Embed Notice

Is there a way to stop search engines crawling a site but still let rich previews work?

My robots.txt was:

User-agent: *
Disallow: /

But when I shared a link on Facebook & other sites, it often wouldn't show the OGP image & description even though the metatags are in there.

I want a site which is hidden from search engines, but still shareable on social networks.

Any tips?