Conversation

Notices

1. Embed this notice
   Terence Eden (edent@mastodon.social)'s status on Wednesday, 29-Jan-2025 18:22:00 JST Terence Eden

   Is there a way to stop search engines crawling a site but still let rich previews work?

   My robots.txt was:

   User-agent: *
   Disallow: /

   But when I shared a link on Facebook & other sites, it often wouldn't show the OGP image & description even though the metatags are in there.

   I want a site which is hidden from search engines, but still shareable on social networks.

   Any tips?

   • Embed this notice
     Eugen Rochko (gargron@mastodon.social)'s status on Wednesday, 29-Jan-2025 18:22:00 JST Eugen Rochko

     in reply to

     • かき@GNUsocialJP

     @Edent robots.txt is the wrong approach. It can stop crawling but not indexing. You want the noindex meta tag in your HTML. In fact, if you have the noindex tag but your robots.txt prevents e.g. Google from crawling your page it will not pick it up and still serve your pages from the index.

   • Embed this notice
     Terence Eden (edent@mastodon.social)'s status on Wednesday, 29-Jan-2025 18:28:44 JST Terence Eden

     in reply to

     • Eugen Rochko

     @Gargron ah, that makes sense. Ta.