Conversation

Notices

1. Embed this notice
   OliverUv (oliveruv@mastodon.social)'s status on Tuesday, 21-Jan-2025 10:27:53 JST OliverUv

   • alcinnz

   @alcinnz any search engine worth its salt honors robots.txt so using that you can protect them. These tools were specifically made to punish those who do not honor robots.txt

   As far as I'm aware, google still honor it

   • Embed this notice
     alcinnz (alcinnz@floss.social)'s status on Tuesday, 21-Jan-2025 10:27:53 JST alcinnz

     in reply to

     @OliverUv The issue as I understand it is that any bot would see the Nepenthes tarpit, even will behaved ones. So unless the crawler's limiting how much of your site it visits...

     Or we could ask well-behaved bots not to crawl the tarpit, I'm guessing the misbehaved ones won't honor that hint...