Conversation

Notices

1. Embed this notice
   Tim Chambers (tchambers@indieweb.social)'s status on Friday, 14-Jun-2024 06:27:04 JST Tim Chambers

   #Admin #IndiewebSocial - Looking for the best list of AI scraper bots to block on this server. Aany others?

   User-agent: GPTBot
   User-agent: ChatGPT-User
   User-agent: Google-Extended
   User-agent: PerplexityBot
   User-agent: Amazonbot
   User-agent: ClaudeBot
   User-agent: Omgilibot
   User-Agent: FacebookBot
   User-Agent: Applebot
   User-agent: anthropic-ai
   User-agent: Bytespider
   User-agent: Claude-Web
   User-agent: Diffbot
   User-agent: ImagesiftBot
   User-agent: Omgilibot
   User-agent: Omgili
   User-agent: YouBot

   • Embed this notice
     Tim Chambers (tchambers@indieweb.social)'s status on Friday, 14-Jun-2024 07:09:55 JST Tim Chambers

     in reply to

     • Bruce Davie

     @Drbruced 🙏 Thank you!

   • Embed this notice
     Bruce Davie (drbruced@aus.social)'s status on Friday, 14-Jun-2024 07:09:56 JST Bruce Davie

     in reply to

     @tchambers I think you want Applebot-Extended as well. This repo might be of interest https://github.com/ai-robots-txt/ai.robots.txt

   • Embed this notice
     Tim Chambers (tchambers@indieweb.social)'s status on Friday, 14-Jun-2024 09:45:32 JST Tim Chambers

     • ken Tucky Swinson

     @kenSwinson You are very welcome: more news on this soon!

   • Embed this notice
     Tim Chambers (tchambers@indieweb.social)'s status on Friday, 14-Jun-2024 22:55:08 JST Tim Chambers

     in reply to

     • Kay Ohtie

     @KayOhtie A good note! Taking it…

   • Embed this notice
     Kay Ohtie (kayohtie@blimps.xyz)'s status on Friday, 14-Jun-2024 22:55:22 JST Kay Ohtie

     in reply to

     @tchambers this may have been corrected but the apple one is Applebot-extended; the short one one just catches the normal quick indexer for previews. Fine if you're pattern matching via user agent to block but might not pass in robots.txt I think?