Conversation

Notices

1. Embed this notice
   Romain Tartière 😈 (smortex@mamot.fr)'s status on Friday, 27-Mar-2026 15:50:41 JST Romain Tartière 😈

   Today, #facebook bots (Meta-ExternalAgent, you know, the one to train AI) took 21 days of wallclock time and 19.9 GB of bandwidth to index the development version of one of our websites for which crawling makes no sense and is explicitly forbidden by robots.txt.

   Maybe it is a signal that we must spend some of _our_ time to make them lose more of _them's_. Any creative idea?

   • Embed this notice
     Aral Balkan (aral@mastodon.ar.al)'s status on Friday, 27-Mar-2026 15:55:52 JST Aral Balkan

     in reply to

     @smortex Don’t know the names off the top of my head but there are ways to make them go into infinitely deep content holes.

   • Embed this notice
     Aral Balkan (aral@mastodon.ar.al)'s status on Saturday, 28-Mar-2026 16:52:20 JST Aral Balkan

     in reply to

     • Davide_Sandini
     • flyinggecko

     @flyinggecko @Davide_Sandini @smortex @algernon Thank you :)

   • Embed this notice
     flyinggecko (flyinggecko@digitalcourage.social)'s status on Saturday, 28-Mar-2026 16:52:22 JST flyinggecko

     in reply to

     • Aral Balkan
     • Davide_Sandini

     @Davide_Sandini @aral @smortex Maybe this? https://iocaine.madhouse-project.org/
     It uses a relative small amount of resources for a lot of work for crawlers.
     Edit: And it's fun following the development toots of @algernon

   • Embed this notice
     Davide_Sandini (davide_sandini@mastodon.uno)'s status on Saturday, 28-Mar-2026 16:52:23 JST Davide_Sandini

     in reply to

     • Aral Balkan

     @aral
     I do not remeber who had a similar project, but for sure it was cited here on mastodon.
     @smortex