Conversation

Notices

1. Embed this notice
   daniel:// stenberg:// (bagder@mastodon.social)'s status on Wednesday, 26-Mar-2025 16:43:40 JST daniel:// stenberg://

   The AI bots that desperately need OSS for code training, are now slowly killing OSS by overloading every site.

   The curl website is now at 77TB/month, or 8GB every five minutes.

   https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/

   • snacks and pistolero like this.
   • Blaise Pabón - controlpl4n3 repeated this.
   • Embed this notice
     Gerard Braad (gbraad@mastodon.social)'s status on Wednesday, 26-Mar-2025 16:44:22 JST Gerard Braad

     in reply to

     @bagder What is the use of them hammering the website over and over again. They do the same for the Fedora wiki... It is not like they need be near real-time.

     Are you considering an IP block ?

     kaia likes this.
   • Embed this notice
     daniel:// stenberg:// (bagder@mastodon.social)'s status on Wednesday, 26-Mar-2025 16:44:28 JST daniel:// stenberg://

     in reply to

     • Gerard Braad

     @gbraad we specifically don't have logs so I can't tell exactly where they come from, but I read others' analyses of the problem and from what I hear they are quite hard to block properly. We are fortunate to have Fastly that hosts the site and thus is the one that handles the onslaught

     kaia likes this.
   • Embed this notice
     daniel:// stenberg:// (bagder@mastodon.social)'s status on Wednesday, 26-Mar-2025 16:44:35 JST daniel:// stenberg://

     in reply to

     I think users (like GitHub/MS and friends) have a responsibility to push back on the AI companies they lean so heavily on and demand they behave. But I have no expectation they will.

     kaia likes this.
   • Embed this notice
     daniel:// stenberg:// (bagder@mastodon.social)'s status on Wednesday, 26-Mar-2025 16:44:44 JST daniel:// stenberg://

     in reply to

     • SkaveRat 🐀 :verified:

     @skaverat three years ago we were at less than 20GB/month, but there is no clear cut-off date nor do I know exactly what amount of this traffic that is AI bots and not

     Haelwenn /элвэн/ :triskell: and kaia like this.
   • Embed this notice
     SkaveRat 🐀 :verified: (skaverat@skaverat.net)'s status on Wednesday, 26-Mar-2025 16:44:46 JST SkaveRat 🐀 :verified:

     in reply to

     @bagder what was the traffic before that?

   • Embed this notice
     Lord Caramac the Clueless, KSC (lordcaramac@discordian.social)'s status on Wednesday, 26-Mar-2025 23:35:39 JST Lord Caramac the Clueless, KSC

     in reply to

     @bagder This is what happens when there is competition where there should be cooperation. AI research and development could be, _should_ be a collaborative project, not owned by anybody and open to everybody, but instead it's a bunch of corporations trying to outrun each other.
     The Tragedy of the Commons only exists when there is competition instead of cooperation. Competition is how we ruin everything by trying to grab it all before anybody else does. Cooperation is how we can give everybody whatever they need for free and still have enough for all of us.
     Why train so many machine learning models that aren't all that different, which are owned and run by private enterprises, when we could instead train much fewer models that aren't owned by anybody and can be used for free?

     prettygood likes this.