Conversation

Notices

1. Embed this notice
   Jan Wildeboer 😷:krulorange: (jwildeboer@social.wildeboer.net)'s status on Tuesday, 30-Jun-2026 03:00:58 JST Jan Wildeboer 😷:krulorange:

   My timeline (which contains a lot of project leaders/sysadmins from big projects) is filling with posts about a new, ongoing wave of what most likely are scrapers collecting training data for „AI“ companies. They seem to be using botnets (or what some call „residential IP proxies“ to make it sound a bit more legitimate) with millions of IP addresses, making it really hard to defend against. Some have decided to take their sites down until this is over. This is now the world we live in :(

   • Embed this notice
     🆘Bill Cole 🇺🇦 (grumpybozo@toad.social)'s status on Tuesday, 30-Jun-2026 03:35:22 JST 🆘Bill Cole 🇺🇦

     in reply to

     • al

     @alan @jwildeboer The attacks have thwarted all of those tactics. They use UAs constructed from real UA tokens with minor variations. They have graduated from cheap VMs on Huawei Cloud and Digital Ocean to random IoTs in millions of households and mobile devices in millions of hands.
     A few days ago I was able to measure over a thousand simultaneous sessions, each from a different /16 network.
     My response to that isn’t taking the site down, but I am shedding load aggressively.

   • Embed this notice
     al (alan@lighthouse.co.im)'s status on Tuesday, 30-Jun-2026 03:35:27 JST al

     in reply to

     @jwildeboer It's weird that the response is to take sites down rather than reach for technical countermeasures -- rate limiting, UA filtering, datacenter ASN blocks. Is the residential proxy problem genuinely that hard to solve at scale, or is the downtime itself the point? A visible protest signal rather than a quiet WAF tweak feels like a different kind of statement about where people think the leverage actually is?

     Rich Felker repeated this.
   • Embed this notice
     al (alan@lighthouse.co.im)'s status on Tuesday, 30-Jun-2026 03:36:10 JST al

     in reply to

     @jwildeboer 2/2
     The deeper problem is upstream. Apple, Google and Microsoft are allowing SDK-injected bandwidth harvesting through their app stores. Until that's addressed at source, we're all playing whack-a-mole with an essentially infinite residential IP pool. This isn't a mail security problem -- it's a platform accountability problem.

   • Embed this notice
     al (alan@lighthouse.co.im)'s status on Tuesday, 30-Jun-2026 03:36:11 JST al

     in reply to

     @jwildeboer 1/2
     Solidarity -- you're not alone in this. The one-attempt-per-IP pattern is specifically designed to be invisible to anything threshold-based. CrowdSec helps at the edges but a fresh residential IP making a single SASL attempt looks like a legitimate user having a bad day. Your manual cronjob approach is the right call. Automation just gives you false confidence.

     Rich Felker repeated this.
   • Embed this notice
     Jan Wildeboer 😷:krulorange: (jwildeboer@social.wildeboer.net)'s status on Tuesday, 30-Jun-2026 03:36:12 JST Jan Wildeboer 😷:krulorange:

     in reply to

     • al

     @alan These botnets are more or less immune to rate limiting, as they use many (and I mean millions) of IP addresses fro a run and each IP address is only used for a few requests before it is being put back in the queue. The IP addresses are also from many different providers, so a (sub-)net wide block also doesn't help. I wrote about those "residential IP proxies in [1] and [2].

     [1] https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
     [2] https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 30-Jun-2026 03:36:59 JST Rich Felker

     in reply to

     • al

     @alan @jwildeboer Yes, it is entirely Apple's and Google's fault that they are hosting botnet malware in their "walled gardens" as legitimate and vetted software. Without that, the botnets would not exist on any viable scale.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 30-Jun-2026 03:39:44 JST Rich Felker

     in reply to

     • Tim Ward ⭐🇪🇺🔶 #FBPE
     • Rupert V/

     @rupert @TimWardCam @jwildeboer Exactly. This is *ideological* - they deem actually-engineered solutions that do things efficiently as a backwards "dirty human" way of doing things. Obviously since their AI slop is superior, they should do the scraping in whatever way the AI slop vomits out code to do it.

   • Embed this notice
     Rupert V/ (rupert@mastodon.nz)'s status on Tuesday, 30-Jun-2026 03:39:45 JST Rupert V/

     in reply to

     • Tim Ward ⭐🇪🇺🔶 #FBPE

     @TimWardCam @jwildeboer Because their trawler is as sloppily coded as everything else they do.

   • Embed this notice
     Tim Ward ⭐🇪🇺🔶 #FBPE (timwardcam@c.im)'s status on Tuesday, 30-Jun-2026 03:39:46 JST Tim Ward ⭐🇪🇺🔶 #FBPE

     in reply to

     @jwildeboer Why would an AI company need millions of copies of the same data?