Conversation

Notices

1. Embed this notice
   ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Saturday, 22-Nov-2025 18:31:56 JST ✧✦Catherine✦✧

   bots have finally knocked a grebedoc runner offline (one bar per 10s, so this was 71 uncached req/s at its peak)

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Saturday, 22-Nov-2025 18:33:15 JST ✧✦Catherine✦✧

     in reply to

     oh, no, i'm wrong: all of this traffic had cached responses. i think the cause was something different, like running out of CPU time due to having to decompress them or something like that

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Saturday, 22-Nov-2025 18:39:13 JST ✧✦Catherine✦✧

     in reply to

     after examining the metrics, it looks like they bottomed out %cpu _and_ a 100 Mbps network uplink

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Saturday, 22-Nov-2025 20:41:07 JST ✧✦Catherine✦✧

     in reply to

     • Paul Khuong

     @pkhuong the "almost" is the problem here, some bots (notably facebook) request identity transformation in response to people using zipbombs

     also curl by default doesn't do --compressed which means you'll not just be screwing over bots

   • Embed this notice
     Paul Khuong (pkhuong@discuss.systems)'s status on Saturday, 22-Nov-2025 20:41:08 JST Paul Khuong

     in reply to

     @whitequark for medium sized docs, I've had a hard time justifying better compression algorithms than gzip, since gz encoding is so widely supported you can almost blindly stick verbatim it in your response. zstd soon though.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Saturday, 22-Nov-2025 21:13:06 JST Rich Felker

     in reply to

     • Paul Khuong

     @whitequark @pkhuong I got rid of Facebook's bots by blocking their whole IP range. It was the single biggest reduction to scraper DoS I made.