Conversation

Notices

1. Embed this notice
   anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 03:13:39 JST anime graf mays ?️?
   just blocked the entirety of hong kong from poast nitter due to somebody using 17 individual /24s (256 IPs * 17) to scrape absolutely everything from veechubas to literal nobodies with 1-2 followers
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 03:13:32 JST anime graf mays ?️?

     in reply to

     • We're Going to Win.
     @pirin04 im fine i dont need to be the best at anything, i just want to be happy with my friends
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 03:13:36 JST anime graf mays ?️?

     in reply to

     • We're Going to Win.
     @pirin04 this has nothing to do with poast actually
   • Embed this notice
     We're Going to Win. (pirin04@poa.st)'s status on Friday, 08-Mar-2024 03:13:36 JST We're Going to Win.

     in reply to
     @graf you're a terrible opportunist...the grift has skipped you
     BowserNoodle ☦️ repeated this.
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 03:13:37 JST anime graf mays ?️?

     in reply to
     got rid of all of them. 7800 requests and 3700 requests per second respectively on both servers which is average traffic. good job poast we defeated the chicom enemy
   • Embed this notice
     We're Going to Win. (pirin04@poa.st)'s status on Friday, 08-Mar-2024 03:13:37 JST We're Going to Win.

     in reply to
     @graf does this mean youre going full torba and start charging us a dollar per jpeg?
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 03:13:38 JST anime graf mays ?️?

     in reply to
     so there's actually way more. 146 /24s and counting so far. somebody really wants this data. thats a fuck of a lot of $$$ for that many IPs. still going about 65k requests/s but im trimming them out
   • Embed this notice
     UnityOstara (unityostara@poa.st)'s status on Friday, 08-Mar-2024 03:14:01 JST UnityOstara

     in reply to

     • We're Going to Win.
     @graf @pirin04 Saying stupid shit like that is why you're one of my favorite people!
     BowserNoodle ☦️ likes this.
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 04:06:49 JST anime graf mays ?️?

     in reply to

     • pwm
     • einmad
     @einmad @pwm they dont hammer fast. it's 10000s of IPs requesting accounts crawling via mentions/reposts 1-2 requests every maybe 30-45 seconds sometimes up to a minute or two. but because there are so many of them it flies under the radar because it doesn't appear as a traffic anomaly
     
     i could use something like that Nginx-LUA-Anti-DDoS maybe. I will look into it github.com/C0nw0nk/Nginx-Lua-Anti-DDoS/blob/master/lua/anti_ddos_challenge.lua
     
     the challenge might be enough. ideally I'd like to redirect traffic to a separate server set up as a tarpit so it keeps the request open forever just to see what it would do
   • Embed this notice
     BowserNoodle ☦️ (bowsacnoodle@poa.st)'s status on Friday, 08-Mar-2024 04:06:49 JST BowserNoodle ☦️

     in reply to

     • pwm
     • einmad
     @graf @einmad @pwm Is there any way to split the Nitter and limit IPs to a certain number of requests per hour without an oauth? I don't understand enough about this stuff to know if that's a dumb idea or not.
   • Embed this notice
     einmad (einmad@poa.st)'s status on Friday, 08-Mar-2024 04:06:50 JST einmad

     in reply to

     • pwm
     @graf @pwm can't you just put a small PoW in front of whatever is being scraped? If a session that passed the PoW starts hammering too fast, they get it again.
   • Embed this notice
     pwm (pwm@crlf.ninja)'s status on Friday, 08-Mar-2024 04:06:51 JST pwm

     in reply to
     @graf For a big thing like this it will probably only sorta help, but I find especially for my email server that you stop being low hanging fruit for those not throwing around cash to rent residential proxies
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 04:06:51 JST anime graf mays ?️?

     in reply to

     • pwm
     @pwm yeah how im doing this definitely isn't sustainable. i'm evaluating ways i can keep it public but also limit access. i wish it was as easy of just setting up an oauth login for poast users but others use it too. even people not on fedi and i feel bad cutting them off to deal with some shitty people. blocking entire ranges is fine for now but people get caught in the crossfire (and iptables/nginx have limits to the amount of IPs they store 'in memory')
     
     this weekend I'm going to tear everything out as far as blocks and stuff are concerned and do it a little more elegantly. it's a lot of work but hopefully in the long run it'll be manageable enough
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 08-Mar-2024 04:06:52 JST anime graf mays ?️?

     in reply to

     • pwm
     @pwm this one actually was also using US residential proxies, a couple US based datacenter ranges.
     
     I wonder if its a mix of servers with /24 (you can rent them for like 150-200$ they call them "SEO servers") and purchased proxies or what but there's a lot of money at play here for that large of a coordinated scrape. 200k requests per second at its peak is crazy. per server
   • Embed this notice
     pwm (pwm@crlf.ninja)'s status on Friday, 08-Mar-2024 04:06:53 JST pwm

     in reply to
     @graf I am never kidding when I tell people this, just drop the entirety of APNIC at the firewall
     There are 0 (zero) negative consequences.
     Chinks gooks and abos out