@blockbot iptables at that point. For stuff like that where someone's just doing the same pattern, I just tail the logs, someone passes a threshold and I drop the traffic. If it's enough to actually flood you, normal usage for that endpoint is about once a minute, so say something like more than ten times in ten seconds you could say is unlikely. I check our logs and in the last week, only two IPs hit 4 times in ten seconds and that only happened six times total, so setting it at ten times in ten seconds is pretty safe. So for your log format, something like this would work:
tail -f /var/log/wherever | mawk -Winteractive ' # Normally you'd just use the $vars but I'm setting them here so the post is easier to read: {ip = $1; timestamp = $4; tenseconds = substr(timestamp, 2, 19); path = $7}
# So, check if they're hitting /api/v1/streaming or I think there's a v3 maybe. path ~ /\/api\/v[0-9]\/streaming/ { # Reasonable key, just cat the fields: ct = a[ip tenseconds]++ # So, ten times in ten seconds from the IP: if(ct > 10) { print "Killing", ip # Or use whatever, ufw or some BSD thing. system("iptables -A INPUT -s " ip " -j DROP") } } '
Here, they were too lazy to even set a UA, so it was probably just firing off nonsense over netcat (basic optimization for flooding: just abandon the connection after the request gets through instead of going through all the overhead of dealing with the HTTP library, which will do stuff like parse the response headers, etc.). Instead of worrying about thresholds, you could just kill anything that hits you more than once in a second and doesn't have a User-Agent set.
One thing I do when testing this kind of script out is to replace "system" with "print" or put "echo" at the beginning of the program it runs, easy way to do a dry run.
Ty. This attack was annoying because it instantly crashed everything running on the server , even my dns. With dns broken, i cant login remotely and have to run to garage and login and diagnose. Cant even use internet to go to cloudflare to stop it
@blockbot You run dnsmasq for the local network? You'll wanna. And by default it only caches 150 entries, you'll want it way bigger, especially for a fedi server.
One kind that I've never seen is a flood of "POST /inbox" with a signature for a pile of servers that don't exist but that do have DNS entries pointing somewhere slow; Revolver accepts provisionally and flags the entry, Mastodon and I *think* Pleroma hold the request open and don't respond until they have actually fetched the key. I haven't looked at how they behave if it's a blocked server, but since most Masto admins don't know what they are doing, I think they wouldn't know what was going on, and maybe it tries to fetch the key even for blocked instances; they go through some pageantry to stop your instance from automatically detecting that it is blocked. Like, if you block an instance on Masto, even if you've never seen it, it will start showing up in your peers (which is how fedilist found a bunch of non-fedi sites).
On the bugout zone, I pretty heavily rate-limit /inbox but I think if you were clever about it, even that wouldn't solve it. Pleroma doesn't bottleneck outbound HTTP requests under most circumstances.
You might actually even have a reflection attack if you were to make a DNS wildcard pointing at your target and then flooding all of fedi with "POST /inbox" and a key URL pointed at the DNS wildcard. Like, it's not super efficient but say you point it at a time-consuming endpoint for the other server (like /api/v1/streaming or the search endpoint or a hashtag; something that returns JSON, though, because a lot of the servers probably close the connection once they get back a header that says it's a video file or whatever without reading the body).
This seems like a big enough problem that someone must have thought of it so either there's something I'm missing or we'll find out when someone decides to make a PoC.
@p@blockbot this is an interesting approach and what's interesting is that the ip in the first snippet [84.239.12.133] actually belongs to a vps company
is this actually an approach they could've taken? is this someone that actually probably is not dumb but is doing dumb shit for some reason
@kirby@blockbot If I leave something out, I keep switching between fedi and acme and I hit "^W" to erase a word and instead it closed the tab, so I am retyping it and sometimes this makes a feller think he has mentioned something but instead, he has said nothing because history has been erased.
> the ip in the first snippet [84.239.12.133] actually belongs to a vps company
Yeah, a lot of VPNs just lease cheap VPSs. It's not Tor, Mullvad, Cloudflare, or ProtonVPN, and those are the only ones I have in my "tell me some shit about this address" script. This one looks like it's from privateinternetaccess.com, which is a paid VPN service.
> is this actually an approach they could've taken?
I don't know. I don't even know if it works. There are a lot of accidental DoSs, it's network code and the network got huge, but the dev teams didn't get more time, and people are always yelling "FEATURES FEATURES" and no one sponsors "Fix the holes" or "Can we make this not absurdly slow" or anything like that.
> is this someone that actually probably is not dumb but is doing dumb shit for some reason
They don't seem dumb, but I don't know why they have decided to try to DoS the pedo instances. It's a moderately clever attack but it's really noisy and it's easy to stop completely and the execution is really lazy.
Pedos have lots of websites, and you can't really remove someone from a decentralized network. I can understand why that might happen if the pedos were doing cop-bait shit and the admin was going out of his way to agitate, but they went after DRC, which is not a pedo instance (maybe confused DRC's Alice with someone on one of the other instances, there are a few people named that on those servers), so it can't be a response to anything they were actively doing, or this person would have started with whoever was actually causing trouble. Bored kid got a wild hair up his ass, no idea. babyfuck.png
This just means really heavy traffic; at the level of the kernel's routing table, it can be difficult to tell the difference between high traffic and a syn flood. ytcracker--strictly_for_my_streamers--05_terminal.mp3
:mgsgb_1::mycomputer::ddr_r::hal9000: SYN :mgsgb_2::hal9000::ddr_r::mycomputer: ACK/SYN :mgsgb_3::mycomputer::ddr_r::hal9000: ACK
After step 3, the connection is established. Sending a SYN flood looks a lot like creating a large number of incoming connections, because it's just repeating step 1 and ignoring the responses in step 2. The from address can be forged, so how do you tell the difference? The syncookie thing is basically embedding some private data in a hash in the TCP sequence number so that the server can skip the connection overhead: when step 3 happens, it validates the hash and then it can assume the connection is really open.
Since actual requests were coming in, it wasn't a syn flood: you can't send the "GET /api/v1/streaming HTTP/1.1\r\n" unless you've finished the handshake.
@blockbot@p@kirby lmao imouto managed to "protect themselves" from the ddos attacks by heavily limiting network traffic but it ended up causing everyone to deal with connectivity issues and a ton of people were unable to access their instance because of their poor mitigations 😭
@blockbot@85f02109a28864e4713d3f972ab66a@kirby Well, DoS unless there were more IPs. But I can see what was happening, and rate-limiting the endpoint and capping the number of open connections per host fixes it even before just killing off the VPN's IPs.
Let me give you an idea of the scale. To test out the software I am working on, I exploited a different bug to DDoS myself: https://git.pleroma.social/pleroma/pleroma/-/issues/2765 . We had a sustained 800+r/s, and that's successful requests, right? I turned off the rate-limiting on /objects for that. Just compulsive posting in the hellthread. 800r/s isn't that amazing, but it's way bigger than 14.22 I don't know how many it would have taken to break it, because I couldn't get up the other instances to fetch posts any faster.
People have hit the streaming endpoint before. We got aggressively scraped by several places, boardreader.com was feeding it to the FBI, we got DDoS'd by a DDoS mitigation company, we got flooded by Gab. Short of saturating the pipe (please don't, that *is* a pain), FSE is really hard to knock down. If you find a bug and *can* knock it down, feel free. Your existing strategy does not work here, but there are lots of holes in fedi, I'm sure you can find another one. koddos.png