Codeberg.org
We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly.
The good news: We're keeping up with the additional load of new users moving to Codeberg. Welcome aboard, we're happy to have you here. After adjusting the AI crawler protections, performance significantly improved again.
翠星石
Codeberg.org
We have a list of explicitly blocked IP ranges. However, a configuration oversight on our part only blocked these ranges on the "normal" routes. The "anubis-protected" routes didn't consider the challenge. It was not a problem while Anubis also protected from the crawlers on the other routes.
However, now that they managed to break through Anubis, there was nothing stopping these armies.
It took us a while to identify and fix the config issue, but we're safe again (for now).
Codeberg.org
However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.
Codeberg.org
It seems like the AI crawlers learned how to solve the Anubis challenges. Anubis is a tool hosted on our infrastructure that requires browsers to do some heavy computation before accessing Codeberg again. It really saved us tons of nerves over the past months, because it saved us from manually maintaining blocklists to having a working detection for "real browsers" and "AI crawlers".
翠星石
Codeberg.org
@Suiseiseki BTW, we're also actively following the work around iocaine, e.g. https://come-from.mad-scientist.club/@algernon/statuses/01K2N54XEVTEYYAASHZ0P48FBT
However, as far as we can see, it does not sufficiently protect from crawling. As the bot armies successfully spread over many servers and addresses, damaging one of them doesn't prevent the next one from doing harmful requests, unfortunately. ~f
Codeberg.org
@Suiseiseki Anubis is the option that saved us a lot of work over the past months. We are not happy about it being open core or using GitHub sponsors, but we acknowledge the position from the maintainer: https://codeberg.org/forgejo/discussions/issues/319#issuecomment-6382369
Calling our usage of anubis an attack on our users is far-fetched. But feel free to move elsewhere, or host an alternative without resorting to extreme measures. We're happy to see working proof that any other protection can be scaled up to the level of Codeberg. ~f
Rich Felker
@Codeberg If some of the attack is coming from Huawei's cloud hosting, it might be worth sending a complaint to their abuse department. IME Chinese companies tend to be scared of breaking rules in international dealings like this.
SuperDicq
@Codeberg@social.anoxinon.de @Suiseiseki@freesoftwareextremist.com A lot of users can not pass Anubis challenges because Anubis does not support every browser and is also incompatible with popular security focussed browser extensions such as JShelter.
Asking your users to enable JavaScript and to disable security extensions like JShelter in order to visit your website is very bad, don't you agree?
I don't think it is far-fetched to call it an attack on your users at all.
翠星石
Stefano Zacchiroli
@Codeberg so, to clarify, do you have evidence that the bots were solving Anubis challenges or not, i.e., it was due to the configuration issue? (I think it's inevitably going to happen if Anubis gets traction. I'm just curious if we're already there or not.) Thanks for your work and transparency on all this.
Codeberg.org
For the load average auction, we offer these numbers from one of our physical servers. Who can offer more?
(It was not the "wildest" moment, but the only for which we have a screenshot)
Phantasm
翠星石
翠星石
Noisytoot
browsers do not offer an interface that allow for that
That sounds like an issue with those browsers, and does not make Anubis proprietary. You could make a more user-freedom-respecting browser. If there was a program that automatically downloaded, say, GIMP, and ran it on your computer, would the existence of that program make GIMP proprietary?
翠星石
Noisytoot
翠星石
翠星石
Marcos Dione
@Suiseiseki @phnt @Codeberg "infidel"? What are you, a pantomime religious extremist? Or an IA? :-P
RedTechEngineer
RedTechEngineer
翠星石
RedTechEngineer
Phantasm
Phantasm
翠星石
翠星石
翠星石
Codeberg.org
@zacchiro Yes, the crawlers completed the challenges. We tried to verify if they are sharing the same cookie value across machines, but that doesn't seem to be the case.
翠星石
翠星石
Dan Jones
Pardon my ignorance, but couldn't they just be using a headless browser, which would still do everything a regular browser does? Just recently, ChatGPT beat Cloudflare's CAPTCHA using a similar system. Is there really any way around this at all? @Codeberg@social.anoxinon.de
daltux
To-BOO-as HELL-gren
@Codeberg Perhaps it's time stop letting robots solve puzzles and instead feed them bombs. Do we know how well a ZIP bomb works on these crawlers?
daltux
To-BOO-as HELL-gren
翠星石
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.