Notices by Codeberg (codeberg@social.anoxinon.de)

@gsuberland

#39c3

Hey everyone, you can now follow @codebergstatus to get more granular notifications about our current system. We decided to keep the main account clear from the noise of smaller issues and allow you to explicitly opt-in to such notifications.

Important and positive news will continue to be shared on this account 😉

Once again, we are fighting with an ongoing DoS attack. We are very sorry for the disruption of the service and hope to bring the service back online soon.

We are currently fighting against a DDoS attack against our service and our status page. We are analyzing network traffic with the help of our ISP at the moment and let you know once we have updates to share.

Every couple of months, a couple of us come together to write down what we've been working on behind the scenes. The last time we did that was June. There are many exciting things (for us, at least!) that have happened since; you can read all about them in the update that we are sharing with you today:

https://blog.codeberg.org/letter-from-codeberg-onwards-and-upwards.html

And since we're sharing background: Codeberg has a working group for matters involving Public Relations—some of us are deeply involved with Codeberg's day-to-day operations, others of us have experience in putting words together for projects and group efforts like ours. (Teamwork makes the dream work!)

We did spend a lot of time on this update (and went over many details, which is why we're posting on here so late 🌃).

Hey everyone! Codeberg will go down for about 10 minutes starting 19.00 UTC (20.00 CET) to create database indices which are expected to improve performance. This requires a short interruption of service.

We apologize for the inconvenience and short notice.

There will be several short interruptions in service during the next 24 hours. Each should not last more than a few minutes.

We are performing scheduled hardware maintenance, replacing 1 of 3 physical servers. You can find the current status at https://status.codeberg.eu/status/codeberg

Great news: Today, we have managed to distribute our Forgejo instance across two servers. A main instance still handles most traffic, but requests from #CodebergPages are now handled by another server. Going forward, we'll try to route more read-only traffic there to distribute the load.

This is an important milestone, because such a setup was never tested with #Forgejo at this scale.

This should improve availability of both Codeberg.org and Codeberg Pages.

Details: https://codeberg.org/forgejo/discussions/issues/259

We apologize for the long performance degradation today.
Finally, we identified all of the 'tricks' that AI crawlers found today. They no longer bypass the anubis proof of work challenges.

A novelty for us was that AI crawlers seem to not only crawl URLs that are actually presented to them by our frontend, but they converted the URLs into a format that bypassed our filter rules.

By the way, you can track the changes we have been doing via

https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/compare/51618~1..e4aac

We have upgraded to Forgejo v13. You can read the official announcement for the release here: https://forgejo.org/2025-10-release-v13-0/

If you have unexpected issues, please don't hesitate to let us know. https://codeberg.org/Codeberg/Community/issues/

Two of our three servers decided to restart at around 04:00 CEST today. This caused Codeberg to be offline for several hours. We have restarted the servers and all services should be working again. We are currently investigating why these servers decided to restart themselves.

If not now, then when?

https://www.theregister.com/2025/09/05/github_copilot_complaints/

If you use Git via HTTPs and get rate-limited now, please provide feedback to us about your usage. Also, feel free to use SSH or extend your usage of caching, if possible.

Today, we're struggling under the load of excessive Git cloning of all repos, and we have made the hard decision to restrict these operations for all users.

Your code is free and libre, but not for sale to large actors.

@thesamesam Unfortunately, I'm not sure if encouraging anyone to reinforce the vendor-lock-in of Microsoft GitHub by making maintainers financially dependent on that platform, is in spirit with our mission. ~f

@zacchiro Yes, the crawlers completed the challenges. We tried to verify if they are sharing the same cookie value across machines, but that doesn't seem to be the case.

For the load average auction, we offer these numbers from one of our physical servers. Who can offer more?

(It was not the "wildest" moment, but the only for which we have a screenshot)

@Suiseiseki Anubis is the option that saved us a lot of work over the past months. We are not happy about it being open core or using GitHub sponsors, but we acknowledge the position from the maintainer: https://codeberg.org/forgejo/discussions/issues/319#issuecomment-6382369

Calling our usage of anubis an attack on our users is far-fetched. But feel free to move elsewhere, or host an alternative without resorting to extreme measures. We're happy to see working proof that any other protection can be scaled up to the level of Codeberg. ~f

@Suiseiseki BTW, we're also actively following the work around iocaine, e.g. https://come-from.mad-scientist.club/@algernon/statuses/01K2N54XEVTEYYAASHZ0P48FBT

However, as far as we can see, it does not sufficiently protect from crawling. As the bot armies successfully spread over many servers and addresses, damaging one of them doesn't prevent the next one from doing harmful requests, unfortunately. ~f

@gturri Anubis sends a challenge. The browser needs to compute the answer with "heavy" work. The server then has "light" work and verifies the challenge.

As far as we can tell, the crawlers actually do the computation and send the correct response. ~f