Status update: we’re 47TB through restoring the 55TB db snapshot of the matrix.org DB, but then have to rebuild the DB and replay the subsequent 17h of DB traffic, which will take several hours. Thank you for your patience, and apologies once again for the outage.
Status update: we've restored the 55TB snapshot and subsequent incremental backups, and are about to replay the remaining traffic since the backup. There are still several unknowns, but if things go well the matrix.org instance should be back in 3-4 hours.
So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.
Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.
We’ve now published the details of the security issues which we addressed in Monday’s coordinated security release of Matrix: read all about Project Hydra at https://matrix.org/blog/2025/08/project-hydra-improving-state-res/ and please upgrade your servers and rooms where applicable.
IMPORTANT: We're pre-disclosing two high severity CVEs in Matrix's federation protocol, which will require a coordinated upgrade on July 22nd of all servers running in untrusted federations. Client developers may need to make minor changes too. Please see https://matrix.org/blog/2025/07/security-predisclosure/
The latest invite-spam-blocking technologies, Matrix Community Summit 2025 announced, and the custom emotes MSC has been proposed for final comment period – this and more happened This Week in Matrix! https://matrix.org/blog/2025/05/30/this-week-in-matrix-2025-05-30/
We're pleased to share this report and the minutes from the first formal meeting of our Governing Board! The GB has made great progress in establishing norms, structures, and processes, and has set up a dedicated Matrix room for public participation.
It's the last spec release of 2024! 🎉 Matrix 1.13 has even more Trust & Safety features for you. Account Suspension, new reporting endpoints, fixes to the existing endpoints, and not-T&S features like EDUs to appservices!
Videos from #MatrixConf, usable implementations of Matrix 2.0, a new Matrix Client called Tammy and lots of changes in Fractal! This and more happened this week in #Matrix!
It's official - Matrix 2.0 implementations are here and ready for mainstream use! 🎉🥳🎊 Come watch the keynote from The Matrix Conference and learn about the joy of instant sync ⚡️, next-gen auth 🔐, native VoIP 🎦 and Invisible Crypto 👻: https://matrix.org/blog/2024/10/29/matrix-2.0-is-here
Matrix is an open protocol for decentralised, secure communications. The Matrix.org Foundation exists to act as a neutral custodian for Matrix and to nurture it as efficiently as possible as a single unfragmented standard.📣 We're currently seeking individual and organizational members to fund our work: https://matrix.org/membership