Conversation

Notices

1. Embed this notice
   The Matrix.org Foundation (matrix@mastodon.matrix.org)'s status on Wednesday, 03-Sep-2025 06:44:09 JST The Matrix.org Foundation

   the matrix.org homeserver is having problems: https://status.matrix.org/incidents/mm9hdm78svgv apologies for the inconvenience…

   • Embed this notice
     The Matrix.org Foundation (matrix@mastodon.matrix.org)'s status on Wednesday, 03-Sep-2025 06:44:06 JST The Matrix.org Foundation

     in reply to

     Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.

     Haelwenn /элвэн/ :triskell: and Johnny Peligro like this.
   • Embed this notice
     The Matrix.org Foundation (matrix@mastodon.matrix.org)'s status on Wednesday, 03-Sep-2025 06:44:07 JST The Matrix.org Foundation

     in reply to

     So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.

     /mastodon/gutocarvalho and Gianmarco Gargiulo repeated this.
   • Embed this notice
     The Matrix.org Foundation (matrix@mastodon.matrix.org)'s status on Wednesday, 03-Sep-2025 20:58:13 JST The Matrix.org Foundation

     in reply to

     Status update: we've restored the 55TB snapshot and subsequent incremental backups, and are about to replay the remaining traffic since the backup. There are still several unknowns, but if things go well the matrix.org instance should be back in 3-4 hours.

   • Embed this notice
     The Matrix.org Foundation (matrix@mastodon.matrix.org)'s status on Wednesday, 03-Sep-2025 20:58:14 JST The Matrix.org Foundation

     in reply to

     Status update: we’re 47TB through restoring the 55TB db snapshot of the matrix.org DB, but then have to rebuild the DB and replay the subsequent 17h of DB traffic, which will take several hours. Thank you for your patience, and apologies once again for the outage.

     13 barn owls in a trenchcoat repeated this.