Conversation

Notices

1. Embed this notice
   evana (evana@hachyderm.io)'s status on Tuesday, 06-Dec-2022 00:40:34 JST evana

   in reply to

   • Kris Nóva

   @nova thanks for the writeup!

   Sitting in my armchair, I'm wondering about the end cause being bad hardware, mostly because I had a slightly-related failure on my own home setup.

   My guess is there was I/O saturation on the largest SSD (looks like they were SATA) due to capacity imbalance. When the largest SSD hit the limit, it could block kernel queues across the system due to the shared ZFS layer.

   I'm wondering if hard partitioning postgres and NFS could have protected postgres.

   • Embed this notice
     Kris Nóva (nova@hachyderm.io)'s status on Tuesday, 06-Dec-2022 00:40:36 JST Kris Nóva

     Sunday morning coffee and pancakes with the family. I decided to put the outage report on hacker news in the hopes of sharing with other admins and raising up the operator mindshare for the entire fediverse.

     Let’s see how long it can stay positive and productive?

     Post Mortem on Mastodon Outage with 30k users https://news.ycombinator.com/item?id=33855250