Conversation

Notices

1. Embed this notice
   Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 10-Jan-2024 19:04:59 JST Drew DeVault

   How's your morning going

   https://status.sr.ht/issues/2023-01-10-network-outage/

   We're working on the issue but there's not a ton we can do until we get the (sleepy) nocs on the phone

   • Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 10-Jan-2024 19:05:42 JST Drew DeVault

     in reply to

     Prognosis is not great, we cannot reach our datacenter provider's emergency line and their office-hours support line doesn't open for another 4 hours. Hang tight

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 10-Jan-2024 23:19:04 JST Drew DeVault

     in reply to

     Got in touch with the NOC and they're investigating the problem.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 00:10:29 JST Drew DeVault

     in reply to

     Update from NOC: it's all hands on deck here, that loud noise you hear in the background is my supervisor yelling at Cogent on the phone, but no ETA yet

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 02:16:01 JST Drew DeVault

     in reply to

     We got another update, the root cause has been identified and there's a plan of action to address it, but, again... needs some more time.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Peter Bindels (dascandy42@mastodon.social)'s status on Thursday, 11-Jan-2024 02:19:57 JST Peter Bindels

     in reply to

     @drewdevault wait, it took *FOUR HOURS* to get the NOC to respond?

     Might as well not have one then.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 02:19:57 JST Drew DeVault

     in reply to

     • Peter Bindels

     @dascandy42 they went through an acquisition recently and along the way did not think to notify us that the old ticketing system was shut down. We got the details on the new ticketing system so hopefully this won't happen again.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 19:21:58 JST Drew DeVault

     in reply to

     Getting there

     Haelwenn /элвэн/ :triskell: repeated this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 19:21:58 JST Drew DeVault

     in reply to

     (08:48 UTC — Jan 11) We are preparing a migration. We do not see a near-term resolution being possible with our upstream network provider and we are preparing to restore service from backups in a new installation. It will take a while, but we are all-hands to address the issue.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 22:30:58 JST Drew DeVault

     in reply to

     (13:19 UTC — Jan 11) PHL network restored. Cogent removed the black hole a few minutes ago, restoring access to our PHL installation. However, we are partway through our migration and have placed PHL into a read-only configuration for the time being. Further updates to come.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Thursday, 11-Jan-2024 22:30:59 JST Drew DeVault

     in reply to

     (11:32 UTC — Jan 11) We are partway through setting up a new installation in one of our secondary datacenters. We do not have an ETA but work is proceeding apace.

     Haelwenn /элвэн/ :triskell: repeated this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 02:54:54 JST Drew DeVault

     in reply to

     Still here, still working on it. Long day.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 02:54:55 JST Drew DeVault

     in reply to

     (13:43 UTC — Jan 11) PHL service unreliable. The network to our PHL installation is unreliable. When available, it is configured in a read-only mode. We are still proceeding with the migration for the time being.

   • Embed this notice
     Codeberg.org (codeberg@social.anoxinon.de)'s status on Friday, 12-Jan-2024 05:20:17 JST Codeberg.org

     in reply to

     @drewdevault Thank you. It started immediately after sending the message. 😞

     Sorry for not serving your status page right now

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 05:20:18 JST Drew DeVault

     in reply to

     • Codeberg.org

     @Codeberg looks like you're having some problems of your own. Good luck. Reach out if you need to coordinate anything

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 05:20:20 JST Drew DeVault

     in reply to

     (18:49 UTC — Jan 11) The migration is proceeding. We are making good progress on the migration. We have a staging environment with production data (with no loss of user data) working with for the following services[...]

     Full update here: https://status.sr.ht/issues/2023-01-10-network-outage/

   • Embed this notice
     Codeberg.org (codeberg@social.anoxinon.de)'s status on Friday, 12-Jan-2024 05:20:20 JST Codeberg.org

     in reply to

     @drewdevault We keep our fingers crossed and wish you no further incidents on the way. All the best from our team!

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 05:21:49 JST Drew DeVault

     in reply to

     Our status page is offline due to a DDoS now targetting Codeberg, who hosts our status page on third-party infrastructure for us. Will continue posting updates on Mastodon as they come.

     Haelwenn /элвэн/ :triskell: likes this.