Conversation

Notices

1. Embed this notice
   Drew DeVault (drewdevault@fosstodon.org)'s status on Friday, 12-Jan-2024 21:03:10 JST Drew DeVault

   Statement regarding the ongoing SourceHut outage

   https://outage.sr.ht

   • Embed this notice
     Gokul Das (goku12@fosstodon.org)'s status on Friday, 12-Jan-2024 21:04:51 JST Gokul Das

     in reply to

     • Codeberg.org

     @drewdevault I'm a paying customer and this may be slightly annoying. But I have the patience to wait and am not going anywhere. I understand how much worse this is for you. Please don't forget to take care of yourself while you deal with this. The same goes for the @Codeberg guys.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 00:31:14 JST Drew DeVault

     in reply to

     We're fairly close to getting some services online

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:42:36 JST Drew DeVault

     in reply to

     We have the following services working in read-only mode:

     meta
     project hub
     git*
     lists
     todo
     paste

     Your access to these is subject to DNS propagation.

     * still restoring backup, not all repos available. We also fixed an issue since the last update with git fetch/clone

     We will have man.sr.ht online tonight, and then we're signing off. Tomorrow we will work on the remaining services according to the priorities documented on the outage page, and work on getting services online in read/write mode.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:42:37 JST Drew DeVault

     in reply to

     We're taking a break, back to work in a few hours.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:42:38 JST Drew DeVault

     in reply to

     lists and todo are coming up now as well, also in read-only mode.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:42:39 JST Drew DeVault

     in reply to

     Update 2024-01-12 16:13 UTC: We are beginning to bring some services online. The following services will become available as DNS propegates to your local nameserver[...]

     Full update here:

     https://outage.sr.ht/

     tl;dr git and meta are coming online in read-only mode now

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:42:40 JST Drew DeVault

     in reply to

     meta.sr.ht is slowly coming online in read-only mode as DNS propegates

   • Embed this notice
     Simon Michael (simonmic@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:43:30 JST Simon Michael

     in reply to

     @drewdevault Superb status update! Best wishes to you and the team. I hope you’ll get investigative support in due course from some of the public cyber sleuths.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 18:44:15 JST Drew DeVault

     in reply to

     We're expecting hg to come online in read-only mode momentarily, and we will be striving to enable read/write support on all services currently available today.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 21:58:37 JST Drew DeVault

     in reply to

     Update 2024-01-13 12:40 UTC: Mail service to lists and todo has been restored, subject to DNS propagation delays. Emails queued during the outage should resume processing now – we do not believe any emails have been lost.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 21:58:37 JST Drew DeVault

     in reply to

     Our new transit provider has notified us that the DDoS followed us to our new network, but that their mitigations are in effect. Ideally there will be no service interruption as a result.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 21:58:38 JST Drew DeVault

     in reply to

     We have an update regarding pages.sr.ht. All custom *sub*domains and *.srht.site should be coming up shortly in read-only mode, subject to DNS propagation delays.

     Users running a website on an apex record (i.e. example.org rather than sub.example.org) can get back online with MANUAL INTERVENTION. For details, see the status update here:

     https://outage.sr.ht/

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 21:58:39 JST Drew DeVault

     in reply to

     read/write support is now enabled for the following services:

     - Project hub
     - meta
     - git
     - todo*
     - lists*
     - paste
     - man

     * todo and lists are online and accept read/write requests via the web, but we're still working on the mail system.

     hg.sr.ht is online and read-only, though there are known issues with hg clone.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 21:58:39 JST Drew DeVault

     in reply to

     pages.sr.ht: we know this one is important to you. Stand by for updates, we're working on this one but it has a number of special challenges to bring back into service.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Saturday, 13-Jan-2024 22:03:17 JST Drew DeVault

     in reply to

     Update: availability is impacted. We are working on the issue.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:04 JST Drew DeVault

     in reply to

     • karl marx & spencer

     @ruben not likely, but many people have expressed their intention to up their subscription to a higher level and this might be a good option if you want to express your support financially

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:13 JST Drew DeVault

     in reply to

     We have restored git.sr.ht and man.sr.ht to full service.

     Details: https://outage.sr.ht

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:13 JST Drew DeVault

     in reply to

     Also, more details to come when the situation is behind us, but we have made the executive decision to credit all paying accounts with service credits for the duration of the outage, whatever that duration ends up being.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     karl marx & spencer (ruben@friendship.quest)'s status on Sunday, 14-Jan-2024 23:28:13 JST karl marx & spencer

     in reply to

     @drewdevault I imagine it’s more cumbersome to do special cases but can we opt out of the credit?

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:15 JST Drew DeVault

     in reply to

     We have restored partial service this morning in a temporary transit configuration, and we hope that it holds. We are working to restore full service in this configuration while we work on a more permanent transit solution.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:15 JST Drew DeVault

     in reply to

     Details on current system status:

     https://outage.sr.ht/

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Sunday, 14-Jan-2024 23:28:16 JST Drew DeVault

     in reply to

     Update 2024-01-13 14:17 UTC: We will need to find a new transit provider to mitigate the problem. We are in talks with a provider to address the problem and we will begin the engineering work shortly. Further updates to come as we have them. Thank you again for your patience.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:46 JST Drew DeVault

     in reply to

     Update 2024-01-15 22:15 UTC: object storage for git.sr.ht (e.g. releases attached to git tags) is now available for read and write operations.

     Haelwenn /элвэн/ :triskell: repeated this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:46 JST Drew DeVault

     in reply to

     We have restored sourcehut.org and consolidated information about the outage:

     https://sourcehut.org/blog/2024-01-16-sourcehut-outage/

     (await DNS propagation if this 404's)

     status.sr.ht has been restored to normal and the details from the temporary outage page moved there:

     https://status.sr.ht

     Today we are working on builds and chat.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:47 JST Drew DeVault

     in reply to

     Update 2024-01-15 15:50 UTC: pages.sr.ht is now available for full service, albeit with degraded performance for publish operations.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:48 JST Drew DeVault

     in reply to

     Update 2024-01-15 08:20 UTC: hg.sr.ht is now available for full service. We are working on builds availability today.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:49 JST Drew DeVault

     in reply to

     We are bringing the mail system back online, which will restore lists and todo to full service. No emails are believed to be lost, and the queue should be processing soon.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Tuesday, 16-Jan-2024 20:03:49 JST Drew DeVault

     in reply to

     We're in pretty good shape and closing out our work for today. We're fairly sure that the services which are online now are fairly stable. Tomorrow we're going to work on getting builds online and possibly chat.sr.ht, as well as planning for the long tail of tidying up loose ends.

   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Tuesday, 16-Jan-2024 20:16:00 JST Haelwenn /элвэн/ :triskell:

     in reply to

     @drewdevault btw for ones using unbound and wanting to flush their DNS cache: unbound-control flush_zone sr.ht

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 17-Jan-2024 18:25:45 JST Drew DeVault

     in reply to

     Done (underpromise, overdeliver) 😄

     Haelwenn /элвэн/ :triskell: likes this.
     Haelwenn /элвэн/ :triskell: repeated this.
   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 17-Jan-2024 18:25:46 JST Drew DeVault

     in reply to

     builds is now running at full capacity.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 17-Jan-2024 18:25:46 JST Drew DeVault

     in reply to

     We're going to do some quick database maintenance, which will cause an outage on all services for no more than 15 minutes

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 17-Jan-2024 18:25:47 JST Drew DeVault

     in reply to

     builds.sr.ht is now available for full service, subject to DNS propagation delays.

     Note that we are currently running it half-capacity, and we expect to have it up to full capacity by tomorrow morning. In the meantime, you may experience longer than usual queue times for builds.

   • Embed this notice
     Drew DeVault (drewdevault@fosstodon.org)'s status on Wednesday, 17-Jan-2024 19:41:59 JST Drew DeVault

     in reply to

     Full service has been restored for all SourceHut services.