Masto.host
There is currently an issue with Masto.host Object Storage provider (OVH) that is causing media uploads to fail.
The information I got just now on a phone call with OVH is that it's a network issue and they working on it.
Sorry for the inconvenience and will update as soon as I know more.
Masto.host
@apps Thank you for letting me know.
Fedilab Apps
@mastohost
Yes, OVH is down. Only media seem impacted.
Masto.host
@jackwilliambell Well, the request for upload are pilling up, so this is causing an overload on the infrastructure but had zero reports of issues in the past days. Could you please get back to me if it happens again after this issue is solved?
Jack William Bell
I don't know if this is related…
I'm seeing gateway timeouts on updates and sometimes the entire site won't load at all. It was on and off for the last week, but today it's happening a lot.
Masto.host
@moiety yes, media uploads are queueing up and causing a massive loas in the infrastructure. Sorry.
Tall Girlfriend as a Service
@mastohost QG has been loading super slow for the last 30 minutes. Is this related?
Masto.host
@webmink Yep, it's likely to be. Sorry :|
Simon Phipps
@mastohost
I had add-to-list fail just now. Is that likely to be related?
Masto.host
@pixel Media uploads are queueing up and causing a massive overload in the infrastructure. That's why things are slow. Again, sorry.
Andy
@mastohost doesn't seem to be just media storage now, my instance is getting pretty slow/timing out entirely now
Masto.host
OVH just pointed to this page https://network.status-ovhcloud.com/incidents/xdklzwrtxyzg as the status page for this problem.
Masto.host
@NeatNit Completely forgot to do that. Just added a note there.
NeatNit
@mastohost good opportunity to examine https://status.masto.host/ in the mean time? "All services are online." ?
Masto.host
OK, I the network is coming back. There will be a large load in the network, so servers will be struggling to deal with it but I will keep monitoring the situation and report back.
Masto.host
@jackwilliambell It wasn't the greatest of moves but nothing major should be broken :)
You just lost some of the pending federated content, remote interactions, etc but nothing you can do now to revert that.
Again sorry for the trouble this caused. Anything I can help with, please let me know.
Jack William Bell
I checked Sidekiq and my queues were hugely backed up, like nearly a thousand jobs, so I just emptied all of them. Hopefully that won't cause any issues with the site, but it certainly seemed to have made things more responsive.
So, assuming this wasn't a stupid thing to do for some reason I don't know about, I recommend doing that if anyone else on masto host are still seeing problems.
Masto.host
@jackwilliambell Jobs in queue don't time out without being processed.
They just wait for threads to be available to process and only if the remote server fails multiple times (several days) does Sidekiq removes moves them out.
Jack William Bell
Sounds like I didn't make anything much worse than it would be if most of those jobs timed out.
But, eh… I mean, it's not like we are talking about money here. If I missed somebody's cat-picture post or someone favoriting one of mine I'm OK with that.
Masto.host
@mawr Thanks :) Appreciate it and sorry for the trouble this caused.
✨Soff✨ Mawr :dragnwitch:
@mastohost Thanks as always for your prompt attention and transparent communication!! :blobheart:
Masto.host
Everything is should be back to normal now. Some instances may still have some jobs queued to be processed but gradually it should all clear.
Really sorry for this long service degradation of service (~3 hours) but something went seriously wrong on OVH today and it was completely out of my hand.
Any issues or questions, please let me know. Thank you for your patience ?
Masto.host
@MLISrevenge Thank you for understanding :)
Jessica 🦕
@mastohost
Thank you for keeping us updated!
Masto.host
@lilithsaintcrow Eventually it did :) Thank you so much, Lili. Appreciate it.
Lili Saintcrow
@mastohost Eventually the network will sort itself out. We’ll all be fine in the meantime. You do wonderful work; anyone who gets shirty with you over a small roadbump like this can sod off.
Masto.host
@joshb No problem. Thanks for understanding.
Josh B
@mastohost Thank you for keeping us up to date on the issues.
Masto.host
@fuzzface Thanks :)
Trapped in the Age of Pisces♓️
@mastohost You rock!
Masto.host
@Kathrin Thank you for understanding.
Kathrin
Thank you so much for keeping us updated!
Masto.host
@david Thank you so much :) Appreciate it.
Masto.host
@leighelse Thank you
David Boles
@mastohost This is why we host with you; so we don’t have to try to figure out why everything is melting in real time! Thank you!
leighelse{}
@mastohost Well managed.
Masto.host
@theradioguy Yes, I was searching last night and found a bunch of reports from other people/businesses being affected by this issue.
Christopher Munz-Michielin??
@mastohost glad to hear it wasn't just us! My workplace has a bunch of gear at OVH and experienced the same massive network disruption
Masto.host
@galooph Sorry I couldn't do more :) thanks
galooph
@mastohost Thanks, Hugo!
Masto.host
@mike Appreciate it :)
Mike Kasprzak ? :checkmark:
@mastohost many thanks for what you do ?
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.