I will be starting the installation of Mastodon v4.1.8 on Masto.host.
There will be about 15 seconds of downtime during the upgrade.
Changelog: https://github.com/mastodon/mastodon/releases/tag/v4.1.8
I will be starting the installation of Mastodon v4.1.8 on Masto.host.
There will be about 15 seconds of downtime during the upgrade.
Changelog: https://github.com/mastodon/mastodon/releases/tag/v4.1.8
I will be starting the installation of Mastodon v4.1.7 on Masto.host.
There will be about 15 seconds of downtime during the upgrade.
Changelog: https://github.com/mastodon/mastodon/releases/tag/v4.1.7
So here we go again! :)
Mastodon v4.1.4 was just released and it brings the patch that I installed earlier today and a couple more.
I will be starting the installation now. Again, it should be around 30 seconds of downtime.
Changelog here: https://github.com/mastodon/mastodon/releases/tag/v4.1.4
I am starting the installation of Mastodon v4.1.3 for everyone on Masto.host now.
During the upgrade there will be around 30 seconds of downtime.
You can see the release changelog here: https://github.com/mastodon/mastodon/releases/tag/v4.1.3
Here is the new status from OVH https://public-cloud.status-ovhcloud.com/incidents/1shkj36zsphs
I saddens me to report that the issue is still not solved. Just reported it and will update once I know more.
Everything is running smoothly again.
I have no words to apologise for the repeated problems these past couple of days.
You can be sure that I am doing everything at my disposal to get to the bottom of the problem and was assured that in the next 48 hours I will get a post-mortem of what exactly happened and how it was solved.
I will share when I have more information.
Thank you for your patience and support. Any issues or questions, please let me know 🐘
OK, service is back online. Will monitor and update soon.
Latest update from OVH:
"I understand that the situation is not reassuring at all since the Keystone authentication API is down again despite the multiple times the incidents were marked as resolved. I will do my best so that a solution to prevent the problem from happening again is applied."
So, let's hope that is the case.
Slowly coming back. Will update as soon as I can say it's stable.
Queues will take a while to catch up.
Really sorry for the long time but there is not much I can do and I am just waiting for OVH to fix the problem. I just got of the phone with them again and I am waiting for a reply with an estimate for when the problem will be solve.
Will keep you updated it.
The issue happened again for about one hour https://public-cloud.status-ovhcloud.com/incidents/8868qm5f4yd9
Sorry for the impact this had on the service.
So, there is still no official confirmation the issue is solved but from my end it's all looking good and I believe we are good now.
I will continue to monitor and will only update this thread if I notice any problem.
Again, really sorry for the long delay in solving this situation but all I could do on my end was try to put some pressure so they fixed the problem as soon as possible.
Please let me know if you notice any issue or have any questions. Thanks 🐘
Finally OVH posted an incident page about the issue: https://public-cloud.status-ovhcloud.com/incidents/853zm0j8q1j4
Will continue to monitor and update once I know more.
Currently Object Storage is down and it's causing server queues to fill up and in extreme cases to become unresponsive.
I have just confirmed with OVH that the issue is happening and they are working on it.
Really sorry for the inconvenience and will update here when I have more information.
There is a cooling problem in Database server 11 and that server will be going down in the next minutes for maintenance.
Only instances using Database server 11 will be impacted (you can check your database server in my.masto.host).
I will update here once the intervention finishes.
Sorry for any trouble this might cause you.
I will be starting the installation of Mastodon v4.1.1 for all Masto.host servers.
Each instance will go down for about 30 seconds during the upgrade installation.
v4.1.1 introduces some fixes and improvements. You can read the full log here https://github.com/mastodon/mastodon/releases/tag/v4.1.1
One of the improvements has a compatibility issue with OpenStack Swift and I will be doing a temporary work around, as I describe here: https://github.com/mastodon/mastodon/issues/24146
Any questions, please let me know.
Upgrade is done!
Everyone on Masto.host is now running Mastodon v4.1.1.
Any issues, please let me know.
Thanks ?
The attack continues and traffic is being filtered, so it might be slower than usual.
Still, from my testing 99,9% of instances hosted are working and over 50% are not being affected at all.
I will continue to monitor and update with more information.
BTW: you can read what a DDoS (Denial-of-service) attack is at https://en.wikipedia.org/wiki/Denial-of-service_attack
There is currently a DDoS attack going on in the Masto.host infrastructure.
Some instances are affected. I am investigating and will report once I know more.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.