Notices by LIBRANET.de Announcements (libranet@mastodon.social)

There's an update for MariaDB - the database server. I'll switch both Friendica nodes into maintenance mode at 12:00 CET (11:00 UTC) and install the update. Expected downtime: approx. 40-60 minutes.

There's an update for the MariaDB database system available. When I install the update the database system will be automatically restarted. This will take around 15-20 minutes.

I'll start the update in a couple of minutes.

libranet.de · venera.social

Linux kernel update installed. I'll reboot the server at 21:30 CEST / 19:30 UTC.

libranet.de · venera.social

Ok. I'll try to solve this. I need to drop old entries from the huge, local workerqueue of venera.social, put libranet.de into maintenance mode and then reenable venera.social. I'll see if the webserver survives the storm this way.

Hmm. venera.social is literally DDoS-ed by the Fediverse. Tons of requests to the local inbox. The webserver is constantly hitting the limit. 😭

Obviously it is not possible to import another database while libranet.de is running. The processes seem to block each other. I have aborted the import of the venera.social database and will try again later. To do this, however, libranet.de must be put back into maintenance mode.

Database import of venera.social is still running. Currently the largest table is being restored.

Update: The database structure update on venera.social is finally done. This node is up and running again. Unfortunately, the structure update on libranet.de isn't completed yet. I can't say how long this will take. The process is painfully slow. I'm very sorry about the terrible situation. But all I can do is waiting until the database update is completed.

I found out that ALTER TABLE statements on MySQL/MariaDB can be very, very slow when the query is executed on very large tables with many indices. The post-user table (that's the one that is responsible for the long running update) is huge. The table has a size of 140 GB and has 36 indices.

The Friendica nodes libranet.de and venera.social are currently in maintenance mode. There is an update for the database structure. This has now been running for 26 hours. There is nothing I can do except waiting until it is completed. Sorry for the long service interruption.

Linux kernel update installed. Rebooting the server at 12:00 CEST / 10:00 UTC.

libranet.de • venera.social

I will upgrade MariaDB (the database server) from v.10.6 to v.10.11 in 20 minutes. The database server is automatically restarted by the upgrade process.

Expected downtime around 10 minutes.

libranet.de · venera.social

🚨 Linux kernel update installed. Reboot required. Will do that at 21:00 CEST (19:00 UTC).

⚠️ Expected downtime approx. 15 minutes.

Can you spot the problem ? 🤦♂️

Server is up again. Checking the logs for problems...

Server is down, unfortunately. I'm on it.

> I'll reboot the server again to check the root filesystem for possible issues. Just to make sure everything is ok with it.

Reboot in 5 minutes. Expected downtime around 20 minutes.

Currently the delivery queue of libranet.de is pretty long. The reason is that the system is removing some very old and dead accounts and all their content. Also, some IP was flooding the webserver with requests.
The first thing will take some hours and I can't do anything about it. The second thing has been mitigated by blocking that IP in the firewall.

There's an update for the MariaDB database server. When I run that update the database server will be restarted automatically. Usually, that takes around 10-15 minutes. The services won't be available during that time.

Will do that at 16:45 CEST / 14:45 UTC.

libranet.de • venera.social • nc.libranet.de

🚨 Linux kernel update installed. Reboot required. Will do that at 13:00 CEST (11:00 UTC).

⚠️ Expected downtime around 15 minutes.