Notices by LIBRANET.de Announcements (libranet@mastodon.social)

Sorry! I have accidentally invoked a shutdown of the database sever. 😭

Please take a coffee and relax.

Starting step 2: 22.04 LTS -> 24.04 LTS upgrade

Ubuntu 20.04 LTS End Of Life – Upgrade necessary for libranet.de and venera.social

The recommended distribution upgrade path is 20.04 LTS -> 22.04 LTS -> 24.04 LTS. Sigh. 🙂

I think I will do that tomorrow around 9:00 CEST / 7:00 UTC. This will take up to 2 hours if everything goes smoothly.

After upgrading the Linux distribution I need to upgrade/convert the database from postgresql 14 to postgresql 16.

I will stop the misskey service - do the database migration - restart the misskey service.

misskey.de

Linux distribution upgrade done. New version is Ubuntu 24.04.2 LTS. Rebooting now. Hope everything is working. :-)

Linux Distribution upgrade on misskey.de

I will stop the server, create a snapshot, restart the server and then perform the release upgrade.

This will take about an hour. There will be several short interruptions during this time.

misskey.de

The issue with the blurred or missing profile pictures has been fixed. 🎉 🚀 ✨

Maybe you need to force-refresh (SHIFT + Reload) the page to load all profile pictures successfully. Alternatively, you may clear your browser cache.

libranet.de · venera.social

In the next days I need to optimize the large database tables of libranet.de and venera.social. The database has grown to an extent where the server would run out of disk space in the next weeks.

This is a time-consuming and resource-intensive process that can block other database queries. That's why I will switch on maintenance mode beforehand.

I'll start with the large database tables of venera.social. When this is done I'll continue with libranet.de.

There's an update for MariaDB - the database server. I'll switch both Friendica nodes into maintenance mode at 12:00 CET (11:00 UTC) and install the update. Expected downtime: approx. 40-60 minutes.

There's an update for the MariaDB database system available. When I install the update the database system will be automatically restarted. This will take around 15-20 minutes.

I'll start the update in a couple of minutes.

libranet.de · venera.social

Linux kernel update installed. I'll reboot the server at 21:30 CEST / 19:30 UTC.

libranet.de · venera.social

Ok. I'll try to solve this. I need to drop old entries from the huge, local workerqueue of venera.social, put libranet.de into maintenance mode and then reenable venera.social. I'll see if the webserver survives the storm this way.

Hmm. venera.social is literally DDoS-ed by the Fediverse. Tons of requests to the local inbox. The webserver is constantly hitting the limit. 😭

Obviously it is not possible to import another database while libranet.de is running. The processes seem to block each other. I have aborted the import of the venera.social database and will try again later. To do this, however, libranet.de must be put back into maintenance mode.

Database import of venera.social is still running. Currently the largest table is being restored.

Update: The database structure update on venera.social is finally done. This node is up and running again. Unfortunately, the structure update on libranet.de isn't completed yet. I can't say how long this will take. The process is painfully slow. I'm very sorry about the terrible situation. But all I can do is waiting until the database update is completed.

I found out that ALTER TABLE statements on MySQL/MariaDB can be very, very slow when the query is executed on very large tables with many indices. The post-user table (that's the one that is responsible for the long running update) is huge. The table has a size of 140 GB and has 36 indices.

The Friendica nodes libranet.de and venera.social are currently in maintenance mode. There is an update for the database structure. This has now been running for 26 hours. There is nothing I can do except waiting until it is completed. Sorry for the long service interruption.

Linux kernel update installed. Rebooting the server at 12:00 CEST / 10:00 UTC.

libranet.de • venera.social

I will upgrade MariaDB (the database server) from v.10.6 to v.10.11 in 20 minutes. The database server is automatically restarted by the upgrade process.

Expected downtime around 10 minutes.

libranet.de · venera.social