Conversation

Notices

1. Embed this notice
   Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:18 JST Ryan Wild

   One of the fall-outs of sorts from the maintenance this morning is it's identified a fairly significant storage issue across our servers. Right now there isn't a huge amount I can realistically do other than invest a lot of money in more servers (And that's not entirely financially viable with the current state of donations from here and Universeodon (And it impacts both)).

   • Embed this notice
     Renaud Chaput (renchap@oisaur.com)'s status on Thursday, 04-Jan-2024 07:54:13 JST Renaud Chaput

     in reply to

     • Emelia 👸🏻

     @wild1145 I am not sure why you need this space for backups. Are you doing SQL backups?
     @thisismissem

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:14 JST Ryan Wild

     in reply to

     • Emelia 👸🏻

     @thisismissem We managed to shave around 80GB in our statuses table today through a vacume full and ultimately exporting and re-importing the database to fix the encoding issue. One of the bigger issues is needing to re-export the DB, re-build the database VM entirely and re-creating it as the disk has ballooned a lot larger than it actually needs, but at the same time we need 2.5x the size of the DB to be free on disk at any given time for effective / full backups of the DB.

   • Embed this notice
     Emelia 👸🏻 (thisismissem@hachyderm.io)'s status on Thursday, 04-Jan-2024 07:54:15 JST Emelia 👸🏻

     in reply to

     @wild1145 it'd be very interesting to see where database storage is being consumed (e.g., is it all non-interacted with / non-addressed statuses?

     Perhaps we need a mechanism in Mastodon to reduce these to just a reference to the remote server. Like you visit that post, it gets refetched & the page shows a "we're refetching this post, if you'd like to view it immediately, click here"

     And this would only apply for non-local-mentioned posts & posts without interactions (likes, reblogs, replies)

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:16 JST Ryan Wild

     in reply to

     MastodonApp.UK and Universeodon won't be going anywhere I want to make that very clear. I just need to figure out if there is a cost effective and scalable way to grow our storage capacity without causing more technical headaches or making things a lot worse. If I can't I will have a look into options to scale back some of our servers and reduce our footprint so we can make investments into new servers with the higher capacity we urgently need.

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:17 JST Ryan Wild

     in reply to

     Our other database host for Universeodon is sitting at a bit over 80% disk usage, and we have one of our more general purpose application servers running at 80% and another at over 90% as well.

     This means I'm starting to have to look at if the current scale we are running the site at is sustainable (For both here and Universeodon) and if reducing the amount of servers we have is going to be the only real option forward so we can invest in a larger capacity server for our DB's.

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:17 JST Ryan Wild

     in reply to

     I'm fortunate enough to get a solid amount per month in Donations, with a relatively small chunk being monthly subscriptions (A lot of our monthly income is you kind folks giving ad-hoc contributions) / payments, however we're just starting to out grow our original architecture and design and I am locked in with some of our infrastructure to contractual obligations / purchase agreements which helped to save us a lot of money in the longer term.

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 07:54:18 JST Ryan Wild

     in reply to

     I'm looking to see if there are any cost effective solutions that don't introduce issues where the sites are more likely to run slow or otherwise crash but it's easier said than done.

     Right now our database server host is running at 99.75% disk usage, and it's having some fairly significant operational impact behind the scene (As well as me loosing a lot of sleep over the fact) and while the container running the database has some headroom, it's not enough.

   • Embed this notice
     Renaud Chaput (renchap@oisaur.com)'s status on Thursday, 04-Jan-2024 18:14:28 JST Renaud Chaput

     in reply to

     • Emelia 👸🏻

     @wild1145 I will repeat what other said, but you really do not want to use dumps for backups at this scale. Use pgbackrest or another similar solution that relies on block backups, and do not require a copy of the data locally when backing up.
     @thisismissem

   • Embed this notice
     Ryan Wild (wild1145@mastodonapp.uk)'s status on Thursday, 04-Jan-2024 18:14:29 JST Ryan Wild

     in reply to

     • Emelia 👸🏻
     • Renaud Chaput

     @renchap @thisismissem Yes, currently it'll dump the database to disk, gzip it and upload it to a bucket. I'm just testing now that I think I have free'd up just enough space that the current VM in it's current config might just be able to perform a single backup at a time (Which was an issue with my old setup as it was backing up different backups but during overlapping times)