Conversation

Notices

1. Embed this notice
   :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 02:44:02 JST :blobcathug:

   • Hanna
   @hanna
   hi, i heard you had db rot at social.platypus-sandbox.com...
   :blobcathuggiessad:
   may i ask if this is due to a bug? or what happened?
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 02:44:21 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @hanna also tagging @ashten
   • Embed this notice
     [evil] lvl. 99 cute lesbian vampire catgirl (ashten@woem.men)'s status on Tuesday, 12-Dec-2023 02:50:37 JST [evil] lvl. 99 cute lesbian vampire catgirl

     in reply to

     • Hanna

     @Jain@blob.cat @hanna@woem.men i mean, it certainly wasn't something we did. basically, the server just got slower and slower, now locking up the entire server for 15-30 seconds whenever a new post is sent.
     
     we're also encountering a bunch of other bugs too:
     
     we can't migrate away from the instance, cuz the migration setting is broken.
     
     we cant export our own posts cuz it just sits there saying its not ready yet.
     
     and the other day, Hanna got a notification that bricked her account until i manually went in and deleted it.

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 02:50:37 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna slower and slower? doesnt sound like data rot...
     wana hear my inputs?
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 02:57:26 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna
     how is bloat? check this, only available for admins:
     https://social.platypus-sandbox.com/phoenix/live_dashboard/ecto_stats?nav=bloat
     how are the vacuum stats?
     https://social.platypus-sandbox.com/phoenix/live_dashboard/ecto_stats?limit=&nav=vacuum_stats&search=&sort_by=rowcount&sort_dir=desc
     how many oban_jobs_args_index_<something> do exist?
     https://social.platypus-sandbox.com/phoenix/live_dashboard/ecto_stats?nav=index_size
   • Embed this notice
     [evil] lvl. 99 cute lesbian vampire catgirl (ashten@woem.men)'s status on Tuesday, 12-Dec-2023 02:57:27 JST [evil] lvl. 99 cute lesbian vampire catgirl

     in reply to

     • Hanna

     @Jain@blob.cat @hanna@woem.men sure! i know that TLA is having very similar problems, so maybe what you have to say could help them out!

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:14:12 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna :blobcatlaugh:
     ok, that looks fine to me... except that oban_jobs::oban_jobs_args_index has a unusual high bloat for such a small size...
     you might want to execute something like
     
     REINDEX INDEX CONCURRENTLY oban_jobs::oban_jobs_args_index;
   • Embed this notice
     [evil] lvl. 99 cute lesbian vampire catgirl (ashten@woem.men)'s status on Tuesday, 12-Dec-2023 03:14:13 JST [evil] lvl. 99 cute lesbian vampire catgirl

     in reply to

     • Hanna

     @Jain@blob.cat @hanna@woem.men
     
     This feels like opening a secret passageway in a house that you've lived in for months.

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:18:56 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna that could be the issue but also not...
     also pg_repack https://reorg.github.io/pg_repack/ https://github.com/reorg/pg_repack can help rebuilding the tables & indexes while beeing live. So you can get rid of the bloat & rebuild all indexes so you will get a neat & fresh db...
     there is one small issue i have with it, i can explain that if you are interested
   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:32:47 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @hanna @ashten well tbh, im not really familiar with that process... but if you have some time available and if you want to research about it, i would recommend turning on debug logging in admin-fe and then take a look at the log
   • Embed this notice
     Hanna (hanna@woem.men)'s status on Tuesday, 12-Dec-2023 03:32:49 JST Hanna

     in reply to

     • [evil] lvl. 99 cute lesbian vampire catgirl

     @Jain@blob.cat @ashten@woem.men another issue is migration fails to recognize any non local account, despite the target account being federated elsewhere such as search and follows. Migration just says "target account not found."

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:41:59 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna :blobcatgoogly: well... do you compile akkoma by source or do you use the pre packaged version (OTP)? im running an older version rn, you are on the latest so... im not sure what that could be... anything would be a more weird guess like the index rebuild :blobcatgiggle2:
   • Embed this notice
     [evil] lvl. 99 cute lesbian vampire catgirl (ashten@woem.men)'s status on Tuesday, 12-Dec-2023 03:42:00 JST [evil] lvl. 99 cute lesbian vampire catgirl

     in reply to

     • Hanna

     @Jain@blob.cat @hanna@woem.men that's... actually another bug we encountered. admin-fe wont allow us to turn on debug logging. it just throws a 500 error when we try to enable it.

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:50:49 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna yeah sure, but then let me tell you some things about pg_repack:
     i do have an issue that it somehow ends up not recreating the oban job index table... it starts creating a new index but because of the frequently locks it stops exchanging the new one with the old one, so keep an eye on that one.
     Beside of that, pg_repack works quite nice, you just have to fullfill its requirements:
     
     > Performing a full-table repack requires free disk space about twice as large as the target table(s) and its indexes. For example, if the total size of the tables and indexes to be reorganized is 1GB, an additional 2GB of disk space is required.
     
     from my point of view i cant really see any issue... also feel free to join the akkoma irc https://akkoma.social/#faqs maybe somebody has a better idea
     
     oh an tell me if it gets better :ablobcatbongo:
   • Embed this notice
     [evil] lvl. 99 cute lesbian vampire catgirl (ashten@woem.men)'s status on Tuesday, 12-Dec-2023 03:50:50 JST [evil] lvl. 99 cute lesbian vampire catgirl

     in reply to

     • Hanna

     @Jain@blob.cat @hanna@woem.men i'll give the database commands and repack a try, but im not sure it'll help with any of the bugs we're encountering. XD

   • Embed this notice
     :blobcathug: (jain@blob.cat)'s status on Tuesday, 12-Dec-2023 03:53:09 JST :blobcathug:

     in reply to

     • Hanna
     • [evil] lvl. 99 cute lesbian vampire catgirl
     @ashten @hanna oh, one more dumb question: does the log say anything? :blobcatgoogly: im so stupid, i should have asked this in the first place