Conversation

Notices

1. Embed this notice
   Matty (matty@gleasonator.com)'s status on Wednesday, 19-Oct-2022 01:56:41 JST Matty

   • Alex Gleason

   Hey @alex , sorry to bother, but I seem to have super duper fucked my server up and I’m not quite sure how to remediate the issue. From what I can gather, there are too many database connections going on for some reason and the overwhelming majority of them are timing out. I upgraded to the latest develop branch of Soapbox-BE (not Rebased) and ran the mix ecto.migrate per the Pleroma administration documents.

   I imagine this error is the one that’s causing the majority of my problems:

   Oct 18 16:08:18 nicecrew mix[19683]: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 153ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by: Oct 18 16:08:18 nicecrew mix[19683]: 1. Ensuring your database is available and that you can connect to it Oct 18 16:08:18 nicecrew mix[19683]: 2. Tracking down slow queries and making sure they are running fast enough Oct 18 16:08:18 nicecrew mix[19683]: 3. Increasing the pool_size (albeit it increases resource consumption) Oct 18 16:08:18 nicecrew mix[19683]: 4. Allowing requests to wait longer by increasing :queue_target and :queue_interval Oct 18 16:08:18 nicecrew mix[19683]: See DBConnection.start_link/2 for more information

   But there’s also this error and I’m not sure if it’s relevant:

   Oct 18 16:08:21 nicecrew mix[19683]: ** (ArgumentError) errors were found at the given arguments: Oct 18 16:08:21 nicecrew mix[19683]: * 1st argument: the table identifier does not refer to an existing ETS table Oct 18 16:08:21 nicecrew mix[19683]: (stdlib 3.17.2) :ets.lookup(Pleroma.Web.Endpoint, :__phoenix_url__)

   I appreciate any help you can provide because I’m at a complete loss right now.

   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Wednesday, 19-Oct-2022 01:56:38 JST Alex Gleason

     in reply to
     Wait what commit is this on? Git log
   • Embed this notice
     Matty (matty@gleasonator.com)'s status on Wednesday, 19-Oct-2022 02:10:40 JST Matty

     in reply to

     • Alex Gleason

     From git log

     commit aba157c24048a1dbe40499d5b223c3bd09412dd4 (HEAD -> develop) Merge: e5afdaf04 044719737 Author: Matty <mattyboombalatty@protonmail.com> Date: Tue Oct 18 13:47:26 2022 +0000 Merge branch 'develop' of https://gitlab.com/soapbox-pub/soapbox-be into develop commit 0447197378e60ff2794acb13d1aa57b17aa6d883 (soapbox-be/develop, soapbox-be/HEAD) Merge: 22a153285 035d9e71e Author: Alex Gleason <alex@alexgleason.me> Date: Mon Oct 10 01:22:54 2022 +0000 Merge branch 'merge-pleroma' into 'develop' Merge Pleroma See merge request soapbox-pub/rebased!184 commit 22a1532850ce595cfc2bad110401a5bc661f09bb Merge: 04ee56407 0ee413400 Author: Alex Gleason <alex@alexgleason.me> Date: Sun Oct 9 22:36:58 2022 +0000 Merge branch 'link-tag-fix' into 'develop' Support FEP-e232 quote posts See merge request soapbox-pub/rebased!185
   • Embed this notice
     Alex Gleason (alex@gleasonator.com)'s status on Wednesday, 19-Oct-2022 02:10:40 JST Alex Gleason

     in reply to

     Okay, so this is Rebased. Your comment about soapbox-be confused me.

     First run htop. Are you using too much CPU/RAM or not enough? If not enough, you need to tweak the settings until you’re using more.

     It’s most likely one of two issues:

     1. There’s a data migration running that’s hammering your db. Query select * from data_migrations; to check its progress.

     2. A new index on activities is not fully built or is corrupted. Are you running Postgres 14?

     As graf mentioned in the other comment, a vacuum can help. So can upgrading to Postgres 14 if you aren’t already.

     The data migration will take time. Once it’s done, performance will also improve.

     If htop says you’re not using enough resources, or you want to scale it up, I can help you tweak postgres to improve your situation. In particular, max_work_mem should be increased if you have the RAM for it.

   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Wednesday, 19-Oct-2022 02:11:26 JST anime graf mays ?️?

     in reply to

     • Alex Gleason

     @alex @matty since your site is down already, I would suggest a full database vacuum (e.g. sudo -Hu postgres psql then \c pleroma; and finally vacuum full verbose;) as a first step to deal with this. you’re likely running into corrupted indexes or similar. we had dealt with this just before the migration to the edit commit on rebased and again when the commit was merged. a repack (since we are live) in both instances resolved the issue. its long queries that are his issue

     that ETS table error it’s throwing we also had, but once the database congestion got under control and we went back down to ~25 load average it stopped

     Alex Gleason likes this.
   • Embed this notice
     Matty (matty@gleasonator.com)'s status on Wednesday, 19-Oct-2022 02:12:15 JST Matty

     in reply to

     • anime graf mays ?️?
     • Alex Gleason
     I've run the vacuum several times, but I'll give what you said a shot. I had considered that I turbo-fucked the database somehow, but I'm *really* hoping that's not the case. Should I stop the pleroma service prior to running the vacuum task? Thanks!
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Wednesday, 19-Oct-2022 02:12:15 JST anime graf mays ?️?

     in reply to

     • Alex Gleason

     @matty @alex yes, also consider looking into pg_repack so you can run this as a scheduled maintenance task while the database is live. ours runs every sunday

     another thing you can look into is temporarily increasing the pool_timeout and pool_size under config :pleroma, Pleroma.Repo, to something more and less conservative

     such as lowering the pool_timeout so pleroma isnt locking workers, start with something like pool_timeout: 15_000,

     and increasing pool_size to something much less conservative pool_size: 150,

     in order to change this –^ you will need to add this to your prod.secret.exs

     config :pleroma, :dangerzone, override_repo_pool_size: true

     in config/prod.secret.exs assuming you’re on the develop branch. you will have to temporarily change your postgres config (/etc/postgresql/(12,13,14)/main/postgresql.conf) to match that

     max_connections = 170

     this may help relieve some stress so you can get a better idea of whats causing it

     Alex Gleason likes this.