Conversation

Notices

1. Embed this notice
   Phantasm (phnt@fluffytail.org)'s status on Tuesday, 06-Aug-2024 13:50:46 JST Phantasm

   • feld

   @feld, is there a possibility that Oban behavior has changed between versions 2.13.4 and 2.17.9 that could break Pleroma's federation (version update commit: dbf29cba)? This commit is in the 03024318..9953b0da commit range from mint and also before the commit in the issue on Gitlab.

   I have a test instance that now runs on that commit to see if it will break. If it doesn't break this week, I'll slowly bisect the commit range.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 06-Aug-2024 13:50:45 JST feld

     in reply to
     @phnt There is one known behavior change: instead of using Postgres insert triggers to tell Oban about new jobs, the Oban process itself tracks them.
     
     Nobody else (Oban users) had reported issues with Oban since this change, but now I've pulled up Oban's Github again and this is a possible candidate. Possible. Unsure though.
     
     https://github.com/sorentwo/oban/issues/1129
      likes this.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 06-Aug-2024 13:53:54 JST 

     in reply to

     • feld
     @phnt @feld Worth noting that someone on [url=https://git.pleroma.social/pleroma/pleroma/-/issues/3286]#3286[/url] reported that updating Oban supposedly fixes the issue. I don't have much faith in that since the issue pops up very randomly and not enough time has passed since to make a full conclusion.
     Screenshot_20240806_075109.png
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Tuesday, 06-Aug-2024 13:53:55 JST Phantasm

     in reply to

     • feld
     @feld This instance currently still runs 2.6.3 and therefore isn't affected in any way, but the test instance runs:
     Erlang: 25.2.3+dfsg-1
     Elixir: 1.14.0.dfsg-2
     PostgreSQL: 15
     
     Both are the default Debian 12 packages. I can switch to a more upstream setup with asdf if you want.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 06-Aug-2024 13:53:56 JST feld

     in reply to

     • feld
     @phnt Can you confirm the Elixir and Erlang versions of the servers where you've seen the issue?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 06-Aug-2024 13:58:12 JST 

     in reply to

     • 
     • feld
     @phnt @feld >but the versions deployed by the users will vary
     Agency and salon: OTP 25, Elixir 1.14.4, Oban 2.17.9
     boymoder.biz (still hasn't frozen since the last time, no updates done): OTP 25, Elixir 1.14.0, Oban 2.17.9
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 06-Aug-2024 14:12:25 JST 

     in reply to

     • 
     • feld
     @feld @phnt Disabled insert_trigger on salon as noted in 2.17 changelog to see if that makes any difference.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 06-Aug-2024 21:52:47 JST feld

     in reply to

     • 
     @mint @phnt It has no effect as we ran the migration which deleted the triggers from the db
      likes this.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Wednesday, 14-Aug-2024 05:19:14 JST Phantasm

     in reply to

     • feld
     @feld It certainly did not do that with Pleroma's configuration at least. There are more cache misses, held locks and increased reads in Postgres. Unless there was a change in Pleroma somewhere between 2.6.3 and the Oban bump commit that would cause exactly this, which is unlikely I think.
     
     
     Pleroma-tan likes this.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Wednesday, 14-Aug-2024 05:19:15 JST feld

     in reply to
     @phnt the change they made was intended to reduce pressure on the database for the heaviest Oban users... so that's odd. 😲
     Pleroma-tan likes this.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Wednesday, 14-Aug-2024 05:19:16 JST Phantasm

     in reply to

     • feld
     @feld You probably won't trigger it on the thinclient. The test 1 core VPS from OVH is still doing fine after 10 days. Whatever change was made to Oban, it caused more iops which is killing the VM OS drive where the DB cache is. And that is causing timeouts which snowball into the DB driver probably restarting which eventually kills Oban. You can clearly see on the graph when I updated the instance and when it died today.
     
     
     
     Pleroma-tan likes this.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Wednesday, 14-Aug-2024 05:19:17 JST feld

     in reply to
     @phnt yeah, I'm trying to reproduce it on slower hardware too
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Wednesday, 14-Aug-2024 05:19:18 JST feld

     in reply to
     @phnt oh wow this is fantastic feedback, please let me know if Oban 2.18 resolves the issue
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Wednesday, 14-Aug-2024 05:19:18 JST Phantasm

     in reply to

     • feld
     @feld Will do. I have my doubts though as it seems to be related to slow hardware where the database can't keep up and eventually everything crawls to a halt when Oban gets into some kind of a bad state.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Wednesday, 14-Aug-2024 05:19:20 JST Phantasm

     in reply to

     • feld
     @feld Can confirm that the upgrade to Oban 2.17 is likely the cause for the stalled federation as the 03024318...dbf29cba range doesn't seem to have anything related. I've updated FluffyTail to the Oban update commit on Sunday and it stalled few hours ago. Now running on your feld/debugging branch with Oban updated to 2.18.
     
     I have pleroma, postgres logs and a full copy of the DB I can restore locally if you want something from that.
     
     Here's also a reply to another thread that didn't federate to your instance: https://annihilation.social/notice/AkvXUKf4HtAFYuZnM0