@mint here's my current working theory:

Older Oban used the db triggers. It has very slight overhead, so they changed to the new method for more performance for the most demanding use cases.

If Postgres can't keep up with work and queries start timing out, Ecto/Postgrex (db driver and connection pooler) crashes and restarts. This would cascade up to Oban. And I think in some edge case it can cause Oban to come back online but not properly start the queue processing.

Now you stopped processing jobs. Super weird.

I have a lower resource test server I'm running now and following some relays -- feld@friedcheese.us. Feel free to flood me with follow requests from a giant bot network, I'll need more followers to stress this further 🤭