after another short down time everything is fixed! #QOTO had to recreate an index that got lost in the migration which was slowing down the DB and everything else. No amount of resources was going to help.
But it is fixed now all ques are empty or very close to it. we will now downgrade the DB to a more sane level now that it is fixed (we upgrades shortly to maintain the system). But it will sill be a pretty hefty system for us, and we can always scale back up when needed.
TL;DR everything is wording fine now.
PS we are now going to work on a staging environment to test upgrades so we can safely start moving the main server through the upgrade cycle. stay tuned.