@dcc I have to get more boxes/crawlers rolling (budget is limited at present): it's currently doing about 20k instances per hour and it was at about 80k with the old setup. 80k was enough to more or less keep up with fedi.

The crawler is now using Revolver for the storage, meaning the crawlers just send back IDs to the workers, way faster, and the DB is a little less than half the size it was.

Aside from getting the throughput back up, the main issue at present is what to do about the ~4k instances that appeared while it was down. They show up as a huge spike in the "new instances" graph on the front page.