This is kind of interesting: the queue-clearing bit, basically if enough instances suddenly go down, it stops crawling. This was written to handle the case where Fedilist itself was broken: it stops crawling when that happens. It used to just stop completely for ten minutes whenever the state changes were all from up to down-ish (nxdomain, etc.). Especially when it was using a byzantine load-balanced proxy setup, a proxy would sputter and Fedilist would start marking all the sites as dead.

That was basically a hack, so it's a little smarter now: passing the threshold for instances that changed state and the proportion that went down triggers a check, and we see

reluctantmediumatlarge.wordpress.com, bethanyhomesforsale.com, these aren't the kind of servers you think of when you think of fedi: they technically speak ActivityPub but it's effectively a broadcast channel. I was trying to think of how to stop burning resources checking and rechecking WordPress sites. I built in a bit that checks less frequently if the information changes less frequently, for example. I hated doing this, but I even special-cased WordPress itself so that FediList would only check them every 60 minutes at minimum. The minimum frequency is around 15 minutes, but not a lot of sites are checked that frequently: the frequency is cranked down if the site responds slowly (so as not to swamp busy sites), if there are a lot of overdue sites, and if the site information doesn't change frequently. Sites that are down are checked even less frequently. (One exception is sites whose domains don't resolve: DNS queries are cheap and don't cause any load on the server we're looking at.)

Special-casing WordPress didn't sit well with me, and neither did dropping WordPress, but there were also :over9k: over 9,000 WordPress sites that had made there way into FediList and FediList was basically wasting its time looking at WordPress "instances" because no one cares. But WordPress basically solved the problem for me. 6.7.2-RC1-59780 must have special-cased FediList, because that is the version number of the 4,149...4,157 instances that FediList just marked as "down".
wordpress278dead.png
wordpressdyingmore.png
wordpress274dead.png
wordpress265dead.png