Adding robots.txt support to FediDB, if you want to block FediDB from crawling your instance, please add a disallow rule to the ./well-known endpoint like GtS is doing.
https://goblin.technology/robots.txt
We will be dropping the Popular Accounts section as those are more harder to support, and I will re-enable them once I can get around to it.