robots.txt é pedir "por favor". Parasitas não ligam para educação.
"Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints [...]"
https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html