@thudfactor my experience early on with robots.txt files was even worse; I've seen bots specifically target the folders I've formally disallowed
I've formally laid hear techbros I worked with say they consider it more of a "suggestion" than even a request... 🤬