Let's say you wanted to block a bunch of rogue web crawlers run by slop companies from your website.
What are some tricks you'd apply?
I already have a list of AS numbers that send garbage traffic around and I'm also looking at non-standard user agents.