If anything has proven the folly of #opt-out systems, it is the fact that robots.txt and spiders.txt are no longer respected by the plagiarism engines stealing the sum total of human knowledge in order to corrupt it and sell it back at a 10,000% markup.
Opt-out systems are ephemeral, and only operate as long as the aggressor chooses to respect it, which is to say, 'until they can get away with not doing so'.