“Robots.txt” is one of those standards only conscientious developers follow, because adherence is voluntary, it’s not backed by law, and requires more work than ignoring the file does. Only slightly more effective than posting a copyright notice on your Facebook page.