“Robots.txt” is one of those standards only conscientious developers follow, because adherence is voluntary, it’s not backed by law, and requires more work than ignoring the file does. Only slightly more effective than posting a copyright notice on your Facebook page.
Conversation
Notices
-
Embed this notice
thudfactor (thudfactor@hachyderm.io)'s status on Saturday, 22-Jun-2024 20:46:17 JST thudfactor
-
Embed this notice
Multiverse Mike (multiverseofbadness@toot.wales)'s status on Saturday, 22-Jun-2024 20:46:16 JST Multiverse Mike
@thudfactor my experience early on with robots.txt files was even worse; I've seen bots specifically target the folders I've formally disallowed
I've formally laid hear techbros I worked with say they consider it more of a "suggestion" than even a request... 🤬
-
Embed this notice
Multiverse Mike (multiverseofbadness@toot.wales)'s status on Saturday, 22-Jun-2024 23:06:56 JST Multiverse Mike
@EndlessMason @thudfactor hah - honeypots rule...
I'm actually working on a phony bologna sentence generator to feed all the ai scrapers when they come sniffing on my message board without being logged in... It's bad enough they think the people playing my game talking about game stuff are worth training off of; wait until they get and even more worthless barrage of word salad!
-
Embed this notice
EndlessMason (endlessmason@hachyderm.io)'s status on Saturday, 22-Jun-2024 23:06:57 JST EndlessMason
@multiverseofbadness
Disallow: /cgi-bin/very-slow-honeypot.pl
@thudfactor -
Embed this notice
EndlessMason (endlessmason@hachyderm.io)'s status on Sunday, 23-Jun-2024 00:55:35 JST EndlessMason
@multiverseofbadness
Don't forget to sprinkle in the emails of some privacy regulation groups and some copyrighted disney text, maybe some embargoed crypto source ... also, you can download the text of Wikipedia and make a nice frequency table out of that with a couple of lines of code -
Embed this notice
Multiverse Mike (multiverseofbadness@toot.wales)'s status on Sunday, 23-Jun-2024 00:55:35 JST Multiverse Mike
-
Embed this notice