Paul Cantrell
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
Paul Cantrell
Same question as the previous post, except for Wkipedia. What would you like to see them do to send a shot across the bow?
Or…well, it’s Wikipedia. Maybe more like a shot to the hull.
3/2
Paul Cantrell
@joe
It is and some of us miiiiight already be doing it.
disregard Joe Groff
@inthehands is "serve LLM poison to googlebot user-agents" on the table
Adam Shostack :donor: :rebelverified:
@inthehands (3) sue on the basis that’s it’s not fair use, and these derivative works clearly have a dramatic impact on the value of the original site
Paul Cantrell
This is clearly how copyright law as written •should• work. Not sure if it’s how it •does• work, but if anybody’s trying, they have my sword.
Paul Cantrell
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled context from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
Paul Cantrell
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
Paul Cantrell
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.
I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
ShadSterling
@joe @inthehands is there a coordinated effort that has a website? And/or server plugins that automate serving coordinated poison?
disregard Joe Groff
@ShadSterling @inthehands i don't know if there's a coordinated movement. there are prefab tools like https://lib.rs/crates/iocaine that are relatively easy to deploy, though i imagine they also lose some of their effectiveness as they become more popular and LLM providers start to counter them
disregard Joe Groff
@inthehands given how eager their summarizer is to incorporate "facts" from even unintentionally adversarial recent posts like satirical blogs, it seems like it wouldn't take much of a coordinated effort to reduce their result quality this way
crystal
@inthehands I think if you really want to sell that point, you should explicitly disallow googlebot in robots.txt, then also setup the middleware to respond with 404 to any other URI requested by googlebot
Korrupt
@inthehands meta noindex it is, definitely. robots disallow can actually hurt the process, since google cannot access the file with the noindex header and therefore won't deindex.
btw, they do indeed respect noindex and robots.txt ATM, since its qute easy to check if pages still get found. Then again, you never know what does not show up in search but is used for training (without giving credit, obv.) anyway. As far as i see, google still remains more standard compliant as e.g. OpenAI.
Paul Cantrell
@blogdiva
I believe that my various name=“___” values specifically target Google.
Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
your auntifa liza 🇵🇷 🦛 🦦
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
your auntifa liza 🇵🇷 🦛 🦦
@inthehands TIL thanks
Paul Cantrell
Keep it in pencil. I’m still learning myself, and not sure I understand everything correctly here.
sean watters
@inthehands i'm spartacus
Paul Cantrell
@swatters
Heck yeah! I hope I got the directives right…. :/
Oliver Jensen
@inthehands i love this idea. I am particularly curious if removing something from their search index also removes the information from the purview of their ai responses. If you have a way to determine this experimentally, I'd love to know the answer.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.