Public
- Public
- Network
- Groups
- Featured
- Popular
- People

Conversation

Notices

Embed this notice
Fuck. (weirdmustard@toot.community)'s status on Sunday, 14-Apr-2024 19:04:17 JST Fuck.

Offenbar klaut OpenAI einfach weiter, auch wenn man sie aktiv von seinen Websites ausschließt.
Edit: Ja, meine Güte, natürlich schließt man sie nicht aus, sondern sagt dem Crawler "Hier nicht crawlen". Der Punkt ist: Die spiders ignorieren die Vorgabe.
In conversation about 7 months ago from toot.community permalink
Attachments
1. John Levine writes Anyone got a contact at OpenAI. They have a spider problem. As I think I have mentioned before, I have the world's lamest content farm at https://www.web.sp.am/. Click on a link or two and you'll get the idea. Unfortunately, GPTBot has found it and has not gotten the idea. It has fetched over 3 million pages today. Before someone tells me to fix my robots.txt, this is a content farm so rather than being one web site with 6,859,000,000 pages, it is 6,859,000,000 web sites each with one page. Of those 3 million page fetches, 1.8 million were for robots.txt. It's not like it's hard to figure out what's going on since the pages all look nearly the same, and they're all on the same IP address with the same wildcard SSL certificate. Amazon's spider got stuck there a month or two ago but fortunately I was able to find someone to pass the word and it stopped. Got any contacts at OpenAI? R's, John PS: If you were wondering what they're using to train GPT-5, well, now you know.
  https://static.toot.community/media_attachments/files/112/268/194/518/126/354/original/cdafe8dbb67a8acb.png
2. To understand what's going on here, know these things: OpenAI is the company that makes ChatGPT A spider is a kind of bot that autonomously crawls the web and sucks up web pages robots.txt is a standard text file that most web sites use to inform spiders whether or not they have permission to crawl the site; basically a No Trespassing sign for robots OpenAI's spider is ignoring robots.txt (very rude!) the web.sp.am site is a research honeypot created to trap ill-behaved spiders, consisting of billions of nonsense garbage pages that look like real content to a dumb robot OpenAI is training its newest ChatGPT model using this incredibly lame content, having consumed over 3 million pages and counting... It's absurd and horrifying at the same time.
  https://static.toot.community/media_attachments/files/112/268/194/879/706/394/original/b2d500df292cc186.png
- GreenSkyOverMe (Monika) repeated this.
- Embed this notice
  always tired (moved to chaos) (project1enigma@wandering.shop)'s status on Sunday, 14-Apr-2024 19:04:42 JST always tired (moved to chaos)
  in reply to
  - Chris Evelyn
  @weirdmustard @chris_evelyn Das bringt mich auf eine Idee. Eine Fake Site, die nur random Markov-Ketten-Content liefert. Und absichtlich kein robots.txt.
  
  In conversation about 7 months ago permalink

Public

Conversation

Notices

Feeds