Conversation

Notices

1. Embed this notice
   Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:22:57 JST Evan Prodromou

   I was reading this article about LLMs making bad citations. I found it pretty interesting, so I decided to try to replicate it with ChatGPT.

   https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:24:33 JST Evan Prodromou

     in reply to

     I tried it with a document I wrote, FEP 5711. It's an enhancement proposal for ActivityPub, adding some inverse relationships for important properties.

     https://codeberg.org/fediverse/fep/src/branch/main/fep/5711/fep-5711.md

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:27:30 JST Evan Prodromou

     in reply to

     Anyway, I took a paragraph out of the document and asked ChatGPT to identify the URL, publisher, publication date, and title. It failed. You can see the transcript here:

     https://chatgpt.com/share/68573fa9-b340-800f-b9b4-7b74fdf0bf46

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:28:46 JST Evan Prodromou

     in reply to

     I was surprised to see that it had really no visibility of the FEPs. After a while, I realized that codeberg.org, the hosting service for FEPs, has ChatGPT blocked.

     https://codeberg.org/robots.txt

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:35:19 JST Evan Prodromou

     in reply to

     I understand the goal; many people don't want their code to be used by LLM code generators. But it also means that this document repository isn't visible for people who use LLMs like a search engine. Numbers vary, but afaict somewhere around 10% of people use LLMs as their primary search engine, and about 50% of people use LLMs some of the time for search.

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 08:40:02 JST Evan Prodromou

     in reply to

     I guess there's maybe some justification like, those people are bad, and they don't deserve nice things like Fediverse Enhancement Proposals? Or, maybe, we have to take a principled stand against LLMs by not providing any training data for them? Such that, perhaps, people disappointed by not having good results in LLMs will return to using traditional search engines like Google or Bing, which are more ethical because reasons.

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Sunday, 22-Jun-2025 11:31:10 JST Evan Prodromou

     in reply to

     • Codeberg.org
     • Mirko Adam

     @elshid @Codeberg that's interesting! Also, really self-destructive.

   • Embed this notice
     Mirko Adam (elshid@social.librem.one)'s status on Sunday, 22-Jun-2025 11:31:11 JST Mirko Adam

     in reply to

     • Codeberg.org

     @evan No, the reason is simply the server load. The AI crawlers have so excessively crawled @Codeberg that their main service, to host a git server, was often very slow.

   • Embed this notice
     Evan Prodromou (evan@cosocial.ca)'s status on Monday, 23-Jun-2025 23:47:29 JST Evan Prodromou

     in reply to

     • Codeberg.org

     @Codeberg Good idea! I set up a mirror at https://fep.swf.pub/ I probably need to automate the sync, and make sure to point back to Codeberg for contributions.

   • Embed this notice
     Codeberg.org (codeberg@social.anoxinon.de)'s status on Monday, 23-Jun-2025 23:47:31 JST Codeberg.org

     in reply to

     @evan The decision to block these scrapers actually originates from an Codeberg-e.V. internal discussion. The argument for "visibility" in the AI language models was considered. However, the cost on Codeberg is immense (also see https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html), and we are also not fans of the Big Tech AI companies.

     Codeberg is a development platform. If your content needs to be known to all scrapers, you are free to publish valuable resources to a normal website additionally.