Conversation

Notices

1. Embed this notice
   Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:01:32 JST Paul Cantrell

   In honor of the dawn of the age of web crawling being for LLMs instead of search engines, I have deleted the robots.txt for https://wookieepedia.org/.

   Let the floodgates open. Crawl away, my friends, crawl away.

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:06:38 JST Paul Cantrell

     in reply to

     I feel like an important thing to mention to put the previous post in context is that the links work.

     ALL the links work.

   • Embed this notice
     Jeff Miller (orange hatband) (jmeowmeow@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:10:24 JST Jeff Miller (orange hatband)

     in reply to

     @inthehands wait a minute there... do I hear echoes of A Plan For Spam?

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:11:47 JST Paul Cantrell

     in reply to

     • Jeff Miller (orange hatband)

     @jmeowmeow To that, I can only say mvuoooo ruuoau raoaaauvuaua ruuauawoaaou rroaaaavoouoa voiahouoaoa noa, wuoaa mouuoaruaourv ruaouu ruiu

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:13:13 JST Paul Cantrell

     in reply to

     • Adam ♿

     @voltagex HRRAOOOOOWAOOO!

   • Embed this notice
     Adam ♿ (voltagex@aus.social)'s status on Saturday, 27-Jan-2024 10:13:14 JST Adam ♿

     in reply to

     @inthehands OUTSTANDING.

   • Embed this notice
     Jeff Miller (orange hatband) (jmeowmeow@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:13:15 JST Jeff Miller (orange hatband)

     in reply to

     @inthehands (loud laughter). Let the Wookiee win!

   • Embed this notice
     Adam ♿ (voltagex@aus.social)'s status on Saturday, 27-Jan-2024 10:14:03 JST Adam ♿

     in reply to

     @inthehands you might also want to match on /*

     https://wookieepedia.org/test3

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:14:03 JST Paul Cantrell

     in reply to

     • Adam ♿

     @voltagex No way, gotta keep the site organized

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:38:45 JST Paul Cantrell

     in reply to

     There’s not a way to explicitly submit one’s site to OpenAI for crawling, right?

     Like…they just do secretive mass crawls on their own schedule, I assume?

   • Embed this notice
     Cory Carson (corycarson@gnu.gl)'s status on Saturday, 27-Jan-2024 10:40:46 JST Cory Carson

     in reply to

     @inthehands it…. It still works….

     https://wookieepedia.org/w/Rrruauou%20rroaanaaouo

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:40:48 JST Paul Cantrell

     in reply to

     • Cory Carson

     @corycarson Amazing

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:48:43 JST Paul Cantrell

     in reply to

     • Cory Carson

     @corycarson It took some cajoling, but I got GPT to start speaking Wookiee:

   • Embed this notice
     Cory Carson (corycarson@gnu.gl)'s status on Saturday, 27-Jan-2024 10:48:45 JST Cory Carson

     in reply to

     @inthehands MUST GO DEEPER

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Saturday, 27-Jan-2024 10:50:12 JST Paul Cantrell

     in reply to

     • Cory Carson

     @corycarson The question is, that is so shockingly similar to the style of “Wookiee” that Wookieepedia uses, it’s hard to believe that Wookieepedia wasn’t the training source. But until a few minutes ago, the site’s robots.txt disallowed crawlers for everything except the homepage….

   • Embed this notice
     Paul Cantrell (inthehands@hachyderm.io)'s status on Monday, 29-Jan-2024 14:20:28 JST Paul Cantrell

     in reply to

     • talby

     @talby One the home page, IIRC, but yeah

   • Embed this notice
     talby (talby@techhub.social)'s status on Monday, 29-Jan-2024 14:20:29 JST talby

     in reply to

     @inthehands I think LLMs mostly use https://commoncrawl.org/ rather than crawling the web themselves. The Internet Archive's Wayback Machine uses Common Crawl as a source and has Wookieepedia so I think it's likely in there already.