Conversation

Notices

1. Embed this notice
   M. Forester (mforester@rollenspiel.social)'s status on Tuesday, 03-Sep-2024 23:44:56 JST M. Forester

   in reply to

   • Justine Smithies

   @justine I wrote a script and pipeline to keep my static website's robots.txt up to date on all the AI madness. Not that I trust any of them to adhere to it, but here's the resource I'm using for that.

   https://github.com/ai-robots-txt/ai.robots.txt

   • Embed this notice
     Justine Smithies (justine@mastodon.bsd.cafe)'s status on Tuesday, 03-Sep-2024 23:44:57 JST Justine Smithies

     Question for all folk that self host their websites. Do you have a robots.txt and if so what's advisable to put in it ???
     #HomeLab #SelfHosted

     Tobias Hellgren repeated this.