Conversation

Notices

1. Embed this notice
   Cuteroll, shark aficionado (ctrl@alico.nexus)'s status on Friday, 10-Jan-2025 15:46:01 JST Cuteroll, shark aficionado

   • Daniel Supernault

   @dansup@mastodon.social ah it doesn't honor robots.txt currently? 💀

   • Embed this notice
     Samuel Hautamäki (sirobsidian@fosstodon.org)'s status on Friday, 10-Jan-2025 16:34:53 JST Samuel Hautamäki

     • Daniel Supernault

     @dansup @ctrl Shouldn't anything that basically crawls websites honor robots.txt? Good job anyway making such a useful service though, even if for now it doesn't honor robots.txt.