Conversation

Notices

1. Embed this notice
   Evan Prodromou (evan@cosocial.ca)'s status on Tuesday, 26-Mar-2024 03:25:55 JST Evan Prodromou

   in reply to

   • Stephen Michael Kellat

   @smkellat "Crawlers are automated clients" is all it says. I think that would encompass API clients like ActivityPub federation peers.

   • Embed this notice
     Stephen Michael Kellat (smkellat@mastodon.social)'s status on Tuesday, 26-Mar-2024 03:25:56 JST Stephen Michael Kellat

     @evan RFC 9309 and linked documents propose definitions that generally would not include ActivityPub software unless there is an AP-enabled spidering package. As such, the proposed standard would indicate robots.txt being irrelevant to most known AP software. https://www.rfc-editor.org/info/rfc9309