Conversation

Notices

1. Embed this notice
   Daniel Supernault (dansup@mastodon.social)'s status on Friday, 10-Jan-2025 15:35:38 JST Daniel Supernault

   Yeah, just because you can scrape the fedi, doesn't mean you should.

   I put my Starter Kit project on hold because even though the Discovery flag that Mastodon/Pixelfed support could be considered consent, I want to make it explicitly opt-in via DM confirmation.

   We can do this right. #fediverse

   • Embed this notice
     Daniel Supernault (dansup@mastodon.social)'s status on Friday, 10-Jan-2025 15:37:23 JST Daniel Supernault

     in reply to

     Disclaimer, I technically scape the fediverse for my https://fedidb.org project, but we only crawl nodeinfo, we do not store any statuses or any PII other than public notable accounts.

   • Embed this notice
     Cuteroll, shark aficionado (ctrl@alico.nexus)'s status on Friday, 10-Jan-2025 15:46:01 JST Cuteroll, shark aficionado

     in reply to

     @dansup@mastodon.social ah it doesn't honor robots.txt currently? 💀

   • Embed this notice
     Samuel Hautamäki (sirobsidian@fosstodon.org)'s status on Friday, 10-Jan-2025 16:34:53 JST Samuel Hautamäki

     • Cuteroll, shark aficionado

     @dansup @ctrl Shouldn't anything that basically crawls websites honor robots.txt? Good job anyway making such a useful service though, even if for now it doesn't honor robots.txt.