Conversation

Notices

1. Embed this notice
   ≠ (amerika@annihilation.social)'s status on Monday, 10-Feb-2025 08:26:39 JST ≠
   This is a tarpit intended to catch web crawlers that scrape data for LLMs.
   
   https://zadzmo.org/code/nepenthes/
   • pistolero likes this.
   • Embed this notice
     pistolero (p@fsebugoutzone.org)'s status on Monday, 10-Feb-2025 08:33:46 JST pistolero

     in reply to
     @amerika You know what those stupid LLM scrapers *love*? cgit. If you can afford the disk and the cycles on a VM, they will read the diffs between every two commits on every repo.
     
     Did you know that you can add basically any other repo as a remote for any git repo? You can take, say, the entire Linux kernel repo, add the Pleroma repo and the Mastodon repo as remotes, fetch them, and git will dutifully treat them like any other branches. And then cgit will show a diff between any random Mastodon commit from 2019 and any random Linux kernel commit. Toss in more repos, toss in every repo.
     
     Google will saturate your pipe crawling that indefinitely (ask how I know). Tarpits usually have a pattern but all of these diffs have enough novelty and length to keep almost every crawler interested.
     Haelwenn /элвэн/ :triskell: and ✙ dcc :pedomustdie: :phear_slackware: like this.
   • Embed this notice
     Zergling_man (zergling_man@sacred.harpy.faith)'s status on Monday, 10-Feb-2025 08:44:59 JST Zergling_man

     in reply to

     • pistolero
     @p @amerika This is easier than robots.txt disallow /lolbot/ and pointing that to a generator that spits out ~200 internal links.
     pistolero likes this.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Monday, 10-Feb-2025 08:47:38 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • pistolero
     @p Haha, even more makes me wish stagit would do branches now, would be really fun to have a tarpit repo that's basically all the various repos I have checked out (and I think with some hardlinks/reflinks the cost of that could be almost zero).
     pistolero likes this.
   • Embed this notice
     pistolero (p@fsebugoutzone.org)'s status on Monday, 10-Feb-2025 12:22:42 JST pistolero

     in reply to

     • Haelwenn /элвэн/ :triskell:
     @lanodan
     
     > (and I think with some hardlinks/reflinks the cost of that could be almost zero)
     
     If you disable repacking, yeah.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Monday, 10-Feb-2025 12:28:32 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • pistolero

     @p Or in fact checking documentation (wanted exact name of GIT_OBJECT_DIRECTORY), just listing them with GIT_ALTERNATE_OBJECT_DIRECTORIES

     pistolero likes this.
   • Embed this notice
     pistolero (p@fsebugoutzone.org)'s status on Monday, 10-Feb-2025 12:42:46 JST pistolero

     in reply to

     • Haelwenn /элвэн/ :triskell:
     @lanodan AH! I didn't know you could add additional object directories.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Monday, 10-Feb-2025 12:46:10 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • pistolero
     @p Meanwhile I knew but forgot, git has too many features sometimes…
     pistolero likes this.