@phnt i saw zipdox repo recently too, but already made a setup to rotate random ua-sigs in my proxy thingy. i just happened to catch a site where i wasn't using it.
really weird that these people don't want "ai" or "bots" to crawl their sites, yet they really thirst for attention.
@jae >really weird that these people don't want "ai" or "bots" to crawl their sites, yet they really thirst for attention. There are some parallels with a group of people that have that kind of behavior and the maintainer. Also quite ironic since the author of the original mascot was never disclosed, so it was probably made with AI.
But my favorite part of Anubis is that it was never meant to actually stop scrapers from the start (pic related). Amazon bypassed it automatically since their UA has no "Mozilla" in it. So did Facebook, OpenAI and Claude. What it might have done, poorly, is stop the Chinese scrapers which were very dumb. image.png
@theorytoe@jae I spent like 2 hours total nullrouting tens of subnets from logs alone with some help of one-liners. Still faster than it would take me to package Anubis and write a policy for it. And guess what after like a week of checking monitoring, the scraping stopped. Or at least it was reduced to such a negligible amount that I stopped caring.
@phnt@theorytoe don't forget if you do get it working, then you have to dig through the maintainer's atrocity of weird custom templating and css, but you have to write him an email to get it working. sad days
@jae Honestly, Anubis got way more annoying than CF boats. At least with boats you don't need to wait for 90 seconds to complete a difficulty 4 challenge to access LKML. It's not like the scrapers can't subscribe to it and get it all sent to them. It's retarded.
Arch also disappointed me by shoving the wiki behind Anubis. Again, what's there to hide. The full wiki pages are already a package.
> There are some parallels with a group of people that have that kind of behavior and the maintainer.
there's an entire book that could be unpacked about the maintainer
> Also quite ironic since the author of the original mascot was never disclosed, so it was probably made with AI.
likely so, the maintainer goes ai heavy on his website.
> But my favorite part of Anubis is that it was never meant to actually stop scrapers from the start (pic related). Amazon bypassed it automatically since their UA has no "Mozilla" in it. So did Facebook, OpenAI and Claude. What it might have done, poorly, is stop the Chinese scrapers which were very dumb.
what's favorite for me is they are mainly complaining about "bandwidth" yet most of the people deploying this earn upwards of 150-200k (whatever currency)
if you don't want to get scraped and packet, build a darknet.
but no, that's impossible because darknet is not full of people, and these types require constant stimulation
> They want attention but also the ability to micromanage every bit of attention. Hence; why you get people who go crazy from everything that someone says about them online.
they must maintain control. you can't control chaos. as it should be. now call be a bitch!
@jae@phnt >really weird that these people don't want "ai" or "bots" to crawl their sites, yet they really thirst for attention.
They want attention but also the ability to micromanage every bit of attention. Hence; why you get people who go crazy from everything that someone says about them online.