Conversation

Notices

Embed this notice
:blank: (i@declin.eu)'s status on Thursday, 13-Mar-2025 21:45:06 JST :blank:
@kirby @p @graf @sun https://github.com/superseriousbusiness/gotosocial/pull/3718 and https://github.com/superseriousbusiness/gotosocial/issues/3723

they fake the stats if you don't respect robots.txt
In conversation about a month ago from declin.eu permalink
Attachments
1. Domain not in remote thumbnail source whitelist: opengraph.githubassets.com
  
  [feature] Add `instance-stats-randomize` config option by tsmethurst · Pull Request #3718 · superseriousbusiness/gotosocial
  
  Description If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements. If this is a documentation change, please briefly describe w...
2. Domain not in remote thumbnail source whitelist: opengraph.githubassets.com
  
  [feature] Change `instance-stats-randomize` to `instance-stats-mode` with multiple options · Issue #3723 · superseriousbusiness/gotosocial
  
  In discussion of #3718 it was pointed out that admins may wish to just serve 0 for all stats instead of serving accurate stats or random stats. We could do this by changing instance-stats-randomize...
- pistolero likes this.
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Thursday, 13-Mar-2025 21:45:05 JST dilbert 1
  in reply to
  @i @kirby @p @graf
  
  > If they want accurate user + status counts, they can parse the html of / if they want
  
  :picardfacepalm:
  
  In conversation about a month ago permalink
  
  narcolepsy and alcoholism :flag:, Phantasm and pistolero like this.
- Embed this notice
  Phantasm (phnt@fluffytail.org)'s status on Thursday, 13-Mar-2025 22:03:51 JST Phantasm
  in reply to
  @i @kirby @p @graf @sun Yeah, they took inspiration from nonce Sharkey instances.
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 06:36:37 JST pistolero
  in reply to
  @i @kirby @graf @sun
  
  > if you don't respect robots.txt
  
  They do not understand robots.txt.
  
  In conversation about a month ago permalink
  
  ✙ dcc :pedomustdie: :phear_slackware: and Phantasm like this.
- Embed this notice
  anime graf mays ?️? (graf@poa.st)'s status on Sunday, 16-Mar-2025 06:36:43 JST anime graf mays ?️?
  in reply to
  @i @kirby @p @sun lmfao
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 06:37:19 JST dilbert 1
  in reply to
  @p @i @kirby @graf why should bots honor robots.txt when fedi servers don't
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 06:47:32 JST pistolero
  in reply to
  @benis_redux @i @graf @kirby @sun It's retarded. It is not how robots.txt works.
  
  Here's the draft of the blog post; I hoped it would clarify things. The only reason I haven't published it yet is that it included links to the "FSE Meets the FBI" article, which I have not published yet. (That one will take some work to finish up; the current draft has some events going in the wrong order, I have to correct them, so it is about 4k words.) This is a draft so I might change some of it, and there are some links missing, and my markdown format is idiosyncratic because it is half Ward's Wiki and half RedCloth.
  about-fedilist
  In conversation about a month ago permalink
  Attachments
  1. about-fedilist
    
    Invalid filename.
  and Phantasm like this.
- Embed this notice
  idk anymore, man (benis_redux@cawfee.club)'s status on Sunday, 16-Mar-2025 06:47:33 JST idk anymore, man
  in reply to
  @i @kirby @p @graf @sun BASED
  
  In conversation about a month ago permalink
- Embed this notice
  Phantasm (phnt@fluffytail.org)'s status on Sunday, 16-Mar-2025 06:48:33 JST Phantasm
  in reply to
  @sun @i @kirby @p @graf Because they don't understand this cat and mouse game. They will poison API data, because a filthy bot scrapes that, but than proudly display the real stats on /.
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 06:59:08 JST dilbert 1
  in reply to
  @p @benis_redux @i @kirby @graf I don't have the right to complain about data poisoning considering I was probably the first person to ever do it on the fediverse
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  anime graf mays ?️? (graf@poa.st)'s status on Sunday, 16-Mar-2025 07:04:44 JST anime graf mays ?️?
  in reply to
  @sun @p @benis_redux @i @kirby It's funny when we do it. Mine was poisoning that bird makeup instance because he was misrepresenting him user agent and aggressively scraping accounts so I fed all his accounts the tweets from the Israel account. Good times
  
  In conversation about a month ago permalink
  
  Phantasm, dilbert 1 and pistolero like this.
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 07:06:03 JST dilbert 1
  in reply to
  @graf @benis_redux @i @kirby @p yeah, the way I did it was, there was a bot that just reported the version string of your server software every time you updated. so I edited the PHP to make the version string the daddies cummies copypasta, which the bot then obediently posted to all its followers.
  
  In conversation about a month ago permalink
  
  Phantasm and pistolero like this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 07:09:55 JST pistolero
  in reply to
  @sun @graf @i @kirby See my rationale in the blog post text file. TL;DR: robots.txt is not for this.
  
  In conversation about a month ago permalink
  
  Phantasm likes this.
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 07:10:21 JST dilbert 1
  in reply to
  @p @i @kirby @graf I agree actually, robots.txt is for web scrapers not protocols that use http
  
  In conversation about a month ago permalink
  
  Phantasm and pistolero like this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 07:13:25 JST pistolero
  in reply to
  @sun @benis_redux @graf @i @kirby Well, there's malicious data poisoning and then there's IDIFTL reasons and the latter is always justified.
  
  If you have a giant thread full of hand-wringing about how making a list of fedi servers is basically the same as doing a Holocaust, then it is safe to assume that the lulz have not been conslutted.
  1185613622217.png
  In conversation about a month ago permalink
  Attachments
  1. 1185613622217.png
    https://fsebugoutzone.org/media/a63733a2-6369-4078-afb7-a583e3f4726d/1185613622217.png?name=1185613622217.png
  ✙ dcc :pedomustdie: :phear_slackware:, Phantasm and dilbert 1 like this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 07:15:21 JST pistolero
  in reply to
  @sun @graf @i @kirby "My robots.txt says you aren't allowed to fetch robots.txt!"
  
  In conversation about a month ago permalink
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 07:18:30 JST dilbert 1
  in reply to
  @i @kirby @p @graf its still an api. I am on the fence about this because it seems valid to exclude api endpoints in a robots.txt for safety.
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  :blank: (i@declin.eu)'s status on Sunday, 16-Mar-2025 07:18:31 JST :blank:
  in reply to
  @sun @kirby @p @graf mastodons api is not part of the protocol
  
  In conversation about a month ago permalink
- Embed this notice
  dilbert 1 (sun@shitposter.world)'s status on Sunday, 16-Mar-2025 07:19:03 JST dilbert 1
  in reply to
  @i @graf @kirby @p one thing I am sure of though is that ap server indexing should not care about robots.txt
  
  In conversation about a month ago permalink
  
  pistolero likes this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Sunday, 16-Mar-2025 07:28:26 JST pistolero
  in reply to
  @i @benis_redux @graf @kirby @sun
  
  > how is your instance metadata search engine, that indexes /api/'s meant for clients, not fit for robots.txt exactly?
  
  "halp whois ignores robots.txt"
  
  Read RFC 9309. I am not indexing web pages. FediList is not a search engine. It does include search but I don't think most people use it. The implication seems to be that if I take out the search box, it's fine to ignore robots.txt.
  
  Consider whether anything you have to say about this leads to an absurd conclusion.
  
  > but what other option do they have
  
  You could read the thing or you could read one paragraph of the thing and then ask me questions answered in other paragraphs of the thing.
  
  > you not caring is fair, but them not caring is fair too
  
  I was responding to them. I don't think they have to care. They are incorrect about why it exists and what it is supposed to do, so for the most part I have laid out my rationale. The rationale was laid out with the hope that this would assuage their fears that I plan to do fediverse Holocaust and that communication could make things work out better. I can't tag them on fedi: they all block FSE. So here's a blog post and they can look at the blog post. They don't have to change their minds and I don't plan to fight human nature, but I have provided information about why I do what I do.
  
  In conversation about a month ago permalink
- Embed this notice
  :blank: (i@declin.eu)'s status on Sunday, 16-Mar-2025 07:28:27 JST :blank:
  in reply to
  @p @benis_redux @kirby @graf @sun >robots.txt is intended to avoid content being indexed
  >The search engine exclusion standard doesn't apply.
  
  how is your instance metadata search engine, that indexes /api/'s meant for clients, not fit for robots.txt exactly?
  
  it's a weak attempt at signaling an opt out, but what other option do they have given that client compatibility would break if it were behind auth
  
  you not caring is fair, but them not caring is fair too
  
  In conversation about a month ago permalink

Public

Conversation

Notices

Feeds