Preface: This is a long, technical post so I apologise in advance. Please avert your eyes if such things offend you.
Hi everybeings!
Overview
@Ada and I have discussed many areas where we see problems in the way the Fediverse currently works and where would like to see improvements made to improve safety of the fediverse.
Safety on the fediverse is one of the topics we hold most dear.
One of these areas (and one that has recently garnered a lot of attention) has been in the area of full-text search, and despite the option for account-wide no-crawl options in many pieces of fediverse softwares, this option is not-federated, non-specific, non-granular and not-for-purpose with regards to fediverse searching. It's specifically designed and worded for crawler bots at a html scraping level, and while we could repurpose it for fedi-searching, it just doesn't feel right or quite fit.
There's been a lot of talk on a lot of different levels out there, but nobody's come to a concensus. A lot of people are talking about how we should do it and what standards we should use, and generally making the whole concept a lot more complicated than it needs to be.
In this post, I want to share with you some of the ideas that I've gathered through my research, and implemented in a way that is simple enough (KISS), not overly complicated (YAGNI) yet still fit for the purpose I need as both a software engineer and an instance admin, and that anyone accessing the content will need.
The idea in sharing this is not to convince me that it shouldn't be done, or to create the absolutely most perfect solution possible that's going to take 20 years to build and will be outdated by the time we get there.
This will be getting built and going out in weeks, not months, not next year. It's needed now. This is my current plan how to implement something we needed months ago, now. This is your chance to change my mind and help refine this plan before I start really coding it up.
Technical/implementation details
A lot of these options will be settable as a default value in your settings as well as at an individual post / file level during composition or afterwards at editing stage.
In terms of inbound federated AP objects which do not support these new fields, we will try to infer the intent based off existing AP fields and other metadata when present.
Specifically, we will be using the Mastodon Account Lookup API (GET /api/v1/accounts/lookup?acct=) to get and store the noindex flag during actor creation/refresh. This will also allow us to put the noindex meta tag on any HTML pages containing that actor's posts.
Searchability
So you don't want people to be able to search for you. But is that everybody? Maybe you want your followers to be able to find your posts? Maybe the people you mention should be able to find your post? What about actual users on your local instance? Maybe a particular post you never want to appear in searches? Maybe a particular post you write contains stuff you want everyone to see?
We will be adding a global default and per-post override federated in AP as hk:searchableBy which will be settable to public, private or a combination of followers, mentions or local. This is different to the currentvisibility in that you can have a public visibility post that is searchable by followers and mentions for example. {
...
"searchableBy": ["followers", "mentions"]
} Interactivity
The other problem is that at the moment you can't control who can reply, boost, vote and/or react to your posts.
To this end, we will be adding per-post overrides federated in AP as hk:canReply and hk:canInteract which will allow you to specify like on searches a combination of public, private, followers, mentions and local for replying to, and boost, voting or reacting to your posts respectively. {
...
"canReply": ["mentions"],
"canInteract": ["public"]
} Licensing
Licensing and attribution can get pretty tricky. On the fediverse, we're assuming that if you're making a post, then you have the rights and willingness to let the content be federated. If you didn't you wouldn't post.
However posting content sometimes requires you to provide the license and attribution that you're using along with the content. At the moment there's no reasonable or standardised way to provide this information on an image or in a post, thus you're in breach if you post a CC-BY licensed image.
So to alleviate this, we will be providing 2 federated AP fields hk:licensing and hk:attribution. The licensing field will contain the URL of the license under which the content is being shared and the attribution will contain any links to the source content/s and/or creator/s. {
...
"license": "https://creativecommons.org/licenses/by-sa-nc/4.0/",
"attribution": [
"https://github.com/supakaity/icons"
]
} Crawlability
So crawlability is about indexing of the content by external parties whether they be search engines or fedicrawlers. It will provide a per-object level flag under the AP field hk:crawlable that lists whether the item may be processed by a bot. The possible values are true or false. {
...
"crawlable": false
} Quotability
Now this is a controvertial one. Sometimes people out there don't mind their posts being boosted, but don't want them to be quoted. You know what we say? If you don't want your posts quoted, we should respect that and not allow people to quote you. So we'll be supplying an AP field hk:canQuote that lists who can quote your posts using the same combination of actor types as above. {
...
"canQuote": ["followers"]
}
@supakaity@ada Great work! To be clear, this is something you are proposing to be added to #hajkey right, federate it out, with the hope that other software will add implementations that respect this?
@supakaity@ada > In terms of inbound federated AP objects which do not support these new fields, we will try to infer the intent based off existing AP fields and other metadata when present. > Specifically, we will be using the Mastodon Account Lookup API (GET /api/v1/accounts/lookup?acct=) to get and store the noindex flag during actor creation/refresh. This will also allow us to put the noindex meta tag on any HTML pages containing that actor’s posts. I guess if the default is not to index. If either an Instance dont use Mastodon or does not implement those flags, one should assume that the user dont want to be indexed. If its intended like this, then im happy.
I dont have an opinion about the Searchability rn but i do have one about Interactivity: > The other problem is that at the moment you can’t control who can reply, boost, vote and/or react to your posts. If you look at the protocol rn, it doesnt make sense to have something like this in the protocol. The current "Interactivity" is built upon the bounds of the protocol... For example the visibility: Public: anyone can fetch the post Follower Only: only servers with followers on it can fetch the post DM: only servers with mentioned accounts can fetch the post Local: the post isnt federated
The only (quite weird) exception here is unlisted: it will get pushed to followers but not to relays and the post wont be shown in lists... That one actually surprises me that it got implemented on a relative wide range tho...
So back to the Topic: Imho is a flag to constraint the replys and Interactions a very bad thing tho. You cant be sure that the receiving server does use the flag correctly or even implement the functionality in the first place. Either that can be intentionally (There were servers which rewrote posts from DMs to Public) or it can be unintentionally like Wildebeest did it or that they just didnt implement it yet. So therefore you need to assume that servers ignore those flags and you need to check those constraints on the local server... In such a case it would lead to a toxic system, like all could fetch a post but the creator wont see all messages... It would lead to way more broken threads than today already exists... And if we talk about beeing in control of the own posts, how do you plan to constraint replys of a thread? Beside of that, how do you wana explain the downside of such things to a user which has no idea how federation works? I hope i could show you why i dont like the Idea of these Proposals... I would prefer to have the protocol as a boundary and not some flags.
I dont feel like talk about Licensing rn but let me add a few words to Crawlability: The same issue with the flags above does also exist within the Crawlability and the Searcheability (mentioned as first Topic)... If you look it the way like Security Peoples does, you need to assume that every public Post will be crawled at some point. It can be a Hint/Flag for a Publicly available Fedi Search Engine and Users could be in control what actually is indexed by them... But dont forget that you need to assume that fedi is crawled and that such Flags also can lead to false assumptions.
So, now to the last thing: Quotability I guess i do have a kinda strange View on this Topic: Quotability is a UI Feature. You cant control that people take Screenshots or Post links to other Posts. A good UI just resolves the link to a different Post and shows the original Post like they generate a Preview of any other link to any other Webresource... If you look at it from this perspective, it is incredibly naive to think that such a flag will ever catch on in the whole network.
All in all, such flags could work on a limited reach tho: Take mastodon.social as an example... If there is a big server and users dont communicate with the rest of the network, such Features and (other Features like Closing Threads and other stuff what Gargron wants to implement) they could be used to moderate and control things but only limited to the local server. I would even consider that realistic, but not for the whole network.
There is one side on having those advisory mechanisms and beeing able to use them and there is the others like having an outdated instance, instances that wont use it or instances that those features simply not exist. Im very worried about such whishes since it is hard to explain that to users and it is impossible to enforce it on the whole network which also lead to (i wouldnt say it like that but i dont know how to express it better) false advertising of features.
> We will of course highlight (or lowlight rather?) bad actor’s actions in the thread view and probably add options to toggle them hidden, collapsed or shown (depending on user preferences), on the main instance That could be difficult... that would mean that a big bunch of software would be marked as bad actor although they can do nothing about it.
> and maybe not even include their actions in the conversation list for onward federation. removing it from the conversation could make things worse and tbh, im not sure what you mean with removing it for onward federation... take a look at misskey, as far as i know, misskey dont even advertise other replies to an instance so fetching / receiving other posts isnt something you can control in the current version of the protocol. To enforce rules, the owner instance would need to act as a proxy for the whole thread... In my big answer I also referred to quotes, it is technically not reliably possible to receive a quote in your own instance at all.... So the whole topic questions your thoughts if bad actors should be marked accordingly... Once apart from the fact that instance blocks exist, who would appear in this list? Instances that are not compatible and are not conscious bad actors and one possibly cant do nothing against it, since even when a instance uses the same software, they could be on a different version which handles certain things different and are therefore marked as a bad actor... In such cases it would become to a very toxic feature then.
> But really, enforcement is not the point at all. At the moment there’s no way for the author to even tell us any of this stuff so that good actor’s can act good. But thats the big issue, even enabling the authors to define such things could lead to false assumptions and one need to explain every author how the system works in the first place...
Moreover, it has not yet been discussed to what extent the answers to a thread should take over these flags. I can well imagine that especially the restriction that not all can answer is generally very difficult. You would either have to accept that the replies have their own/different values for the flags, or you would have to share whole follower/blocklists to be able to restrict a post really cleanly.
These are advisory mechanisms that allow you to express your wishes.
We want to be good citizens, we want our users to be able to be good citizens. At the moment there's no real way for someone to tell us how they want their content treated.
We will apply them on our instance.
Hopefully other well behaved instances will as well.
Deliberately misbehaving has other recourses available.
We will of course highlight (or lowlight rather?) bad actor's actions in the thread view and probably add options to toggle them hidden, collapsed or shown (depending on user preferences), on the main instance, and maybe not even include their actions in the conversation list for onward federation.
But really, enforcement is not the point at all. At the moment there's no way for the author to even tell us any of this stuff so that good actor's can act good.
@supakaity@blahaj.zone@Ada@blahaj.zone Not opposed to any of this, very exciting! Just wonder how you plan to deal with quotes, because there isn't really a practical way to prevent people from quoting. Even if you manage to get your instance and compatible federated software to "prevent-quote", other instances which do not support the fields will still be able to quote, you just won't see it, which is arguably worse... And even if you did completely prevent quoting, nothing prevents us from posting a link and relying on link embedding to do the job... Mastodon-style "quoting" has been a thing since forever, it just wasn't called quoting...
Imo, bad actors will bad act... you just ban them... that's what you do... As for no-crawl, I really like the idea of federating no-crawl, but again, you do rely on everyone else implementing your no-crawl field, if I understood your post right...
@cody@ada@supakaity > I was thinking maybe you could have a toggle to allow federating only to instances that support the features you’re selecting, i.e. if the user selects they don’t want crawlers, only federate to instances that announce no-crawl federation support :blobcatgooglynotlikethis: thats a very very bad idea, if you are interested in why, read my other 2 posts
@supakaity@blahaj.zone@Ada@blahaj.zone That sounds like a very good approach, honestly I really hope this gets widely implemented, also, for enforcement (I know it's not the point but it doesn't seem like a bad idea), I was thinking maybe you could have a toggle to allow federating only to instances that support the features you're selecting, i.e. if the user selects they don't want crawlers, only federate to instances that announce no-crawl federation support (? Anyway, just giving ideas, I'd honestly love to help develop this stuff but I'm literally too dumb to even migrate my dockerized misskey instance to calckey, so ye xd
@supakaity@ada@cody I get the feeling you might have interpreted certain things wrong... There is a spectrum of bad actors to good actors, obviously if it is too much, they will be blocked on instance/user level All my points are more related to - Does using a different Version/Software makes someone a bad actor and what is bad? - Is it possible to fulfill those things withing the specification / the network? - Will it harm more than it solves things? - Is it User friendly / understandable?
> I don’t know your wishes, because there’s no way for you to tell me that you don’t want me to be able to full-text search your posts. Yes i agree. Currently the reality and the current Boundaries do match in the perspective of Security, Simplicity and how it is implemented. Flags lead to multiple disadvantages (Security, Simplicity, False Assumptions, etc) which i described.
> How do I be a good actor if I don’t know how? People use their profile page for that...
You're absolutely right in that it's not going to be possible to stop actors deliberately doing bad things on the fedi.
The whole point is that at the moment, as a good actor on the fedi. I don't know your wishes, because there's no way for you to tell me that you don't want me to be able to full-text search your posts, or that you're not interested in my opinion on a particular topic if I wasn't mentioned.
@Jain@blob.cat@ada@blahaj.zone@cody@misskey.codingneko.com Moreover, it has not yet been discussed to what extent the answers to a thread should take over these flags. I can well imagine that especially the restriction that not all can answer is generally very difficult. You would either have to accept that the replies have their own/different values for the flags, or you would have to share whole follower/blocklists to be able to restrict a post really cleanly.Just like you can't expand visibility on replies, you shouldn't be able to expand rights on replies. There's a whole bag of worms waiting under the "but should you be able to futher restrict on replies" and the answer is probably yes... but that's gonna make my brain sore when I start thinking about it...
@cody@ada@supakaity ok but technically you cant assume that the very friendly other instance with potentially the same software has the exact same implementation, that it actually works and that it wont lead to false assumptions on your and the other servers... I gave an example in a other post: Cloudflares Wildeest https://github.com/cloudflare/wildebeest/issues/303
@Jain@blob.cat@ada@blahaj.zone@supakaity@blahaj.zone I mean it's just an option though, I wouldn't personally mark most of my posts for that, but the same way you can send instance only notes, I don't see an issue with having an option to send "supports X" instances only option
@toni@supakaity@ada@cody Here a few flaws: > I also think it is a misunderstanding to think users will even need to be aware of this beyond selecting their preferences in personal settings. - A Bad Actor could do anything over Activity Pub. - A Bad Actor could do anything without Activity Pub. - One need to assume that everything possible is already done > Once it is implemented and supported by Calckey, ALL updated Calckey instances will support it by default. Once other fediverse adds support to it, ALL updated instances of THAT software will support it be default. - Only the ones which are updated. - One cant assume that complex Software have no flaws - One cant assume that every Software has the same implementation - One cant assume that every Software will ever implement things > Getting Mastodon to add support for it will be the biggy Ok thats very subjective: No, mastodon has already implementations which arent thought till the end and they plan to add more things like that. > but once it does this will pretty much become a standard, - No, to get a common ground, things need to be specified in the official specifications which defines the standard > and individual users will not really be able to bipass it - Individual Users can copy a link and post that = Bypass - Individual Users can open the other instances website = Bypass > unless they self-host and are prepared to hack the software - no, selfhosting or buying a instance is enough for example to change a posts visibility > but given they’d likely be blocked once found out, it is hard to see why they would even bother to go to the effort. As you can see, there are enough possibilities for good people which uses good software to unintentionally became a "bad actor". All in all you cant trust a checkbox to solve such problems, everyone who does has false assumptions.
@supakaity@Jain@blob.cat@ada@blahaj.zone@cody@misskey.codingneko.com I also think it is a misunderstanding to think users will even need to be aware of this beyond selecting their preferences in personal settings. Once it is implemented and supported by Calckey, ALL updated Calckey instances will support it by default. Once other fediverse adds support to it, ALL updated instances of THAT software will support it be default. Getting Mastodon to add support for it will be the biggy, but once it does this will pretty much become a standard, and individual users will not really be able to bipass it unless they self-host and are prepared to hack the software, but given they'd likely be blocked once found out, it is hard to see why they would even bother to go to the effort.
Does using software that doesn't support it make you a bad actor? Not necessarily. It may makes you an unaware user. And that's not necessarily bad. Or you may be a user that's deliberatley using old software to bypass these restrictions. It's not an uncomplicated situation, and someone may chose to block you if you keep posting after being informed, that's dependent upon the two user involved. If that user keeps doing it, maybe it goes up the chain. We already have moderation processes to deal with these scenarios.
Is it possible in the current framework? I don't believe so. I've done substantial research into this, so I'm not just pulling this out of nowhere.
Will it harm more than it solves things? There's significant harm being done at the moment, and no movement on solving the issue. Is this the right answer? I think it's a good start.
Is it user friendly and understandable? Explaining the advisory nature of this feature, documenting it and raising awareness of it is a big part of the implementation that needs substantial consideration.
Unfortunately automatically parsing a profile page is a very poor way to determine if I should show a specific federated post on my instance to users who are entering a search term.
I'd prefer to be given a clear direction on if a certain user can see your post in their search results rather than guessing and even worse guessing wrong.
@toni@ada@cody@supakaity a half hearted approach would be [ ] i dont want to be indexed (we cant prevent it tho) [ ] i dont want to be quoted (its impossible to prevent & verify that)
@Jain@blob.cat@supakaity@blahaj.zone@ada@blahaj.zone@cody@misskey.codingneko.com Yeah, but all of the problems you outline EXIST NOW! Implementing these protocols will just make it easier for users to behave nice. Sure not all software will implement these things. So what? Does that mean none of us should? I don't see why not. Maybe its because I've seen how the 'standards' evolved on the web since I began programming it in 1994, but standards evolved out of attempts to unify competing implementations, not the other way around. Sometimes things need to be implemented first, as a proof of concept, and to demonstrate what is being attempted. Yes it can be a messy approach, but getting everyone to agree on it all before hand may mean it never gets done.
@toni@supakaity@ada@cody No they exists forever. Yes imho they should be not implemented because they lead to false assumptions. The way better solution is to define such stuff in your Profile Page. If you think like that, have you already contributed your opinion to the "show me only follower stuff" flag what mastodon implemented and now leads to unnecessary admin bothering? Anyone who knows a bunch of protocols immediately can see a bunch of issues from similiar protocols like smtp and mail bouncing... Yes im in the opinion to leave the protocol as it is because its simpler, it fulfills its purpose & it doesnt lead to false assumptions.
@Jain@blob.cat@supakaity@blahaj.zone@ada@blahaj.zone@cody@misskey.codingneko.com Okay, I see where you are coming from. As a user I disagree because I think I would benefit from tools that helped me be a 'good actor' because I want to be able to just 'hit boost' etc without having to check profiles. And with search feature that preference makes no sense at a user level, because if I see the post in my search results, I've seen it, it makes no sense to expect me to then check the results profile and pretend I didn't see it. Having sites that implement search respect that setting means it would never appear for me. Sure some other user on a site that doesn't respect the setting might find it, but that is the case right now. If this gained critical mass, then sites that don't respect it would soon find themselves ostracised and subject to fediblock. And frankly I would have no problem with that.
@toni@supakaity@ada@cody Im glad we had the discussion then. I hope you wont get the wrong view of me, if such features get implemented i will use them too, im just in the opinion that they should better not exist.
Only thing i dont agree now is the whole fediblock topic...
But i guess i now need to explain that (im trying to get it short): I guess we kinda have similiar values for blob.cat like blahaj.zone does have in the term of our rules. But there is a big difference, the rules are enforced locally but we almost dont block any server, the idea is to give the user the authority to block stuff. This is the only reason why one cant consider blob.cat as a safe space since one have to block domains themself.
Can you assume what this means in terms of fediblock & blocking? I guess you can. Even tho i talk to users / ban them to solve conflicts and unwanted content, we ended up beeing on many blocklists due to simply other admins/mods are impatient / dont verify things / or do not discuss at all. Yes, many instances are legitimately on such blocklists, but one could never show me content on blob.cat which justifies a block.
I guess on certain topics, within features and authority i personally do disagree but due to what happened in the past, im against fediblock because i strongly belive that there shouldnt be an authority on a blocklist and there shouldnt be lazy admins just copying them.