volkris
I’m talking about the underlying protocol, ActivityPub itself.
No matter what platform is implementing ActivityPub It has to make all of these connections and send all of these messages that just don’t scale particularly well.
It’s not about the language or how it’s written. It’s about the requirements of the system itself.
helge
@darnell @volkris @gimulnautti @jan @berkes@bitcoinhackers.org @nickapos @Sujiyan
This number should be widely known. Given these servers have together about 40k monthly active users, we get a little over 10$ per user per year.
Suji Yan
@helgek @darnell @volkris @gimulnautti @jan @nickapos cost can be largely reduced if we do more optimization but it’s gonna be harder for the OAuth reason.
Also we don’t cap media size so that’s one reason.
Darnell Clayton :verified:
@volkris @gimulnautti @jan @berkes @helgek @nickapos We definately need to figure out how to operate #Mastodon / #Fediverse at scale.
@Sujiyan revealed that running Pawoo (https://pawoo.net), MSTDN Japan ?? (https://mstdn.jp) & Mastodon Cloud (https://mastodon.cloud) surpassed $1 million over 2 years (legal, staff, servers, etcetera).
?? https://darnell.day/zooming-with-pawoo-net-new-owner-suji-yan
volkris
@gimulnautti @jan @berkes @helgek @darnell @nickapos
Remember, this is not theoretical. This is actual experience hearing from real people running instances and finding themselves having to unexpectedly shell out more money for higher hosting prices that they weren’t expecting.
The protocol requires poorly scaling processing and bandwidth.
And that’s not even getting into expensive design decisions that Mastodon in particular put on top of everything else. For example the intentional decision not to redistribute image previews but instead require each instance to go out and pull its own image preview, duplicating that effort throughout the whole platform.
It seriously sounds like [almost] nobody involved in this from protocol design up through platform implementation gives a second thought to what’s going to happen at scale.
And I may have said it in this thread, but when I was in school for computer science we were hammered with big-O analysis of algorithmic scaling but someone recently told me that’s not emphasized in school these days. It sure looks like that’s the case.
Toni Aittoniemi
@jan @berkes @helgek @darnell @volkris @nickapos I also don’t think it can be so much an issue with raw compute power. That only affects how the server performs within it’s own workloads, not between computers at all.
Network IO takes magnitudes longer than compute. But while doing it, the thread often waits most of the time, doing very little compute.
Maybe a layer of distributed caches? Delivery is the key.
Jan
@berkes @helgek @darnell @volkris @nickapos Rails is not the main reason for the scaling issues. Mainly background jobs where Ruby is just a wrapper for transcoding libs etc. You can scale Puma and Sidekiq horizontal by adding more compute/workers. Same for storage. Meaning its not the language or the framework, its the protocol and overall architecture which leads to more resource consumption.
helge
@darnell @volkris @nickapos @jan
Putting a focus here on #ActivityPub scaling to big instances is misleading. The number of connections involved scales with
1.) the number of posts (duh! if people post twice as much, you should expect twice as much traffic).
2.) The number of instances you have followers on. Each post gets transmitted to the sharedInbox the poster has followers on.
The only obvious thing missing in ActivityPub to reduce the amount of requests is batching.
berkes ? ? ? ?
@helgek @darnell @volkris @nickapos @jan that is purely technical.
For now we are stuck with Mastodon which is a Rails app, which comes with all sorts of performance characteristics. The larger alternatives (elixir+pg, php+pg) rather similar.
Performance and scaling are stuck there now. Regardless of the amount of people or amount of followers, the limits are imposed by the tech stack , today.
So that's where work is needed.
Darnell Clayton :verified:
@volkris @nickapos @jan Let’s say for the sake of arguement that #ActivityPub fails to scale for larger instances, say over one million accounts (I am just using this number as an example).
How would the #Fediverse fix this issue & should it be fixed‽ On the latter I think it should, but not doing so would force instances to remain small.
Suji Yan
@helgek @darnell @volkris @gimulnautti @jan @nickapos but I do agree that activityPub & mastodon are not well designed for scale. It never reached this size (not small size, 1.5m? active user) previously.
Overall decentralization sacrifice efficiency (whether mastodon is decentralized is another Q) but some sacrifices are caused by clumsy decisions in early stage
Suji Yan
@helgek @darnell @volkris @gimulnautti @jan @nickapos also - Fediverse community (now) lack of master of distributed system / decentralization architects imo.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.