Aral Balkan
“We rebuilt the Mastodon backend from scratch and made it able to handle Twitter-scale (500M users, 20M writes/second, unbalanced social graph, etc.)”
Congratulations, you reinvented Big Tech.
Did you stop to consider that fediverse (emphasis “diverse”) servers should not be trying to reach “Twitter scale?” That they should be kept small on purpose? That the goal isn’t to recreate Twitter but with some other asshole in charge?
(Of course, it’s a VC-backed startup.)
Aral Balkan
@koen_hufkens It is but they state they rebuilt it from scratch. If so, it doesn’t apply. There isn’t much you can hit them with. These things were always going to happen given that ActivityPub has the same inherent economies of scale that the Big Web has. The only chance the fediverse had of resisting enclosure with its current architecture was social pressure but we’ve all but thrown that away with some of the most prominent folks rolling out the red carpet for Silicon Valley.
Koen Hufkens, PhD
@aral Next question would be: is Mastodon's backend (A)GPLv3. Might be mighty fine to slap VC bros with some licensing.
Aral Balkan
@Paul100 @koen_hufkens Indeed. Thank fuck for Facebook/Meta, Twitter, heck ExxonMobil and let’s not forget the Saudi Royal family… where would be without the billionaires and trillion-dollar corporations?
Paul100
@koen_hufkens@mastodon.social @aral@mastodon.ar.al
If someone wants to become big, why clip his wings?
Opportunities are good.
Aral Balkan
@kdekooter Doesn’t sound good to me in any way.
Kees de Kooter 🍉
@aral if it sounds too good to be true...
Aral Balkan
@snaggen Yeah, that’s exactly what I’m complaining about. Clearly nothing gets by you, Mattias. Good chap! Let’s just add them to long list of VC-funded philanthropically enterprises, shall we?
Mattias Eriksson 🦀🚵♂️⛵
@aral
So, you are complaining that they are not consuming enough electricity and resources per user?
Just think it is a weird thing to rant about, that they claim to be performant. They also seems to intend to go opensource by Aug 29, and if so it sounds like a good thing to have someone using VC money to improve the mastodon ecosystem.
nicolas ⁂
@aral Their terms of service if someone was considering trying out their instance…
Aral Balkan
@nclm Nothing that a VC-funded corporation does is meant to make life easier for the little guy. Factory farms aren’t spas for the livestock.
nicolas ⁂
@aral And yes, they are completely missing the point.
One question is if a server that capable means that it’s lighter and cheaper to run for small instances? In that case it could still be somewhat interesting…
All in all it seems that this is just a tech demo for their “Rama” product, whatever that is.
Aral Balkan
@nil This is not that.
Anil
@aral I think there's a balance. If mastodon could run more cheaply then e.g. community servers would be easier to maintain.
Aral Balkan
@Loeil_du_tigre With all due respect, I’m not your personal idiot whisperer.
L'oeil du tigre :flag_QC: :verified:
@aral what the actual fuck are you on about?? They literally made a much more efficient fediverse server than what we have now and youre out there complaining that their vc money is being used to make cool open source projects. Would you rather they used that money to make a privacy invading, user hostile software that is full of DRM and has a subscription tied to it?
L'oeil du tigre :flag_QC: :verified:
@aral if a company contributes to the Linux kernel, is that bad because they are a corporation and they make profits? You're not making any sense by basically just saying VC bad without any depth to your accusations
Aral Balkan
@myrmidon Yes, you’re right, some of the best engineers in the world work in surveillance capitalism.
This isn’t about engineers. It’s about funding and business models.
myrmidon
@ttiurani @hrefna @maegul @aral l’m reading this thread and may I humbly ask who are we to strike down without any material the work of Nathan Marz? The guy is an engineering legend!
It feels like a bunch of tiktok’ers criticizing Spielberg new movie that nobody has seen yet.
Timo Tiuraniemi
Yes, I'm also interested in the actual numbers and operational complexity.
Personally, for the reasons outlined above, I'd be very surprised that the result will be anywhere close to an optimally resource-efficient and cost-efficient solution to host a server for a small group.
Better than Rails, definitely.
So good that the pros outweigh the cons of making an Big Tech scalable Mastodon server? I'm doubtful.
Hrefna (DHC)
Is a clojure based system going to be better? Maybe, maybe not, depends on a wide variety of factors, but I'd be very curious what their _actual numbers_ are like and what the scaling breakpoints are for them and, at a minimum, their work in this field contributes to a sense of what is possible and what is easy or hard in this space.
That's the important question. Not how much memory java takes up, which is only relevant as a choice after you've added constraints
Hrefna (DHC)
Like we could sit here and break down the exact memory requirements you think would be "reasonable" but the truth is I don't care what you think is reasonable because it is going to be purely theoretical and your entire approach goes against how the cost of these systems scale anyways, as well as the major cost sinks in this space.
Instead put the focus where it should be: The cost of the systems today is too high for small groups to run. It needs to be improved.
Hrefna (DHC)
I have run JVM based applications for nearly 20 years in production with sizes ranging between "less than a dozen users," millions of QPS, and "we're rewriting it in C++, ouch" I have run applications in Java, Kotlin, scala, and helped maintain systems in clojure.
I am intimately familiar with the overhead required.
I also can pretty much solve a problem of a few hundred users on a social media app with a rolodeck.
You are optimizing the wrong point of the problem.
Timo Tiuraniemi
Fosstodon is *way* too big. Looking into the future, fediverse ought not optimise for that scale.
No JVM GC, including G1, works without a noticeable memory overhead. That directly contributes to the total memory requirements of the server, and it does make a difference that it's by default higher than needed. As you said, there are a bunch of other things on that server as well that use memory.
Hrefna (DHC)
1. There's a huge gap between "big tech" and "100 users." Fosstodon—the server you are on right now—claims 16k users.
2. That is often true, but it _often_ also has to do more with the _rest of the system_ and not just the worker process. For example: caching and dbs.
3. Using the GC is, again, like Z on the list of problems (maybe S) for the scale we're talking about and has been in Java basically since before the G1 became the default five years ago.
Timo Tiuraniemi
Again, the official Rails app isn't good, but I'd be very surprised if this is operationally anywhere close to optimal for the small-to-medium scale. If you build for Twitter-scale, you will make decisions that will almost certainly hurt small scale.
And Clojure relies *heavily* on the GC. Using it is not a good way to build low memory JVM servers.
Hrefna (DHC)
I've run plenty of JVM systems with nice memory footprints per user, especially on modern JVMs. Sure, it isn't the most memory efficient compared to C++, but that is also the least of my concerns when you compare it against:
1. A queue with rails worker processes.
2. A database.
3. An _ephemeral_ database (Redis)
3a. Times 2.
4. A rails API.
etc.
A JVM worker "not being memory efficient" is like Z on the cost list and trivial to lower costs on at small scale.
Timo Tiuraniemi
There's a difference between resource efficiency wrt scaling up vs a constant (small) scale.
What the fediverse could use (as you highlighted) is a resource-efficient server for small-to-medium scale, think max a few hundred users. While the official Rails app has problem, what Red Planet Labs has built is a JVM server that runs Clojure. That by design comes with a large memory footprint, which is not resource efficient if you don't have enough use for it.
Hrefna (DHC)
I'd also like to underscore the point there about the _cost_.
Right now mastodon is prohibitively expensive to run in terms of both time and resources. I'd see efforts to reduce this to be a net positive to the fediverse in most situations?
The lessons at large scale don't always apply at small scale and vice versa, but the point remains that right now mastodon is a bit of a trash fire in that regard and it doesn't have to be.
maegul
Their comments here (https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/#Our_Mastodon_instance) don't seem to me unreasonable.
They echo your point and respond that it's a community decision and shouldn't be a tech limitation (I agree, especially as they're gonna open it). Plus it mostly seems to demonstrate their Rama thing.
Dunno, bringing more tech to the ecosystem is probably a net good. Big and small places can both exist, and big ones, I'd wager, probably have value, so we should make them ourselves no?
Aral Balkan
\@dschwarz Thanks man, lotsa luck to you too doing whatever it is you do.
Daniel Schwarz :toad:
@aral I’ve seen Fediverse gatekeeping but… attempting to gatekeep users of an open protocol? Lotsa luck with that.
Aral Balkan
@andrei_chiffa @mitsuhiko That’s a very different kind of optimisation. And no VC-funded startup is going to be interested in solving that problem.
Andrei Kucharavy
@aral tbh @mitsuhiko had a really good point that Mastodon back-end needs to be rewritten for efficiency so that 10-15$ in compute can run a 1000-10 000 person instance, making it much more easy to spin new instances for communities that want to but do not have the resources for now.
Aral Balkan
@kielkontrovers Nice red herring you have there but I’m afraid I’m not hungry at the moment. Goodbye.
kielkontrovers
@aral do you think killing all mastodon servers would be better than servers that can withstand the threat. I think this is rather stupid. it's a good thing to have other servers and protocols especially if they are more robust.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.