Aral Balkan
I hope I never have to rely on my Mastodon instance in an emergency because given the state of my Sidekiq queue, I’d likely have been dead for several hours before anyone even hears my cry for help :)
(I say this with a smile but it’s actually really quite concerning that I can’t rely on it being real-time. Going to have to see what I can do about it. Can’t really afford to spend more than the ~€50/month I’m spending on my instance of one and my posts are routinely delayed by hours these days.)
Jackson Palmer
@aral @dominick @mastohost I'd be curious to know how many outgoing HTTP requests per second are occurring and whether *that* is the bottleneck vs. Sidekiq itself.
It's hard to get away from chattiness if you want to remain decentralized - but 3k requests per second should be a walk in the park even on fairly modest hardware.
Masto.host
@ummjackson @aral @dominick There are more things going on than just sending 3k requests. For example, if a new user interacts with the post, the profile needs to be "federated", meaning avatar/header/bio/keys/pinned posts need to be copied. Also, media files (avatar/header/pinned media) need to be converted to make sure they have the correct dimensions/format. Also, remote servers can be slow to respond and threads need to wait for a response, that also does causes the process to be slower.
Jackson Palmer
@dominick Wow, that's unsustainable and a huge problem.
Jackson Palmer
@dominick Actually... reading into this a bit more. Sending 3k HTTP requests shouldn't be a lot of work, this sounds more like a Sidekiq issue specifically. Hmm.
Aral Balkan
@ummjackson @dominick Yeah, it’s one that doesn’t seem to be easy to solve apparently.
CC @mastohost
Jackson Palmer
@aral It looks like you're running a single user instance, how could performance be that bad? Is it because of your # of followers?
Dom DeLorenzo
@ummjackson Not to respond for Aral, but yeah, basically. Aral wrote about the phenomena himself back in November: https://ar.al/2022/11/09/is-the-fediverse-about-to-get-fryed-or-why-every-toot-is-also-a-potential-denial-of-service-attack/
Masto.host
@aral @matt Yes, it's a single process because it's limited to 20 threads. On my testing only with 30 or more threads does creating multiple processes make a difference.
Aral Balkan
@matt Not sure. I believe so. It looks like the tasks are handled in batches of ~20 at a time.
CC @mastohost
Matt :idic: 🌌
@aral are you running concurrent processes to consume the queues? default is 1 for all queues.
https://www.kujoe.blog/2023/01/optimizing-sidekiq-for-mastodon-40x.html
Masto.host
@ummjackson Yep, I see Sidekiq threads waiting for media encoding and remote server responses. So, yes I do think they are synchronous.
Jackson Palmer
@aral @mastohost @dominick Those extra background tasks make sense, but are you saying the architecture is such that those operations block concurrent processing of outgoing HTTP requests? I imagine performance could also be improved by not sitting around waiting for a response on all 3k of those HTTP requests.
Aral Balkan
@mastohost @ummjackson @dominick Yeah, I don’t think it’s a coincidence that we see this every time there’s an influx of new people. It’s definitely not just about my server.
Jackson Palmer
@mastohost @aral @dominick That feels like an optimization waiting to happen? I had assumed things were either happening concurrently/async or being offloaded to different queues...
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.