Notices by 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)

@pixelfed
Hello @dansup

Does this release fix this bug? I see the issue is still open... Thanks!

https://github.com/pixelfed/pixelfed/issues/5529

I am using Postgres db.

Or in glitch? @mehdi_benadel

@dansup
Great, thanks.

Now my issue is solved! This is how I achieved it and I'm unsure why this actually worked, but it appears the problem has disappeared.

1. Updated the code in account-post-count-stat-update to call getAllPostCountIncr(1000) with an arbitrary limit.

2. Executed manually the updated routine, took about one minute but executed successfully.

3. Restored the original code in account-post-count-stat-update to call getAllPostCountIncr() with no limit.

4. Now it executes in less than a second, and the scheduled job no longer hogs any resources.

Maybe this eliminated a rogue entry in DB/Redis which led to an infinite loop?

I will monitor the situation but it seems ok so far.

Thanks for your answers, and many thanks to all your contributions and efforts to the Fediverse!

@dansup
Hi Dan. I see you committed changes to Apiv1Controller.php, is this a fix for this issue? Thanks.

@dansup
Nope, applied it, didn't change anything.

So the issue is clearly isolated : it's the scheduled task eating all memory every 6 hours before getting oom killed: /usr/bin/php8.4 artisan app:account-post-count-stat-update

Since it eats up 28+ GB of RAM, adding more is not the solution.

Is there a way to limit the memory used by this job without blocking subsequent php processes?

Why did this suddenly occur despite not changing anything on my side? Pixelfed is at 0.12.4 version and was not updated recently on my side.

Thanks! Am I the only one to experience this issue?

#pixelfed #pixelfedadmin

@dansup
Hey Dan. Had a look into the code, didn't see anything obvious but I am no expert in PHP/Laravel.

Fact:

The scheduled job (every 6 hours) App/Console/Commands/AccountPostCountStatUpdate.php is the culprit. Eats up all RAM (28+ GB) and gets oom killed.

Hypothesis:

Nothing changed on my side, but I did get a burst of requests from a rogue server which I blocked at Iptables level. My impression (can't be sure) is that it happens since. Could it be that status updates are too many and hog the job which gets killed before it can finish?

Proposal:

- App/Services/Account/AccountStatService::getAllPostCountIncr gets called with no limit (-1), could this induce too big a loop?

- Could we limit this loop to a number that is manageable by just setting limit = 10 000 (or whatever number) in this routine? Would this have other consequences?

Shooting in the dark here as I'm not expert here, any suggestions welcome!

@dansup
Hey Dan, thanks a lot if you can help!

I isolated the process eating all the memory, currently 27 GB and still growing, with over 60% of CPU (which in turn is consuming postgres and redis resources also), it is:

/usr/bin/php8.4 artisan app:account-post-count-stat-update

Yet my server has 8 vCPU, 32 GB of RAM, runs on Ubuntu 24.04 and Postgres 17 (but is started a few days ago when I was still on 4 vCPU, 16 GB of RAM, Ubuntu 22.04 and Postgres 16, with exactly the same pattern, before I migrated on this fresh install).

Thanks in advance for any pointers and why this job triggers. It runs every 6 hours and finally gets killed by oom when RAM is full (increases by about 0.5 GB by minute so it takes almost one hour before it gets killed).

Message for #Pixelfed admins

Do any of you encounter the following? Every 6 hours and consistently, my pixelfed server goes into a high memory usage, process is php-fpm triggered by pixelfed, before being killed by oom. Clearly a bot, never happened before.

Now I changed server with a full fresh install and new IP but it still goes on (even changed OS from 22.04 to 24.04, bumped RAM from 16 to 32 GB and php8.3 version to 8.4).

Do you administrators see similar behavior? See attached Grafana/Prometheus dashboard for illustration on a 24-hours period.

Any clue on how to mitigate such DoS attacks?

@dansup

#pixelfedadmin #admin

🇫🇷 J'ai activé un robot pour faciliter la création de comptes XMPP à partir de Mastodon (fonctionne aussi depuis Pixelfed). Essayez ! C'est sympa, la messagerie instantanée.

🤖 Suivez ou mentionnez @xmpp_bot pour créer votre compte.

🔄 N'hésitez pas à le faire savoir et à inviter vos potes pour des interactions plus riches !

🇬🇧 I activated a robot to facilitate XMPP account creation from Mastodon (works also from Pixelfed). Try it out! Instant messaging is cool.

🤖 Follow or mention @xmpp_bot to create your account.

🔄 Spead the word and invite your friends to join for richer interactions!

#xmpp #gayxmpp #movim #gaymovim

@dansup @pixelfed @loops
Great news! What are the main backend server requirements? PHP, Postgres, Redis...?

@dansup
Sorry but definitely no, it goes against a philosophy of decentralization and free software. I am happy to volunteer contributions, not to pay a fee to a central authority. Just my pick, sorry if I sound harsh but it's definitely against my principles.

@dansup
And I guess the underlying server dependencies: postgres, redis,...?

@dansup
Great initiative. What is the technical stack used by Loops?

@dansup
I'm more worried about an exploit on a user (pixel, www-data) which is in the redis group.

@dansup
Do we actually need the ability to edit S3 keys? This is a very rare task and multiplying the key storage to several locations does not seem good for security, to enable a task which does not require to be done from the UI in my opinion.

@dansup
Even the S3 portal UI will only show you the secret once (on creation of the API key). Compromising that key leaves all your files open to leakage or deletion, so I'm very cautious in storing it in a redis cache unencrypted.

@dansup @pixelfed
Great!

A higher priority for me is localization: any way we can help in templating and avoiding to have to edit PHP source code to translate all texts? And fear of redoing some of the job when updating.