Conversation

Notices

1. Embed this notice
   🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Wednesday, 12-Mar-2025 03:44:37 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

   • Daniel Supernault

   Message for #Pixelfed admins

   Do any of you encounter the following? Every 6 hours and consistently, my pixelfed server goes into a high memory usage, process is php-fpm triggered by pixelfed, before being killed by oom. Clearly a bot, never happened before.

   Now I changed server with a full fresh install and new IP but it still goes on (even changed OS from 22.04 to 24.04, bumped RAM from 16 to 32 GB and php8.3 version to 8.4).

   Do you administrators see similar behavior? See attached Grafana/Prometheus dashboard for illustration on a 24-hours period.

   Any clue on how to mitigate such DoS attacks?

   @dansup

   #pixelfedadmin #admin

   • Embed this notice
     Daniel Supernault (dansup@mastodon.social)'s status on Wednesday, 12-Mar-2025 03:44:33 JST Daniel Supernault

     in reply to

     @barbapulpe Looks like it could be a scheduled task, I will look into this.

   • Embed this notice
     Daniel Supernault (dansup@mastodon.social)'s status on Wednesday, 12-Mar-2025 04:30:26 JST Daniel Supernault

     in reply to

     @barbapulpe Yeah sorry about that, I'm working on a fix!

   • Embed this notice
     🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Wednesday, 12-Mar-2025 04:30:27 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

     in reply to

     • Daniel Supernault

     @dansup
     Hey Dan, thanks a lot if you can help!

     I isolated the process eating all the memory, currently 27 GB and still growing, with over 60% of CPU (which in turn is consuming postgres and redis resources also), it is:

     /usr/bin/php8.4 artisan app:account-post-count-stat-update

     Yet my server has 8 vCPU, 32 GB of RAM, runs on Ubuntu 24.04 and Postgres 17 (but is started a few days ago when I was still on 4 vCPU, 16 GB of RAM, Ubuntu 22.04 and Postgres 16, with exactly the same pattern, before I migrated on this fresh install).

     Thanks in advance for any pointers and why this job triggers. It runs every 6 hours and finally gets killed by oom when RAM is full (increases by about 0.5 GB by minute so it takes almost one hour before it gets killed).

   • Embed this notice
     Daniel Supernault (dansup@mastodon.social)'s status on Friday, 14-Mar-2025 05:22:16 JST Daniel Supernault

     in reply to

     @barbapulpe Hey, yeah I'm going to refactor that to use less resources

   • Embed this notice
     🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Friday, 14-Mar-2025 05:22:23 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

     in reply to

     • Daniel Supernault

     @dansup
     Hey Dan. Had a look into the code, didn't see anything obvious but I am no expert in PHP/Laravel.

     Fact:

     The scheduled job (every 6 hours) App/Console/Commands/AccountPostCountStatUpdate.php is the culprit. Eats up all RAM (28+ GB) and gets oom killed.

     Hypothesis:

     Nothing changed on my side, but I did get a burst of requests from a rogue server which I blocked at Iptables level. My impression (can't be sure) is that it happens since. Could it be that status updates are too many and hog the job which gets killed before it can finish?

     Proposal:

     - App/Services/Account/AccountStatService::getAllPostCountIncr gets called with no limit (-1), could this induce too big a loop?

     - Could we limit this loop to a number that is manageable by just setting limit = 10 000 (or whatever number) in this routine? Would this have other consequences?

     Shooting in the dark here as I'm not expert here, any suggestions welcome!

   • Embed this notice
     🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Friday, 14-Mar-2025 05:22:29 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

     in reply to

     • Daniel Supernault

     @dansup
     Nope, applied it, didn't change anything.

     So the issue is clearly isolated : it's the scheduled task eating all memory every 6 hours before getting oom killed: /usr/bin/php8.4 artisan app:account-post-count-stat-update

     Since it eats up 28+ GB of RAM, adding more is not the solution.

     Is there a way to limit the memory used by this job without blocking subsequent php processes?

     Why did this suddenly occur despite not changing anything on my side? Pixelfed is at 0.12.4 version and was not updated recently on my side.

     Thanks! Am I the only one to experience this issue?

     #pixelfed #pixelfedadmin

   • Embed this notice
     🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Friday, 14-Mar-2025 05:22:36 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

     in reply to

     • Daniel Supernault

     @dansup
     Hi Dan. I see you committed changes to Apiv1Controller.php, is this a fix for this issue? Thanks.

   • Embed this notice
     Daniel Supernault (dansup@mastodon.social)'s status on Friday, 14-Mar-2025 18:28:22 JST Daniel Supernault

     in reply to

     @barbapulpe I did ship a fix in https://github.com/pixelfed/pixelfed/commit/134eb6324ed44cf85f90780053917fa9ced6918e

   • Embed this notice
     🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ (barbapulpe@gayfr.social)'s status on Friday, 14-Mar-2025 18:28:23 JST 🌈 BarbaPulpe 😇 ᴹᵃˢᵗᵒᵈᵒⁿ

     in reply to

     • Daniel Supernault

     @dansup
     Great, thanks.

     Now my issue is solved! This is how I achieved it and I'm unsure why this actually worked, but it appears the problem has disappeared.

     1. Updated the code in account-post-count-stat-update to call getAllPostCountIncr(1000) with an arbitrary limit.

     2. Executed manually the updated routine, took about one minute but executed successfully.

     3. Restored the original code in account-post-count-stat-update to call getAllPostCountIncr() with no limit.

     4. Now it executes in less than a second, and the scheduled job no longer hogs any resources.

     Maybe this eliminated a rogue entry in DB/Redis which led to an infinite loop?

     I will monitor the situation but it seems ok so far.

     Thanks for your answers, and many thanks to all your contributions and efforts to the Fediverse!