Conversation

Notices

1. Embed this notice
   sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:27:15 JST sj_zero
   [admin mode] completed some maintenance tasks on the database(hence the inaccessibility today), it shrank the database a shocking amount so I'd call that a win. Should improve performance, I've still got more to do later but it's fine for now.
   • ✙ dcc :pedomustdie: :phear_slackware: likes this.
   • Embed this notice
     sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:27:38 JST sj_zero

     in reply to

     • Soul Dessin
     The database cleanup was just running a couple of built in pleroma scripts that clean up old data (cleaning out 25GB of remote posts!), but the real issue seems to have been a number of faulty packets with invalid addresses being aimed at fbxl social in particular (once I stopped accepting packets for fbxl domains the server came back and once I started again)
     
     My hypothesis so far is that either intentionally due to malicious attack or unintentionally due to a misconfigured server a bunch of these malformed packets were sent my way, and filled up connection slots in the kernel, locking up not just http but telnet and icmp. Once I added some configuration changes to increase the number of connection slots and also to filter any packets with bad addresses (sysctl calls them martian packets since they come from "alien" address spaces) the connection issues ceased.
   • Embed this notice
     ✙ dcc :pedomustdie: :phear_slackware: (dcc@annihilation.social)'s status on Tuesday, 11-Jun-2024 10:27:38 JST ✙ dcc :pedomustdie: :phear_slackware:

     in reply to

     • Soul Dessin
     @sj_zero @souldessin Did you use repack? (if not you should)
   • Embed this notice
     Soul Dessin (souldessin@noauthority.social)'s status on Tuesday, 11-Jun-2024 10:27:39 JST Soul Dessin

     in reply to

     @sj_zero
     I'm curious, what was the core of the issue?

   • Embed this notice
     sj_zero (sj_zero@social.fbxl.net)'s status on Tuesday, 11-Jun-2024 10:29:08 JST sj_zero

     in reply to

     • ✙ dcc :pedomustdie: :phear_slackware:
     • Soul Dessin
     I've had an automated weekly repack for a year or two now, it has a big impact.
     ✙ dcc :pedomustdie: :phear_slackware: likes this.