Conversation

Notices

1. Embed this notice
   Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Tuesday, 23-Sep-2025 17:37:19 JST Stefano Marinelli

   Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.

   The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.

   The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.

   The result:

   - systemd stops Nginx.
   - Port 80 becomes free.
   - certbot, in standalone mode, immediately grabs it for validation.
   - systemd tries to restart Nginx, which fails with "Address already in use".

   The web server was knocked offline by its own certificate renewal script.

   I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.

   Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.

   #SysAdmin #Linux #SystemD #Rant #KISS

   • Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Tuesday, 23-Sep-2025 17:43:32 JST Haelwenn /элвэн/ :triskell:

     in reply to
     @stefano Says more about certbot than systemd though.
     Like web server can just stay up with using the other ACME challenges (which can be DNS or reverse-proxying the acme client), so web server never has to go down.