Conversation

Notices

1. Embed this notice
   Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Friday, 08-Nov-2024 18:46:30 JST Stefano Marinelli

   I woke up this morning:

   - I have a bit of a headache
   - The BSD Cafe snac instance had crashed, so it was down - I modified the rc.d script to run under daemon(8) to restart automatically (and I sent a PR to include it)
   - a critical virtualized Windows server for a client was in total crash (resolved by doing a stop/start from the Proxmox console, but this will also move to bhyve soon)
   - I spilled half of my coffee on the floor - and being an espresso, I have very little left

   I’m waiting for the locusts...

   #TGIF #snac #BSDCafe #Proxmox #bhve #coffee

   • Embed this notice
     zirias (on snac) (zirias@snac.bsd.cafe)'s status on Friday, 08-Nov-2024 18:46:26 JST zirias (on snac)

     in reply to

     • vermaden
     Seriously? The purpose of daemon(8) is to run something as a daemon that actually isn't one. And I'd define a daemon as "a program able to fork itself into background while properly detaching from the current session and terminal, and optionally also handling a pidfile". Such programs should never be wrapped into daemon(8).
     
     Now, if snac can't do that by itself, it's fine. But the "automatic restart" feature of daemon(8) is IMHO very questionable, because it's a workaround just hiding a problem that could go worse. Imagine a broken program that, while triggering broken behavior, deletes or corrupts important data. You don't want that to automatically restart, giving it the chance to wreak more havoc.
     
     CC: @vermaden@bsd.cafe
     
   • Embed this notice
     Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Friday, 08-Nov-2024 18:46:27 JST Stefano Marinelli

     in reply to

     • vermaden

     @vermaden I agree. I created that snac rc.d script one year ago, and didn't use daemon to keep it simple. But there's no reason to avoid using it. I'm going to modify all my rc.d scripts and ALWAYS use daemon

   • Embed this notice
     vermaden (vermaden@mastodon.bsd.cafe)'s status on Friday, 08-Nov-2024 18:46:29 JST vermaden

     in reply to

     @stefano

     That always amaze me - WHY FreeBSD DOES NOT use daemon(8) more - especially in /etc/rc.d startup scripts - to make sure the service is running ...

   • Embed this notice
     zirias (on snac) (zirias@snac.bsd.cafe)'s status on Friday, 08-Nov-2024 19:07:14 JST zirias (on snac)

     in reply to

     • vermaden
     Yes, that's the problem daemon(8) solves. You shouldn't write rc scripts that just fire something that isn't a daemon "into background" indeed. Just wanted to clarify you also shouldn't use daemon(8) on, well, actual daemons.
     
     In my personal opinion, the auto-restart is a misfeature here, it's a completely different concern and belongs into some kind of "service manager/supervisor" instead. And there, it should be used with care, as a temporary workaround for crashes already analyzed, but not fixed yet. Unfortunately, we're trained to just accept broken software more and more (e.g. systemd allows for easy respawning, even whole containers are just respawned), forgetting about the risks attached. 😞
     
     On a side note, this discussion gives me an idea for a blog article: Clearly define the terms "daemon", "service" and "server", so people stop confusing them 😁
     
     CC: @vermaden@bsd.cafe
     
   • Embed this notice
     Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Friday, 08-Nov-2024 19:07:15 JST Stefano Marinelli

     in reply to

     • vermaden
     • zirias (on snac)

     @zirias @vermaden I agree. Generally speaking I want to know when things crash, and why.
     Programs should never crash and they should deal with problems gracefully.
     Snac doesn't daemonize by itself, so daemon could be a nice choice here.

   • Embed this notice
     zirias (on snac) (zirias@snac.bsd.cafe)'s status on Friday, 08-Nov-2024 19:19:29 JST zirias (on snac)

     in reply to

     • Joel Carnat ♑ 🤪 :runbsd:
     • vermaden
     A crashing program obviously has undefined behavior, which means it could do anything, including very harmful things. A quite typical outcome would be corrupting its own data. As a responsible operator of services, you'll have a good backup strategy of course, so the impact of such behavior would be limited, but still you should be aware that just restarting something that crashed before you know why and how it crashed is quite risky. You should weigh that risk against the (business?) risk of the particular service being down for a while...
     
     CC: @joel@piou.foolbazar.eu @stefano@bsd.cafe
     
   • Embed this notice
     vermaden (vermaden@mastodon.bsd.cafe)'s status on Friday, 08-Nov-2024 19:19:31 JST vermaden

     in reply to

     • Joel Carnat ♑ 🤪 :runbsd:

     @joel @stefano

     If this is production and You are responsible for the thing to 'just' work - then its probably anyway the first thing You would do anyway - probably checking for free disk space or network problems first.

     Then after production is back online you have time to dig in the problem, check logs, submit bugs.

     Not the other way around.

   • Embed this notice
     Joel Carnat ♑ 🤪 :runbsd: (joel@piou.foolbazar.eu)'s status on Friday, 08-Nov-2024 19:19:32 JST Joel Carnat ♑ 🤪 :runbsd:

     in reply to

     • vermaden

     @stefano @vermaden some say it is not a good idea to automatically restart your crashed daemons without looking at why it crashed first.