Conversation

Notices

1. Embed this notice
   Chris Siebenmann (cks@mastodon.social)'s status on Friday, 22-Nov-2024 01:19:09 JST Chris Siebenmann

   Ah yes, the one problem that our Prometheus based alert system can't send us alert email about: when the central mail server explodes. Who rings the bell to tell you that the bell isn't working?

   • Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Friday, 22-Nov-2024 01:22:23 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • fsniper
     @fsniper @cks Yeah, I have that with my backup emails which tend to be in verbose mode so I can quickly eyeball verify that things went right.
     (That said my machines are pets, not industrial cattle)
   • Embed this notice
     fsniper (fsniper@mastodon.social)'s status on Friday, 22-Nov-2024 01:22:24 JST fsniper

     in reply to

     @cks what about setting up a periodic email and if it does not arrive timely your email server is dead? A dead 'mail'man's switch if you will

   • Embed this notice
     Chris Siebenmann (cks@mastodon.social)'s status on Friday, 22-Nov-2024 01:22:52 JST Chris Siebenmann

     in reply to

     • fsniper

     @fsniper We generally notice core mail server problems fairly fast (we did this time too), but in general I feel humans are very bad at noticing when a regular thing stops happening. To fix this we'd have to build some additional sort of alert notification other than email, and we don't really have that available.

     (In theory we could SMS or something, but the budget for SMS alerting is $0 and I don't think there are any free gateways.)

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Friday, 22-Nov-2024 01:25:57 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • fsniper
     @cks @fsniper Maybe check if their/your phone operator offers an SMS alert system to your own number? At least mine has that option at no extra charge with an HTTP endpoint + token.