It has been '0' days since a periodically flaky alert check that flickered very briefly every so often and tempted me to disable it was actually a sign of a real problem (intermittent packet loss, causing ping checks to flake briefly).
Conversation
Notices
-
Embed this notice
Chris Siebenmann (cks@mastodon.social)'s status on Friday, 14-Feb-2025 03:05:46 JST Chris Siebenmann
- Haelwenn /элвэн/ :triskell: likes this.
-
Embed this notice
Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Friday, 14-Feb-2025 03:07:34 JST Haelwenn /элвэн/ :triskell:
@cks makes me wonder how you debugged that because finding the cause of issues that happen irregularly can be quite annoying. -
Embed this notice
Chris Siebenmann (cks@mastodon.social)'s status on Friday, 14-Feb-2025 03:18:10 JST Chris Siebenmann
@lanodan The automated ping check is infrequent (~30s) so a failure had to line up with the check, but it turned out actual ping flakes were common enough that mtr in strip mode running from another machine on the same network showed me some '?' markers (which was entirely too many on a local network where other things were fine).
Haelwenn /элвэн/ :triskell: likes this. -
Embed this notice
Chris Siebenmann (cks@mastodon.social)'s status on Friday, 14-Feb-2025 03:21:43 JST Chris Siebenmann
@lanodan Also I would never have noticed this enough if I hadn't put together a desktop tool to pop up to show me the state of our alerts, and the tool showed me pending alerts too, so every so often the tool's check lined up with the metric system's ping check that had failed and a little window flickered irritatingly into existence on my desktop.
Haelwenn /элвэн/ :triskell: likes this.