Conversation

Notices

1. Embed this notice
   malevolent dictator for life (astrid@fedi.astrid.tech)'s status on Sunday, 22-Jan-2023 16:28:41 JST malevolent dictator for life
   oh no, my grafana server is shitting itself! i should go check the graphs to see what's causing it.
   
   ...
   
   oh.
   • Embed this notice
     @Dwarf (dwarf@borg.social)'s status on Sunday, 22-Jan-2023 16:28:36 JST @Dwarf

     in reply to

     • rqsd

     @rqsd@borg.social @astrid@fedi.astrid.tech I'm not sure about the requirements for Prometheus but I wouldn't be surprised it's not very lightweight, especially once you connect it to Grafana or Alertmanager and start querying.

   • Embed this notice
     rqsd (rqsd@borg.social)'s status on Sunday, 22-Jan-2023 16:28:38 JST rqsd

     in reply to

     @astrid@fedi.astrid.tech I'm gonna be real with you: If a dual-core 1 GiB of RAM node *isn't* enough to run a monitoring stack, I can't help but wonder what that monitoring stack even does. That seems like plenty of resources.

   • Embed this notice
     malevolent dictator for life (astrid@fedi.astrid.tech)'s status on Sunday, 22-Jan-2023 16:28:39 JST malevolent dictator for life

     in reply to
     perhaps running a monitoring stack on a 2core 1gb free oracle node was a mistake
   • Embed this notice
     @Dwarf (dwarf@borg.social)'s status on Sunday, 22-Jan-2023 16:33:01 JST @Dwarf

     in reply to

     • rqsd

     @rqsd@borg.social @astrid@fedi.astrid.tech Prometheus stores a lot of time based metrics, so crunching that into something that's understandable by humans is probably kinda heavy, especially if you do a lot of calculations and mutations on that data.

   • Embed this notice
     rqsd (rqsd@borg.social)'s status on Sunday, 22-Jan-2023 16:33:02 JST rqsd

     in reply to

     • @Dwarf

     @dwarf @astrid@fedi.astrid.tech Doesn't that seem, like, off to you? What even is it doing?

   • Embed this notice
     @Dwarf (dwarf@borg.social)'s status on Sunday, 22-Jan-2023 16:50:46 JST @Dwarf

     in reply to

     • rqsd
     • 4censord

     @4censord@mstdn.social @rqsd@borg.social @astrid@fedi.astrid.tech comparable yes! I've borked many a query and crashed Prometheus that way, probably user error though :blobcatblep:

   • Embed this notice
     4censord (4censord@mstdn.social)'s status on Sunday, 22-Jan-2023 16:50:47 JST 4censord

     in reply to

     • @Dwarf
     • rqsd

     @rqsd @dwarf @astrid isn't essentially a database server with a very predictable load?
     So depending on how many things you monitor, this might be completely expected.

   • Embed this notice
     @Dwarf (dwarf@borg.social)'s status on Sunday, 22-Jan-2023 16:52:05 JST @Dwarf

     in reply to

     • rqsd
     • 4censord

     @4censord@mstdn.social @rqsd@borg.social @astrid@fedi.astrid.tech and before @rqsd asks, no, not the crappy borg.social setup, but at work on the production cluster. oopsie.