Conversation

Notices

1. Embed this notice
   Mauricio Teixeira 🇧🇷🇺🇲 (badnetmask@hachyderm.io)'s status on Thursday, 19-Dec-2024 02:53:15 JST Mauricio Teixeira 🇧🇷🇺🇲

   tired -- write some bad automation and mess up your system

   wired -- use that as a great excuse to confirm that your backups are working

   #HomeLab #Babkup #SysAdminLife

   • Embed this notice
     Mauricio Teixeira 🇧🇷🇺🇲 (badnetmask@hachyderm.io)'s status on Saturday, 21-Dec-2024 22:28:39 JST Mauricio Teixeira 🇧🇷🇺🇲

     in reply to

     The other day I bragged about how a home lab backup was tested due to bad automation, so yesterday I wanted to manually run the backup job on a different server, before doing a different automation. Then I realized that backup had been failing for at least 20 days, after some package updates. Now I need to figure out a *monitoring* system for my backups.

     You can't ever win. Everything needs to be monitored!

     #HomeLab

   • Embed this notice
     ItzTrain :aim_logo: (train@hachyderm.io)'s status on Sunday, 22-Dec-2024 00:22:05 JST ItzTrain :aim_logo:

     in reply to

     @badnetmask The way i accomplished this was that I stated using a cron management software called Dagu! Then I use curl to send me a notification to pushover when the backup start and finish. I use the software to kick off ansible as well. It’s pretty great.

   • Embed this notice
     Mauricio Teixeira 🇧🇷🇺🇲 (badnetmask@hachyderm.io)'s status on Sunday, 22-Dec-2024 00:39:48 JST Mauricio Teixeira 🇧🇷🇺🇲

     in reply to

     • ItzTrain :aim_logo:

     @train
     I'll take a look at it. Thanks!

   • Embed this notice
     Mauricio Teixeira 🇧🇷🇺🇲 (badnetmask@hachyderm.io)'s status on Sunday, 22-Dec-2024 00:59:53 JST Mauricio Teixeira 🇧🇷🇺🇲

     in reply to

     • Awesome New Year Robot

     @StompyRobot
     Yeah, I do have ways to send notifications to myself that do not include email. I just need to figure out what to use to generate such notification.

   • Embed this notice
     Awesome New Year Robot (stompyrobot@mastodon.gamedev.place)'s status on Sunday, 22-Dec-2024 00:59:54 JST Awesome New Year Robot

     in reply to

     @badnetmask it used to be, you could have the monitor send email to yourself, but with cloud email, that likely will be refused.

   • Embed this notice
     Mauricio Teixeira 🇧🇷🇺🇲 (badnetmask@hachyderm.io)'s status on Sunday, 22-Dec-2024 07:06:22 JST Mauricio Teixeira 🇧🇷🇺🇲

     in reply to

     • -rb

     @dashrb
     Yeah, been there. It's a slippery slope.

   • Embed this notice
     -rb (dashrb@awscommunity.social)'s status on Sunday, 22-Dec-2024 07:06:24 JST -rb

     in reply to

     @badnetmask yep! So tells you when there’s a failure.

     Weeks later, you have not seen any failure messages but you look manually anyway & discover that the monitoring system itself has been failing & thus, no notification.

     So you modify the monitoring system to notify you of success & failure, both.

     Weeks later you are numb to the success messages so you haven’t really been paying attention, but you look and discover that there have been repeated failure messages you ignored. 😒