Mauricio Teixeira 🇧🇷🇺🇲
One of my Raspberry Pis is freezing every night, because the amount of I/O and CPU to run incremental backups is too high, and is causing other processes to misbehave.
Trying to decide whether I want to run backups from a different system (all data is in NFS), or run a different backup software (currently using Deja Dup). 🤔
Mauricio Teixeira 🇧🇷🇺🇲
@stefan
Yeah, since everything is already in NFS, I am considering just doing it from a different machine that has access to the same NFS share. Thanks.
Stefan :veritrek:
@badnetmask besides Proxmox backup backing my VM, I have jobs defined on my NAS which are doing just rsync from devices like a pi to local. This way I can directly see and use the files. Real backup happens than by the NAS every night to other locations.
Mauricio Teixeira 🇧🇷🇺🇲
For now, since rebooting every morning is getting tiresome, I just created a script that will shutdown all the services before the backup starts, and bring everything up again after it completes. Should survive a couple of nights without intervention. #HomeLab.
Miroslav Stankic
@badnetmask I've been running raspiBackup on all my Pis for years now.
Mauricio Teixeira 🇧🇷🇺🇲
@pax0707
That's a very interesting project, no doubt, but doesn't work for me, since I don't run anything on the SD card other than the OS itself. Thanks for the pointer anyway, could be useful in the future.
Mauricio Teixeira 🇧🇷🇺🇲
Stopping all the services while running the backup not only stabilized the system, but it also cut the backup to 1/4 of the time. There's definitely some improvement to be made here. #HomeLab
Mauricio Teixeira 🇧🇷🇺🇲
Uh-oh. Spoke too soon. The Raspberry Pi just froze 8 hours after the backup has completed (with all services down). Poking around the logs, it seems something related to InfluxDB trying to compact old files, which overloads the CPU and saturates the I/O on the SD card (temp files?).
Suggestions? Should I upgrade to 1.11 or 2.7? Maybe convert everything to Victoria Metrics?
I would prefer to retain the data, if possible (historical knowledge).
slamp
@badnetmask I'm using influxdb 2.7 without issue since 1+ year. the raspberry pi 4 has an SSD for the OS + data.
but the pi is slow for some tasks : docker pull homeassistant takes around 15 min. docker up -d 15 to 30 min.
Mauricio Teixeira 🇧🇷🇺🇲
@slamp
How big is your database? Mine is around 10GB now.
offdutypirate
@badnetmask I've had OK luck using USB disks instead of SD cards for more I/O intensive stuff on the Pi. Maybe that's an option?
Mauricio Teixeira 🇧🇷🇺🇲
@offdutypirate
Not really. The apps are running out of an NFS mount, so in this case it's just a matter of redirecting the temp dir out of the SD card.
slamp
@badnetmask Home Assistant sqlite: 753M
Influxdb : 116G (oh s* ! I need to clean it)
Mauricio Teixeira 🇧🇷🇺🇲
Wow. My "erratic" Raspberry Pi never really recovered (see thread). I've been trying different things to no avail, only changing how long it takes to freeze.
This evening I decided to reboot it while I was physically next to it, and the house was silent, which allowed me to hear a weird buzzing noise coming out of it. The buzzing coincided with access to the memory card (the when the green light in the back blinks).
I really don't know what's going on. It's the first time I hear a buzzing sound coming out of a Raspberry Pi. Even weirder is the fact that it only happens when accessing the memory card.
Gladly, the way I configured things, the memory card is hot swappable, so I simply put it in another Pi that I had laying around doing nothing (from an abandoned project). It's been working fine, without the buzz.
Hopefully that was the underlying reason since the start, and now I can get rid of the problem for good. I just don't know what to do with this "half-working Pi" (it works well under light load, just freezes up under high load).
Fingers crossed.
Mauricio Teixeira 🇧🇷🇺🇲
@noiq
I don't have such a tool, and I am really really bad with soldering. But thanks for chiming in!
NoIQ
@badnetmask just a wild asses guess, but maybe a short in the solder of the flash? If you have access to a electronic hot air gun, you might try a resolder.
Alex Kretzschmar
@badnetmask @noiq i do
Mauricio Teixeira 🇧🇷🇺🇲
@ironicbadger @noiq
We'll talk offline. 😄
Mauricio Teixeira 🇧🇷🇺🇲
@dwsharp @noiq
It's a 4. I guess I could make it boot out of a USB drive, but that's a project for another day. Thanks for the idea!
dwsharp
@badnetmask @noiq What type of pi is it? Could you run it off a USB drive so that you don’t need to use the memory card?
Anthony
@badnetmask My two most common failure modes on Pis is a failing SD card or a failing external power adapter.
Mauricio Teixeira 🇧🇷🇺🇲
@ironicbadger @noiq
So, looking at my host monitoring (Telegraf + InfluxDB + Grafana), it seems like the SD card IO is much more constant/homogeneous now. Before it was spiky, especially when running backups. Looks like the card interface was really the issue, but I won't do a victory lap just yet. Let's give it a few more days.
#HomeLab #RaspberryPi
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.