Peter Krefting
It's never a good sign when fsck leaves 55 Gbytes worth of files in /lost+found
I guess it doesn't help to have the file system on RAID when the logical file system structures break. Seems mostly to be backups that are affected, though, those should repair themselves on the next backup run. That and a couple of Git repositories that lost some objects and needed to be recloned from checkouts/backups.
Peter Krefting
@adrinux Is that somehow less resilient to driver bugs/memory corruption/cosmic radiation or whatever happened (I have no idea) than mdraid+ext4?
This is the first time I have had something like this happening after running mdraid+ext4 for well over a decade on multiple machines. I have not tested zfs, so I have no point of comparison.
Peter Krefting
Something is obviosly not right with the file system, as it has remounted itself to read-only again :-/
Peter Krefting
I have a theory as to what is going wrong...
popey
@nafmo I don't think in all my years, I've ever actually seen what a memtest fail looks like.
Peter Krefting
@popey I've had a few. After harddisks failing, memory chips going bad is the second most common error
Peter Krefting
The file system corruption runs deeper. After a couple of hours, the system bailed with
EXT4-fs error (device md0): ext4_lookup:1856: inode #120979457: comm updatedb.plocat: iget: bad extra_isize 9234 (inode size 256)
Aborting journal on device md0-8.
EXT4-fs error (device md0): ext4_journal_check_start:83: comm rs:main Q:Reg: Detected aborted journal
EXT4-fs error (device md0): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal
EXT4-fs (md0): Remounting filesystem read-only
which most definitely does not look good :-/
Peter Krefting
Turns out I took the wrong memory chip out. Now I have a combination of chips (including some older, smaller, ones) that earned me a "PASS" banner.
I am still going to rebuild the file system, I'm afraid there are deeper corruptions, but that will have to wait a little while. I've bought a new disk to replace the one that has been constantly-on since 2016 (the other one in the RAID is from 2019, so practically still brand-new).
Peter Krefting
@adrinux Thanks for the pointers.
I think I will stick with something I know how to work with (ext4 or xfs over mdraid) as I have use that setup extensively both at home and at work. But if I get around to it, I might make me a toy VM to play with it a bit to learn more.
Dr Adrian Simmons
@nafmo If you have zfs questions https://discourse.practicalzfs.com/latest is a good place to ask. People far more knowledgeable than me there.
(Was set up as an exodus from r/zfs when Reddit started behaving badly. But tends not to come up in searches yet unless you look for it specifically.)
Peter Krefting
Looks like it's the 2019 drive that has uncorrectable errors, while the 2016 drive is fine. Looks like I need to replace it sooner rather than later.
Peter Krefting
The 6+ hours overnight memtest gave me a PASS on the new memory configuration, so now I am *copying* the files over to a new hard disk (configured as a single-disk RAID1, to be updated with the disk I'm copying *from* afterwards).
Copying 1.5 Tbyte of files does take a while (5 hours and counting so far), but since I don't trust the metadata of the old filesystem, I am not taking the chance to just have Linux RAID mirror it over.
I need to clean up the filesystem, it is heavy with backups of backups of old root filesystem copies which had backups of old machines with backups on them...
Peter Krefting
The file copying finished eventually. The tricky part was to switch to booting from the new Raid instead of the old, getting grub to read the correct kernel and pass the correct root. After a few rounds through booting from an USB image (#Ventoy ftw), and updating grub inside a chroot, it eventually worked.
Then I zeroed out the old RAID device, and ran mdadm with --grow and --raid-devices=2 to sync the new fs back to the old drive.
Took all day, and syncing those 2 terabytes takes a while longer still.
Let's hope this works out now...
Peter Krefting
One power outage later, I've learned that #update-grub isn't enough when booting in #EFI mode and changing the partition that holds the grub menu configuration. One also needs to run #grub-install to update the EFI grub.cfg to point to the first grub.cfg
https://wiki.debian.org/GrubEFIReinstall
And of course, the power outage had to happen just as I had left for Germany and Spiel. Good thing the rest of the family wasn't home either...
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.