Conversation

Notices

1. Embed this notice
   DarkMahesvara (darkmahesvara@varishangout.net)'s status on Sunday, 15-Oct-2023 09:10:22 JST DarkMahesvara
   >copy torrent file with rsync
   >qbittorrent complaints new file is only 99,9% instead of finished like the original
   >compare copy with original shows hash is the same
   
   :hyperconfused:
   • Embed this notice
     翠星石 (suiseiseki@freesoftwareextremist.com)'s status on Sunday, 15-Oct-2023 09:10:20 JST 翠星石

     in reply to
     @DarkMahesvara Random bitrot on a large files or a collection of files occurs fairly commonly due deficiencies in RAM, HDD and SDD technology.
     
     Most likely you don't have ECC RAM and HDD's+SDD's also allocate the minimum amount of space possible to ECC, plus data density is so small, the reading of charges and turning those charges into bytes is a statistical operation.
     
     Usually such bitflips go unnoticed, as are you usually going to notice a small colour change in a chunk in a PNG or video or a bitflip in an elf file in struct padding bytes, or a slight CPU instruction change in a function that doesn't end up causing a change in execution by chance?
     
     
     It looks like rsync copied the file correctly, but when qbitorrent compared the sha1 hashes of all the chunks, it found one with a bitflip.
     
     What's good about bittorrent is that as you're seeding a torrent, each chunk is automatically checked and corrected (if needed) as they're seeded, plus even if the torrent isn't very popular, you just need to hit verify in the torrent client every few months to automatically correct any bitflips.
     
     Btrfs is the filesystem attempt to provide something similar, but for recovery to work a second drive in RAID with a duplicate of the corrupted block is needed.
   • Embed this notice
     翠星石 (suiseiseki@freesoftwareextremist.com)'s status on Sunday, 15-Oct-2023 16:39:31 JST 翠星石

     in reply to
     @DarkMahesvara Filesystem based checksumming and ECC RAM will reduce the error rate a huge amount, but as far as I'm aware, there is still a slim, but not "basically impossible" chance a triple RAM bitflip occurs (undetectable by ECC), or one while the data travels down the memory bus into the CPU and into the cache (I'm pretty sure there's error checking for memory busses, but not cryptographic grade I believe) - or qbittorent had a bug and didn't actually finish the download.
   • Embed this notice
     DarkMahesvara (darkmahesvara@varishangout.net)'s status on Sunday, 15-Oct-2023 16:39:32 JST DarkMahesvara

     in reply to

     • 翠星石
     @Suiseiseki i do use ZFS and ECC so bitrot is basically impossible. the only thing i can think of is that somehow the file never finished but QBT wrongly said it did or there is some padding but i don't know QBT good enough to say for sure.