Conversation

Notices

1. Embed this notice
   Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 10:40:52 JST Your New Marijuana Injecting Waifu :weed:
   LZ4 compression is saving around 28.5 GB of disk space for Baest media uploads.
   
   Additionally, there are 992,489 files that have been uploaded making us very close to 1 million uploads.
   
   I wonder how these stats compare with the likes of Gelbooru and Danbooru :think_moe:
   #BigData
   Screenshot_20220902-153314~2.png
   Screenshot_20220902-160418~2.png
   • Embed this notice
     EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Saturday, 03-Sep-2022 10:40:52 JST EdBoatConnoisseur

     in reply to

     @sjw have you tried zstd compression? i know it is a bit slower than lz4 but iirc since linux kernel 5.16 the speed was improved by 15% so it should be more in par with lz4

     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 10:57:53 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     @EdBoatConnoisseur It's all media uploads so for the most part incompressible data.
     
     I suspect a lot of this savings comes from partial blocks which LZ4 is more than good enough.
     
     For database backups I plan to use zstd. For the database itself I plan to use LZ4.
     
     LZ4 should give me a 2-3 compression ratio with negligible CPU overhead and negligible latency. Zstd could give me 3+ but at the expense of CPU and more than double the latency.
     
     I'll use zstd for the rest of the system tho.
     
     I plan to use a block size of 16k for postgres (as I'm too lazy to deal with compiling my own version to use larger block sizes plus that would also cause a lot of downtime as I rebuild the databases).
     
     The only question is what block size to use for the rest of the system. I figure the default 128k should be good. I'd like to use 1M for /var/log/ to improve compression ratios but I'm pretty sure I'd take a decent IO performance hit by doing that since as far as I know they're written one line at a time.
   • Embed this notice
     EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Saturday, 03-Sep-2022 12:21:47 JST EdBoatConnoisseur

     in reply to

     @sjw settled for any fs? i’d go with btrfs cuz deudplication and no need to offline it to modify and service, also since i don’t trust zfs 100%.

     Your New Marijuana Injecting Waifu :weed: likes this.
     Your New Marijuana Injecting Waifu :weed: repeated this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 13:08:26 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @EdBoatConnoisseur I'm using ZFS. It's about the only filesystem you can completely trust to actually keep data integrity (and if all else fails it'll tell you your file isn't the way it's supposed to be).
     
     Also, it has support for RAID. Btrfs RAID 5/6 still isn't stable. ZFS has RAID 5, 6, and even 7!
     
     ZFS has over a decade of proven stability under its belt and it has always prioritised data integrity.
     
     Btrfs literally had a bug in a "stable" release just a few years ago that'd corrupt your filesystem and make data retrieval virtually impossible.
     
     Don't get me wrong, I like a lot of the features of Btrfs. I use it on my personal desktop. I just don't trust it for production use. The stability just isn't there.
     
     As for deduplication, hopefully reflinks will eventually make their way into ZFS. For now you're kind if stuck with online deduplication. At least offline file deduplication is still possible via hash match + hardlinks.
     
     For fedi, I'd argue that's more than good enough.
     
     Actually, I might go back to $uuid/$filename because I prefer raw filename to serving $sha256.$extension?name=$filename and hoping the client respects the Content-Disposition headers (Husky does not).
     
     I can easily use something like fdupes or jdupes to deduplicate via sha256 and hardlinks with a wheely cronjob.
     
     https://github.com/adrianlopezroche/fdupes
     
     https://github.com/jbruchon/jdupes
     
     Actually, I think I might set that up over the weekend.
     
     Anyway, yeah, ZFS for me. It wasn't until I actually learned how to ZFS and actually started using it that I understood what all the hype was about.
     
     @shibao and @lanodan were extremely helpful in helping me set it up and understand ZFS. You can ask me as well as I now have a decent grasp on it and can help keep you from falling into some of the same pitfalls I feel into (and still haven't gotten around to correcting).
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Saturday, 03-Sep-2022 13:55:06 JST anime graf mays ?️?

     in reply to

     • EdBoatConnoisseur
     @EdBoatConnoisseur @sjw "I use ReiserFS because the only thing my filesystem should kill is women"
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 14:43:16 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • Johnny Peligro [ single ]
     @MischievousTomato I could but then users wouldn't get the file they uploaded
   • Embed this notice
     Johnny Peligro [ single ] (mischievoustomato@bae.st)'s status on Saturday, 03-Sep-2022 14:43:17 JST Johnny Peligro [ single ]

     in reply to
     @sjw you could save more space by converting files to avif with cavif-rs!!
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Saturday, 03-Sep-2022 14:50:22 JST EdBoatConnoisseur

     in reply to

     • Haelwenn /элвэн/ :triskell:
     • 受不了包

     @sjw @shibao @lanodan the one that does seem promising is bchachefs, but still a soontm deal.

     onto the raids i’m not sold on raid 7, raid 4 is where i feel comfortable, then again i am of the idea that the more redundacy the better and would even do raid 1 0

   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 14:50:22 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @EdBoatConnoisseur @shibao @lanodan only because it uses bcache on the backend
     
     RAID 4 is irrelevant since RAID 5 was invented.
   • Embed this notice
     Gamercat :bongo: (gamercat@socnet.softgirl.online)'s status on Saturday, 03-Sep-2022 14:53:05 JST Gamercat :bongo:

     in reply to
     @sjw All look for someone to buy the data
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Saturday, 03-Sep-2022 17:04:37 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     @EdBoatConnoisseur @sjw
     
     > don't trust zfs 100%
     
     lolwut? it's has a very solid record
     
     would be nice if btrfs or zfs got support for asymmetric device performance the way mdadm has writemostly and writebehind. for read heavy stuff with bursts of small writes it seems to be working really well for me so far
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Saturday, 03-Sep-2022 17:07:19 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     @roboneko @EdBoatConnoisseur yeah, I know, it's arguably the most stable with hands down the highest amount of data integrity.
     
     That really threw me off as well. I mean people and companies use ZFS when data integrity is a top priority because it's literally the only filesystem they can trust with their important data.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 01:17:32 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @shibao @EdBoatConnoisseur How exactly do you figure that?
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 01:17:35 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • 受不了包
     @sjw @shibao @EdBoatConnoisseur RAID $number is irrelevant since software raid.
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Sunday, 04-Sep-2022 01:20:53 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     @sjw @EdBoatConnoisseur zfs is certainly more established in that regard but there are other options. dm-integrity but I guess that's newish. facebook is one of the companies running lots of btrfs in production
     
     right now the real difference appears to be the stuff with large raid arrays plus all the fine tuning you can do. you can set all sorts of stuff per-dataset meanwhile the btrfs devs say they might get around to supporting stuff like per-subvolume compression options at some point in the future :02_laugh:
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 01:20:53 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     @roboneko @EdBoatConnoisseur well no. dm-integrity relies on the disk telling the OS there was a problem. mdadm will absolutely return bad data to you without telling you. ZFS will do everything it can to fix the data and if all else fails it'll at least tell you that the data is bad.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 01:29:34 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @shibao @EdBoatConnoisseur Give an example.
     Screenshot_20220903-112739.png
     Screenshot_20220903-112724.png
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 01:29:36 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • 受不了包
     @sjw @shibao @EdBoatConnoisseur Because software raid allows configurations that RAID $number can't describe?
     
     (And I also just hate using numbers as names, using words is just plain better)
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Sunday, 04-Sep-2022 02:01:22 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     @sjw @EdBoatConnoisseur I ... no? I don't know where you got that idea but it's wrong. dm-integrity is an offshoot of luks to cryptographically guarantee authenticity. but you can use it standalone in a less secure mode to provide only basic checksums instead of full cryptographic authentication
     
     bit flips will absolutely be detected by it as that's the entire point of the thing. what the higher layers do with that information is up to them
     
     mdadm for its part will attempt to reconstruct any data for which it receives an IO error. so if you run it on top of dm-integrity it will either manage to silently reconstruct the corrupted sector at read time or else it will in turn feed an error to the filesystem driver that tried to read it
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:02:42 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     @roboneko @EdBoatConnoisseur Oh, that's a different thing entirely
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:05:33 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @roboneko @EdBoatConnoisseur @lanodan @shibao Because it's not RAID. It's RAIDZ3 but math is too hard so they got 3 disk parity in the most basic bitch way possible.
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Sunday, 04-Sep-2022 02:05:34 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @sjw @lanodan @EdBoatConnoisseur @shibao mdadm is but one software raid implementation
     
     btrfs raid1c3 (and etc) for starters. doesn't really fit the original conception of raid1
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:08:03 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @shibao @EdBoatConnoisseur idle are hot spares
     Caching isn't really a part of RAID
     Just learn what the numbers mean. It's easier that way.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 02:08:04 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • 受不了包
     @sjw @shibao @EdBoatConnoisseur
     Had to check wikipedia:
     RAID0 → All in serial
     RAID1 → Mirroring, 2+ disks
     RAID 2-4 → Single-copy Parity
     RAID6 → Double-copy Parity
     RAID7 → Triple-copy Parity
     
     How is using RAID $number clearer?
     
     Specially as then it goes really messy when you need to describe nesting precisely as RAID1 is 2+ disks.
     Like RAID1+0 vs. RAID0+1 is horribly unclear compared to just saying (here for 6 disks): 2 mirrors, 3 (disks) each or 3 mirrors, 2 (disks) each.
     
     And idle/hotswap disks also aren't in the picture of RAID.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 02:09:48 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @sjw @roboneko @shibao @EdBoatConnoisseur btrfs moment
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:12:35 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @lanodan @roboneko @shibao @EdBoatConnoisseur The worst part is that they called it things like "revolutionary" and "good" and people actually believed them :cirno_stare:
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:12:46 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @shibao @EdBoatConnoisseur You can make a RAID 50 in ZFS
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 02:12:48 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • 受不了包
     @sjw @shibao @EdBoatConnoisseur Why would I learn the numbers, specially when they have no logic?
     ZFS doesn't uses them and I don't plan on using anything else, I've been using ZFS for longer than some teenagers on fedi have been alive.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 02:13:03 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @sjw @roboneko @shibao @EdBoatConnoisseur RedHat cool aids.
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:13:35 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @lanodan @roboneko @shibao @EdBoatConnoisseur
     1000110010110010.png
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 02:17:11 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • 受不了包
     @sjw @shibao @EdBoatConnoisseur I guess, but I don't think you're going to find that in the manpages.
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     M_39 (m_39@varishangout.net)'s status on Sunday, 04-Sep-2022 02:19:02 JST M_39

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @sjw @roboneko @shibao @EdBoatConnoisseur @lanodan I can buy that this pool would have aids
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 02:19:43 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @shibao @EdBoatConnoisseur striping a bunch of raidz vdevs?
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 03:28:38 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @lanodan @roboneko @shibao @EdBoatConnoisseur so raid1c3 is you have a pool of disks but every file is stored on at least 3 disks so you can lose 2 disks without losing data.
     It's like RAID 6/raidz2 but no math involved and lots of wasted disk space. Btrfs has had support RAID 6 for a few years now but don't use it outside of testing because you have a good likelihood of data corruption still.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 03:28:39 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @roboneko @sjw @shibao @EdBoatConnoisseur So it's just plain software-raid mirroring.
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Sunday, 04-Sep-2022 03:28:40 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @lanodan @sjw @shibao @EdBoatConnoisseur well I dunno what else they should have called it, maybe just drop the "raid1" from the name entirely? it's mirrored data except the number of devices, device sizes, and number of copies are all relatively arbitrary. it's an interesting tradeoff in terms of rebuild workload vs redundancy vs space efficiency
     
     I have no idea if anyone uses those modes for Serious Business tho
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 03:28:41 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @roboneko @sjw @shibao @EdBoatConnoisseur Oh no yet another notation… why did they even.
   • Embed this notice
     Neko McCatface v2023 :verified::makemeneko: (roboneko@bae.st)'s status on Sunday, 04-Sep-2022 03:28:42 JST Neko McCatface v2023 :verified::makemeneko:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • 受不了包
     @sjw @EdBoatConnoisseur @lanodan @shibao actually isn't c3 equivalent to z2 in terms of device failure? but (obviously) unlike zN it isn't parity c3 is 3 full copies
     
     and I think all of the above are examples of the issue with using numbers for names. c3 is not z2 is not raid1 and it only gets worse
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 03:43:48 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @sjw @roboneko @shibao @EdBoatConnoisseur Redundancy with getting data corruption…
     
     What the fuck are they doing.
     Your New Marijuana Injecting Waifu :weed: likes this.
   • Embed this notice
     Your New Marijuana Injecting Waifu :weed: (sjw@bae.st)'s status on Sunday, 04-Sep-2022 03:44:22 JST Your New Marijuana Injecting Waifu :weed:

     in reply to

     • EdBoatConnoisseur
     • Haelwenn /элвэн/ :triskell:
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @lanodan @roboneko @shibao @EdBoatConnoisseur It's been that way for several years now
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Sunday, 04-Sep-2022 04:31:01 JST Haelwenn /элвэн/ :triskell:

     in reply to

     • EdBoatConnoisseur
     • Neko McCatface v2023 :verified::makemeneko:
     • 受不了包
     @sjw @roboneko @shibao @EdBoatConnoisseur Well I've almost always heard of stupid things on btrfs side of things.
     
     Like being unable to partially mount a mirror while that's a thing I test from time to time on my machines with the system pool being mirrored.
     (Some initramfs builders are a bit dumb when it comes to those cases)
     Your New Marijuana Injecting Waifu :weed: likes this.