Notices by Infoseepage (infoseepage@mastodon.social)

These days a lot of systems have a mix of "Performance" cores and "Efficiency cores." What's interesting about the idea of a background process that you use to essentially cold archive data is that the greatest compression might be achieved by having one or two very fast cores to work the problem.

This runs somewhat contrary to the way CPU performance increases have been made in the modern era. You used to have few cores and systems got faster over time as you were able to increase the clockspeed. The fastest Pentium 4 chips were about 4 GHZ. Increases in performance have been achieved through architectural improvements, reduced heat due to ever smaller transistors and increasing the number of cores.

Basically, you can find more redundancies/space savings in a large block of data than in a small one, so the ideal situation would be to do the compression on a system with one or two VERY fast cores with the algorithm having access to loads of ram.

Also wondering how much more some of these .7zip archives could be made if I did them with only 2 threads instead of letting all my CPU cores in on the process. Why does this matter? Well, because the way 7 zip processes data changes based on the number assigned threads. You compress the file in less time with more threads, but it achieves this speedup by splitting the data that is being worked on into smaller chunks to feed to the threads.

I wonder if a next generation .zim2 format could be introduced with better compression while still being able to browse and search articles efficiently.

I keep pretty much every offline copy of Wikipedia I've ever downloaded for historical / change over time reference. They take up a lot of space. I don't actually access them that frequently, so I decided to 7zip them on LZMA 2 Ultra setting. The smallest is about 50 GB and the latest is up to about 120 GB. I'm seeing about 91% of original size, so across all my files I can probably squeeze out another 100 GB of free space. Impressive, given .zim files are already compressed.

@alfiekohn The word for this is false or implied endorsement. They making use of an image of you to represent to someone you know that you approve of a product/service who it can very easily be argued is being mislead.

Celebrities sue companies for this shit all the time.

@kbm0 There very clearly are gains to be had via time and ram to compress / access speed tradeoffs. They are real tradeoffs, though, and this isn't something to be applied willy-nilly, but I think a deep archiving mode that still allows files to be accessed transparently (but much more slowly) would be very useful to a lot of people. It wouldn't surprise me given my careful scheme for storing data that I could save 10% or so in space through deep archiving.

@kbm0 Maybe if you have a lot of small files, but most of the files I'm dealing with are usually 5-50 MB in size. Default block size on modern NTFS is like 4k. I think space actually lost to a 4k block only being partially filled is pretty minimal these days and I've looked at the folder properties which list "size" vs "size on disk" (the actual space needing to be allocated) often enough that I don't think there is much in the way of gains to be had there.

@kbm0 I don't actually want some sort of scan service that has to run in the background to ingest and compress new data as you gradually add more to your file hierarchy, what makes sense to me would be a second checkbox and a user trigger-able "rescan this folder now" button in the folder properties dialog.

@kbm0 The NTFS compression offer via the Windows GUI is MUCH worse in terms of what it offers, relying on algorithms that were no joke first published in the late 70's before I was born. I've tested that as well and the actual space savings are extremely nominal and not at all worth it, IMO. IMO MS should deprecate it or at least add a second GUI checkbox and functionality for automatically LZX compressing at time of copy/move anything added to a folder which has the checkbox turned on.

@kbm0 LZX at least seems worthwhile to run against folder hierarchies of static files that don't need frequent access. Note, this requires you to run a command line tool against the folder you want to transparently compress once and any new files added to the folder don't get compressed without yu rerunning the tool, which is not efficiently designed to quickly find and compress just the new files which need compressing.

@kbm0 I think the transparent filesystem compression that MS offers via the Compact tool (with the LZX algorithm offering the most gains) compresses files on a per file basis. There should be no duplicate files in that directory path.

This is the sort of fundamental architectural feature that might actually make me interested in Windows again, instead of waking up wondering what they've chosen to enshitify today.

But, what I really want is an even deeper transparent archiving mode. I compressed a sample of 20 GB of those same raw files with 7z LZMA2 on Ultra and got a post compression size of about 83% of original, so the LZX compression MS offers (but hides behind a command line tool - out of site of most users) is leaving 10% on the bone. Implies if properly implemented, I could have 421 GB of space freed up.

The MS compact tool is finished running and my 2.4 TB photo archive of raw files is now taking up 93.1% of the space they used to on disk. Not going to turn down 169 GB of free space that I didn't have before.

Might be wrong about this. Going to let the compact command do its magic overnight and check again once its finished.

@carnage4life

Wouldn't surprise me if the "links that weren't publicly listed" were extremely private or intimate images or at least things like family gatherings with photos of children. Many people these days make the conscious choice to keep photos of their children and their development to adulthood offline.

@carnage4life How much you want to bet that these were people who were using some personal photo sharing site and that site decided to sell out their userbase by selling their users data to OpenAI and changed their use policy for users to allow them to do so, with some sort of opt-out.

Interestingly, if I examine the folder properties I don't see a change in the amount of space it takes to store the files on disk, but I am seeing a slow increase in available disk space. It isn't huge and a far cry from the 10% savings with LZMA2 Ultra, but it isn't nothing and is basically transparent for static files.