Conversation

Notices

1. Embed this notice
   Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 05-Nov-2024 12:20:44 JST Rich Felker

   Lazyweb, what's a reasonable removable media filesystem to use for a ridiculously large number of smallish files, hundreds of GB total? FAT32 is just incredibly slow.

   • Embed this notice
     aburka 🫣 (aburka@hachyderm.io)'s status on Tuesday, 05-Nov-2024 12:28:41 JST aburka 🫣

     in reply to

     @dalias probably depends on what operating systems you need to be able to read it

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 05-Nov-2024 12:45:38 JST Rich Felker

     in reply to

     • aburka 🫣

     @aburka Really only *needs* to be Linux, but in a pinch it's preferable to be more widely readable.

   • Embed this notice
     purple 👊✊💨 (purple@nya.social)'s status on Tuesday, 05-Nov-2024 13:08:18 JST purple 👊✊💨

     in reply to

     @dalias@hachyderm.io ext4 is the fastest of the reasonable filesystems for this use case.
     
     dont use exfat, it can be every bit as slow as fat32.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 05-Nov-2024 13:32:01 JST Rich Felker

     in reply to

     FFS, copying speed is like in the tens of MB/min range...

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 05-Nov-2024 13:44:34 JST Rich Felker

     in reply to

     🤬 it's not the fs it's the actual drive being a piece of 💩. dd from it to /dev/null is 5-7 MB/s... vs over 100 with a different drive.

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Michael Gurski (emag@strangeplace.me)'s status on Tuesday, 05-Nov-2024 13:57:37 JST Michael Gurski

     in reply to

     • aburka 🫣

     @dalias @aburka that many small files is going to be the worst performance ever. If you can consolidate in a tarball, a zip, a rat, even without compression, before copying, you'll see a huge performance increase. This was my bane when I had to do backups on a PB of data across many multi-TB volumes. Weeks. When they moved to "archive" and I consolidated them, hours to write the same data to WORM tapes. It's the open/read/close cycle. Open/close are fixed times, so more is slower.

   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Tuesday, 05-Nov-2024 13:57:37 JST Rich Felker

     in reply to

     • aburka 🫣
     • Michael Gurski

     @emag @aburka It's not the open/close taking up time. It's the actual data transfer. But even if it were, putting the data into archives would just move the cost to archiving and unarchiving it. All the costs here except the disk being stupidly slow are pretty much fundamental.

   • Embed this notice
     GreenSkyOverMe (Monika) (greenskyoverme@ohai.social)'s status on Monday, 11-Nov-2024 14:48:39 JST GreenSkyOverMe (Monika)

     in reply to

     @dalias I asked in the local Unix User Group, one person said ext4, one said fuse fs with tar back-end and linked this https://github.com/google/fuse-archive and one person said ext2 with this reasoning, but the ext4 person disagreed (I hope your instance has autotranslate):

   • Embed this notice
     GreenSkyOverMe (Monika) (greenskyoverme@ohai.social)'s status on Monday, 11-Nov-2024 14:49:17 JST GreenSkyOverMe (Monika)

     in reply to

     @dalias
     Zu der Frage wegen vielen kleinen Dateien und removable Media filesystem: mir würde hier als Erstes ext2 einfallen. Wahrscheinlich könnte man auf Journaling und Dateiintegrität verzichten.
     FAT32 ist unter Linux wirklich super langsam bis hart zu unbrauchbar, die Erfahrung habe ich selbst schon gemacht.