Conversation

Notices

1. Embed this notice
   Jason Tubnor 🇦🇺 (tubsta@soc.feditime.com)'s status on Monday, 14-Apr-2025 09:23:31 JST Jason Tubnor 🇦🇺
   Memory management on #Linux is a bit borked. I tried to ingest 4GB of records from a file into a list (array) with #Python. Instead of the machine releasing RAM, it decided just to use swap and before I knew it, swapped was consumed and the Python process fell over.
   
   I'll move the workload over to one of my #FreeBSD lab hosts tonight and re-run the script.
   • Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Monday, 14-Apr-2025 09:23:29 JST feld

     in reply to

     • Ian Wagner
     @ianthetechie @Tubsta there may be a new allocator in FreeBSD in the not too distant future too
     
     https://github.com/microsoft/snmalloc
   • Embed this notice
     Ian Wagner (ianthetechie@fosstodon.org)'s status on Monday, 14-Apr-2025 09:23:30 JST Ian Wagner

     in reply to

     @Tubsta curious to hear a postmortem report on that + how it runs on FreeBSD. Can’t say I’ve ever hit that issue before. (While I really wish Linux would adopt jemalloc, I rather doubt that’s to blame..)

   • Embed this notice
     feld (feld@friedcheese.us)'s status on Monday, 14-Apr-2025 09:28:10 JST feld

     in reply to

     • Ian Wagner
     • feld
     @ianthetechie @Tubsta the CFT for it was announced here
     
     https://lists.freebsd.org/archives/freebsd-hackers/2023-March/001890.html
   • Embed this notice
     Ian Wagner (ianthetechie@fosstodon.org)'s status on Wednesday, 16-Apr-2025 08:27:44 JST Ian Wagner

     in reply to

     • feld

     @feld @Tubsta ooh wow! I didn’t hear about this! Thanks for sharing!

   • Embed this notice
     Jason Tubnor 🇦🇺 (tubsta@soc.feditime.com)'s status on Wednesday, 16-Apr-2025 08:27:44 JST Jason Tubnor 🇦🇺

     in reply to

     • Ian Wagner
     • feld
     @ianthetechie @feld I can confirm that #Python on #FreeBSD behaves as one would expect. It consumes all RAM (with #ZFS releasing ARC as expected) and then dips into swap. As soon as Python releases memory after the ingestion routine, the swap is purged to near zero and the RAM then becomes available (and used) by the system. Far more predictable and reliable.
     
     If you have big, vertical workloads, FreeBSD is where it is at.
     Haelwenn /элвэн/ :triskell: and feld like this.