Conversation

Notices

1. Embed this notice
   Michael A. Murphy :system76: (mmstick@fosstodon.org)'s status on Thursday, 06-Feb-2025 09:00:13 JST Michael A. Murphy :system76:

   PSA to #rustlang and #linux developers: there is a long-standing 20y bug in the system allocator (glibc malloc) which causes it to hoard large sbrk buffers in arenas. By default, it uses heuristics to dynamically increase the mmap threshold—the point where it switches from using sbrk to mmap—which causes a memory "leak". Some #libcosmic apps were affected by this bug, causing them to use 10-30x more memory than they need. See the PR below for how to tame malloc in Rust:

   https://github.com/pop-os/cosmic-bg/pull/73

   • Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Brodie Robertson (brodieonlinux@mstdn.social)'s status on Thursday, 06-Feb-2025 09:00:12 JST Brodie Robertson

     in reply to

     @mmstick You mention this is a bug in the system allocator, is there an upstream report I can see about that.

   • Embed this notice
     Brodie Robertson (brodieonlinux@mstdn.social)'s status on Thursday, 06-Feb-2025 10:35:01 JST Brodie Robertson

     in reply to

     @mmstick Awesome thanks for the links

   • Embed this notice
     Michael A. Murphy :system76: (mmstick@fosstodon.org)'s status on Thursday, 06-Feb-2025 10:35:02 JST Michael A. Murphy :system76:

     in reply to

     • Brodie Robertson

     @BrodieOnLinux There are a handful of related bug reports on their tracker.

     https://sourceware.org/bugzilla/show_bug.cgi?id=14827

     https://sourceware.org/bugzilla/show_bug.cgi?id=26969

     https://sourceware.org/bugzilla/show_bug.cgi?id=30769

     https://sourceware.org/bugzilla/show_bug.cgi?id=15321

     You'll find a lot of blog articles on the subject, too.

     https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory

     https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/

     https://www.linkedin.com/blog/engineering/infrastructure/taming-memory-fragmentation-in-venice-with-jemalloc

     https://www.softwareatscale.dev/p/run-python-servers-more-efficiently

     https://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html

     Many simply result to switching to jemalloc since it handles allocations across multiple threads more efficiently, with less fragmentation.