Re: [pve-devel] applied: [RFC pve-qemu] disable jemalloc

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Hi,

Am 10/03/2023 um 19:05 schrieb DERUMIER, Alexandre:
> I'm currently benching again qemu with librbd and memory allocator.
> 
> 
> It's seem that they are still performance problem with default glibc
> allocator, around 20-25% less iops and bigger latency.

Are those numbers compared to jemalloc or tcmalloc?

Also, a key problem with allocator tuning is that its heavily dependent on
the workload of each specific library (i.e., not only QEMU itself but also
the specific block backend (library).

> 
> From my bench, i'm around 60k iops vs 80-90k iops with 4k randread.
> 
> Redhat have also notice it
> https://bugzilla.redhat.com/show_bug.cgi?id=1717414
> https://sourceware.org/bugzilla/show_bug.cgi?id=28050
> 
> 
> I known than jemalloc was buggy with rust lib  && pbs block driver,
> but did you have evaluated tcmalloc ?

Yes, for PBS once - was way worse in how it generally worked than either
jemalloc and default glibc IIRC, but I don't think I checked for latency,
as then we tracked down freed memory that the allocator did not give back
to the system to how they internally try to keep a pool of available memory
around.

So for latency it might be a win, but IMO not to sure if the other effects
it has are worth that.

> 
> Note that it's possible to load it dynamically with LD_PRELOAD,
> so maybe could we add an option in vm config to enable it ? 
> 

I'm not 100% sure if QEMU copes well with preloading it via the dynlinker
as is, or if we need to hard-disable malloc_trim support for it then.
As currently with the "system" allocator (glibc) there's malloc_trim called
(semi-) periodically via call_rcu_thread - and at least qemu's meson build
system config disables malloc_trim for tcmalloc or jemalloc.


Or did you already test this directly on QEMU, not just rbd bench? As then
I'd be open to add some tuning config with a allocator sub-property in there
to our CFGs.