[pve-devel] [RFC pve-qemu] disable jemalloc

[pve-devel] [RFC pve-qemu] disable jemalloc

[Stefan Reiter]

jemalloc does not play nice with our Rust library (proxmox-backup-qemu),
specifically it never releases memory allocated from Rust to the OS.
This leads to a problem with larger caches (e.g. for the PBS block driver).

It appears to be related to this GitHub issue:
https://github.com/jemalloc/jemalloc/issues/1398

The background_thread solution seems weirdly hacky, so let's disable
jemalloc entirely for now.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

@Alexandre: you were the one to introduce jemalloc into our QEMU builds a long
time ago - does it still provide a measurable benefit? If the performance loss
would be too great in removing it, we could maybe figure out some workarounds as
well.

Its current behaviour does seem rather broken to me though...

 debian/rules | 1 -
 1 file changed, 1 deletion(-)

diff --git a/debian/rules b/debian/rules
index c73d6a1..57e1c91 100755
--- a/debian/rules
+++ b/debian/rules
@@ -60,7 +60,6 @@ config.status: configure
 	--enable-docs \
 	--enable-glusterfs \
 	--enable-gnutls \
-	--enable-jemalloc \
 	--enable-libiscsi \
 	--enable-libusb \
 	--enable-linux-aio \
-- 
2.20.1

Re: [pve-devel] [RFC pve-qemu] disable jemalloc

[alexandre derumier]

AFAIK, It was mainly for rbd performance, but performance is fixed now 
(with glibc from buster).
So, I think we don't need it anymore.


On 10/12/2020 16:23, Stefan Reiter wrote:
> jemalloc does not play nice with our Rust library (proxmox-backup-qemu),
> specifically it never releases memory allocated from Rust to the OS.
> This leads to a problem with larger caches (e.g. for the PBS block driver).
>
> It appears to be related to this GitHub issue:
> https://github.com/jemalloc/jemalloc/issues/1398
>
> The background_thread solution seems weirdly hacky, so let's disable
> jemalloc entirely for now.
>
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
>
> @Alexandre: you were the one to introduce jemalloc into our QEMU builds a long
> time ago - does it still provide a measurable benefit? If the performance loss
> would be too great in removing it, we could maybe figure out some workarounds as
> well.
>
> Its current behaviour does seem rather broken to me though...
>
>   debian/rules | 1 -
>   1 file changed, 1 deletion(-)
>
> diff --git a/debian/rules b/debian/rules
> index c73d6a1..57e1c91 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -60,7 +60,6 @@ config.status: configure
>   	--enable-docs \
>   	--enable-glusterfs \
>   	--enable-gnutls \
> -	--enable-jemalloc \
>   	--enable-libiscsi \
>   	--enable-libusb \
>   	--enable-linux-aio \

[pve-devel] applied: [RFC pve-qemu] disable jemalloc

[Thomas Lamprecht]

On 10.12.20 16:23, Stefan Reiter wrote:
> jemalloc does not play nice with our Rust library (proxmox-backup-qemu),
> specifically it never releases memory allocated from Rust to the OS.
> This leads to a problem with larger caches (e.g. for the PBS block driver).
> 
> It appears to be related to this GitHub issue:
> https://github.com/jemalloc/jemalloc/issues/1398
> 
> The background_thread solution seems weirdly hacky, so let's disable
> jemalloc entirely for now.
> 
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
> 
> @Alexandre: you were the one to introduce jemalloc into our QEMU builds a long
> time ago - does it still provide a measurable benefit? If the performance loss
> would be too great in removing it, we could maybe figure out some workarounds as
> well.
> 
> Its current behaviour does seem rather broken to me though...
> 
>  debian/rules | 1 -
>  1 file changed, 1 deletion(-)
> 
>

applied, thanks!

Re: [pve-devel] applied: [RFC pve-qemu] disable jemalloc

[DERUMIER, Alexandre]

Hi,
sorry for bumping this old thread.

I'm currently benching again qemu with librbd and memory allocator.


It's seem that they are still performance problem with default glibc
allocator, around 20-25% less iops and bigger latency.

From my bench, i'm around 60k iops vs 80-90k iops with 4k randread.

Redhat have also notice it
https://bugzilla.redhat.com/show_bug.cgi?id=1717414
https://sourceware.org/bugzilla/show_bug.cgi?id=28050


I known than jemalloc was buggy with rust lib  && pbs block driver,
but did you have evaluated tcmalloc ?


Note that it's possible to load it dynamically with LD_PRELOAD,
so maybe could we add an option in vm config to enable it ? 



Le mardi 15 décembre 2020 à 14:43 +0100, Thomas Lamprecht a écrit :
> On 10.12.20 16:23, Stefan Reiter wrote:
> > jemalloc does not play nice with our Rust library (proxmox-backup-
> > qemu),
> > specifically it never releases memory allocated from Rust to the
> > OS.
> > This leads to a problem with larger caches (e.g. for the PBS block
> > driver).
> > 
> > It appears to be related to this GitHub issue:
> > https://github.com/jemalloc/jemalloc/issues/1398
> > 
> > The background_thread solution seems weirdly hacky, so let's
> > disable
> > jemalloc entirely for now.
> > 
> > Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> > ---
> > 
> > @Alexandre: you were the one to introduce jemalloc into our QEMU
> > builds a long
> > time ago - does it still provide a measurable benefit? If the
> > performance loss
> > would be too great in removing it, we could maybe figure out some
> > workarounds as
> > well.
> > 
> > Its current behaviour does seem rather broken to me though...
> > 
> >  debian/rules | 1 -
> >  1 file changed, 1 deletion(-)
> > 
> > 
> 
> applied, thanks!
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 

Re: [pve-devel] applied: [RFC pve-qemu] disable jemalloc

[Thomas Lamprecht]

Hi,

Am 10/03/2023 um 19:05 schrieb DERUMIER, Alexandre:
> I'm currently benching again qemu with librbd and memory allocator.
> 
> 
> It's seem that they are still performance problem with default glibc
> allocator, around 20-25% less iops and bigger latency.

Are those numbers compared to jemalloc or tcmalloc?

Also, a key problem with allocator tuning is that its heavily dependent on
the workload of each specific library (i.e., not only QEMU itself but also
the specific block backend (library).

> 
> From my bench, i'm around 60k iops vs 80-90k iops with 4k randread.
> 
> Redhat have also notice it
> https://bugzilla.redhat.com/show_bug.cgi?id=1717414
> https://sourceware.org/bugzilla/show_bug.cgi?id=28050
> 
> 
> I known than jemalloc was buggy with rust lib  && pbs block driver,
> but did you have evaluated tcmalloc ?

Yes, for PBS once - was way worse in how it generally worked than either
jemalloc and default glibc IIRC, but I don't think I checked for latency,
as then we tracked down freed memory that the allocator did not give back
to the system to how they internally try to keep a pool of available memory
around.

So for latency it might be a win, but IMO not to sure if the other effects
it has are worth that.

> 
> Note that it's possible to load it dynamically with LD_PRELOAD,
> so maybe could we add an option in vm config to enable it ? 
> 

I'm not 100% sure if QEMU copes well with preloading it via the dynlinker
as is, or if we need to hard-disable malloc_trim support for it then.
As currently with the "system" allocator (glibc) there's malloc_trim called
(semi-) periodically via call_rcu_thread - and at least qemu's meson build
system config disables malloc_trim for tcmalloc or jemalloc.


Or did you already test this directly on QEMU, not just rbd bench? As then
I'd be open to add some tuning config with a allocator sub-property in there
to our CFGs.