Re: [PATCH pve-qemu 0/2] Re-enable tcmalloc as the memory allocator

DERUMIER, Alexandre
2026-04-13T13:14:09Z

Hi,

I have done a small test on a free cluster (this was with old xeon v3
with 18 ssd).

For 4k randwrite, I don't see difference (but I think I'm write limited
by the ssd), I'm around 40k~50k iops

for 4k randread, I see 30% improvement (~70k iops -> ~100kiops)

test: virtio-scsi + iothread + cache=none

without tcmalloc:

# fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=64
--ioengine=libaio --name test
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
^Cbs: 1 (f=1): [r(1)][20%][r=273MiB/s][r=69.9k IOPS][eta 05m:12s]

# fio --filename=/dev/sda --direct=1 --rw=randread --bs=4k --iodepth=64
--ioengine=libaio --name test

test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
^Cbs: 1 (f=1): [r(1)][13.6%][r=273MiB/s][r=69.9k IOPS][eta 05m:12s]
fio: terminating on signal 2

test: (groupid=0, jobs=1): err= 0: pid=929282: Mon Apr 13 15:01:33 2026
read: IOPS=72.7k, BW=284MiB/s (298MB/s)(13.7GiB/49316msec)
slat (usec): min=3, max=917, avg= 6.00, stdev= 3.66
clat (usec): min=169, max=117369, avg=873.83, stdev=521.46
lat (usec): min=180, max=117377, avg=879.83, stdev=521.44
clat percentiles (usec):
| 1.00th=[ 408], 5.00th=[ 510], 10.00th=[ 578], 20.00th=[
668],
| 30.00th=[ 734], 40.00th=[ 799], 50.00th=[ 848], 60.00th=[
906],
| 70.00th=[ 971], 80.00th=[ 1057], 90.00th=[ 1172], 95.00th=[
1303],
| 99.00th=[ 1598], 99.50th=[ 1762], 99.90th=[ 2311], 99.95th=[
2638],
| 99.99th=[ 4490]
bw ( KiB/s): min=228520, max=315464, per=100.00%, avg=291059.43,
stdev=14202.45, samples=98
iops : min=57130, max=78866, avg=72764.90, stdev=3550.61,
samples=98
lat (usec) : 250=0.01%, 500=4.32%, 750=27.98%, 1000=41.61%
lat (msec) : 2=25.85%, 4=0.21%, 10=0.01%, 100=0.01%, 250=0.01%
cpu : usr=15.15%, sys=50.34%, ctx=226303, majf=0, minf=4301
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%,
>=64=0.0%
issued rwts: total=3584067,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s),
io=13.7GiB (14.7GB), run=49316-49316msec

Disk stats (read/write):
sdc: ios=3571207/0, merge=0/0, ticks=2868237/0, in_queue=2868237,
util=99.90%

with tcmalloc
-------------

fio --filename=/dev/sda --direct=1 --rw=randread --bs=4k --iodepth=64 -
-ioengine=libaio --name test
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
^Cbs: 1 (f=1): [r(1)][19.0%][r=383MiB/s][r=98.0k IOPS][eta 03m:42s]

test: (groupid=0, jobs=1): err= 0: pid=1293: Mon Apr 13 15:05:04 2026
read: IOPS=95.7k, BW=374MiB/s (392MB/s)(19.1GiB/52256msec)
slat (usec): min=2, max=1605, avg= 5.94, stdev= 3.93
clat (usec): min=150, max=7426, avg=661.83, stdev=204.22
lat (usec): min=157, max=7433, avg=667.77, stdev=204.42
clat percentiles (usec):
| 1.00th=[ 347], 5.00th=[ 412], 10.00th=[ 453], 20.00th=[
506],
| 30.00th=[ 553], 40.00th=[ 586], 50.00th=[ 627], 60.00th=[
668],
| 70.00th=[ 717], 80.00th=[ 791], 90.00th=[ 906], 95.00th=[
1020],
| 99.00th=[ 1336], 99.50th=[ 1500], 99.90th=[ 1909], 99.95th=[
2114],
| 99.99th=[ 3359]
bw ( KiB/s): min=324264, max=442120, per=100.00%, avg=383405.95,
stdev=20263.50, samples=104
iops : min=81066, max=110530, avg=95851.60, stdev=5065.87,
samples=104
lat (usec) : 250=0.02%, 500=18.47%, 750=56.25%, 1000=19.54%
lat (msec) : 2=5.64%, 4=0.06%, 10=0.01%
cpu : usr=19.61%, sys=65.54%, ctx=262129, majf=0, minf=4296
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%,
>=64=0.0%
issued rwts: total=5002932,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=374MiB/s (392MB/s), 374MiB/s-374MiB/s (392MB/s-392MB/s),
io=19.1GiB (20.5GB), run=52256-52256msec

Disk stats (read/write):
sda: ios=4990801/0, merge=0/0, ticks=2862837/0, in_queue=2862837,
util=99.91%

-------- Message initial --------
De: Kefu Chai <k.chai@proxmox.com>
À: Fiona Ebner <f.ebner@proxmox.com>, "DERUMIER, Alexandre"
<alexandre.derumier@groupe-cyllene.com>, pve-devel@lists.proxmox.com
<pve-devel@lists.proxmox.com>
Objet: Re: [PATCH pve-qemu 0/2] Re-enable tcmalloc as the memory
allocator
Date: 13/04/2026 13:13:12

On Mon Apr 13, 2026 at 4:14 PM CST, Fiona Ebner wrote:
> Am 10.04.26 um 12:44 PM schrieb DERUMIER, Alexandre:
> >
> > > > How does the performance change when doing IO within a QEMU
> > > > guest?
> > > >
> > > > How does this affect the performance for other storage types,
> > > > like
> > > > ZFS,
> > > > qcow2 on top of directory-based storages, qcow2 on top of LVM,
> > > > LVM-
> > > > thin,
> > > > etc. and other workloads like saving VM state during snapshot,
> > > > transfer
> > > > during migration, maybe memory hotplug/ballooning, network
> > > > performance
> > > > for vNICs?

Hi Fiona,

Thanks for the questions.

I traced QEMU's source code. It turns out that guest's RAM is allocated
via direct mmap() calls, which completely bypassing QEMU's C library
allocator. The path looks like:

-m 4G on command line:

memory_region_init_ram_flags_nomigrate()
qemu_ram_alloc()
qemu_ram_alloc_internal()
g_malloc0(sizeof(*new_block))  <-- only the RAMBlock metadata
struct, about 512 bytes
ram_block_add()
qemu_anon_ram_alloc()
qemu_ram_mmap(-1, size, ...)
mmap_reserve(total)
mmap(0, size, PROT_NONE, ...)  <-- reserve address space
mmap_activate(ptr, size, ...)
mmap(ptr, size, PROT_RW, ...)  <-- actual guest RAM

So the gigabytes of guest memory go straight to the kernel via mmap()
and never touch malloc or tcmalloc. Only the small RAMBlock metadata
structure (~512 bytes per region) goes through g_malloc0().

In other words, tcmalloc's scope is limited to QEMU's own working
memory, among other things, block layer buffers, coroutine stacks,
internal data structs. It does not touch gues RAM at all.

The inflat/deflate path of balloon code does involve glib malloc. It's
virtio_balloon_pbp_alloc(), which calls bitmap_new() via g_new0() to
track partially-ballooned pages. But these bitmaps tracks only
metadata,
whose memory footprints are relatively small. And this only happens in
infrequent operations.

Hopefully, this addresses the balloon concern. If I've missed something
or misread the code, please help point it out.

When it comes to different storage types and workloads, I ran
benchmarks
covering some scenarios you listed. All comparison use the same binary
(pve-qemu-kvm 10.1.2-7) with and without LD_PRELOAD=libtcmalloc.so.4,
so
the allocator is the only variable.

Storage backends (block layer, via qemu-img bench)

4K reads, depth=32, io_uring, cache=none, 5M ops per run, best
of 3. Host NVMe (ext4) or LVM-thin backed by NVMe.

backend              glibc ops/s   tcmalloc ops/s   delta
qcow2 on ext4        1,188,495     1,189,343        +0.1%
qcow2 on ext4 write  1,036,914     1,036,699         0.0%
raw on ext4          1,263,583     1,277,465        +1.1%
raw on LVM-thin        433,727       433,576         0.0%

The reason why raw/LVM-thin is slower, is that, I *think* it actually
hits the dm-thin layer rather than page cache. The allocator delta is
noise in all cases.

RBD tested via a local vstart cluster (3 OSDs, bluestore on NVMe):

path                        glibc       tcmalloc    delta
qemu-img bench + librbd     19,111      19,156      +0.2%
rbd bench (librbd direct)   35,329      36,622      +3.7%

I don't have ZFS configured on this (host) machine, but QEMU's file
I/O
path to ZFS goes through the same code as ext4, so the difference
is in the kernel. I'd expect the same null result,
though I could be wrong and am happy to set up a test pool if
you'd like to see the numbers.

Guest I/O (Debian 13 guest, virtio-blk)

Ran dd inside a Debian 13 cloud image guest (4 vCPU, 2 GB RAM,
qcow2 on ext4, cache=none, aio=native) against a second virtio
disk (/dev/vdb, 8 GB qcow2).

workload                              glibc     tcmalloc   delta
dd if=/dev/vdb bs=4M count=1024       15.3 GB/s 15.5 GB/s  +1.3%
iflag=direct (sequential)
8x parallel dd if=/dev/vdb bs=4k      787 MB/s  870 MB/s  +10.5%
count=100k iflag=direct
(each starting at a different offset)

Migration and savevm

Tested with a 4 GB guest where about 2 GB of RAM was dirtied
(filled /dev/shm with urandom, 8x256 MB) before triggering each
operation.

scenario                glibc      tcmalloc   delta
migrate (exec: URI)     0.622 s    0.622 s    0.0%
savevm (qcow2 snap)     0.503 s    0.504 s    +0.2%

What I didn't measure

I did'nt test ZFS/vNIC throughput, as the host path lives in the
kernel
side, while QEMU only handles the control plane. Hence I'd expect
very
little allocator impact there, though I could be wrong.

I also didn't test memory hotplug on ballooning cases, because, as
explained above, these are rare one-shot operations. and as the
source
code trace shows malloc is not involved. But again, happy to look
into
it, if it's still a concern.

> >
> > Hi Fiona,
> >
> > I'm stil running in production (I have keeped tcmalloc after the
> > removal some year ago from the pve build), and I didn't notice
> > problem.
> > (but I still don't use pbs).
> >
> > But I never have done bench with/without it since 5/6 year.

And thanks Alexandre for sharing your production experience, that's
very
valuable context.

> >
> > Maybe vm memory should be checked too, I'm thinking about RSS
> > memory
> > with balloon free_page_reporting, to see if it's correcting freeing
> > memory.

For free_page_reporting, the VIRTIO_BALLOON_F_REPORTING path:

virtual_balloon_handle_report()
ram_block_discard_range()
madvise(host_startaddr, kebngth, QEMU_MADV_DONTNEED)

This operates directly on the mmap'ed guest RAM region. Still no malloc
involvement anywhere in this path.

In short, the tests above reveal a consistent pattern: tcmalloc helps
where allocation pressure is high, and is neutral everywhere else.
Please let me know if you'd like more data on any specific workload, or
if there is anything I overlooked. I appreciate the careful review and
insights.

>
> Thanks! If it was running fine for you for this long, that is a very
> good data point :) Still, testing different scenarios/configurations
> would be nice to rule out that there is a (performance) regression
> somewhere else.

cheers,
Kefu