Re: [PATCH v2 pve-qemu 0/2] Re-enable tcmalloc as the memory allocator

["Kefu Chai" <k.chai@proxmox.com>]

On Tue Apr 14, 2026 at 11:36 PM CST, Fiona Ebner wrote:
> Am 14.04.26 um 1:08 PM schrieb Fiona Ebner:
>> Note that I did play around with memory hotplug and ballooning before as
>> well, not sure if related.
>> 
>> Unfortunately, I don't have the debug symbols for librbd.so.1 right now:
>> 
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0  0x00007ea8da6442d0 in tc_memalign () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
>>> [Current thread is 1 (Thread 0x7ea8ca66a6c0 (LWP 109157))]

Hi Fiona,

Thank you for the backtrace.

I dug into the segfault, but was not able to reprudce it locally after
performing over 3300 snapshot ops on two RBD drives, including
concurrent and batch ops.

I also searched over internet to see if we are alone. And here is what I
found: 

The crash site (SLL_Next in linked_list.h) is a known pattern in
gperftools. It's what happens when a *prior* operation corrupts a freed
block's embedded freelist pointer, and a later allocation follows the
garbage and segfaults. Essentially, tc_memalign() is the victim, not the
culprit. RHBZ #1430223 [1] and gperftools issues #1036 [2] and #1096 [3]
all describe the same crash pattern with Ceph. RHBZ #1494309 [4] is also
worth noting -- tcmalloc didn't intercept aligned_alloc() until
gperftools 2.6.1-5, causing a mixed-allocator situation where glibc
allocated but tcmalloc freed. That one's long fixed in our 2.16, but it
shows this corner of the allocator has had real bugs before.

If it happens again, probably the way to catch the actual corruption
at its source would be:

  LD_PRELOAD=libtcmalloc_debug.so.4 qemu-system-x86_64 ...

This adds guard words around allocations and checks them on free,
so it'd point straight at whatever is doing the corrupting write.
This comes with 2-5x overhead, but guess it's fine for debugging.

If you manage to reproduce it, I am more than happy to debug it with
your reproducer.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1430223
[2] https://github.com/gperftools/gperftools/issues/1036
[3] https://github.com/gperftools/gperftools/issues/1096
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1494309

>
> I had added malloc_stats(); calls around
> MallocExtension_ReleaseFreeMemory(); to better see the effects, which
> also requires including malloc.h in pve-backup.c when building for
> tcmalloc. I also did a few backups before, so I can't rule out that it's
> related to that. I did a build of librbd1 and librados2 with the debug
> symbols now, but haven't been able to reproduce the issue yet. Will try
> more tomorrow.