From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 1F4291FF140 for ; Fri, 10 Apr 2026 06:30:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 933A612F92; Fri, 10 Apr 2026 06:31:09 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH pve-qemu 1/2] PVE: use tcmalloc as the memory allocator Date: Fri, 10 Apr 2026 12:30:26 +0800 Message-ID: <20260410043027.3621673-2-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260410043027.3621673-1-k.chai@proxmox.com> References: <20260410043027.3621673-1-k.chai@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1775795364754 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.393 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RWAICNJZB3RR5FX4DEPJW3A3XATRHQNC X-Message-ID-Hash: RWAICNJZB3RR5FX4DEPJW3A3XATRHQNC X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add allocator-aware memory release in the backup completion path: since tcmalloc does not provide glibc's malloc_trim(), use the tcmalloc-specific MallocExtension_ReleaseFreeMemory() API instead. This function walks tcmalloc's page heap free span lists and calls madvise(MADV_DONTNEED) -- it does not walk allocated memory or compact the heap, so latency impact is negligible. Also adds a CONFIG_TCMALLOC meson define so the conditional compilation in pve-backup.c can detect the allocator choice. Signed-off-by: Kefu Chai --- ...use-tcmalloc-as-the-memory-allocator.patch | 77 +++++++++++++++++++ debian/patches/series | 1 + 2 files changed, 78 insertions(+) create mode 100644 debian/patches/pve/0048-PVE-use-tcmalloc-as-the-memory-allocator.patch diff --git a/debian/patches/pve/0048-PVE-use-tcmalloc-as-the-memory-allocator.patch b/debian/patches/pve/0048-PVE-use-tcmalloc-as-the-memory-allocator.patch new file mode 100644 index 0000000..719d522 --- /dev/null +++ b/debian/patches/pve/0048-PVE-use-tcmalloc-as-the-memory-allocator.patch @@ -0,0 +1,77 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Kefu Chai +Date: Thu, 9 Apr 2026 17:29:10 +0800 +Subject: [PATCH] PVE: use tcmalloc as the memory allocator + +Use tcmalloc (from gperftools) as the memory allocator for improved +performance with workloads that create many small, short-lived +allocations -- particularly Ceph/librbd I/O paths. + +tcmalloc uses per-thread caches and size-class freelists that handle +this allocation pattern more efficiently than glibc's allocator. Ceph +benchmarks show ~50% IOPS improvement on 16KB random reads. + +Since tcmalloc does not provide glibc's malloc_trim(), use the +tcmalloc-specific MallocExtension_ReleaseFreeMemory() API to release +cached memory back to the OS after backup completion. This function +walks tcmalloc's page heap free span lists and calls +madvise(MADV_DONTNEED) -- it does not walk allocated memory or compact +the heap, so latency impact is negligible. + +Historical context: +- tcmalloc was originally enabled in 2015 but removed due to + performance issues with gperftools 2.2's default settings (low + TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES and aggressive decommit + disabled). These issues were resolved in gperftools 2.4+. +- jemalloc replaced tcmalloc but was removed in 2020 because it didn't + release memory allocated from Rust (proxmox-backup-qemu) back to the + OS. The allocator-specific release API addresses this. +- PVE 9 ships gperftools 2.16, so the old tuning issues are moot. + +Signed-off-by: Kefu Chai +--- + meson.build | 1 + + pve-backup.c | 8 ++++++++ + 2 files changed, 9 insertions(+) + +diff --git a/meson.build b/meson.build +index 0b28d2ec39..c6de2464d6 100644 +--- a/meson.build ++++ b/meson.build +@@ -2567,6 +2567,7 @@ config_host_data.set('CONFIG_CRYPTO_SM4', crypto_sm4.found()) + config_host_data.set('CONFIG_CRYPTO_SM3', crypto_sm3.found()) + config_host_data.set('CONFIG_HOGWEED', hogweed.found()) + config_host_data.set('CONFIG_MALLOC_TRIM', has_malloc_trim) ++config_host_data.set('CONFIG_TCMALLOC', get_option('malloc') == 'tcmalloc') + config_host_data.set('CONFIG_ZSTD', zstd.found()) + config_host_data.set('CONFIG_QPL', qpl.found()) + config_host_data.set('CONFIG_UADK', uadk.found()) +diff --git a/pve-backup.c b/pve-backup.c +index ad0f8668fd..d5556f152b 100644 +--- a/pve-backup.c ++++ b/pve-backup.c +@@ -19,6 +19,8 @@ + + #if defined(CONFIG_MALLOC_TRIM) + #include ++#elif defined(CONFIG_TCMALLOC) ++#include + #endif + + #include +@@ -303,6 +305,12 @@ static void coroutine_fn pvebackup_co_cleanup(void) + * Won't happen by default if there is fragmentation. + */ + malloc_trim(4 * 1024 * 1024); ++#elif defined(CONFIG_TCMALLOC) ++ /* ++ * Release free memory from tcmalloc's page cache back to the OS. This is ++ * allocator-aware and efficiently returns cached spans via madvise(). ++ */ ++ MallocExtension_ReleaseFreeMemory(); + #endif + } + +-- +2.47.3 + diff --git a/debian/patches/series b/debian/patches/series index 8ed0c52..468df6c 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -81,3 +81,4 @@ pve/0044-PVE-backup-get-device-info-allow-caller-to-specify-f.patch pve/0045-PVE-backup-implement-backup-access-setup-and-teardow.patch pve/0046-PVE-backup-prepare-for-the-switch-to-using-blockdev-.patch pve/0047-savevm-async-reuse-migration-blocker-check-for-snaps.patch +pve/0048-PVE-use-tcmalloc-as-the-memory-allocator.patch -- 2.47.3