From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 93B1C1FF09C for ; Mon, 21 Sep 2026 11:55:18 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1F58021657; Mon, 21 Sep 2026 11:54:37 +0200 (CEST) From: Elias Huhsovitz To: pve-devel@lists.proxmox.com Subject: [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Date: Mon, 21 Sep 2026 11:53:59 +0200 Message-ID: <20260921095404.61552-1-e.huhsovitz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789984465274 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.220 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5YNDMBWWKKMRRUIOKORMVMPASZERXE2P X-Message-ID-Hash: 5YNDMBWWKKMRRUIOKORMVMPASZERXE2P X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Elias Huhsovitz X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A different take on the RFC [0] sent by Dominik. Overview -------- Introduce a new 'pinning' property, with 3 methods for vCPU pinning: * balanced: Automatically selects the host NUMA node(s) with the most uncommitted memory, using as many as needed to fit the VM's CPU and memory requirements. Binds the VM's memory to the selected host node(s) with policy=bind. The QEMU process is confined to the selected host nodes CPUs, but individual vCPU threads are NOT pinned. The host scheduler may move them freely within the selected nodes. Use new 'hostnodes' field to restrict which host nodes are considered. * numa: Pins the sets of vCPUs of each virtual NUMA node to a corresponding host NUMA node. This makes memory access consistent for each vCPU because the scheduler will not move it to a different NUMA node. It considers the 'numaX' setting when binding vCPUs to host nodes. * one-to-one: Pins each vCPU to a specific host core. This prevents vCPUs from being rescheduled on other cores entirely. It provides the highest performance but is the least flexible for the host scheduler. It considers the virtual and physical NUMA layout and avoids SMT collisions by spreading vCPUs across distinct physical cores whenever possible. When used with numa: 1, memory is also bound to host NUMA nodes. When used without numa: 1, only CPU pinning is performed, memory is not bound to specific host nodes. All settings are affected by 'affinity', to only consider those cores. All 3 pinning options respect the cpu topology to a certain degree. Background ---------- I read the tutorial by a community member on resource isolation and specifically CPU pinnging [1]. I strongly agreed with the poster that proper CPU pinnig must respect the underlying CPU topology. The new `balanced` pinning mode is directly inspired by the incus option limits.cpu.nodes [6], which tries to efficiently spead the load on the host-node on VM startup. Code is available at [2]. (See balanceNUMANodes()). Changes v1 -> v2: ----------------- * Add host CPU topology discovery (sysfs) to respect physical sockets, cores, and SMT siblings. * Add heterogeneous CPU detection (e.g., Intel P/E cores, ARM big.LITTLE). (Inspired by midr_el1 and cache geometry grouping in Incus's startupCPUSet & cpuType [7]). * Confine QEMU process to the largest same-type CPU subset at startup to prevent KVM initialization failures. (Inspired by Incus's startupCPUSet [7]). * Allow one-to-one binding without NUMA enabled. * Add topology validation for 'one-to-one' mode. If the selected host CPUs form an asymmetric layout, the guest topology falls back to a flat layout (sockets=1, cores=N) to avoid guest scheduler penalties. * Emit more detailed NUMA topology to the guest vm. * Automatic memory binding: 'balanced' and 'numa' modes now automatically bind guest memory to the selected host NUMA nodes using policy=bind, even if explicit 'hostnodes' are not set. * Add 'balanced' pinning mode. It automatically selects host NUMA nodes with the most uncommitted memory and binds the VM's memory to them, without pinning individualvCPU threads. (Inspired by Incus's balanceNUMANodes[2]) * Add JSON-based reservation tracking with file locking to prevent NUMA placement collisions during concurrent VM starts. Previous Versions ----------------- v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ Benchmark summary ================= Host System ------------- OS: Proxmox VE 9.2.18 x86_64 Kernel: Linux 7.0.14-16-pve CPU: AMD EPYC 7351P 16-Core Processor Memory: 64 GB DDR4 memory with 2667MT/s (4x16GB sticks) numactl -H: available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 16 17 18 19 node 0 size: 15891 MB node 0 free: 14693 MB node 1 cpus: 4 5 6 7 20 21 22 23 node 1 size: 16124 MB node 1 free: 15174 MB node 2 cpus: 8 9 10 11 24 25 26 27 node 2 size: 16124 MB node 2 free: 14838 MB node 3 cpus: 12 13 14 15 28 29 30 31 node 3 size: 16123 MB node 3 free: 14978 MB node distances: node 0 1 2 3 0: 10 16 16 16 1: 16 10 16 16 2: 16 16 10 16 3: 16 16 16 10 Guest System ----------- OS: Fedora Linux 44 (Server Edition) x86_64 Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0) Kernel: Linux 7.1.13-200.fc44.x86_64 Memory: 8GB Benchmark Tools --------------- * STREAM [3] compiled via: gcc -O3 -fopenmp -march=native -mcmodel=medium \ -DSTREAM_ARRAY_SIZE=160000000 -DNTIMES=40 stream.c -o stream executed via: OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores stream * sc-membench [4] compiled via: make numa executed via: OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores \ membench-numa -s 262144 -r 2 -t 10 * sysbench [5] executed via (sysbench_mem_bw): sysbench memory --threads=8 --time=10 \ --memory-block-size=512M --memory-total-size=100G run executed via (sysbench_mem_bw_local): sysbench memory --threads=8 --time=10 \ --memory-block-size=256M --memory-total-size=100G \ --memory-scope=local run Configuration ------------- sockets=4, cores=2, numa=1, memory=8192 MiB affinity=0,1,4,5,8,9,12,13 (2 physical cores per host NUMA node, no SMT siblings) * 0_numa_no_pinning: baseline. guest NUMA topology defined with implicit binding (no hostnodes, no policy). no pinning, no affinity. * 1_pinning_balanced: pinning=balanced + affinity. allocator auto-selects host NUMA nodes by uncommitted memory, binds guest memory to them with policy=bind, confines the QEMU process to the selected nodes' CPUs. individual vCPU threads are NOT pinned. * 2_pinning_numa_auto: pinning=numa + affinity, no explicit hostnodes. allocator auto-assigns each guest NUMA node to a host NUMA node (least-loaded first), pins vCPU sets per node. * 3_pinning_one_to_one_auto: pinning=one-to-one + affinity, no explicit hostnodes. same auto-assignment as 2, but each vCPU is pinned to exactly one host core. * 4_pinning_numa: pinning=numa + affinity + explicit hostnodes/policy=bind. user manually maps each guest NUMA node to a specific host NUMA node. * 5_pinning_one_to_one: pinning=one-to-one + affinity + explicit hostnodes/policy=bind. same as 4, but each vCPU is pinned to exactly one host core. Results ======= Metric 0_numa_no_pinning 1_pinning_balanced 2_pinning_numa_auto 3_pinning_one_to_one_auto 4_pinning_numa 5_pinning_one_to_one --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- sc_membench: copy MB/s 8790.55 6840.53 9075.65 9081.70 9087.27 9010.15 sc_membench: largest size KB 262144 262144 262144 262144 262144 262144 sc_membench: latency ns 117.74 168.22 113.21 113.36 113.06 112.71 sc_membench: latency sd ns 0.50 0.27 0.31 0.09 0.09 0.11 sc_membench: read MB/s 38295.46 33668.66 60821.28 60623.70 60965.07 60387.27 sc_membench: write MB/s 19172.32 24095.66 30706.31 30995.34 30730.69 30607.34 stream: Add MB/s 44952.8 44471.7 52853.9 52989.6 53007.2 52369.6 stream: Copy MB/s 55028.7 62880.5 67564.1 67612.2 67588.7 67544.5 stream: Scale MB/s 43047.6 41157.5 46950.8 46937.1 47002.6 46552.3 stream: Triad MB/s 40861.5 44506.2 52943.8 53016.1 53051.9 52361.6 sysbench_mem_bw: MiB/sec 14881.21 12499.01 15839.22 15673.10 14101.67 13875.53 sysbench_mem_bw: lat 95th ms 390.30 502.20 502.20 434.83 580.02 475.79 sysbench_mem_bw: lat avg ms 264.59 285.98 242.82 244.62 262.01 266.14 sysbench_mem_bw_local: MiB/sec 11076.00 15128.00 12817.04 13579.28 11712.35 13013.29 sysbench_mem_bw_local: lat 95th ms 580.02 450.77 356.70 356.70 467.30 363.18 sysbench_mem_bw_local: lat avg ms 159.01 108.40 126.76 117.12 138.46 127.62 Interpretation of results ------------------------- The new addition of the pinning=balanced mode seemed promising on paper, but yielded unsatisfying results. To my understanding this is because I used an 8GB VM on for testing, on a processor where each NUMA node is 16GB. So VM-Memory < NUMA-Node Memory, which causes all memory to be allocated on a single NUMA node. If the pinning=balanced mode is here to stay, then further testing with different VM sizes makes sense IMO. The other 2 modes (numa & one-to-one) see an improvement in memory sensitive tasks, altough there doesn't seem to be a major performance difference between the two modes. References ---------- [0] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ [1] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/ [2] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_common.go [3] https://www.cs.virginia.edu/stream/ [4] https://github.com/SpareCores/sc-membench [5] https://github.com/akopytov/sysbench [6] https://linuxcontainers.org/incus/docs/main/reference/instance_options/#cpu-pinning [7] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_qemu_machine.go Summary of changes ------------------ Elias Huhsovitz (4): pinning: add topology discovery and config parsing pinning: add NUMA allocator and reservation tracking memory: integrate pinning-aware NUMA memory binding pinning: integrate cpu pinning into vm lifecycle src/PVE/QemuServer.pm | 97 +++- src/PVE/QemuServer/Makefile | 2 + src/PVE/QemuServer/Memory.pm | 240 +++++++--- src/PVE/QemuServer/Pinning.pm | 396 ++++++++++++++++ src/PVE/QemuServer/Pinning/Allocator.pm | 590 ++++++++++++++++++++++++ src/PVE/QemuServer/Pinning/Config.pm | 244 ++++++++++ src/PVE/QemuServer/Pinning/Makefile | 9 + src/PVE/QemuServer/Pinning/Topology.pm | 462 +++++++++++++++++++ 8 files changed, 1948 insertions(+), 92 deletions(-) create mode 100644 src/PVE/QemuServer/Pinning.pm create mode 100644 src/PVE/QemuServer/Pinning/Allocator.pm create mode 100644 src/PVE/QemuServer/Pinning/Config.pm create mode 100644 src/PVE/QemuServer/Pinning/Makefile create mode 100644 src/PVE/QemuServer/Pinning/Topology.pm -- 2.47.3