From: Elias Huhsovitz <e.huhsovitz@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Elias Huhsovitz <e.huhsovitz@proxmox.com>
Subject: [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning
Date: Mon, 21 Sep 2026 11:53:59 +0200 [thread overview]
Message-ID: <20260921095404.61552-1-e.huhsovitz@proxmox.com> (raw)
A different take on the RFC [0] sent by Dominik.
Overview
--------
Introduce a new 'pinning' property, with 3 methods for
vCPU pinning:
* balanced: Automatically selects the host NUMA node(s) with the most
uncommitted memory, using as many as needed to fit the VM's CPU and memory
requirements. Binds the VM's memory to the selected host node(s) with
policy=bind. The QEMU process is confined to the selected host nodes CPUs,
but individual vCPU threads are NOT pinned. The host scheduler may move
them freely within the selected nodes.
Use new 'hostnodes' field to restrict which host nodes are considered.
* numa: Pins the sets of vCPUs of each virtual NUMA node to a corresponding
host NUMA node. This makes memory access consistent for each vCPU because
the scheduler will not move it to a different NUMA node. It considers the
'numaX' setting when binding vCPUs to host nodes.
* one-to-one: Pins each vCPU to a specific host core. This prevents vCPUs
from being rescheduled on other cores entirely. It provides the highest
performance but is the least flexible for the host scheduler. It considers
the virtual and physical NUMA layout and avoids SMT collisions by spreading
vCPUs across distinct physical cores whenever possible. When used with
numa: 1, memory is also bound to host NUMA nodes. When used without numa: 1,
only CPU pinning is performed, memory is not bound to specific host nodes.
All settings are affected by 'affinity', to only consider those cores.
All 3 pinning options respect the cpu topology to a certain degree.
Background
----------
I read the tutorial by a community member on resource isolation
and specifically CPU pinnging [1]. I strongly agreed with the poster
that proper CPU pinnig must respect the underlying CPU topology.
The new `balanced` pinning mode is directly inspired by the incus
option limits.cpu.nodes [6], which tries to efficiently spead the load
on the host-node on VM startup. Code is available at [2].
(See balanceNUMANodes()).
Changes v1 -> v2:
-----------------
* Add host CPU topology discovery (sysfs) to respect physical
sockets, cores, and SMT siblings.
* Add heterogeneous CPU detection (e.g., Intel P/E cores, ARM
big.LITTLE). (Inspired by midr_el1 and cache geometry grouping in
Incus's startupCPUSet & cpuType [7]).
* Confine QEMU process to the largest same-type CPU subset at
startup to prevent KVM initialization failures. (Inspired by
Incus's startupCPUSet [7]).
* Allow one-to-one binding without NUMA enabled.
* Add topology validation for 'one-to-one' mode. If the selected
host CPUs form an asymmetric layout, the guest topology falls
back to a flat layout (sockets=1, cores=N) to avoid guest
scheduler penalties.
* Emit more detailed NUMA topology to the guest vm.
* Automatic memory binding: 'balanced' and 'numa' modes now
automatically bind guest memory to the selected host NUMA nodes
using policy=bind, even if explicit 'hostnodes' are not set.
* Add 'balanced' pinning mode. It automatically selects host NUMA nodes with
the most uncommitted memory and binds the VM's memory to them,
without pinning individualvCPU threads. (Inspired by Incus's
balanceNUMANodes[2])
* Add JSON-based reservation tracking with file locking to
prevent NUMA placement collisions during concurrent VM starts.
Previous Versions
-----------------
v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
Benchmark summary
=================
Host System
-------------
OS: Proxmox VE 9.2.18 x86_64
Kernel: Linux 7.0.14-16-pve
CPU: AMD EPYC 7351P 16-Core Processor
Memory: 64 GB DDR4 memory with 2667MT/s (4x16GB sticks)
numactl -H:
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 16 17 18 19
node 0 size: 15891 MB
node 0 free: 14693 MB
node 1 cpus: 4 5 6 7 20 21 22 23
node 1 size: 16124 MB
node 1 free: 15174 MB
node 2 cpus: 8 9 10 11 24 25 26 27
node 2 size: 16124 MB
node 2 free: 14838 MB
node 3 cpus: 12 13 14 15 28 29 30 31
node 3 size: 16123 MB
node 3 free: 14978 MB
node distances:
node 0 1 2 3
0: 10 16 16 16
1: 16 10 16 16
2: 16 16 10 16
3: 16 16 16 10
Guest System
-----------
OS: Fedora Linux 44 (Server Edition) x86_64
Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0)
Kernel: Linux 7.1.13-200.fc44.x86_64
Memory: 8GB
Benchmark Tools
---------------
* STREAM [3]
compiled via:
gcc -O3 -fopenmp -march=native -mcmodel=medium \
-DSTREAM_ARRAY_SIZE=160000000 -DNTIMES=40 stream.c -o stream
executed via:
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores stream
* sc-membench [4]
compiled via:
make numa
executed via:
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores \
membench-numa -s 262144 -r 2 -t 10
* sysbench [5]
executed via (sysbench_mem_bw):
sysbench memory --threads=8 --time=10 \
--memory-block-size=512M --memory-total-size=100G run
executed via (sysbench_mem_bw_local):
sysbench memory --threads=8 --time=10 \
--memory-block-size=256M --memory-total-size=100G \
--memory-scope=local run
Configuration
-------------
sockets=4, cores=2, numa=1, memory=8192 MiB
affinity=0,1,4,5,8,9,12,13
(2 physical cores per host NUMA node, no SMT siblings)
* 0_numa_no_pinning:
baseline. guest NUMA topology defined with implicit binding
(no hostnodes, no policy). no pinning, no affinity.
* 1_pinning_balanced:
pinning=balanced + affinity. allocator auto-selects host
NUMA nodes by uncommitted memory, binds guest memory to
them with policy=bind, confines the QEMU process to the
selected nodes' CPUs. individual vCPU threads are NOT pinned.
* 2_pinning_numa_auto:
pinning=numa + affinity, no explicit hostnodes. allocator
auto-assigns each guest NUMA node to a host NUMA node
(least-loaded first), pins vCPU sets per node.
* 3_pinning_one_to_one_auto:
pinning=one-to-one + affinity, no explicit hostnodes.
same auto-assignment as 2, but each vCPU is pinned to
exactly one host core.
* 4_pinning_numa:
pinning=numa + affinity + explicit hostnodes/policy=bind.
user manually maps each guest NUMA node to a specific
host NUMA node.
* 5_pinning_one_to_one:
pinning=one-to-one + affinity + explicit hostnodes/policy=bind.
same as 4, but each vCPU is pinned to exactly one host core.
Results
=======
Metric 0_numa_no_pinning 1_pinning_balanced 2_pinning_numa_auto 3_pinning_one_to_one_auto 4_pinning_numa 5_pinning_one_to_one
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sc_membench: copy MB/s 8790.55 6840.53 9075.65 9081.70 9087.27 9010.15
sc_membench: largest size KB 262144 262144 262144 262144 262144 262144
sc_membench: latency ns 117.74 168.22 113.21 113.36 113.06 112.71
sc_membench: latency sd ns 0.50 0.27 0.31 0.09 0.09 0.11
sc_membench: read MB/s 38295.46 33668.66 60821.28 60623.70 60965.07 60387.27
sc_membench: write MB/s 19172.32 24095.66 30706.31 30995.34 30730.69 30607.34
stream: Add MB/s 44952.8 44471.7 52853.9 52989.6 53007.2 52369.6
stream: Copy MB/s 55028.7 62880.5 67564.1 67612.2 67588.7 67544.5
stream: Scale MB/s 43047.6 41157.5 46950.8 46937.1 47002.6 46552.3
stream: Triad MB/s 40861.5 44506.2 52943.8 53016.1 53051.9 52361.6
sysbench_mem_bw: MiB/sec 14881.21 12499.01 15839.22 15673.10 14101.67 13875.53
sysbench_mem_bw: lat 95th ms 390.30 502.20 502.20 434.83 580.02 475.79
sysbench_mem_bw: lat avg ms 264.59 285.98 242.82 244.62 262.01 266.14
sysbench_mem_bw_local: MiB/sec 11076.00 15128.00 12817.04 13579.28 11712.35 13013.29
sysbench_mem_bw_local: lat 95th ms 580.02 450.77 356.70 356.70 467.30 363.18
sysbench_mem_bw_local: lat avg ms 159.01 108.40 126.76 117.12 138.46 127.62
Interpretation of results
-------------------------
The new addition of the pinning=balanced mode seemed promising on
paper, but yielded unsatisfying results. To my understanding this is
because I used an 8GB VM on for testing, on a processor where each
NUMA node is 16GB. So VM-Memory < NUMA-Node Memory, which causes
all memory to be allocated on a single NUMA node.
If the pinning=balanced mode is here to stay, then further testing with
different VM sizes makes sense IMO.
The other 2 modes (numa & one-to-one) see an improvement in memory
sensitive tasks, altough there doesn't seem to be a major performance
difference between the two modes.
References
----------
[0] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
[1] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/
[2] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_common.go
[3] https://www.cs.virginia.edu/stream/
[4] https://github.com/SpareCores/sc-membench
[5] https://github.com/akopytov/sysbench
[6] https://linuxcontainers.org/incus/docs/main/reference/instance_options/#cpu-pinning
[7] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_qemu_machine.go
Summary of changes
------------------
Elias Huhsovitz (4):
pinning: add topology discovery and config parsing
pinning: add NUMA allocator and reservation tracking
memory: integrate pinning-aware NUMA memory binding
pinning: integrate cpu pinning into vm lifecycle
src/PVE/QemuServer.pm | 97 +++-
src/PVE/QemuServer/Makefile | 2 +
src/PVE/QemuServer/Memory.pm | 240 +++++++---
src/PVE/QemuServer/Pinning.pm | 396 ++++++++++++++++
src/PVE/QemuServer/Pinning/Allocator.pm | 590 ++++++++++++++++++++++++
src/PVE/QemuServer/Pinning/Config.pm | 244 ++++++++++
src/PVE/QemuServer/Pinning/Makefile | 9 +
src/PVE/QemuServer/Pinning/Topology.pm | 462 +++++++++++++++++++
8 files changed, 1948 insertions(+), 92 deletions(-)
create mode 100644 src/PVE/QemuServer/Pinning.pm
create mode 100644 src/PVE/QemuServer/Pinning/Allocator.pm
create mode 100644 src/PVE/QemuServer/Pinning/Config.pm
create mode 100644 src/PVE/QemuServer/Pinning/Makefile
create mode 100644 src/PVE/QemuServer/Pinning/Topology.pm
--
2.47.3
next reply other threads:[~2026-09-21 9:55 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-21 9:53 Elias Huhsovitz [this message]
2026-09-21 9:54 ` [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 2/4] pinning: add NUMA allocator and reservation tracking Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 3/4] memory: integrate pinning-aware NUMA memory binding Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 4/4] pinning: integrate cpu pinning into vm lifecycle Elias Huhsovitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260921095404.61552-1-e.huhsovitz@proxmox.com \
--to=e.huhsovitz@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox