* [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning
@ 2026-09-21 9:53 Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing Elias Huhsovitz
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Elias Huhsovitz @ 2026-09-21 9:53 UTC (permalink / raw)
To: pve-devel; +Cc: Elias Huhsovitz
A different take on the RFC [0] sent by Dominik.
Overview
--------
Introduce a new 'pinning' property, with 3 methods for
vCPU pinning:
* balanced: Automatically selects the host NUMA node(s) with the most
uncommitted memory, using as many as needed to fit the VM's CPU and memory
requirements. Binds the VM's memory to the selected host node(s) with
policy=bind. The QEMU process is confined to the selected host nodes CPUs,
but individual vCPU threads are NOT pinned. The host scheduler may move
them freely within the selected nodes.
Use new 'hostnodes' field to restrict which host nodes are considered.
* numa: Pins the sets of vCPUs of each virtual NUMA node to a corresponding
host NUMA node. This makes memory access consistent for each vCPU because
the scheduler will not move it to a different NUMA node. It considers the
'numaX' setting when binding vCPUs to host nodes.
* one-to-one: Pins each vCPU to a specific host core. This prevents vCPUs
from being rescheduled on other cores entirely. It provides the highest
performance but is the least flexible for the host scheduler. It considers
the virtual and physical NUMA layout and avoids SMT collisions by spreading
vCPUs across distinct physical cores whenever possible. When used with
numa: 1, memory is also bound to host NUMA nodes. When used without numa: 1,
only CPU pinning is performed, memory is not bound to specific host nodes.
All settings are affected by 'affinity', to only consider those cores.
All 3 pinning options respect the cpu topology to a certain degree.
Background
----------
I read the tutorial by a community member on resource isolation
and specifically CPU pinnging [1]. I strongly agreed with the poster
that proper CPU pinnig must respect the underlying CPU topology.
The new `balanced` pinning mode is directly inspired by the incus
option limits.cpu.nodes [6], which tries to efficiently spead the load
on the host-node on VM startup. Code is available at [2].
(See balanceNUMANodes()).
Changes v1 -> v2:
-----------------
* Add host CPU topology discovery (sysfs) to respect physical
sockets, cores, and SMT siblings.
* Add heterogeneous CPU detection (e.g., Intel P/E cores, ARM
big.LITTLE). (Inspired by midr_el1 and cache geometry grouping in
Incus's startupCPUSet & cpuType [7]).
* Confine QEMU process to the largest same-type CPU subset at
startup to prevent KVM initialization failures. (Inspired by
Incus's startupCPUSet [7]).
* Allow one-to-one binding without NUMA enabled.
* Add topology validation for 'one-to-one' mode. If the selected
host CPUs form an asymmetric layout, the guest topology falls
back to a flat layout (sockets=1, cores=N) to avoid guest
scheduler penalties.
* Emit more detailed NUMA topology to the guest vm.
* Automatic memory binding: 'balanced' and 'numa' modes now
automatically bind guest memory to the selected host NUMA nodes
using policy=bind, even if explicit 'hostnodes' are not set.
* Add 'balanced' pinning mode. It automatically selects host NUMA nodes with
the most uncommitted memory and binds the VM's memory to them,
without pinning individualvCPU threads. (Inspired by Incus's
balanceNUMANodes[2])
* Add JSON-based reservation tracking with file locking to
prevent NUMA placement collisions during concurrent VM starts.
Previous Versions
-----------------
v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
Benchmark summary
=================
Host System
-------------
OS: Proxmox VE 9.2.18 x86_64
Kernel: Linux 7.0.14-16-pve
CPU: AMD EPYC 7351P 16-Core Processor
Memory: 64 GB DDR4 memory with 2667MT/s (4x16GB sticks)
numactl -H:
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 16 17 18 19
node 0 size: 15891 MB
node 0 free: 14693 MB
node 1 cpus: 4 5 6 7 20 21 22 23
node 1 size: 16124 MB
node 1 free: 15174 MB
node 2 cpus: 8 9 10 11 24 25 26 27
node 2 size: 16124 MB
node 2 free: 14838 MB
node 3 cpus: 12 13 14 15 28 29 30 31
node 3 size: 16123 MB
node 3 free: 14978 MB
node distances:
node 0 1 2 3
0: 10 16 16 16
1: 16 10 16 16
2: 16 16 10 16
3: 16 16 16 10
Guest System
-----------
OS: Fedora Linux 44 (Server Edition) x86_64
Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-11.0)
Kernel: Linux 7.1.13-200.fc44.x86_64
Memory: 8GB
Benchmark Tools
---------------
* STREAM [3]
compiled via:
gcc -O3 -fopenmp -march=native -mcmodel=medium \
-DSTREAM_ARRAY_SIZE=160000000 -DNTIMES=40 stream.c -o stream
executed via:
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores stream
* sc-membench [4]
compiled via:
make numa
executed via:
OMP_NUM_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores \
membench-numa -s 262144 -r 2 -t 10
* sysbench [5]
executed via (sysbench_mem_bw):
sysbench memory --threads=8 --time=10 \
--memory-block-size=512M --memory-total-size=100G run
executed via (sysbench_mem_bw_local):
sysbench memory --threads=8 --time=10 \
--memory-block-size=256M --memory-total-size=100G \
--memory-scope=local run
Configuration
-------------
sockets=4, cores=2, numa=1, memory=8192 MiB
affinity=0,1,4,5,8,9,12,13
(2 physical cores per host NUMA node, no SMT siblings)
* 0_numa_no_pinning:
baseline. guest NUMA topology defined with implicit binding
(no hostnodes, no policy). no pinning, no affinity.
* 1_pinning_balanced:
pinning=balanced + affinity. allocator auto-selects host
NUMA nodes by uncommitted memory, binds guest memory to
them with policy=bind, confines the QEMU process to the
selected nodes' CPUs. individual vCPU threads are NOT pinned.
* 2_pinning_numa_auto:
pinning=numa + affinity, no explicit hostnodes. allocator
auto-assigns each guest NUMA node to a host NUMA node
(least-loaded first), pins vCPU sets per node.
* 3_pinning_one_to_one_auto:
pinning=one-to-one + affinity, no explicit hostnodes.
same auto-assignment as 2, but each vCPU is pinned to
exactly one host core.
* 4_pinning_numa:
pinning=numa + affinity + explicit hostnodes/policy=bind.
user manually maps each guest NUMA node to a specific
host NUMA node.
* 5_pinning_one_to_one:
pinning=one-to-one + affinity + explicit hostnodes/policy=bind.
same as 4, but each vCPU is pinned to exactly one host core.
Results
=======
Metric 0_numa_no_pinning 1_pinning_balanced 2_pinning_numa_auto 3_pinning_one_to_one_auto 4_pinning_numa 5_pinning_one_to_one
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sc_membench: copy MB/s 8790.55 6840.53 9075.65 9081.70 9087.27 9010.15
sc_membench: largest size KB 262144 262144 262144 262144 262144 262144
sc_membench: latency ns 117.74 168.22 113.21 113.36 113.06 112.71
sc_membench: latency sd ns 0.50 0.27 0.31 0.09 0.09 0.11
sc_membench: read MB/s 38295.46 33668.66 60821.28 60623.70 60965.07 60387.27
sc_membench: write MB/s 19172.32 24095.66 30706.31 30995.34 30730.69 30607.34
stream: Add MB/s 44952.8 44471.7 52853.9 52989.6 53007.2 52369.6
stream: Copy MB/s 55028.7 62880.5 67564.1 67612.2 67588.7 67544.5
stream: Scale MB/s 43047.6 41157.5 46950.8 46937.1 47002.6 46552.3
stream: Triad MB/s 40861.5 44506.2 52943.8 53016.1 53051.9 52361.6
sysbench_mem_bw: MiB/sec 14881.21 12499.01 15839.22 15673.10 14101.67 13875.53
sysbench_mem_bw: lat 95th ms 390.30 502.20 502.20 434.83 580.02 475.79
sysbench_mem_bw: lat avg ms 264.59 285.98 242.82 244.62 262.01 266.14
sysbench_mem_bw_local: MiB/sec 11076.00 15128.00 12817.04 13579.28 11712.35 13013.29
sysbench_mem_bw_local: lat 95th ms 580.02 450.77 356.70 356.70 467.30 363.18
sysbench_mem_bw_local: lat avg ms 159.01 108.40 126.76 117.12 138.46 127.62
Interpretation of results
-------------------------
The new addition of the pinning=balanced mode seemed promising on
paper, but yielded unsatisfying results. To my understanding this is
because I used an 8GB VM on for testing, on a processor where each
NUMA node is 16GB. So VM-Memory < NUMA-Node Memory, which causes
all memory to be allocated on a single NUMA node.
If the pinning=balanced mode is here to stay, then further testing with
different VM sizes makes sense IMO.
The other 2 modes (numa & one-to-one) see an improvement in memory
sensitive tasks, altough there doesn't seem to be a major performance
difference between the two modes.
References
----------
[0] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
[1] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/
[2] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_common.go
[3] https://www.cs.virginia.edu/stream/
[4] https://github.com/SpareCores/sc-membench
[5] https://github.com/akopytov/sysbench
[6] https://linuxcontainers.org/incus/docs/main/reference/instance_options/#cpu-pinning
[7] https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_qemu_machine.go
Summary of changes
------------------
Elias Huhsovitz (4):
pinning: add topology discovery and config parsing
pinning: add NUMA allocator and reservation tracking
memory: integrate pinning-aware NUMA memory binding
pinning: integrate cpu pinning into vm lifecycle
src/PVE/QemuServer.pm | 97 +++-
src/PVE/QemuServer/Makefile | 2 +
src/PVE/QemuServer/Memory.pm | 240 +++++++---
src/PVE/QemuServer/Pinning.pm | 396 ++++++++++++++++
src/PVE/QemuServer/Pinning/Allocator.pm | 590 ++++++++++++++++++++++++
src/PVE/QemuServer/Pinning/Config.pm | 244 ++++++++++
src/PVE/QemuServer/Pinning/Makefile | 9 +
src/PVE/QemuServer/Pinning/Topology.pm | 462 +++++++++++++++++++
8 files changed, 1948 insertions(+), 92 deletions(-)
create mode 100644 src/PVE/QemuServer/Pinning.pm
create mode 100644 src/PVE/QemuServer/Pinning/Allocator.pm
create mode 100644 src/PVE/QemuServer/Pinning/Config.pm
create mode 100644 src/PVE/QemuServer/Pinning/Makefile
create mode 100644 src/PVE/QemuServer/Pinning/Topology.pm
--
2.47.3
^ permalink raw reply [flat|nested] 5+ messages in thread
* [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing
2026-09-21 9:53 [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Elias Huhsovitz
@ 2026-09-21 9:54 ` Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 2/4] pinning: add NUMA allocator and reservation tracking Elias Huhsovitz
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Elias Huhsovitz @ 2026-09-21 9:54 UTC (permalink / raw)
To: pve-devel; +Cc: Elias Huhsovitz
Introduce Topology.pm and Config.pm to parse host CPU layouts from sysfs
and handle guest pinning configuration.
The config module parses 3 new modes that buid the mapping between guest
vCPUs and virtual NUMA nodes:
balanced: Automatically selects the host NUMA node(s) with the most
uncommitted memory, using as many as needed to fit the VM's CPU and
memory requirements.
1. numa: Pins the sets of vCPUs of each virtual NUMA node to a
corresponding host NUMA node.
2. one-to-one: Pins each vCPU to a specific host core. This prevents
vCPUs from being rescheduled on other cores entirely.
3. The topology module reads physical packages, cores, and SMT siblings,
and detects heterogeneous CPUs (e.g., Intel P/E cores, ARM big.LITTLE).
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
src/PVE/QemuServer/Pinning/Config.pm | 244 +++++++++++++
src/PVE/QemuServer/Pinning/Topology.pm | 462 +++++++++++++++++++++++++
2 files changed, 706 insertions(+)
create mode 100644 src/PVE/QemuServer/Pinning/Config.pm
create mode 100644 src/PVE/QemuServer/Pinning/Topology.pm
diff --git a/src/PVE/QemuServer/Pinning/Config.pm b/src/PVE/QemuServer/Pinning/Config.pm
new file mode 100644
index 00000000..e71fdd84
--- /dev/null
+++ b/src/PVE/QemuServer/Pinning/Config.pm
@@ -0,0 +1,244 @@
+package PVE::QemuServer::Pinning::Config;
+
+use v5.36;
+
+use PVE::JSONSchema qw(parse_property_string);
+use PVE::QemuServer::Helpers;
+use PVE::QemuServer::Memory;
+
+use constant DEFAULT_VCPU_PINNING => 'none';
+
+our $pinning_fmt = {
+ mode => {
+ type => 'string',
+ enum => ['balanced', 'one-to-one', 'numa', DEFAULT_VCPU_PINNING],
+ description => 'Set the vCPU pinning mode.',
+ verbose_description => <<'EODESCR',
+There are multiple ways to pin vCPUs to host cores:
+
+* none (default): Applies no pinning.
+
+* balanced: Automatically selects the host NUMA node(s) with the most
+ uncommitted memory, using as many as needed to fit the VM's CPU and memory
+ requirements. Binds the VM's memory to the selected host node(s) with
+ policy=bind. The QEMU process is confined to the selected host nodes CPUs,
+ but individual vCPU threads are NOT pinned. The host scheduler may move
+ them freely within the selected nodes. Requires numa: 1 to be enabled.
+ Use 'hostnodes' to restrict which host nodes are considered.
+
+* numa: Pins the sets of vCPUs of each virtual NUMA node to a corresponding
+ host NUMA node. This makes memory access consistent for each vCPU because
+ the scheduler will not move it to a different NUMA node. It considers the
+ 'numaX' setting when binding vCPUs to host nodes.
+ Requires numa: 1 to be enabled.
+
+* one-to-one: Pins each vCPU to a specific host core. This prevents vCPUs
+ from being rescheduled on other cores entirely. It provides the highest
+ performance but is the least flexible for the host scheduler. It considers
+ the virtual and physical NUMA layout and avoids SMT collisions by spreading
+ vCPUs across distinct physical cores whenever possible. When used with
+ numa: 1, memory is also bound to host NUMA nodes. When used without numa: 1,
+ only CPU pinning is performed, memory is not bound to specific host nodes.
+
+All settings are affected by 'affinity', to only consider those cores.
+
+These options do not consider pinning settings from other virtual machines or
+containers. To achieve the best and most consistent performance, use the
+combination of 'numaX' and 'affinity' options to ensure host cores are not
+crowded with vCPU assignments.
+EODESCR
+ default => DEFAULT_VCPU_PINNING,
+ default_key => 1,
+ },
+ hostnodes => {
+ type => 'string',
+ pattern => qr/\d+(?:-\d+)?(?:;\d+(?:-\d+)?)*/,
+ description => 'Restrict host NUMA nodes the VM may use for pinning.',
+ verbose_description => <<'EODESCR',
+Semicolon-separated list of host NUMA node IDs (or ranges) that the VM is
+allowed to use for vCPU pinning and memory allocation. When set, the
+pinning allocator only considers these host nodes as candidates for
+auto-assignment.
+
+If not set, all host NUMA nodes are considered.
+
+Example: pinning=balanced,hostnodes=0;1 restricts the VM to host NUMA
+nodes 0 and 1 only.
+EODESCR
+ format_description => 'id[-id];...',
+ optional => 1,
+ },
+};
+
+PVE::JSONSchema::register_format('pve-qm-cpu-pinning', $pinning_fmt);
+
+our $pinning_desc = {
+ optional => 1,
+ type => 'string',
+ format => 'pve-qm-cpu-pinning',
+ description => 'CPU pinning settings.',
+};
+
+PVE::JSONSchema::register_standard_option('pve-qm-cpu-pinning', $pinning_desc);
+
+# Parses a semicolon-separated list of host NUMA node IDs/ranges,
+# into a sorted arrayref of node ID
+# in: 3-5;0;1 out: [0,1,3,4,5]
+my sub parse_host_node_list($id_ranges) {
+ return undef if !defined($id_ranges) || $id_ranges eq '';
+
+ my $ranges = PVE::QemuServer::Helpers::parse_number_sets($id_ranges);
+ my @ids;
+ for my $range ($ranges->@*) {
+ my ($start, $end) = $range->@*;
+ $end //= $start;
+ push @ids, ($start .. $end);
+ }
+ @ids = sort { $a <=> $b } @ids;
+ return @ids ? \@ids : undef;
+}
+
+# in: "balanced,hostnodes=0;1";
+# out: {mode => "balanced", hostnodes => [0,1]}
+sub parse_pinning($setting_str) {
+ return { mode => DEFAULT_VCPU_PINNING } if !defined($setting_str);
+
+ my $settings_map = parse_property_string($pinning_fmt, $setting_str);
+ $settings_map->{mode} = DEFAULT_VCPU_PINNING if !defined($settings_map->{mode});
+ $settings_map->{hostnodes} = parse_host_node_list($settings_map->{hostnodes}) if (defined($settings_map->{hostnodes}));
+
+ return $settings_map;
+}
+
+# convert NUMA node range list to vCPU set
+# in: {cpus => [ [0, 1], [3]]}
+# out: {0, 1, 3}
+my sub expand_numa_vcpus($numa_map) {
+ my %vcpus;
+ for my $range ($numa_map->{cpus}->@*) {
+ my ($start, $end) = $range->@*;
+ $vcpus{$_} = 1 for $start .. $end // $start;
+ }
+ return \%vcpus;
+}
+
+# in: {hostnodes => [[1, 1]]};
+# out: 1
+my sub extract_numa_hostnode($numa_map) {
+ my $hostnodes = $numa_map->{hostnodes};
+ return undef if !defined($hostnodes);
+
+ die "Pinning only available for 1-to-1 NUMA node mapping\n"
+ if $hostnodes->@* > 1 || defined($hostnodes->[0]->[1]);
+
+ return $hostnodes->[0]->[0];
+}
+
+# in:
+# {
+# numa => 1,
+# numa0 => "cpus=0-1,hostnodes=0",
+# numa1 => "cpus=2-3,hostnodes=1",
+# }
+# out:
+# {
+# 0 => {vcpus => {0, 1} hostnode => 0},
+# 1 => {vcpus => {2, 3}, hostnode => 1,}
+# }
+my sub parse_numa_vcpu_map($conf) {
+ my $map = {};
+
+ for my $i (0 .. $PVE::QemuServer::Memory::MAX_NUMA - 1) {
+ my $entry = $conf->{"numa$i"} or next;
+ my $numa = PVE::QemuServer::Memory::parse_numa($entry) or next;
+
+ $map->{$i} = {
+ vcpus => expand_numa_vcpus($numa),
+ hostnode => extract_numa_hostnode($numa)
+ };
+ }
+
+ return $map;
+}
+
+# Parse cpu settings to map keyed by numa node.
+# in: ($sockets, $cores, $threads_per_core) = (2, 2, 1);
+# out: { 0 => {vcpus => {0, 1}}, 1 => {vcpus => {2, 3}} }
+my sub build_socket_vcpu_map($sockets, $cores, $threads_per_core) {
+ my $map = {};
+ my $vcpu_id = 0;
+
+ for my $socket (0 .. $sockets - 1) {
+ my %vcpus;
+ for (1 .. ($cores * $threads_per_core)) {
+ $vcpus{ $vcpu_id++ } = 1;
+ }
+ $map->{$socket} = { vcpus => \%vcpus };
+ }
+
+ return $map;
+}
+
+# in: ($sockets, $cores, $threads_per_core) = (2, 2, 1);
+# out: { 0 => {vcpus => {0, 1, 2, 3}} }
+my sub build_flat_vcpu_map($sockets, $cores, $threads_per_core) {
+ my $total = $sockets * $cores * $threads_per_core;
+ return {
+ 0 => { vcpus => { map { $_ => 1 } 0 .. $total - 1 } },
+ };
+}
+
+# Parse qm.conf cpu settings to map.
+# The map is keyed by numa-node
+# in: {
+# sockets => 1,
+# cores => 4,
+# vcpus => 4,
+# numa => 1,
+# numa0 => "cpus=0-1,hostnodes=0",
+# numa1 => "cpus=2-3,hostnodes=1",
+# }
+# out: {
+# 0 => {vcpus => {0, 1}, hostnode => 0},
+# 1 => {vcpus => {2, 3}, hostnode => 1},
+# }
+sub build_vnuma_vcpu_map($conf) {
+ my $sockets = $conf->{sockets} // 1;
+ my $cores = $conf->{cores} // 1;
+ # Fix 4.4: honour $conf->{vcpus} when it exceeds sockets*cores (SMT/hotplug).
+ my $vcpus = $conf->{vcpus} // ($sockets * $cores);
+ my $threads_per_core = $vcpus > ($sockets * $cores)
+ ? int($vcpus / ($sockets * $cores))
+ : 1;
+
+ my $map;
+ if ($conf->{numa}) {
+ $map = parse_numa_vcpu_map($conf);
+ $map = build_socket_vcpu_map($sockets, $cores, $threads_per_core)
+ if !keys $map->%*;
+ } else {
+ $map = build_flat_vcpu_map($sockets, $cores, $threads_per_core);
+ }
+
+ my $assigned = 0;
+ $assigned += scalar keys $_->{vcpus}->%* for values $map->%*;
+
+ my $expected = $sockets * $cores * $threads_per_core;
+ die "Invalid NUMA configuration for pinning, some vCPUs missing in numa"
+ . " binding (assigned=$assigned, expected=$expected)\n"
+ if $assigned != $expected;
+
+ return $map;
+}
+
+# in: $conf = { sockets => 2, cores => 4, vcpus => 6 };
+# out: (2, 4, 8, 6)
+sub get_config_topology($conf) {
+ my $sockets = $conf->{sockets} || $conf->{smp} || 1;
+ my $cores = $conf->{cores} || 1;
+ my $maxcpus = $sockets * $cores;
+ my $vcpus = $conf->{vcpus} || $maxcpus;
+ return ($sockets, $cores, $maxcpus, $vcpus);
+}
+
+1;
diff --git a/src/PVE/QemuServer/Pinning/Topology.pm b/src/PVE/QemuServer/Pinning/Topology.pm
new file mode 100644
index 00000000..daddea0b
--- /dev/null
+++ b/src/PVE/QemuServer/Pinning/Topology.pm
@@ -0,0 +1,462 @@
+package PVE::QemuServer::Pinning::Topology;
+
+use v5.36;
+
+use PVE::CpuSet;
+use PVE::Tools qw(file_read_firstline dir_glob_foreach);
+
+# Build { cpu_id => numa_node_id } from /sys/devices/system/node.
+# key: cpu, val: numa-node
+# e.g. reads: 8 core CPU with 2 NUMA nodes
+# out: { 0 => 0, 1 => 0, 2 => 0, 3 => 0, 4 => 1, 5 => 1, 6 => 1, 7 => 1 }
+my sub cpu_to_numa_map() {
+ my $cpu_to_numa = {};
+ my $numa_path = "/sys/devices/system/node";
+
+ return $cpu_to_numa if !-d $numa_path;
+
+ dir_glob_foreach($numa_path, 'node(\d+)', sub ($node_dir, $node_id) {
+ dir_glob_foreach("$numa_path/$node_dir", 'cpu(\d+)', sub ($cpu_dir, $cpu_id) {
+ $cpu_to_numa->{$cpu_id} = $node_id;
+ });
+ });
+
+ return $cpu_to_numa;
+}
+
+# Build { cpu_id => topology_dir_path } for all CPUs under /sys.
+# in: (no args, reads sysfs)
+# out: { 0 => "/sys/devices/system/cpu/cpu0/topology",
+# 1 => "/sys/devices/system/cpu/cpu1/topology", ... }
+my sub cpu_topology_dirs() {
+ my $base_path = "/sys/devices/system/cpu";
+ opendir(my $dirfd, $base_path) || die "cannot open $base_path\n";
+
+ my $dirs = {};
+ for my $entry (readdir($dirfd)) {
+ next if $entry !~ m/^cpu(\d+)$/;
+ my $topo_path = "$base_path/$entry/topology";
+ next if !-d $topo_path;
+ $dirs->{$1} = $topo_path;
+ }
+ closedir($dirfd);
+
+ return $dirs;
+}
+
+# Read physical_package_id and core_id, normalizing sentinel values to 0.
+# in: "/sys/devices/system/cpu/cpu1/topology"
+# out: (0, 1)
+# core_id seems to be set by the vendor and non-linear.
+# e.g. for the AMD EPYC 7351P 16-Core Processor cpu 4
+# and its thread sibling cpu 16 reside on core 8
+# e.g. /sys/devices/system/cpu/cpu4/topology/
+# out: (0, 8)
+my sub read_cpu_topology($topo_path) {
+ my $socket_id = file_read_firstline("$topo_path/physical_package_id") // 0;
+ my $core_id = file_read_firstline("$topo_path/core_id") // 0;
+
+ # missing topology info is represented as sential value (e.g. nested virtualization)
+ $socket_id = 0 if $socket_id < 0 || $socket_id == 65535;
+ $core_id = 0 if $core_id < 0 || $core_id == 65535;
+
+ return ($socket_id, $core_id);
+}
+
+my sub make_cpu_info($socket_id, $core_id, $numa_node) {
+ return {
+ socket => $socket_id,
+ core => "${socket_id}_${core_id}",
+ numa_node => $numa_node,
+ };
+}
+
+# Add one CPU into topology (mutates cpus/cores/numa_nodes).
+# in: {cpus=>{},cores=>{},numa_nodes=>{}}, 3, {socket=>0,core=>"0_3",numa_node=>1}
+# out: (mutated topology)
+# cpus => {3 => {socket=>0, core=>"0_3", numa_node=>1}}
+# cores => {"0_3" => {numa_node=>1, cpus=>[3]}}
+# numa_nodes => {1 => {cores=>{"0_3"}, cpus=>{3}}}
+my sub add_cpu_to_topology($topology, $cpu_id, $cpu_info) {
+ $topology->{cpus}->{$cpu_id} = $cpu_info;
+
+ my $core_key = $cpu_info->{core};
+ my $numa_node = $cpu_info->{numa_node};
+
+ $topology->{cores}->{$core_key} //= {
+ numa_node => $numa_node,
+ cpus => [],
+ };
+
+ push $topology->{cores}->{$core_key}->{cpus}->@*, $cpu_id;
+
+ $topology->{numa_nodes}->{$numa_node} //= {
+ cores => {},
+ cpus => {},
+ };
+ $topology->{numa_nodes}->{$numa_node}->{cores}->{$core_key} = 1;
+ $topology->{numa_nodes}->{$numa_node}->{cpus}->{$cpu_id} = 1;
+}
+
+=head2 get_host_topology
+
+Reads the host CPU topology from sysfs and returns it as a hashref with
+three keys:
+
+=over
+
+=item * C<cpus> - C<< { $cpu_id => { socket, core, numa_node } } >>, one
+entry per CPU. C<core> is a C<"${socket}_${core_id}"> string.
+
+=item * C<cores> - C<< { $core_key => { numa_node, cpus } } >>, where
+C<cpus> is an arrayref of the CPU ids sharing that physical core.
+
+=item * C<numa_nodes> - C<< { $node_id => { cores, cpus } } >>, where
+C<cores> and C<cpus> are sets (hashrefs of id => 1).
+
+=back
+
+Missing or unavailable topology information is normalized to socket and
+core C<0>, and CPUs without an entry under C</sys/devices/system/node> are
+placed on NUMA node C<0>. B<Dies> if C</sys/devices/system/cpu> cannot be
+opened.
+
+B<Example> - x86 CPU:
+1 socket, 8 cores, 2 CPUs per core (SMT siblings), 2 NUMA nodes
+
+ {
+ # Keyed by CPU id. socket = physical package, core = "socket_coreid"
+ # (the key used to index C<cores> below), numa_node = host NUMA node.
+ #
+ # CPUs 8..15 are the SMT siblings of 0..7: same socket and
+ # core, therefore same NUMA node (e.g. 0 and 8 both on node 0).
+ cpus => {
+ 0 => { socket => 0, core => "0_0", numa_node => 0 },
+ 1 => { socket => 0, core => "0_1", numa_node => 0 },
+ 2 => { socket => 0, core => "0_2", numa_node => 0 },
+ 3 => { socket => 0, core => "0_3", numa_node => 0 },
+ 4 => { socket => 0, core => "0_4", numa_node => 1 },
+ 5 => { socket => 0, core => "0_5", numa_node => 1 },
+ 6 => { socket => 0, core => "0_6", numa_node => 1 },
+ 7 => { socket => 0, core => "0_7", numa_node => 1 },
+ 8 => { socket => 0, core => "0_0", numa_node => 0 },
+ 9 => { socket => 0, core => "0_1", numa_node => 0 },
+ 10 => { socket => 0, core => "0_2", numa_node => 0 },
+ 11 => { socket => 0, core => "0_3", numa_node => 0 },
+ 12 => { socket => 0, core => "0_4", numa_node => 1 },
+ 13 => { socket => 0, core => "0_5", numa_node => 1 },
+ 14 => { socket => 0, core => "0_6", numa_node => 1 },
+ 15 => { socket => 0, core => "0_7", numa_node => 1 },
+ },
+
+ # Keyed by core key ("socket_coreid").
+ cores => {
+ # CPUs 0 and 8 are SMT siblings
+ "0_0" => { numa_node => 0, cpus => [0, 8] },
+ "0_1" => { numa_node => 0, cpus => [1, 9] },
+ "0_2" => { numa_node => 0, cpus => [2, 10] },
+ # CPUs 3 and 11 are SMT siblings (see their entries above).
+ "0_3" => { numa_node => 0, cpus => [3, 11] },
+ "0_4" => { numa_node => 1, cpus => [4, 12] },
+ "0_5" => { numa_node => 1, cpus => [5, 13] },
+ "0_6" => { numa_node => 1, cpus => [6, 14] },
+ "0_7" => { numa_node => 1, cpus => [7, 15] },
+ },
+
+ # Keyed by NUMA node id. Each value holds two sets (id => 1): the
+ # core keys on that node, and the CPU ids on that node. The 1 values
+ # are not counts; only the keys are meaningful.
+ numa_nodes => {
+ 0 => {
+ cores => { "0_0" => 1, "0_1" => 1, "0_2" => 1, "0_3" => 1 },
+ cpus => { 0 => 1, 1 => 1, 2 => 1, 3 => 1,
+ 8 => 1, 9 => 1, 10 => 1, 11 => 1 },
+ },
+ 1 => {
+ cores => { "0_4" => 1, "0_5" => 1, "0_6" => 1, "0_7" => 1 },
+ cpus => { 4 => 1, 5 => 1, 6 => 1, 7 => 1,
+ 12 => 1, 13 => 1, 14 => 1, 15 => 1 },
+ },
+ },
+ }
+=cut
+sub get_host_topology() {
+ my $cpu_to_numa = cpu_to_numa_map();
+
+ my $topology = {
+ cpus => {},
+ cores => {},
+ numa_nodes => {},
+ };
+
+ my $cpu_dirs = cpu_topology_dirs();
+
+ for my $cpu_id (sort { $a <=> $b } keys $cpu_dirs->%*) {
+ my ($socket_id, $core_id) = read_cpu_topology($cpu_dirs->{$cpu_id});
+
+ my $numa_node = $cpu_to_numa->{$cpu_id} // 0;
+
+ my $cpu_info = make_cpu_info($socket_id, $core_id, $numa_node);
+ add_cpu_to_topology($topology, $cpu_id, $cpu_info);
+ }
+
+ return $topology;
+}
+
+my sub read_sysfs_trim($path) {
+ return undef if !-e $path;
+ my $val = file_read_firstline($path);
+ return undef if !defined($val);
+ $val =~ s/^\s+|\s+$//g;
+ return $val;
+}
+
+# in: /sys/devices/system/cpu/online = 0-7
+# out: [0, 1, 2, 3, 4, 5, 6, 7]
+my sub online_cpu_ids() {
+ my $content = read_sysfs_trim("/sys/devices/system/cpu/online");
+ return [] if !$content;
+
+ my @ids;
+ for my $part (split(/,/, $content)) {
+ if ($part =~ /^(\d+)-(\d+)$/) {
+ push @ids, ($1 .. $2);
+ } elsif ($part =~ /^(\d+)$/) {
+ push @ids, $1;
+ }
+ }
+ return \@ids;
+}
+
+# Build a stable "CPU type" key for a single online CPU.
+# On ARM: MIDR register + cache geometry.
+# On x86 (no MIDR): cache geometry alone (level/type/line/ways/sets).
+# On intel CPUs different P/E cores or big.LITTLE cores produce different keys because
+# they have different cache geometries even when /proc/cpuinfo model names
+# are identical or absent.
+# Note: Only tested on AMD x86 CPUs, so take this with a grain of salt.
+my sub cpu_type_key($cpu_id) {
+ my $cpu_path = "/sys/devices/system/cpu/cpu$cpu_id";
+
+ my $key = '';
+
+ # MIDR (ARM only). Absent on x86.
+ my $midr = read_sysfs_trim("$cpu_path/regs/identification/midr_el1");
+ $key .= defined($midr) ? $midr : 'no-midr';
+
+ # Cache geometry: sorted cache indices, each with level/type/line/ways/sets.
+ # This distinguishes P-cores from E-cores on x86 (different L2/L3 sizes)
+ # and provides a secondary signal on ARM.
+ my $cache_base = "$cpu_path/cache";
+ if (-d $cache_base) {
+ opendir(my $dh, $cache_base) or return $key;
+ my @indices = sort { $a <=> $b }
+ map { /^index(\d+)$/ ? $1 : () }
+ readdir($dh);
+ closedir($dh);
+
+ for my $idx (@indices) {
+ my $idx_path = "$cache_base/index$idx";
+ for my $field (qw(level type coherency_line_size ways_of_associativity number_of_sets)) {
+ my $val = read_sysfs_trim("$idx_path/$field");
+ $key .= '/' . (defined($val) ? $val : '');
+ }
+ }
+ }
+
+ return $key;
+}
+
+# Group online CPUs by cpu_type_key. capacity defaults to 1024/CPU.
+# out: ({ "abc/..." => {cpus => {0, 1, 2, 3}, capacity => 4096} },
+# [ "abc/..." ])
+sub group_cpus_by_type() {
+ my $online_ids = online_cpu_ids();
+ return ({}, []) if !$online_ids || !@$online_ids;
+
+ my %groups;
+ my @keys;
+
+ for my $cpu_id (@$online_ids) {
+ my $key = cpu_type_key($cpu_id);
+ if (!$groups{$key}) {
+ $groups{$key} = { cpus => [], capacity => 0 };
+ push @keys, $key;
+ }
+ push $groups{$key}{cpus}->@*, $cpu_id;
+
+ my $cap = 1024;
+ my $cap_val = read_sysfs_trim("/sys/devices/system/cpu/cpu$cpu_id/cpu_capacity");
+ if (defined($cap_val) && $cap_val =~ /^(\d+)$/) {
+ $cap = int($1);
+ }
+ $groups{$key}{capacity} += $cap;
+ }
+
+ # Deterministic key order: by first CPU id in each group.
+ @keys = sort { $groups{$a}{cpus}[0] <=> $groups{$b}{cpus}[0] } @keys;
+
+ return (\%groups, \@keys);
+}
+
+sub are_cpus_heterogenous($cpu_ids_arrayref) {
+ my ($groups, $keys) = group_cpus_by_type();
+ return 0 if !@$keys || @$keys <= 1;
+
+ my %requested = map { $_ => 1 } @$cpu_ids_arrayref;
+ my $seen_key;
+
+ for my $key (@$keys) {
+ for my $cpu_id (@{$groups->{$key}{cpus}}) {
+ next if !$requested{$cpu_id};
+ if (!defined $seen_key) {
+ $seen_key = $key;
+ } elsif ($key ne $seen_key) {
+ return 1;
+ }
+ last;
+ }
+ }
+ return 0;
+}
+
+sub filter_by_affinity($topology, $affinity_string) {
+ return $topology if !defined($affinity_string) || $affinity_string eq '';
+
+ my ($_count, $affinity_members) = PVE::CpuSet::parse_cpuset($affinity_string);
+
+ my $filtered = {
+ cpus => {},
+ cores => {},
+ numa_nodes => {},
+ };
+
+ for my $cpu_id (sort { $a <=> $b } keys $topology->{cpus}->%*) {
+ next if !$affinity_members->{$cpu_id};
+ my $cpu_info = $topology->{cpus}->{$cpu_id};
+ add_cpu_to_topology($filtered, $cpu_id, $cpu_info);
+ }
+
+ return $filtered;
+}
+
+# Group selected host CPUs by socket -> core -> threads.
+# in: $topology, [0, 1, 4, 5]
+# out: ({ 0 => { "0_0" => [0, 4], "0_1" => [1, 5] } }, undef)
+# in: $topology, [0, 99]
+# out: (undef, "CPU 99 not found in host topology")
+my sub group_cpus_by_socket_core($topology, $selected_cpus) {
+ my %sockets;
+ for my $cpu_id (@$selected_cpus) {
+ my $info = $topology->{cpus}->{$cpu_id};
+ return (undef, "CPU $cpu_id not found in host topology") if !$info;
+ push @{$sockets{$info->{socket}}{$info->{core}}}, $cpu_id;
+ }
+ return (\%sockets, undef);
+}
+
+# Check every socket has same core count and every core same thread count.
+# in: { 0 => { "0_0" => [0, 8], "0_4" => [4, 12] } }
+# out: (2, 2)
+# in: { 0 => { "0_0" => [0, 8] }, 1 => { "1_0" => [1, 9], "1_1" => [2, 10] } }
+# out: (undef, undef)
+my sub validate_consistency($sockets) {
+ my ($expected_cores, $expected_threads) = (-1, -1);
+
+ for my $socket (keys %$sockets) {
+ my $core_count = scalar keys %{$sockets->{$socket}};
+ return (undef, undef)
+ if $expected_cores != -1 && $core_count != $expected_cores;
+ $expected_cores = $core_count;
+
+ for my $core (keys %{$sockets->{$socket}}) {
+ my $thread_count = scalar @{$sockets->{$socket}->{$core}};
+ return (undef, undef)
+ if $expected_threads != -1 && $thread_count != $expected_threads;
+ $expected_threads = $thread_count;
+ }
+ }
+
+ return ($expected_cores, $expected_threads);
+}
+
+my sub flat_fallback_result($total, $warning = undef) {
+ $warning //= "Instance uses a CPU pinning profile which doesn't match"
+ . " hardware layout; falling back to flat topology"
+ . " (sockets=1, cores=$total, threads=1)";
+ return {
+ valid => 0,
+ sockets => 1,
+ cores => $total,
+ threads => 1,
+ warning => $warning,
+ };
+}
+
+=encoding utf8
+
+=head2 validate_pinned_topology
+
+Validates that the host CPUs in C<$selected_cpus> (an arrayref of host CPU
+IDs) form a balanced topology matching the guest configuration.
+
+=over 4
+
+=item 1. Groups selected host CPUs by physical socket and physical core.
+
+=item 2. Checks that every socket contains the same number of cores.
+
+=item 3. Checks that every core contains the same number of threads.
+
+=item 4. Checks that C<sockets * cores_per_socket * threads_per_core> equals
+the total number of selected CPUs.
+
+=back
+
+Returns a hashref:
+
+ {
+ valid => 1 | 0,
+ sockets => $nr_sockets,
+ cores => $nr_cores_per_socket,
+ threads => $nr_threads_per_core,
+ warning => $msg | undef,
+ }
+
+If the topology is invalid, the result contains a flat fallback
+(C<sockets=1, cores=N, threads=1>) and a warning string.
+
+=cut
+sub validate_pinned_topology($topology, $selected_cpus, $guest_sockets, $guest_cores) {
+ my $total = scalar @$selected_cpus;
+
+ # Empty selection: trust the guest-configured topology.
+ return {
+ valid => 1,
+ sockets => $guest_sockets,
+ cores => $guest_cores,
+ threads => 1,
+ warning => undef,
+ } if $total == 0;
+
+ my ($sockets, $err) = group_cpus_by_socket_core($topology, $selected_cpus);
+ return flat_fallback_result($total, $err) if defined($err);
+
+ my ($expected_cores, $expected_threads) = validate_consistency($sockets);
+ return flat_fallback_result($total)
+ if !defined $expected_cores || !defined $expected_threads;
+
+ my $nr_sockets = scalar keys %$sockets;
+ return flat_fallback_result($total)
+ if $nr_sockets * $expected_cores * $expected_threads != $total;
+
+ return {
+ valid => 1,
+ sockets => $nr_sockets,
+ cores => $expected_cores,
+ threads => $expected_threads,
+ warning => undef,
+ };
+}
+
+1;
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [RFC qemu-server v2 2/4] pinning: add NUMA allocator and reservation tracking
2026-09-21 9:53 [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing Elias Huhsovitz
@ 2026-09-21 9:54 ` Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 3/4] memory: integrate pinning-aware NUMA memory binding Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 4/4] pinning: integrate cpu pinning into vm lifecycle Elias Huhsovitz
3 siblings, 0 replies; 5+ messages in thread
From: Elias Huhsovitz @ 2026-09-21 9:54 UTC (permalink / raw)
To: pve-devel; +Cc: Elias Huhsovitz
Introduce Allocator.pm to handle NUMA node placement and CPU thread
allocation.
The allocator assigns guest NUMA nodes to host NUMA nodes based on
uncommitted memory and CPU load. To prevent placement collisions during
concurrent VM starts, it uses a file lock and a JSON tracker in /run to
record pending reservations.
Three allocation strategies are supported:
1. assign_numa_nodes: explicit or auto-placed 1:1 guest-to-host mapping.
2. balance_numa_nodes: spreads vCPUs across multiple nodes proportional
to their capacity (used for 'balanced' mode).
3. allocate_one_to_one: spreads vCPUs across distinct physical cores to
avoid SMT collisions.
Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
src/PVE/QemuServer/Pinning/Allocator.pm | 590 ++++++++++++++++++++++++
1 file changed, 590 insertions(+)
create mode 100644 src/PVE/QemuServer/Pinning/Allocator.pm
diff --git a/src/PVE/QemuServer/Pinning/Allocator.pm b/src/PVE/QemuServer/Pinning/Allocator.pm
new file mode 100644
index 00000000..813c5aca
--- /dev/null
+++ b/src/PVE/QemuServer/Pinning/Allocator.pm
@@ -0,0 +1,590 @@
+package PVE::QemuServer::Pinning::Allocator;
+
+use v5.36;
+
+use JSON;
+use PVE::Tools qw(lock_file file_get_contents file_set_contents file_read_firstline);
+use PVE::QemuConfig;
+use PVE::QemuServer::Pinning::Config;
+use PVE::QemuServer::Memory;
+
+use constant RESERVATIONS_FILE => '/run/qemu-server/numa-reservations.json';
+use constant LOCK_FILE => '/var/lock/pve-numa-pin.lck';
+
+my sub host_cpu_capacity($topology) {
+ return {
+ map { $_ => scalar keys $topology->{numa_nodes}->{$_}->{cpus}->%* }
+ keys $topology->{numa_nodes}->%*
+ };
+}
+
+# Return MemTotal per NUMA node from
+# /sys/devices/system/node/nodeN/meminfo (bytes).
+# out: { 0 => 16907931648, 1 => 16907931648, 2 => 16907931648, 3 => 16907931648 }
+my sub host_memory_capacity() {
+ my $mem = {};
+ my $base = "/sys/devices/system/node";
+ opendir(my $dh, $base) or return $mem;
+ while (my $entry = readdir($dh)) {
+ next if $entry !~ /^node(\d+)$/;
+ my $node_id = $1;
+ my $meminfo_path = "$base/$entry/meminfo";
+ my $line = file_read_firstline($meminfo_path);
+ next if !defined($line);
+ if ($line =~ /MemTotal:\s+(\d+)\s+kB/) {
+ $mem->{$node_id} = $1 * 1024;
+ } else {
+ $mem->{$node_id} = 0;
+ }
+ }
+ closedir($dh);
+ return $mem;
+}
+
+# Read persisted reservations from /run, empty hash if missing/corrupt.
+# in: (no args)
+# out: { 100 => {nodes => {0 => 4}, memory => {0 => 4294967296}} }
+my sub load_reservations() {
+ return {} if !-e RESERVATIONS_FILE;
+ my $data = eval { file_get_contents(RESERVATIONS_FILE) };
+ return {} if !defined($data) || $data eq '';
+ my $reservations = eval { decode_json($data) };
+ return $reservations // {};
+}
+
+my sub save_reservations($reservations) {
+ file_set_contents(RESERVATIONS_FILE, encode_json($reservations));
+}
+
+# Drop reservation entries where VM has no running pid file.
+my sub prune_stale_reservations($reservations, $vm_id) {
+ for my $other_vmid (keys %$reservations) {
+ next if $other_vmid == $vm_id;
+ my $pid_file = "/run/qemu-server/$other_vmid.pid";
+ delete $reservations->{$other_vmid} if !-e $pid_file;
+ }
+}
+
+sub clear_reservation($vmid) {
+ lock_file(LOCK_FILE, 10, sub {
+ my $reservations = load_reservations();
+ delete $reservations->{$vmid};
+ save_reservations($reservations);
+ });
+}
+
+# Sum vCPU counts and memory per host node across all running pinned VMs.
+# in: 100
+# out: ({ 0 => 8, 1 => 4 }, { 0 => 8589934592, 1 => 4294967296 }
+my sub calculate_running_load($current_vmid) {
+ my $cpu_load = {};
+ my $mem_load = {};
+
+ my $conf_dir = "/etc/pve/qemu-server";
+ opendir(my $dh, $conf_dir) or return ($cpu_load, $mem_load);
+
+ while (my $file = readdir($dh)) {
+ next if $file !~ /^(\d+)\.conf$/;
+ my $other_vmid = $1;
+ next if $other_vmid == $current_vmid;
+
+ my $pid_file = "/run/qemu-server/$other_vmid.pid";
+ next if !-e $pid_file;
+
+ eval {
+ my $other_conf = PVE::QemuConfig->load_config($other_vmid);
+ my $pinning = PVE::QemuServer::Pinning::Config::parse_pinning(
+ $other_conf->{pinning},
+ );
+ return if $pinning->{mode} eq PVE::QemuServer::Pinning::Config::DEFAULT_VCPU_PINNING;
+
+ my $numa_map = PVE::QemuServer::Pinning::Config::build_vnuma_vcpu_map($other_conf);
+ my $vm_memory_mib = PVE::QemuServer::Memory::get_current_memory($other_conf->{memory});
+ my $vm_memory_bytes = $vm_memory_mib * 1024 * 1024;
+ my $node_count = scalar keys %$numa_map;
+ my $memory_per_node = $node_count > 0
+ ? int($vm_memory_bytes / $node_count)
+ : 0;
+
+ for my $numa_node (keys %$numa_map) {
+ my $hostnode = $numa_map->{$numa_node}->{hostnode};
+ next if !defined($hostnode);
+
+ my $vcpu_count = scalar keys $numa_map->{$numa_node}->{vcpus}->%*;
+ $cpu_load->{$hostnode} = ($cpu_load->{$hostnode} // 0) + $vcpu_count;
+ $mem_load->{$hostnode} = ($mem_load->{$hostnode} // 0) + $memory_per_node;
+ }
+ } or do {
+ my $error = $@ || 'Unknown failure';
+ warn "Failed to read NUMA load from running VM $other_vmid: $error\n";
+ }
+ }
+ closedir($dh);
+
+ return ($cpu_load, $mem_load);
+}
+
+# Fold other VMs' pending reservations into load maps.
+# in: ({0=>4}, {0=>4294967296}, {101=>{nodes=>{0=>4},memory=>{0=>4294967296}}}, 100)
+# mutated: cpu_load={0=>8}, mem_load={0=>8589934592}
+my sub apply_pending_load($cpu_load, $mem_load, $reservations, $current_vmid) {
+ for my $other_vmid (keys %$reservations) {
+ next if $other_vmid == $current_vmid;
+ my $entry = $reservations->{$other_vmid};
+ for my $node (keys %{$entry->{nodes} // {}}) {
+ $cpu_load->{$node} = ($cpu_load->{$node} // 0) + $entry->{nodes}->{$node};
+ }
+ for my $node (keys %{$entry->{memory} // {}}) {
+ $mem_load->{$node} = ($mem_load->{$node} // 0) + $entry->{memory}->{$node};
+ }
+ }
+}
+
+my sub run_vm_with_load_data($vmid, $function) {
+ my $worker = sub {
+ my $reservations = load_reservations();
+ prune_stale_reservations($reservations, $vmid);
+
+ my ($cpu_used, $mem_used) = calculate_running_load($vmid);
+ apply_pending_load($cpu_used, $mem_used, $reservations, $vmid);
+
+ $function->($cpu_used, $mem_used, $reservations);
+
+ save_reservations($reservations);
+ };
+ eval { lock_file(LOCK_FILE, 10, $worker); };
+ die $@ if $@;
+}
+
+# Restrict node list to pinning=balanced,hostnodes=... if set.
+# in: {pinning => "balanced,hostnodes=0;1"}, (0, 1, 2, 3)
+# out: (0, 1)
+my sub filter_by_hostnodes($conf, @nodes) {
+ my $pinning_conf = PVE::QemuServer::Pinning::Config::parse_pinning($conf->{pinning});
+ my $allowed = $pinning_conf->{hostnodes};
+ return @nodes if !defined($allowed);
+
+ my %allowed_set = map { $_ => 1 } @$allowed;
+ my @filtered = grep { $allowed_set{$_} } @nodes;
+ die "None of the host-nodes specified in pinning exist on"
+ . " this host (requested: " . join(';', @$allowed) . ")\n"
+ if !@filtered;
+ return @filtered;
+}
+
+# Pick host node with
+# 1. most free memory
+# 2. then least CPU load
+# that fits $vcpu_count.
+my sub pick_host_node($cpu_capacity, $mem_capacity, $cpu_used, $mem_used, $order, $vcpu_count) {
+ my @candidates;
+
+ for my $node (@$order) {
+ my $cpu_load = $cpu_used->{$node} // 0;
+ next if ($cpu_capacity->{$node} // 0) - $cpu_load < $vcpu_count;
+ my $free_mem = ($mem_capacity->{$node} // 0) - ($mem_used->{$node} // 0);
+ push @candidates, { node => $node, cpu_load => $cpu_load, free_mem => $free_mem };
+ }
+
+ return undef if !@candidates;
+
+ @candidates = sort {
+ $b->{free_mem} <=> $a->{free_mem}
+ || $a->{cpu_load} <=> $b->{cpu_load}
+ } @candidates;
+
+ return $candidates[0]->{node};
+}
+
+# Bump vCPU and memory used counters for a node
+my sub reserve_host_node($cpu_used, $mem_used, $cpu_capacity, $node, $vcpu_count, $memory_bytes) {
+ my $free = ($cpu_capacity->{$node} // 0) - ($cpu_used->{$node} // 0);
+ die "Not enough CPUs available on NUMA node $node\n" if $free < $vcpu_count;
+ $cpu_used->{$node} = ($cpu_used->{$node} // 0) + $vcpu_count;
+ $mem_used->{$node} = ($mem_used->{$node} // 0) + $memory_bytes;
+}
+
+# Rotate a node list by $seed so different VMs start at different nodes.
+# in: [0, 1, 2, 3], 2
+# out: (2, 3, 0, 1)
+my sub rotation_order($nodes, $seed) {
+ return @$nodes if !@$nodes;
+ my $start = $seed % @$nodes;
+ return (
+ @$nodes[$start .. $#$nodes],
+ @$nodes[0 .. $start - 1],
+ );
+}
+
+my sub compute_memory_per_node($conf, $guest_node_count) {
+ my $vm_memory_bytes =
+ PVE::QemuServer::Memory::get_current_memory($conf->{memory}) * 1024 * 1024;
+ return $guest_node_count > 0
+ ? int($vm_memory_bytes / $guest_node_count)
+ : 0;
+}
+
+my sub count_vcpus($vcpu_to_numa_map) {
+ return scalar keys %$vcpu_to_numa_map;
+}
+
+# Host node search order: affinity-restricted, rotated by vmid.
+# in: {pinning => "balanced,hostnodes=0;1"}, $topology, 100
+# out: (1, 0) # 100 % 2 = 0, so rotated for vmid 101 → (1, 0)
+my sub build_search_order($conf, $topology, $vmid) {
+ my @available = filter_by_hostnodes($conf,
+ sort keys $topology->{numa_nodes}->%*);
+ return rotation_order(\@available, $vmid);
+}
+
+# Reserve host nodes that are explicitly pinned using: numaX hostnodes=)
+my sub reserve_explicit_nodes($numa_vcpu_map, $cpu_used, $mem_used, $cpu_capacity, $memory_per_node) {
+ for my $numa_node (sort keys $numa_vcpu_map->%*) {
+ my $hostnode = $numa_vcpu_map->{$numa_node}->{hostnode};
+ next if !defined($hostnode);
+
+ my $vcpu_count = scalar keys $numa_vcpu_map->{$numa_node}->{vcpus}->%*;
+ reserve_host_node(
+ $cpu_used, $mem_used, $cpu_capacity,
+ $hostnode, $vcpu_count, $memory_per_node,
+ );
+ }
+}
+
+=head2 assign_unassigned_nodes
+
+Assigns a host NUMA node to every guest NUMA node in C<$numa_vcpu_map>
+that does not already have a C<hostnode> set. Guest nodes with an explicit
+binding (from C<numaX=...,hostnodes=N>) are skipped; they are reserved
+separately by C<reserve_explicit_nodes()>.
+
+For each unbound guest node, C<pick_host_node()> selects the best host node
+that can fit the guest node's vCPU count. The order is defined by C<$search_order>.
+The chosen node's CPU and memory usage are then recorded.
+
+B<Mutates:> C<$numa_vcpu_map> (sets C<hostnode>), C<$cpu_used>, C<$mem_used>.
+
+B<Dies> if no host node has enough free CPUs for a guest node.
+
+=over
+
+=item C<$numa_vcpu_map> - C<< { $guest_node => { vcpus => { $v => 1, ... }, hostnode => $host_node } } >>
+
+=item C<$cpu_capacity> - C<< { $host_node => $total_cpus } >>
+
+=item C<$mem_capacity> - C<< { $host_node => $total_bytes } >>
+
+=item C<$cpu_used>, C<$mem_used> - current per-host-node usage, updated in place.
+
+=item C<$search_order> - arrayref of host node ids, tried in this order.
+
+=item C<$memory_per_node> - bytes charged to the host node for this guest node.
+
+=back
+
+=cut
+my sub assign_unassigned_nodes(
+ $numa_vcpu_map, $cpu_capacity, $mem_capacity,
+ $cpu_used, $mem_used, $search_order, $memory_per_node
+) {
+ for my $numa_node (sort keys $numa_vcpu_map->%*) {
+ next if defined($numa_vcpu_map->{$numa_node}->{hostnode});
+
+ my $vcpu_count = scalar keys $numa_vcpu_map->{$numa_node}->{vcpus}->%*;
+ my $hostnode = pick_host_node(
+ $cpu_capacity, $mem_capacity,
+ $cpu_used, $mem_used,
+ $search_order, $vcpu_count,
+ ) // die "Could not find a fitting host NUMA node for guest"
+ . " NUMA node $numa_node\n";
+
+ reserve_host_node(
+ $cpu_used, $mem_used, $cpu_capacity,
+ $hostnode, $vcpu_count, $memory_per_node,
+ );
+ $numa_vcpu_map->{$numa_node}->{hostnode} = $hostnode;
+ }
+}
+
+my sub build_reservation_from_guest_map($numa_vcpu_map, $memory_per_node) {
+ my ($nodes, $memory) = ({}, {});
+ for my $numa_node (keys %$numa_vcpu_map) {
+ my $hostnode = $numa_vcpu_map->{$numa_node}->{hostnode};
+ my $vcpu_count = scalar keys $numa_vcpu_map->{$numa_node}->{vcpus}->%*;
+ $nodes->{$hostnode} = ($nodes->{$hostnode} // 0) + $vcpu_count;
+ $memory->{$hostnode} = ($memory->{$hostnode} // 0) + $memory_per_node;
+ }
+ return { nodes => $nodes, memory => $memory };
+}
+
+my sub flatten_vcpu_to_host_map($numa_vcpu_map) {
+ my %vcpu_to_host;
+ for my $numa_node (keys %$numa_vcpu_map) {
+ my $hostnode = $numa_vcpu_map->{$numa_node}->{hostnode};
+ $vcpu_to_host{$_} = $hostnode
+ for keys $numa_vcpu_map->{$numa_node}->{vcpus}->%*;
+ }
+ return \%vcpu_to_host;
+}
+
+# Distribute vCPUs across nodes proportionally to CPU capacity.
+# in: [0, 1], {0=>8, 1=>8}, 6
+# out: {0=>0, 1=>0, 2=>0, 3=>1, 4=>1, 5=>1} # remainder given to first node
+# with key: vcpu, val: hostcpu
+# in: [0, 1, 2, 3], {0=>8, 1=>8, 2=>4, 3=>4}, 5
+# out: {0=>0, 1=>0, 2=>0, 3=>1, 4=>1} # floor + remainder to node 0
+my sub distribute_vcpus_proportional($selected_nodes, $cpu_capacity, $total_vcpus) {
+ my $total_capacity = 0;
+ $total_capacity += $cpu_capacity->{$_} // 0 for @$selected_nodes;
+
+ my @per_node;
+ my $assigned = 0;
+ for my $node (@$selected_nodes) {
+ my $count = int($total_vcpus * ($cpu_capacity->{$node} // 0) / $total_capacity);
+ push @per_node, $count;
+ $assigned += $count;
+ }
+ $per_node[0] += $total_vcpus - $assigned;
+
+ my %vcpu_to_host;
+ my $vcpu_id = 0;
+ for my $i (0 .. $#$selected_nodes) {
+ for (1 .. $per_node[$i]) {
+ $vcpu_to_host{$vcpu_id++} = $selected_nodes->[$i];
+ }
+ }
+ return \%vcpu_to_host;
+}
+
+my sub build_reservation_from_vcpu_map($vcpu_to_host, $selected_nodes, $memory_per_node) {
+ my ($nodes, $memory) = ({}, {});
+ for my $node (@$selected_nodes) {
+ my $node_vcpus = scalar grep { $vcpu_to_host->{$_} == $node }
+ keys %$vcpu_to_host;
+ $nodes->{$node} = $node_vcpus;
+ $memory->{$node} = $memory_per_node;
+ }
+ return { nodes => $nodes, memory => $memory };
+}
+
+# Assign each guest NUMA node to a host NUMA node.
+#
+# Guest nodes with an explicit "hostnodes=N" in their numaX entry are
+# reserved first. Guest nodes without one are auto-placed by
+# assign_unassigned_nodes().
+#
+# in: $conf = {
+# numa => 1,
+# numa0 => "cpus=0-1,hostnodes=0",
+# numa1 => "cpus=2-3,hostnodes=1",
+# },
+# $topology, 100
+# out: {0=>0, 1=>0, 2=>1, 3=>1} with vcpu id => host node id
+sub assign_numa_nodes($conf, $topology, $vmid) {
+ my $numa_vcpu_map = PVE::QemuServer::Pinning::Config::build_vnuma_vcpu_map($conf);
+ my $cpu_capacity = host_cpu_capacity($topology);
+ my $mem_capacity = host_memory_capacity();
+ my $memory_per_node = compute_memory_per_node($conf, scalar keys %$numa_vcpu_map);
+ my @search_order = build_search_order($conf, $topology, $vmid);
+
+ run_vm_with_load_data($vmid, sub ($cpu_used, $mem_used, $reservations) {
+ reserve_explicit_nodes(
+ $numa_vcpu_map, $cpu_used, $mem_used, $cpu_capacity, $memory_per_node);
+
+ assign_unassigned_nodes(
+ $numa_vcpu_map, $cpu_capacity, $mem_capacity,
+ $cpu_used, $mem_used, \@search_order, $memory_per_node);
+
+ $reservations->{$vmid} = build_reservation_from_guest_map(
+ $numa_vcpu_map, $memory_per_node);
+ });
+
+ return flatten_vcpu_to_host_map($numa_vcpu_map);
+}
+
+=head2 balance_numa_nodes
+
+ my $vcpu_to_host = balance_numa_nodes($conf, $topology, $vmid);
+
+Selects the smallest set of host NUMA nodes that fits the VM's vCPUs and
+memory, then spreads vCPUs across them proportionally to each node's CPU
+capacity. Used for C<pinning=balanced>. Inspired by Incus's
+C<balanceNUMANodes()> in F<incus/internal/server/instance/drivers/driver_common.go>.
+
+=over
+
+=item 1. Filter by C<hostnodes> restriction.
+
+=item 2. Sort by uncommitted memory (desc), then CPU load (asc).
+
+=item 3. Use as many nodes as needed to fit CPU + memory.
+
+=item 4. Distribute vCPUs proportionally.
+
+=item 5. Persist reservation.
+
+=back
+
+Returns C<< { $vcpu_id => $host_node_id } >>.
+
+B<Mutates:> writes the VM's reservation to the reservations file under the
+global NUMA lock; stale entries are pruned in the same critical section.
+
+B<Dies> if the pinning restriction names non-existent host nodes, or if no
+set of nodes can fit the VM.
+
+Example (2 equal nodes, 4 vCPUs, 4 GiB):
+
+ balance_numa_nodes({vcpus=>4, memory=>4096, pinning=>'balanced'}, $topo, 100);
+ # => { 0 => 0, 1 => 0, 2 => 1, 3 => 1 }
+
+=cut
+sub balance_numa_nodes($conf, $topology, $vmid) {
+ my $cpu_capacity = host_cpu_capacity($topology);
+ my $mem_capacity = host_memory_capacity();
+
+ my $vcpus = $conf->{vcpus}
+ // (($conf->{sockets} // 1) * ($conf->{cores} // 1));
+ my $vm_memory_mib = PVE::QemuServer::Memory::get_current_memory($conf->{memory});
+ my $vm_memory_bytes = $vm_memory_mib * 1024 * 1024;
+
+ my @nodes = filter_by_hostnodes($conf,
+ sort keys $topology->{numa_nodes}->%*);
+
+ my $vcpu_to_numa_map;
+
+ run_vm_with_load_data($vmid, sub ($cpu_used, $mem_used, $reservations) {
+ # Sort: most uncommitted memory first, then least CPU load.
+ my @sorted = sort {
+ (($mem_capacity->{$b} // 0) - ($mem_used->{$b} // 0))
+ <=>
+ (($mem_capacity->{$a} // 0) - ($mem_used->{$a} // 0))
+ ||
+ ($cpu_used->{$a} // 0) <=> ($cpu_used->{$b} // 0)
+ } @nodes;
+
+ # Use as many nodes as needed to fit CPU and memory.
+ my @selected;
+ my $cpu_total = 0;
+ my $mem_free = 0;
+ for my $node (@sorted) {
+ $cpu_total += $cpu_capacity->{$node} // 0;
+ $mem_free += ($mem_capacity->{$node} // 0) - ($mem_used->{$node} // 0);
+ push @selected, $node;
+ last if $vcpus <= $cpu_total && $vm_memory_bytes <= $mem_free;
+ }
+
+ die "Could not fit VM ($vcpus vCPUs, $vm_memory_mib MiB) across"
+ . " available host NUMA nodes\n"
+ if $vcpus > $cpu_total || $vm_memory_bytes > $mem_free;
+
+ $vcpu_to_numa_map = distribute_vcpus_proportional(
+ \@selected, $cpu_capacity, $vcpus);
+
+ my $memory_per_node = int($vm_memory_bytes / scalar(@selected));
+ $reservations->{$vmid} = build_reservation_from_vcpu_map(
+ $vcpu_to_numa_map, \@selected, $memory_per_node);
+ });
+
+ return $vcpu_to_numa_map;
+}
+
+# Returns: {node => {core_key => [cpu_ids]}}
+my sub build_numa_node_core_pool($topology) {
+ my $pool = {};
+ for my $node (keys $topology->{numa_nodes}->%*) {
+ for my $core_key (keys $topology->{numa_nodes}->{$node}->{cores}->%*) {
+ my @cpus = $topology->{cores}->{$core_key}->{cpus}->@*;
+ $pool->{$node}->{$core_key} = [@cpus];
+ }
+ }
+ return $pool;
+}
+
+# Pop one CPU from the core with the most remaining CPUs (SMT spread).
+# in: {"0_0" => [0, 4], "0_1" => [1, 5]}, 0
+# out: 0 # pool now {"0_0" => [4], "0_1" => [1, 5]}
+my sub take_cpu_from_core_pool($core_pool, $vcpu) {
+ my @cores = sort {
+ $core_pool->{$b}->@* <=> $core_pool->{$a}->@*
+ || $a cmp $b
+ } keys %$core_pool;
+
+ die "No available cores for vcpu $vcpu\n" if !@cores;
+
+ my $core_key = $cores[0];
+ my $cpus = $core_pool->{$core_key};
+ my $cpu = shift $cpus->@*;
+ delete $core_pool->{$core_key} if !$cpus->@*;
+ return $cpu;
+}
+
+my sub take_cpu_from_node($pool, $node, $vcpu) {
+ die "No available cores on NUMA node $node for vcpu $vcpu\n"
+ if !keys %{$pool->{$node} // {}};
+ return take_cpu_from_core_pool($pool->{$node}, $vcpu);
+}
+
+# Flat per-vCPU pinning: spread across all cores, ignoring NUMA boundaries.
+# take_cpu_from_core_pool() sees all 16 cores in one pool, so the spread
+# is over physical cores first (lowest core key wins each tie).
+# in: $topology (above), $vcpu_count = 4
+# out: $vcpu_to_cpu_map = { vcpu_id => host_cpu_id } = {0=>0, 1=>1, 2=>2, 3=>3}
+sub allocate_one_to_one($topology, $vcpu_to_numa_map, $vcpu_count) {
+ my $node_cores_pool = build_numa_node_core_pool($topology);
+ my $final_map = {};
+
+ for my $vcpu (0 .. $vcpu_count - 1) {
+ my $node = $vcpu_to_numa_map->{$vcpu};
+ if (!defined($node) || !defined($node_cores_pool->{$node})) {
+ die "No available CPUs for pinning\n" if !keys %$node_cores_pool;
+ ($node) = sort keys %$node_cores_pool;
+ }
+ $final_map->{$vcpu} = take_cpu_from_node($node_cores_pool, $node, $vcpu);
+ }
+
+ return $final_map;
+}
+
+# Flat per-vCPU pinning: spread across all cores, ignoring NUMA.
+# in: $topology, 4
+# out: {0=>0, 1=>1, 2=>2, 3=>3}
+sub allocate_one_to_one_flat($topology, $vcpu_count) {
+ my $node_cores_pool = build_numa_node_core_pool($topology);
+
+ my %flat_pool;
+ for my $node (keys %$node_cores_pool) {
+ for my $core_key (keys %{$node_cores_pool->{$node}}) {
+ $flat_pool{$core_key} = $node_cores_pool->{$node}->{$core_key};
+ }
+ }
+
+ my $final_map = {};
+ for my $vcpu (0 .. $vcpu_count - 1) {
+ die "No available CPUs for pinning\n" if !keys %flat_pool;
+ $final_map->{$vcpu} = take_cpu_from_core_pool(\%flat_pool, $vcpu);
+ }
+
+ return $final_map;
+}
+
+# Each vCPU gets ALL CPUs of its assigned host node as a comma-joined set.
+# in: $topology,
+# $vcpu_to_numa_map = { vcpu_id => host_node_id } = {0=>0, 1=>0, 2=>1, 3=>1}
+# out: $vcpu_to_cpuset_map = { vcpu_id => "cpu,cpu,..." } = {
+# 0 => "0,1,2,3,16,17,18,19",
+# 1 => "0,1,2,3,16,17,18,19",
+# 2 => "4,5,6,7,20,21,22,23",
+# 3 => "4,5,6,7,20,21,22,23",
+# }
+sub allocate_numa($topology, $vcpu_to_numa_map) {
+ my $final_map = {};
+ for my $vcpu (keys $vcpu_to_numa_map->%*) {
+ my $node = $vcpu_to_numa_map->{$vcpu};
+ my @cpus = sort keys $topology->{numa_nodes}->{$node}->{cpus}->%*;
+ $final_map->{$vcpu} = join(',', @cpus);
+ }
+ return $final_map;
+}
+
+1;
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [RFC qemu-server v2 3/4] memory: integrate pinning-aware NUMA memory binding
2026-09-21 9:53 [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 2/4] pinning: add NUMA allocator and reservation tracking Elias Huhsovitz
@ 2026-09-21 9:54 ` Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 4/4] pinning: integrate cpu pinning into vm lifecycle Elias Huhsovitz
3 siblings, 0 replies; 5+ messages in thread
From: Elias Huhsovitz @ 2026-09-21 9:54 UTC (permalink / raw)
To: pve-devel; +Cc: Elias Huhsovitz
Extend config to accept pinning decisions and emit the corresponding
QEMU NUMA arguments.
Currently memory binding is not automatically configured and must be set
via policy=bind. In the context of NUMA aware CPU Pinning, omitting
memory binding rarely makes sense.
When NUMA + pinning is active, the memory module now automatically binds
guest memory to the host NUMA nodes selected by the allocator using
policy=bind, even if the user did not explicitly configure 'hostnodes'
in the numaX settings.
Emit detailed '-numa cpu' arguments to pass the exact socket, core, and
thread topology to the guest OS.
Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
src/PVE/QemuServer/Memory.pm | 240 +++++++++++++++++++++++++----------
1 file changed, 171 insertions(+), 69 deletions(-)
diff --git a/src/PVE/QemuServer/Memory.pm b/src/PVE/QemuServer/Memory.pm
index 27738cd9..310b6b04 100644
--- a/src/PVE/QemuServer/Memory.pm
+++ b/src/PVE/QemuServer/Memory.pm
@@ -364,8 +364,165 @@ sub qemu_memdevices_list {
return $dimms;
}
+my sub pass_detailed_numa_topology {
+ my ($cmd, $conf, $sockets, $cores, $numa_totalmemory, $pinning_info) = @_;
+
+ my $threads_per_core = 1;
+ if ($conf->{vcpus} && $conf->{vcpus} > $sockets * $cores) {
+ $threads_per_core = int($conf->{vcpus} / ($sockets * $cores));
+ }
+
+ my $pinning_active = $pinning_info
+ && $pinning_info->{vcpu_to_numa_map}
+ && %{$pinning_info->{vcpu_to_numa_map}};
+
+ my $vcpu_id = 0;
+ for (my $s = 0; $s < $sockets; $s++) {
+ for (my $c = 0; $c < $cores; $c++) {
+ for (my $t = 0; $t < $threads_per_core; $t++) {
+ my $nodeid;
+
+ if ($numa_totalmemory) {
+ # Custom numaX topology: find which numaX contains this vcpu_id.
+ for (my $i = 0; $i < $MAX_NUMA; $i++) {
+ next if !$conf->{"numa$i"};
+ my $numa = parse_numa($conf->{"numa$i"});
+ next if !$numa || !$numa->{cpus};
+
+ for my $range (@{$numa->{cpus}}) {
+ my ($start, $end) = @$range;
+ $end //= $start;
+ if ($vcpu_id >= $start && $vcpu_id <= $end) {
+ $nodeid = $i;
+ last;
+ }
+ }
+ last if defined $nodeid;
+ }
+ } elsif ($pinning_active) {
+ # No numaX, but pinning decided the layout: use it.
+ # vcpu_to_numa_map values are host node IDs; with our 1:1
+ # guest_node_to_host_node scheme the guest nodeid equals
+ # the host node id.
+ my $host_node = $pinning_info->{vcpu_to_numa_map}->{$vcpu_id};
+ $nodeid = $host_node if defined $host_node;
+ } else {
+ # Default topology: 1 node per socket.
+ $nodeid = $s;
+ }
+
+ if (defined $nodeid) {
+ push @$cmd, '-numa', "cpu,node-id=$nodeid,socket-id=$s,core-id=$c,thread-id=$t";
+ }
+ $vcpu_id++;
+ }
+ }
+ }
+}
+
+# Emit one -object memory-backend + one -numa node for guest node $i
+# Returns: node's memory size, or undef if numa$i is not set
+my sub emit_explicit_numa_node {
+ my ($cmd, $conf, $node, $virtiofs_enabled, $pinning_active, $pinning_info) = @_;
+ my $numa = parse_numa($conf->{"numa$node"});
+ return if !$numa;
+
+ die "missing NUMA node$node memory value\n" if !$numa->{memory};
+ my $numa_memory = $numa->{memory};
+
+ my $memdev = $virtiofs_enabled ? "virtiofs-mem$node" : "ram-node$node";
+ my $mem_object = print_mem_object($conf, $memdev, $numa_memory);
+
+ my $cpulists = $numa->{cpus};
+ die "missing NUMA node$node cpus\n" if !defined($cpulists);
+ my $cpus = join(';',
+ map {
+ my ($start, $end) = @$_;
+ defined($end) ? "$start-$end" : $start
+ } @$cpulists,
+ );
+
+ my $hostnodelists = $numa->{hostnodes};
+ if (defined($hostnodelists)) {
+ my $hostnodes = print_numa_hostnodes($hostnodelists);
+ my $policy = $numa->{policy};
+ die "you need to define a policy for hostnode $hostnodes\n" if !$policy;
+ $mem_object .= ",host-nodes=$hostnodes,policy=$policy";
+ } elsif ($pinning_active
+ && defined($pinning_info->{guest_node_to_host_node}->{$node})) {
+ # No explicit hostnodes, but pinning is set: bind memory to the
+ # host node the pinning decision placed guest node $i on.
+ my $host_node = $pinning_info->{guest_node_to_host_node}->{$node};
+ $mem_object .= ",host-nodes=$host_node,policy=bind";
+ } else {
+ die "numa hostnodes need to be defined to use hugepages" if $conf->{hugepages};
+ }
+
+ push @$cmd, '-object', $mem_object;
+ push @$cmd, '-numa', "node,nodeid=$node,cpus=$cpus,memdev=$memdev";
+
+ return $numa_memory;
+}
+
+# Emit one -object memory-backend + one -numa node per guest node chosen
+# by the pinning allocator. Memory is split evenly (remainder to the first
+# node) and bound with policy=bind.
+my sub emit_pinning_aware_numa {
+ my ($cmd, $conf, $pinning_info, $virtiofs_enabled, $static_memory) = @_;
+ my @guest_nodes = sort { $a <=> $b }
+ keys %{$pinning_info->{guest_node_to_host_node}};
+ my $num_nodes = scalar @guest_nodes;
+ die "pinning selected no host NUMA nodes\n" if $num_nodes == 0;
+
+ my $numa_memory = int($static_memory / $num_nodes);
+ my $remainder = $static_memory - ($numa_memory * $num_nodes);
+
+ my $first = 1;
+ for my $guest_node (@guest_nodes) {
+ my $host_node = $pinning_info->{guest_node_to_host_node}->{$guest_node};
+ my $mem_for_node = $numa_memory + ($first ? $remainder : 0);
+ $first = 0;
+
+ my $memdev = $virtiofs_enabled
+ ? "virtiofs-mem$guest_node"
+ : "ram-node$guest_node";
+ my $mem_object = print_mem_object($conf, $memdev, $mem_for_node);
+ $mem_object .= ",host-nodes=$host_node,policy=bind";
+
+ push @$cmd, '-object', $mem_object;
+ push @$cmd, '-numa', "node,nodeid=$guest_node,memdev=$memdev";
+ }
+}
+
+# Emit one -object memory-backend & one -numa node per guest socket.
+# Used when numa: 1 but no numaX entries and no pinning info. No host
+# binding. Memory is split evenly; cpus= covers each socket's vCPU range.
+my sub emit_per_socket_numa {
+ my ($cmd, $conf, $sockets, $cores, $virtiofs_enabled, $static_memory) = @_;
+ my $numa_memory = ($static_memory / $sockets);
+
+ for (my $i = 0; $i < $sockets; $i++) {
+ die "host NUMA node$i doesn't exist\n"
+ if !host_numanode_exists($i) && $conf->{hugepages};
+
+ my $cpus = ($cores * $i);
+ $cpus .= "-" . ($cpus + $cores - 1) if $cores > 1;
+
+ my $memdev = $virtiofs_enabled ? "virtiofs-mem$i" : "ram-node$i";
+ my $mem_object = print_mem_object($conf, $memdev, $numa_memory);
+ push @$cmd, '-object', $mem_object;
+ push @$cmd, '-numa', "node,nodeid=$i,cpus=$cpus,memdev=$memdev";
+ }
+}
+
sub config {
- my ($conf, $vmid, $sockets, $cores, $hotplug, $virtiofs_enabled, $cmd, $machine_flags) = @_;
+ my ($conf, $vmid, $sockets, $cores, $hotplug, $virtiofs_enabled, $cmd, $machine_flags, $pinning_info) = @_;
+
+ $pinning_info //= {};
+ my $pinning_active = $pinning_info->{vcpu_to_numa_map}
+ && %{$pinning_info->{vcpu_to_numa_map}}
+ && $pinning_info->{guest_node_to_host_node}
+ && %{$pinning_info->{guest_node_to_host_node}};
my $memory = get_current_memory($conf->{memory});
my $static_memory = 0;
@@ -381,94 +538,46 @@ sub config {
}
my $sockets = $conf->{sockets} || 1;
-
$static_memory = $STATICMEM;
$static_memory = $static_memory * $sockets
if ($conf->{hugepages} && $conf->{hugepages} == 1024);
die "minimum memory must be ${static_memory}MB\n" if ($memory < $static_memory);
push @$cmd, '-m', "size=${static_memory},slots=255,maxmem=${MAX_MEM}M";
-
} else {
-
$static_memory = $memory;
push @$cmd, '-m', $static_memory;
}
die "numa needs to be enabled to use hugepages" if $conf->{hugepages} && !$conf->{numa};
-
die "Memory hotplug does not work in combination with virtio-fs.\n"
if $hotplug && $virtiofs_enabled;
if ($conf->{numa}) {
-
my $numa_totalmemory = undef;
+
for (my $i = 0; $i < $MAX_NUMA; $i++) {
next if !$conf->{"numa$i"};
- my $numa = parse_numa($conf->{"numa$i"});
- next if !$numa;
- # memory
- die "missing NUMA node$i memory value\n" if !$numa->{memory};
- my $numa_memory = $numa->{memory};
- $numa_totalmemory += $numa_memory;
-
- my $memdev = $virtiofs_enabled ? "virtiofs-mem$i" : "ram-node$i";
- my $mem_object = print_mem_object($conf, $memdev, $numa_memory);
-
- # cpus
- my $cpulists = $numa->{cpus};
- die "missing NUMA node$i cpus\n" if !defined($cpulists);
- my $cpus = join(
- ',cpus=',
- map {
- my ($start, $end) = @$_;
- defined($end) ? "$start-$end" : $start
- } @$cpulists,
- );
-
- # hostnodes
- my $hostnodelists = $numa->{hostnodes};
- if (defined($hostnodelists)) {
-
- my $hostnodes = print_numa_hostnodes($hostnodelists);
-
- # policy - note that the value 'default' is not exposed, because it can't be used in
- # combination with host-nodes:
- # > kvm: host-nodes must be empty for policy default, or you should explicitly
- # > specify a policy other than default
- my $policy = $numa->{policy};
- die "you need to define a policy for hostnode $hostnodes\n" if !$policy;
- $mem_object .= ",host-nodes=$hostnodes,policy=$policy";
- } else {
- die "numa hostnodes need to be defined to use hugepages" if $conf->{hugepages};
- }
-
- push @$cmd, '-object', $mem_object;
- push @$cmd, '-numa', "node,nodeid=$i,cpus=$cpus,memdev=$memdev";
+ $numa_totalmemory += emit_explicit_numa_node(
+ $cmd, $conf, $i, $virtiofs_enabled, $pinning_active, $pinning_info);
}
die "total memory for NUMA nodes must be equal to vm static memory\n"
if $numa_totalmemory && $numa_totalmemory != $static_memory;
- #if no custom tology, we split memory and cores across numa nodes
if (!$numa_totalmemory) {
- my $numa_memory = ($static_memory / $sockets);
-
- for (my $i = 0; $i < $sockets; $i++) {
- die "host NUMA node$i doesn't exist\n"
- if !host_numanode_exists($i) && $conf->{hugepages};
-
- my $cpus = ($cores * $i);
- $cpus .= "-" . ($cpus + $cores - 1) if $cores > 1;
-
- my $memdev = $virtiofs_enabled ? "virtiofs-mem$i" : "ram-node$i";
- my $mem_object = print_mem_object($conf, $memdev, $numa_memory);
- push @$cmd, '-object', $mem_object;
- push @$cmd, '-numa', "node,nodeid=$i,cpus=$cpus,memdev=$memdev";
+ if ($pinning_active) {
+ emit_pinning_aware_numa(
+ $cmd, $conf, $pinning_info, $virtiofs_enabled, $static_memory);
+ } else {
+ emit_per_socket_numa(
+ $cmd, $conf, $sockets, $cores, $virtiofs_enabled, $static_memory);
}
}
+
+ pass_detailed_numa_topology($cmd, $conf, $sockets, $cores, $numa_totalmemory, $pinning_info);
+
} elsif ($virtiofs_enabled) {
- # kvm: '-machine memory-backend' and '-numa memdev' properties are mutually exclusive
push @$cmd, '-object',
'memory-backend-memfd,id=virtiofs-mem' . ",size=$conf->{memory}M,share=on";
push @$machine_flags, 'memory-backend=virtiofs-mem';
@@ -476,18 +585,12 @@ sub config {
if ($hotplug) {
foreach_dimm(
- $conf,
- $vmid,
- $memory,
- $static_memory,
+ $conf, $vmid, $memory, $static_memory,
sub {
my ($conf, $vmid, $name, $dimm_size, $numanode, $current_size, $memory) = @_;
-
my $mem_object = print_mem_object($conf, "mem-$name", $dimm_size);
-
push @$cmd, "-object", $mem_object;
push @$cmd, "-device", "pc-dimm,id=$name,memdev=mem-$name,node=$numanode";
-
die "memory size ($memory) must be aligned to $dimm_size for hotplugging\n"
if $current_size > $memory;
},
@@ -816,4 +919,3 @@ sub hugepages_update_locked {
return $res;
}
1;
-
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [RFC qemu-server v2 4/4] pinning: integrate cpu pinning into vm lifecycle
2026-09-21 9:53 [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Elias Huhsovitz
` (2 preceding siblings ...)
2026-09-21 9:54 ` [RFC qemu-server v2 3/4] memory: integrate pinning-aware NUMA memory binding Elias Huhsovitz
@ 2026-09-21 9:54 ` Elias Huhsovitz
3 siblings, 0 replies; 5+ messages in thread
From: Elias Huhsovitz @ 2026-09-21 9:54 UTC (permalink / raw)
To: pve-devel; +Cc: Elias Huhsovitz
Add new Pinning.pm to handle the core logic of pinning via taskset.
Integrate the new pinning system into QemuServer.pm so that CPU
placement actually takes effect when VMs start, stop, or change.
config_to_command now computes the effective topology and startup CPU
set before building the QEMU command line. The taskset wrapper uses this
computed set instead of the raw affinity value. This confines QEMU qemu
to the correct CPUs immediately.
vm_start_nolock now receives a pinning map from config_to_command and
applies per-vCPU thread affinity via QMP once the guest is running.
qemu_cpu_hotplug recalculates the pinning map after adding or
removing vCPUs and reapplies thread affinity.
vm_stop_cleanup clears the VM's entry from the NUMA reservation tracker
so that freed resources are immediately available for other VMs instead
of waiting for the next start cycle to prune stale entries.
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
src/PVE/QemuServer.pm | 97 +++++--
src/PVE/QemuServer/Makefile | 2 +
src/PVE/QemuServer/Pinning.pm | 396 ++++++++++++++++++++++++++++
src/PVE/QemuServer/Pinning/Makefile | 9 +
4 files changed, 481 insertions(+), 23 deletions(-)
create mode 100644 src/PVE/QemuServer/Pinning.pm
create mode 100644 src/PVE/QemuServer/Pinning/Makefile
diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 759f7db2..ba007915 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -85,6 +85,7 @@ use PVE::QemuServer::Memory qw(get_current_memory);
use PVE::QemuServer::MetaInfo;
use PVE::QemuServer::Monitor qw(mon_cmd qmp_cmd vm_qmp_peer);
use PVE::QemuServer::Network;
+use PVE::QemuServer::Pinning;
use PVE::QemuServer::OVMF;
use PVE::QemuServer::PCI qw(print_pci_addr print_pcie_addr parse_hostpci get_pci_bridges);
use PVE::QemuServer::PCI::Mdev;
@@ -739,6 +740,12 @@ EODESCR
optional => 1,
default => 1,
},
+ pinning => {
+ type => 'string',
+ format => $PVE::QemuServer::Pinning::pinning_fmt,
+ description => "Set pinning options for the guest",
+ optional => 1,
+ },
};
my $cicustom_fmt = {
@@ -3202,8 +3209,27 @@ sub config_to_command {
($use_old_bios_files, $machine_type) = qemu_use_old_bios_files($machine_type);
my $cmd = [];
- if ($conf->{affinity}) {
- push @$cmd, '/usr/bin/taskset', '--cpu-list', '--all-tasks', $conf->{affinity};
+
+ my ($sockets, $cores, $maxcpus, $vcpus, $startup_cpuset, $pinning_map);
+
+ my $topo;
+ if ($dry_run) {
+ ($sockets, $cores, $maxcpus, $vcpus) =
+ PVE::QemuServer::Pinning::Config::get_config_topology($conf);
+ $startup_cpuset = $conf->{affinity};
+ $pinning_map = {};
+ } else {
+ $topo = PVE::QemuServer::Pinning::get_effective_topology($conf, $vmid);
+ ($sockets, $cores, $maxcpus, $vcpus) = @$topo{qw(sockets cores maxcpus vcpus)};
+ $startup_cpuset = $topo->{startup_cpuset};
+ $pinning_map = $topo->{pinning_map};
+ }
+
+ my $allowed_vcpus = $cpuinfo->{cpus};
+ die "MAX $allowed_vcpus vcpus allowed per VM on this node\n" if ($allowed_vcpus < $maxcpus);
+
+ if (defined($startup_cpuset) && $startup_cpuset ne '') {
+ push @$cmd, '/usr/bin/taskset', '--cpu-list', '--all-tasks', $startup_cpuset;
}
push @$cmd, $kvm_binary;
@@ -3375,20 +3401,6 @@ sub config_to_command {
# to support backing up template VMs even if the TPM disk is write-protected.
add_tpm_device($vmid, $devices, $conf) if !$is_template;
- my $sockets = 1;
- $sockets = $conf->{smp} if $conf->{smp}; # old style - no longer iused
- $sockets = $conf->{sockets} if $conf->{sockets};
-
- my $cores = $conf->{cores} || 1;
-
- my $maxcpus = $sockets * $cores;
-
- my $vcpus = $conf->{vcpus} ? $conf->{vcpus} : $maxcpus;
-
- my $allowed_vcpus = $cpuinfo->{cpus};
-
- die "MAX $allowed_vcpus vcpus allowed per VM on this node\n" if ($allowed_vcpus < $maxcpus);
-
if ($hotplug_features->{cpu} && min_version($machine_version, 2, 7)) {
push @$cmd, '-smp', "1,sockets=$sockets,cores=$cores,maxcpus=$maxcpus";
for (my $i = 2; $i <= $vcpus; $i++) {
@@ -3448,6 +3460,18 @@ sub config_to_command {
my $virtiofs_enabled = PVE::QemuServer::Virtiofs::virtiofs_enabled($conf);
+ my $pinning_info = {};
+ if (!$dry_run) {
+ my $vmap = $topo->{vcpu_to_numa_map} // {};
+ my $nmap = $topo->{guest_node_to_host_node} // {};
+ if (%$pinning_map || %$vmap) {
+ $pinning_info = {
+ vcpu_to_numa_map => $vmap,
+ guest_node_to_host_node => $nmap,
+ };
+ }
+ }
+
PVE::QemuServer::Memory::config(
$conf,
$vmid,
@@ -3457,6 +3481,7 @@ sub config_to_command {
$virtiofs_enabled,
$cmd,
$machineFlags,
+ $pinning_info
);
push @$cmd, '-S' if $conf->{freeze};
@@ -3809,7 +3834,7 @@ sub config_to_command {
push @$cmd, @$aa;
}
- return wantarray ? ($cmd, $spice_port, $pci_devices, $conf) : $cmd;
+ return wantarray ? ($cmd, $spice_port, $pci_devices, $conf, $pinning_map) : $cmd;
}
sub spice_port {
@@ -4261,11 +4286,8 @@ sub qemu_cpu_hotplug {
my $machine_type = PVE::QemuServer::Machine::get_current_qemu_machine($vmid);
- my $sockets = 1;
- $sockets = $conf->{smp} if $conf->{smp}; # old style - no longer iused
- $sockets = $conf->{sockets} if $conf->{sockets};
- my $cores = $conf->{cores} || 1;
- my $maxcpus = $sockets * $cores;
+ my $topo = PVE::QemuServer::Pinning::get_effective_topology($conf, $vmid);
+ my $maxcpus = $topo->{maxcpus};
$vcpus = $maxcpus if !$vcpus;
@@ -4330,6 +4352,18 @@ sub qemu_cpu_hotplug {
mon_cmd($vmid, "cpu-add", id => int($i));
}
}
+
+ # Recalculate the pinning map AFTER the hotplug operations have changed the vCPU count.
+ my $new_vcpus = $conf->{vcpus} || $maxcpus;
+ eval {
+ my ($new_pinning_map, undef, undef, undef) = PVE::QemuServer::Pinning::get_pinning_map($conf, $vmid, $new_vcpus);
+ PVE::QemuServer::Pinning::pin_vcpu_threads_to_host_cpus($vmid, $new_pinning_map);
+ 1;
+ } or do {
+ my $err = $@ || 'unknown error';
+ log_warn("could not re-pin vCPU threads after CPU hotplug - $err");
+ };
+
}
sub qemu_volume_snapshot {
@@ -5619,6 +5653,7 @@ sub vm_start_nolock {
my ($cmd, $spice_port, $start_timeout);
my $pci_reserve_list = [];
+ my $pinning_map = {};
eval {
# With -blockdev, it is necessary to activate the volumes before generating the command line
# Plugins can safely deactivate already-active volumes here if needed
@@ -5629,7 +5664,7 @@ sub vm_start_nolock {
# Note that for certain cases like templates, the configuration is minimized, so need to ensure
# the rest of the function here uses the same configuration that was used to build the command
- ($cmd, $spice_port, my $pci_devices, $conf) = config_to_command(
+ ($cmd, $spice_port, my $pci_devices, $conf, $pinning_map) = config_to_command(
$storecfg,
$vmid,
$conf,
@@ -5836,6 +5871,14 @@ sub vm_start_nolock {
my $qemu_pkg_string = PVE::QemuServer::QMPHelpers::get_qemu_package_string($vmid);
syslog("info", "VM $vmid started with PID $pid using $qemu_pkg_string.");
+ eval {
+ PVE::QemuServer::Pinning::pin_vcpu_threads_to_host_cpus($vmid, $pinning_map);
+ 1;
+ } or do {
+ my $err = $@ || 'unknown error';
+ log_warn("could not pin vCPU threads - $err");
+ };
+
PVE::QemuServer::RunState::create_cleanup_flag($vmid);
if (defined(my $migrate = $res->{migrate})) {
@@ -6186,6 +6229,14 @@ sub vm_stop_cleanup {
warn $err;
}
+ eval {
+ PVE::QemuServer::Pinning::Allocator::clear_reservation($vmid);
+ 1;
+ } or do {
+ my $err = $@ || 'Unknown failure';
+ warn "Failed to clear NUMA reservation for VM $vmid: $err";
+ };
+
# under the new mechanism the hookscript is fired here, so all callers of
# vm_stop_cleanup observe it; with the old one 'qm cleanup' still drives it.
# callers that should not trigger the post-stop hook (e.g. the live-migration
diff --git a/src/PVE/QemuServer/Makefile b/src/PVE/QemuServer/Makefile
index 060fac23..f5981868 100644
--- a/src/PVE/QemuServer/Makefile
+++ b/src/PVE/QemuServer/Makefile
@@ -22,6 +22,7 @@ SOURCES=Agent.pm \
Network.pm \
OVMF.pm \
PCI.pm \
+ Pinning.pm \
QemuImage.pm \
QMPHelpers.pm \
QSD.pm \
@@ -36,4 +37,5 @@ SOURCES=Agent.pm \
install: $(SOURCES)
for i in $(SOURCES); do install -D -m 0644 $$i $(DESTDIR)$(PERLDIR)/PVE/QemuServer/$$i; done
$(MAKE) -C Cfg2Cmd install
+ $(MAKE) -C Pinning install
$(MAKE) -C PCI install
diff --git a/src/PVE/QemuServer/Pinning.pm b/src/PVE/QemuServer/Pinning.pm
new file mode 100644
index 00000000..36cb743d
--- /dev/null
+++ b/src/PVE/QemuServer/Pinning.pm
@@ -0,0 +1,396 @@
+package PVE::QemuServer::Pinning;
+
+use v5.36;
+
+use PVE::CpuSet;
+use PVE::QemuServer::Monitor;
+use PVE::QemuServer::Pinning::Config;
+use PVE::QemuServer::Pinning::Topology;
+use PVE::QemuServer::Pinning::Allocator;
+use PVE::Tools qw(run_command);
+
+
+# Returns: sorted cpu ids
+# Warn if the assigned host CPUs span multiple CPU models.
+# in: $pinning_map = {0=>"0", 1=>"1", 2=>"4"}
+# out: [0, 1, 4]
+my sub collect_assigned_cpu_ids($pinning_map) {
+ my %unique_cpus;
+ for my $vcpu_id (keys $pinning_map->%*) {
+ my (undef, $members) = PVE::CpuSet::parse_cpuset($pinning_map->{$vcpu_id});
+ $unique_cpus{$_} = 1 for keys %$members;
+ }
+ my @cpu_ids = sort { $a <=> $b } keys %unique_cpus;
+
+ if (PVE::QemuServer::Pinning::Topology::are_cpus_heterogenous(\@cpu_ids)) {
+ warn "WARNING: The assigned CPU pinning spans multiple CPU models."
+ . " This may cause KVM initialization failures or performance degradation.\n";
+ }
+
+ return \@cpu_ids;
+}
+
+# On heterogeneous systems (ARM big.LITTLE, Intel P+E), KVM vCPU init can
+# fail if the QEMU process schedules across CPU types at startup.
+# Restrict a base cpuset to the largest same-type subset.
+# Returns undef on homogeneous systems or empty input.
+my sub restrict_to_largest_cpu_type($cpuset_str) {
+ my ($groups, $keys) = PVE::QemuServer::Pinning::Topology::group_cpus_by_type();
+ return undef if !@$keys || @$keys <= 1;
+ return undef if !defined($cpuset_str) || $cpuset_str eq '';
+
+ my (undef, $members) = PVE::CpuSet::parse_cpuset($cpuset_str);
+ return undef if !keys %$members;
+
+ my ($best_key, $best_count) = (undef, -1);
+ for my $key (@$keys) {
+ my @matches = grep { $members->{$_} } @{$groups->{$key}{cpus}};
+ if (scalar @matches > $best_count) {
+ $best_count = scalar @matches;
+ $best_key = $key;
+ }
+ }
+ return undef if !defined $best_key || $best_count == 0;
+
+ my @sorted = sort { $a <=> $b }
+ grep { $members->{$_} } @{$groups->{$best_key}{cpus}};
+ return @sorted ? join(',', @sorted) : undef;
+}
+
+# Used when there's no existing cpuset to restrict (pinning: none, no affinity).
+my sub pick_highest_capacity_cpu_type() {
+ my ($groups, $keys) = PVE::QemuServer::Pinning::Topology::group_cpus_by_type();
+ return undef if !@$keys || @$keys <= 1;
+
+ my ($best_key, $best_capacity) = (undef, -1);
+ for my $key (@$keys) {
+ my $cap = $groups->{$key}{capacity};
+ if ($cap > $best_capacity) {
+ $best_capacity = $cap;
+ $best_key = $key;
+ }
+ }
+ return undef if !defined $best_key;
+
+ my @sorted = sort { $a <=> $b } @{$groups->{$best_key}{cpus}};
+ return @sorted ? join(',', @sorted) : undef;
+}
+
+my sub get_filtered_topology($conf) {
+ my $topology = PVE::QemuServer::Pinning::Topology::get_host_topology();
+ return PVE::QemuServer::Pinning::Topology::filter_by_affinity(
+ $topology, $conf->{affinity});
+}
+
+my sub allocate_for_mode($mode, $topology, $vcpu_to_numa_map, $vcpu_count, $has_numa) {
+ return {} if $mode eq 'balanced';
+ return PVE::QemuServer::Pinning::Allocator::allocate_numa(
+ $topology, $vcpu_to_numa_map)
+ if $mode eq 'numa';
+ if ($mode eq 'one-to-one') {
+ return PVE::QemuServer::Pinning::Allocator::allocate_one_to_one_flat(
+ $topology, $vcpu_count)
+ if !$has_numa;
+ return PVE::QemuServer::Pinning::Allocator::allocate_one_to_one(
+ $topology, $vcpu_to_numa_map, $vcpu_count);
+ }
+ return undef;
+}
+
+my sub validate_topology_for_mode($mode, $topology, $assigned_cpu_ids, $conf) {
+ return { valid => 1 } if $mode ne 'one-to-one';
+
+ my $guest_sockets = $conf->{sockets} // 1;
+ my $guest_cores = $conf->{cores} // 1;
+
+ my $result = PVE::QemuServer::Pinning::Topology::validate_pinned_topology(
+ $topology, $assigned_cpu_ids, $guest_sockets, $guest_cores);
+
+ warn "WARNING: $result->{warning}\n" if !$result->{valid};
+ return $result;
+}
+
+=head2 get_pinning_map
+
+Parses the VM's C<pinning> property, reads host topology,
+assigns host NUMA nodes to guest nodes if needed, computes
+the per-vCPU CPU pinning map for the configured mode.
+
+Returns four values:
+
+=over
+
+=item C<$pinning_map> - C<< { $vcpu_id => $cpus } >>. For C<one-to-one>
+and C<numa>, C<$cpus> is a comma-joined CPU set. For C<balanced>, C<{}>:
+memory binding is done by the caller using C<$vcpu_to_numa_map>.
+
+=item C<$topo_result> - validation result from C<validate_pinned_topology()>.
+
+=item C<$vcpu_to_numa_map> - C<< { $vcpu_id => $host_node_id } >>.
+
+=item C<$topology> - affinity-filtered host topology. Returned so callers
+do not have to re-read sysfs.
+
+=back
+
+Returns C<({}, {valid=>1}, {}, undef)> immediately when pinning mode is
+C<none>. Dies if C<balanced> or C<numa> is used without C<numa: 1>.
+
+=cut
+sub get_pinning_map($conf, $vmid, $vcpu_count) {
+ my $pinning = PVE::QemuServer::Pinning::Config::parse_pinning($conf->{pinning});
+ my $mode = $pinning->{mode};
+
+ return ({}, { valid => 1 }, {}, undef)
+ if $mode eq PVE::QemuServer::Pinning::Config::DEFAULT_VCPU_PINNING;
+
+ my $topology = get_filtered_topology($conf);
+ my $has_numa = $conf->{numa};
+
+ # balanced and numa modes require numa: 1 for memory binding.
+ if (($mode eq 'balanced' || $mode eq 'numa') && !$has_numa) {
+ die "pinning mode '$mode' requires numa: 1 to be enabled\n";
+ }
+
+ my $vcpu_to_numa_map;
+
+ if ($mode eq 'balanced') {
+ $vcpu_to_numa_map = PVE::QemuServer::Pinning::Allocator::balance_numa_nodes(
+ $conf, $topology, $vmid);
+ } elsif ($mode eq 'one-to-one' && !$has_numa) {
+ $vcpu_to_numa_map = {};
+ } else {
+ $vcpu_to_numa_map = PVE::QemuServer::Pinning::Allocator::assign_numa_nodes(
+ $conf, $topology, $vmid);
+ }
+
+ my $pinning_map = allocate_for_mode(
+ $mode, $topology, $vcpu_to_numa_map, $vcpu_count, $has_numa);
+ return ({}, { valid => 1 }, {}, $topology) if !defined $pinning_map;
+
+ my $assigned_cpu_ids = collect_assigned_cpu_ids($pinning_map);
+ my $topo_result = validate_topology_for_mode(
+ $mode, $topology, $assigned_cpu_ids, $conf);
+
+ return ($pinning_map, $topo_result, $vcpu_to_numa_map, $topology);
+}
+
+# QEMU requires -numa node,nodeid=N to start from 0 and be sequential.
+# If host nodes are [1, 3], guest IDs become {0 => 1, 1 => 3}.
+my sub build_sequential_guest_node_to_host_node($vcpu_to_numa_map) {
+ return {} if !defined($vcpu_to_numa_map) || !%$vcpu_to_numa_map;
+
+ my %seen;
+ $seen{$_} = 1 for values %$vcpu_to_numa_map;
+ my @host_nodes = sort { $a <=> $b } keys %seen;
+
+ my $map = {};
+ my $guest_id = 0;
+ for my $host_node (@host_nodes) {
+ $map->{$guest_id} = $host_node;
+ $guest_id++;
+ }
+ return $map;
+}
+
+# Rewrite vcpu→host_node values to vcpu→guest_node values.
+#
+# Why this is needed: the allocator works in host node id space, because
+# that is what the kernel and the reservations file use. QEMU, however,
+# requires -numa node,nodeid=N to start at 0 and be contiguous. On a host
+# where the allocator picked nodes {2, 3}, QEMU must still see nodeid=0
+# and nodeid=1. This function translates between the two id spaces.
+#
+# On the common case where the allocator picked {0, 1, 2, ...},
+# the output equals the input.
+#
+# in: {0=>1, 1=>1, 2=>3}, {0=>1, 1=>3}
+# out: {0=>0, 1=>0, 2=>1}
+my sub remap_to_guest_ids($vcpu_to_numa_map, $guest_node_to_host_node) {
+ return {} if !%$vcpu_to_numa_map || !%$guest_node_to_host_node;
+
+ my %host_to_guest;
+ for my $guest_id (keys %$guest_node_to_host_node) {
+ my $host_id = $guest_node_to_host_node->{$guest_id};
+ $host_to_guest{$host_id} = $guest_id;
+ }
+
+ my %remapped;
+ for my $vcpu (keys %$vcpu_to_numa_map) {
+ my $host_node = $vcpu_to_numa_map->{$vcpu};
+ $remapped{$vcpu} = $host_to_guest{$host_node} // $host_node;
+ }
+ return \%remapped;
+}
+
+# Union of all host CPUs referenced by the pinning map.
+# in: {0=>"0", 1=>"1", 2=>"4,5"}
+# out: "0,1,4,5"
+my sub cpuset_from_pinning_map($pinning_map) {
+ return undef if !%$pinning_map;
+
+ my %unique_cpus;
+ for my $vcpu_id (keys %$pinning_map) {
+ my (undef, $members) = PVE::CpuSet::parse_cpuset($pinning_map->{$vcpu_id});
+ $unique_cpus{$_} = 1 for keys %$members;
+ }
+
+ return undef if !%unique_cpus;
+ return join(',', sort { $a <=> $b } keys %unique_cpus);
+}
+
+# Union of CPUs on the host nodes selected in balanced mode.
+# in: {0=>1, 1=>3}, $topology
+# out: "4,5,6,7,20,21,22,23,12,13,14,15,28,29,30,31"
+my sub cpuset_for_balanced_mode($guest_node_to_host_node, $topology) {
+ my %host_nodes = map { $_ => 1 } values %$guest_node_to_host_node;
+ my @cpus;
+ for my $host_node (sort { $a <=> $b } keys %host_nodes) {
+ next if !defined($topology->{numa_nodes}->{$host_node});
+ push @cpus, sort { $a <=> $b }
+ keys $topology->{numa_nodes}->{$host_node}->{cpus}->%*;
+ }
+ return @cpus ? join(',', @cpus) : undef;
+}
+
+=head2 compute_startup_cpuset
+
+ my $cpuset = compute_startup_cpuset(
+ $pinning_map, $guest_node_to_host_node, $affinity, $topology);
+
+Determines the cpuset that the QEMU main thread is confined to at startup.
+The scheduler only ever runs QEMU on these CPUs, so this must cover every
+CPU that any vCPU might be pinned to.
+
+Priority:
+
+=over
+
+=item 1. Union of CPUs in C<$pinning_map> (numa, one-to-one).
+
+=item 2. Union of CPUs on selected host nodes (balanced).
+
+=item 3. VM's C<affinity> setting.
+
+=item 4. All online CPUs.
+
+=back
+
+The chosen set is then restricted to a single CPU type if the host is
+heterogeneous (ARM big.LITTLE, Intel P+E). KVM vCPU init can fail if the
+QEMU process is allowed to migrate across CPU types during startup, so on
+such hosts we narrow the cpuset to the largest same-type subset — or, if
+no cpuset has been established yet, to the highest-capacity type.
+
+Returns a comma-joined cpuset string, or the original C<$affinity> if no
+restriction is possible.
+
+=cut
+my sub compute_startup_cpuset($pinning_map, $guest_node_to_host_node, $affinity, $topology) {
+ my $base_cpuset = $affinity;
+
+ if (%$pinning_map) {
+ my $pinning_cpuset = cpuset_from_pinning_map($pinning_map);
+ $base_cpuset = $pinning_cpuset if defined($pinning_cpuset);
+ } elsif (%$guest_node_to_host_node) {
+ my $balanced = cpuset_for_balanced_mode($guest_node_to_host_node, $topology);
+ $base_cpuset = $balanced if defined($balanced);
+ }
+
+ if (defined($base_cpuset) && $base_cpuset ne '') {
+ my $restricted = restrict_to_largest_cpu_type($base_cpuset);
+ $base_cpuset = $restricted if defined($restricted) && $restricted ne '';
+ } else {
+ my $best_type = pick_highest_capacity_cpu_type();
+ $base_cpuset = $best_type if defined($best_type);
+ }
+
+ return $base_cpuset;
+}
+
+my sub taskset_thread($cpus, $thread_id) {
+ run_command(
+ ['/usr/bin/taskset', '-c', '-p', $cpus, $thread_id],
+ errmsg => "failed to pin thread $thread_id to cpus $cpus",
+ quiet => 1,
+ );
+}
+
+sub pin_vcpu_threads_to_host_cpus($vmid, $pinning_map) {
+ return if !%$pinning_map;
+
+ my $cpuinfo = PVE::QemuServer::Monitor::mon_cmd($vmid, 'query-cpus-fast');
+
+ for my $vcpu ($cpuinfo->@*) {
+ my $vcpu_index = $vcpu->{'cpu-index'};
+ my $cpus = $pinning_map->{$vcpu_index};
+
+ die "no cpus selected for pinning vcpu $vcpu_index\n"
+ if !defined($cpus);
+
+ my $tid = $vcpu->{'thread-id'};
+ print "pinning vcpu $vcpu_index (thread $tid) to cpu(s) $cpus\n";
+ taskset_thread($cpus, $tid);
+ }
+}
+
+=head2 get_effective_topology
+
+Computes the effective CPU topology and pinning for a VM start.
+
+Returns a hashref with:
+
+=over
+
+=item C<sockets>, C<cores>, C<maxcpus>, C<vcpus> - the guest CPU topology
+to pass to QEMU.
+
+=item C<startup_cpuset> - cpuset string for the QEMU main thread.
+
+=item C<pinning_map> - C<< { $vcpu_id => $cpus } >> for C<numa> and
+C<one-to-one>; empty for C<balanced> and C<none>.
+
+=item C<vcpu_to_numa_map> - C<< { $vcpu_id => $guest_node_id } >> with
+sequential guest node ids.
+
+=item C<guest_node_to_host_node> - C<< { $guest_node_id => $host_node_id } >>.
+
+=back
+
+Dies via C<get_pinning_map()> on invalid pinning configurations.
+
+=cut
+sub get_effective_topology($conf, $vmid) {
+ my ($sockets, $cores, $maxcpus, $vcpus) =
+ PVE::QemuServer::Pinning::Config::get_config_topology($conf);
+
+ my ($pinning_map, $topo_result, $vcpu_to_numa_map, $topology) =
+ get_pinning_map($conf, $vmid, $vcpus);
+
+ my $guest_node_to_host_node = build_sequential_guest_node_to_host_node($vcpu_to_numa_map);
+ $vcpu_to_numa_map = remap_to_guest_ids($vcpu_to_numa_map, $guest_node_to_host_node);
+
+ my $startup_cpuset = compute_startup_cpuset(
+ $pinning_map, $guest_node_to_host_node, $conf->{affinity}, $topology);
+
+ # Fall back to flat topology if one-to-one pinning didn't match hardware.
+ if (%$pinning_map && !$topo_result->{valid}) {
+ $sockets = $topo_result->{sockets};
+ $cores = $topo_result->{cores};
+ $maxcpus = $sockets * $cores;
+ $vcpus = $maxcpus;
+ }
+
+ return {
+ sockets => $sockets,
+ cores => $cores,
+ maxcpus => $maxcpus,
+ vcpus => $vcpus,
+ startup_cpuset => $startup_cpuset,
+ pinning_map => $pinning_map,
+ vcpu_to_numa_map => $vcpu_to_numa_map,
+ guest_node_to_host_node => $guest_node_to_host_node,
+ };
+}
+
+1;
diff --git a/src/PVE/QemuServer/Pinning/Makefile b/src/PVE/QemuServer/Pinning/Makefile
new file mode 100644
index 00000000..aa2c9603
--- /dev/null
+++ b/src/PVE/QemuServer/Pinning/Makefile
@@ -0,0 +1,9 @@
+DESTDIR=
+PREFIX=/usr
+PERLDIR=$(PREFIX)/share/perl5
+
+SOURCES=Allocator.pm Config.pm Topology.pm
+
+.PHONY: install
+install: $(SOURCES)
+ for i in $(SOURCES); do install -D -m 0644 $$i $(DESTDIR)$(PERLDIR)/PVE/QemuServer/Pinning/$$i; done
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-09-21 9:55 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-21 9:53 [RFC qemu-server v2 0/4] fix #7282: allow (NUMA aware) vCPU pinning Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 1/4] pinning: add topology discovery and config parsing Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 2/4] pinning: add NUMA allocator and reservation tracking Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 3/4] memory: integrate pinning-aware NUMA memory binding Elias Huhsovitz
2026-09-21 9:54 ` [RFC qemu-server v2 4/4] pinning: integrate cpu pinning into vm lifecycle Elias Huhsovitz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.