From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 120031FF0AF for ; Thu, 22 Oct 2026 11:22:45 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DF4452163C; Thu, 22 Oct 2026 11:22:38 +0200 (CEST) From: Elias Huhsovitz To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v3 3/3] cpu: add CPU binding Date: Thu, 22 Oct 2026 11:22:25 +0200 Message-ID: <20261022092227.58871-4-e.huhsovitz@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261022092227.58871-1-e.huhsovitz@proxmox.com> References: <20261022092227.58871-1-e.huhsovitz@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792660953516 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.516 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: XSVYWC7F7XSUCQQ5KRUVO4ZIOBALGNLS X-Message-ID-Hash: XSVYWC7F7XSUCQQ5KRUVO4ZIOBALGNLS X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Elias Huhsovitz X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add two new VM config options: * cpu-binding-mode none | manual | one-to-one | numa | numa-manual * cpu-binding-cfg mode-dependent syntax When cpu-binding-mode is unset or "none", behavior is unchanged. Otherwise, vCPU threads are pinned to host CPUs after QEMU has spawned them, driven by the query-cpus-fast QMP response. Five modes are supported: * manual - explicit vCPU:hostCPU mapping, e.g. "0:0;1:1;2:2;3:3" * one-to-one - each vCPU gets its own physical core from a user pool, SMT stripped, e.g. "0-3" * numa - per-vNUMA-node set-to-set binding, sequential, e.g. "3;4" * numa-manual - explicit vNUMA->host NUMA mapping, e.g. "0:2;1:3" Two entry points are provided in PVE::QemuServer::CPUBinding: * validate_binding() - called at VM start, before QEMU spawns. Rejects invalid configs against the current topology. * bind_vcpus() - called after QEMU is up. Encodes the QMP cpu information as JSON and hands it to Rust together with the mode and cfg string. Signed-off-by: Elias Huhsovitz --- src/PVE/QemuServer.pm | 36 +++++++++ src/PVE/QemuServer/CPUBinding.pm | 133 +++++++++++++++++++++++++++++++ src/PVE/QemuServer/Makefile | 1 + 3 files changed, 170 insertions(+) create mode 100644 src/PVE/QemuServer/CPUBinding.pm diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index 1efeeb66..3ba266be 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -60,6 +60,7 @@ use PVE::QemuServer::Helpers qw(config_aware_timeout get_iscsi_initiator_name get_host_arch min_version kvm_user_version windows_version); use PVE::QemuServer::Cloudinit; use PVE::QemuServer::CGroup; +use PVE::QemuServer::CPUBinding; use PVE::QemuServer::CPUConfig qw( print_cpu_device get_cpu_options @@ -739,6 +740,18 @@ EODESCR optional => 1, default => 1, }, + 'cpu-binding-mode' => { + type => 'string', + format => $PVE::QemuServer::Bind::bind_mode_fmt, + description => "Set CPU binding mode for the guest", + optional => 1, + }, + 'cpu-binding-cfg' => { + type => 'string', + format => $PVE::QemuServer::Bind::bind_cfg_fmt, + description => "Configuration for CPU binding for the guest", + optional => 1, + }, }; my $cicustom_fmt = { @@ -3218,6 +3231,15 @@ sub config_to_command { push @$cmd, '/usr/bin/taskset', '--cpu-list', '--all-tasks', $conf->{affinity}; } + if ($conf->{'cpu-binding-mode'}) { + PVE::QemuServer::CPUBinding::validate_binding( + $conf->{'cpu-binding-mode'}, + $conf->{'sockets'}, + $conf->{'cores'}, + $conf->{'cpu-binding-cfg'}, + ); + } + push @$cmd, $kvm_binary; push @$cmd, '-id', $vmid; @@ -5715,6 +5737,20 @@ sub vm_start_nolock { my $qemu_pkg_string = PVE::QemuServer::QMPHelpers::get_qemu_package_string($vmid); syslog("info", "VM $vmid started with PID $pid using $qemu_pkg_string."); + if ($conf->{'cpu-binding-mode'}) { + eval { + PVE::QemuServer::CPUBinding::bind_vcpus( + $vmid, + $conf->{'cpu-binding-mode'}, + $conf->{'cpu-binding-cfg'}, + ); + 1; + } or do { + my $error = $@ || 'Unknown failure'; + log_warn("unable to bind vCPU threads - $error"); + }; + } + PVE::QemuServer::RunState::create_cleanup_flag($vmid); if (defined(my $migrate = $res->{migrate})) { diff --git a/src/PVE/QemuServer/CPUBinding.pm b/src/PVE/QemuServer/CPUBinding.pm new file mode 100644 index 00000000..3ce3653b --- /dev/null +++ b/src/PVE/QemuServer/CPUBinding.pm @@ -0,0 +1,133 @@ +package PVE::QemuServer::CPUBinding; + +use v5.36; + +use JSON; + +use PVE::QemuServer::Monitor; +use PVE::RS::CPU::Bind; + +our $bind_mode_fmt = { + mode => { + type => 'string', + enum => [qw(none manual one-to-one numa numa-manual)], + description => "Set the CPU binding mode for the guest.", + verbose_description => <<'EODESCR', +Selects how guest vCPUs are pinned to host CPUs. The mode determines +how cpu-binding-cfg is interpreted. + +* none (default): no binding. + +* manual: explicit vCPU-to-hostCPU mapping. The config string is a + semicolon-separated list of `:` entries. + Example: `0:0;1:1;2:2;3:3` pins vCPU 0 to CPU 0, vCPU 1 to CPU 1, + etc. Multi-value sets use `a-b` ranges or `{a,b,c}` unions. + +* one-to-one: pin each vCPU to its own physical core, chosen from a + user-supplied pool. The config string is an upper bound: the list of + host cores the vCPUs may be assigned to. SMT siblings are stripped + from that pool before assignment, so each vCPU gets its own distinct + physical core, not a hyperthread. + + If the pool has N cores and the VM has M vCPUs, each vCPU is pinned + to exactly one of the N cores (N must be >= M, otherwise validation + fails). For example, with `0-3` on a 4-vCPU VM, vCPU 0 gets core 0, + vCPU 1 gets core 1, and so on. Each vCPU is fixed to a single core, + it does not float within the pool. With `0;1;4;5` on the same VM, + vCPU 2 is pinned to core 4 and vCPU 3 to core 5, skipping cores 2-3. + + An empty string means "use all physical cores on the host" as the + upper bound. On a host with 16 cores, that pins an N-vCPU VM to + cores 0, 1, ..., N-1 — one core per vCPU, SMT stripped. + + Unlike `manual`, you do not specify which core each vCPU gets; that + assignment is up to hwloc. Use `manual` if you need a specific + vCPU-to-core mapping. + +* numa: bind the set of vCPUs of each guest vNUMA node to the set of + CPUs of a host NUMA node. The config string is a `;`-separated list + of host NUMA node indices, consumed in order: guest vNUMA 0 gets the + first host node, guest vNUMA 1 the second, and so on. + + Each guest vNUMA node's vCPUs form a group that may run on any CPU + of the assigned host NUMA node — including its cores and SMT threads. + The host scheduler is free to migrate vCPUs within the node, but not + outside it. This is the intended trade-off: memory allocated by a + vCPU stays local to the host node it runs on (preserving NUMA + memory locality), while the scheduler still has room to balance + load across the node's cores and hyperthreads. + + This is a set-to-set binding, not a per-vCPU assignment. All vCPUs + of a guest vNUMA node share the same host CPU pool. If you need + each vCPU pinned to its own distinct core, use one-to-one or manual + instead. + + Example: `3;4` lets all vCPUs on guest vNUMA 0 run on any CPU of + host NUMA node 3, and all vCPUs on guest vNUMA 1 run on any CPU of + host NUMA node 4. + + An empty string uses host NUMA nodes in ascending index order: + guest vNUMA 0 → host node 0, guest vNUMA 1 → host node 1, etc. + +* numa-manual: explicit variant of numa. The config string is a + `;`-separated list of `:` pairs. Each guest + vNUMA node's set of vCPUs is bound to the set of CPUs of the + specified host NUMA node. + + Semantics are identical to numa — vCPUs of a guest vNUMA node may + be scheduled on any CPU of the assigned host node, but cannot leave + it. Only the node mapping is explicit here. + + Example: `0:2;1:3` binds all vCPUs on guest vNUMA 0 to host NUMA + node 2's CPUs, and all vCPUs on guest vNUMA 1 to host NUMA node 3's + CPUs. +EODESCR + default => 'none', + default_key => 1, + }, +}; + +our $bind_cfg_fmt = { + cfg => { + type => 'string', + description => "CPU binding configuration (syntax depends on cpu-binding-mode).", + verbose_description => <<'EODESCR', +The exact syntax depends on cpu-binding-mode: + +* manual, one-to-one: + `;`-separated list of `:` entries (manual) + or a plain host CPU list (one-to-one). CPU sets accept `a-b` ranges + and `{a,b,c}` unions. + +* numa: + `;`-separated list of host NUMA node indices, e.g. `3;4`. + +* numa-manual: + `;`-separated list of `:` pairs, e.g. `0:2;1:3`. + +Host CPU and NUMA node indices are OS indices as reported by `lstopo -p`. +EODESCR + default => '', + default_key => 1, + }, +}; + +sub validate_binding($mode, $sockets, $cores, $binding_cfg) { + return if $mode eq 'none'; + + $binding_cfg //= ''; + + PVE::RS::CPU::Bind::validate_binding($mode, $sockets, $cores, $binding_cfg); +} + +sub bind_vcpus($vmid, $mode, $binding_cfg) { + return if $mode eq 'none'; + + $binding_cfg //= ''; + + my $cpuinfo = PVE::QemuServer::Monitor::mon_cmd($vmid, 'query-cpus-fast'); + my $cpuinfo_json = JSON::encode_json($cpuinfo); + PVE::RS::CPU::Bind::bind_vcpus($mode, $binding_cfg, $cpuinfo_json); +} + +1; diff --git a/src/PVE/QemuServer/Makefile b/src/PVE/QemuServer/Makefile index 61ed1a4e..ee123cf5 100644 --- a/src/PVE/QemuServer/Makefile +++ b/src/PVE/QemuServer/Makefile @@ -8,6 +8,7 @@ SOURCES=Agent.pm \ Cfg2Cmd.pm \ CGroup.pm \ Cloudinit.pm \ + CPUBinding.pm \ CPUConfig.pm \ CPUFlags.pm \ DBusVMState.pm \ -- 2.47.3