From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B30A31FF0B7 for ; Fri, 30 Oct 2026 14:47:11 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 20A1F2166A; Fri, 30 Oct 2026 14:46:59 +0100 (CET) Message-ID: <447abbd2-ee7b-49b0-9ebc-d7c52c76b9d1@proxmox.com> Date: Fri, 30 Oct 2026 14:46:52 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH qemu-server v3 3/3] cpu: add CPU binding To: Elias Huhsovitz , pve-devel@lists.proxmox.com References: <20261022092227.58871-1-e.huhsovitz@proxmox.com> <20261022092227.58871-4-e.huhsovitz@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: <20261022092227.58871-4-e.huhsovitz@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793368012704 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.342 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GR5U4UR57ZXU343NAALZSP255TBY7YUW X-Message-ID-Hash: GR5U4UR57ZXU343NAALZSP255TBY7YUW X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: some comments inline On 10/22/26 11:22 AM, Elias Huhsovitz wrote: > Add two new VM config options: > * cpu-binding-mode none | manual | one-to-one | numa | numa-manual > * cpu-binding-cfg mode-dependent syntax > > When cpu-binding-mode is unset or "none", behavior is unchanged. > Otherwise, vCPU threads are pinned to host CPUs after QEMU has spawned > them, driven by the query-cpus-fast QMP response. > > Five modes are supported: > * manual - explicit vCPU:hostCPU mapping, e.g. "0:0;1:1;2:2;3:3" > * one-to-one - each vCPU gets its own physical core from a user pool, > SMT stripped, e.g. "0-3" > * numa - per-vNUMA-node set-to-set binding, sequential, e.g. "3;4" > * numa-manual - explicit vNUMA->host NUMA mapping, e.g. "0:2;1:3" > > Two entry points are provided in PVE::QemuServer::CPUBinding: > * validate_binding() - called at VM start, before QEMU spawns. Rejects > invalid configs against the current topology. > > * bind_vcpus() - called after QEMU is up. Encodes the QMP cpu > information as JSON and hands it to Rust together with the mode and cfg > string. > > Signed-off-by: Elias Huhsovitz > --- > src/PVE/QemuServer.pm | 36 +++++++++ > src/PVE/QemuServer/CPUBinding.pm | 133 +++++++++++++++++++++++++++++++ > src/PVE/QemuServer/Makefile | 1 + > 3 files changed, 170 insertions(+) > create mode 100644 src/PVE/QemuServer/CPUBinding.pm > > diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm > index 1efeeb66..3ba266be 100644 > --- a/src/PVE/QemuServer.pm > +++ b/src/PVE/QemuServer.pm > @@ -60,6 +60,7 @@ use PVE::QemuServer::Helpers > qw(config_aware_timeout get_iscsi_initiator_name get_host_arch min_version kvm_user_version windows_version); > use PVE::QemuServer::Cloudinit; > use PVE::QemuServer::CGroup; > +use PVE::QemuServer::CPUBinding; > use PVE::QemuServer::CPUConfig qw( > print_cpu_device > get_cpu_options > @@ -739,6 +740,18 @@ EODESCR > optional => 1, > default => 1, > }, > + 'cpu-binding-mode' => { > + type => 'string', > + format => $PVE::QemuServer::Bind::bind_mode_fmt, > + description => "Set CPU binding mode for the guest", > + optional => 1, > + }, > + 'cpu-binding-cfg' => { > + type => 'string', > + format => $PVE::QemuServer::Bind::bind_cfg_fmt, > + description => "Configuration for CPU binding for the guest", > + optional => 1, > + }, i don't like that they are two different settings. simplifying and having one property string would be better > }; > > my $cicustom_fmt = { > @@ -3218,6 +3231,15 @@ sub config_to_command { > push @$cmd, '/usr/bin/taskset', '--cpu-list', '--all-tasks', $conf->{affinity}; > } > > + if ($conf->{'cpu-binding-mode'}) { > + PVE::QemuServer::CPUBinding::validate_binding( > + $conf->{'cpu-binding-mode'}, > + $conf->{'sockets'}, > + $conf->{'cores'}, > + $conf->{'cpu-binding-cfg'}, > + ); > + } > + > push @$cmd, $kvm_binary; > > push @$cmd, '-id', $vmid; > @@ -5715,6 +5737,20 @@ sub vm_start_nolock { > my $qemu_pkg_string = PVE::QemuServer::QMPHelpers::get_qemu_package_string($vmid); > syslog("info", "VM $vmid started with PID $pid using $qemu_pkg_string."); > > + if ($conf->{'cpu-binding-mode'}) { > + eval { > + PVE::QemuServer::CPUBinding::bind_vcpus( > + $vmid, > + $conf->{'cpu-binding-mode'}, > + $conf->{'cpu-binding-cfg'}, > + ); > + 1; > + } or do { > + my $error = $@ || 'Unknown failure'; > + log_warn("unable to bind vCPU threads - $error"); > + }; > + } > + > PVE::QemuServer::RunState::create_cleanup_flag($vmid); > > if (defined(my $migrate = $res->{migrate})) { > diff --git a/src/PVE/QemuServer/CPUBinding.pm b/src/PVE/QemuServer/CPUBinding.pm > new file mode 100644 > index 00000000..3ce3653b > --- /dev/null > +++ b/src/PVE/QemuServer/CPUBinding.pm > @@ -0,0 +1,133 @@ > +package PVE::QemuServer::CPUBinding; > + > +use v5.36; > + > +use JSON; > + > +use PVE::QemuServer::Monitor; > +use PVE::RS::CPU::Bind; > + > +our $bind_mode_fmt = { > + mode => { > + type => 'string', > + enum => [qw(none manual one-to-one numa numa-manual)], > + description => "Set the CPU binding mode for the guest.", > + verbose_description => <<'EODESCR', > +Selects how guest vCPUs are pinned to host CPUs. The mode determines > +how cpu-binding-cfg is interpreted. > + > +* none (default): no binding. > + > +* manual: explicit vCPU-to-hostCPU mapping. The config string is a > + semicolon-separated list of `:` entries. > + Example: `0:0;1:1;2:2;3:3` pins vCPU 0 to CPU 0, vCPU 1 to CPU 1, > + etc. Multi-value sets use `a-b` ranges or `{a,b,c}` unions. > + > +* one-to-one: pin each vCPU to its own physical core, chosen from a > + user-supplied pool. The config string is an upper bound: the list of > + host cores the vCPUs may be assigned to. SMT siblings are stripped > + from that pool before assignment, so each vCPU gets its own distinct > + physical core, not a hyperthread. > + > + If the pool has N cores and the VM has M vCPUs, each vCPU is pinned > + to exactly one of the N cores (N must be >= M, otherwise validation > + fails). For example, with `0-3` on a 4-vCPU VM, vCPU 0 gets core 0, > + vCPU 1 gets core 1, and so on. Each vCPU is fixed to a single core, > + it does not float within the pool. With `0;1;4;5` on the same VM, > + vCPU 2 is pinned to core 4 and vCPU 3 to core 5, skipping cores 2-3. > + > + An empty string means "use all physical cores on the host" as the > + upper bound. On a host with 16 cores, that pins an N-vCPU VM to > + cores 0, 1, ..., N-1 — one core per vCPU, SMT stripped. > + > + Unlike `manual`, you do not specify which core each vCPU gets; that > + assignment is up to hwloc. Use `manual` if you need a specific > + vCPU-to-core mapping. > + > +* numa: bind the set of vCPUs of each guest vNUMA node to the set of > + CPUs of a host NUMA node. The config string is a `;`-separated list > + of host NUMA node indices, consumed in order: guest vNUMA 0 gets the > + first host node, guest vNUMA 1 the second, and so on. > + > + Each guest vNUMA node's vCPUs form a group that may run on any CPU > + of the assigned host NUMA node — including its cores and SMT threads. > + The host scheduler is free to migrate vCPUs within the node, but not > + outside it. This is the intended trade-off: memory allocated by a > + vCPU stays local to the host node it runs on (preserving NUMA > + memory locality), while the scheduler still has room to balance > + load across the node's cores and hyperthreads. > + > + This is a set-to-set binding, not a per-vCPU assignment. All vCPUs > + of a guest vNUMA node share the same host CPU pool. If you need > + each vCPU pinned to its own distinct core, use one-to-one or manual > + instead. > + > + Example: `3;4` lets all vCPUs on guest vNUMA 0 run on any CPU of > + host NUMA node 3, and all vCPUs on guest vNUMA 1 run on any CPU of > + host NUMA node 4. > + > + An empty string uses host NUMA nodes in ascending index order: > + guest vNUMA 0 → host node 0, guest vNUMA 1 → host node 1, etc. > + > +* numa-manual: explicit variant of numa. The config string is a > + `;`-separated list of `:` pairs. Each guest > + vNUMA node's set of vCPUs is bound to the set of CPUs of the > + specified host NUMA node. > + > + Semantics are identical to numa — vCPUs of a guest vNUMA node may > + be scheduled on any CPU of the assigned host node, but cannot leave > + it. Only the node mapping is explicit here. > + > + Example: `0:2;1:3` binds all vCPUs on guest vNUMA 0 to host NUMA > + node 2's CPUs, and all vCPUs on guest vNUMA 1 to host NUMA node 3's > + CPUs. > +EODESCR > + default => 'none', > + default_key => 1, > + }, > +}; > + > +our $bind_cfg_fmt = { > + cfg => { > + type => 'string', > + description => "CPU binding configuration (syntax depends on cpu-binding-mode).", > + verbose_description => <<'EODESCR', > +The exact syntax depends on cpu-binding-mode: > + > +* manual, one-to-one: > + `;`-separated list of `:` entries (manual) > + or a plain host CPU list (one-to-one). CPU sets accept `a-b` ranges > + and `{a,b,c}` unions. > + > +* numa: > + `;`-separated list of host NUMA node indices, e.g. `3;4`. > + > +* numa-manual: > + `;`-separated list of `:` pairs, e.g. `0:2;1:3`. > + > +Host CPU and NUMA node indices are OS indices as reported by `lstopo -p`. > +EODESCR > + default => '', > + default_key => 1, > + }, > +}; > + > +sub validate_binding($mode, $sockets, $cores, $binding_cfg) { > + return if $mode eq 'none'; > + > + $binding_cfg //= ''; > + > + PVE::RS::CPU::Bind::validate_binding($mode, $sockets, $cores, $binding_cfg); mhmm the config does not require a sockets or cores entry but the rust binding take a u32, we should read the default from the schema here > +} > + > +sub bind_vcpus($vmid, $mode, $binding_cfg) { > + return if $mode eq 'none'; > + > + $binding_cfg //= ''; > + > + my $cpuinfo = PVE::QemuServer::Monitor::mon_cmd($vmid, 'query-cpus-fast'); > + my $cpuinfo_json = JSON::encode_json($cpuinfo); > + PVE::RS::CPU::Bind::bind_vcpus($mode, $binding_cfg, $cpuinfo_json); > +} > + > +1; > diff --git a/src/PVE/QemuServer/Makefile b/src/PVE/QemuServer/Makefile > index 61ed1a4e..ee123cf5 100644 > --- a/src/PVE/QemuServer/Makefile > +++ b/src/PVE/QemuServer/Makefile > @@ -8,6 +8,7 @@ SOURCES=Agent.pm \ > Cfg2Cmd.pm \ > CGroup.pm \ > Cloudinit.pm \ > + CPUBinding.pm \ > CPUConfig.pm \ > CPUFlags.pm \ > DBusVMState.pm \