From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 53BB61FF0DF for ; Thu, 19 Nov 2026 10:35:49 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 90634216A7; Thu, 19 Nov 2026 10:35:47 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Nov 2026 10:35:40 +0100 Message-Id: Subject: Re: [PATCH qemu-server v4 4/4] cpu: add cpu binding From: "Elias Huhsovitz" To: "Dominik Csapak" , Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: aerc 0.20.0 References: <20261111152027.260117-1-e.huhsovitz@proxmox.com> <20261111152027.260117-5-e.huhsovitz@proxmox.com> <0c1ed99f-7379-4cb9-94f9-1b564b78242a@proxmox.com> In-Reply-To: <0c1ed99f-7379-4cb9-94f9-1b564b78242a@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795080940201 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.408 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: SF6UMIP6NVRBHICDH2IBUF7FPI5NZ3J7 X-Message-ID-Hash: SF6UMIP6NVRBHICDH2IBUF7FPI5NZ3J7 X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue Nov 17, 2026 at 5:36 PM CET, Dominik Csapak wrote: > comments inline > > On 11/11/26 4:21 PM, Elias Huhsovitz wrote: >> Add a new qm.conf option `cpu-binding-mode`: none | one-to-one | numa >>=20 >> When cpu-binding-mode is unset or "none", behavior is unchanged. >> Otherwise, vCPU threads are pinned to host CPUs after QEMU has spawned >> them. >>=20 >> Two functional modes are supported: (1) one-to-one - each vCPU gets its >> own cpu, defined by the affinity-pool. (2) numa - per-vNUMA-node >> set-to-set binding, defined by numa [N] config. >>=20 >> Two entry points are provided in PVE::QemuServer::CPUBind. (1) >> gen_vcpu_bindings() - called before VM QEMU spawns. Generates a binding >> map, specifying vCPU:host cpu(s). (2) bind_vcpus() - called after QEMU >> started. Binds the vCPUs according to the generated binding map. >> information as JSON and hands it to Rust together with the mode and cfg >> string. >>=20 >> Originally-by: Dominik Csapak >> Signed-off-by: Elias Huhsovitz >> --- >> src/PVE/QemuServer.pm | 32 ++++++++++- >> src/PVE/QemuServer/CPUBind.pm | 103 ++++++++++++++++++++++++++++++++++ >> src/PVE/QemuServer/Makefile | 1 + >> src/PVE/QemuServer/Memory.pm | 52 ++++++++++++++++- >> 4 files changed, 186 insertions(+), 2 deletions(-) >> create mode 100644 src/PVE/QemuServer/CPUBind.pm >>=20 >> diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm >> index 8149184e..a8269e36 100644 >> --- a/src/PVE/QemuServer.pm >> +++ b/src/PVE/QemuServer.pm >> @@ -59,6 +59,7 @@ use PVE::QemuServer::Helpers >> qw(config_aware_timeout get_iscsi_initiator_name get_host_arch min= _version kvm_user_version windows_version); >> use PVE::QemuServer::Cloudinit; >> use PVE::QemuServer::CGroup; >> +use PVE::QemuServer::CPUBind; >> use PVE::QemuServer::CPUConfig qw( >> print_cpu_device >> get_cpu_options >> @@ -741,6 +742,12 @@ EODESCR >> optional =3D> 1, >> default =3D> 1, >> }, >> + 'cpu-binding-mode' =3D> { > > this here is 'cpu-binding-*mode*' and the first > value there is 'mode' so a full explicit config looks like: > > cpu-binding-mode: mode=3D<...> > > IMO one "mode" too much > ( i know it's not there most of the time, but i gues on the > top level prop we can still drop it) > This is a remnant of a previous version. In my testing I set cpu-binding-mode: one-to-one | numa and it worked fine, so I didn't notice. Thanks! Actually having a format like this, might be the way to go: cpu-binding: mode=3D<...> since this allows us to opt for more nuanced settings (e.g. avoiding SMT siblings) cpu-binding: mode=3Done-to-one,allow-smt-siblings=3D0 >> + type =3D> 'string', >> + format =3D> $PVE::QemuServer::Bind::bind_mode_fmt, >> + description =3D> "Set CPU binding mode for the guest", >> + optional =3D> 1, >> + }, >> }; >> =20 >> my $cicustom_fmt =3D { >> @@ -3189,7 +3196,7 @@ sub config_to_command { >> ($use_old_bios_files, $machine_type) =3D qemu_use_old_bios_files($= machine_type); >> =20 >> my $cmd =3D []; >> - if ($conf->{affinity}) { >> + if ($conf->{affinity} && !$conf->{'cpu-binding-mode'}) { >> push @$cmd, '/usr/bin/taskset', '--cpu-list', '--all-tasks', $= conf->{affinity}; >> } > > what i meant in the crate: here we skip the taskset if cpu-binding-mode > is 'none' (explicitely, this is a valid value in the config)... > Perhaps removing the 'none' state is the optimal solution. I implemented the 'none' state, since I was unsure what the convention for "no config" is, since: e.g. disabling "Freeze CPU at startup" causes the=20 freeze: 1 to disappear but disabling NUMA causes numa: 0 to be set. >> =20 >> @@ -5514,6 +5521,8 @@ sub vm_start_nolock { >> =20 >> my ($cmd, $spice_port, $start_timeout); >> my $pci_reserve_list =3D []; >> + my $vcpu_binding_map; >> + my $freeze_for_binding =3D 0; > > i can guess why this is helpful, but can you please > either explain it in a comment or in the commit message? Will do! > > FWICT this is so the cpus never execute on threads intended for them? Yes! I have noticed that since we pin the vCPUs after startup, there is a small time frame where vCPUs are online and can be schedlued on host all host cpus within the affinity mask. It is a minor optimization, but I figured it might be relevant. > >> eval { >> # With -blockdev, it is necessary to activate the volumes befo= re generating the command line >> # Plugins can safely deactivate already-active volumes here if= needed >> @@ -5539,10 +5548,21 @@ sub vm_start_nolock { >> }, >> ); >> =20 >> + if ($conf->{'cpu-binding-mode'}) { >> + $vcpu_binding_map =3D PVE::QemuServer::CPUBind::conf_to_vcp= u_bindings($conf); > > ... but here int he crate we don't do any cpu pinning > > we could also always do it in the crate > > then the taskset call would vanish here. > > we'd have to do > > my $binding =3D parse_cpu_binding($conf-{'pcu- > I am not quite sure what is meant by this.=20 Do you mean that we simply hand all affinity related logic to the code responsible for the cpu pinning logic?=20 Or do you mean that we should init qemu using taskset anyways and then later set the affinity of the vcpus again when it is time to pin? Since I chose to freeze the vm on startup, until pinning is applied, I figured the original taskset to be redundant. >> + } >> + >> + $freeze_for_binding =3D >> + $vcpu_binding_map >> + && !$params->{paused} >> + && !$statefile >> + && !$conf->{freeze}; >> + >> my $memory =3D get_current_memory($conf->{memory}); >> $start_timeout =3D $params->{timeout} // config_aware_timeout(= $conf, $memory, $resume); >> =20 >> push $cmd->@*, $state_cmdline->@*; >> + push @$cmd, '-S' if $freeze_for_binding; >> =20 >> ($pci_reserve_list, my $uuid) =3D >> PVE::QemuServer::PCI::prepare_pci_devices($conf, $vmid, $p= ci_devices, $start_timeout); >> @@ -5759,6 +5779,16 @@ sub vm_start_nolock { >> my $qemu_pkg_string =3D PVE::QemuServer::QMPHelpers::get_qemu_pack= age_string($vmid); >> syslog("info", "VM $vmid started with PID $pid using $qemu_pkg_str= ing."); >> =20 >> + if ($vcpu_binding_map) { >> + eval { PVE::QemuServer::CPUBind::bind_vcpus($vmid, $vcpu_bindin= g_map) }; >> + log_warn("unable to bind vCPU threads - $@") if $@; >> + } >> + >> + if ($freeze_for_binding) { >> + eval { mon_cmd($vmid, "cont") }; >> + log_warn("failed to resume VM after CPU binding: $@") if $@; > > mhmm not sure about that. if the cont call fails, the vm might not > continue and it looks like it froze > > stopping again seems also overkill though... > maybe we could still fail the task but leave the guest "running" > Hmmm I am not sure what is best here . I guess leaving the guest running with a loud warning might be best.=20 Or we could also disregard the freezing approach entirely. >> + } >> + >> PVE::QemuServer::RunState::create_cleanup_flag($vmid); >> =20 >> if (defined(my $migrate =3D $res->{migrate})) { >> diff --git a/src/PVE/QemuServer/CPUBind.pm b/src/PVE/QemuServer/CPUBind.= pm >> new file mode 100644 >> index 00000000..0566b08b >> --- /dev/null >> +++ b/src/PVE/QemuServer/CPUBind.pm >> @@ -0,0 +1,103 @@ >> +package PVE::QemuServer::CPUBind; >> + >> +use v5.36; >> + >> +use JSON; >> + >> +use PVE::QemuServer::Memory; >> +use PVE::QemuServer::Monitor; >> +use PVE::RS::CPU::Bind; >> + >> +our $bind_mode_fmt =3D { >> + mode =3D> { >> + type =3D> 'string', >> + enum =3D> [qw(none one-to-one numa)], >> + description =3D> "Set the CPU binding mode for the guest.", >> + verbose_description =3D> <<'EODESCR', >> +Determines how guest vCPUs are bound/pinned to host CPUs. >> + >> +* none (default): no binding. Regular scheduling. >> + >> +* one-to-one: pin each vCPU to its own physical core, chosen from the > > not physical core anymore, i'd just drop the 'physical' here or use=20 > 'logical' Thanks! I missed this >> + pool configured by affinity. The configured affinity string acts as >> + an upper bound on the host cpus, that can be used for binding. >> + >> + If the guest has N vCPUS and the affinity-pool has M host cpus, each = vCPU >> + is fixed to a single one of the M host cpus. (M >=3D N, must be true)= . >> + >> + Each vCPU is fixed to a single host cpu, it does not float within >> + the pool. >> + >> + Example 1: 4-vCPU guest with affinity=3D4-9: >> + The resulting mapping would be: vCPU0:CPU4, vCPU1:CPU5, vCPU2:CPU6, >> + vCPU3:CPU7. >> + >> + Example 2: 4-vCPU guest with affinity=3D0,2: >> + The resulting mapping would be: vCPU0:CPU0, vCPU1:CPU2 >> + >> + An empty affinity string means all host cpus will be considered. >> + On a host with 16 cores, that pins an N-vCPU VM to host cpus >> + 0, 1, ..., N-1. >> + >> +* numa: bind the set of vCPUs of each vNUMA node to the set of >> + CPUs of a host NUMA node. The cpu bindings are determined by the >> + numa[N] configuration of the guest. If no numa[N] config is >> + explicitly provided, then for each socket a numa[N] node is created. >> + >> + Each guest vNUMA node's vCPUs form a group that may run on any CPU >> + of the assigned host NUMA node. The host scheduler is free to migrate >> + the vCPUs within the host node. This is the intended trade-off: >> + memory allocated by a vCPU stays local to the host node it runs on >> + (preserving NUMA memory locality), while the scheduler still has room >> + to balance load across the node's cores and hyperthreads. >> + >> + Examples: >> + numa0: cpus=3D0-1,hostnodes=3D0,memory=3D2048,policy=3Dbind >> + Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0 >> + >> + numa1: cpus=3D6-7,hostnodes=3D3,memory=3D2048,policy=3Dbind >> + Allows vCPUs 6-7 to execute on all cpus of host NUMA node 3 >> + >> + numa:1, no numa[N], sockets: 2, cores: 2, total vCPUs: 4 >> + Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0 >> + Allows vCPUs 2-3 to execute on all cpus of host NUMA node 1 >> + >> +EODESCR >> + default =3D> 'none', >> + default_key =3D> 1, >> + }, >> +}; >> + >> +my sub gen_vcpu_bindings($mode, $sockets, $cores, $affinity, $cfg) { >> + my $cfg_json =3D ''; # currently only relevant for numa mode. >> + if ($mode eq 'numa') { >> + $cfg_json =3D JSON::encode_json($cfg); >> + } >> + >> + # PVE::RS::CPU::Bind::gen_vcpu_bindings returns the map as JSON. Th= is is currently >> + # not decoded, because at the moment we are simply passing the JSON= bind map to >> + # bind_vcpus. Should this change in the future, decode the JSON str= ing before returning. >> + return PVE::RS::CPU::Bind::gen_vcpu_bindings($mode, $sockets, $core= s, $affinity, $cfg_json); >> +} >> + >> +sub conf_to_vcpu_bindings($conf) { >> + my $mode =3D $conf->{'cpu-binding-mode'}; >> + return if $mode eq 'none'; >> + >> + my $cfg =3D ''; >> + $cfg =3D PVE::QemuServer::Memory::numa_conf_to_map($conf) if $mode = eq 'numa'; >> + >> + my $vcpu_binding_map =3D gen_vcpu_bindings( >> + $mode, $conf->{sockets}, $conf->{cores}, $conf->{affinity} // '= ', $cfg, >> + ); >> + >> + return $vcpu_binding_map; >> +} >> + >> +sub bind_vcpus($vmid, $bind_map_json) { >> + my $cpuinfo =3D PVE::QemuServer::Monitor::mon_cmd($vmid, 'query-cpu= s-fast'); >> + my $qmp_cpuinfo_json =3D JSON::encode_json($cpuinfo); >> + PVE::RS::CPU::Bind::bind_vcpus($bind_map_json, $qmp_cpuinfo_json); >> +} >> + >> +1; >> diff --git a/src/PVE/QemuServer/Makefile b/src/PVE/QemuServer/Makefile >> index 61ed1a4e..c4cd9077 100644 >> --- a/src/PVE/QemuServer/Makefile >> +++ b/src/PVE/QemuServer/Makefile >> @@ -8,6 +8,7 @@ SOURCES=3DAgent.pm \ >> Cfg2Cmd.pm \ >> CGroup.pm \ >> Cloudinit.pm \ >> + CPUBind.pm \ >> CPUConfig.pm \ >> CPUFlags.pm \ >> DBusVMState.pm \ >> diff --git a/src/PVE/QemuServer/Memory.pm b/src/PVE/QemuServer/Memory.pm >> index 8cbbac84..282d5e23 100644 >> --- a/src/PVE/QemuServer/Memory.pm >> +++ b/src/PVE/QemuServer/Memory.pm >> @@ -87,6 +87,57 @@ our $memory_fmt =3D { >> }, >> }; >> =20 >> +sub numa_conf_to_map { >> + my ($conf) =3D @_; >> + >> + my $numa_cfg =3D {}; >> + my $has_custom_topology =3D 0; >> + >> + for (my $i =3D 0; $i < $PVE::QemuServer::Memory::MAX_NUMA; $i++) { >> + my $entry =3D $conf->{"numa$i"} or next; >> + $has_custom_topology =3D 1; >> + >> + my $numa =3D PVE::QemuServer::Memory::parse_numa($entry); >> + >> + $numa_cfg->{"numa$i"} =3D { >> + vcpus =3D> $numa->{cpus} // [], >> + hostnodes =3D> $numa->{hostnodes} // [], >> + memory =3D> 0 + ($numa->{memory} // 0), >> + policy =3D> $numa->{policy}, >> + }; >> + } >> + >> + return $numa_cfg if $has_custom_topology; >> + >> + die "NUMA must be enabled in the VM configuration\n" >> + if !$conf->{numa}; >> + >> + my $sockets =3D $conf->{sockets} || 1; >> + my $cores =3D $conf->{cores} || 1; >> + my $memory =3D get_current_memory($conf->{memory}); >> + my $numa_memory =3D int($memory / $sockets); >> + >> + for (my $i =3D 0; $i < $sockets; $i++) { >> + my $start_cpu =3D $cores * $i; >> + my $end_cpu =3D $start_cpu + $cores - 1; >> + >> + # Format matches parse_number_sets output: [[start, end]] or [[= start, undef]] >> + my $cpus =3D $cores > 1 ? [[$start_cpu, $end_cpu]] : [[$start_c= pu, undef]]; >> + >> + # Fallback maps guest node $i directly to host node $i >> + my $hostnodes =3D [[$i, undef]]; >> + >> + $numa_cfg->{"numa$i"} =3D { >> + vcpus =3D> $cpus, >> + hostnodes =3D> $hostnodes, >> + memory =3D> 0 + $numa_memory, >> + policy =3D> undef, >> + }; >> + } >> + >> + return $numa_cfg; >> +} >> + >> sub print_memory { >> my $memory =3D shift; >> =20 >> @@ -825,4 +876,3 @@ sub hugepages_update_locked { >> return $res; >> } >> 1; >> -