From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 3815E1FF0B7 for ; Fri, 30 Oct 2026 14:47:02 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A3AF72161C; Fri, 30 Oct 2026 14:46:58 +0100 (CET) Message-ID: Date: Fri, 30 Oct 2026 14:46:49 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH proxmox-perl-rs/qemu-server/qemu-server-rs v3 0/3] vCPU binding for qemu guests To: Elias Huhsovitz , pve-devel@lists.proxmox.com References: <20261022092227.58871-1-e.huhsovitz@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: <20261022092227.58871-1-e.huhsovitz@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793368010166 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.058 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QAJJFPICH2KPHKV2GX7JSII662OZOXEY X-Message-ID-Hash: QAJJFPICH2KPHKV2GX7JSII662OZOXEY X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, here a few high level comments: On 10/22/26 11:22 AM, Elias Huhsovitz wrote: > 1. Overview > ----------- > This series adds a convinient way to achive vCPU:CPU binding in PVE. > A new Rust crate, pve-qemu-server-cpu, exposes binding and validation via > PVE::RS::CPU::Bind, called from the VM start path after QEMU has > spawned its vCPU threads. > > Five modes are added: IMO for starters, we should keep the scope smaller and focus on the one-to-one and numa mappings. Manual can be nice, but can still be done later on, no need to expose such things from the beginning if it's not clear anybody needs that (also one can also do this manually with a hookscript and a bit of scripting) > > * none (default): no binding. > > * manual: explicit vCPU-to-hostCPU mapping. The config string is a > semicolon-separated list of `:` entries. > Example: `0:0;1:1;2:2;3:3` pins vCPU 0 to CPU 0, vCPU 1 to CPU 1, > etc. Multi-value sets use `a-b` ranges or `{a,b,c}` unions. > > * one-to-one: pin each vCPU to its own physical core, chosen from a > user-supplied pool. The config string is an upper bound: the list of > host cores the vCPUs may be assigned to. SMT siblings are stripped > from that pool before assignment, so each vCPU gets its own distinct > physical core, not a hyperthread. I would not do it that way, using hyperthreads can be useful when one wants to give most of the host to a vm. Restricting can be done using the affinity setting, but more on that below. > > If the pool has N cores and the VM has M vCPUs, each vCPU is pinned > to exactly one of the N cores (N must be >= M, otherwise validation > fails). For example, with `0-3` on a 4-vCPU VM, vCPU 0 gets core 0, > vCPU 1 gets core 1, and so on. Each vCPU is fixed to a single core, > it does not float within the pool. With `0;1;4;5` on the same VM, > vCPU 2 is pinned to core 4 and vCPU 3 to core 5, skipping cores 2-3. > > An empty string means "use all physical cores on the host" as the > upper bound. On a host with 16 cores, that pins an N-vCPU VM to > cores 0, 1, ..., N-1 — one core per vCPU, SMT stripped. > > Unlike `manual`, you do not specify which core each vCPU gets; that > assignment is up to hwloc. Use `manual` if you need a specific > vCPU-to-core mapping. > > * numa: bind the set of vCPUs of each guest vNUMA node to the set of > CPUs of a host NUMA node. The config string is a `;`-separated list > of host NUMA node indices, consumed in order: guest vNUMA 0 gets the > first host node, guest vNUMA 1 the second, and so on. > > Each guest vNUMA node's vCPUs form a group that may run on any CPU > of the assigned host NUMA node — including its cores and SMT threads. > The host scheduler is free to migrate vCPUs within the node, but not > outside it. This is the intended trade-off: memory allocated by a > vCPU stays local to the host node it runs on (preserving NUMA > memory locality), while the scheduler still has room to balance > load across the node's cores and hyperthreads. > > This is a set-to-set binding, not a per-vCPU assignment. All vCPUs > of a guest vNUMA node share the same host CPU pool. If you need > each vCPU pinned to its own distinct core, use one-to-one or manual > instead. > > Example: `3;4` lets all vCPUs on guest vNUMA 0 run on any CPU of > host NUMA node 3, and all vCPUs on guest vNUMA 1 run on any CPU of > host NUMA node 4. > > An empty string uses host NUMA nodes in ascending index order: > guest vNUMA 0 → host node 0, guest vNUMA 1 → host node 1, etc. here we should simply reuse the existing numaX config instead? if that is not available just guess or choose at random, no need to duplicate a guest <-> host numa node mapping. In the numaX setting we already can have a vcpu -> host numa node mapping, introducing a new setting where we can define that as well is confusing and not necessary. > > * numa-manual: explicit variant of numa. The config string is a > `;`-separated list of `:` pairs. Each guest > vNUMA node's set of vCPUs is bound to the set of CPUs of the > specified host NUMA node. > > Semantics are identical to numa — vCPUs of a guest vNUMA node may > be scheduled on any CPU of the assigned host node, but cannot leave > it. Only the node mapping is explicit here. > > Example: `0:2;1:3` binds all vCPUs on guest vNUMA 0 to host NUMA > node 2's CPUs, and all vCPUs on guest vNUMA 1 to host NUMA node 3's > CPUs. > > Configuration is via two new VM config keys, cpu-binding-mode > and cpu-binding-cfg. Host CPU and NUMA indices are OS indices > (PU P#N from lstopo -p). > > 2. Prerequisites > ---------------- > The series is based on Dominik's PCI series [0]. > > This patch adds a new build dependency: libhwloc-dev. It is > declared in pve-qemu-server-rs/pve-qemu-server-cpu/debian/ > debcargo.toml. > > 3. Request for feedback > ----------------------- > I am looking for feedback, especially on: > > * whether adding hwloc as a dependency is acceptable for > qemu-server. > * whether binding belongs in a dedicated crate or should live > closer to qemu-server. > * the config schema and the chosen mode split. > * naming. > * general design decisions (e.g. if rust is the right choice here) > > 4. Why hwloc > ------------ > hwloc [4] is widely used, well maintained, small and already available > on Debian (ver 2.12.0-4 at the time of writing this). > > It abstracts hardware topology discovery across architectures and > CPU generations, including SMT, cache hierarchy, NUMA nodes, and > CPU kinds. > > Without hwloc we would parse topology ourselves, from sysfs or > from arch-specific interfaces. Every new CPU family would > become our maintenance burden, and parser bugs would surface > only as binding failures on customer hardware. > > hwloc also unlocks further performance work requiring topology > data: > > * NUMA-aware memory placement (bind memory to a node) > * cache-aware vCPU placement (group by shared L2/L3) > * heterogeneous CPU support (Intel P/E cores) via CPU kinds > * distance-aware placement (latency/bandwidth between nodes) > > The hwloc FAQ has a section for exactly this question: > "I only need binding, or the number of cores, why should I use > hwloc?" [1]. my 2 cents, we could probably save us the trouble of having hwloc bindings and wrapper code by simply using the infos from sysfs, for the proposed features we really don't need more. Since the bindings and safe wrapper does not look too bad, it can be ok though, so no very hard feelings here. IMO this is more of a taste question, and maybe someone else would like to chime in here. > > 5. Why not hwlocality > --------------------- > hwlocality [2,3] is the idiomatic high-level Rust binding to > hwloc. It is still very new (1.0.0-alpha) and not packaged in > Debian. Packaging it would require packaging its full > transitive dependency tree, several of which are also not yet > in Debian. That scope is too much for this feature IMO. > > Our FFI bindings are a minimal hand-rolled subset of hwloc's > public C API, matching what hwlocality-sys exposes (I have > literally taken the code from hwlocality-sys [5]). Migration > to hwlocality later is straightforward: swap the bindings > module for the crate and delete the wrapper types. > > 6. Affinity settings > -------------------- > The existing 'affinity' option is not integrated with this > feature yet. IMO these two are conceptually different: affinity > in PVE is a list of allowed CPUs; binding is a per-vCPU placement. > A single flat list cannot express per-vCPU intent. Also, the affinity > code lives purely in perl. > This is wrong in my opinion: we shouldn't ignore a setting when setting something else, especially when they overlap: affinity could be used to limit the set of available host cpus, with that one can represent things like physical cores only, or limit to specific set of 'big' or 'little' cores manually. Granted, we should improve the UI/UX for that when exposing this feature on the UI, but that is a different problem to solve. > For now this binding code is experimental and should be used > without 'affinity'. Running taskset alongside the Rust binder > would race on the same threads and is hard to reason about. exactly the reason why we shouldn't ignore the combination. > > Long-term I would like to move affinity handling into Rust as > well, so both features share topology-based validation and a > common binding backend. For this patch, shipping binding > independently and harmonizing with affinity later seems > cleanest. couldn't that be done already with the hwloc bindings? an affinity only setting would just set every thread to the set of the cpus in the affinity config? > > 7. Reproducibility > ------------------ > The original RFC [6] relied on 'affinity' to limit the CPU set and > kept the vCPU-to-CPU assignment in a Perl hash. Two configs > that looked identical could produce different placement, > depending on hash ordering and how affinity was set. > > To make placement reproducible from the config file alone, the > binding logic lives entirely in Rust and the assignment is > read from cpu-binding-cfg. Two VMs with the same config on the same host > should get the same placement. (unless there is some randomness in the > hwloc_bitmap_singlify_per_core function that i am unaware of) > > 8. Basic usage > -------------- > # Explicit per-vCPU binding > qm set --cpu-binding-mode manual > qm set --cpu-binding-cfg "0:0;1:1;2:2;3:3" > > # One physical core per vCPU, let hwloc place them > qm set --cpu-binding-mode one-to-one > qm set --cpu-binding-cfg "0-3" > > # Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3 > qm set --cpu-binding-mode numa-manual > qm set --cpu-binding-cfg "0:2;1:3" > > # Automatic numa binding: > #Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3 > qm set --cpu-binding-mode numa > qm set --cpu-binding-cfg "2-3" > > Without cpu-binding-mode the VM behaves as before. Validation > runs at qm set time and at VM start; binding itself runs after > QEMU spawns the vCPU threads. > > 9. What is not in this PATCH > ---------------------------- > * memory_binding: IMO this is best handled by qemu directly and already > possible via, e.g.: > > numa0: cpus=0-1,hostnodes=0,memory=2048,policy=bind > numa1: cpus=2-3,hostnodes=1,memory=2048,policy=bind > numa2: cpus=4-5,hostnodes=2,memory=2048,policy=bind > numa3: cpus=6-7,hostnodes=3,memory=2048,policy=bind > > What we need is a convinient way to automatically apply the correct > memory binding for a given CPU Binding config. > > * Qemu I/O Thread control/binding > > * CPU Isolation > > I plan on adding most of the features requested in the forum post [7]. > > 10. Previous versions > --------------------- > v2: https://lore.proxmox.com/pve-devel/20260921095404.61552-1-e.huhsovitz@proxmox.com/ > v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ > (Credits to Dominik) > > 11. References > ------------- > [0] https://lore.proxmox.com/pve-devel/20261015081041.743422-1-d.csapak@proxmox.com > [1] https://hwloc.readthedocs.io/en/hwloc-2.15.0/doxygen/html/faq.html > [2] https://crates.io/crates/hwlocality > [3] https://github.com/HadrienG2/hwlocality > [4] https://github.com/open-mpi/hwloc > [5] https://github.com/HadrienG2/hwlocality/tree/main/hwlocality-sys > [6] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ > [7] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/ > > Thank you to whoever reviews this! :) > > Summary of changes > ------------- > pve-qemu-server-rs: > > Elias Huhsovitz (1): > cpu: add pve-qemu-server-cpu crate for vCPU binding > > Cargo.toml | 2 +- > build.sh | 2 +- > pve-qemu-server-cpu/Cargo.toml | 16 + > pve-qemu-server-cpu/debian/changelog | 5 + > pve-qemu-server-cpu/debian/control | 41 ++ > pve-qemu-server-cpu/debian/copyright | 18 + > pve-qemu-server-cpu/debian/debcargo.toml | 9 + > pve-qemu-server-cpu/src/bind.rs | 478 ++++++++++++++++++++++ > pve-qemu-server-cpu/src/error.rs | 38 ++ > pve-qemu-server-cpu/src/hwloc/bindings.rs | 171 ++++++++ > pve-qemu-server-cpu/src/hwloc/mod.rs | 222 ++++++++++ > pve-qemu-server-cpu/src/lib.rs | 92 +++++ > pve-qemu-server-cpu/src/qmp_cpu.rs | 99 +++++ > 13 files changed, 1191 insertions(+), 2 deletions(-) > create mode 100644 pve-qemu-server-cpu/Cargo.toml > create mode 100644 pve-qemu-server-cpu/debian/changelog > create mode 100644 pve-qemu-server-cpu/debian/control > create mode 100644 pve-qemu-server-cpu/debian/copyright > create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml > create mode 100644 pve-qemu-server-cpu/src/bind.rs > create mode 100644 pve-qemu-server-cpu/src/error.rs > create mode 100644 pve-qemu-server-cpu/src/hwloc/bindings.rs > create mode 100644 pve-qemu-server-cpu/src/hwloc/mod.rs > create mode 100644 pve-qemu-server-cpu/src/lib.rs > create mode 100644 pve-qemu-server-cpu/src/qmp_cpu.rs > > > proxmox-perl-rs: > > Elias Huhsovitz (1): > pve: cpu: add PVE::RS::CPU::Bind perl bindings > > pve-rs/Cargo.toml | 1 + > pve-rs/Makefile | 1 + > pve-rs/src/bindings/cpu/bind.rs | 40 +++++++++++++++++++++++++++++++++ > pve-rs/src/bindings/cpu/mod.rs | 2 ++ > pve-rs/src/bindings/mod.rs | 3 +++ > 5 files changed, 47 insertions(+) > create mode 100644 pve-rs/src/bindings/cpu/bind.rs > create mode 100644 pve-rs/src/bindings/cpu/mod.rs > > > qemu-server: > > Elias Huhsovitz (1): > cpu: add CPU binding > > src/PVE/QemuServer.pm | 36 +++++++++ > src/PVE/QemuServer/CPUBinding.pm | 133 +++++++++++++++++++++++++++++++ > src/PVE/QemuServer/Makefile | 1 + > 3 files changed, 170 insertions(+) > create mode 100644 src/PVE/QemuServer/CPUBinding.pm > > > Summary over all repositories: > 21 files changed, 1408 insertions(+), 2 deletions(-) >