From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 2A7411FF0AF for ; Thu, 22 Oct 2026 11:22:53 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 0A09721669; Thu, 22 Oct 2026 11:22:40 +0200 (CEST) From: Elias Huhsovitz To: pve-devel@lists.proxmox.com Subject: [PATCH proxmox-perl-rs/qemu-server/qemu-server-rs v3 0/3] vCPU binding for qemu guests Date: Thu, 22 Oct 2026 11:22:22 +0200 Message-ID: <20261022092227.58871-1-e.huhsovitz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792660953136 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.112 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: AKOZMOQW7LDZRAC62DK6TVXRSYYAP7H6 X-Message-ID-Hash: AKOZMOQW7LDZRAC62DK6TVXRSYYAP7H6 X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Elias Huhsovitz X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: 1. Overview ----------- This series adds a convinient way to achive vCPU:CPU binding in PVE. A new Rust crate, pve-qemu-server-cpu, exposes binding and validation via PVE::RS::CPU::Bind, called from the VM start path after QEMU has spawned its vCPU threads. Five modes are added: * none (default): no binding. * manual: explicit vCPU-to-hostCPU mapping. The config string is a semicolon-separated list of `:` entries. Example: `0:0;1:1;2:2;3:3` pins vCPU 0 to CPU 0, vCPU 1 to CPU 1, etc. Multi-value sets use `a-b` ranges or `{a,b,c}` unions. * one-to-one: pin each vCPU to its own physical core, chosen from a user-supplied pool. The config string is an upper bound: the list of host cores the vCPUs may be assigned to. SMT siblings are stripped from that pool before assignment, so each vCPU gets its own distinct physical core, not a hyperthread. If the pool has N cores and the VM has M vCPUs, each vCPU is pinned to exactly one of the N cores (N must be >= M, otherwise validation fails). For example, with `0-3` on a 4-vCPU VM, vCPU 0 gets core 0, vCPU 1 gets core 1, and so on. Each vCPU is fixed to a single core, it does not float within the pool. With `0;1;4;5` on the same VM, vCPU 2 is pinned to core 4 and vCPU 3 to core 5, skipping cores 2-3. An empty string means "use all physical cores on the host" as the upper bound. On a host with 16 cores, that pins an N-vCPU VM to cores 0, 1, ..., N-1 — one core per vCPU, SMT stripped. Unlike `manual`, you do not specify which core each vCPU gets; that assignment is up to hwloc. Use `manual` if you need a specific vCPU-to-core mapping. * numa: bind the set of vCPUs of each guest vNUMA node to the set of CPUs of a host NUMA node. The config string is a `;`-separated list of host NUMA node indices, consumed in order: guest vNUMA 0 gets the first host node, guest vNUMA 1 the second, and so on. Each guest vNUMA node's vCPUs form a group that may run on any CPU of the assigned host NUMA node — including its cores and SMT threads. The host scheduler is free to migrate vCPUs within the node, but not outside it. This is the intended trade-off: memory allocated by a vCPU stays local to the host node it runs on (preserving NUMA memory locality), while the scheduler still has room to balance load across the node's cores and hyperthreads. This is a set-to-set binding, not a per-vCPU assignment. All vCPUs of a guest vNUMA node share the same host CPU pool. If you need each vCPU pinned to its own distinct core, use one-to-one or manual instead. Example: `3;4` lets all vCPUs on guest vNUMA 0 run on any CPU of host NUMA node 3, and all vCPUs on guest vNUMA 1 run on any CPU of host NUMA node 4. An empty string uses host NUMA nodes in ascending index order: guest vNUMA 0 → host node 0, guest vNUMA 1 → host node 1, etc. * numa-manual: explicit variant of numa. The config string is a `;`-separated list of `:` pairs. Each guest vNUMA node's set of vCPUs is bound to the set of CPUs of the specified host NUMA node. Semantics are identical to numa — vCPUs of a guest vNUMA node may be scheduled on any CPU of the assigned host node, but cannot leave it. Only the node mapping is explicit here. Example: `0:2;1:3` binds all vCPUs on guest vNUMA 0 to host NUMA node 2's CPUs, and all vCPUs on guest vNUMA 1 to host NUMA node 3's CPUs. Configuration is via two new VM config keys, cpu-binding-mode and cpu-binding-cfg. Host CPU and NUMA indices are OS indices (PU P#N from lstopo -p). 2. Prerequisites ---------------- The series is based on Dominik's PCI series [0]. This patch adds a new build dependency: libhwloc-dev. It is declared in pve-qemu-server-rs/pve-qemu-server-cpu/debian/ debcargo.toml. 3. Request for feedback ----------------------- I am looking for feedback, especially on: * whether adding hwloc as a dependency is acceptable for qemu-server. * whether binding belongs in a dedicated crate or should live closer to qemu-server. * the config schema and the chosen mode split. * naming. * general design decisions (e.g. if rust is the right choice here) 4. Why hwloc ------------ hwloc [4] is widely used, well maintained, small and already available on Debian (ver 2.12.0-4 at the time of writing this). It abstracts hardware topology discovery across architectures and CPU generations, including SMT, cache hierarchy, NUMA nodes, and CPU kinds. Without hwloc we would parse topology ourselves, from sysfs or from arch-specific interfaces. Every new CPU family would become our maintenance burden, and parser bugs would surface only as binding failures on customer hardware. hwloc also unlocks further performance work requiring topology data: * NUMA-aware memory placement (bind memory to a node) * cache-aware vCPU placement (group by shared L2/L3) * heterogeneous CPU support (Intel P/E cores) via CPU kinds * distance-aware placement (latency/bandwidth between nodes) The hwloc FAQ has a section for exactly this question: "I only need binding, or the number of cores, why should I use hwloc?" [1]. 5. Why not hwlocality --------------------- hwlocality [2,3] is the idiomatic high-level Rust binding to hwloc. It is still very new (1.0.0-alpha) and not packaged in Debian. Packaging it would require packaging its full transitive dependency tree, several of which are also not yet in Debian. That scope is too much for this feature IMO. Our FFI bindings are a minimal hand-rolled subset of hwloc's public C API, matching what hwlocality-sys exposes (I have literally taken the code from hwlocality-sys [5]). Migration to hwlocality later is straightforward: swap the bindings module for the crate and delete the wrapper types. 6. Affinity settings -------------------- The existing 'affinity' option is not integrated with this feature yet. IMO these two are conceptually different: affinity in PVE is a list of allowed CPUs; binding is a per-vCPU placement. A single flat list cannot express per-vCPU intent. Also, the affinity code lives purely in perl. For now this binding code is experimental and should be used without 'affinity'. Running taskset alongside the Rust binder would race on the same threads and is hard to reason about. Long-term I would like to move affinity handling into Rust as well, so both features share topology-based validation and a common binding backend. For this patch, shipping binding independently and harmonizing with affinity later seems cleanest. 7. Reproducibility ------------------ The original RFC [6] relied on 'affinity' to limit the CPU set and kept the vCPU-to-CPU assignment in a Perl hash. Two configs that looked identical could produce different placement, depending on hash ordering and how affinity was set. To make placement reproducible from the config file alone, the binding logic lives entirely in Rust and the assignment is read from cpu-binding-cfg. Two VMs with the same config on the same host should get the same placement. (unless there is some randomness in the hwloc_bitmap_singlify_per_core function that i am unaware of) 8. Basic usage -------------- # Explicit per-vCPU binding qm set --cpu-binding-mode manual qm set --cpu-binding-cfg "0:0;1:1;2:2;3:3" # One physical core per vCPU, let hwloc place them qm set --cpu-binding-mode one-to-one qm set --cpu-binding-cfg "0-3" # Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3 qm set --cpu-binding-mode numa-manual qm set --cpu-binding-cfg "0:2;1:3" # Automatic numa binding: #Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3 qm set --cpu-binding-mode numa qm set --cpu-binding-cfg "2-3" Without cpu-binding-mode the VM behaves as before. Validation runs at qm set time and at VM start; binding itself runs after QEMU spawns the vCPU threads. 9. What is not in this PATCH ---------------------------- * memory_binding: IMO this is best handled by qemu directly and already possible via, e.g.: numa0: cpus=0-1,hostnodes=0,memory=2048,policy=bind numa1: cpus=2-3,hostnodes=1,memory=2048,policy=bind numa2: cpus=4-5,hostnodes=2,memory=2048,policy=bind numa3: cpus=6-7,hostnodes=3,memory=2048,policy=bind What we need is a convinient way to automatically apply the correct memory binding for a given CPU Binding config. * Qemu I/O Thread control/binding * CPU Isolation I plan on adding most of the features requested in the forum post [7]. 10. Previous versions --------------------- v2: https://lore.proxmox.com/pve-devel/20260921095404.61552-1-e.huhsovitz@proxmox.com/ v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ (Credits to Dominik) 11. References ------------- [0] https://lore.proxmox.com/pve-devel/20261015081041.743422-1-d.csapak@proxmox.com [1] https://hwloc.readthedocs.io/en/hwloc-2.15.0/doxygen/html/faq.html [2] https://crates.io/crates/hwlocality [3] https://github.com/HadrienG2/hwlocality [4] https://github.com/open-mpi/hwloc [5] https://github.com/HadrienG2/hwlocality/tree/main/hwlocality-sys [6] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ [7] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/ Thank you to whoever reviews this! :) Summary of changes ------------- pve-qemu-server-rs: Elias Huhsovitz (1): cpu: add pve-qemu-server-cpu crate for vCPU binding Cargo.toml | 2 +- build.sh | 2 +- pve-qemu-server-cpu/Cargo.toml | 16 + pve-qemu-server-cpu/debian/changelog | 5 + pve-qemu-server-cpu/debian/control | 41 ++ pve-qemu-server-cpu/debian/copyright | 18 + pve-qemu-server-cpu/debian/debcargo.toml | 9 + pve-qemu-server-cpu/src/bind.rs | 478 ++++++++++++++++++++++ pve-qemu-server-cpu/src/error.rs | 38 ++ pve-qemu-server-cpu/src/hwloc/bindings.rs | 171 ++++++++ pve-qemu-server-cpu/src/hwloc/mod.rs | 222 ++++++++++ pve-qemu-server-cpu/src/lib.rs | 92 +++++ pve-qemu-server-cpu/src/qmp_cpu.rs | 99 +++++ 13 files changed, 1191 insertions(+), 2 deletions(-) create mode 100644 pve-qemu-server-cpu/Cargo.toml create mode 100644 pve-qemu-server-cpu/debian/changelog create mode 100644 pve-qemu-server-cpu/debian/control create mode 100644 pve-qemu-server-cpu/debian/copyright create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml create mode 100644 pve-qemu-server-cpu/src/bind.rs create mode 100644 pve-qemu-server-cpu/src/error.rs create mode 100644 pve-qemu-server-cpu/src/hwloc/bindings.rs create mode 100644 pve-qemu-server-cpu/src/hwloc/mod.rs create mode 100644 pve-qemu-server-cpu/src/lib.rs create mode 100644 pve-qemu-server-cpu/src/qmp_cpu.rs proxmox-perl-rs: Elias Huhsovitz (1): pve: cpu: add PVE::RS::CPU::Bind perl bindings pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 1 + pve-rs/src/bindings/cpu/bind.rs | 40 +++++++++++++++++++++++++++++++++ pve-rs/src/bindings/cpu/mod.rs | 2 ++ pve-rs/src/bindings/mod.rs | 3 +++ 5 files changed, 47 insertions(+) create mode 100644 pve-rs/src/bindings/cpu/bind.rs create mode 100644 pve-rs/src/bindings/cpu/mod.rs qemu-server: Elias Huhsovitz (1): cpu: add CPU binding src/PVE/QemuServer.pm | 36 +++++++++ src/PVE/QemuServer/CPUBinding.pm | 133 +++++++++++++++++++++++++++++++ src/PVE/QemuServer/Makefile | 1 + 3 files changed, 170 insertions(+) create mode 100644 src/PVE/QemuServer/CPUBinding.pm Summary over all repositories: 21 files changed, 1408 insertions(+), 2 deletions(-) -- Generated by murpp 0.12.0