[PATCH proxmox-perl-rs/qemu-server/qemu-server-rs v3 0/3] vCPU binding for qemu guests

[Elias Huhsovitz <e.huhsovitz@proxmox.com>]

1. Overview
-----------
This series adds a convinient way to achive vCPU:CPU binding in PVE.
A new Rust crate, pve-qemu-server-cpu, exposes binding and validation via
PVE::RS::CPU::Bind, called from the VM start path after QEMU has
spawned its vCPU threads.

Five modes are added:

* none (default): no binding.

* manual: explicit vCPU-to-hostCPU mapping. The config string is a
  semicolon-separated list of `<vcpu-set>:<hostcpu-set>` entries.
  Example: `0:0;1:1;2:2;3:3` pins vCPU 0 to CPU 0, vCPU 1 to CPU 1,
  etc. Multi-value sets use `a-b` ranges or `{a,b,c}` unions.

* one-to-one: pin each vCPU to its own physical core, chosen from a
  user-supplied pool. The config string is an upper bound: the list of
  host cores the vCPUs may be assigned to. SMT siblings are stripped
  from that pool before assignment, so each vCPU gets its own distinct
  physical core, not a hyperthread.

  If the pool has N cores and the VM has M vCPUs, each vCPU is pinned
  to exactly one of the N cores (N must be >= M, otherwise validation
  fails). For example, with `0-3` on a 4-vCPU VM, vCPU 0 gets core 0,
  vCPU 1 gets core 1, and so on. Each vCPU is fixed to a single core,
  it does not float within the pool. With `0;1;4;5` on the same VM,
  vCPU 2 is pinned to core 4 and vCPU 3 to core 5, skipping cores 2-3.

  An empty string means "use all physical cores on the host" as the
  upper bound. On a host with 16 cores, that pins an N-vCPU VM to
  cores 0, 1, ..., N-1 — one core per vCPU, SMT stripped.

  Unlike `manual`, you do not specify which core each vCPU gets; that
  assignment is up to hwloc. Use `manual` if you need a specific
  vCPU-to-core mapping.

* numa: bind the set of vCPUs of each guest vNUMA node to the set of
  CPUs of a host NUMA node. The config string is a `;`-separated list
  of host NUMA node indices, consumed in order: guest vNUMA 0 gets the
  first host node, guest vNUMA 1 the second, and so on.

  Each guest vNUMA node's vCPUs form a group that may run on any CPU
  of the assigned host NUMA node — including its cores and SMT threads.
  The host scheduler is free to migrate vCPUs within the node, but not
  outside it. This is the intended trade-off: memory allocated by a
  vCPU stays local to the host node it runs on (preserving NUMA
  memory locality), while the scheduler still has room to balance
  load across the node's cores and hyperthreads.

  This is a set-to-set binding, not a per-vCPU assignment. All vCPUs
  of a guest vNUMA node share the same host CPU pool. If you need
  each vCPU pinned to its own distinct core, use one-to-one or manual
  instead.

  Example: `3;4` lets all vCPUs on guest vNUMA 0 run on any CPU of
  host NUMA node 3, and all vCPUs on guest vNUMA 1 run on any CPU of
  host NUMA node 4.

  An empty string uses host NUMA nodes in ascending index order:
  guest vNUMA 0 → host node 0, guest vNUMA 1 → host node 1, etc.

* numa-manual: explicit variant of numa. The config string is a
  `;`-separated list of `<guest-vnuma>:<host-numa>` pairs. Each guest
  vNUMA node's set of vCPUs is bound to the set of CPUs of the
  specified host NUMA node.

  Semantics are identical to numa — vCPUs of a guest vNUMA node may
  be scheduled on any CPU of the assigned host node, but cannot leave
  it. Only the node mapping is explicit here.

  Example: `0:2;1:3` binds all vCPUs on guest vNUMA 0 to host NUMA
  node 2's CPUs, and all vCPUs on guest vNUMA 1 to host NUMA node 3's
  CPUs.

Configuration is via two new VM config keys, cpu-binding-mode
and cpu-binding-cfg. Host CPU and NUMA indices are OS indices
(PU P#N from lstopo -p).

2. Prerequisites
----------------
The series is based on Dominik's PCI series [0].

This patch adds a new build dependency: libhwloc-dev. It is
declared in pve-qemu-server-rs/pve-qemu-server-cpu/debian/
debcargo.toml.

3. Request for feedback
-----------------------
I am looking for feedback, especially on:

* whether adding hwloc as a dependency is acceptable for
  qemu-server.
* whether binding belongs in a dedicated crate or should live
  closer to qemu-server.
* the config schema and the chosen mode split.
* naming.
* general design decisions (e.g. if rust is the right choice here)

4. Why hwloc
------------
hwloc [4] is widely used, well maintained, small and already available
on Debian (ver 2.12.0-4 at the time of writing this).

It abstracts hardware topology discovery across architectures and
CPU generations, including SMT, cache hierarchy, NUMA nodes, and
CPU kinds.

Without hwloc we would parse topology ourselves, from sysfs or
from arch-specific interfaces. Every new CPU family would
become our maintenance burden, and parser bugs would surface
only as binding failures on customer hardware.

hwloc also unlocks further performance work requiring topology
data:

* NUMA-aware memory placement (bind memory to a node)
* cache-aware vCPU placement (group by shared L2/L3)
* heterogeneous CPU support (Intel P/E cores) via CPU kinds
* distance-aware placement (latency/bandwidth between nodes)

The hwloc FAQ has a section for exactly this question:
"I only need binding, or the number of cores, why should I use
hwloc?" [1].

5. Why not hwlocality
---------------------
hwlocality [2,3] is the idiomatic high-level Rust binding to
hwloc. It is still very new (1.0.0-alpha) and not packaged in
Debian. Packaging it would require packaging its full
transitive dependency tree, several of which are also not yet
in Debian. That scope is too much for this feature IMO.

Our FFI bindings are a minimal hand-rolled subset of hwloc's
public C API, matching what hwlocality-sys exposes (I have
literally taken the code from hwlocality-sys [5]). Migration
to hwlocality later is straightforward: swap the bindings
module for the crate and delete the wrapper types.

6. Affinity settings
--------------------
The existing 'affinity' option is not integrated with this
feature yet. IMO these two are conceptually different: affinity
in PVE is a list of allowed CPUs; binding is a per-vCPU placement.
A single flat list cannot express per-vCPU intent. Also, the affinity
code lives purely in perl.

For now this binding code is experimental and should be used
without 'affinity'. Running taskset alongside the Rust binder
would race on the same threads and is hard to reason about.

Long-term I would like to move affinity handling into Rust as
well, so both features share topology-based validation and a
common binding backend. For this patch, shipping binding
independently and harmonizing with affinity later seems
cleanest.

7. Reproducibility
------------------
The original RFC [6] relied on 'affinity' to limit the CPU set and
kept the vCPU-to-CPU assignment in a Perl hash. Two configs
that looked identical could produce different placement,
depending on hash ordering and how affinity was set.

To make placement reproducible from the config file alone, the
binding logic lives entirely in Rust and the assignment is
read from cpu-binding-cfg. Two VMs with the same config on the same host
should get the same placement. (unless there is some randomness in the
hwloc_bitmap_singlify_per_core function that i am unaware of)

8. Basic usage
--------------
# Explicit per-vCPU binding
qm set <vmid> --cpu-binding-mode manual
qm set <vmid> --cpu-binding-cfg "0:0;1:1;2:2;3:3"

# One physical core per vCPU, let hwloc place them
qm set <vmid> --cpu-binding-mode one-to-one
qm set <vmid> --cpu-binding-cfg "0-3"

# Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3
qm set <vmid> --cpu-binding-mode numa-manual
qm set <vmid> --cpu-binding-cfg "0:2;1:3"

# Automatic numa binding:
#Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3
qm set <vmid> --cpu-binding-mode numa
qm set <vmid> --cpu-binding-cfg "2-3"

Without cpu-binding-mode the VM behaves as before. Validation
runs at qm set time and at VM start; binding itself runs after
QEMU spawns the vCPU threads.

9. What is not in this PATCH
----------------------------
* memory_binding: IMO this is best handled by qemu directly and already
possible via, e.g.:

numa0: cpus=0-1,hostnodes=0,memory=2048,policy=bind
numa1: cpus=2-3,hostnodes=1,memory=2048,policy=bind
numa2: cpus=4-5,hostnodes=2,memory=2048,policy=bind
numa3: cpus=6-7,hostnodes=3,memory=2048,policy=bind

What we need is a convinient way to automatically apply the correct
memory binding for a given CPU Binding config.

* Qemu I/O Thread control/binding

* CPU Isolation

I plan on adding most of the features requested in the forum post [7].

10. Previous versions
---------------------
v2: https://lore.proxmox.com/pve-devel/20260921095404.61552-1-e.huhsovitz@proxmox.com/
v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
(Credits to Dominik)

11. References
-------------
[0] https://lore.proxmox.com/pve-devel/20261015081041.743422-1-d.csapak@proxmox.com
[1] https://hwloc.readthedocs.io/en/hwloc-2.15.0/doxygen/html/faq.html
[2] https://crates.io/crates/hwlocality
[3] https://github.com/HadrienG2/hwlocality
[4] https://github.com/open-mpi/hwloc
[5] https://github.com/HadrienG2/hwlocality/tree/main/hwlocality-sys
[6] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
[7] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/

Thank you to whoever reviews this! :)

Summary of changes
-------------
pve-qemu-server-rs:

Elias Huhsovitz (1):
  cpu: add pve-qemu-server-cpu crate for vCPU binding

 Cargo.toml                                |   2 +-
 build.sh                                  |   2 +-
 pve-qemu-server-cpu/Cargo.toml            |  16 +
 pve-qemu-server-cpu/debian/changelog      |   5 +
 pve-qemu-server-cpu/debian/control        |  41 ++
 pve-qemu-server-cpu/debian/copyright      |  18 +
 pve-qemu-server-cpu/debian/debcargo.toml  |   9 +
 pve-qemu-server-cpu/src/bind.rs           | 478 ++++++++++++++++++++++
 pve-qemu-server-cpu/src/error.rs          |  38 ++
 pve-qemu-server-cpu/src/hwloc/bindings.rs | 171 ++++++++
 pve-qemu-server-cpu/src/hwloc/mod.rs      | 222 ++++++++++
 pve-qemu-server-cpu/src/lib.rs            |  92 +++++
 pve-qemu-server-cpu/src/qmp_cpu.rs        |  99 +++++
 13 files changed, 1191 insertions(+), 2 deletions(-)
 create mode 100644 pve-qemu-server-cpu/Cargo.toml
 create mode 100644 pve-qemu-server-cpu/debian/changelog
 create mode 100644 pve-qemu-server-cpu/debian/control
 create mode 100644 pve-qemu-server-cpu/debian/copyright
 create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml
 create mode 100644 pve-qemu-server-cpu/src/bind.rs
 create mode 100644 pve-qemu-server-cpu/src/error.rs
 create mode 100644 pve-qemu-server-cpu/src/hwloc/bindings.rs
 create mode 100644 pve-qemu-server-cpu/src/hwloc/mod.rs
 create mode 100644 pve-qemu-server-cpu/src/lib.rs
 create mode 100644 pve-qemu-server-cpu/src/qmp_cpu.rs


proxmox-perl-rs:

Elias Huhsovitz (1):
  pve: cpu: add PVE::RS::CPU::Bind perl bindings

 pve-rs/Cargo.toml               |  1 +
 pve-rs/Makefile                 |  1 +
 pve-rs/src/bindings/cpu/bind.rs | 40 +++++++++++++++++++++++++++++++++
 pve-rs/src/bindings/cpu/mod.rs  |  2 ++
 pve-rs/src/bindings/mod.rs      |  3 +++
 5 files changed, 47 insertions(+)
 create mode 100644 pve-rs/src/bindings/cpu/bind.rs
 create mode 100644 pve-rs/src/bindings/cpu/mod.rs


qemu-server:

Elias Huhsovitz (1):
  cpu: add CPU binding

 src/PVE/QemuServer.pm            |  36 +++++++++
 src/PVE/QemuServer/CPUBinding.pm | 133 +++++++++++++++++++++++++++++++
 src/PVE/QemuServer/Makefile      |   1 +
 3 files changed, 170 insertions(+)
 create mode 100644 src/PVE/QemuServer/CPUBinding.pm


Summary over all repositories:
  21 files changed, 1408 insertions(+), 2 deletions(-)

-- 
Generated by murpp 0.12.0