[RFC proxmox-perl-rs/qemu-server/qemu-server-rs v4 0/4] vCPU binding for qemu guests

[Elias Huhsovitz <e.huhsovitz@proxmox.com>]

1. Overview
-----------
This series adds a more convinient way to achive vCPU:CPU binding in PVE.
A new Rust crate, pve-qemu-server-cpu, exposes binding and validation via
PVE::RS::CPU::Bind, called from the VM start path after QEMU has
spawned its vCPU threads.

2 functional modes are added:

* one-to-one: pin each vCPU to its own host cpu, chosen from the
  pool configured by affinity. The configured affinity string acts as
  an upper bound on the host cpus, that can be used for binding.

  If the guest has N vCPUs and the affinity-pool has M host cpus, each vCPU
  is fixed to a single one of the M host cpus. (M >= N, must be true).

  Each vCPU is fixed to a single host cpu, it does not float within
  the pool.

  Example 1: 4-vCPU guest with affinity=4-9:
  The resulting mapping would be: vCPU0:CPU4, vCPU1:CPU5, vCPU2:CPU6,
  vCPU3:CPU7.

  Example 2: 4-vCPU guest with affinity=0,2:
  The resulting mapping would be: vCPU0:CPU0, vCPU1:CPU2

  An empty affinity string means all host cpus will be considered.
  On a host with 16 cores, that pins an N-vCPU VM to host cpus
  0, 1, ..., N-1.

* numa: bind the set of vCPUs of each vNUMA node to the set of
  CPUs of a host NUMA node. The cpu bindings are determined by the
  numa[N] configuration of the guest.

  Each guest vNUMA node's vCPUs form a group that may run on any CPU
  of the assigned host NUMA node. The host scheduler is free to migrate
  the vCPUs within the host node. This is the intended trade-off:
  memory allocated by a vCPU stays local to the host node it runs on
  (preserving NUMA memory locality), while the scheduler still has room
  to balance load across the node's cores and hyperthreads.

  Examples:
  numa0: cpus=0-1,hostnodes=0,memory=2048,policy=bind
  Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0

  numa1: cpus=6-7,hostnodes=3,memory=2048,policy=bind
  Allows vCPUs 6-7 to execute on all cpus of host NUMA node 3

  no numa[N]: sockets: 2, cores: 2, total vCPUs: 4
  Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0
  Allows vCPUs 2-3 to execute on all cpus of host NUMA node 1

Both settings respect the cpu affinity.
The mode can be configured via na VM config key `cpu-binding-mode`.

2. Prerequisites
----------------
The new crate `pve-qemu-server-cpu` and patches for `proxmox-perl-rs`
is based on the patches introduced by Dominik's PCI series [0].

The qemu-server is applied on top of the current master, since I had
trouble applying [0].

3. Reproducibility
------------------
To make vCPU placement reproduceable, the config is deterministically
generated using the 3 configuration keys:
- affinity
- cpu-binding-mode
- numa[N] config keys

Two VMs with the same config on the same host
should get the same placement. (unless there is some randomness that I
am unaware of)

4. Basic usage
--------------
# One vCPU executes on exactly one host cpu
qm set <vmid> --cpu-binding-mode one-to-one
qm set <vmid> --affinity 0-3

# NUMA binding (explicit):
#Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3
qm set <vmid> --numa 1
qm set <vmid> --cpu-binding-mode numa
qm set <vmid> --numa0 cpus=0-1,hostnodes=0,memory=1024,policy=bind
qm set <vmid> --numa1 cpus=2-3,hostnodes=1,memory=1024,policy=bind

# NUMA binding (implicit):
# for each socket: one numa node is created and bound
# e.g., 2 sockets: numa0:host-numa0, numa1:host-numa:1
qm set <vmid> --numa 1

Validation runs at VM start. Binding itself runs after QEMU spawns the
vCPU threads (The VM is frozen at start, until the affinity of the
vCPUs has been set)

5. Changes v3->v4
-----------------* Drop the manual modes (manual, numa-manual).
* All modes respect the affnity setting.
* numa mode utilizes the numa[N] config.
* Freeze guest until sched_setaffinity has finished to avoid using
non-specified host cpus during startup sequence.
* Build cpu toplogy from sysfs instead of hwloc [4].


6. What is not in this PATCH
----------------------------
* memory_binding: This is best handled by qemu directly and already
possible via, e.g.:

numa0: cpus=0-1,hostnodes=1,memory=2048,policy=bind
numa1: cpus=2-3,hostnodes=2,memory=2048,policy=bind
numa2: cpus=4-5,hostnodes=3,memory=2048,policy=bind
numa3: cpus=6-7,hostnodes=4,memory=2048,policy=bind

What we need is a convinient way to automatically apply the correct
memory bindingings for the numa node. This is will be handled in an
upcoming adjacent patch.

* Qemu I/O Thread control/binding

* CPU Isolation

I plan on adding most of the features requested in the forum post [3].

7. Previous versions
---------------------
v3: https://lore.proxmox.com/pve-devel/20261022092227.58871-1-e.huhsovitz@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20260921095404.61552-1-e.huhsovitz@proxmox.com/
v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/

v1 was originally introduced by Dominik in [1].

8. References
-------------
[0] https://lore.proxmox.com/pve-devel/20261106132604.2556840-1-d.csapak@proxmox.com/
[1] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/
[2] https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html
[3] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/
[4] https://hwloc.readthedocs.io/en/hwloc-2.15.0/doxygen/html/index.html

Thank you to whoever reviews this! :)

Summary of changes
------------------

pve-qemu-server-rs:

Elias Huhsovitz (1):
  cpu: add pve-qemu-server-cpu crate for vCPU binding

 Cargo.toml                                |   2 +-
 build.sh                                  |   2 +-
 pve-qemu-server-cpu/Cargo.toml            |  16 ++
 pve-qemu-server-cpu/debian/changelog      |   5 +
 pve-qemu-server-cpu/debian/control        |  41 ++++
 pve-qemu-server-cpu/debian/copyright      |  18 ++
 pve-qemu-server-cpu/debian/debcargo.toml  |   9 +
 pve-qemu-server-cpu/src/bind.rs           | 136 +++++++++++++
 pve-qemu-server-cpu/src/config.rs         | 146 ++++++++++++++
 pve-qemu-server-cpu/src/cpuset.rs         | 125 ++++++++++++
 pve-qemu-server-cpu/src/error.rs          |  32 ++++
 pve-qemu-server-cpu/src/lib.rs            |  92 +++++++++
 pve-qemu-server-cpu/src/qmp.rs            |  91 +++++++++
 pve-qemu-server-cpu/src/syscall.rs        |  24 +++
 pve-qemu-server-cpu/src/topology/mod.rs   | 222 ++++++++++++++++++++++
 pve-qemu-server-cpu/src/topology/sysfs.rs |   6 +
 16 files changed, 965 insertions(+), 2 deletions(-)
 create mode 100644 pve-qemu-server-cpu/Cargo.toml
 create mode 100644 pve-qemu-server-cpu/debian/changelog
 create mode 100644 pve-qemu-server-cpu/debian/control
 create mode 100644 pve-qemu-server-cpu/debian/copyright
 create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml
 create mode 100644 pve-qemu-server-cpu/src/bind.rs
 create mode 100644 pve-qemu-server-cpu/src/config.rs
 create mode 100644 pve-qemu-server-cpu/src/cpuset.rs
 create mode 100644 pve-qemu-server-cpu/src/error.rs
 create mode 100644 pve-qemu-server-cpu/src/lib.rs
 create mode 100644 pve-qemu-server-cpu/src/qmp.rs
 create mode 100644 pve-qemu-server-cpu/src/syscall.rs
 create mode 100644 pve-qemu-server-cpu/src/topology/mod.rs
 create mode 100644 pve-qemu-server-cpu/src/topology/sysfs.rs


proxmox-perl-rs:

Elias Huhsovitz (1):
  pve: cpu: add binding for pve-qemu-server-cpu crate

 pve-rs/Cargo.toml               |  1 +
 pve-rs/Makefile                 |  1 +
 pve-rs/src/bindings/cpu/bind.rs | 42 +++++++++++++++++++++++++++++++++
 pve-rs/src/bindings/cpu/mod.rs  |  2 ++
 pve-rs/src/bindings/mod.rs      |  3 +++
 5 files changed, 49 insertions(+)
 create mode 100644 pve-rs/src/bindings/cpu/bind.rs
 create mode 100644 pve-rs/src/bindings/cpu/mod.rs


qemu-server:

Elias Huhsovitz (2):
  helpers: return integer values for parse_number_sets
  cpu: add cpu binding

 src/PVE/QemuServer.pm         |  32 ++++++++++-
 src/PVE/QemuServer/CPUBind.pm | 103 ++++++++++++++++++++++++++++++++++
 src/PVE/QemuServer/Helpers.pm |   6 +-
 src/PVE/QemuServer/Makefile   |   1 +
 src/PVE/QemuServer/Memory.pm  |  52 ++++++++++++++++-
 5 files changed, 191 insertions(+), 3 deletions(-)
 create mode 100644 src/PVE/QemuServer/CPUBind.pm


Summary over all repositories:
  26 files changed, 1205 insertions(+), 5 deletions(-)

-- 
Generated by murpp 0.12.0