From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9F2AA1FF0B3 for ; Wed, 11 Nov 2026 16:21:36 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A3A7D216D3; Wed, 11 Nov 2026 16:21:08 +0100 (CET) From: Elias Huhsovitz To: pve-devel@lists.proxmox.com Subject: [RFC proxmox-perl-rs/qemu-server/qemu-server-rs v4 0/4] vCPU binding for qemu guests Date: Wed, 11 Nov 2026 16:20:23 +0100 Message-ID: <20261111152027.260117-1-e.huhsovitz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794410459624 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.426 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YJPLYKHRCIRYZDRDQMRYPHUGHIN7LJ2K X-Message-ID-Hash: YJPLYKHRCIRYZDRDQMRYPHUGHIN7LJ2K X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Elias Huhsovitz X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: 1. Overview ----------- This series adds a more convinient way to achive vCPU:CPU binding in PVE. A new Rust crate, pve-qemu-server-cpu, exposes binding and validation via PVE::RS::CPU::Bind, called from the VM start path after QEMU has spawned its vCPU threads. 2 functional modes are added: * one-to-one: pin each vCPU to its own host cpu, chosen from the pool configured by affinity. The configured affinity string acts as an upper bound on the host cpus, that can be used for binding. If the guest has N vCPUs and the affinity-pool has M host cpus, each vCPU is fixed to a single one of the M host cpus. (M >= N, must be true). Each vCPU is fixed to a single host cpu, it does not float within the pool. Example 1: 4-vCPU guest with affinity=4-9: The resulting mapping would be: vCPU0:CPU4, vCPU1:CPU5, vCPU2:CPU6, vCPU3:CPU7. Example 2: 4-vCPU guest with affinity=0,2: The resulting mapping would be: vCPU0:CPU0, vCPU1:CPU2 An empty affinity string means all host cpus will be considered. On a host with 16 cores, that pins an N-vCPU VM to host cpus 0, 1, ..., N-1. * numa: bind the set of vCPUs of each vNUMA node to the set of CPUs of a host NUMA node. The cpu bindings are determined by the numa[N] configuration of the guest. Each guest vNUMA node's vCPUs form a group that may run on any CPU of the assigned host NUMA node. The host scheduler is free to migrate the vCPUs within the host node. This is the intended trade-off: memory allocated by a vCPU stays local to the host node it runs on (preserving NUMA memory locality), while the scheduler still has room to balance load across the node's cores and hyperthreads. Examples: numa0: cpus=0-1,hostnodes=0,memory=2048,policy=bind Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0 numa1: cpus=6-7,hostnodes=3,memory=2048,policy=bind Allows vCPUs 6-7 to execute on all cpus of host NUMA node 3 no numa[N]: sockets: 2, cores: 2, total vCPUs: 4 Allows vCPUs 0-1 to execute on all cpus of host NUMA node 0 Allows vCPUs 2-3 to execute on all cpus of host NUMA node 1 Both settings respect the cpu affinity. The mode can be configured via na VM config key `cpu-binding-mode`. 2. Prerequisites ---------------- The new crate `pve-qemu-server-cpu` and patches for `proxmox-perl-rs` is based on the patches introduced by Dominik's PCI series [0]. The qemu-server is applied on top of the current master, since I had trouble applying [0]. 3. Reproducibility ------------------ To make vCPU placement reproduceable, the config is deterministically generated using the 3 configuration keys: - affinity - cpu-binding-mode - numa[N] config keys Two VMs with the same config on the same host should get the same placement. (unless there is some randomness that I am unaware of) 4. Basic usage -------------- # One vCPU executes on exactly one host cpu qm set --cpu-binding-mode one-to-one qm set --affinity 0-3 # NUMA binding (explicit): #Guest NUMA 0 -> host NUMA 2, guest NUMA 1 -> host NUMA 3 qm set --numa 1 qm set --cpu-binding-mode numa qm set --numa0 cpus=0-1,hostnodes=0,memory=1024,policy=bind qm set --numa1 cpus=2-3,hostnodes=1,memory=1024,policy=bind # NUMA binding (implicit): # for each socket: one numa node is created and bound # e.g., 2 sockets: numa0:host-numa0, numa1:host-numa:1 qm set --numa 1 Validation runs at VM start. Binding itself runs after QEMU spawns the vCPU threads (The VM is frozen at start, until the affinity of the vCPUs has been set) 5. Changes v3->v4 -----------------* Drop the manual modes (manual, numa-manual). * All modes respect the affnity setting. * numa mode utilizes the numa[N] config. * Freeze guest until sched_setaffinity has finished to avoid using non-specified host cpus during startup sequence. * Build cpu toplogy from sysfs instead of hwloc [4]. 6. What is not in this PATCH ---------------------------- * memory_binding: This is best handled by qemu directly and already possible via, e.g.: numa0: cpus=0-1,hostnodes=1,memory=2048,policy=bind numa1: cpus=2-3,hostnodes=2,memory=2048,policy=bind numa2: cpus=4-5,hostnodes=3,memory=2048,policy=bind numa3: cpus=6-7,hostnodes=4,memory=2048,policy=bind What we need is a convinient way to automatically apply the correct memory bindingings for the numa node. This is will be handled in an upcoming adjacent patch. * Qemu I/O Thread control/binding * CPU Isolation I plan on adding most of the features requested in the forum post [3]. 7. Previous versions --------------------- v3: https://lore.proxmox.com/pve-devel/20261022092227.58871-1-e.huhsovitz@proxmox.com/ v2: https://lore.proxmox.com/pve-devel/20260921095404.61552-1-e.huhsovitz@proxmox.com/ v1: https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ v1 was originally introduced by Dominik in [1]. 8. References ------------- [0] https://lore.proxmox.com/pve-devel/20261106132604.2556840-1-d.csapak@proxmox.com/ [1] https://lore.proxmox.com/pve-devel/20260217114813.2063770-1-d.csapak@proxmox.com/ [2] https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html [3] https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/ [4] https://hwloc.readthedocs.io/en/hwloc-2.15.0/doxygen/html/index.html Thank you to whoever reviews this! :) Summary of changes ------------------ pve-qemu-server-rs: Elias Huhsovitz (1): cpu: add pve-qemu-server-cpu crate for vCPU binding Cargo.toml | 2 +- build.sh | 2 +- pve-qemu-server-cpu/Cargo.toml | 16 ++ pve-qemu-server-cpu/debian/changelog | 5 + pve-qemu-server-cpu/debian/control | 41 ++++ pve-qemu-server-cpu/debian/copyright | 18 ++ pve-qemu-server-cpu/debian/debcargo.toml | 9 + pve-qemu-server-cpu/src/bind.rs | 136 +++++++++++++ pve-qemu-server-cpu/src/config.rs | 146 ++++++++++++++ pve-qemu-server-cpu/src/cpuset.rs | 125 ++++++++++++ pve-qemu-server-cpu/src/error.rs | 32 ++++ pve-qemu-server-cpu/src/lib.rs | 92 +++++++++ pve-qemu-server-cpu/src/qmp.rs | 91 +++++++++ pve-qemu-server-cpu/src/syscall.rs | 24 +++ pve-qemu-server-cpu/src/topology/mod.rs | 222 ++++++++++++++++++++++ pve-qemu-server-cpu/src/topology/sysfs.rs | 6 + 16 files changed, 965 insertions(+), 2 deletions(-) create mode 100644 pve-qemu-server-cpu/Cargo.toml create mode 100644 pve-qemu-server-cpu/debian/changelog create mode 100644 pve-qemu-server-cpu/debian/control create mode 100644 pve-qemu-server-cpu/debian/copyright create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml create mode 100644 pve-qemu-server-cpu/src/bind.rs create mode 100644 pve-qemu-server-cpu/src/config.rs create mode 100644 pve-qemu-server-cpu/src/cpuset.rs create mode 100644 pve-qemu-server-cpu/src/error.rs create mode 100644 pve-qemu-server-cpu/src/lib.rs create mode 100644 pve-qemu-server-cpu/src/qmp.rs create mode 100644 pve-qemu-server-cpu/src/syscall.rs create mode 100644 pve-qemu-server-cpu/src/topology/mod.rs create mode 100644 pve-qemu-server-cpu/src/topology/sysfs.rs proxmox-perl-rs: Elias Huhsovitz (1): pve: cpu: add binding for pve-qemu-server-cpu crate pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 1 + pve-rs/src/bindings/cpu/bind.rs | 42 +++++++++++++++++++++++++++++++++ pve-rs/src/bindings/cpu/mod.rs | 2 ++ pve-rs/src/bindings/mod.rs | 3 +++ 5 files changed, 49 insertions(+) create mode 100644 pve-rs/src/bindings/cpu/bind.rs create mode 100644 pve-rs/src/bindings/cpu/mod.rs qemu-server: Elias Huhsovitz (2): helpers: return integer values for parse_number_sets cpu: add cpu binding src/PVE/QemuServer.pm | 32 ++++++++++- src/PVE/QemuServer/CPUBind.pm | 103 ++++++++++++++++++++++++++++++++++ src/PVE/QemuServer/Helpers.pm | 6 +- src/PVE/QemuServer/Makefile | 1 + src/PVE/QemuServer/Memory.pm | 52 ++++++++++++++++- 5 files changed, 191 insertions(+), 3 deletions(-) create mode 100644 src/PVE/QemuServer/CPUBind.pm Summary over all repositories: 26 files changed, 1205 insertions(+), 5 deletions(-) -- Generated by murpp 0.12.0