all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Elias Huhsovitz <e.huhsovitz@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Elias Huhsovitz <e.huhsovitz@proxmox.com>
Subject: [PATCH qemu-server-rs v4 1/4] cpu: add pve-qemu-server-cpu crate for vCPU binding
Date: Wed, 11 Nov 2026 16:20:24 +0100	[thread overview]
Message-ID: <20261111152027.260117-2-e.huhsovitz@proxmox.com> (raw)
In-Reply-To: <20261111152027.260117-1-e.huhsovitz@proxmox.com>

Add a new library crate that implements vCPU to host CPU binding of qemu
guests.

2 main functions are exposed: gen_vcpu_bindings: (1) Reads the CPU
Toplogy from sysfs, validates the configured cpu binding config and
Generates a vCPU:hostCPU binding map. This function should be called
before the guest starts. (2) bind_vcpus: Applies the binding map
generated by gen_vcpu_bindings to the specified vm. The cpu binding is
applied using sched_setaffinity(2).

2 modes are implemented: (1) one-to-one: pin each vCPU to its own host
cpu. (2) numa: bind the set of vCPUs of each vNUMA node to the set of
CPUs of a host NUMA node. The cpu bindings are determined by the numa[N]
configuration of the guest.

Both settings respect the configured affinity.

Thread binding uses sched_setaffinity(2), which accepts a kernel
TID directly. The TID used is reported by qemu via query-cpus-fast.

Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
 Cargo.toml                                |   2 +-
 build.sh                                  |   2 +-
 pve-qemu-server-cpu/Cargo.toml            |  16 ++
 pve-qemu-server-cpu/debian/changelog      |   5 +
 pve-qemu-server-cpu/debian/control        |  41 ++++
 pve-qemu-server-cpu/debian/copyright      |  18 ++
 pve-qemu-server-cpu/debian/debcargo.toml  |   9 +
 pve-qemu-server-cpu/src/bind.rs           | 136 +++++++++++++
 pve-qemu-server-cpu/src/config.rs         | 146 ++++++++++++++
 pve-qemu-server-cpu/src/cpuset.rs         | 125 ++++++++++++
 pve-qemu-server-cpu/src/error.rs          |  32 ++++
 pve-qemu-server-cpu/src/lib.rs            |  92 +++++++++
 pve-qemu-server-cpu/src/qmp.rs            |  91 +++++++++
 pve-qemu-server-cpu/src/syscall.rs        |  24 +++
 pve-qemu-server-cpu/src/topology/mod.rs   | 222 ++++++++++++++++++++++
 pve-qemu-server-cpu/src/topology/sysfs.rs |   6 +
 16 files changed, 965 insertions(+), 2 deletions(-)
 create mode 100644 pve-qemu-server-cpu/Cargo.toml
 create mode 100644 pve-qemu-server-cpu/debian/changelog
 create mode 100644 pve-qemu-server-cpu/debian/control
 create mode 100644 pve-qemu-server-cpu/debian/copyright
 create mode 100644 pve-qemu-server-cpu/debian/debcargo.toml
 create mode 100644 pve-qemu-server-cpu/src/bind.rs
 create mode 100644 pve-qemu-server-cpu/src/config.rs
 create mode 100644 pve-qemu-server-cpu/src/cpuset.rs
 create mode 100644 pve-qemu-server-cpu/src/error.rs
 create mode 100644 pve-qemu-server-cpu/src/lib.rs
 create mode 100644 pve-qemu-server-cpu/src/qmp.rs
 create mode 100644 pve-qemu-server-cpu/src/syscall.rs
 create mode 100644 pve-qemu-server-cpu/src/topology/mod.rs
 create mode 100644 pve-qemu-server-cpu/src/topology/sysfs.rs

diff --git a/Cargo.toml b/Cargo.toml
index 5eefc79..80f87ff 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,5 +1,6 @@
 [workspace]
 members = [
+    "pve-qemu-server-cpu",
     "pve-qemu-server-pci"
 ]
 exclude = [
@@ -25,4 +26,3 @@ strum = { version = "0.26", features = ["derive"] }
 
 # proxmox dependencies
 pve-api-types = "8.1"
-
diff --git a/build.sh b/build.sh
index 3c2293b..c3ee7b6 100755
--- a/build.sh
+++ b/build.sh
@@ -28,7 +28,7 @@ debcargo package \
     --no-overlay-write-back \
     --directory "$PWD/"${BUILDDIR}"/${CRATE}" \
     "${CRATE}" \
-    "$(dpkg-parsechangelog -l "${CRATE}/debian/changelog" -SVersion | sed -e 's/-.*//')"
+    "$(dpkg-parsechangelog -l "${CRATE}/debian/changelog" -SVersion | sed -e 's/-.*//' -e 's/~/-/')"
 
 cd "${BUILDDIR}/${CRATE}"
 rm -f debian/source/format.debcargo.hint
diff --git a/pve-qemu-server-cpu/Cargo.toml b/pve-qemu-server-cpu/Cargo.toml
new file mode 100644
index 0000000..b236796
--- /dev/null
+++ b/pve-qemu-server-cpu/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "pve-qemu-server-cpu"
+version = "0.1.0"
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+homepage.workspace = true
+exclude.workspace = true
+rust-version.workspace = true
+
+[dependencies]
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+thiserror = "2"
+libc = "0.2"
diff --git a/pve-qemu-server-cpu/debian/changelog b/pve-qemu-server-cpu/debian/changelog
new file mode 100644
index 0000000..3731cfd
--- /dev/null
+++ b/pve-qemu-server-cpu/debian/changelog
@@ -0,0 +1,5 @@
+rust-pve-qemu-server-cpu (0.1.0) trixie; urgency=medium
+
+  * initial release
+
+ -- Proxmox Support Team <support@proxmox.com>  Tue, 01 Sep 2026 14:02:25 +0200
diff --git a/pve-qemu-server-cpu/debian/control b/pve-qemu-server-cpu/debian/control
new file mode 100644
index 0000000..1298d61
--- /dev/null
+++ b/pve-qemu-server-cpu/debian/control
@@ -0,0 +1,41 @@
+Source: rust-pve-qemu-server-cpu
+Section: rust
+Priority: optional
+Build-Depends: debhelper-compat (= 13),
+ dh-sequence-cargo,
+ libhwloc-dev (>= 2.12)
+Build-Depends-Arch: cargo:native <!nocheck>,
+ rustc:native (>= 1.94) <!nocheck>,
+ libstd-rust-dev <!nocheck>,
+ librust-libc-0.2+default-dev <!nocheck>,
+ librust-serde-1+default-dev <!nocheck>,
+ librust-serde-1+derive-dev <!nocheck>,
+ librust-serde-json-1+default-dev <!nocheck>,
+ librust-thiserror-2+default-dev <!nocheck>
+Maintainer: Proxmox Support Team <support@proxmox.com>
+Standards-Version: 4.7.2
+Vcs-Git: git://git.proxmox.com/git/pve-qemu-server-rs.git
+Vcs-Browser: https://git.proxmox.com/?p=pve-qemu-server-rs.git
+Homepage: https://proxmox.com
+X-Cargo-Crate: pve-qemu-server-cpu
+
+Package: librust-pve-qemu-server-cpu-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-libc-0.2+default-dev,
+ librust-serde-1+default-dev,
+ librust-serde-1+derive-dev,
+ librust-serde-json-1+default-dev,
+ librust-thiserror-2+default-dev
+Provides:
+ librust-pve-qemu-server-cpu+default-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0+default-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0.1-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0.1+default-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0.1.0-dev (= ${binary:Version}),
+ librust-pve-qemu-server-cpu-0.1.0+default-dev (= ${binary:Version})
+Description: Rust crate "pve-qemu-server-cpu" - Rust source code
+ Source code for Debianized Rust crate "pve-qemu-server-cpu"
diff --git a/pve-qemu-server-cpu/debian/copyright b/pve-qemu-server-cpu/debian/copyright
new file mode 100644
index 0000000..01138fa
--- /dev/null
+++ b/pve-qemu-server-cpu/debian/copyright
@@ -0,0 +1,18 @@
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+
+Files:
+ *
+Copyright: 2026 Proxmox Server Solutions GmbH <support@proxmox.com>
+License: AGPL-3.0-or-later
+ This program is free software: you can redistribute it and/or modify it under
+ the terms of the GNU Affero General Public License as published by the Free
+ Software Foundation, either version 3 of the License, or (at your option) any
+ later version.
+ .
+ This program is distributed in the hope that it will be useful, but WITHOUT
+ ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+ FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+ details.
+ .
+ You should have received a copy of the GNU Affero General Public License along
+ with this program. If not, see <https://www.gnu.org/licenses/>.
diff --git a/pve-qemu-server-cpu/debian/debcargo.toml b/pve-qemu-server-cpu/debian/debcargo.toml
new file mode 100644
index 0000000..23feea4
--- /dev/null
+++ b/pve-qemu-server-cpu/debian/debcargo.toml
@@ -0,0 +1,9 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+allow_prerelease_deps = true
+
+[source]
+vcs_git = "git://git.proxmox.com/git/pve-qemu-server-rs.git"
+vcs_browser = "https://git.proxmox.com/?p=pve-qemu-server-rs.git"
+build_depends = ["libhwloc-dev (>= 2.12)"]
diff --git a/pve-qemu-server-cpu/src/bind.rs b/pve-qemu-server-cpu/src/bind.rs
new file mode 100644
index 0000000..d7f52f8
--- /dev/null
+++ b/pve-qemu-server-cpu/src/bind.rs
@@ -0,0 +1,136 @@
+use std::collections::BTreeMap;
+
+use crate::cpuset::CpuSet;
+use crate::config::NumaConfig;
+use crate::qmp::QmpCpu;
+use crate::syscall::sched_setaffinity;
+use crate::topology::Topology;
+use crate::error::CpuError;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Mode {
+    None,
+    OneToOne,
+    Numa,
+}
+
+impl std::str::FromStr for Mode {
+    type Err = CpuError;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s {
+            "none" => Ok(Mode::None),
+            "one-to-one" => Ok(Mode::OneToOne),
+            "numa" => Ok(Mode::Numa),
+            other => Err(CpuError::BadSyntax(format!(
+                "unknown cpu-bind-mode '{other}'"
+            ))),
+        }
+    }
+}
+
+pub fn gen_one_to_one_bindings(
+    sockets: u32,
+    cores: u32,
+    affinity: &Option<CpuSet>,
+    only_real_cores: bool,
+) -> Result<BTreeMap<u32, CpuSet>, CpuError> {
+    let topo = Topology::new()?;
+    let mut allowed = topo.online_cpus().clone();
+
+    if let Some(aff) = affinity {
+        allowed.intersect(aff);
+    }
+
+    let nr_vcpus = sockets * cores;
+
+    if only_real_cores {
+        allowed = topo.singlify_per_core(&allowed);
+    }
+
+    if allowed.size() < nr_vcpus as usize {
+        return Err(CpuError::BadSyntax(format!(
+            "one-to-one mode requires {nr_vcpus} host cpus but only {} are available",
+            allowed.size()
+        )));
+    }
+
+    let available: Vec<u32> = allowed.to_vec();
+
+    let mut bindings: BTreeMap<u32, CpuSet> = BTreeMap::new();
+
+    // qmp::parse_cpuinfo should yield vCPU IDs that are 0..N-1.
+    // If not: we proably messed up somewhere else
+    for (vcpu_id, cpu) in available.iter().take(nr_vcpus as usize).enumerate() {
+        let mut single_cpu = CpuSet::new();
+        single_cpu.set(*cpu);
+        bindings.insert(vcpu_id as u32, single_cpu);
+    }
+
+    Ok(bindings)
+}
+
+pub fn gen_numa_bindings(
+    numa_cfg: &Vec<NumaConfig>,
+    affinity: &Option<CpuSet>,
+) -> Result<BTreeMap<u32, CpuSet>, CpuError> {
+    let topo = Topology::new()?;
+    let mut bindings = BTreeMap::new();
+
+    for (vnode_id, cfg) in numa_cfg.iter().enumerate() {
+        if cfg.hostnodes.len() != 1 {
+            return Err(CpuError::BadSyntax(format!(
+                "vNUMA node {vnode_id} must be bound to exactly one host NUMA node"
+            )));
+        }
+
+        let host_node_id = *cfg.hostnodes.first().unwrap();
+        let host_cpus = topo.numa_cpus(host_node_id).ok_or_else(|| {
+            CpuError::BadSyntax(format!(
+                "host NUMA node {host_node_id} does not exist on this host"
+            ))
+        })?;
+
+        let mut allowed = host_cpus.clone();
+
+        if let Some(aff) = affinity {
+            allowed.intersect(aff);
+        }
+
+        if allowed.is_empty() {
+            return Err(CpuError::BadSyntax(format!(
+                "no allowed CPUs available for host NUMA node {host_node_id} \
+                after applying affinity mask."
+            )));
+        }
+
+        for &vcpu_id in &cfg.vcpus {
+            bindings.insert(vcpu_id, allowed.clone());
+        }
+    }
+
+    Ok(bindings)
+}
+
+/// Applies the generated CPU bindings to the running vCPU threads.
+pub fn apply_bindings(
+    bindings: &BTreeMap<u32, CpuSet>,
+    qmp_cpuinfo: &[QmpCpu],
+) -> Result<(), CpuError> {
+    for (&vcpu_id, cpuset) in bindings {
+        let vcpu_index = vcpu_id as usize;
+        if vcpu_index >= qmp_cpuinfo.len() {
+            return Err(CpuError::VcpuOutOfRange {
+                vcpu: vcpu_id,
+                max: qmp_cpuinfo.len() as u32,
+            });
+        }
+
+        let tid = qmp_cpuinfo[vcpu_index].thread_id;
+        sched_setaffinity(tid as libc::pid_t, cpuset).map_err(|e| CpuError::Bind {
+            vcpu: vcpu_id,
+            message: e.to_string(),
+        })?;
+    }
+    Ok(())
+}
diff --git a/pve-qemu-server-cpu/src/config.rs b/pve-qemu-server-cpu/src/config.rs
new file mode 100644
index 0000000..a89c62c
--- /dev/null
+++ b/pve-qemu-server-cpu/src/config.rs
@@ -0,0 +1,146 @@
+//! Parsing of the Proxmox VE CPU and memory configuration.
+
+use serde::de::{self, Deserializer, SeqAccess, Visitor};
+use serde::Deserialize;
+use std::collections::{BTreeMap, BTreeSet, HashMap};
+use std::fmt;
+
+use crate::cpuset::CpuSet;
+use crate::error::CpuError;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub enum NumaPolicy {
+    Bind,
+    Interleave,
+    Preferred,
+}
+
+/// Configuration for a single numa[N] node.
+#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
+pub struct NumaConfig {
+    /// Guest vCPU IDs assigned to this node.
+    #[serde(default, deserialize_with = "deserialize_range_set")]
+    pub vcpus: BTreeSet<u32>,
+
+    /// Host NUMA node IDs.
+    #[serde(default, deserialize_with = "deserialize_range_set")]
+    pub hostnodes: BTreeSet<u32>,
+
+    /// Memory provided by this node, in MiB.
+    pub memory: u32,
+
+    /// NUMA allocation policy.
+    pub policy: Option<NumaPolicy>,
+}
+
+/// Parses the NUMA configuration from a JSON string.
+///
+/// Returns a `Vec` where the index corresponds to the NUMA node ID.
+/// Assumes the Perl layer has already validated that node IDs are
+/// contiguous from `0` to `N-1`.
+/// Parses the NUMA configuration from a JSON string.
+///
+/// Returns a `Vec` where the index corresponds to the NUMA node ID.
+/// Assumes the Perl layer has already validated that node IDs are
+/// contiguous from `0` to `N-1`.
+pub fn numa_config_from_json(cfg_json: &str) -> Result<Vec<NumaConfig>, CpuError> {
+    let raw_map: HashMap<String, NumaConfig> = serde_json::from_str(cfg_json)
+        .map_err(|e| CpuError::BadSyntax(format!("invalid NUMA JSON: {e}")))?;
+
+    let len = raw_map.len();
+
+    let mut configs: Vec<Option<NumaConfig>> = vec![None; len];
+
+    for (key, config) in raw_map {
+        let id_str = key.strip_prefix("numa").unwrap_or(&key);
+        let id = id_str.parse::<usize>().map_err(|e| {
+            CpuError::BadSyntax(format!("invalid NUMA node ID '{key}': {e}"))
+        })?;
+
+        configs[id] = Some(config);
+    }
+
+    // Unwrap the options. If Perl did its job, every slot will be filled.
+    // We return an error instead of panicking just in case the invariant is broken.
+    configs
+        .into_iter()
+        .enumerate()
+        .map(|(i, opt)| {
+            opt.ok_or_else(|| CpuError::BadSyntax(format!("missing NUMA node {i}")))
+        })
+        .collect()
+}
+
+/// Expands a JSON array of ranges into a `BTreeSet<u32>`.
+///
+/// Perl outputs ranges as `[[0, 3], [8, null]]`. This visitor expands them
+/// into a flat set of integers: `{0, 1, 2, 3, 8}`.
+fn deserialize_range_set<'de, D>(deserializer: D) -> Result<BTreeSet<u32>, D::Error>
+where
+    D: Deserializer<'de>,
+{
+    struct RangeSetVisitor;
+
+    impl<'de> Visitor<'de> for RangeSetVisitor {
+        type Value = BTreeSet<u32>;
+
+        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+            formatter.write_str("a JSON array of ranges (e.g., [[0, 3], [8, null]])")
+        }
+
+        fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
+        where
+            A: SeqAccess<'de>,
+        {
+            let mut set = BTreeSet::new();
+
+            while let Some((start, end_opt)) = seq.next_element::<(u32, Option<u32>)>()? {
+                let end = end_opt.unwrap_or(start);
+                if end < start {
+                    return Err(de::Error::custom(format!(
+                        "invalid range: end ({end}) < start ({start})"
+                    )));
+                }
+                set.extend(start..=end);
+            }
+
+            Ok(set)
+        }
+
+        fn visit_unit<E>(self) -> Result<Self::Value, E> {
+            Ok(BTreeSet::new())
+        }
+
+        fn visit_none<E>(self) -> Result<Self::Value, E> {
+            Ok(BTreeSet::new())
+        }
+    }
+
+    deserializer.deserialize_any(RangeSetVisitor)
+}
+
+pub fn bindings_from_json(bind_map_json: &str) -> Result<BTreeMap<u32, CpuSet>, CpuError> {
+    let raw_map: BTreeMap<u32, Vec<u32>> = serde_json::from_str(bind_map_json)
+        .map_err(|e| CpuError::BadSyntax(format!("invalid binding map JSON: {e}")))?;
+
+    let mut bindings = BTreeMap::new();
+    for (vcpu, cpus) in raw_map {
+        let mut set = CpuSet::new();
+        for cpu in cpus {
+            set.set(cpu);
+        }
+        bindings.insert(vcpu, set);
+    }
+    Ok(bindings)
+}
+
+pub fn bindings_to_json(bindings: &BTreeMap<u32, CpuSet>) -> Result<String, CpuError> {
+    let map: BTreeMap<u32, Vec<u32>> = bindings
+        .iter()
+        .map(|(&vcpu, cpuset)| (vcpu, cpuset.iter().collect()))
+        .collect();
+
+    serde_json::to_string(&map)
+        .map_err(|e| CpuError::BadSyntax(format!("failed to serialize bindings: {e}")))
+}
diff --git a/pve-qemu-server-cpu/src/cpuset.rs b/pve-qemu-server-cpu/src/cpuset.rs
new file mode 100644
index 0000000..d2ffcf8
--- /dev/null
+++ b/pve-qemu-server-cpu/src/cpuset.rs
@@ -0,0 +1,125 @@
+//! Arbitrary-precision CPU bitmask for efficient set operations.
+
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct CpuSet {
+    bits: Vec<u64>,
+}
+
+struct WordBits {
+    base: u32,
+    remaining: u64,
+}
+
+impl Iterator for WordBits {
+    type Item = u32;
+
+    fn next(&mut self) -> Option<u32> {
+        if self.remaining == 0 {
+            return None;
+        }
+        let bit = self.remaining.trailing_zeros();
+        // Clear the lowest set bit so the next call finds the following one.
+        self.remaining &= self.remaining - 1;
+        Some(self.base + bit)
+    }
+}
+
+impl CpuSet {
+    pub fn new() -> Self {
+        Self { bits: Vec::new() }
+    }
+
+    pub fn set(&mut self, cpu: u32) {
+        let idx = (cpu / 64) as usize;
+        let bit = cpu % 64;
+        if idx >= self.bits.len() {
+            self.bits.resize(idx + 1, 0);
+        }
+        self.bits[idx] |= 1u64 << bit;
+    }
+
+    pub fn intersect(&mut self, other: &CpuSet) {
+        for (i, word) in self.bits.iter_mut().enumerate() {
+            if i < other.bits.len() {
+                *word &= other.bits[i];
+            } else {
+                *word = 0;
+            }
+        }
+    }
+
+    pub fn union(&mut self, other: &CpuSet) {
+        if other.bits.len() > self.bits.len() {
+            self.bits.resize(other.bits.len(), 0);
+        }
+        for (i, word) in other.bits.iter().enumerate() {
+            self.bits[i] |= word;
+        }
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.bits.iter().all(|&w| w == 0)
+    }
+
+    pub fn size(&self) -> usize {
+        self.bits.iter().map(|w| w.count_ones() as usize).sum()
+    }
+
+    pub fn first_set(&self) -> Option<u32> {
+        self.iter().next()
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = u32> + '_ {
+        self.bits
+            .iter()
+            .enumerate()
+            .flat_map(|(idx, &word)| WordBits {
+                base: (idx as u32) * 64,
+                remaining: word,
+            })
+    }
+
+    /// Returns the set members as an ascending vector.
+    pub fn to_vec(&self) -> Vec<u32> {
+        self.iter().collect()
+    }
+
+    /// Converts the bitmask to a `libc::cpu_set_t` for `sched_setaffinity`.
+    pub fn to_cpu_set_t(&self) -> libc::cpu_set_t {
+        unsafe {
+            let mut set: libc::cpu_set_t = std::mem::zeroed();
+            for cpu in self.iter() {
+                libc::CPU_SET(cpu as usize, &mut set);
+            }
+            set
+        }
+    }
+
+    /// Parses sysfs CPU lists, for example `online`, `cpulist`, `core_cpus_list`.
+    ///
+    /// Format: comma-separated CPU IDs and inclusive ranges, such as `0-15,18-31`.
+    pub fn from_list(list: &str) -> Self {
+        let mut set = CpuSet::new();
+        let list = list.trim();
+        if list.is_empty() {
+            return set;
+        }
+
+        for part in list.split(',') {
+            let part = part.trim();
+            if part.is_empty() {
+                continue;
+            }
+            if let Some((start_raw, end_raw)) = part.split_once('-') {
+                let start: u32 = start_raw.trim().parse().unwrap_or(0);
+                let end: u32 = end_raw.trim().parse().unwrap_or(start);
+                for i in start..=end {
+                    set.set(i);
+                }
+            } else if let Ok(cpu) = part.parse::<u32>() {
+                set.set(cpu);
+            }
+        }
+        set
+    }
+}
diff --git a/pve-qemu-server-cpu/src/error.rs b/pve-qemu-server-cpu/src/error.rs
new file mode 100644
index 0000000..2efbc12
--- /dev/null
+++ b/pve-qemu-server-cpu/src/error.rs
@@ -0,0 +1,32 @@
+use thiserror::Error;
+
+#[derive(Debug, Error)]
+#[error("cpu binding: {0}")]
+pub enum CpuError {
+    #[error("invalid syntax in: '{0}'")]
+    BadSyntax(String),
+
+    // Validation errors (pre-start)
+    #[error("host CPU {0} does not exist in this topology")]
+    UnknownHostCpu(u32),
+
+    #[error("vCPU {vcpu} is out of range (VM has vCPUs 0..{max})")]
+    VcpuOutOfRange { vcpu: u32, max: u32 },
+
+    // Runtime errors (post-start)
+    #[error("no running thread found for vCPU {0}")]
+    ThreadNotFound(u32),
+
+    #[error("invalid kernel TID {tid} for vCPU {vcpu}")]
+    BadThreadId { vcpu: u32, tid: u32 },
+
+    #[error("failed to init host topology: '{0}'")]
+    TopologyInit(String),
+
+    #[error("binding vCPU {vcpu} failed: {message}")]
+    Bind {
+        vcpu: u32,
+        /// Human-readable error from sched_setaffinity(2)
+        message: String,
+    },
+}
diff --git a/pve-qemu-server-cpu/src/lib.rs b/pve-qemu-server-cpu/src/lib.rs
new file mode 100644
index 0000000..cf19aa8
--- /dev/null
+++ b/pve-qemu-server-cpu/src/lib.rs
@@ -0,0 +1,92 @@
+mod bind;
+mod cpuset;
+mod error;
+mod config;
+mod qmp;
+mod syscall;
+mod topology;
+
+/// Generate vcpu:cpu(s) binding map for a guest
+///
+/// # Arguments
+///
+/// * `mode` - Pinning mode.
+/// * `sockets` - number of sockets configured in <vmid.conf>
+/// * `cores` - number of cores configured in <vmid.conf>
+/// * `affinity` - Upper bound of host cpus that can be used for binding.
+/// * `cfg_json` - Additional config options. (e.g. numa config)
+/// * `cpuinfo_json` - JSON output from QMP `query-cpus-fast`.
+///
+/// # Modes
+///
+/// | Mode | Meaning |
+/// |---|---|
+/// | `none` | Do not pin vCPUs. |
+/// | `one-to-one` | Assign each vCPU to its own cpu. |
+/// | `numa` | Map guest NUMA nodes to host NUMA nodes |
+///
+/// # Examples
+///
+/// ```ignore
+/// // One-to-one mode:
+/// // Distribute vCPUs over host CPUs 0-7.
+/// gen_vcpu_bindings("one-to-one", 1, 8, "0-7", "", &cpuinfo_json)?;
+///
+/// // One-to-one mode over all allowed host CPUs.
+/// gen_vcpu_bindings("one-to-one", 1, 4 "", "", &cpuinfo_json)?;
+///
+/// // NUMA mode:
+/// // NUMA mode over all allowed host NUMA nodes.
+/// gen_vcpu_bindings("numa", 2, 4, "", &cfg_json, &cpuinfo_json)?;
+///
+/// // NUMA mode respecting the set affinity
+/// gen_vcpu_bindings("numa", 4, 2, "0-7", &cfg_json, &cpuinfo_json)?;
+/// ```
+///
+/// # Errors
+///
+/// Returns [`CpuError`] if the mode or binding syntax is invalid, if the
+/// requested host CPUs or NUMA nodes do not exist, or if binding fails.
+pub fn gen_vcpu_bindings(
+    mode: &str,
+    sockets: u32,
+    cores: u32,
+    affinity: &str,
+    cfg_json: &str,
+) -> Result<String, error::CpuError> {
+    let cpu_affinity = if affinity.is_empty() {
+        None
+    } else {
+        Some(cpuset::CpuSet::from_list(affinity))
+    };
+
+    let bindings: std::collections::BTreeMap<u32, cpuset::CpuSet> = match mode.parse::<bind::Mode>()? {
+        bind::Mode::None => return Ok("{}".to_string()),
+        bind::Mode::OneToOne => {
+            // only_real_cores is hardcoded to false for now, until we device if we actually want this setting
+            bind::gen_one_to_one_bindings(sockets, cores, &cpu_affinity, false)?
+        }
+        bind::Mode::Numa => {
+            let numa_cfg = config::numa_config_from_json(cfg_json)?;
+            bind::gen_numa_bindings(&numa_cfg, &cpu_affinity)?
+        }
+    };
+
+    // This shouldn't happen, but we should still avoid passing empty config to perl
+    if bindings.is_empty() {
+        return Err(error::CpuError::BadSyntax(
+            "cpu-binding-mode '{mode}' is enabled but no vCPU bindings were generated".to_string(),
+        ));
+    }
+
+    config::bindings_to_json(&bindings)
+}
+
+/// Apply vcpu:cpu(s) the binding map
+/// The function parses the QMP `query-cpus-fast` response and binds each
+/// vCPU thread according to the selected mode.
+pub fn bind_vcpus(bind_map_json: &str, qmp_cpuinfo_json: &str) -> Result<(), error::CpuError> {
+    let bindings = config::bindings_from_json(bind_map_json)?;
+    let qmp_cpuinfo = qmp::parse_cpuinfo(qmp_cpuinfo_json)?;
+    bind::apply_bindings(&bindings, &qmp_cpuinfo)
+}
diff --git a/pve-qemu-server-cpu/src/qmp.rs b/pve-qemu-server-cpu/src/qmp.rs
new file mode 100644
index 0000000..78fb8bd
--- /dev/null
+++ b/pve-qemu-server-cpu/src/qmp.rs
@@ -0,0 +1,91 @@
+//! Parsing of the QEMU `query-cpus-fast` QMP response.
+//!
+//! The response is a JSON array; each element describes one guest vCPU
+//! and carries both its guest-visible topology (socket/core/NUMA node)
+//! and its host kernel TID. Every binding mode consumes this shape, so
+//! the parsing lives here rather than inside any individual mode.
+//!
+//! The full QEMU schema is modeled, including fields not currently used
+//! by any binding mode. Serde ignores fields it doesn't know about, but
+//! keeping them explicit documents the wire format and makes future
+//! modes cheaper to implement.
+
+use serde::Deserialize;
+
+use crate::error::CpuError;
+
+/// One entry of the `query-cpus-fast` array.
+#[derive(Debug, Clone, Deserialize)]
+pub struct QmpCpu {
+    /// Guest-visible vCPU index (0 .. sockets*cores).
+    #[serde(rename = "cpu-index")]
+    pub cpu_index: u32,
+
+    /// Host kernel TID of the QEMU vCPU thread.
+    #[serde(rename = "thread-id")]
+    pub thread_id: u32,
+
+    /// QOM path of the CPU object, e.g. `/machine/unattached/device[0]`.
+    #[expect(dead_code)]
+    #[serde(rename = "qom-path")]
+    pub qom_path: String,
+
+    /// QOM type of the CPU object, e.g. `host-x86_64-cpu`.
+    #[expect(dead_code)]
+    #[serde(rename = "qom-type")]
+    pub qom_type: String,
+
+    /// Target architecture, e.g. `x86_64`.
+    #[expect(dead_code)]
+    pub target: String,
+
+    /// Guest-visible topology properties of the vCPU.
+    #[expect(dead_code)]
+    pub props: QmpCpuProps,
+}
+
+/// Guest-visible topology properties of a vCPU.
+#[derive(Debug, Clone, Deserialize)]
+pub struct QmpCpuProps {
+    /// Guest socket the vCPU belongs to.
+    #[expect(dead_code)]
+    #[serde(rename = "socket-id")]
+    pub socket_id: u32,
+
+    /// Guest core within that socket.
+    #[expect(dead_code)]
+    #[serde(rename = "core-id")]
+    pub core_id: u32,
+
+    /// Guest NUMA node the vCPU belongs to.
+    ///
+    /// QEMU reports this even when `numa` is disabled in the VM config,
+    /// in which case every vCPU is on node 0.
+    #[expect(dead_code)]
+    #[serde(rename = "node-id")]
+    pub node_id: u32,
+
+    /// Guest thread within the core.
+    ///
+    /// Always 0 today: QEMU's CPU object model carries this field for
+    /// SMT support in the guest, but PVE does not expose SMT to guests,
+    /// so it is always zero in practice. Modeled for completeness.
+    /// TODO: PVE should expose SMT to the guests
+    #[expect(dead_code)]
+    #[serde(rename = "thread-id")]
+    pub thread_id: u32,
+}
+
+// returned vec is sorted by vCPU id (0,1,2,3...)
+// Guarantees vec[id] == vec[id].cpu_index
+pub fn parse_cpuinfo(cpuinfo_json: &str) -> Result<Vec<QmpCpu>, CpuError> {
+    let mut cpus: Vec<QmpCpu> = serde_json::from_str(cpuinfo_json)
+        .map_err(|e| CpuError::BadSyntax(format!("invalid query-cpus-fast JSON: {e}")))?;
+
+    // QEMU assigns cpu-index sequentially from 0. Sort by it so
+    // `vec[i]` indexes vCPU i,
+    // value should be 0..N-1 so that assumption holds.
+    cpus.sort_unstable_by_key(|c| c.cpu_index);
+
+    Ok(cpus)
+}
diff --git a/pve-qemu-server-cpu/src/syscall.rs b/pve-qemu-server-cpu/src/syscall.rs
new file mode 100644
index 0000000..59b74a9
--- /dev/null
+++ b/pve-qemu-server-cpu/src/syscall.rs
@@ -0,0 +1,24 @@
+//! Safe wrappers around Linux syscalls.
+
+#[cfg(target_os = "linux")]
+use std::io;
+
+use crate::cpuset::CpuSet;
+
+pub fn sched_setaffinity(tid: libc::pid_t, cpus: &CpuSet) -> io::Result<()> {
+    let set = cpus.to_cpu_set_t();
+
+    unsafe {
+        let rc = libc::sched_setaffinity(
+            tid,
+            std::mem::size_of::<libc::cpu_set_t>(),
+            &set,
+        );
+
+        if rc != 0 {
+            return Err(io::Error::last_os_error());
+        }
+    }
+
+    Ok(())
+}
diff --git a/pve-qemu-server-cpu/src/topology/mod.rs b/pve-qemu-server-cpu/src/topology/mod.rs
new file mode 100644
index 0000000..3d7ffaa
--- /dev/null
+++ b/pve-qemu-server-cpu/src/topology/mod.rs
@@ -0,0 +1,222 @@
+//! Hierarchical hardware topology discovered from sysfs.
+//!
+//! The topology is a tree rooted at the machine. Every node owns its
+//! children and there are no parent pointers, so traversal is top-down.
+//!
+//! On modern systems (like AMD EPYC), the hierarchy typically looks like:
+//!
+//! ```text
+//! Machine -> Package -> Die -> NumaNode -> Core -> Pu
+//! ```
+
+mod sysfs;
+
+use std::collections::HashMap;
+use std::fs;
+use std::path::Path;
+
+use crate::cpuset::CpuSet;
+use crate::error::CpuError;
+
+const CPU_DIR: &str = "/sys/devices/system/cpu";
+const NODE_DIR: &str = "/sys/devices/system/node";
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ObjectKind {
+    Machine,
+    Package,
+    Die,
+    NumaNode,
+    // Cache, // Future extension
+    Core,
+    Pu,
+}
+
+/// A node in the hardware topology tree.
+///
+/// Each node owns its children. The `cpuset` covers every CPU at or below
+/// this node. `os_index` is the operating-system index (hwloc's `P#`).
+#[derive(Debug, Clone)]
+pub struct TopoObject {
+    pub kind: ObjectKind,
+    pub os_index: u32,
+    pub cpuset: CpuSet,
+    pub children: Vec<TopoObject>,
+}
+
+impl TopoObject {
+    fn new(kind: ObjectKind, os_index: u32) -> Self {
+        Self {
+            kind,
+            os_index,
+            cpuset: CpuSet::new(),
+            children: Vec::new(),
+        }
+    }
+
+    /// Finds an existing child by kind and index, or creates and appends a new one.
+    fn get_or_add_child(&mut self, kind: ObjectKind, os_index: u32) -> &mut TopoObject {
+        let pos = self
+            .children
+            .iter()
+            .position(|c| c.kind == kind && c.os_index == os_index)
+            .unwrap_or_else(|| {
+                self.children.push(TopoObject::new(kind, os_index));
+                self.children.len() - 1
+            });
+
+        &mut self.children[pos]
+    }
+
+    /// Finds the first node matching `kind` and `os_index` in this subtree.
+    pub fn find(&self, kind: ObjectKind, os_index: u32) -> Option<&TopoObject> {
+        if self.kind == kind && self.os_index == os_index {
+            return Some(self);
+        }
+        for child in &self.children {
+            if let Some(found) = child.find(kind, os_index) {
+                return Some(found);
+            }
+        }
+        None
+    }
+
+    /// Calls `f` for every node matching `kind` in this subtree.
+    pub fn for_each(&self, kind: ObjectKind, f: &mut dyn FnMut(&TopoObject)) {
+        if self.kind == kind {
+            f(self);
+        }
+        for child in &self.children {
+            child.for_each(kind, &mut *f);
+        }
+    }
+
+    /// Computes this node's cpuset bottom-up from its children.
+    fn compute_cpuset(&mut self) {
+        self.cpuset = CpuSet::new();
+        if self.kind == ObjectKind::Pu {
+            self.cpuset.set(self.os_index);
+        } else {
+            for child in &mut self.children {
+                child.compute_cpuset();
+                self.cpuset.union(&child.cpuset);
+            }
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct Topology {
+    online_cpus: CpuSet,
+    root: TopoObject,
+}
+
+impl Topology {
+    pub fn new() -> Result<Self, CpuError> {
+        let online_cpus = sysfs::read_file(&Path::new(CPU_DIR).join("online"))
+            .map(|s| CpuSet::from_list(&s))
+            .ok_or_else(|| {
+                CpuError::TopologyInit("cannot read the online CPU mask".to_string())
+            })?;
+
+        let cpu_to_numa = Self::read_cpu_to_numa();
+
+        let mut root = TopoObject::new(ObjectKind::Machine, 0);
+        Self::build_tree(&mut root, &online_cpus, &cpu_to_numa)?;
+
+        root.compute_cpuset();
+
+        Ok(Topology { online_cpus, root })
+    }
+
+    fn build_tree(
+        root: &mut TopoObject,
+        online_cpus: &CpuSet,
+        cpu_to_numa: &HashMap<u32, u32>,
+    ) -> Result<(), CpuError> {
+        let cpu_dir = Path::new(CPU_DIR);
+
+        for cpu_id in online_cpus.iter() {
+            let topo_dir = cpu_dir.join(format!("cpu{cpu_id}")).join("topology");
+
+            let pkg_id = sysfs::read_file(&topo_dir.join("physical_package_id"))
+                .and_then(|s| s.parse().ok())
+                .unwrap_or(0);
+
+            let die_id = sysfs::read_file(&topo_dir.join("die_id"))
+                .and_then(|s| s.parse().ok())
+                .unwrap_or(0);
+
+            let core_id = sysfs::read_file(&topo_dir.join("core_id"))
+                .and_then(|s| s.parse().ok())
+                .unwrap_or(cpu_id);
+
+            let pkg = root.get_or_add_child(ObjectKind::Package, pkg_id);
+            let die = pkg.get_or_add_child(ObjectKind::Die, die_id);
+
+            // NumaNode is optional. If present, it sits between Die and Core.
+            let parent_of_core = if let Some(numa_id) = cpu_to_numa.get(&cpu_id).copied() {
+                die.get_or_add_child(ObjectKind::NumaNode, numa_id)
+            } else {
+                &mut *die
+            };
+
+            let core = parent_of_core.get_or_add_child(ObjectKind::Core, core_id);
+            core.children.push(TopoObject::new(ObjectKind::Pu, cpu_id));
+        }
+
+        Ok(())
+    }
+
+    pub fn online_cpus(&self) -> &CpuSet {
+        &self.online_cpus
+    }
+
+    pub fn numa_cpus(&self, os_index: u32) -> Option<&CpuSet> {
+        self.root
+            .find(ObjectKind::NumaNode, os_index)
+            .map(|node| &node.cpuset)
+    }
+
+    /// Reduces `cpus` to at most one thread per physical core.
+    /// (Avoids SMT siblings)
+    pub fn singlify_per_core(&self, cpus: &CpuSet) -> CpuSet {
+        let mut result = CpuSet::new();
+        self.root.for_each(ObjectKind::Core, &mut |core| {
+            let mut core_cpus = core.cpuset.clone();
+            core_cpus.intersect(cpus);
+            if let Some(first) = core_cpus.first_set() {
+                result.set(first);
+            }
+        });
+        result
+    }
+
+    fn read_cpu_to_numa() -> HashMap<u32, u32> {
+        let mut map = HashMap::new();
+        let node_dir = Path::new(NODE_DIR);
+
+        let entries = match fs::read_dir(node_dir) {
+            Ok(entries) => entries,
+            Err(_) => return map,
+        };
+
+        for entry in entries.flatten() {
+            let name = entry.file_name();
+            let name_str = name.to_string_lossy();
+            let node_id = match name_str.strip_prefix("node").and_then(|s| s.parse::<u32>().ok()) {
+                Some(id) => id,
+                None => continue,
+            };
+
+            let cpuset = sysfs::read_file(&entry.path().join("cpulist"))
+                .map(|s| CpuSet::from_list(&s))
+                .unwrap_or_else(CpuSet::new);
+
+            for cpu in cpuset.iter() {
+                map.insert(cpu, node_id);
+            }
+        }
+        map
+    }
+}
diff --git a/pve-qemu-server-cpu/src/topology/sysfs.rs b/pve-qemu-server-cpu/src/topology/sysfs.rs
new file mode 100644
index 0000000..a1ff67a
--- /dev/null
+++ b/pve-qemu-server-cpu/src/topology/sysfs.rs
@@ -0,0 +1,6 @@
+use std::fs;
+use std::path::Path;
+
+pub fn read_file(path: &Path) -> Option<String> {
+    fs::read_to_string(path).ok().map(|s| s.trim().to_string())
+}
-- 
2.47.3





  reply	other threads:[~2026-11-11 15:21 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-11 15:20 [RFC proxmox-perl-rs/qemu-server/qemu-server-rs v4 0/4] vCPU binding for qemu guests Elias Huhsovitz
2026-11-11 15:20 ` Elias Huhsovitz [this message]
2026-11-11 15:20 ` [PATCH proxmox-perl-rs v4 2/4] pve: cpu: add binding for pve-qemu-server-cpu crate Elias Huhsovitz
2026-11-11 15:20 ` [PATCH qemu-server v4 3/4] helpers: return integer values for parse_number_sets Elias Huhsovitz
2026-11-11 15:20 ` [PATCH qemu-server v4 4/4] cpu: add cpu binding Elias Huhsovitz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261111152027.260117-2-e.huhsovitz@proxmox.com \
    --to=e.huhsovitz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal