From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host
Date: Thu, 19 Nov 2026 14:43:45 +0100 [thread overview]
Message-ID: <DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-12-d.csapak@proxmox.com>
>From a high level view, I think the design is fitting.
It would be very interesting to measure the real world performance
impact of the 3 layouts.
Comments inline.
On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> Guest software sometimes uses the PCI topology to choose CPUs and DMA
> paths. Since we put every hostpcie device on a root port of the root
> bus, the guest can't see which devices are close to which NUMA node or
> to each other.
>
> So add a hostpci topology setting for machines with a PCIe root bus
> (q35 and aarch64 virt) and the grouped layout:
> * 'flat': the current placement, a root port on the root bus per device
> * 'numa': puts the root ports of each guest NUMA node's devices behind
> a pxb-pcie for that node
> * 'host': additionally replicates the host PCIe switches that more
> than one passed through device is behind. We skip the other
> switches, since they don't add any information but cost bus numbers.
>
> The guest NUMA node and the host switches of each device are passed as
> placements separately from the hardware config, since the caller only
> knows them after choosing the host devices.
>
> All buses share the 256 bus numbers of a single PCI domain. We pack
> the bus number ranges of the expander bridges by demand and align them
> to 16, so a changed device normally only shifts its own node, and even
> 16 devices always fit.
>
> Each port gets its own slot, numbered after the lowest hostpcie index
> behind it, so its address does not depend on its siblings or on a
> function 0 being present.
>
> In the bindings, add 'hostpcie_bus_devices', which takes the placements
> and returns the expander bridges, switches and root ports, and let
> 'addr' take them as an optional parameter, so existing callers don't
> have to change. Since the topology is a required field, qemu-server
> passes 'flat' until the option to choose it is added.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
[...]
> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result<PciAddress, LayoutError> {
> - let layout = layout(hw.has_pcie());
> - layout.find_device(&id.parse()?)
> +pub(crate) fn address(
> + hw: &HardwareConfig,
> + id: &str,
> + placements: Option<&[HostpciePlacement]>,
> +) -> Result<PciAddress, LayoutError> {
> + let device = id.parse()?;
> + match device {
> + Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => {
> + let placements = placements.ok_or(LayoutError::MissingPlacement(index))?;
> + let topology = Topology::new(hw, placements)?;
Isn't building a new topology for each address() call a bit excessive?
IMO we should rebuild the topology only when necesarry.
Or do we run into some freshness issues in this case?
> + let addr = topology.addresses.get(&index).copied();
> + addr.ok_or(LayoutError::NoAddressFound(device))
> + }
> + _ => layout(hw.has_pcie()).find_device(&device),
> + }
> }
>
> pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option<Bus> {
> - let addr = address(hw, id).ok()?;
> + let addr = address(hw, id, None).ok()?;
> Some(addr.bus)
> }
>
[...]
> +
> +/// How passed through PCIe devices are placed in the guest.
> +///
> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer
> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device.
> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub enum HostpciTopology {
> + /// Each device gets its own root port on the root bus.
> + #[default]
> + Flat,
> + /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node.
> + Numa,
> + /// Like `Numa`, but also replicates host switches with more than one device behind them.
> + Host,
> +}
> +
> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other
> +/// than [`HostpciTopology::Flat`].
> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct HostpciePlacement {
> + /// The hostpci index of the device (e.g. '3' from 'hostpci3').
> + pub index: u8,
> + /// The guest NUMA node backed by the host NUMA node of the device.
> + #[serde(default)]
> + pub numa_node: u8,
> + /// The host bridges above the device, starting at the root port. Each one needs a unique id,
> + /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch,
> + /// so the devices have the same common upstream bridge as on the host. Only used with
> + /// [`HostpciTopology::Host`].
> + #[serde(default)]
> + pub switches: Vec<String>,
> +}
> +
> +/// A device or replicated switch in the guest.
> +enum Node {
> + Device(u8),
> + Switch(BTreeMap<u8, Node>),
> +}
> +
> +impl Node {
> + /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the
> + /// address of a port does not depend on its siblings.
> + fn key(&self) -> u8 {
> + match self {
> + Node::Device(index) => *index,
> + Node::Switch(children) => *children.keys().next().unwrap(),
> + }
> + }
> +}
> +
> +/// The devices and switches below a NUMA node or a switch on the host.
> +#[derive(Default)]
> +struct HostTree<'a> {
> + devices: Vec<u8>,
> + switches: BTreeMap<&'a str, HostTree<'a>>,
> +}
nit: naming: IMO something like "HostNode" or "PciNode" fits better.
Usually a "Tree" struct just wraps the root node (In Java at least)
> +impl<'a> HostTree<'a> {
> + /// Adds `hostpcie<index>` below the given host switches.
> + fn insert(&mut self, index: u8, switches: &'a [String]) {
> + match switches.split_first() {
> + None => self.devices.push(index),
> + Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest),
> + }
> + }
> +
> + /// Returns the guest nodes directly below this one, keyed by [`Node::key`].
> + fn children(self) -> BTreeMap<u8, Node> {
> + let devices = self.devices.into_iter().map(Node::Device);
> + let switches = self.switches.into_values().map(HostTree::into_node);
> + devices
> + .chain(switches)
> + .map(|node| (node.key(), node))
> + .collect()
> + }
> +
> + /// Converts the switch to a guest node. Switches with only one child are replaced by that
> + /// child, since they only use up bus numbers without giving the guest any information.
> + fn into_node(self) -> Node {
> + let mut children = self.children();
> + if children.len() == 1 {
> + children.pop_first().unwrap().1
> + } else {
> + Node::Switch(children)
> + }
> + }
> +}
> +
> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured
> +/// are ignored.
> +fn numa_nodes<'a>(
> + hw: &HardwareConfig,
> + placements: &'a [HostpciePlacement],
> +) -> Result<BTreeMap<u8, HostTree<'a>>, LayoutError> {
> + let placements: BTreeMap<u8, &HostpciePlacement> =
> + placements.iter().map(|p| (p.index, p)).collect();
> + let mut nodes: BTreeMap<u8, HostTree> = BTreeMap::new();
> + let mut parents = BTreeMap::new();
> +
> + for &index in hw.hostpcie_devices.keys() {
> + let device = placements
> + .get(&index)
> + .ok_or(LayoutError::MissingPlacement(index))?;
> + if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES {
> + return Err(LayoutError::InvalidPlacement(index));
> + }
> + let switches: &[String] = match hw.hostpci_topology {
> + HostpciTopology::Host => &device.switches,
> + _ => &[],
> + };
> + // switches must be nested identically for all devices (e.g. switch X can't be behind Y for
> + // one device and behind Z for another)
> + for (depth, switch) in switches.iter().enumerate() {
> + let parent = (device.numa_node, &switches[..depth]);
> + if *parents.entry(switch).or_insert(parent) != parent {
> + return Err(LayoutError::InvalidPlacement(index));
> + }
> + }
> + nodes
> + .entry(device.numa_node)
> + .or_default()
> + .insert(index, switches);
> + }
> +
> + Ok(nodes)
> +}
> +
> +/// Returns the `-device` value for the device creating `bus` at `addr`.
> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String {
> + // all buses of the topology are created by a device
> + let model = bus.bridge_model(addr.bus).unwrap();
> + format!("{model},id={bus}{}{params}", addr.to_qemu_addr())
> +}
> +
> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses.
> +pub(crate) struct Topology {
> + /// `-device` values for all expander bridges, ports and switches, each bus before the devices
> + /// on it.
> + pub(crate) devices: Vec<String>,
> + /// Addresses of the `hostpcie` devices, keyed by index.
> + pub(crate) addresses: BTreeMap<u8, PciAddress>,
> + switches: u8,
> +}
> +
> +impl Topology {
> + /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`].
> + pub(crate) fn new(
> + hw: &HardwareConfig,
> + placements: &[HostpciePlacement],
> + ) -> Result<Self, LayoutError> {
> + if !hw.has_pcie() || hw.layout != PciLayout::Grouped {
> + return Err(LayoutError::UnsupportedTopology);
> + }
> +
> + let mut topology = Self {
> + devices: Vec::new(),
> + addresses: BTreeMap::new(),
> + switches: 0,
> + };
> + let mut bus_nr = FIRST_NUMA_BUS;
> + for (numa_node, tree) in numa_nodes(hw, placements)? {
> + let start = topology.devices.len();
> + let bus = Bus::Numa(numa_node);
> + let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?;
Why is the PCIE_Layout always the grouped layout here? I understand a
non-flat topology currently only makes sense in the context of a
grouped layout, but shouldnt we keep this somewhat generic so we
aren't locked into the grouped layout?
IMO we can just pass the layout in the constructor of Topology.
> + let params = format!(",bus_nr={bus_nr},numa_node={numa_node}");
> + topology.devices.push(device(bus, addr, ¶ms));
> + for (key, node) in tree.children() {
> + // keep slot 0 free, like on all other bridges
> + let addr = PciAddress::new(bus, key + 1, 0);
> + topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node);
> + }
> +
> + // each device we added creates exactly one bus
> + let buses = topology.devices.len() - start;
> + if bus_nr + buses > 256 {
> + return Err(LayoutError::BusNumbersExhausted);
> + }
> + bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN);
> + }
> +
> + Ok(topology)
> + }
> +
> + /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are
> + /// root ports, on switches downstream ports.
> + fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) {
> + let root_port = matches!(addr.bus, Bus::Numa(_));
> + let id = match node {
> + Node::Device(index) if root_port => Bus::RootPort(*index),
> + Node::Device(index) => Bus::SwitchPort(*index),
> + Node::Switch(_) => Bus::SwitchUplink(self.switches),
> + };
> + let slot = node.key();
> + let mut params = format!(",port={slot},chassis={chassis},slot={slot}");
> + if root_port {
> + params.push_str(",x-speed=16,x-width=32");
> + }
> + self.devices.push(device(id, addr, ¶ms));
> +
> + match node {
> + Node::Device(index) => {
> + self.addresses.insert(*index, PciAddress::new(id, 0, 0));
> + }
> + Node::Switch(children) => {
> + let nr = self.switches;
> + self.switches += 1;
> +
> + let switch = Bus::Switch(nr);
> + self.devices
> + .push(device(switch, PciAddress::new(id, 0, 0), ""));
> + for (key, child) in children {
> + let addr = PciAddress::new(switch, *key, 0);
> + self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child);
> + }
> + }
> + }
> + }
> +}
> +
[... tests ...]
[... perlmod ..]
next prev parent reply other threads:[~2026-11-19 13:43 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47 ` Elias Huhsovitz
2026-11-19 14:51 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43 ` Elias Huhsovitz [this message]
2026-11-19 15:10 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42 ` Elias Huhsovitz
2026-11-19 15:12 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33 ` Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com \
--to=e.huhsovitz@proxmox.com \
--cc=d.csapak@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox