public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host
Date: Thu, 19 Nov 2026 14:43:45 +0100	[thread overview]
Message-ID: <DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-12-d.csapak@proxmox.com>

>From a high level view, I think the design is fitting.

It would be very interesting to measure the real world performance
impact of the 3 layouts.

Comments inline.

On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> Guest software sometimes uses the PCI topology to choose CPUs and DMA
> paths. Since we put every hostpcie device on a root port of the root
> bus, the guest can't see which devices are close to which NUMA node or
> to each other.
>
> So add a hostpci topology setting for machines with a PCIe root bus
> (q35 and aarch64 virt) and the grouped layout:
> * 'flat': the current placement, a root port on the root bus per device
> * 'numa': puts the root ports of each guest NUMA node's devices behind
>   a pxb-pcie for that node
> * 'host': additionally replicates the host PCIe switches that more
>   than one passed through device is behind. We skip the other
>   switches, since they don't add any information but cost bus numbers.
>
> The guest NUMA node and the host switches of each device are passed as
> placements separately from the hardware config, since the caller only
> knows them after choosing the host devices.
>
> All buses share the 256 bus numbers of a single PCI domain. We pack
> the bus number ranges of the expander bridges by demand and align them
> to 16, so a changed device normally only shifts its own node, and even
> 16 devices always fit.
>
> Each port gets its own slot, numbered after the lowest hostpcie index
> behind it, so its address does not depend on its siblings or on a
> function 0 being present.
>
> In the bindings, add 'hostpcie_bus_devices', which takes the placements
> and returns the expander bridges, switches and root ports, and let
> 'addr' take them as an optional parameter, so existing callers don't
> have to change. Since the topology is a required field, qemu-server
> passes 'flat' until the option to choose it is added.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
  
[...]

> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result<PciAddress, LayoutError> {
> -    let layout = layout(hw.has_pcie());
> -    layout.find_device(&id.parse()?)
> +pub(crate) fn address(
> +    hw: &HardwareConfig,
> +    id: &str,
> +    placements: Option<&[HostpciePlacement]>,
> +) -> Result<PciAddress, LayoutError> {
> +    let device = id.parse()?;
> +    match device {
> +        Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => {
> +            let placements = placements.ok_or(LayoutError::MissingPlacement(index))?;
> +            let topology = Topology::new(hw, placements)?;

Isn't building a new topology for each address() call a bit excessive?
IMO we should rebuild the topology only when necesarry.

Or do we run into some freshness issues in this case?

> +            let addr = topology.addresses.get(&index).copied();
> +            addr.ok_or(LayoutError::NoAddressFound(device))
> +        }
> +        _ => layout(hw.has_pcie()).find_device(&device),
> +    }
>  }
>  
>  pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option<Bus> {
> -    let addr = address(hw, id).ok()?;
> +    let addr = address(hw, id, None).ok()?;
>      Some(addr.bus)
>  }
>  

[...]

> +
> +/// How passed through PCIe devices are placed in the guest.
> +///
> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer
> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device.
> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub enum HostpciTopology {
> +    /// Each device gets its own root port on the root bus.
> +    #[default]
> +    Flat,
> +    /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node.
> +    Numa,
> +    /// Like `Numa`, but also replicates host switches with more than one device behind them.
> +    Host,
> +}
> +
> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other
> +/// than [`HostpciTopology::Flat`].
> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct HostpciePlacement {
> +    /// The hostpci index of the device (e.g. '3' from 'hostpci3').
> +    pub index: u8,
> +    /// The guest NUMA node backed by the host NUMA node of the device.
> +    #[serde(default)]
> +    pub numa_node: u8,
> +    /// The host bridges above the device, starting at the root port. Each one needs a unique id,
> +    /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch,
> +    /// so the devices have the same common upstream bridge as on the host. Only used with
> +    /// [`HostpciTopology::Host`].
> +    #[serde(default)]
> +    pub switches: Vec<String>,
> +}
> +
> +/// A device or replicated switch in the guest.
> +enum Node {
> +    Device(u8),
> +    Switch(BTreeMap<u8, Node>),
> +}
> +
> +impl Node {
> +    /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the
> +    /// address of a port does not depend on its siblings.
> +    fn key(&self) -> u8 {
> +        match self {
> +            Node::Device(index) => *index,
> +            Node::Switch(children) => *children.keys().next().unwrap(),
> +        }
> +    }
> +}
> +
> +/// The devices and switches below a NUMA node or a switch on the host.
> +#[derive(Default)]
> +struct HostTree<'a> {
> +    devices: Vec<u8>,
> +    switches: BTreeMap<&'a str, HostTree<'a>>,
> +}

nit: naming: IMO something like "HostNode" or "PciNode" fits better.
Usually a "Tree" struct just wraps the root node (In Java at least)

> +impl<'a> HostTree<'a> {
> +    /// Adds `hostpcie<index>` below the given host switches.
> +    fn insert(&mut self, index: u8, switches: &'a [String]) {
> +        match switches.split_first() {
> +            None => self.devices.push(index),
> +            Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest),
> +        }
> +    }
> +
> +    /// Returns the guest nodes directly below this one, keyed by [`Node::key`].
> +    fn children(self) -> BTreeMap<u8, Node> {
> +        let devices = self.devices.into_iter().map(Node::Device);
> +        let switches = self.switches.into_values().map(HostTree::into_node);
> +        devices
> +            .chain(switches)
> +            .map(|node| (node.key(), node))
> +            .collect()
> +    }
> +
> +    /// Converts the switch to a guest node. Switches with only one child are replaced by that
> +    /// child, since they only use up bus numbers without giving the guest any information.
> +    fn into_node(self) -> Node {
> +        let mut children = self.children();
> +        if children.len() == 1 {
> +            children.pop_first().unwrap().1
> +        } else {
> +            Node::Switch(children)
> +        }
> +    }
> +}
> +
> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured
> +/// are ignored.
> +fn numa_nodes<'a>(
> +    hw: &HardwareConfig,
> +    placements: &'a [HostpciePlacement],
> +) -> Result<BTreeMap<u8, HostTree<'a>>, LayoutError> {
> +    let placements: BTreeMap<u8, &HostpciePlacement> =
> +        placements.iter().map(|p| (p.index, p)).collect();
> +    let mut nodes: BTreeMap<u8, HostTree> = BTreeMap::new();
> +    let mut parents = BTreeMap::new();
> +
> +    for &index in hw.hostpcie_devices.keys() {
> +        let device = placements
> +            .get(&index)
> +            .ok_or(LayoutError::MissingPlacement(index))?;
> +        if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES {
> +            return Err(LayoutError::InvalidPlacement(index));
> +        }
> +        let switches: &[String] = match hw.hostpci_topology {
> +            HostpciTopology::Host => &device.switches,
> +            _ => &[],
> +        };
> +        // switches must be nested identically for all devices (e.g. switch X can't be behind Y for
> +        // one device and behind Z for another)
> +        for (depth, switch) in switches.iter().enumerate() {
> +            let parent = (device.numa_node, &switches[..depth]);
> +            if *parents.entry(switch).or_insert(parent) != parent {
> +                return Err(LayoutError::InvalidPlacement(index));
> +            }
> +        }
> +        nodes
> +            .entry(device.numa_node)
> +            .or_default()
> +            .insert(index, switches);
> +    }
> +
> +    Ok(nodes)
> +}
> +
> +/// Returns the `-device` value for the device creating `bus` at `addr`.
> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String {
> +    // all buses of the topology are created by a device
> +    let model = bus.bridge_model(addr.bus).unwrap();
> +    format!("{model},id={bus}{}{params}", addr.to_qemu_addr())
> +}
> +
> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses.
> +pub(crate) struct Topology {
> +    /// `-device` values for all expander bridges, ports and switches, each bus before the devices
> +    /// on it.
> +    pub(crate) devices: Vec<String>,
> +    /// Addresses of the `hostpcie` devices, keyed by index.
> +    pub(crate) addresses: BTreeMap<u8, PciAddress>,
> +    switches: u8,
> +}
> +
> +impl Topology {
> +    /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`].
> +    pub(crate) fn new(
> +        hw: &HardwareConfig,
> +        placements: &[HostpciePlacement],
> +    ) -> Result<Self, LayoutError> {
> +        if !hw.has_pcie() || hw.layout != PciLayout::Grouped {
> +            return Err(LayoutError::UnsupportedTopology);
> +        }
> +
> +        let mut topology = Self {
> +            devices: Vec::new(),
> +            addresses: BTreeMap::new(),
> +            switches: 0,
> +        };
> +        let mut bus_nr = FIRST_NUMA_BUS;
> +        for (numa_node, tree) in numa_nodes(hw, placements)? {
> +            let start = topology.devices.len();
> +            let bus = Bus::Numa(numa_node);
> +            let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?;

Why is the PCIE_Layout always the grouped layout here? I understand a 
non-flat topology currently only makes sense in the context of a
grouped layout, but shouldnt we keep this somewhat generic so we
aren't locked into the grouped layout?

IMO we can just pass the layout in the constructor of Topology.

> +            let params = format!(",bus_nr={bus_nr},numa_node={numa_node}");
> +            topology.devices.push(device(bus, addr, &params));
> +            for (key, node) in tree.children() {
> +                // keep slot 0 free, like on all other bridges
> +                let addr = PciAddress::new(bus, key + 1, 0);
> +                topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node);
> +            }
> +
> +            // each device we added creates exactly one bus
> +            let buses = topology.devices.len() - start;
> +            if bus_nr + buses > 256 {
> +                return Err(LayoutError::BusNumbersExhausted);
> +            }
> +            bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN);
> +        }
> +
> +        Ok(topology)
> +    }
> +
> +    /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are
> +    /// root ports, on switches downstream ports.
> +    fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) {
> +        let root_port = matches!(addr.bus, Bus::Numa(_));
> +        let id = match node {
> +            Node::Device(index) if root_port => Bus::RootPort(*index),
> +            Node::Device(index) => Bus::SwitchPort(*index),
> +            Node::Switch(_) => Bus::SwitchUplink(self.switches),
> +        };
> +        let slot = node.key();
> +        let mut params = format!(",port={slot},chassis={chassis},slot={slot}");
> +        if root_port {
> +            params.push_str(",x-speed=16,x-width=32");
> +        }
> +        self.devices.push(device(id, addr, &params));
> +
> +        match node {
> +            Node::Device(index) => {
> +                self.addresses.insert(*index, PciAddress::new(id, 0, 0));
> +            }
> +            Node::Switch(children) => {
> +                let nr = self.switches;
> +                self.switches += 1;
> +
> +                let switch = Bus::Switch(nr);
> +                self.devices
> +                    .push(device(switch, PciAddress::new(id, 0, 0), ""));
> +                for (key, child) in children {
> +                    let addr = PciAddress::new(switch, *key, 0);
> +                    self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child);
> +                }
> +            }
> +        }
> +    }
> +}
> +

[... tests ...]
[... perlmod ..]




  reply	other threads:[~2026-11-19 13:43 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47   ` Elias Huhsovitz
2026-11-19 14:51     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43   ` Elias Huhsovitz [this message]
2026-11-19 15:10     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42   ` Elias Huhsovitz
2026-11-19 15:12     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33   ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com \
    --to=e.huhsovitz@proxmox.com \
    --cc=d.csapak@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal