all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Csapak <d.csapak@proxmox.com>
To: Elias Huhsovitz <e.huhsovitz@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host
Date: Thu, 19 Nov 2026 16:10:04 +0100	[thread overview]
Message-ID: <36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com> (raw)
In-Reply-To: <DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com>



On 11/19/26 2:43 PM, Elias Huhsovitz wrote:
>  From a high level view, I think the design is fitting.
> 
> It would be very interesting to measure the real world performance
> impact of the 3 layouts.
> 

yes, i'll benchmark that soon, hopefully can post results in the next 
few days...

will also include your current series then (will adopt it to this
series here locally if i can)

> Comments inline.
> 
> On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
>> Guest software sometimes uses the PCI topology to choose CPUs and DMA
>> paths. Since we put every hostpcie device on a root port of the root
>> bus, the guest can't see which devices are close to which NUMA node or
>> to each other.
>>
>> So add a hostpci topology setting for machines with a PCIe root bus
>> (q35 and aarch64 virt) and the grouped layout:
>> * 'flat': the current placement, a root port on the root bus per device
>> * 'numa': puts the root ports of each guest NUMA node's devices behind
>>    a pxb-pcie for that node
>> * 'host': additionally replicates the host PCIe switches that more
>>    than one passed through device is behind. We skip the other
>>    switches, since they don't add any information but cost bus numbers.
>>
>> The guest NUMA node and the host switches of each device are passed as
>> placements separately from the hardware config, since the caller only
>> knows them after choosing the host devices.
>>
>> All buses share the 256 bus numbers of a single PCI domain. We pack
>> the bus number ranges of the expander bridges by demand and align them
>> to 16, so a changed device normally only shifts its own node, and even
>> 16 devices always fit.
>>
>> Each port gets its own slot, numbered after the lowest hostpcie index
>> behind it, so its address does not depend on its siblings or on a
>> function 0 being present.
>>
>> In the bindings, add 'hostpcie_bus_devices', which takes the placements
>> and returns the expander bridges, switches and root ports, and let
>> 'addr' take them as an optional parameter, so existing callers don't
>> have to change. Since the topology is a required field, qemu-server
>> passes 'flat' until the option to choose it is added.
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>    
> [...]
> 
>> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result<PciAddress, LayoutError> {
>> -    let layout = layout(hw.has_pcie());
>> -    layout.find_device(&id.parse()?)
>> +pub(crate) fn address(
>> +    hw: &HardwareConfig,
>> +    id: &str,
>> +    placements: Option<&[HostpciePlacement]>,
>> +) -> Result<PciAddress, LayoutError> {
>> +    let device = id.parse()?;
>> +    match device {
>> +        Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => {
>> +            let placements = placements.ok_or(LayoutError::MissingPlacement(index))?;
>> +            let topology = Topology::new(hw, placements)?;
> 
> Isn't building a new topology for each address() call a bit excessive?
> IMO we should rebuild the topology only when necesarry.
> 
> Or do we run into some freshness issues in this case?

well we build it at max for (currently) 16 pcie devices, once per
startup. the overhead should't be that big (i guess a few µs to ms),
but yes, as a future improvement caching that would be good

> 
>> +            let addr = topology.addresses.get(&index).copied();
>> +            addr.ok_or(LayoutError::NoAddressFound(device))
>> +        }
>> +        _ => layout(hw.has_pcie()).find_device(&device),
>> +    }
>>   }
>>   
>>   pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option<Bus> {
>> -    let addr = address(hw, id).ok()?;
>> +    let addr = address(hw, id, None).ok()?;
>>       Some(addr.bus)
>>   }
>>   
> 
> [...]
> 
>> +
>> +/// How passed through PCIe devices are placed in the guest.
>> +///
>> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer
>> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device.
>> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub enum HostpciTopology {
>> +    /// Each device gets its own root port on the root bus.
>> +    #[default]
>> +    Flat,
>> +    /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node.
>> +    Numa,
>> +    /// Like `Numa`, but also replicates host switches with more than one device behind them.
>> +    Host,
>> +}
>> +
>> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other
>> +/// than [`HostpciTopology::Flat`].
>> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub struct HostpciePlacement {
>> +    /// The hostpci index of the device (e.g. '3' from 'hostpci3').
>> +    pub index: u8,
>> +    /// The guest NUMA node backed by the host NUMA node of the device.
>> +    #[serde(default)]
>> +    pub numa_node: u8,
>> +    /// The host bridges above the device, starting at the root port. Each one needs a unique id,
>> +    /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch,
>> +    /// so the devices have the same common upstream bridge as on the host. Only used with
>> +    /// [`HostpciTopology::Host`].
>> +    #[serde(default)]
>> +    pub switches: Vec<String>,
>> +}
>> +
>> +/// A device or replicated switch in the guest.
>> +enum Node {
>> +    Device(u8),
>> +    Switch(BTreeMap<u8, Node>),
>> +}
>> +
>> +impl Node {
>> +    /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the
>> +    /// address of a port does not depend on its siblings.
>> +    fn key(&self) -> u8 {
>> +        match self {
>> +            Node::Device(index) => *index,
>> +            Node::Switch(children) => *children.keys().next().unwrap(),
>> +        }
>> +    }
>> +}
>> +
>> +/// The devices and switches below a NUMA node or a switch on the host.
>> +#[derive(Default)]
>> +struct HostTree<'a> {
>> +    devices: Vec<u8>,
>> +    switches: BTreeMap<&'a str, HostTree<'a>>,
>> +}
> 
> nit: naming: IMO something like "HostNode" or "PciNode" fits better.
> Usually a "Tree" struct just wraps the root node (In Java at least)

sure, i have no strong opinion on naming these things anyway ;)

> 
>> +impl<'a> HostTree<'a> {
>> +    /// Adds `hostpcie<index>` below the given host switches.
>> +    fn insert(&mut self, index: u8, switches: &'a [String]) {
>> +        match switches.split_first() {
>> +            None => self.devices.push(index),
>> +            Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest),
>> +        }
>> +    }
>> +
>> +    /// Returns the guest nodes directly below this one, keyed by [`Node::key`].
>> +    fn children(self) -> BTreeMap<u8, Node> {
>> +        let devices = self.devices.into_iter().map(Node::Device);
>> +        let switches = self.switches.into_values().map(HostTree::into_node);
>> +        devices
>> +            .chain(switches)
>> +            .map(|node| (node.key(), node))
>> +            .collect()
>> +    }
>> +
>> +    /// Converts the switch to a guest node. Switches with only one child are replaced by that
>> +    /// child, since they only use up bus numbers without giving the guest any information.
>> +    fn into_node(self) -> Node {
>> +        let mut children = self.children();
>> +        if children.len() == 1 {
>> +            children.pop_first().unwrap().1
>> +        } else {
>> +            Node::Switch(children)
>> +        }
>> +    }
>> +}
>> +
>> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured
>> +/// are ignored.
>> +fn numa_nodes<'a>(
>> +    hw: &HardwareConfig,
>> +    placements: &'a [HostpciePlacement],
>> +) -> Result<BTreeMap<u8, HostTree<'a>>, LayoutError> {
>> +    let placements: BTreeMap<u8, &HostpciePlacement> =
>> +        placements.iter().map(|p| (p.index, p)).collect();
>> +    let mut nodes: BTreeMap<u8, HostTree> = BTreeMap::new();
>> +    let mut parents = BTreeMap::new();
>> +
>> +    for &index in hw.hostpcie_devices.keys() {
>> +        let device = placements
>> +            .get(&index)
>> +            .ok_or(LayoutError::MissingPlacement(index))?;
>> +        if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES {
>> +            return Err(LayoutError::InvalidPlacement(index));
>> +        }
>> +        let switches: &[String] = match hw.hostpci_topology {
>> +            HostpciTopology::Host => &device.switches,
>> +            _ => &[],
>> +        };
>> +        // switches must be nested identically for all devices (e.g. switch X can't be behind Y for
>> +        // one device and behind Z for another)
>> +        for (depth, switch) in switches.iter().enumerate() {
>> +            let parent = (device.numa_node, &switches[..depth]);
>> +            if *parents.entry(switch).or_insert(parent) != parent {
>> +                return Err(LayoutError::InvalidPlacement(index));
>> +            }
>> +        }
>> +        nodes
>> +            .entry(device.numa_node)
>> +            .or_default()
>> +            .insert(index, switches);
>> +    }
>> +
>> +    Ok(nodes)
>> +}
>> +
>> +/// Returns the `-device` value for the device creating `bus` at `addr`.
>> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String {
>> +    // all buses of the topology are created by a device
>> +    let model = bus.bridge_model(addr.bus).unwrap();
>> +    format!("{model},id={bus}{}{params}", addr.to_qemu_addr())
>> +}
>> +
>> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses.
>> +pub(crate) struct Topology {
>> +    /// `-device` values for all expander bridges, ports and switches, each bus before the devices
>> +    /// on it.
>> +    pub(crate) devices: Vec<String>,
>> +    /// Addresses of the `hostpcie` devices, keyed by index.
>> +    pub(crate) addresses: BTreeMap<u8, PciAddress>,
>> +    switches: u8,
>> +}
>> +
>> +impl Topology {
>> +    /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`].
>> +    pub(crate) fn new(
>> +        hw: &HardwareConfig,
>> +        placements: &[HostpciePlacement],
>> +    ) -> Result<Self, LayoutError> {
>> +        if !hw.has_pcie() || hw.layout != PciLayout::Grouped {
>> +            return Err(LayoutError::UnsupportedTopology);
>> +        }
>> +
>> +        let mut topology = Self {
>> +            devices: Vec::new(),
>> +            addresses: BTreeMap::new(),
>> +            switches: 0,
>> +        };
>> +        let mut bus_nr = FIRST_NUMA_BUS;
>> +        for (numa_node, tree) in numa_nodes(hw, placements)? {
>> +            let start = topology.devices.len();
>> +            let bus = Bus::Numa(numa_node);
>> +            let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?;
> 
> Why is the PCIE_Layout always the grouped layout here? I understand a
> non-flat topology currently only makes sense in the context of a
> grouped layout, but shouldnt we keep this somewhat generic so we
> aren't locked into the grouped layout?
> 
> IMO we can just pass the layout in the constructor of Topology.

i mean we can still change that later down the line if necessary.
The interface is internal to this crate, so changing it
wouldn't break any outside caller

if i have to send a new version i can do it though

> 
>> +            let params = format!(",bus_nr={bus_nr},numa_node={numa_node}");
>> +            topology.devices.push(device(bus, addr, &params));
>> +            for (key, node) in tree.children() {
>> +                // keep slot 0 free, like on all other bridges
>> +                let addr = PciAddress::new(bus, key + 1, 0);
>> +                topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node);
>> +            }
>> +
>> +            // each device we added creates exactly one bus
>> +            let buses = topology.devices.len() - start;
>> +            if bus_nr + buses > 256 {
>> +                return Err(LayoutError::BusNumbersExhausted);
>> +            }
>> +            bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN);
>> +        }
>> +
>> +        Ok(topology)
>> +    }
>> +
>> +    /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are
>> +    /// root ports, on switches downstream ports.
>> +    fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) {
>> +        let root_port = matches!(addr.bus, Bus::Numa(_));
>> +        let id = match node {
>> +            Node::Device(index) if root_port => Bus::RootPort(*index),
>> +            Node::Device(index) => Bus::SwitchPort(*index),
>> +            Node::Switch(_) => Bus::SwitchUplink(self.switches),
>> +        };
>> +        let slot = node.key();
>> +        let mut params = format!(",port={slot},chassis={chassis},slot={slot}");
>> +        if root_port {
>> +            params.push_str(",x-speed=16,x-width=32");
>> +        }
>> +        self.devices.push(device(id, addr, &params));
>> +
>> +        match node {
>> +            Node::Device(index) => {
>> +                self.addresses.insert(*index, PciAddress::new(id, 0, 0));
>> +            }
>> +            Node::Switch(children) => {
>> +                let nr = self.switches;
>> +                self.switches += 1;
>> +
>> +                let switch = Bus::Switch(nr);
>> +                self.devices
>> +                    .push(device(switch, PciAddress::new(id, 0, 0), ""));
>> +                for (key, child) in children {
>> +                    let addr = PciAddress::new(switch, *key, 0);
>> +                    self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child);
>> +                }
>> +            }
>> +        }
>> +    }
>> +}
>> +
> 
> [... tests ...]
> [... perlmod ..]





  reply	other threads:[~2026-11-19 15:10 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47   ` Elias Huhsovitz
2026-11-19 14:51     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43   ` Elias Huhsovitz
2026-11-19 15:10     ` Dominik Csapak [this message]
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42   ` Elias Huhsovitz
2026-11-19 15:12     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33   ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com \
    --to=d.csapak@proxmox.com \
    --cc=e.huhsovitz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal