public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Csapak <d.csapak@proxmox.com>
To: Elias Huhsovitz <e.huhsovitz@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host
Date: Thu, 19 Nov 2026 16:10:04 +0100	[thread overview]
Message-ID: <36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com> (raw)
In-Reply-To: <DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com>



On 11/19/26 2:43 PM, Elias Huhsovitz wrote:
>  From a high level view, I think the design is fitting.
> 
> It would be very interesting to measure the real world performance
> impact of the 3 layouts.
> 

yes, i'll benchmark that soon, hopefully can post results in the next 
few days...

will also include your current series then (will adopt it to this
series here locally if i can)

> Comments inline.
> 
> On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
>> Guest software sometimes uses the PCI topology to choose CPUs and DMA
>> paths. Since we put every hostpcie device on a root port of the root
>> bus, the guest can't see which devices are close to which NUMA node or
>> to each other.
>>
>> So add a hostpci topology setting for machines with a PCIe root bus
>> (q35 and aarch64 virt) and the grouped layout:
>> * 'flat': the current placement, a root port on the root bus per device
>> * 'numa': puts the root ports of each guest NUMA node's devices behind
>>    a pxb-pcie for that node
>> * 'host': additionally replicates the host PCIe switches that more
>>    than one passed through device is behind. We skip the other
>>    switches, since they don't add any information but cost bus numbers.
>>
>> The guest NUMA node and the host switches of each device are passed as
>> placements separately from the hardware config, since the caller only
>> knows them after choosing the host devices.
>>
>> All buses share the 256 bus numbers of a single PCI domain. We pack
>> the bus number ranges of the expander bridges by demand and align them
>> to 16, so a changed device normally only shifts its own node, and even
>> 16 devices always fit.
>>
>> Each port gets its own slot, numbered after the lowest hostpcie index
>> behind it, so its address does not depend on its siblings or on a
>> function 0 being present.
>>
>> In the bindings, add 'hostpcie_bus_devices', which takes the placements
>> and returns the expander bridges, switches and root ports, and let
>> 'addr' take them as an optional parameter, so existing callers don't
>> have to change. Since the topology is a required field, qemu-server
>> passes 'flat' until the option to choose it is added.
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>    
> [...]
> 
>> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result<PciAddress, LayoutError> {
>> -    let layout = layout(hw.has_pcie());
>> -    layout.find_device(&id.parse()?)
>> +pub(crate) fn address(
>> +    hw: &HardwareConfig,
>> +    id: &str,
>> +    placements: Option<&[HostpciePlacement]>,
>> +) -> Result<PciAddress, LayoutError> {
>> +    let device = id.parse()?;
>> +    match device {
>> +        Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => {
>> +            let placements = placements.ok_or(LayoutError::MissingPlacement(index))?;
>> +            let topology = Topology::new(hw, placements)?;
> 
> Isn't building a new topology for each address() call a bit excessive?
> IMO we should rebuild the topology only when necesarry.
> 
> Or do we run into some freshness issues in this case?

well we build it at max for (currently) 16 pcie devices, once per
startup. the overhead should't be that big (i guess a few µs to ms),
but yes, as a future improvement caching that would be good

> 
>> +            let addr = topology.addresses.get(&index).copied();
>> +            addr.ok_or(LayoutError::NoAddressFound(device))
>> +        }
>> +        _ => layout(hw.has_pcie()).find_device(&device),
>> +    }
>>   }
>>   
>>   pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option<Bus> {
>> -    let addr = address(hw, id).ok()?;
>> +    let addr = address(hw, id, None).ok()?;
>>       Some(addr.bus)
>>   }
>>   
> 
> [...]
> 
>> +
>> +/// How passed through PCIe devices are placed in the guest.
>> +///
>> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer
>> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device.
>> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub enum HostpciTopology {
>> +    /// Each device gets its own root port on the root bus.
>> +    #[default]
>> +    Flat,
>> +    /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node.
>> +    Numa,
>> +    /// Like `Numa`, but also replicates host switches with more than one device behind them.
>> +    Host,
>> +}
>> +
>> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other
>> +/// than [`HostpciTopology::Flat`].
>> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub struct HostpciePlacement {
>> +    /// The hostpci index of the device (e.g. '3' from 'hostpci3').
>> +    pub index: u8,
>> +    /// The guest NUMA node backed by the host NUMA node of the device.
>> +    #[serde(default)]
>> +    pub numa_node: u8,
>> +    /// The host bridges above the device, starting at the root port. Each one needs a unique id,
>> +    /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch,
>> +    /// so the devices have the same common upstream bridge as on the host. Only used with
>> +    /// [`HostpciTopology::Host`].
>> +    #[serde(default)]
>> +    pub switches: Vec<String>,
>> +}
>> +
>> +/// A device or replicated switch in the guest.
>> +enum Node {
>> +    Device(u8),
>> +    Switch(BTreeMap<u8, Node>),
>> +}
>> +
>> +impl Node {
>> +    /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the
>> +    /// address of a port does not depend on its siblings.
>> +    fn key(&self) -> u8 {
>> +        match self {
>> +            Node::Device(index) => *index,
>> +            Node::Switch(children) => *children.keys().next().unwrap(),
>> +        }
>> +    }
>> +}
>> +
>> +/// The devices and switches below a NUMA node or a switch on the host.
>> +#[derive(Default)]
>> +struct HostTree<'a> {
>> +    devices: Vec<u8>,
>> +    switches: BTreeMap<&'a str, HostTree<'a>>,
>> +}
> 
> nit: naming: IMO something like "HostNode" or "PciNode" fits better.
> Usually a "Tree" struct just wraps the root node (In Java at least)

sure, i have no strong opinion on naming these things anyway ;)

> 
>> +impl<'a> HostTree<'a> {
>> +    /// Adds `hostpcie<index>` below the given host switches.
>> +    fn insert(&mut self, index: u8, switches: &'a [String]) {
>> +        match switches.split_first() {
>> +            None => self.devices.push(index),
>> +            Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest),
>> +        }
>> +    }
>> +
>> +    /// Returns the guest nodes directly below this one, keyed by [`Node::key`].
>> +    fn children(self) -> BTreeMap<u8, Node> {
>> +        let devices = self.devices.into_iter().map(Node::Device);
>> +        let switches = self.switches.into_values().map(HostTree::into_node);
>> +        devices
>> +            .chain(switches)
>> +            .map(|node| (node.key(), node))
>> +            .collect()
>> +    }
>> +
>> +    /// Converts the switch to a guest node. Switches with only one child are replaced by that
>> +    /// child, since they only use up bus numbers without giving the guest any information.
>> +    fn into_node(self) -> Node {
>> +        let mut children = self.children();
>> +        if children.len() == 1 {
>> +            children.pop_first().unwrap().1
>> +        } else {
>> +            Node::Switch(children)
>> +        }
>> +    }
>> +}
>> +
>> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured
>> +/// are ignored.
>> +fn numa_nodes<'a>(
>> +    hw: &HardwareConfig,
>> +    placements: &'a [HostpciePlacement],
>> +) -> Result<BTreeMap<u8, HostTree<'a>>, LayoutError> {
>> +    let placements: BTreeMap<u8, &HostpciePlacement> =
>> +        placements.iter().map(|p| (p.index, p)).collect();
>> +    let mut nodes: BTreeMap<u8, HostTree> = BTreeMap::new();
>> +    let mut parents = BTreeMap::new();
>> +
>> +    for &index in hw.hostpcie_devices.keys() {
>> +        let device = placements
>> +            .get(&index)
>> +            .ok_or(LayoutError::MissingPlacement(index))?;
>> +        if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES {
>> +            return Err(LayoutError::InvalidPlacement(index));
>> +        }
>> +        let switches: &[String] = match hw.hostpci_topology {
>> +            HostpciTopology::Host => &device.switches,
>> +            _ => &[],
>> +        };
>> +        // switches must be nested identically for all devices (e.g. switch X can't be behind Y for
>> +        // one device and behind Z for another)
>> +        for (depth, switch) in switches.iter().enumerate() {
>> +            let parent = (device.numa_node, &switches[..depth]);
>> +            if *parents.entry(switch).or_insert(parent) != parent {
>> +                return Err(LayoutError::InvalidPlacement(index));
>> +            }
>> +        }
>> +        nodes
>> +            .entry(device.numa_node)
>> +            .or_default()
>> +            .insert(index, switches);
>> +    }
>> +
>> +    Ok(nodes)
>> +}
>> +
>> +/// Returns the `-device` value for the device creating `bus` at `addr`.
>> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String {
>> +    // all buses of the topology are created by a device
>> +    let model = bus.bridge_model(addr.bus).unwrap();
>> +    format!("{model},id={bus}{}{params}", addr.to_qemu_addr())
>> +}
>> +
>> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses.
>> +pub(crate) struct Topology {
>> +    /// `-device` values for all expander bridges, ports and switches, each bus before the devices
>> +    /// on it.
>> +    pub(crate) devices: Vec<String>,
>> +    /// Addresses of the `hostpcie` devices, keyed by index.
>> +    pub(crate) addresses: BTreeMap<u8, PciAddress>,
>> +    switches: u8,
>> +}
>> +
>> +impl Topology {
>> +    /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`].
>> +    pub(crate) fn new(
>> +        hw: &HardwareConfig,
>> +        placements: &[HostpciePlacement],
>> +    ) -> Result<Self, LayoutError> {
>> +        if !hw.has_pcie() || hw.layout != PciLayout::Grouped {
>> +            return Err(LayoutError::UnsupportedTopology);
>> +        }
>> +
>> +        let mut topology = Self {
>> +            devices: Vec::new(),
>> +            addresses: BTreeMap::new(),
>> +            switches: 0,
>> +        };
>> +        let mut bus_nr = FIRST_NUMA_BUS;
>> +        for (numa_node, tree) in numa_nodes(hw, placements)? {
>> +            let start = topology.devices.len();
>> +            let bus = Bus::Numa(numa_node);
>> +            let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?;
> 
> Why is the PCIE_Layout always the grouped layout here? I understand a
> non-flat topology currently only makes sense in the context of a
> grouped layout, but shouldnt we keep this somewhat generic so we
> aren't locked into the grouped layout?
> 
> IMO we can just pass the layout in the constructor of Topology.

i mean we can still change that later down the line if necessary.
The interface is internal to this crate, so changing it
wouldn't break any outside caller

if i have to send a new version i can do it though

> 
>> +            let params = format!(",bus_nr={bus_nr},numa_node={numa_node}");
>> +            topology.devices.push(device(bus, addr, &params));
>> +            for (key, node) in tree.children() {
>> +                // keep slot 0 free, like on all other bridges
>> +                let addr = PciAddress::new(bus, key + 1, 0);
>> +                topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node);
>> +            }
>> +
>> +            // each device we added creates exactly one bus
>> +            let buses = topology.devices.len() - start;
>> +            if bus_nr + buses > 256 {
>> +                return Err(LayoutError::BusNumbersExhausted);
>> +            }
>> +            bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN);
>> +        }
>> +
>> +        Ok(topology)
>> +    }
>> +
>> +    /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are
>> +    /// root ports, on switches downstream ports.
>> +    fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) {
>> +        let root_port = matches!(addr.bus, Bus::Numa(_));
>> +        let id = match node {
>> +            Node::Device(index) if root_port => Bus::RootPort(*index),
>> +            Node::Device(index) => Bus::SwitchPort(*index),
>> +            Node::Switch(_) => Bus::SwitchUplink(self.switches),
>> +        };
>> +        let slot = node.key();
>> +        let mut params = format!(",port={slot},chassis={chassis},slot={slot}");
>> +        if root_port {
>> +            params.push_str(",x-speed=16,x-width=32");
>> +        }
>> +        self.devices.push(device(id, addr, &params));
>> +
>> +        match node {
>> +            Node::Device(index) => {
>> +                self.addresses.insert(*index, PciAddress::new(id, 0, 0));
>> +            }
>> +            Node::Switch(children) => {
>> +                let nr = self.switches;
>> +                self.switches += 1;
>> +
>> +                let switch = Bus::Switch(nr);
>> +                self.devices
>> +                    .push(device(switch, PciAddress::new(id, 0, 0), ""));
>> +                for (key, child) in children {
>> +                    let addr = PciAddress::new(switch, *key, 0);
>> +                    self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child);
>> +                }
>> +            }
>> +        }
>> +    }
>> +}
>> +
> 
> [... tests ...]
> [... perlmod ..]





  reply	other threads:[~2026-11-19 15:10 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47   ` Elias Huhsovitz
2026-11-19 14:51     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43   ` Elias Huhsovitz
2026-11-19 15:10     ` Dominik Csapak [this message]
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42   ` Elias Huhsovitz
2026-11-19 15:12     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33   ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com \
    --to=d.csapak@proxmox.com \
    --cc=e.huhsovitz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal