From: Dominik Csapak <d.csapak@proxmox.com>
To: Elias Huhsovitz <e.huhsovitz@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host
Date: Thu, 19 Nov 2026 16:10:04 +0100 [thread overview]
Message-ID: <36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com> (raw)
In-Reply-To: <DMZ8D1J1B8NQ.71RHPHMICRU7@proxmox.com>
On 11/19/26 2:43 PM, Elias Huhsovitz wrote:
> From a high level view, I think the design is fitting.
>
> It would be very interesting to measure the real world performance
> impact of the 3 layouts.
>
yes, i'll benchmark that soon, hopefully can post results in the next
few days...
will also include your current series then (will adopt it to this
series here locally if i can)
> Comments inline.
>
> On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
>> Guest software sometimes uses the PCI topology to choose CPUs and DMA
>> paths. Since we put every hostpcie device on a root port of the root
>> bus, the guest can't see which devices are close to which NUMA node or
>> to each other.
>>
>> So add a hostpci topology setting for machines with a PCIe root bus
>> (q35 and aarch64 virt) and the grouped layout:
>> * 'flat': the current placement, a root port on the root bus per device
>> * 'numa': puts the root ports of each guest NUMA node's devices behind
>> a pxb-pcie for that node
>> * 'host': additionally replicates the host PCIe switches that more
>> than one passed through device is behind. We skip the other
>> switches, since they don't add any information but cost bus numbers.
>>
>> The guest NUMA node and the host switches of each device are passed as
>> placements separately from the hardware config, since the caller only
>> knows them after choosing the host devices.
>>
>> All buses share the 256 bus numbers of a single PCI domain. We pack
>> the bus number ranges of the expander bridges by demand and align them
>> to 16, so a changed device normally only shifts its own node, and even
>> 16 devices always fit.
>>
>> Each port gets its own slot, numbered after the lowest hostpcie index
>> behind it, so its address does not depend on its siblings or on a
>> function 0 being present.
>>
>> In the bindings, add 'hostpcie_bus_devices', which takes the placements
>> and returns the expander bridges, switches and root ports, and let
>> 'addr' take them as an optional parameter, so existing callers don't
>> have to change. Since the topology is a required field, qemu-server
>> passes 'flat' until the option to choose it is added.
>>
>> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
>> ---
>
> [...]
>
>> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result<PciAddress, LayoutError> {
>> - let layout = layout(hw.has_pcie());
>> - layout.find_device(&id.parse()?)
>> +pub(crate) fn address(
>> + hw: &HardwareConfig,
>> + id: &str,
>> + placements: Option<&[HostpciePlacement]>,
>> +) -> Result<PciAddress, LayoutError> {
>> + let device = id.parse()?;
>> + match device {
>> + Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => {
>> + let placements = placements.ok_or(LayoutError::MissingPlacement(index))?;
>> + let topology = Topology::new(hw, placements)?;
>
> Isn't building a new topology for each address() call a bit excessive?
> IMO we should rebuild the topology only when necesarry.
>
> Or do we run into some freshness issues in this case?
well we build it at max for (currently) 16 pcie devices, once per
startup. the overhead should't be that big (i guess a few µs to ms),
but yes, as a future improvement caching that would be good
>
>> + let addr = topology.addresses.get(&index).copied();
>> + addr.ok_or(LayoutError::NoAddressFound(device))
>> + }
>> + _ => layout(hw.has_pcie()).find_device(&device),
>> + }
>> }
>>
>> pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option<Bus> {
>> - let addr = address(hw, id).ok()?;
>> + let addr = address(hw, id, None).ok()?;
>> Some(addr.bus)
>> }
>>
>
> [...]
>
>> +
>> +/// How passed through PCIe devices are placed in the guest.
>> +///
>> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer
>> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device.
>> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub enum HostpciTopology {
>> + /// Each device gets its own root port on the root bus.
>> + #[default]
>> + Flat,
>> + /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node.
>> + Numa,
>> + /// Like `Numa`, but also replicates host switches with more than one device behind them.
>> + Host,
>> +}
>> +
>> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other
>> +/// than [`HostpciTopology::Flat`].
>> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)]
>> +#[serde(rename_all = "kebab-case")]
>> +pub struct HostpciePlacement {
>> + /// The hostpci index of the device (e.g. '3' from 'hostpci3').
>> + pub index: u8,
>> + /// The guest NUMA node backed by the host NUMA node of the device.
>> + #[serde(default)]
>> + pub numa_node: u8,
>> + /// The host bridges above the device, starting at the root port. Each one needs a unique id,
>> + /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch,
>> + /// so the devices have the same common upstream bridge as on the host. Only used with
>> + /// [`HostpciTopology::Host`].
>> + #[serde(default)]
>> + pub switches: Vec<String>,
>> +}
>> +
>> +/// A device or replicated switch in the guest.
>> +enum Node {
>> + Device(u8),
>> + Switch(BTreeMap<u8, Node>),
>> +}
>> +
>> +impl Node {
>> + /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the
>> + /// address of a port does not depend on its siblings.
>> + fn key(&self) -> u8 {
>> + match self {
>> + Node::Device(index) => *index,
>> + Node::Switch(children) => *children.keys().next().unwrap(),
>> + }
>> + }
>> +}
>> +
>> +/// The devices and switches below a NUMA node or a switch on the host.
>> +#[derive(Default)]
>> +struct HostTree<'a> {
>> + devices: Vec<u8>,
>> + switches: BTreeMap<&'a str, HostTree<'a>>,
>> +}
>
> nit: naming: IMO something like "HostNode" or "PciNode" fits better.
> Usually a "Tree" struct just wraps the root node (In Java at least)
sure, i have no strong opinion on naming these things anyway ;)
>
>> +impl<'a> HostTree<'a> {
>> + /// Adds `hostpcie<index>` below the given host switches.
>> + fn insert(&mut self, index: u8, switches: &'a [String]) {
>> + match switches.split_first() {
>> + None => self.devices.push(index),
>> + Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest),
>> + }
>> + }
>> +
>> + /// Returns the guest nodes directly below this one, keyed by [`Node::key`].
>> + fn children(self) -> BTreeMap<u8, Node> {
>> + let devices = self.devices.into_iter().map(Node::Device);
>> + let switches = self.switches.into_values().map(HostTree::into_node);
>> + devices
>> + .chain(switches)
>> + .map(|node| (node.key(), node))
>> + .collect()
>> + }
>> +
>> + /// Converts the switch to a guest node. Switches with only one child are replaced by that
>> + /// child, since they only use up bus numbers without giving the guest any information.
>> + fn into_node(self) -> Node {
>> + let mut children = self.children();
>> + if children.len() == 1 {
>> + children.pop_first().unwrap().1
>> + } else {
>> + Node::Switch(children)
>> + }
>> + }
>> +}
>> +
>> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured
>> +/// are ignored.
>> +fn numa_nodes<'a>(
>> + hw: &HardwareConfig,
>> + placements: &'a [HostpciePlacement],
>> +) -> Result<BTreeMap<u8, HostTree<'a>>, LayoutError> {
>> + let placements: BTreeMap<u8, &HostpciePlacement> =
>> + placements.iter().map(|p| (p.index, p)).collect();
>> + let mut nodes: BTreeMap<u8, HostTree> = BTreeMap::new();
>> + let mut parents = BTreeMap::new();
>> +
>> + for &index in hw.hostpcie_devices.keys() {
>> + let device = placements
>> + .get(&index)
>> + .ok_or(LayoutError::MissingPlacement(index))?;
>> + if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES {
>> + return Err(LayoutError::InvalidPlacement(index));
>> + }
>> + let switches: &[String] = match hw.hostpci_topology {
>> + HostpciTopology::Host => &device.switches,
>> + _ => &[],
>> + };
>> + // switches must be nested identically for all devices (e.g. switch X can't be behind Y for
>> + // one device and behind Z for another)
>> + for (depth, switch) in switches.iter().enumerate() {
>> + let parent = (device.numa_node, &switches[..depth]);
>> + if *parents.entry(switch).or_insert(parent) != parent {
>> + return Err(LayoutError::InvalidPlacement(index));
>> + }
>> + }
>> + nodes
>> + .entry(device.numa_node)
>> + .or_default()
>> + .insert(index, switches);
>> + }
>> +
>> + Ok(nodes)
>> +}
>> +
>> +/// Returns the `-device` value for the device creating `bus` at `addr`.
>> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String {
>> + // all buses of the topology are created by a device
>> + let model = bus.bridge_model(addr.bus).unwrap();
>> + format!("{model},id={bus}{}{params}", addr.to_qemu_addr())
>> +}
>> +
>> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses.
>> +pub(crate) struct Topology {
>> + /// `-device` values for all expander bridges, ports and switches, each bus before the devices
>> + /// on it.
>> + pub(crate) devices: Vec<String>,
>> + /// Addresses of the `hostpcie` devices, keyed by index.
>> + pub(crate) addresses: BTreeMap<u8, PciAddress>,
>> + switches: u8,
>> +}
>> +
>> +impl Topology {
>> + /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`].
>> + pub(crate) fn new(
>> + hw: &HardwareConfig,
>> + placements: &[HostpciePlacement],
>> + ) -> Result<Self, LayoutError> {
>> + if !hw.has_pcie() || hw.layout != PciLayout::Grouped {
>> + return Err(LayoutError::UnsupportedTopology);
>> + }
>> +
>> + let mut topology = Self {
>> + devices: Vec::new(),
>> + addresses: BTreeMap::new(),
>> + switches: 0,
>> + };
>> + let mut bus_nr = FIRST_NUMA_BUS;
>> + for (numa_node, tree) in numa_nodes(hw, placements)? {
>> + let start = topology.devices.len();
>> + let bus = Bus::Numa(numa_node);
>> + let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?;
>
> Why is the PCIE_Layout always the grouped layout here? I understand a
> non-flat topology currently only makes sense in the context of a
> grouped layout, but shouldnt we keep this somewhat generic so we
> aren't locked into the grouped layout?
>
> IMO we can just pass the layout in the constructor of Topology.
i mean we can still change that later down the line if necessary.
The interface is internal to this crate, so changing it
wouldn't break any outside caller
if i have to send a new version i can do it though
>
>> + let params = format!(",bus_nr={bus_nr},numa_node={numa_node}");
>> + topology.devices.push(device(bus, addr, ¶ms));
>> + for (key, node) in tree.children() {
>> + // keep slot 0 free, like on all other bridges
>> + let addr = PciAddress::new(bus, key + 1, 0);
>> + topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node);
>> + }
>> +
>> + // each device we added creates exactly one bus
>> + let buses = topology.devices.len() - start;
>> + if bus_nr + buses > 256 {
>> + return Err(LayoutError::BusNumbersExhausted);
>> + }
>> + bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN);
>> + }
>> +
>> + Ok(topology)
>> + }
>> +
>> + /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are
>> + /// root ports, on switches downstream ports.
>> + fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) {
>> + let root_port = matches!(addr.bus, Bus::Numa(_));
>> + let id = match node {
>> + Node::Device(index) if root_port => Bus::RootPort(*index),
>> + Node::Device(index) => Bus::SwitchPort(*index),
>> + Node::Switch(_) => Bus::SwitchUplink(self.switches),
>> + };
>> + let slot = node.key();
>> + let mut params = format!(",port={slot},chassis={chassis},slot={slot}");
>> + if root_port {
>> + params.push_str(",x-speed=16,x-width=32");
>> + }
>> + self.devices.push(device(id, addr, ¶ms));
>> +
>> + match node {
>> + Node::Device(index) => {
>> + self.addresses.insert(*index, PciAddress::new(id, 0, 0));
>> + }
>> + Node::Switch(children) => {
>> + let nr = self.switches;
>> + self.switches += 1;
>> +
>> + let switch = Bus::Switch(nr);
>> + self.devices
>> + .push(device(switch, PciAddress::new(id, 0, 0), ""));
>> + for (key, child) in children {
>> + let addr = PciAddress::new(switch, *key, 0);
>> + self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child);
>> + }
>> + }
>> + }
>> + }
>> +}
>> +
>
> [... tests ...]
> [... perlmod ..]
next prev parent reply other threads:[~2026-11-19 15:10 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47 ` Elias Huhsovitz
2026-11-19 14:51 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43 ` Elias Huhsovitz
2026-11-19 15:10 ` Dominik Csapak [this message]
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42 ` Elias Huhsovitz
2026-11-19 15:12 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33 ` Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=e.huhsovitz@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox