From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 88A6E1FF0DE for ; Thu, 19 Nov 2026 14:43:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 21CE72169B; Thu, 19 Nov 2026 14:43:51 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Nov 2026 14:43:45 +0100 Message-Id: Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host From: "Elias Huhsovitz" To: "Dominik Csapak" , Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 X-Mailer: aerc 0.20.0 References: <20261118152803.3843758-1-d.csapak@proxmox.com> <20261118152803.3843758-12-d.csapak@proxmox.com> In-Reply-To: <20261118152803.3843758-12-d.csapak@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795095825158 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.403 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WOCTKB34S4JB3AXR3NKTKD6XN5I3RM7L X-Message-ID-Hash: WOCTKB34S4JB3AXR3NKTKD6XN5I3RM7L X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: >>From a high level view, I think the design is fitting. It would be very interesting to measure the real world performance impact of the 3 layouts. Comments inline. On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: > Guest software sometimes uses the PCI topology to choose CPUs and DMA > paths. Since we put every hostpcie device on a root port of the root > bus, the guest can't see which devices are close to which NUMA node or > to each other. > > So add a hostpci topology setting for machines with a PCIe root bus > (q35 and aarch64 virt) and the grouped layout: > * 'flat': the current placement, a root port on the root bus per device > * 'numa': puts the root ports of each guest NUMA node's devices behind > a pxb-pcie for that node > * 'host': additionally replicates the host PCIe switches that more > than one passed through device is behind. We skip the other > switches, since they don't add any information but cost bus numbers. > > The guest NUMA node and the host switches of each device are passed as > placements separately from the hardware config, since the caller only > knows them after choosing the host devices. > > All buses share the 256 bus numbers of a single PCI domain. We pack > the bus number ranges of the expander bridges by demand and align them > to 16, so a changed device normally only shifts its own node, and even > 16 devices always fit. > > Each port gets its own slot, numbered after the lowest hostpcie index > behind it, so its address does not depend on its siblings or on a > function 0 being present. > > In the bindings, add 'hostpcie_bus_devices', which takes the placements > and returns the expander bridges, switches and root ports, and let > 'addr' take them as an optional parameter, so existing callers don't > have to change. Since the topology is a required field, qemu-server > passes 'flat' until the option to choose it is added. > > Signed-off-by: Dominik Csapak > --- =20 [...] > -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result { > - let layout =3D layout(hw.has_pcie()); > - layout.find_device(&id.parse()?) > +pub(crate) fn address( > + hw: &HardwareConfig, > + id: &str, > + placements: Option<&[HostpciePlacement]>, > +) -> Result { > + let device =3D id.parse()?; > + match device { > + Device::Hostpcie(index) if hw.hostpci_topology !=3D HostpciTopol= ogy::Flat =3D> { > + let placements =3D placements.ok_or(LayoutError::MissingPlac= ement(index))?; > + let topology =3D Topology::new(hw, placements)?; Isn't building a new topology for each address() call a bit excessive? IMO we should rebuild the topology only when necesarry. Or do we run into some freshness issues in this case? > + let addr =3D topology.addresses.get(&index).copied(); > + addr.ok_or(LayoutError::NoAddressFound(device)) > + } > + _ =3D> layout(hw.has_pcie()).find_device(&device), > + } > } > =20 > pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Op= tion { > - let addr =3D address(hw, id).ok()?; > + let addr =3D address(hw, id, None).ok()?; > Some(addr.bus) > } > =20 [...] > + > +/// How passed through PCIe devices are placed in the guest. > +/// > +/// Drivers for accelerators and RDMA NICs use the PCI topology to choos= e CPUs and peer-to-peer > +/// paths, so we replicate the host topology from the [`HostpciePlacemen= t`] of each device. > +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)] > +#[serde(rename_all =3D "kebab-case")] > +pub enum HostpciTopology { > + /// Each device gets its own root port on the root bus. > + #[default] > + Flat, > + /// Groups the root ports behind a `pxb-pcie` expander bridge per gu= est NUMA node. > + Numa, > + /// Like `Numa`, but also replicates host switches with more than on= e device behind them. > + Host, > +} > + > +/// Where a passed through PCIe device is connected on the host. Only ne= eded for topologies other > +/// than [`HostpciTopology::Flat`]. > +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)] > +#[serde(rename_all =3D "kebab-case")] > +pub struct HostpciePlacement { > + /// The hostpci index of the device (e.g. '3' from 'hostpci3'). > + pub index: u8, > + /// The guest NUMA node backed by the host NUMA node of the device. > + #[serde(default)] > + pub numa_node: u8, > + /// The host bridges above the device, starting at the root port. Ea= ch one needs a unique id, > + /// e.g. its PCI address. Bridges shared by more than one device are= replicated as a switch, > + /// so the devices have the same common upstream bridge as on the ho= st. Only used with > + /// [`HostpciTopology::Host`]. > + #[serde(default)] > + pub switches: Vec, > +} > + > +/// A device or replicated switch in the guest. > +enum Node { > + Device(u8), > + Switch(BTreeMap), > +} > + > +impl Node { > + /// Returns the lowest `hostpcie` index below this node. We use it a= s slot number, so the > + /// address of a port does not depend on its siblings. > + fn key(&self) -> u8 { > + match self { > + Node::Device(index) =3D> *index, > + Node::Switch(children) =3D> *children.keys().next().unwrap()= , > + } > + } > +} > + > +/// The devices and switches below a NUMA node or a switch on the host. > +#[derive(Default)] > +struct HostTree<'a> { > + devices: Vec, > + switches: BTreeMap<&'a str, HostTree<'a>>, > +} nit: naming: IMO something like "HostNode" or "PciNode" fits better. Usually a "Tree" struct just wraps the root node (In Java at least) > +impl<'a> HostTree<'a> { > + /// Adds `hostpcie` below the given host switches. > + fn insert(&mut self, index: u8, switches: &'a [String]) { > + match switches.split_first() { > + None =3D> self.devices.push(index), > + Some((first, rest)) =3D> self.switches.entry(first).or_defau= lt().insert(index, rest), > + } > + } > + > + /// Returns the guest nodes directly below this one, keyed by [`Node= ::key`]. > + fn children(self) -> BTreeMap { > + let devices =3D self.devices.into_iter().map(Node::Device); > + let switches =3D self.switches.into_values().map(HostTree::into_= node); > + devices > + .chain(switches) > + .map(|node| (node.key(), node)) > + .collect() > + } > + > + /// Converts the switch to a guest node. Switches with only one chil= d are replaced by that > + /// child, since they only use up bus numbers without giving the gue= st any information. > + fn into_node(self) -> Node { > + let mut children =3D self.children(); > + if children.len() =3D=3D 1 { > + children.pop_first().unwrap().1 > + } else { > + Node::Switch(children) > + } > + } > +} > + > +/// Returns the host tree for each guest NUMA node. Placements of device= s that are not configured > +/// are ignored. > +fn numa_nodes<'a>( > + hw: &HardwareConfig, > + placements: &'a [HostpciePlacement], > +) -> Result>, LayoutError> { > + let placements: BTreeMap =3D > + placements.iter().map(|p| (p.index, p)).collect(); > + let mut nodes: BTreeMap =3D BTreeMap::new(); > + let mut parents =3D BTreeMap::new(); > + > + for &index in hw.hostpcie_devices.keys() { > + let device =3D placements > + .get(&index) > + .ok_or(LayoutError::MissingPlacement(index))?; > + if index >=3D MAX_HOSTPCI_DEVICES || device.numa_node >=3D MAX_N= UMA_NODES { > + return Err(LayoutError::InvalidPlacement(index)); > + } > + let switches: &[String] =3D match hw.hostpci_topology { > + HostpciTopology::Host =3D> &device.switches, > + _ =3D> &[], > + }; > + // switches must be nested identically for all devices (e.g. swi= tch X can't be behind Y for > + // one device and behind Z for another) > + for (depth, switch) in switches.iter().enumerate() { > + let parent =3D (device.numa_node, &switches[..depth]); > + if *parents.entry(switch).or_insert(parent) !=3D parent { > + return Err(LayoutError::InvalidPlacement(index)); > + } > + } > + nodes > + .entry(device.numa_node) > + .or_default() > + .insert(index, switches); > + } > + > + Ok(nodes) > +} > + > +/// Returns the `-device` value for the device creating `bus` at `addr`. > +fn device(bus: Bus, addr: PciAddress, params: &str) -> String { > + // all buses of the topology are created by a device > + let model =3D bus.bridge_model(addr.bus).unwrap(); > + format!("{model},id=3D{bus}{}{params}", addr.to_qemu_addr()) > +} > + > +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpc= ie` addresses. > +pub(crate) struct Topology { > + /// `-device` values for all expander bridges, ports and switches, e= ach bus before the devices > + /// on it. > + pub(crate) devices: Vec, > + /// Addresses of the `hostpcie` devices, keyed by index. > + pub(crate) addresses: BTreeMap, > + switches: u8, > +} > + > +impl Topology { > + /// Computes the topology of `hw`, which must not be [`HostpciTopolo= gy::Flat`]. > + pub(crate) fn new( > + hw: &HardwareConfig, > + placements: &[HostpciePlacement], > + ) -> Result { > + if !hw.has_pcie() || hw.layout !=3D PciLayout::Grouped { > + return Err(LayoutError::UnsupportedTopology); > + } > + > + let mut topology =3D Self { > + devices: Vec::new(), > + addresses: BTreeMap::new(), > + switches: 0, > + }; > + let mut bus_nr =3D FIRST_NUMA_BUS; > + for (numa_node, tree) in numa_nodes(hw, placements)? { > + let start =3D topology.devices.len(); > + let bus =3D Bus::Numa(numa_node); > + let addr =3D PCIE_LAYOUT.find_device(&Device::Bridge(bus))?; Why is the PCIE_Layout always the grouped layout here? I understand a=20 non-flat topology currently only makes sense in the context of a grouped layout, but shouldnt we keep this somewhat generic so we aren't locked into the grouped layout? IMO we can just pass the layout in the constructor of Topology. > + let params =3D format!(",bus_nr=3D{bus_nr},numa_node=3D{numa= _node}"); > + topology.devices.push(device(bus, addr, ¶ms)); > + for (key, node) in tree.children() { > + // keep slot 0 free, like on all other bridges > + let addr =3D PciAddress::new(bus, key + 1, 0); > + topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &= node); > + } > + > + // each device we added creates exactly one bus > + let buses =3D topology.devices.len() - start; > + if bus_nr + buses > 256 { > + return Err(LayoutError::BusNumbersExhausted); > + } > + bus_nr +=3D buses.next_multiple_of(BUS_RANGE_ALIGN); > + } > + > + Ok(topology) > + } > + > + /// Adds a port for `node` at `addr`, plus everything behind it. On = expander bridges these are > + /// root ports, on switches downstream ports. > + fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) { > + let root_port =3D matches!(addr.bus, Bus::Numa(_)); > + let id =3D match node { > + Node::Device(index) if root_port =3D> Bus::RootPort(*index), > + Node::Device(index) =3D> Bus::SwitchPort(*index), > + Node::Switch(_) =3D> Bus::SwitchUplink(self.switches), > + }; > + let slot =3D node.key(); > + let mut params =3D format!(",port=3D{slot},chassis=3D{chassis},s= lot=3D{slot}"); > + if root_port { > + params.push_str(",x-speed=3D16,x-width=3D32"); > + } > + self.devices.push(device(id, addr, ¶ms)); > + > + match node { > + Node::Device(index) =3D> { > + self.addresses.insert(*index, PciAddress::new(id, 0, 0))= ; > + } > + Node::Switch(children) =3D> { > + let nr =3D self.switches; > + self.switches +=3D 1; > + > + let switch =3D Bus::Switch(nr); > + self.devices > + .push(device(switch, PciAddress::new(id, 0, 0), ""))= ; > + for (key, child) in children { > + let addr =3D PciAddress::new(switch, *key, 0); > + self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child)= ; > + } > + } > + } > + } > +} > + [... tests ...] [... perlmod ..]