From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 87B9F1FF0DA for ; Thu, 19 Nov 2026 16:10:13 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 946AC216D7; Thu, 19 Nov 2026 16:10:11 +0100 (CET) Message-ID: <36bd1129-7d72-4733-8bd2-fedca7d69e40@proxmox.com> Date: Thu, 19 Nov 2026 16:10:04 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host To: Elias Huhsovitz , pve-devel@lists.proxmox.com References: <20261118152803.3843758-1-d.csapak@proxmox.com> <20261118152803.3843758-12-d.csapak@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795101005140 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.274 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: JCC5THJUAE26GMI5FR7KTHNMQRPWKGJQ X-Message-ID-Hash: JCC5THJUAE26GMI5FR7KTHNMQRPWKGJQ X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 11/19/26 2:43 PM, Elias Huhsovitz wrote: > From a high level view, I think the design is fitting. > > It would be very interesting to measure the real world performance > impact of the 3 layouts. > yes, i'll benchmark that soon, hopefully can post results in the next few days... will also include your current series then (will adopt it to this series here locally if i can) > Comments inline. > > On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: >> Guest software sometimes uses the PCI topology to choose CPUs and DMA >> paths. Since we put every hostpcie device on a root port of the root >> bus, the guest can't see which devices are close to which NUMA node or >> to each other. >> >> So add a hostpci topology setting for machines with a PCIe root bus >> (q35 and aarch64 virt) and the grouped layout: >> * 'flat': the current placement, a root port on the root bus per device >> * 'numa': puts the root ports of each guest NUMA node's devices behind >> a pxb-pcie for that node >> * 'host': additionally replicates the host PCIe switches that more >> than one passed through device is behind. We skip the other >> switches, since they don't add any information but cost bus numbers. >> >> The guest NUMA node and the host switches of each device are passed as >> placements separately from the hardware config, since the caller only >> knows them after choosing the host devices. >> >> All buses share the 256 bus numbers of a single PCI domain. We pack >> the bus number ranges of the expander bridges by demand and align them >> to 16, so a changed device normally only shifts its own node, and even >> 16 devices always fit. >> >> Each port gets its own slot, numbered after the lowest hostpcie index >> behind it, so its address does not depend on its siblings or on a >> function 0 being present. >> >> In the bindings, add 'hostpcie_bus_devices', which takes the placements >> and returns the expander bridges, switches and root ports, and let >> 'addr' take them as an optional parameter, so existing callers don't >> have to change. Since the topology is a required field, qemu-server >> passes 'flat' until the option to choose it is added. >> >> Signed-off-by: Dominik Csapak >> --- > > [...] > >> -pub(crate) fn address(hw: &HardwareConfig, id: &str) -> Result { >> - let layout = layout(hw.has_pcie()); >> - layout.find_device(&id.parse()?) >> +pub(crate) fn address( >> + hw: &HardwareConfig, >> + id: &str, >> + placements: Option<&[HostpciePlacement]>, >> +) -> Result { >> + let device = id.parse()?; >> + match device { >> + Device::Hostpcie(index) if hw.hostpci_topology != HostpciTopology::Flat => { >> + let placements = placements.ok_or(LayoutError::MissingPlacement(index))?; >> + let topology = Topology::new(hw, placements)?; > > Isn't building a new topology for each address() call a bit excessive? > IMO we should rebuild the topology only when necesarry. > > Or do we run into some freshness issues in this case? well we build it at max for (currently) 16 pcie devices, once per startup. the overhead should't be that big (i guess a few µs to ms), but yes, as a future improvement caching that would be good > >> + let addr = topology.addresses.get(&index).copied(); >> + addr.ok_or(LayoutError::NoAddressFound(device)) >> + } >> + _ => layout(hw.has_pcie()).find_device(&device), >> + } >> } >> >> pub(crate) fn pci_bridge_for_device(hw: &HardwareConfig, id: &str) -> Option { >> - let addr = address(hw, id).ok()?; >> + let addr = address(hw, id, None).ok()?; >> Some(addr.bus) >> } >> > > [...] > >> + >> +/// How passed through PCIe devices are placed in the guest. >> +/// >> +/// Drivers for accelerators and RDMA NICs use the PCI topology to choose CPUs and peer-to-peer >> +/// paths, so we replicate the host topology from the [`HostpciePlacement`] of each device. >> +#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize)] >> +#[serde(rename_all = "kebab-case")] >> +pub enum HostpciTopology { >> + /// Each device gets its own root port on the root bus. >> + #[default] >> + Flat, >> + /// Groups the root ports behind a `pxb-pcie` expander bridge per guest NUMA node. >> + Numa, >> + /// Like `Numa`, but also replicates host switches with more than one device behind them. >> + Host, >> +} >> + >> +/// Where a passed through PCIe device is connected on the host. Only needed for topologies other >> +/// than [`HostpciTopology::Flat`]. >> +#[derive(Default, Clone, Debug, PartialEq, Eq, Deserialize)] >> +#[serde(rename_all = "kebab-case")] >> +pub struct HostpciePlacement { >> + /// The hostpci index of the device (e.g. '3' from 'hostpci3'). >> + pub index: u8, >> + /// The guest NUMA node backed by the host NUMA node of the device. >> + #[serde(default)] >> + pub numa_node: u8, >> + /// The host bridges above the device, starting at the root port. Each one needs a unique id, >> + /// e.g. its PCI address. Bridges shared by more than one device are replicated as a switch, >> + /// so the devices have the same common upstream bridge as on the host. Only used with >> + /// [`HostpciTopology::Host`]. >> + #[serde(default)] >> + pub switches: Vec, >> +} >> + >> +/// A device or replicated switch in the guest. >> +enum Node { >> + Device(u8), >> + Switch(BTreeMap), >> +} >> + >> +impl Node { >> + /// Returns the lowest `hostpcie` index below this node. We use it as slot number, so the >> + /// address of a port does not depend on its siblings. >> + fn key(&self) -> u8 { >> + match self { >> + Node::Device(index) => *index, >> + Node::Switch(children) => *children.keys().next().unwrap(), >> + } >> + } >> +} >> + >> +/// The devices and switches below a NUMA node or a switch on the host. >> +#[derive(Default)] >> +struct HostTree<'a> { >> + devices: Vec, >> + switches: BTreeMap<&'a str, HostTree<'a>>, >> +} > > nit: naming: IMO something like "HostNode" or "PciNode" fits better. > Usually a "Tree" struct just wraps the root node (In Java at least) sure, i have no strong opinion on naming these things anyway ;) > >> +impl<'a> HostTree<'a> { >> + /// Adds `hostpcie` below the given host switches. >> + fn insert(&mut self, index: u8, switches: &'a [String]) { >> + match switches.split_first() { >> + None => self.devices.push(index), >> + Some((first, rest)) => self.switches.entry(first).or_default().insert(index, rest), >> + } >> + } >> + >> + /// Returns the guest nodes directly below this one, keyed by [`Node::key`]. >> + fn children(self) -> BTreeMap { >> + let devices = self.devices.into_iter().map(Node::Device); >> + let switches = self.switches.into_values().map(HostTree::into_node); >> + devices >> + .chain(switches) >> + .map(|node| (node.key(), node)) >> + .collect() >> + } >> + >> + /// Converts the switch to a guest node. Switches with only one child are replaced by that >> + /// child, since they only use up bus numbers without giving the guest any information. >> + fn into_node(self) -> Node { >> + let mut children = self.children(); >> + if children.len() == 1 { >> + children.pop_first().unwrap().1 >> + } else { >> + Node::Switch(children) >> + } >> + } >> +} >> + >> +/// Returns the host tree for each guest NUMA node. Placements of devices that are not configured >> +/// are ignored. >> +fn numa_nodes<'a>( >> + hw: &HardwareConfig, >> + placements: &'a [HostpciePlacement], >> +) -> Result>, LayoutError> { >> + let placements: BTreeMap = >> + placements.iter().map(|p| (p.index, p)).collect(); >> + let mut nodes: BTreeMap = BTreeMap::new(); >> + let mut parents = BTreeMap::new(); >> + >> + for &index in hw.hostpcie_devices.keys() { >> + let device = placements >> + .get(&index) >> + .ok_or(LayoutError::MissingPlacement(index))?; >> + if index >= MAX_HOSTPCI_DEVICES || device.numa_node >= MAX_NUMA_NODES { >> + return Err(LayoutError::InvalidPlacement(index)); >> + } >> + let switches: &[String] = match hw.hostpci_topology { >> + HostpciTopology::Host => &device.switches, >> + _ => &[], >> + }; >> + // switches must be nested identically for all devices (e.g. switch X can't be behind Y for >> + // one device and behind Z for another) >> + for (depth, switch) in switches.iter().enumerate() { >> + let parent = (device.numa_node, &switches[..depth]); >> + if *parents.entry(switch).or_insert(parent) != parent { >> + return Err(LayoutError::InvalidPlacement(index)); >> + } >> + } >> + nodes >> + .entry(device.numa_node) >> + .or_default() >> + .insert(index, switches); >> + } >> + >> + Ok(nodes) >> +} >> + >> +/// Returns the `-device` value for the device creating `bus` at `addr`. >> +fn device(bus: Bus, addr: PciAddress, params: &str) -> String { >> + // all buses of the topology are created by a device >> + let model = bus.bridge_model(addr.bus).unwrap(); >> + format!("{model},id={bus}{}{params}", addr.to_qemu_addr()) >> +} >> + >> +/// The QEMU devices for a [`HostpciTopology`] and the resulting `hostpcie` addresses. >> +pub(crate) struct Topology { >> + /// `-device` values for all expander bridges, ports and switches, each bus before the devices >> + /// on it. >> + pub(crate) devices: Vec, >> + /// Addresses of the `hostpcie` devices, keyed by index. >> + pub(crate) addresses: BTreeMap, >> + switches: u8, >> +} >> + >> +impl Topology { >> + /// Computes the topology of `hw`, which must not be [`HostpciTopology::Flat`]. >> + pub(crate) fn new( >> + hw: &HardwareConfig, >> + placements: &[HostpciePlacement], >> + ) -> Result { >> + if !hw.has_pcie() || hw.layout != PciLayout::Grouped { >> + return Err(LayoutError::UnsupportedTopology); >> + } >> + >> + let mut topology = Self { >> + devices: Vec::new(), >> + addresses: BTreeMap::new(), >> + switches: 0, >> + }; >> + let mut bus_nr = FIRST_NUMA_BUS; >> + for (numa_node, tree) in numa_nodes(hw, placements)? { >> + let start = topology.devices.len(); >> + let bus = Bus::Numa(numa_node); >> + let addr = PCIE_LAYOUT.find_device(&Device::Bridge(bus))?; > > Why is the PCIE_Layout always the grouped layout here? I understand a > non-flat topology currently only makes sense in the context of a > grouped layout, but shouldnt we keep this somewhat generic so we > aren't locked into the grouped layout? > > IMO we can just pass the layout in the constructor of Topology. i mean we can still change that later down the line if necessary. The interface is internal to this crate, so changing it wouldn't break any outside caller if i have to send a new version i can do it though > >> + let params = format!(",bus_nr={bus_nr},numa_node={numa_node}"); >> + topology.devices.push(device(bus, addr, ¶ms)); >> + for (key, node) in tree.children() { >> + // keep slot 0 free, like on all other bridges >> + let addr = PciAddress::new(bus, key + 1, 0); >> + topology.add_port(addr, NUMA_CHASSIS_BASE + numa_node, &node); >> + } >> + >> + // each device we added creates exactly one bus >> + let buses = topology.devices.len() - start; >> + if bus_nr + buses > 256 { >> + return Err(LayoutError::BusNumbersExhausted); >> + } >> + bus_nr += buses.next_multiple_of(BUS_RANGE_ALIGN); >> + } >> + >> + Ok(topology) >> + } >> + >> + /// Adds a port for `node` at `addr`, plus everything behind it. On expander bridges these are >> + /// root ports, on switches downstream ports. >> + fn add_port(&mut self, addr: PciAddress, chassis: u8, node: &Node) { >> + let root_port = matches!(addr.bus, Bus::Numa(_)); >> + let id = match node { >> + Node::Device(index) if root_port => Bus::RootPort(*index), >> + Node::Device(index) => Bus::SwitchPort(*index), >> + Node::Switch(_) => Bus::SwitchUplink(self.switches), >> + }; >> + let slot = node.key(); >> + let mut params = format!(",port={slot},chassis={chassis},slot={slot}"); >> + if root_port { >> + params.push_str(",x-speed=16,x-width=32"); >> + } >> + self.devices.push(device(id, addr, ¶ms)); >> + >> + match node { >> + Node::Device(index) => { >> + self.addresses.insert(*index, PciAddress::new(id, 0, 0)); >> + } >> + Node::Switch(children) => { >> + let nr = self.switches; >> + self.switches += 1; >> + >> + let switch = Bus::Switch(nr); >> + self.devices >> + .push(device(switch, PciAddress::new(id, 0, 0), "")); >> + for (key, child) in children { >> + let addr = PciAddress::new(switch, *key, 0); >> + self.add_port(addr, SWITCH_CHASSIS_BASE + nr, child); >> + } >> + } >> + } >> + } >> +} >> + > > [... tests ...] > [... perlmod ..]