From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 56DF01FF0DE for ; Thu, 19 Nov 2026 13:42:36 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 104912169D; Thu, 19 Nov 2026 13:42:35 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Nov 2026 13:42:30 +0100 Message-Id: From: "Elias Huhsovitz" To: "Dominik Csapak" , Subject: Re: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: aerc 0.20.0 References: <20261118152803.3843758-1-d.csapak@proxmox.com> <20261118152803.3843758-13-d.csapak@proxmox.com> In-Reply-To: <20261118152803.3843758-13-d.csapak@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795092150354 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.596 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GFWH4E5TZLXH2ZHB55FYYPAJD52W22WB X-Message-ID-Hash: GFWH4E5TZLXH2ZHB55FYYPAJD52W22WB X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Comments inline. On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: > Makes the hostpci topology of the PCI bindings configurable, so that > hostpci devices with pcie=3D1 can be placed in the guest like on the > host. The default is 'flat', which keeps the current placement. > > For the 'numa' and 'host' topologies, read the NUMA node of each device > from sysfs, and for 'host' also the bridges above it, and pass them as > placements to the bindings. This happens after choosing the host > devices, since only then it's known which ones are used, and choosing > them reserves them. For mediated devices, the actual device is only > chosen when starting the guest, so all alternatives must be on the same > host NUMA node and only their common bridges are used. > > The guest must have NUMA enabled with a guest node bound to the host > node of each device, as the expander bridges can only be assigned to > existing guest NUMA nodes. Both topologies also need the 'grouped' PCI > layout. > > Since the placement depends on the hardware of the host, prevent live > migration for all topologies except 'flat'. > > Signed-off-by: Dominik Csapak > --- [...] > =20 > +# Returns the addresses of all bridges between the host bridge and the g= iven host device, starting > +# at the root port. A PCIe switch shows up as its upstream and downstrea= m port. > +sub get_host_pci_bridges { > + my ($pciid) =3D @_; > + > + # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0= /0000:02:04.0/0000:03:00.0 > + my $sysfs_path =3D "/sys/bus/pci/devices/$pciid"; > + die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $= sysfs_path; > + my $path =3D abs_path($sysfs_path); > + > + my @bridges; > + for my $part (split('/', $path)) { > + push @bridges, $1 if $part =3D~ m/^($PCIRE)$/; # filter relevant= parts and untaint > + } > + pop @bridges; # remove the device itself > + > + return \@bridges; > +} > + > +# Returns the host NUMA node the given host device is connected to. > +sub get_host_numa_node { > + my ($pciid) =3D @_; > + > + my $node =3D PVE::File::file_read_first_line("/sys/bus/pci/devices/$= pciid/numa_node"); > + > + # hosts with only one node report -1 > + return defined($node) && $node =3D~ m/^(\d+)$/ ? int($1) : 0; > +} > + > +# Returns where the chosen hostpci devices with pcie=3D1 are connected o= n the host, as needed to > +# place them in the guest with the 'numa' and 'host' PCI topologies. Q: What is the point of implementing get_hostpcie_placements in perl, when we are just reading the info from sysfs to pass it back to rust almost immediately? Is the goal to avoid any I/O in pve-qemu-pci? nit: I feel like get_hostpcie_placements and the function that calls it (print_hostpci_devices) are doing a bit too much. Especially since variables like ($host_node, $bridges) are used and mutated inside nested loops. IMO they would benefit from seperating the concerns a bit more. > +sub get_hostpcie_placements { > + my ($conf, $pci_devices, $topology) =3D @_; > + > + return [] if !scalar(keys $pci_devices->%*); > + > + # the expander bridges can only be assigned to existing guest NUMA n= odes > + die "PCI topology '$topology' requires NUMA to be enabled\n" if !$co= nf->{numa}; > + > + # we need a clear host to guest numa node mapping > + my $guest_to_host =3D PVE::QemuServer::Memory::get_numa_guest_to_hos= t_map($conf); > + my $host_to_guest =3D {}; > + for my $guest_node (sort { $a <=3D> $b } keys $guest_to_host->%*) { > + my $host_node =3D $guest_to_host->{$guest_node}; > + next if !defined($host_node) || $host_node !~ m/^\d+$/; > + $host_to_guest->{$host_node} //=3D $guest_node; > + } > + > + my $placements =3D []; > + for (my $i =3D 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { > + my $d =3D $pci_devices->{"hostpci$i"}; > + next if !$d || !$d->{pcie}; > + > + # For mediated devices, the actual device is only chosen when st= arting the guest, so only > + # use numa nodes/bridges that all alternatives have in common. > + my @pciids =3D map { $_->{id} } $d->{ids}->@*; > + @pciids =3D ($pciids[0]) if !$d->{mdev}; > + > + my ($host_node, $bridges); > + for my $pciid (@pciids) { > + my $node =3D get_host_numa_node($pciid); > + $host_node //=3D $node; > + die "hostpci$i: PCI topology '$topology' requires all altern= ative devices to be on" > + . " the same host NUMA node\n" > + if $node !=3D $host_node; > + > + next if $topology ne 'host'; > + > + my $current =3D get_host_pci_bridges($pciid); > + if (!defined($bridges)) { > + $bridges =3D $current; > + next; > + } > + # keep the bridges shared by all alternatives > + my $common =3D []; > + for (my $j =3D 0; $j < scalar(@$bridges) && $j < scalar(@$cu= rrent); $j++) { > + last if $bridges->[$j] ne $current->[$j]; > + push @$common, $bridges->[$j]; > + } > + $bridges =3D $common; > + } > + > + my $guest_node =3D $host_to_guest->{$host_node}; > + die "hostpci$i: no guest NUMA node is bound to host NUMA node $h= ost_node\n" > + if !defined($guest_node); > + > + my $placement =3D { index =3D> $i, 'numa-node' =3D> int($guest_n= ode) }; > + $placement->{switches} =3D $bridges if defined($bridges); > + push $placements->@*, $placement; > + } > + > + return $placements; > +} > + > sub print_hostpci_devices { > my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = =3D @_; > =20 > @@ -376,6 +473,15 @@ sub print_hostpci_devices { > my $pciaddr; > my $pci_devices =3D choose_hostpci_devices(parse_hostpci_devices($co= nf), $vmid, $dry_run); > =20 > + # the placements can only be derived after we chose the pci devices > + my $machine_conf =3D PVE::QemuServer::Machine::parse_machine($conf->= {machine}); > + my $topology =3D $machine_conf->{'pci-topology'} // 'flat'; > + my $placements; > + if ($topology ne 'flat') { > + $placements =3D get_hostpcie_placements($conf, $pci_devices, $to= pology); > + push @$devices, $hw_config->hostpcie_bus_devices($placements)->@= *; > + } > + > for (my $i =3D 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { > my $id =3D "hostpci$i"; > my $d =3D $pci_devices->{$id}; > @@ -384,8 +490,7 @@ sub print_hostpci_devices { > $legacy_igd =3D 1 if $d->{'legacy-igd'}; > =20 > if (my $pcie =3D $d->{pcie}) { > - # the root ports come from pci_bus_devices > - $pciaddr =3D $hw_config->addr("hostpcie$i"); > + $pciaddr =3D $hw_config->addr("hostpcie$i", $placements); > } else { > my $pci_name =3D $d->{'legacy-igd'} ? 'legacy-igd' : $id; > $pciaddr =3D $hw_config->addr($pci_name); > @@ -711,6 +816,12 @@ sub hardware_config { > $mainboard =3D "i440fx"; > } > =20 > + my $topology =3D $machine_conf->{'pci-topology'} // 'flat'; > + my $layout =3D $machine_conf->{'pci-layout'} // 'legacy'; > + > + die "PCI topology '$topology' requires the 'grouped' PCI layout\n" > + if $topology ne 'flat' && $layout eq 'legacy'; > + > my ($major, $minor, $patch, $pve) =3D PVE::QemuServer::Helpers::vers= ion_parts($machine_version); > =20 > return PVE::QemuServer::RS::PCI::HardwareConfig->new({ > @@ -729,9 +840,9 @@ sub hardware_config { > }, > 'legacy-igd' =3D> $legacy_igd, > 'ostype' =3D> $conf->{ostype} // 'other', > - 'layout' =3D> $machine_conf->{'pci-layout'} // 'legacy', > + 'layout' =3D> $layout, > 'hostpcie-devices' =3D> $hostpcie, > - 'hostpci-topology' =3D> 'flat', > + 'hostpci-topology' =3D> $topology, > }); > } > =20 > diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuS= erverTest/CommandLineMocks.pm > index 7fd4cc23..1ede3512 100644 > --- a/src/test/QemuServerTest/CommandLineMocks.pm > +++ b/src/test/QemuServerTest/CommandLineMocks.pm > @@ -168,6 +168,15 @@ my $pci_devs =3D [ > "0000:02:00.5", > ]; > =20 > +# where the devices on a host bus are connected: the NUMA node and the b= ridges above them, > +# starting at the root port. Buses not listed here are on node 0 without= any bridge. > +my $pci_host_topology =3D { > + '01' =3D> { node =3D> 0, bridges =3D> ['0000:00:01.0'] }, > + '07' =3D> { node =3D> 0, bridges =3D> ['0000:00:03.0'] }, > + # behind a switch > + 'd0' =3D> { node =3D> 1, bridges =3D> ['0000:c0:01.0', '0000:ce:00.0= ', '0000:cf:08.0'] }, > +}; > + > my $pci_map_config =3D { > ids =3D> { > someGpu =3D> { > @@ -615,6 +624,18 @@ $pci_module->mock( > reserve_pci_usage =3D> sub { > die "reserve_pci_usage should not be called for 'qm showcmd'\n"; > }, > + get_host_numa_node =3D> sub { > + my ($pciid) =3D @_; > + > + my ($bus) =3D $pciid =3D~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; > + return $pci_host_topology->{$bus}->{node} // 0; > + }, > + get_host_pci_bridges =3D> sub { > + my ($pciid) =3D @_; > + > + my ($bus) =3D $pciid =3D~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; > + return [($pci_host_topology->{$bus}->{bridges} // [])->@*]; > + }, > ); > =20 [...]