From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 11A571FF0DA for ; Thu, 19 Nov 2026 16:12:48 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id CB7F2216BF; Thu, 19 Nov 2026 16:12:46 +0100 (CET) Message-ID: <50a97314-7b8a-46d7-b5ea-591e3cb88150@proxmox.com> Date: Thu, 19 Nov 2026 16:12:41 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option To: Elias Huhsovitz , pve-devel@lists.proxmox.com References: <20261118152803.3843758-1-d.csapak@proxmox.com> <20261118152803.3843758-13-d.csapak@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795101161532 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.726 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GJNMBXZXHURVPOA5HQSZNL2A7VGVZ2VB X-Message-ID-Hash: GJNMBXZXHURVPOA5HQSZNL2A7VGVZ2VB X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 11/19/26 1:42 PM, Elias Huhsovitz wrote: > Comments inline. > > On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: >> Makes the hostpci topology of the PCI bindings configurable, so that >> hostpci devices with pcie=1 can be placed in the guest like on the >> host. The default is 'flat', which keeps the current placement. >> >> For the 'numa' and 'host' topologies, read the NUMA node of each device >> from sysfs, and for 'host' also the bridges above it, and pass them as >> placements to the bindings. This happens after choosing the host >> devices, since only then it's known which ones are used, and choosing >> them reserves them. For mediated devices, the actual device is only >> chosen when starting the guest, so all alternatives must be on the same >> host NUMA node and only their common bridges are used. >> >> The guest must have NUMA enabled with a guest node bound to the host >> node of each device, as the expander bridges can only be assigned to >> existing guest NUMA nodes. Both topologies also need the 'grouped' PCI >> layout. >> >> Since the placement depends on the hardware of the host, prevent live >> migration for all topologies except 'flat'. >> >> Signed-off-by: Dominik Csapak >> --- > > [...] > >> >> +# Returns the addresses of all bridges between the host bridge and the given host device, starting >> +# at the root port. A PCIe switch shows up as its upstream and downstream port. >> +sub get_host_pci_bridges { >> + my ($pciid) = @_; >> + >> + # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.0 >> + my $sysfs_path = "/sys/bus/pci/devices/$pciid"; >> + die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $sysfs_path; >> + my $path = abs_path($sysfs_path); >> + >> + my @bridges; >> + for my $part (split('/', $path)) { >> + push @bridges, $1 if $part =~ m/^($PCIRE)$/; # filter relevant parts and untaint >> + } >> + pop @bridges; # remove the device itself >> + >> + return \@bridges; >> +} >> + >> +# Returns the host NUMA node the given host device is connected to. >> +sub get_host_numa_node { >> + my ($pciid) = @_; >> + >> + my $node = PVE::File::file_read_first_line("/sys/bus/pci/devices/$pciid/numa_node"); >> + >> + # hosts with only one node report -1 >> + return defined($node) && $node =~ m/^(\d+)$/ ? int($1) : 0; >> +} >> + >> +# Returns where the chosen hostpci devices with pcie=1 are connected on the host, as needed to >> +# place them in the guest with the 'numa' and 'host' PCI topologies. > > Q: What is the point of implementing get_hostpcie_placements in perl, > when we are just reading the info from sysfs to pass it back to rust > almost immediately? Is the goal to avoid any I/O in pve-qemu-pci? well the reason was that the function needs part of the reserved pci-devices hash and i didn't want to invent a new interface type for that too. IMO it would be better to port the whole hostpcie handling to rust in one go, then the types can be internal to the crate > > nit: I feel like get_hostpcie_placements and the function that calls it > (print_hostpci_devices) are doing a bit too much. Especially since > variables like ($host_node, $bridges) are used and mutated inside nested > loops. > > IMO they would benefit from seperating the concerns a bit more. fair point, i'll see what i can do there, without increasing the scope too much ;) > >> +sub get_hostpcie_placements { >> + my ($conf, $pci_devices, $topology) = @_; >> + >> + return [] if !scalar(keys $pci_devices->%*); >> + >> + # the expander bridges can only be assigned to existing guest NUMA nodes >> + die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa}; >> + >> + # we need a clear host to guest numa node mapping >> + my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf); >> + my $host_to_guest = {}; >> + for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) { >> + my $host_node = $guest_to_host->{$guest_node}; >> + next if !defined($host_node) || $host_node !~ m/^\d+$/; >> + $host_to_guest->{$host_node} //= $guest_node; >> + } >> + >> + my $placements = []; >> + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { >> + my $d = $pci_devices->{"hostpci$i"}; >> + next if !$d || !$d->{pcie}; >> + >> + # For mediated devices, the actual device is only chosen when starting the guest, so only >> + # use numa nodes/bridges that all alternatives have in common. >> + my @pciids = map { $_->{id} } $d->{ids}->@*; >> + @pciids = ($pciids[0]) if !$d->{mdev}; >> + >> + my ($host_node, $bridges); >> + for my $pciid (@pciids) { >> + my $node = get_host_numa_node($pciid); >> + $host_node //= $node; >> + die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on" >> + . " the same host NUMA node\n" >> + if $node != $host_node; >> + >> + next if $topology ne 'host'; >> + >> + my $current = get_host_pci_bridges($pciid); >> + if (!defined($bridges)) { >> + $bridges = $current; >> + next; >> + } >> + # keep the bridges shared by all alternatives >> + my $common = []; >> + for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) { >> + last if $bridges->[$j] ne $current->[$j]; >> + push @$common, $bridges->[$j]; >> + } >> + $bridges = $common; >> + } >> + >> + my $guest_node = $host_to_guest->{$host_node}; >> + die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n" >> + if !defined($guest_node); >> + >> + my $placement = { index => $i, 'numa-node' => int($guest_node) }; >> + $placement->{switches} = $bridges if defined($bridges); >> + push $placements->@*, $placement; >> + } >> + >> + return $placements; >> +} >> + >> sub print_hostpci_devices { >> my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_; >> >> @@ -376,6 +473,15 @@ sub print_hostpci_devices { >> my $pciaddr; >> my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run); >> >> + # the placements can only be derived after we chose the pci devices >> + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine}); >> + my $topology = $machine_conf->{'pci-topology'} // 'flat'; >> + my $placements; >> + if ($topology ne 'flat') { >> + $placements = get_hostpcie_placements($conf, $pci_devices, $topology); >> + push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*; >> + } >> + >> for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { >> my $id = "hostpci$i"; >> my $d = $pci_devices->{$id}; >> @@ -384,8 +490,7 @@ sub print_hostpci_devices { >> $legacy_igd = 1 if $d->{'legacy-igd'}; >> >> if (my $pcie = $d->{pcie}) { >> - # the root ports come from pci_bus_devices >> - $pciaddr = $hw_config->addr("hostpcie$i"); >> + $pciaddr = $hw_config->addr("hostpcie$i", $placements); >> } else { >> my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id; >> $pciaddr = $hw_config->addr($pci_name); >> @@ -711,6 +816,12 @@ sub hardware_config { >> $mainboard = "i440fx"; >> } >> >> + my $topology = $machine_conf->{'pci-topology'} // 'flat'; >> + my $layout = $machine_conf->{'pci-layout'} // 'legacy'; >> + >> + die "PCI topology '$topology' requires the 'grouped' PCI layout\n" >> + if $topology ne 'flat' && $layout eq 'legacy'; >> + >> my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version); >> >> return PVE::QemuServer::RS::PCI::HardwareConfig->new({ >> @@ -729,9 +840,9 @@ sub hardware_config { >> }, >> 'legacy-igd' => $legacy_igd, >> 'ostype' => $conf->{ostype} // 'other', >> - 'layout' => $machine_conf->{'pci-layout'} // 'legacy', >> + 'layout' => $layout, >> 'hostpcie-devices' => $hostpcie, >> - 'hostpci-topology' => 'flat', >> + 'hostpci-topology' => $topology, >> }); >> } >> >> diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm >> index 7fd4cc23..1ede3512 100644 >> --- a/src/test/QemuServerTest/CommandLineMocks.pm >> +++ b/src/test/QemuServerTest/CommandLineMocks.pm >> @@ -168,6 +168,15 @@ my $pci_devs = [ >> "0000:02:00.5", >> ]; >> >> +# where the devices on a host bus are connected: the NUMA node and the bridges above them, >> +# starting at the root port. Buses not listed here are on node 0 without any bridge. >> +my $pci_host_topology = { >> + '01' => { node => 0, bridges => ['0000:00:01.0'] }, >> + '07' => { node => 0, bridges => ['0000:00:03.0'] }, >> + # behind a switch >> + 'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] }, >> +}; >> + >> my $pci_map_config = { >> ids => { >> someGpu => { >> @@ -615,6 +624,18 @@ $pci_module->mock( >> reserve_pci_usage => sub { >> die "reserve_pci_usage should not be called for 'qm showcmd'\n"; >> }, >> + get_host_numa_node => sub { >> + my ($pciid) = @_; >> + >> + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; >> + return $pci_host_topology->{$bus}->{node} // 0; >> + }, >> + get_host_pci_bridges => sub { >> + my ($pciid) = @_; >> + >> + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; >> + return [($pci_host_topology->{$bus}->{bridges} // [])->@*]; >> + }, >> ); >> > [...]