From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option
Date: Thu, 19 Nov 2026 13:42:30 +0100 [thread overview]
Message-ID: <DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-13-d.csapak@proxmox.com>
Comments inline.
On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> Makes the hostpci topology of the PCI bindings configurable, so that
> hostpci devices with pcie=1 can be placed in the guest like on the
> host. The default is 'flat', which keeps the current placement.
>
> For the 'numa' and 'host' topologies, read the NUMA node of each device
> from sysfs, and for 'host' also the bridges above it, and pass them as
> placements to the bindings. This happens after choosing the host
> devices, since only then it's known which ones are used, and choosing
> them reserves them. For mediated devices, the actual device is only
> chosen when starting the guest, so all alternatives must be on the same
> host NUMA node and only their common bridges are used.
>
> The guest must have NUMA enabled with a guest node bound to the host
> node of each device, as the expander bridges can only be assigned to
> existing guest NUMA nodes. Both topologies also need the 'grouped' PCI
> layout.
>
> Since the placement depends on the hardware of the host, prevent live
> migration for all topologies except 'flat'.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
[...]
>
> +# Returns the addresses of all bridges between the host bridge and the given host device, starting
> +# at the root port. A PCIe switch shows up as its upstream and downstream port.
> +sub get_host_pci_bridges {
> + my ($pciid) = @_;
> +
> + # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.0
> + my $sysfs_path = "/sys/bus/pci/devices/$pciid";
> + die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $sysfs_path;
> + my $path = abs_path($sysfs_path);
> +
> + my @bridges;
> + for my $part (split('/', $path)) {
> + push @bridges, $1 if $part =~ m/^($PCIRE)$/; # filter relevant parts and untaint
> + }
> + pop @bridges; # remove the device itself
> +
> + return \@bridges;
> +}
> +
> +# Returns the host NUMA node the given host device is connected to.
> +sub get_host_numa_node {
> + my ($pciid) = @_;
> +
> + my $node = PVE::File::file_read_first_line("/sys/bus/pci/devices/$pciid/numa_node");
> +
> + # hosts with only one node report -1
> + return defined($node) && $node =~ m/^(\d+)$/ ? int($1) : 0;
> +}
> +
> +# Returns where the chosen hostpci devices with pcie=1 are connected on the host, as needed to
> +# place them in the guest with the 'numa' and 'host' PCI topologies.
Q: What is the point of implementing get_hostpcie_placements in perl,
when we are just reading the info from sysfs to pass it back to rust
almost immediately? Is the goal to avoid any I/O in pve-qemu-pci?
nit: I feel like get_hostpcie_placements and the function that calls it
(print_hostpci_devices) are doing a bit too much. Especially since
variables like ($host_node, $bridges) are used and mutated inside nested
loops.
IMO they would benefit from seperating the concerns a bit more.
> +sub get_hostpcie_placements {
> + my ($conf, $pci_devices, $topology) = @_;
> +
> + return [] if !scalar(keys $pci_devices->%*);
> +
> + # the expander bridges can only be assigned to existing guest NUMA nodes
> + die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa};
> +
> + # we need a clear host to guest numa node mapping
> + my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf);
> + my $host_to_guest = {};
> + for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) {
> + my $host_node = $guest_to_host->{$guest_node};
> + next if !defined($host_node) || $host_node !~ m/^\d+$/;
> + $host_to_guest->{$host_node} //= $guest_node;
> + }
> +
> + my $placements = [];
> + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
> + my $d = $pci_devices->{"hostpci$i"};
> + next if !$d || !$d->{pcie};
> +
> + # For mediated devices, the actual device is only chosen when starting the guest, so only
> + # use numa nodes/bridges that all alternatives have in common.
> + my @pciids = map { $_->{id} } $d->{ids}->@*;
> + @pciids = ($pciids[0]) if !$d->{mdev};
> +
> + my ($host_node, $bridges);
> + for my $pciid (@pciids) {
> + my $node = get_host_numa_node($pciid);
> + $host_node //= $node;
> + die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on"
> + . " the same host NUMA node\n"
> + if $node != $host_node;
> +
> + next if $topology ne 'host';
> +
> + my $current = get_host_pci_bridges($pciid);
> + if (!defined($bridges)) {
> + $bridges = $current;
> + next;
> + }
> + # keep the bridges shared by all alternatives
> + my $common = [];
> + for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) {
> + last if $bridges->[$j] ne $current->[$j];
> + push @$common, $bridges->[$j];
> + }
> + $bridges = $common;
> + }
> +
> + my $guest_node = $host_to_guest->{$host_node};
> + die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n"
> + if !defined($guest_node);
> +
> + my $placement = { index => $i, 'numa-node' => int($guest_node) };
> + $placement->{switches} = $bridges if defined($bridges);
> + push $placements->@*, $placement;
> + }
> +
> + return $placements;
> +}
> +
> sub print_hostpci_devices {
> my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_;
>
> @@ -376,6 +473,15 @@ sub print_hostpci_devices {
> my $pciaddr;
> my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run);
>
> + # the placements can only be derived after we chose the pci devices
> + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine});
> + my $topology = $machine_conf->{'pci-topology'} // 'flat';
> + my $placements;
> + if ($topology ne 'flat') {
> + $placements = get_hostpcie_placements($conf, $pci_devices, $topology);
> + push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*;
> + }
> +
> for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
> my $id = "hostpci$i";
> my $d = $pci_devices->{$id};
> @@ -384,8 +490,7 @@ sub print_hostpci_devices {
> $legacy_igd = 1 if $d->{'legacy-igd'};
>
> if (my $pcie = $d->{pcie}) {
> - # the root ports come from pci_bus_devices
> - $pciaddr = $hw_config->addr("hostpcie$i");
> + $pciaddr = $hw_config->addr("hostpcie$i", $placements);
> } else {
> my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id;
> $pciaddr = $hw_config->addr($pci_name);
> @@ -711,6 +816,12 @@ sub hardware_config {
> $mainboard = "i440fx";
> }
>
> + my $topology = $machine_conf->{'pci-topology'} // 'flat';
> + my $layout = $machine_conf->{'pci-layout'} // 'legacy';
> +
> + die "PCI topology '$topology' requires the 'grouped' PCI layout\n"
> + if $topology ne 'flat' && $layout eq 'legacy';
> +
> my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version);
>
> return PVE::QemuServer::RS::PCI::HardwareConfig->new({
> @@ -729,9 +840,9 @@ sub hardware_config {
> },
> 'legacy-igd' => $legacy_igd,
> 'ostype' => $conf->{ostype} // 'other',
> - 'layout' => $machine_conf->{'pci-layout'} // 'legacy',
> + 'layout' => $layout,
> 'hostpcie-devices' => $hostpcie,
> - 'hostpci-topology' => 'flat',
> + 'hostpci-topology' => $topology,
> });
> }
>
> diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm
> index 7fd4cc23..1ede3512 100644
> --- a/src/test/QemuServerTest/CommandLineMocks.pm
> +++ b/src/test/QemuServerTest/CommandLineMocks.pm
> @@ -168,6 +168,15 @@ my $pci_devs = [
> "0000:02:00.5",
> ];
>
> +# where the devices on a host bus are connected: the NUMA node and the bridges above them,
> +# starting at the root port. Buses not listed here are on node 0 without any bridge.
> +my $pci_host_topology = {
> + '01' => { node => 0, bridges => ['0000:00:01.0'] },
> + '07' => { node => 0, bridges => ['0000:00:03.0'] },
> + # behind a switch
> + 'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] },
> +};
> +
> my $pci_map_config = {
> ids => {
> someGpu => {
> @@ -615,6 +624,18 @@ $pci_module->mock(
> reserve_pci_usage => sub {
> die "reserve_pci_usage should not be called for 'qm showcmd'\n";
> },
> + get_host_numa_node => sub {
> + my ($pciid) = @_;
> +
> + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> + return $pci_host_topology->{$bus}->{node} // 0;
> + },
> + get_host_pci_bridges => sub {
> + my ($pciid) = @_;
> +
> + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> + return [($pci_host_topology->{$bus}->{bridges} // [])->@*];
> + },
> );
>
[...]
next prev parent reply other threads:[~2026-11-19 12:42 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47 ` Elias Huhsovitz
2026-11-19 14:51 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43 ` Elias Huhsovitz
2026-11-19 15:10 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42 ` Elias Huhsovitz [this message]
2026-11-19 15:12 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33 ` Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com \
--to=e.huhsovitz@proxmox.com \
--cc=d.csapak@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox