public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option
Date: Thu, 19 Nov 2026 13:42:30 +0100	[thread overview]
Message-ID: <DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-13-d.csapak@proxmox.com>

Comments inline.

On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> Makes the hostpci topology of the PCI bindings configurable, so that
> hostpci devices with pcie=1 can be placed in the guest like on the
> host. The default is 'flat', which keeps the current placement.
>
> For the 'numa' and 'host' topologies, read the NUMA node of each device
> from sysfs, and for 'host' also the bridges above it, and pass them as
> placements to the bindings. This happens after choosing the host
> devices, since only then it's known which ones are used, and choosing
> them reserves them. For mediated devices, the actual device is only
> chosen when starting the guest, so all alternatives must be on the same
> host NUMA node and only their common bridges are used.
>
> The guest must have NUMA enabled with a guest node bound to the host
> node of each device, as the expander bridges can only be assigned to
> existing guest NUMA nodes. Both topologies also need the 'grouped' PCI
> layout.
>
> Since the placement depends on the hardware of the host, prevent live
> migration for all topologies except 'flat'.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---

[...]

>  
> +# Returns the addresses of all bridges between the host bridge and the given host device, starting
> +# at the root port. A PCIe switch shows up as its upstream and downstream port.
> +sub get_host_pci_bridges {
> +    my ($pciid) = @_;
> +
> +    # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.0
> +    my $sysfs_path = "/sys/bus/pci/devices/$pciid";
> +    die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $sysfs_path;
> +    my $path = abs_path($sysfs_path);
> +
> +    my @bridges;
> +    for my $part (split('/', $path)) {
> +        push @bridges, $1 if $part =~ m/^($PCIRE)$/; # filter relevant parts and untaint
> +    }
> +    pop @bridges; # remove the device itself
> +
> +    return \@bridges;
> +}
> +
> +# Returns the host NUMA node the given host device is connected to.
> +sub get_host_numa_node {
> +    my ($pciid) = @_;
> +
> +    my $node = PVE::File::file_read_first_line("/sys/bus/pci/devices/$pciid/numa_node");
> +
> +    # hosts with only one node report -1
> +    return defined($node) && $node =~ m/^(\d+)$/ ? int($1) : 0;
> +}
> +
> +# Returns where the chosen hostpci devices with pcie=1 are connected on the host, as needed to
> +# place them in the guest with the 'numa' and 'host' PCI topologies.

Q: What is the point of implementing get_hostpcie_placements in perl,
when we are just reading the info from sysfs to pass it back to rust
almost immediately? Is the goal to avoid any I/O in pve-qemu-pci?

nit: I feel like get_hostpcie_placements and the function that calls it
(print_hostpci_devices) are doing a bit too much. Especially since
variables like ($host_node, $bridges) are used and mutated inside nested
loops.

IMO they would benefit from seperating the concerns a bit more.

> +sub get_hostpcie_placements {
> +    my ($conf, $pci_devices, $topology) = @_;
> +
> +    return [] if !scalar(keys $pci_devices->%*);
> +
> +    # the expander bridges can only be assigned to existing guest NUMA nodes
> +    die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa};
> +
> +    # we need a clear host to guest numa node mapping
> +    my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf);
> +    my $host_to_guest = {};
> +    for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) {
> +        my $host_node = $guest_to_host->{$guest_node};
> +        next if !defined($host_node) || $host_node !~ m/^\d+$/;
> +        $host_to_guest->{$host_node} //= $guest_node;
> +    }
> +
> +    my $placements = [];
> +    for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
> +        my $d = $pci_devices->{"hostpci$i"};
> +        next if !$d || !$d->{pcie};
> +
> +        # For mediated devices, the actual device is only chosen when starting the guest, so only
> +        # use numa nodes/bridges that all alternatives have in common.
> +        my @pciids = map { $_->{id} } $d->{ids}->@*;
> +        @pciids = ($pciids[0]) if !$d->{mdev};
> +
> +        my ($host_node, $bridges);
> +        for my $pciid (@pciids) {
> +            my $node = get_host_numa_node($pciid);
> +            $host_node //= $node;
> +            die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on"
> +                . " the same host NUMA node\n"
> +                if $node != $host_node;
> +
> +            next if $topology ne 'host';
> +
> +            my $current = get_host_pci_bridges($pciid);
> +            if (!defined($bridges)) {
> +                $bridges = $current;
> +                next;
> +            }
> +            # keep the bridges shared by all alternatives
> +            my $common = [];
> +            for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) {
> +                last if $bridges->[$j] ne $current->[$j];
> +                push @$common, $bridges->[$j];
> +            }
> +            $bridges = $common;
> +        }
> +
> +        my $guest_node = $host_to_guest->{$host_node};
> +        die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n"
> +            if !defined($guest_node);
> +
> +        my $placement = { index => $i, 'numa-node' => int($guest_node) };
> +        $placement->{switches} = $bridges if defined($bridges);
> +        push $placements->@*, $placement;
> +    }
> +
> +    return $placements;
> +}
> +
>  sub print_hostpci_devices {
>      my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_;
>  
> @@ -376,6 +473,15 @@ sub print_hostpci_devices {
>      my $pciaddr;
>      my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run);
>  
> +    # the placements can only be derived after we chose the pci devices
> +    my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine});
> +    my $topology = $machine_conf->{'pci-topology'} // 'flat';
> +    my $placements;
> +    if ($topology ne 'flat') {
> +        $placements = get_hostpcie_placements($conf, $pci_devices, $topology);
> +        push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*;
> +    }
> +
>      for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
>          my $id = "hostpci$i";
>          my $d = $pci_devices->{$id};
> @@ -384,8 +490,7 @@ sub print_hostpci_devices {
>          $legacy_igd = 1 if $d->{'legacy-igd'};
>  
>          if (my $pcie = $d->{pcie}) {
> -            # the root ports come from pci_bus_devices
> -            $pciaddr = $hw_config->addr("hostpcie$i");
> +            $pciaddr = $hw_config->addr("hostpcie$i", $placements);
>          } else {
>              my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id;
>              $pciaddr = $hw_config->addr($pci_name);
> @@ -711,6 +816,12 @@ sub hardware_config {
>          $mainboard = "i440fx";
>      }
>  
> +    my $topology = $machine_conf->{'pci-topology'} // 'flat';
> +    my $layout = $machine_conf->{'pci-layout'} // 'legacy';
> +
> +    die "PCI topology '$topology' requires the 'grouped' PCI layout\n"
> +        if $topology ne 'flat' && $layout eq 'legacy';
> +
>      my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version);
>  
>      return PVE::QemuServer::RS::PCI::HardwareConfig->new({
> @@ -729,9 +840,9 @@ sub hardware_config {
>          },
>          'legacy-igd' => $legacy_igd,
>          'ostype' => $conf->{ostype} // 'other',
> -        'layout' => $machine_conf->{'pci-layout'} // 'legacy',
> +        'layout' => $layout,
>          'hostpcie-devices' => $hostpcie,
> -        'hostpci-topology' => 'flat',
> +        'hostpci-topology' => $topology,
>      });
>  }
>  
> diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm
> index 7fd4cc23..1ede3512 100644
> --- a/src/test/QemuServerTest/CommandLineMocks.pm
> +++ b/src/test/QemuServerTest/CommandLineMocks.pm
> @@ -168,6 +168,15 @@ my $pci_devs = [
>      "0000:02:00.5",
>  ];
>  
> +# where the devices on a host bus are connected: the NUMA node and the bridges above them,
> +# starting at the root port. Buses not listed here are on node 0 without any bridge.
> +my $pci_host_topology = {
> +    '01' => { node => 0, bridges => ['0000:00:01.0'] },
> +    '07' => { node => 0, bridges => ['0000:00:03.0'] },
> +    # behind a switch
> +    'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] },
> +};
> +
>  my $pci_map_config = {
>      ids => {
>          someGpu => {
> @@ -615,6 +624,18 @@ $pci_module->mock(
>      reserve_pci_usage => sub {
>          die "reserve_pci_usage should not be called for 'qm showcmd'\n";
>      },
> +    get_host_numa_node => sub {
> +        my ($pciid) = @_;
> +
> +        my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> +        return $pci_host_topology->{$bus}->{node} // 0;
> +    },
> +    get_host_pci_bridges => sub {
> +        my ($pciid) = @_;
> +
> +        my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> +        return [($pci_host_topology->{$bus}->{bridges} // [])->@*];
> +    },
>  );
>  
[...]




  reply	other threads:[~2026-11-19 12:42 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47   ` Elias Huhsovitz
2026-11-19 14:51     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43   ` Elias Huhsovitz
2026-11-19 15:10     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42   ` Elias Huhsovitz [this message]
2026-11-19 15:12     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33   ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com \
    --to=e.huhsovitz@proxmox.com \
    --cc=d.csapak@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal