all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option
Date: Thu, 19 Nov 2026 13:42:30 +0100	[thread overview]
Message-ID: <DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-13-d.csapak@proxmox.com>

Comments inline.

On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> Makes the hostpci topology of the PCI bindings configurable, so that
> hostpci devices with pcie=1 can be placed in the guest like on the
> host. The default is 'flat', which keeps the current placement.
>
> For the 'numa' and 'host' topologies, read the NUMA node of each device
> from sysfs, and for 'host' also the bridges above it, and pass them as
> placements to the bindings. This happens after choosing the host
> devices, since only then it's known which ones are used, and choosing
> them reserves them. For mediated devices, the actual device is only
> chosen when starting the guest, so all alternatives must be on the same
> host NUMA node and only their common bridges are used.
>
> The guest must have NUMA enabled with a guest node bound to the host
> node of each device, as the expander bridges can only be assigned to
> existing guest NUMA nodes. Both topologies also need the 'grouped' PCI
> layout.
>
> Since the placement depends on the hardware of the host, prevent live
> migration for all topologies except 'flat'.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---

[...]

>  
> +# Returns the addresses of all bridges between the host bridge and the given host device, starting
> +# at the root port. A PCIe switch shows up as its upstream and downstream port.
> +sub get_host_pci_bridges {
> +    my ($pciid) = @_;
> +
> +    # resolves to e.g. /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.0
> +    my $sysfs_path = "/sys/bus/pci/devices/$pciid";
> +    die "unable to resolve sysfs path of PCI device '$pciid'\n" if !-e $sysfs_path;
> +    my $path = abs_path($sysfs_path);
> +
> +    my @bridges;
> +    for my $part (split('/', $path)) {
> +        push @bridges, $1 if $part =~ m/^($PCIRE)$/; # filter relevant parts and untaint
> +    }
> +    pop @bridges; # remove the device itself
> +
> +    return \@bridges;
> +}
> +
> +# Returns the host NUMA node the given host device is connected to.
> +sub get_host_numa_node {
> +    my ($pciid) = @_;
> +
> +    my $node = PVE::File::file_read_first_line("/sys/bus/pci/devices/$pciid/numa_node");
> +
> +    # hosts with only one node report -1
> +    return defined($node) && $node =~ m/^(\d+)$/ ? int($1) : 0;
> +}
> +
> +# Returns where the chosen hostpci devices with pcie=1 are connected on the host, as needed to
> +# place them in the guest with the 'numa' and 'host' PCI topologies.

Q: What is the point of implementing get_hostpcie_placements in perl,
when we are just reading the info from sysfs to pass it back to rust
almost immediately? Is the goal to avoid any I/O in pve-qemu-pci?

nit: I feel like get_hostpcie_placements and the function that calls it
(print_hostpci_devices) are doing a bit too much. Especially since
variables like ($host_node, $bridges) are used and mutated inside nested
loops.

IMO they would benefit from seperating the concerns a bit more.

> +sub get_hostpcie_placements {
> +    my ($conf, $pci_devices, $topology) = @_;
> +
> +    return [] if !scalar(keys $pci_devices->%*);
> +
> +    # the expander bridges can only be assigned to existing guest NUMA nodes
> +    die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa};
> +
> +    # we need a clear host to guest numa node mapping
> +    my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf);
> +    my $host_to_guest = {};
> +    for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) {
> +        my $host_node = $guest_to_host->{$guest_node};
> +        next if !defined($host_node) || $host_node !~ m/^\d+$/;
> +        $host_to_guest->{$host_node} //= $guest_node;
> +    }
> +
> +    my $placements = [];
> +    for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
> +        my $d = $pci_devices->{"hostpci$i"};
> +        next if !$d || !$d->{pcie};
> +
> +        # For mediated devices, the actual device is only chosen when starting the guest, so only
> +        # use numa nodes/bridges that all alternatives have in common.
> +        my @pciids = map { $_->{id} } $d->{ids}->@*;
> +        @pciids = ($pciids[0]) if !$d->{mdev};
> +
> +        my ($host_node, $bridges);
> +        for my $pciid (@pciids) {
> +            my $node = get_host_numa_node($pciid);
> +            $host_node //= $node;
> +            die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on"
> +                . " the same host NUMA node\n"
> +                if $node != $host_node;
> +
> +            next if $topology ne 'host';
> +
> +            my $current = get_host_pci_bridges($pciid);
> +            if (!defined($bridges)) {
> +                $bridges = $current;
> +                next;
> +            }
> +            # keep the bridges shared by all alternatives
> +            my $common = [];
> +            for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) {
> +                last if $bridges->[$j] ne $current->[$j];
> +                push @$common, $bridges->[$j];
> +            }
> +            $bridges = $common;
> +        }
> +
> +        my $guest_node = $host_to_guest->{$host_node};
> +        die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n"
> +            if !defined($guest_node);
> +
> +        my $placement = { index => $i, 'numa-node' => int($guest_node) };
> +        $placement->{switches} = $bridges if defined($bridges);
> +        push $placements->@*, $placement;
> +    }
> +
> +    return $placements;
> +}
> +
>  sub print_hostpci_devices {
>      my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_;
>  
> @@ -376,6 +473,15 @@ sub print_hostpci_devices {
>      my $pciaddr;
>      my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run);
>  
> +    # the placements can only be derived after we chose the pci devices
> +    my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine});
> +    my $topology = $machine_conf->{'pci-topology'} // 'flat';
> +    my $placements;
> +    if ($topology ne 'flat') {
> +        $placements = get_hostpcie_placements($conf, $pci_devices, $topology);
> +        push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*;
> +    }
> +
>      for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) {
>          my $id = "hostpci$i";
>          my $d = $pci_devices->{$id};
> @@ -384,8 +490,7 @@ sub print_hostpci_devices {
>          $legacy_igd = 1 if $d->{'legacy-igd'};
>  
>          if (my $pcie = $d->{pcie}) {
> -            # the root ports come from pci_bus_devices
> -            $pciaddr = $hw_config->addr("hostpcie$i");
> +            $pciaddr = $hw_config->addr("hostpcie$i", $placements);
>          } else {
>              my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id;
>              $pciaddr = $hw_config->addr($pci_name);
> @@ -711,6 +816,12 @@ sub hardware_config {
>          $mainboard = "i440fx";
>      }
>  
> +    my $topology = $machine_conf->{'pci-topology'} // 'flat';
> +    my $layout = $machine_conf->{'pci-layout'} // 'legacy';
> +
> +    die "PCI topology '$topology' requires the 'grouped' PCI layout\n"
> +        if $topology ne 'flat' && $layout eq 'legacy';
> +
>      my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version);
>  
>      return PVE::QemuServer::RS::PCI::HardwareConfig->new({
> @@ -729,9 +840,9 @@ sub hardware_config {
>          },
>          'legacy-igd' => $legacy_igd,
>          'ostype' => $conf->{ostype} // 'other',
> -        'layout' => $machine_conf->{'pci-layout'} // 'legacy',
> +        'layout' => $layout,
>          'hostpcie-devices' => $hostpcie,
> -        'hostpci-topology' => 'flat',
> +        'hostpci-topology' => $topology,
>      });
>  }
>  
> diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm
> index 7fd4cc23..1ede3512 100644
> --- a/src/test/QemuServerTest/CommandLineMocks.pm
> +++ b/src/test/QemuServerTest/CommandLineMocks.pm
> @@ -168,6 +168,15 @@ my $pci_devs = [
>      "0000:02:00.5",
>  ];
>  
> +# where the devices on a host bus are connected: the NUMA node and the bridges above them,
> +# starting at the root port. Buses not listed here are on node 0 without any bridge.
> +my $pci_host_topology = {
> +    '01' => { node => 0, bridges => ['0000:00:01.0'] },
> +    '07' => { node => 0, bridges => ['0000:00:03.0'] },
> +    # behind a switch
> +    'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] },
> +};
> +
>  my $pci_map_config = {
>      ids => {
>          someGpu => {
> @@ -615,6 +624,18 @@ $pci_module->mock(
>      reserve_pci_usage => sub {
>          die "reserve_pci_usage should not be called for 'qm showcmd'\n";
>      },
> +    get_host_numa_node => sub {
> +        my ($pciid) = @_;
> +
> +        my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> +        return $pci_host_topology->{$bus}->{node} // 0;
> +    },
> +    get_host_pci_bridges => sub {
> +        my ($pciid) = @_;
> +
> +        my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/;
> +        return [($pci_host_topology->{$bus}->{bridges} // [])->@*];
> +    },
>  );
>  
[...]




  reply	other threads:[~2026-11-19 12:42 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47   ` Elias Huhsovitz
2026-11-19 14:51     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43   ` Elias Huhsovitz
2026-11-19 15:10     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42   ` Elias Huhsovitz [this message]
2026-11-19 15:12     ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Elias Huhsovitz
2026-11-19 12:33   ` Dominik Csapak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZ725CQIUNC.NVRVO5XKJQF9@proxmox.com \
    --to=e.huhsovitz@proxmox.com \
    --cc=d.csapak@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal