From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 8136F1FF0B7 for ; Fri, 13 Nov 2026 15:41:45 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 216212183D; Fri, 13 Nov 2026 15:40:18 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v4 13/14] machine: add 'pci-topology' option Date: Fri, 13 Nov 2026 15:39:29 +0100 Message-ID: <20261113143939.3012964-14-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261113143939.3012964-1-d.csapak@proxmox.com> References: <20261113143939.3012964-1-d.csapak@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.694 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RGN7P5YIPO6XNJCARXEYWAQD7BU6NO5S X-Message-ID-Hash: RGN7P5YIPO6XNJCARXEYWAQD7BU6NO5S X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Guest software sometimes uses the PCI topology to choose CPUs and DMA paths. Since every hostpci device with pcie=1 gets a root port on the root bus, the guest can't see which devices are close to which NUMA node or to each other. The new option selects how these devices are placed in the guest: * 'flat': the current behavior and the default * 'numa': puts the devices behind an expander bridge for the guest NUMA node that is bound to their host NUMA node * 'host': additionally replicates the bridges that more than one device is behind on the host as switches For that, read the NUMA node and the bridges of each device from sysfs and pass them to the bindings. This is done after choosing the host devices, since only then it's known which ones are used. For mediated devices, the actual device is only chosen when starting the guest, so only use what all alternatives have in common. The guest must have NUMA enabled with a guest node bound to the host node of each device, as the expander bridges can only be assigned to existing guest NUMA nodes. The bindings only support the topologies with the 'grouped' PCI layout. Since the placement depends on the hardware of the host, prevent live migration for all topologies except 'flat'. Signed-off-by: Dominik Csapak --- src/PVE/QemuMigrate.pm | 6 + src/PVE/QemuServer/Machine.pm | 20 +++ src/PVE/QemuServer/PCI.pm | 118 +++++++++++++++++- src/test/QemuServerTest/CommandLineMocks.pm | 21 ++++ ...-grouped-layout-hostpci-host-topology.conf | 19 +++ ...uped-layout-hostpci-host-topology.conf.cmd | 57 +++++++++ 6 files changed, 238 insertions(+), 3 deletions(-) create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm index 8176afe5..476719bc 100644 --- a/src/PVE/QemuMigrate.pm +++ b/src/PVE/QemuMigrate.pm @@ -268,6 +268,12 @@ sub prepare { } } + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine}); + if (my $topology = $machine_conf->{'pci-topology'}) { + die "can't live migrate running VM with PCI topology '$topology'\n" + if $running && $topology ne 'flat'; + } + my ($loc_res, $mapped_res, $missing_mappings_by_node) = PVE::QemuMigrate::Helpers::check_local_resources($conf, $running, 1); my $blocking_resources = []; diff --git a/src/PVE/QemuServer/Machine.pm b/src/PVE/QemuServer/Machine.pm index f30c1ef3..01673be0 100644 --- a/src/PVE/QemuServer/Machine.pm +++ b/src/PVE/QemuServer/Machine.pm @@ -111,6 +111,26 @@ my $machine_fmt = { . " its own set of bridges. Changing it changes the addresses of all devices. This" . " option is experimental.", }, + 'pci-topology' => { + type => 'string', + optional => 1, + default => 'flat', + enum => [qw(flat numa host)], + description => + "Determines how hostpci devices with pcie set are laid out on the guest.", + verbose_description => <%*); + + # the expander bridges can only be assigned to existing guest NUMA nodes + die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa}; + + # we need a clear host to guest numa node mapping + my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf); + my $host_to_guest = {}; + for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) { + my $host_node = $guest_to_host->{$guest_node}; + next if !defined($host_node) || $host_node !~ m/^\d+$/; + $host_to_guest->{$host_node} //= $guest_node; + } + + my $placements = []; + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { + my $d = $pci_devices->{"hostpci$i"}; + next if !$d || !$d->{pcie}; + + # For mediated devices, the actual device is only chosen when starting the guest, so only + # use numa nodes/bridges that all alternatives have in common. + my @pciids = map { $_->{id} } $d->{ids}->@*; + @pciids = ($pciids[0]) if !$d->{mdev}; + + my ($host_node, $bridges); + for my $pciid (@pciids) { + my $node = get_host_numa_node($pciid); + $host_node //= $node; + die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on" + . " the same host NUMA node\n" + if $node != $host_node; + + next if $topology ne 'host'; + + my $current = get_host_pci_bridges($pciid); + if (!defined($bridges)) { + $bridges = $current; + next; + } + # keep the bridges shared by all alternatives + my $common = []; + for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) { + last if $bridges->[$j] ne $current->[$j]; + push @$common, $bridges->[$j]; + } + $bridges = $common; + } + + my $guest_node = $host_to_guest->{$host_node}; + die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n" + if !defined($guest_node); + + my $placement = { index => $i, 'numa-node' => int($guest_node) }; + $placement->{switches} = $bridges if defined($bridges); + push $placements->@*, $placement; + } + + return $placements; +} + sub print_hostpci_devices { my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_; @@ -376,6 +473,15 @@ sub print_hostpci_devices { my $pciaddr; my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run); + # the placements can only be derived after we chose the pci devices + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine}); + my $topology = $machine_conf->{'pci-topology'} // 'flat'; + my $placements; + if ($topology ne 'flat') { + $placements = get_hostpcie_placements($conf, $pci_devices, $topology); + push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*; + } + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { my $id = "hostpci$i"; my $d = $pci_devices->{$id}; @@ -384,8 +490,7 @@ sub print_hostpci_devices { $legacy_igd = 1 if $d->{'legacy-igd'}; if (my $pcie = $d->{pcie}) { - # the root ports come from pci_bus_devices - $pciaddr = $hw_config->addr("hostpcie$i"); + $pciaddr = $hw_config->addr("hostpcie$i", $placements); } else { my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id; $pciaddr = $hw_config->addr($pci_name); @@ -711,6 +816,12 @@ sub hardware_config { $mainboard = "i440fx"; } + my $topology = $machine_conf->{'pci-topology'} // 'flat'; + my $layout = $machine_conf->{'pci-layout'} // 'legacy'; + + die "PCI topology '$topology' requires the 'grouped' PCI layout\n" + if $topology ne 'flat' && $layout eq 'legacy'; + my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version); return PVE::RS::PCI::HardwareConfig->new({ @@ -729,8 +840,9 @@ sub hardware_config { }, 'legacy-igd' => $legacy_igd, 'ostype' => $conf->{ostype} // 'other', - 'layout' => $machine_conf->{'pci-layout'} // 'legacy', + 'layout' => $layout, 'hostpcie-devices' => $hostpcie, + 'hostpci-topology' => $topology, }); } diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm index 0ae5e6b5..361f1f9a 100644 --- a/src/test/QemuServerTest/CommandLineMocks.pm +++ b/src/test/QemuServerTest/CommandLineMocks.pm @@ -168,6 +168,15 @@ my $pci_devs = [ "0000:02:00.5", ]; +# where the devices on a host bus are connected: the NUMA node and the bridges above them, +# starting at the root port. Buses not listed here are on node 0 without any bridge. +my $pci_host_topology = { + '01' => { node => 0, bridges => ['0000:00:01.0'] }, + '07' => { node => 0, bridges => ['0000:00:03.0'] }, + # behind a switch + 'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] }, +}; + my $pci_map_config = { ids => { someGpu => { @@ -620,6 +629,18 @@ $pci_module->mock( create_nvidia_device => sub { die "create_nvidia_device should not be called for 'qm showcmd'\n"; }, + get_host_numa_node => sub { + my ($pciid) = @_; + + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; + return $pci_host_topology->{$bus}->{node} // 0; + }, + get_host_pci_bridges => sub { + my ($pciid) = @_; + + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; + return [($pci_host_topology->{$bus}->{bridges} // [])->@*]; + }, ); 1; diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf new file mode 100644 index 00000000..db2efa90 --- /dev/null +++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf @@ -0,0 +1,19 @@ +# TEST: Config with q35, NUMA, grouped pci-layout & hostpci passthrough with host pci-topology +bios: ovmf +bootdisk: scsi0 +cores: 1 +efidisk0: local:100/vm-100-disk-1.qcow2,size=128K +hostpci0: d0:13.0,pcie=1 +hostpci1: d0:15.1,pcie=1 +hostpci2: mapping=someNic,pcie=1 +hostpci3: mapping=someGpu,mdev=some-model,pcie=1 +hostpci4: 00:f4.0 +machine: q35,pci-layout=grouped,pci-topology=host +memory: 512 +net0: virtio=2E:01:68:F9:9C:87,bridge=vmbr0 +numa: 1 +ostype: l26 +scsihw: virtio-scsi-pci +smbios1: uuid=3dd750ce-d910-44d0-9493-525c0be4e687 +sockets: 2 +vmgenid: 54d1c06c-8f5b-440f-b5b2-6eab1380e13d diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd new file mode 100644 index 00000000..f1f9b6e6 --- /dev/null +++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd @@ -0,0 +1,57 @@ +/usr/bin/kvm \ + -id 8006 \ + -name vm8006 \ + -no-shutdown \ + -chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' \ + -object 'monitor-qmp,chardev=qmp,id=monitor-qmp' \ + -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \ + -object 'monitor-qmp,chardev=qmp-event,id=monitor-qmp-event' \ + -pidfile /var/run/qemu-server/8006.pid \ + -daemonize \ + -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \ + -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \ + -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \ + -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"}' \ + -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \ + -smp '2,sockets=2,cores=1,maxcpus=2' \ + -nodefaults \ + -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \ + -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \ + -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \ + -m 512 \ + -object 'memory-backend-ram,id=ram-node0,size=256M' \ + -numa 'node,nodeid=0,cpus=0,memdev=ram-node0' \ + -object 'memory-backend-ram,id=ram-node1,size=256M' \ + -numa 'node,nodeid=1,cpus=1,memdev=ram-node1' \ + -global 'ICH9-LPC.disable_s3=1' \ + -global 'ICH9-LPC.disable_s4=1' \ + -device 'pcie-pci-bridge,id=pve.sys0,bus=pcie.0,addr=0x4' \ + -device 'pcie-pci-bridge,id=pve.virtio-fs0,bus=pcie.0,addr=0x5' \ + -device 'pcie-pci-bridge,id=pve.net0,bus=pcie.0,addr=0x6,multifunction=on' \ + -device 'pcie-pci-bridge,id=pve.net1,bus=pcie.0,addr=0x6.1' \ + -device 'pcie-pci-bridge,id=pve.scsi0,bus=pcie.0,addr=0x7' \ + -device 'pcie-pci-bridge,id=pve.virtio-blk0,bus=pcie.0,addr=0x8' \ + -device 'pcie-pci-bridge,id=pve.hostpci0,bus=pcie.0,addr=0x9' \ + -device 'vmgenid,guid=54d1c06c-8f5b-440f-b5b2-6eab1380e13d' \ + -device 'qemu-xhci,p2=15,p3=15,id=xhci,bus=pve.sys0,addr=0x3' \ + -device 'usb-hub,id=hub,bus=xhci.0,port=15' \ + -device 'usb-tablet,id=tablet,bus=xhci.0,port=15.7' \ + -device 'pxb-pcie,id=pve.numa0,bus=pcie.0,addr=0x10,bus_nr=32,numa_node=0' \ + -device 'pcie-root-port,id=pve.root-port2,bus=pve.numa0,addr=0x3,port=2,chassis=128,slot=2,x-speed=16,x-width=32' \ + -device 'pcie-root-port,id=pve.root-port3,bus=pve.numa0,addr=0x4,port=3,chassis=128,slot=3,x-speed=16,x-width=32' \ + -device 'pxb-pcie,id=pve.numa1,bus=pcie.0,addr=0x11,bus_nr=48,numa_node=1' \ + -device 'pcie-root-port,id=pve.switch-uplink0,bus=pve.numa1,addr=0x1,port=0,chassis=129,slot=0,x-speed=16,x-width=32' \ + -device 'x3130-upstream,id=pve.switch0,bus=pve.switch-uplink0,addr=0x0' \ + -device 'xio3130-downstream,id=pve.switch-port0,bus=pve.switch0,addr=0x0,port=0,chassis=144,slot=0' \ + -device 'xio3130-downstream,id=pve.switch-port1,bus=pve.switch0,addr=0x1,port=1,chassis=144,slot=1' \ + -device 'vfio-pci,host=0000:d0:13.0,id=hostpci0,bus=pve.switch-port0,addr=0x0' \ + -device 'vfio-pci,host=0000:d0:15.1,id=hostpci1,bus=pve.switch-port1,addr=0x0' \ + -device 'vfio-pci,host=0000:07:10.0,id=hostpci2,bus=pve.root-port2,addr=0x0' \ + -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000003-0000-0000-0000-000000008006,id=hostpci3,bus=pve.root-port3,addr=0x0' \ + -device 'vfio-pci,host=0000:00:f4.0,id=hostpci4,bus=pve.hostpci0,addr=0x5' \ + -device 'VGA,id=vga,bus=pcie.0,addr=0x2' \ + -device 'virtio-balloon-pci,id=balloon0,bus=pve.sys0,addr=0x2,free-page-reporting=on' \ + -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \ + -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \ + -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pve.net0,addr=0x1,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300,host_mtu=1500,host_tunnel=off' \ + -machine 'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=q35+pve0' -- 2.47.3