From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 63DA31FF0C0 for ; Wed, 18 Nov 2026 16:29:31 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1141C2186C; Wed, 18 Nov 2026 16:29:05 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Date: Wed, 18 Nov 2026 16:27:54 +0100 Message-ID: <20261118152803.3843758-13-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118152803.3843758-1-d.csapak@proxmox.com> References: <20261118152803.3843758-1-d.csapak@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.717 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 2RB73OVUGO4QMPQGOLIMGY5NIS5UVGJN X-Message-ID-Hash: 2RB73OVUGO4QMPQGOLIMGY5NIS5UVGJN X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Makes the hostpci topology of the PCI bindings configurable, so that hostpci devices with pcie=1 can be placed in the guest like on the host. The default is 'flat', which keeps the current placement. For the 'numa' and 'host' topologies, read the NUMA node of each device from sysfs, and for 'host' also the bridges above it, and pass them as placements to the bindings. This happens after choosing the host devices, since only then it's known which ones are used, and choosing them reserves them. For mediated devices, the actual device is only chosen when starting the guest, so all alternatives must be on the same host NUMA node and only their common bridges are used. The guest must have NUMA enabled with a guest node bound to the host node of each device, as the expander bridges can only be assigned to existing guest NUMA nodes. Both topologies also need the 'grouped' PCI layout. Since the placement depends on the hardware of the host, prevent live migration for all topologies except 'flat'. Signed-off-by: Dominik Csapak --- src/PVE/QemuMigrate.pm | 6 + src/PVE/QemuServer/Machine.pm | 20 +++ src/PVE/QemuServer/PCI.pm | 119 +++++++++++++++++- src/test/QemuServerTest/CommandLineMocks.pm | 21 ++++ ...-grouped-layout-hostpci-host-topology.conf | 19 +++ ...uped-layout-hostpci-host-topology.conf.cmd | 57 +++++++++ 6 files changed, 238 insertions(+), 4 deletions(-) create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm index 2d362140..73a49a50 100644 --- a/src/PVE/QemuMigrate.pm +++ b/src/PVE/QemuMigrate.pm @@ -270,6 +270,12 @@ sub prepare { } } + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine}); + if (my $topology = $machine_conf->{'pci-topology'}) { + die "can't live migrate running VM with PCI topology '$topology'\n" + if $running && $topology ne 'flat'; + } + my ($loc_res, $mapped_res, $missing_mappings_by_node) = PVE::QemuMigrate::Helpers::check_local_resources($conf, $running, 1); my $blocking_resources = []; diff --git a/src/PVE/QemuServer/Machine.pm b/src/PVE/QemuServer/Machine.pm index 466538de..77a9e5b3 100644 --- a/src/PVE/QemuServer/Machine.pm +++ b/src/PVE/QemuServer/Machine.pm @@ -111,6 +111,26 @@ my $machine_fmt = { . " its own set of bridges. Changing it changes the addresses of all devices. This" . " option is experimental.", }, + 'pci-topology' => { + type => 'string', + optional => 1, + default => 'flat', + enum => [qw(flat numa host)], + description => + "Determines how hostpci devices with pcie set are laid out on the guest.", + verbose_description => <%*); + + # the expander bridges can only be assigned to existing guest NUMA nodes + die "PCI topology '$topology' requires NUMA to be enabled\n" if !$conf->{numa}; + + # we need a clear host to guest numa node mapping + my $guest_to_host = PVE::QemuServer::Memory::get_numa_guest_to_host_map($conf); + my $host_to_guest = {}; + for my $guest_node (sort { $a <=> $b } keys $guest_to_host->%*) { + my $host_node = $guest_to_host->{$guest_node}; + next if !defined($host_node) || $host_node !~ m/^\d+$/; + $host_to_guest->{$host_node} //= $guest_node; + } + + my $placements = []; + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { + my $d = $pci_devices->{"hostpci$i"}; + next if !$d || !$d->{pcie}; + + # For mediated devices, the actual device is only chosen when starting the guest, so only + # use numa nodes/bridges that all alternatives have in common. + my @pciids = map { $_->{id} } $d->{ids}->@*; + @pciids = ($pciids[0]) if !$d->{mdev}; + + my ($host_node, $bridges); + for my $pciid (@pciids) { + my $node = get_host_numa_node($pciid); + $host_node //= $node; + die "hostpci$i: PCI topology '$topology' requires all alternative devices to be on" + . " the same host NUMA node\n" + if $node != $host_node; + + next if $topology ne 'host'; + + my $current = get_host_pci_bridges($pciid); + if (!defined($bridges)) { + $bridges = $current; + next; + } + # keep the bridges shared by all alternatives + my $common = []; + for (my $j = 0; $j < scalar(@$bridges) && $j < scalar(@$current); $j++) { + last if $bridges->[$j] ne $current->[$j]; + push @$common, $bridges->[$j]; + } + $bridges = $common; + } + + my $guest_node = $host_to_guest->{$host_node}; + die "hostpci$i: no guest NUMA node is bound to host NUMA node $host_node\n" + if !defined($guest_node); + + my $placement = { index => $i, 'numa-node' => int($guest_node) }; + $placement->{switches} = $bridges if defined($bridges); + push $placements->@*, $placement; + } + + return $placements; +} + sub print_hostpci_devices { my ($vmid, $conf, $devices, $vga, $hw_config, $bootorder, $dry_run) = @_; @@ -376,6 +473,15 @@ sub print_hostpci_devices { my $pciaddr; my $pci_devices = choose_hostpci_devices(parse_hostpci_devices($conf), $vmid, $dry_run); + # the placements can only be derived after we chose the pci devices + my $machine_conf = PVE::QemuServer::Machine::parse_machine($conf->{machine}); + my $topology = $machine_conf->{'pci-topology'} // 'flat'; + my $placements; + if ($topology ne 'flat') { + $placements = get_hostpcie_placements($conf, $pci_devices, $topology); + push @$devices, $hw_config->hostpcie_bus_devices($placements)->@*; + } + for (my $i = 0; $i < $MAX_HOSTPCI_DEVICES; $i++) { my $id = "hostpci$i"; my $d = $pci_devices->{$id}; @@ -384,8 +490,7 @@ sub print_hostpci_devices { $legacy_igd = 1 if $d->{'legacy-igd'}; if (my $pcie = $d->{pcie}) { - # the root ports come from pci_bus_devices - $pciaddr = $hw_config->addr("hostpcie$i"); + $pciaddr = $hw_config->addr("hostpcie$i", $placements); } else { my $pci_name = $d->{'legacy-igd'} ? 'legacy-igd' : $id; $pciaddr = $hw_config->addr($pci_name); @@ -711,6 +816,12 @@ sub hardware_config { $mainboard = "i440fx"; } + my $topology = $machine_conf->{'pci-topology'} // 'flat'; + my $layout = $machine_conf->{'pci-layout'} // 'legacy'; + + die "PCI topology '$topology' requires the 'grouped' PCI layout\n" + if $topology ne 'flat' && $layout eq 'legacy'; + my ($major, $minor, $patch, $pve) = PVE::QemuServer::Helpers::version_parts($machine_version); return PVE::QemuServer::RS::PCI::HardwareConfig->new({ @@ -729,9 +840,9 @@ sub hardware_config { }, 'legacy-igd' => $legacy_igd, 'ostype' => $conf->{ostype} // 'other', - 'layout' => $machine_conf->{'pci-layout'} // 'legacy', + 'layout' => $layout, 'hostpcie-devices' => $hostpcie, - 'hostpci-topology' => 'flat', + 'hostpci-topology' => $topology, }); } diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm index 7fd4cc23..1ede3512 100644 --- a/src/test/QemuServerTest/CommandLineMocks.pm +++ b/src/test/QemuServerTest/CommandLineMocks.pm @@ -168,6 +168,15 @@ my $pci_devs = [ "0000:02:00.5", ]; +# where the devices on a host bus are connected: the NUMA node and the bridges above them, +# starting at the root port. Buses not listed here are on node 0 without any bridge. +my $pci_host_topology = { + '01' => { node => 0, bridges => ['0000:00:01.0'] }, + '07' => { node => 0, bridges => ['0000:00:03.0'] }, + # behind a switch + 'd0' => { node => 1, bridges => ['0000:c0:01.0', '0000:ce:00.0', '0000:cf:08.0'] }, +}; + my $pci_map_config = { ids => { someGpu => { @@ -615,6 +624,18 @@ $pci_module->mock( reserve_pci_usage => sub { die "reserve_pci_usage should not be called for 'qm showcmd'\n"; }, + get_host_numa_node => sub { + my ($pciid) = @_; + + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; + return $pci_host_topology->{$bus}->{node} // 0; + }, + get_host_pci_bridges => sub { + my ($pciid) = @_; + + my ($bus) = $pciid =~ m/^[0-9a-f]{4}:([0-9a-f]{2}):/; + return [($pci_host_topology->{$bus}->{bridges} // [])->@*]; + }, ); our $mdev_module = Test::MockModule->new("PVE::QemuServer::PCI::Mdev"); diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf new file mode 100644 index 00000000..db2efa90 --- /dev/null +++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf @@ -0,0 +1,19 @@ +# TEST: Config with q35, NUMA, grouped pci-layout & hostpci passthrough with host pci-topology +bios: ovmf +bootdisk: scsi0 +cores: 1 +efidisk0: local:100/vm-100-disk-1.qcow2,size=128K +hostpci0: d0:13.0,pcie=1 +hostpci1: d0:15.1,pcie=1 +hostpci2: mapping=someNic,pcie=1 +hostpci3: mapping=someGpu,mdev=some-model,pcie=1 +hostpci4: 00:f4.0 +machine: q35,pci-layout=grouped,pci-topology=host +memory: 512 +net0: virtio=2E:01:68:F9:9C:87,bridge=vmbr0 +numa: 1 +ostype: l26 +scsihw: virtio-scsi-pci +smbios1: uuid=3dd750ce-d910-44d0-9493-525c0be4e687 +sockets: 2 +vmgenid: 54d1c06c-8f5b-440f-b5b2-6eab1380e13d diff --git a/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd new file mode 100644 index 00000000..f1f9b6e6 --- /dev/null +++ b/src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd @@ -0,0 +1,57 @@ +/usr/bin/kvm \ + -id 8006 \ + -name vm8006 \ + -no-shutdown \ + -chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' \ + -object 'monitor-qmp,chardev=qmp,id=monitor-qmp' \ + -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \ + -object 'monitor-qmp,chardev=qmp-event,id=monitor-qmp-event' \ + -pidfile /var/run/qemu-server/8006.pid \ + -daemonize \ + -smbios 'type=1,uuid=3dd750ce-d910-44d0-9493-525c0be4e687' \ + -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \ + -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE.fd"},"node-name":"pflash0","read-only":true}' \ + -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/100/vm-100-disk-1.qcow2","node-name":"e70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"f70e3017c5a79fdee5a04aa92ac1e9c","read-only":false},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"}' \ + -global 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' \ + -smp '2,sockets=2,cores=1,maxcpus=2' \ + -nodefaults \ + -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \ + -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \ + -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \ + -m 512 \ + -object 'memory-backend-ram,id=ram-node0,size=256M' \ + -numa 'node,nodeid=0,cpus=0,memdev=ram-node0' \ + -object 'memory-backend-ram,id=ram-node1,size=256M' \ + -numa 'node,nodeid=1,cpus=1,memdev=ram-node1' \ + -global 'ICH9-LPC.disable_s3=1' \ + -global 'ICH9-LPC.disable_s4=1' \ + -device 'pcie-pci-bridge,id=pve.sys0,bus=pcie.0,addr=0x4' \ + -device 'pcie-pci-bridge,id=pve.virtio-fs0,bus=pcie.0,addr=0x5' \ + -device 'pcie-pci-bridge,id=pve.net0,bus=pcie.0,addr=0x6,multifunction=on' \ + -device 'pcie-pci-bridge,id=pve.net1,bus=pcie.0,addr=0x6.1' \ + -device 'pcie-pci-bridge,id=pve.scsi0,bus=pcie.0,addr=0x7' \ + -device 'pcie-pci-bridge,id=pve.virtio-blk0,bus=pcie.0,addr=0x8' \ + -device 'pcie-pci-bridge,id=pve.hostpci0,bus=pcie.0,addr=0x9' \ + -device 'vmgenid,guid=54d1c06c-8f5b-440f-b5b2-6eab1380e13d' \ + -device 'qemu-xhci,p2=15,p3=15,id=xhci,bus=pve.sys0,addr=0x3' \ + -device 'usb-hub,id=hub,bus=xhci.0,port=15' \ + -device 'usb-tablet,id=tablet,bus=xhci.0,port=15.7' \ + -device 'pxb-pcie,id=pve.numa0,bus=pcie.0,addr=0x10,bus_nr=32,numa_node=0' \ + -device 'pcie-root-port,id=pve.root-port2,bus=pve.numa0,addr=0x3,port=2,chassis=128,slot=2,x-speed=16,x-width=32' \ + -device 'pcie-root-port,id=pve.root-port3,bus=pve.numa0,addr=0x4,port=3,chassis=128,slot=3,x-speed=16,x-width=32' \ + -device 'pxb-pcie,id=pve.numa1,bus=pcie.0,addr=0x11,bus_nr=48,numa_node=1' \ + -device 'pcie-root-port,id=pve.switch-uplink0,bus=pve.numa1,addr=0x1,port=0,chassis=129,slot=0,x-speed=16,x-width=32' \ + -device 'x3130-upstream,id=pve.switch0,bus=pve.switch-uplink0,addr=0x0' \ + -device 'xio3130-downstream,id=pve.switch-port0,bus=pve.switch0,addr=0x0,port=0,chassis=144,slot=0' \ + -device 'xio3130-downstream,id=pve.switch-port1,bus=pve.switch0,addr=0x1,port=1,chassis=144,slot=1' \ + -device 'vfio-pci,host=0000:d0:13.0,id=hostpci0,bus=pve.switch-port0,addr=0x0' \ + -device 'vfio-pci,host=0000:d0:15.1,id=hostpci1,bus=pve.switch-port1,addr=0x0' \ + -device 'vfio-pci,host=0000:07:10.0,id=hostpci2,bus=pve.root-port2,addr=0x0' \ + -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000003-0000-0000-0000-000000008006,id=hostpci3,bus=pve.root-port3,addr=0x0' \ + -device 'vfio-pci,host=0000:00:f4.0,id=hostpci4,bus=pve.hostpci0,addr=0x5' \ + -device 'VGA,id=vga,bus=pcie.0,addr=0x2' \ + -device 'virtio-balloon-pci,id=balloon0,bus=pve.sys0,addr=0x2,free-page-reporting=on' \ + -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \ + -netdev 'type=tap,id=net0,ifname=tap8006i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \ + -device 'virtio-net-pci,mac=2E:01:68:F9:9C:87,netdev=net0,bus=pve.net0,addr=0x1,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300,host_mtu=1500,host_tunnel=off' \ + -machine 'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=q35+pve0' -- 2.47.3