From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9B2561FF0B7 for ; Fri, 13 Nov 2026 15:40:45 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 821D221788; Fri, 13 Nov 2026 15:40:07 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [PATCH manager/proxmox-perl-rs/qemu-server/qemu-server-rs v4 00/14] pci-handling rewrite Date: Fri, 13 Nov 2026 15:39:16 +0100 Message-ID: <20261113143939.3012964-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.306 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: JJSBOJLBCPBHYNALF4C5Z6QGARODY3HV X-Message-ID-Hash: JJSBOJLBCPBHYNALF4C5Z6QGARODY3HV X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: # Motivation We want to extend the PCI layout mechanism to be easily extendable, especially for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0]). This is necessary for some performance gains in guests, especially with high-end hardware where users pay extra for higher performance. # The Problem PCI addresses (and hardware layout in general) must be fixed for our qemu guests, since this is part of the machine state and must be stable for live-migration suspend/resume, etc. Currently, these addresses are hardcoded in perl hashes, with a long history of devs (me included) piling new addresses on top, using whatever address was free. This led to a very convoluted and scattered assignment list, without any underlying structure, which makes it harder the more new devices are added. (Using a wrong address only surfaces when running some tests or starting a guest with it) # My Proposed Solution So to fix the current situation I propose a plan in multiple phases, which are now all part of this series, so i didn't mark it as RFC anymore. (The rust part should be mostly fine now sans naming etc.) ## Phase 1 - untangling perl code to rust To extend/switch between multiple PCI layouts, we first must untangle the current situation and make it easier to see what is actually used. I did this in rust, by using the type system to represent the PCI layout, and inferring the addresses from the layout, not the other way around. That makes it easier to see where there is actually free space and where we can add new things. It also highlights how messy the current layout is. (See the legacy layout in the `pve-qemu-server-pci` crate) I also introduced here a generic `HardwareConfig` struct that's intended to hold information useful to more call sites in contrast to passing a lot of arguments everywhere, similar to the Cfg2Cmd class, but usable in the rust codebase. ## Phase 2 - extending pci layouts This introduces a new layout ("grouped") that can be used in an opt-in manner, and is vastly more logically constructed, which makes it easy to extend and reason about. This part entails restructuring the addr calls in qemu-server's perl code, so that there is only a single entry point ("address" for example) instead of splitting between 'print_pci_addr' and 'print_pcie_addr'. This is folded into the commit to using the rust bindings, since it does not make sense to recreate the old interface in rust, just to replace it with the next patches anyway. This now also creates the commandline for the q35 devices instead of using 'readconfig' (guarded behind a new machine version). ## Phase 3 - more perl to rust code port The HardwareConfig also contains a version comparison, so we can widely reuse it in many methods, since we need it for the 'addr' interface anyway. These changes are also folded for the same reason as above. ## Phase 4 - Host and NUMA layouts (the next part) With all the pieces in place, we can then easily extend the new layout by PCI switches that are needed for assigning to NUMA nodes and replicating the host topology for passed through devices. This is a separate setting on top of the grouped layout instead of more layouts: machine: q35,pci-layout=grouped,pci-topology= * flat (default): every hostpci device with pcie=1 gets its own root port on the root bus * numa: adds a pxb-pcie expander bridge per guest NUMA node and puts the root ports of that node's devices behind it * host: additionally replicates the host PCIe switches that more than one passed through device is behind The expander bridges go into slots of the grouped layout that were unused before, so enabling a topology does not move any of the other devices. qemu-server reads the NUMA node and the bridges of each device from sysfs, so nothing has to be configured per device. Since the placement depends on the hardware of the host, live migration is prevented for all topologies except 'flat'. The pve-manager patch only makes sure that the two new options don't get lost when editing the machine in the UI, they are not exposed there (yet). # Notes * pve-qemu-server-rs is a new git repo, but the crate in it could easily be integrated somewhere else. Having this as a separate repo could replace 'qemu-server' at one point though. * This is an RFC, so comments about the general design/plan are desired. * Names and crate placement are not fixed, I know that some names I chose are not optimal, if you do have better names/places, please suggest them. * Opted to do this in rust for now, since I think that is the best way forward. If the consensus is to do the layouting changes in perl, or keep some parts in perl, that's fine with me too and I'll change my efforts accordingly * The repos depend on each other in the order pve-qemu-server-rs -> proxmox-perl-rs -> qemu-server. The parameters of the bindings are all required and unknown ones are rejected, so the qemu-server patches only work with bindings that are built from the matching state of the crate. E.g. the 'hostpci-topology' parameter is only passed with the last qemu-server patch, so with bindings built from the whole series, the earlier qemu-server patches don't work on their own. * Neither 'pci-layout' nor 'pci-topology' is bound to a (new) machine version, both are opt-in via the machine property and work with every machine version. That means that all nodes in a cluster must know the options before they are used, since older nodes can't parse the machine property with them. * The options are only checked when starting the guest, not when setting them, e.g. a non-flat topology with the legacy layout or without q35 only fails on start. Could be added later still though. * For the 'numa' and 'host' topologies, the guest to host NUMA node mapping is taken from the 'numaN' entries. If there are none, we assume that guest node N is on host node N. Not sure if we should require an explicit binding here, opinions welcome. (Similar for devices where the host does not report a NUMA node, these currently count as node 0.) * The bus numbers behind the expander bridges can shift when hostpci devices are added or removed. Since hostpci devices can't be hotplugged and the topologies prevent live migration, this only happens on a fresh start. * Hostpci devices with pcie=1 now always get a root port on function 0 of their slot, also for existing machine versions, see the commit message of 'pci: legacy: generate q35 chipset devices and hostpcie root ports'. * @Fiona: i did not implement every change: - the checks in `PciAddress::new` and `AddressAssignment::new` are only `debug_assert!`s, to keep the constructors free of `Result`s - no additional devices on the root bus/no multiple instances of the builtin devices (as written in my replies, we can add more functions per slot or more bridges if we ever need to, and there is 'args') - unknown 'scsihw' and 'ostype' values are still not an error in the crate, since the types from the api-types crate accept them. Perl validates and defaults them before though. - the maximal configs are already tested by the 'full-*' cfg2cmd tests that are on master now changes from v3: - pve-qemu-server-rs: * new patches for the 'numa' and 'host' topologies: * `HardwareConfigParams` only takes explicit values and has no fallbacks anymore: * the mainboard is no longer derived from the architecture, but must be given explicitly * new `machine` module with `Machine`, `MachineVersion` and `Mainboard` instead of the version tuple in `HardwareConfig` * group the scsi options of `HardwareConfig` in `ScsiOptions` * add `AddressAssignment::new` that checks that multifunction is only set on function 0, and check the function number in `PciAddress::new` * rename `FUNCTIONS_PER_DEVICE` to `MAX_FUNCTIONS_PER_DEVICE_SLOT` * drop the outdated documentation about the fallbacks of `HardwareConfig::new` * move the comment about the generic 'pve.' bus prefix to the patch that introduces the prefix * add Fiona's R-b to 'pci: legacy: generate pci bridges' (only adapted to the changed structs) * adapt tests - proxmox-perl-rs: * rebased on current master * adapt to the changed parameters of the crate * new patch: add `hostpcie_bus_devices` and an optional placements parameter for `addr` - qemu-server: * rebased on current master * build the nested parameter hash for the HardwareConfig, derive the mainboard from the machine type and pass explicit defaults for 'scsihw' and 'ostype' * adapt the expected output of the cfg2cmd tests that were added on master in the meantime * drop the now unused `$winversion` parameter of `print_hostpci_devices` * drop the virtiofs dir mapping mock, master has one now * new patch: add the 'pci-topology' machine option - pve-manager: * new patch: don't lose the pci options when editing the machine changes from v2: - pve-qemu-server-rs: * `HardwareConfig::new` now takes a `HardwareConfigParams` struct, which can be deserialized directly and rejects unknown keys * the layout is deserialized directly into `PciLayout`, so an unknown layout is now an error instead of silently falling back to the legacy one * hostpcie devices are passed as a list of objects (`{ index => N }`) instead of plain indices, so they can be extended later without changing the interface * use an internal `Arch` enum, so the config can't hold an unknown architecture * grouped layout: only set 'multifunction=on' on bridges that share their slot with other bridges (currently only the net bridges) * the `Display` implementation of `Device` now prints the configuration ids (e.g. 'qga0', 'balloon0', 'virtioN') and is exhaustive * the xhci controller has no index anymore (we can add more ports with hubs) * document that `version_at_least` records the required pve version, and use relaxed ordering for it * check the bridge count with `assert!` instead of `debug_assert!` * drop the unused anyhow and log dependencies * code style and doc fixes (`'static` layout lifetime, `match` in `map_legacy_pci_id`, comments) - proxmox-perl-rs: * `new` takes a single hash with the parameters instead of a list of positional arguments * `addr` returns undef for devices that are not part of the layout instead of dying * use 'this' for the object parameter and add the 'Class method'/'Method' doc prefixes * doc fixes - qemu-server: * rebased on current master * construct the HardwareConfig with a parameter hash, and pass the hostpcie devices as objects * drop the now unused `PVE::QemuServer::Machine` import from USB.pm * grouped layout tests: adapt to the bridges without 'multifunction=on' 0: https://bugzilla.proxmox.com/show_bug.cgi?id=7283 pve-qemu-server-rs: Dominik Csapak (6): add pve-qemu-server-pci crate for guest PCI address generation pci: legacy: generate pci bridges pci: legacy: generate q35 chipset devices and hostpcie root ports pci: layout: add grouped PCI and PCIe layouts pci: bus: add buses for numa expander bridges and pcie switches pci: topology: place hostpcie devices like on the host proxmox-perl-rs: Dominik Csapak (2): pve: add bindings for `pve-qemu-server-pci` crate pve: pci: add host placements for the hostpci topology pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 1 + pve-rs/src/bindings/mod.rs | 3 + pve-rs/src/bindings/pci/hardware_config.rs | 143 +++++++++++++++++++++ pve-rs/src/bindings/pci/mod.rs | 2 + 5 files changed, 150 insertions(+) create mode 100644 pve-rs/src/bindings/pci/hardware_config.rs create mode 100644 pve-rs/src/bindings/pci/mod.rs qemu-server: Dominik Csapak (5): pci: use PVE::RS::PCI bindings for addresses and bus devices tree-wide: use HardwareConfig for the machine version checks cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 machine: add 'pci-layout' option machine: add 'pci-topology' option src/PVE/API2/Qemu.pm | 9 +- src/PVE/QemuMigrate.pm | 6 + src/PVE/QemuServer.pm | 264 +++++----- src/PVE/QemuServer/Drive.pm | 7 +- src/PVE/QemuServer/DriveDevice.pm | 12 +- src/PVE/QemuServer/Machine.pm | 36 ++ src/PVE/QemuServer/PCI.pm | 459 ++++++------------ src/PVE/QemuServer/RNG.pm | 6 +- src/PVE/QemuServer/USB.pm | 62 ++- src/PVE/QemuServer/Virtiofs.pm | 7 +- src/test/Makefile | 5 +- src/test/QemuServerTest/CommandLineMocks.pm | 21 + src/test/cfg2cmd/audio-spice.conf.cmd | 21 +- .../cfg2cmd/disk-frontend-properties.conf.cmd | 21 +- .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd | 21 +- src/test/cfg2cmd/full-q35.conf.cmd | 45 +- .../machine-power-state-override.conf.cmd | 21 +- ...-grouped-layout-hostpci-host-topology.conf | 19 + ...uped-layout-hostpci-host-topology.conf.cmd | 57 +++ src/test/cfg2cmd/q35-grouped-layout.conf | 17 + src/test/cfg2cmd/q35-grouped-layout.conf.cmd | 47 ++ .../cfg2cmd/q35-grouped-virtiofs-viommu.conf | 14 + .../q35-grouped-virtiofs-viommu.conf.cmd | 46 ++ src/test/cfg2cmd/q35-ide.conf.cmd | 21 +- .../q35-linux-created-qemu-6.1.conf.cmd | 21 +- .../cfg2cmd/q35-linux-hostpci-11.1+pve1.conf | 21 + .../q35-linux-hostpci-11.1+pve1.conf.cmd | 61 +++ .../q35-linux-hostpci-driver-keep.conf.cmd | 25 +- .../q35-linux-hostpci-mapping.conf.cmd | 21 +- .../q35-linux-hostpci-multifunction.conf.cmd | 21 +- .../q35-linux-hostpci-nvidia-vgpu.conf.cmd | 21 +- src/test/cfg2cmd/q35-linux-hostpci-wrong.conf | 16 + .../cfg2cmd/q35-linux-hostpci-wrong.conf.cmd | 41 ++ ...q35-linux-hostpci-x-pci-overrides.conf.cmd | 21 +- src/test/cfg2cmd/q35-linux-hostpci.conf.cmd | 25 +- src/test/cfg2cmd/q35-linux-ivshmem.conf.cmd | 21 +- src/test/cfg2cmd/q35-scsihw2.conf.cmd | 21 +- src/test/cfg2cmd/q35-simple.conf.cmd | 21 +- .../cfg2cmd/q35-viommu-intel-aw-bits.conf.cmd | 21 +- .../q35-viommu-intel-guest-phys-bits.conf.cmd | 21 +- src/test/cfg2cmd/q35-viommu-intel.conf.cmd | 21 +- .../q35-viommu-virtio-aw-bits.conf.cmd | 21 +- src/test/cfg2cmd/q35-viommu-virtio.conf.cmd | 21 +- src/test/cfg2cmd/q35-win10-hostpci.conf.cmd | 2 +- .../simple-grouped-virtiofs-viommu.conf | 16 + .../simple-grouped-virtiofs-viommu.conf.cmd | 45 ++ src/test/cfg2cmd/simple-grouped.conf | 15 + src/test/cfg2cmd/simple-grouped.conf.cmd | 41 ++ src/test/cfg2cmd/spice-win-grouped.conf | 14 + src/test/cfg2cmd/spice-win-grouped.conf.cmd | 48 ++ src/test/cfg2cmd/tdx-attestation.conf.cmd | 21 +- src/test/cfg2cmd/virtiofs-viommu.conf | 16 + src/test/cfg2cmd/virtiofs-viommu.conf.cmd | 40 ++ src/test/run_pci_addr_checks.pl | 141 ------ 54 files changed, 1401 insertions(+), 704 deletions(-) create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf.cmd create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf.cmd create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf.cmd create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf.cmd create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf.cmd create mode 100644 src/test/cfg2cmd/simple-grouped.conf create mode 100644 src/test/cfg2cmd/simple-grouped.conf.cmd create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf.cmd create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf.cmd delete mode 100755 src/test/run_pci_addr_checks.pl pve-manager: Dominik Csapak (1): ui: qemu machine: don't lose pci options on edit cycle www/manager6/qemu/MachineEdit.js | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) Summary over all repositories: 60 files changed, 1575 insertions(+), 709 deletions(-) -- Generated by murpp 0.11.0