From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A57D11FF0DF for ; Thu, 15 Oct 2026 10:11:15 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7ECA521606; Thu, 15 Oct 2026 10:10:52 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [RFC proxmox-perl-rs/qemu-server/qemu-server-rs v2 0/9] pci-handling rewrite (part 1/2/3) Date: Thu, 15 Oct 2026 10:10:22 +0200 Message-ID: <20261015081041.743422-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.366 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 2NGWHTA6KUGBNWPHONXOYQU5LTAPPWDZ X-Message-ID-Hash: 2NGWHTA6KUGBNWPHONXOYQU5LTAPPWDZ X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: # Motivation We want to extend the PCI layout mechanism to be easily extendable, especially for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0]). This is necessary for some performance gains in guests, especially with high-end hardware where users pay extra for higher performance. # The Problem PCI addresses (and hardware layout in general) must be fixed for our qemu guests, since this is part of the machine state and must be stable for live-migration suspend/resume, etc. Currently, these addresses are hardcoded in perl hashes, with a long history of devs (me included) piling new addresses on top, using whatever address was free. This led to a very convoluted and scattered assignment list, without any underlying structure, which makes it harder the more new devices are added. (Using a wrong address only surfaces when running some tests or starting a guest with it) # My Proposed Solution So to fix the current situation I propose a plan in multiple phases: ## Phase 1 - untangling perl code to rust (this series) To extend/switch between multiple PCI layouts, we first must untangle the current situation and make it easier to see what is actually used. I did this in rust, by using the type system to represent the PCI layout, and inferring the addresses from the layout, not the other way around. That makes it easier to see where there is actually free space and where we can add new things. It also highlights how messy the current layout is. (See the legacy layout in the `pve-qemu-server-pci` crate) I also introduced here a generic `HardwareConfig` struct that's intended to hold information useful to more call sites in contrast to passing a lot of arguments everywhere, similar to the Cfg2Cmd class, but usable in the rust codebase. ## Phase 2 - extending pci layouts (this series) This introduces a new layout ("grouped") that can be used in an opt-in manner, and is vastly more logically constructed, which makes it easy to extend and reason about. This part entails restructuring the addr calls in qemu-server's perl code, so that there is only a single entry point ("address" for example) instead of splitting between 'print_pci_addr' and 'print_pcie_addr'. This is folded into the commit to using the rust bindings, since it does not make sense to recreate the old interface in rust, just to replace it with the next patches anyway. This now also creates the commandline for the q35 devices instead of using 'readconfig' (guarded behind a new machine version). ## Phase 3 - more perl to rust code port (this series) The HardwareConfig also contains a version comparison, so we can widely reuse it in many methods, since we need it for the 'addr' interface anyway. These changes are also folded for the same reason as above. ## Phase 4 - Host and NUMA layouts (the next part) With all the pieces in place, we can then easily extend the new layout by PCI switches that are needed for assigning to NUMA nodes and replicating the host topology for passed through devices. # Notes * pve-qemu-server-rs is a new git repo, but the crate in it could easily be integrated somewhere else. Having this as a separate repo could replace 'qemu-server' at one point though. * This is an RFC, so comments about the general design/plan are desired. * Names and crate placement are not fixed, I know that some names I chose are not optimal, if you do have better names/places, please suggest them. * I based this on my recent qemu-server hotplug v3 series [1], so keep that in mind when testing/applying * Opted to do this in rust for now, since I think that is the best way forward. If the consensus is to do the layouting changes in perl, or keep some parts in perl, that's fine with me too and I'll change my efforts accordingly changes from rfc v1: * rebased on the hotplug v3[1] and cleanup[2] series * split out the qemu-server cleanups/fixes into a separate series[2] * this now also includes phase 2 and 3 of the plan - pve-qemu-server-rs: * renamed `Machine` to `HardwareConfig` and extended it (mainboard, layout, hostpcie devices, windows version) * version_at_least now records the required +pveX version like the version guard in qemu-server, and treats a missing pve version as +pve0 * use a single `address` entry point instead of print_pci_addr and print_pcie_addr, the win7 quirk is now handled internally * replaced get_pci_bridges with pci_bus_devices, which also returns the hostpcie root ports and the q35 chipset devices (from 11.1+pve1 on) * add the root port on function 0 of used slots, otherwise the guest does not see the other functions * renamed the 'v2' layout to 'grouped', removed the gaps between the bridges and moved the root ports * take the device limits from pve-api-types instead of hardcoding them * renamed some types and added a `LegacyRootPort` bus variant * more tests - proxmox-perl-rs: * replaced the free functions and the `Machine` class with a single `PVE::RS::PCI::HardwareConfig` class - qemu-server: * removed the print_pci_addr/print_pcie_addr wrappers, construct the HardwareConfig once and pass it around instead of the arch * root ports now come from the rust side, which fixes the invisible hostpcie devices without a root port on function 0 * use the HardwareConfig for the machine version checks * generate the q35 chipset devices with 11.1+pve1 instead of using the readconfig file * add the 'pci-layout' machine option to opt-in to the grouped layout 0: https://bugzilla.proxmox.com/show_bug.cgi?id=7283 1: https://lore.proxmox.com/pve-devel/20261007142145.984970-1-d.csapak@proxmox.com/ 2: https://lore.proxmox.com/pve-devel/20261015075428.572882-1-d.csapak@proxmox.com/ pve-qemu-server-rs: Dominik Csapak (4): add pve-qemu-server-pci crate for guest PCI address generation pci: legacy: generate pci bridges pci: legacy: generate q35 chipset devices and hostpcie root ports pci: layout: add grouped PCI and PCIe layouts proxmox-perl-rs: Dominik Csapak (1): pve: add bindings for `pve-qemu-server-pci` crate pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 1 + pve-rs/src/bindings/mod.rs | 3 + pve-rs/src/bindings/pci/hardware_config.rs | 150 +++++++++++++++++++++ pve-rs/src/bindings/pci/mod.rs | 2 + 5 files changed, 157 insertions(+) create mode 100644 pve-rs/src/bindings/pci/hardware_config.rs create mode 100644 pve-rs/src/bindings/pci/mod.rs qemu-server: Dominik Csapak (4): pci: use PVE::RS::PCI bindings for addresses and bus devices tree-wide: use HardwareConfig for the machine version checks cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 machine: add 'pci-layout' option src/PVE/API2/Qemu.pm | 9 +- src/PVE/QemuServer.pm | 250 +++++++------ src/PVE/QemuServer/Drive.pm | 7 +- src/PVE/QemuServer/DriveDevice.pm | 12 +- src/PVE/QemuServer/Machine.pm | 16 + src/PVE/QemuServer/PCI.pm | 340 ++---------------- src/PVE/QemuServer/RNG.pm | 6 +- src/PVE/QemuServer/USB.pm | 58 +-- src/PVE/QemuServer/Virtiofs.pm | 7 +- src/test/Makefile | 5 +- src/test/TestsCommon/CommandLineMocks.pm | 27 ++ .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd | 21 +- src/test/cfg2cmd/q35-grouped-layout.conf | 17 + src/test/cfg2cmd/q35-grouped-layout.conf.cmd | 47 +++ .../cfg2cmd/q35-grouped-virtiofs-viommu.conf | 14 + .../q35-grouped-virtiofs-viommu.conf.cmd | 46 +++ src/test/cfg2cmd/q35-ide.conf.cmd | 21 +- .../cfg2cmd/q35-linux-hostpci-11.1+pve1.conf | 21 ++ .../q35-linux-hostpci-11.1+pve1.conf.cmd | 61 ++++ .../q35-linux-hostpci-driver-keep.conf.cmd | 25 +- .../q35-linux-hostpci-mapping.conf.cmd | 21 +- .../q35-linux-hostpci-multifunction.conf.cmd | 21 +- src/test/cfg2cmd/q35-linux-hostpci-wrong.conf | 16 + .../cfg2cmd/q35-linux-hostpci-wrong.conf.cmd | 41 +++ ...q35-linux-hostpci-x-pci-overrides.conf.cmd | 21 +- src/test/cfg2cmd/q35-linux-hostpci.conf.cmd | 25 +- src/test/cfg2cmd/q35-linux-ivshmem.conf.cmd | 21 +- src/test/cfg2cmd/q35-scsihw2.conf.cmd | 21 +- src/test/cfg2cmd/q35-simple.conf.cmd | 21 +- .../cfg2cmd/q35-viommu-intel-aw-bits.conf.cmd | 21 +- .../q35-viommu-intel-guest-phys-bits.conf.cmd | 21 +- src/test/cfg2cmd/q35-viommu-intel.conf.cmd | 21 +- .../q35-viommu-virtio-aw-bits.conf.cmd | 21 +- src/test/cfg2cmd/q35-viommu-virtio.conf.cmd | 21 +- src/test/cfg2cmd/q35-win10-hostpci.conf.cmd | 2 +- .../simple-grouped-virtiofs-viommu.conf | 16 + .../simple-grouped-virtiofs-viommu.conf.cmd | 45 +++ src/test/cfg2cmd/simple-grouped.conf | 15 + src/test/cfg2cmd/simple-grouped.conf.cmd | 41 +++ src/test/cfg2cmd/spice-win-grouped.conf | 14 + src/test/cfg2cmd/spice-win-grouped.conf.cmd | 48 +++ src/test/cfg2cmd/virtiofs-viommu.conf | 16 + src/test/cfg2cmd/virtiofs-viommu.conf.cmd | 40 +++ src/test/run_pci_addr_checks.pl | 141 -------- 44 files changed, 1039 insertions(+), 662 deletions(-) create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf.cmd create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf.cmd create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf.cmd create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf.cmd create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf.cmd create mode 100644 src/test/cfg2cmd/simple-grouped.conf create mode 100644 src/test/cfg2cmd/simple-grouped.conf.cmd create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf.cmd create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf.cmd delete mode 100755 src/test/run_pci_addr_checks.pl Summary over all repositories: 49 files changed, 1196 insertions(+), 662 deletions(-) -- Generated by murpp 0.11.0