From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 62D671FF0DE for ; Thu, 19 Nov 2026 13:18:58 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 33A50216EB; Thu, 19 Nov 2026 13:18:56 +0100 (CET) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Nov 2026 13:18:50 +0100 Message-Id: Subject: Re: [PATCH manager/qemu-server v5 00/13] pci-handling rewrite From: "Elias Huhsovitz" To: "Dominik Csapak" , X-Mailer: aerc 0.20.0 References: <20261118152803.3843758-1-d.csapak@proxmox.com> In-Reply-To: <20261118152803.3843758-1-d.csapak@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795090730692 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.406 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ADINBMENBS4PEKLWOQTJENXY577HAYHQ X-Message-ID-Hash: ADINBMENBS4PEKLWOQTJENXY577HAYHQ X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Some comments inline. On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: > # Motivation > > We want to extend the PCI layout mechanism to be easily extendable, espec= ially > for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0])= . > > This is necessary for some performance gains in guests, especially with > high-end hardware where users pay extra for higher performance. > > # The Problem > > PCI addresses (and hardware layout in general) must be fixed for our qemu > guests, since this is part of the machine state and must be stable for > live-migration suspend/resume, etc. > > Currently, these addresses are hardcoded in perl hashes, with a long hist= ory of > devs (me included) piling new addresses on top, using whatever address wa= s > free. > > This led to a very convoluted and scattered assignment list, without any > underlying structure, which makes it harder the more new devices are adde= d. > (Using a wrong address only surfaces when running some tests or starting = a > guest with it) > > # My Proposed Solution > > So to fix the current situation I propose a plan in multiple phases, whic= h > are now all part of this series, so i didn't mark it as RFC anymore. > (The rust part should be mostly fine now sans naming etc.) > > ## Phase 1 - untangling perl code to rust > > To extend/switch between multiple PCI layouts, we first must untangle the > current situation and make it easier to see what is actually used. > > I did this in rust, by using the type system to represent the PCI layout,= and > inferring the addresses from the layout, not the other way around. > > That makes it easier to see where there is actually free space and where = we can > add new things. > > It also highlights how messy the current layout is. (See the legacy layou= t in > the `pve-qemu-server-pci` crate) > > I also introduced here a generic `HardwareConfig` struct that's intended = to > hold information useful to more call sites in contrast to passing a lot o= f > arguments everywhere, similar to the Cfg2Cmd class, but usable in the rus= t > codebase. > > ## Phase 2 - extending pci layouts > > This introduces a new layout ("grouped") that can be used in an opt-in ma= nner, > and is vastly more logically constructed, which makes it easy to extend a= nd > reason about. > > This part entails restructuring the addr calls in qemu-server's perl code= , so > that there is only a single entry point ("address" for example) instead o= f > splitting between 'print_pci_addr' and 'print_pcie_addr'. > > This is folded into the commit to using the rust bindings, since it does = not > make sense to recreate the old interface in rust, just to replace it with > the next patches anyway. > > This now also creates the commandline for the q35 devices instead > of using 'readconfig' (guarded behind a new machine version). > > ## Phase 3 - more perl to rust code port > > The HardwareConfig also contains a version comparison, so we can widely r= euse > it in many methods, since we need it for the 'addr' interface anyway. > > These changes are also folded for the same reason as above. > > ## Phase 4 - Host and NUMA layouts (the next part) Isn't this Phase 4 (this part) Or is there more coming? > > With all the pieces in place, we can then easily extend the new layout by= PCI > switches that are needed for assigning to NUMA nodes and replicating the = host > topology for passed through devices. > > This is a separate setting on top of the grouped layout instead of more > layouts: > > machine: q35,pci-layout=3Dgrouped,pci-topology=3D > > * flat (default): every hostpci device with pcie=3D1 gets its own root po= rt on > the root bus > * numa: adds a pxb-pcie expander bridge per guest NUMA node and puts the = root > ports of that node's devices behind it > * host: additionally replicates the host PCIe switches that more than one > passed through device is behind > > The expander bridges go into slots of the grouped layout that were unused > before, so enabling a topology does not move any of the other devices. > qemu-server reads the NUMA node and the bridges of each device from sysfs= , so > nothing has to be configured per device. > > Since the placement depends on the hardware of the host, live migration i= s > prevented for all topologies except 'flat'. Wouldnt it theoretically be possible support live migration for nearly identical host hardware, e.g. for datacenter use cases. Or are there potential pitfalls in comparing topology between hosts? The topology provided by tools like lostop (hwloc) seem consistent enough. > > The pve-manager patch only makes sure that the two new options don't get = lost > when editing the machine in the UI, they are not exposed there (yet). > > # Notes > > * Neither 'pci-layout' nor 'pci-topology' is bound to a (new) machine > version, both are opt-in via the machine property and work with every > machine version. That means that all nodes in a cluster must know the > options before they are used, since older nodes can't parse the machine > property with them. > * The options are only checked when starting the guest, not when setting > them, e.g. a non-flat topology with the legacy layout or without q35 on= ly > fails on start. Could be added later still though. IMO this is the right approach. Otherwise you would have to apply settings in the same "time" step to avoid errors, i.e. when do you know that configuration has finished and it is safe to validate the config? I guess it would be possible, but implementing this cleanly would be a maintenance from my understanding. > * For the 'numa' and 'host' topologies, the guest to host NUMA node mappi= ng > is taken from the 'numaN' entries. If there are none, we assume that gu= est > node N is on host node N. Not sure if we should require an explicit > binding here, opinions welcome. (Similar for devices where the host doe= s > not report a NUMA node, these currently count as node 0.) I think the fix here is to make our numa handling more robust in case no/incomplete numa[N] entires are provided. IMO if numa is enabled but the config is incomplete we should complete the numa[N] automatically internally. As you mentioned we can make reasonable assumptions on what these values should be: * cpus * hostnodes * memory Generally idea on vm start: If no numa[N] is provided: generate a sensible perl hash based on the sockets/cores and host nodes. If numa[N] is povided but incomplete: fill in the gaps (missing cpus,hostnodes,memory) Make the perl hash available to other callers (e.g. pci), so everyone opera= tes on a consistent understanding of the numa config is. (AFAIK this is already somewhat implemented but the generated config is directly passed to the qemu command and no "completed" layout is provided to other callers) I was thinking of doing something like this for the cpu pinning serires. > * The bus numbers behind the expander bridges can shift when hostpci devi= ces > are added or removed. Since hostpci devices can't be hotplugged and the > topologies prevent live migration, this only happens on a fresh start. > * Hostpci devices with pcie=3D1 now always get a root port on function 0 = of > their slot, also for existing machine versions, see the commit message = of > 'pci: legacy: generate q35 chipset devices and hostpcie root ports'. > > changes from v4 (most are packaging related, not code): > * fixed the hotplug allowed check for usb > * move the crate into the qemu-server repo as 'pve-qemu-rs' workspace > * rename the crate from 'pve-qemu-server-pci' to 'pve-qemu-pci' > * move the perl bindings from proxmox-perl-rs into a qemu-server perlmod = library > * rename the bindings from 'PVE::RS::*' to 'PVE::QemuServer::RS::*' > * use an own 'pve-qemu' lib tag for the library > * add the bindings together with the crate features they expose > * tests use the built library directly, no install needed > * new build-dependencies: cargo, rustc, dh-cargo, perlmod-bin, librust-*-= dev Side Note: I gave this a quick test by building+installing the package and doing some basic PCI operations. So this seems fine so far.