From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Dominik Csapak" <d.csapak@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH manager/qemu-server v5 00/13] pci-handling rewrite
Date: Thu, 19 Nov 2026 13:18:50 +0100 [thread overview]
Message-ID: <DMZ6K15N4372.2WQLDHULCW5EK@proxmox.com> (raw)
In-Reply-To: <20261118152803.3843758-1-d.csapak@proxmox.com>
Some comments inline.
On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote:
> # Motivation
>
> We want to extend the PCI layout mechanism to be easily extendable, especially
> for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0]).
>
> This is necessary for some performance gains in guests, especially with
> high-end hardware where users pay extra for higher performance.
>
> # The Problem
>
> PCI addresses (and hardware layout in general) must be fixed for our qemu
> guests, since this is part of the machine state and must be stable for
> live-migration suspend/resume, etc.
>
> Currently, these addresses are hardcoded in perl hashes, with a long history of
> devs (me included) piling new addresses on top, using whatever address was
> free.
>
> This led to a very convoluted and scattered assignment list, without any
> underlying structure, which makes it harder the more new devices are added.
> (Using a wrong address only surfaces when running some tests or starting a
> guest with it)
>
> # My Proposed Solution
>
> So to fix the current situation I propose a plan in multiple phases, which
> are now all part of this series, so i didn't mark it as RFC anymore.
> (The rust part should be mostly fine now sans naming etc.)
>
> ## Phase 1 - untangling perl code to rust
>
> To extend/switch between multiple PCI layouts, we first must untangle the
> current situation and make it easier to see what is actually used.
>
> I did this in rust, by using the type system to represent the PCI layout, and
> inferring the addresses from the layout, not the other way around.
>
> That makes it easier to see where there is actually free space and where we can
> add new things.
>
> It also highlights how messy the current layout is. (See the legacy layout in
> the `pve-qemu-server-pci` crate)
>
> I also introduced here a generic `HardwareConfig` struct that's intended to
> hold information useful to more call sites in contrast to passing a lot of
> arguments everywhere, similar to the Cfg2Cmd class, but usable in the rust
> codebase.
>
> ## Phase 2 - extending pci layouts
>
> This introduces a new layout ("grouped") that can be used in an opt-in manner,
> and is vastly more logically constructed, which makes it easy to extend and
> reason about.
>
> This part entails restructuring the addr calls in qemu-server's perl code, so
> that there is only a single entry point ("address" for example) instead of
> splitting between 'print_pci_addr' and 'print_pcie_addr'.
>
> This is folded into the commit to using the rust bindings, since it does not
> make sense to recreate the old interface in rust, just to replace it with
> the next patches anyway.
>
> This now also creates the commandline for the q35 devices instead
> of using 'readconfig' (guarded behind a new machine version).
>
> ## Phase 3 - more perl to rust code port
>
> The HardwareConfig also contains a version comparison, so we can widely reuse
> it in many methods, since we need it for the 'addr' interface anyway.
>
> These changes are also folded for the same reason as above.
>
> ## Phase 4 - Host and NUMA layouts (the next part)
Isn't this Phase 4 (this part) Or is there more coming?
>
> With all the pieces in place, we can then easily extend the new layout by PCI
> switches that are needed for assigning to NUMA nodes and replicating the host
> topology for passed through devices.
>
> This is a separate setting on top of the grouped layout instead of more
> layouts:
>
> machine: q35,pci-layout=grouped,pci-topology=<flat|numa|host>
>
> * flat (default): every hostpci device with pcie=1 gets its own root port on
> the root bus
> * numa: adds a pxb-pcie expander bridge per guest NUMA node and puts the root
> ports of that node's devices behind it
> * host: additionally replicates the host PCIe switches that more than one
> passed through device is behind
>
> The expander bridges go into slots of the grouped layout that were unused
> before, so enabling a topology does not move any of the other devices.
> qemu-server reads the NUMA node and the bridges of each device from sysfs, so
> nothing has to be configured per device.
>
> Since the placement depends on the hardware of the host, live migration is
> prevented for all topologies except 'flat'.
Wouldnt it theoretically be possible support live migration for nearly
identical host hardware, e.g. for datacenter use cases.
Or are there potential pitfalls in comparing topology between hosts?
The topology provided by tools like lostop (hwloc) seem consistent
enough.
>
> The pve-manager patch only makes sure that the two new options don't get lost
> when editing the machine in the UI, they are not exposed there (yet).
>
> # Notes
>
> * Neither 'pci-layout' nor 'pci-topology' is bound to a (new) machine
> version, both are opt-in via the machine property and work with every
> machine version. That means that all nodes in a cluster must know the
> options before they are used, since older nodes can't parse the machine
> property with them.
> * The options are only checked when starting the guest, not when setting
> them, e.g. a non-flat topology with the legacy layout or without q35 only
> fails on start. Could be added later still though.
IMO this is the right approach. Otherwise you would have to apply
settings in the same "time" step to avoid errors, i.e. when do you know
that configuration has finished and it is safe to validate the config?
I guess it would be possible, but implementing this cleanly would be a
maintenance from my understanding.
> * For the 'numa' and 'host' topologies, the guest to host NUMA node mapping
> is taken from the 'numaN' entries. If there are none, we assume that guest
> node N is on host node N. Not sure if we should require an explicit
> binding here, opinions welcome. (Similar for devices where the host does
> not report a NUMA node, these currently count as node 0.)
I think the fix here is to make our numa handling more robust in case
no/incomplete numa[N] entires are provided. IMO if numa is enabled but
the config is incomplete we should complete the numa[N] automatically
internally.
As you mentioned we can make reasonable assumptions on what these values
should be:
* cpus
* hostnodes
* memory
Generally idea on vm start:
If no numa[N] is provided:
generate a sensible perl hash based on the sockets/cores and host
nodes.
If numa[N] is povided but incomplete:
fill in the gaps (missing cpus,hostnodes,memory)
Make the perl hash available to other callers (e.g. pci), so everyone operates
on a consistent understanding of the numa config is.
(AFAIK this is already somewhat implemented but the generated config is
directly passed to the qemu command and no "completed" layout is
provided to other callers)
I was thinking of doing something like this for the cpu pinning serires.
> * The bus numbers behind the expander bridges can shift when hostpci devices
> are added or removed. Since hostpci devices can't be hotplugged and the
> topologies prevent live migration, this only happens on a fresh start.
> * Hostpci devices with pcie=1 now always get a root port on function 0 of
> their slot, also for existing machine versions, see the commit message of
> 'pci: legacy: generate q35 chipset devices and hostpcie root ports'.
>
> changes from v4 (most are packaging related, not code):
> * fixed the hotplug allowed check for usb
> * move the crate into the qemu-server repo as 'pve-qemu-rs' workspace
> * rename the crate from 'pve-qemu-server-pci' to 'pve-qemu-pci'
> * move the perl bindings from proxmox-perl-rs into a qemu-server perlmod library
> * rename the bindings from 'PVE::RS::*' to 'PVE::QemuServer::RS::*'
> * use an own 'pve-qemu' lib tag for the library
> * add the bindings together with the crate features they expose
> * tests use the built library directly, no install needed
> * new build-dependencies: cargo, rustc, dh-cargo, perlmod-bin, librust-*-dev
Side Note: I gave this a quick test by building+installing the package
and doing some basic PCI operations. So this seems fine so far.
next prev parent reply other threads:[~2026-11-19 12:18 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:27 [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 01/13] pve-qemu-rs: add pve-qemu-pci crate for guest PCI address generation Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 02/13] pve-qemu-pci: legacy: generate pci bridges Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 03/13] pve-qemu-pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 04/13] pve-qemu-rs: build perlmod library with bindings for pve-qemu-pci Dominik Csapak
2026-11-19 12:47 ` Elias Huhsovitz
2026-11-19 14:51 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 05/13] pci: use PVE::QemuServer::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 06/13] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 07/13] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 08/13] pve-qemu-pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 09/13] machine: add 'pci-layout' option Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 10/13] pve-qemu-pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 11/13] pve-qemu-pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-19 13:43 ` Elias Huhsovitz
2026-11-19 15:10 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH qemu-server v5 12/13] machine: add 'pci-topology' option Dominik Csapak
2026-11-19 12:42 ` Elias Huhsovitz
2026-11-19 15:12 ` Dominik Csapak
2026-11-18 15:27 ` [PATCH pve-manager v5 13/13] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak
2026-11-19 12:18 ` Elias Huhsovitz [this message]
2026-11-19 12:33 ` [PATCH manager/qemu-server v5 00/13] pci-handling rewrite Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ6K15N4372.2WQLDHULCW5EK@proxmox.com \
--to=e.huhsovitz@proxmox.com \
--cc=d.csapak@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox