public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH manager/proxmox-perl-rs/qemu-server/qemu-server-rs v4 00/14] pci-handling rewrite
@ 2026-11-13 14:39 Dominik Csapak
  2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 01/14] add pve-qemu-server-pci crate for guest PCI address generation Dominik Csapak
                   ` (13 more replies)
  0 siblings, 14 replies; 15+ messages in thread
From: Dominik Csapak @ 2026-11-13 14:39 UTC (permalink / raw)
  To: pve-devel

# Motivation

We want to extend the PCI layout mechanism to be easily extendable, especially
for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0]).

This is necessary for some performance gains in guests, especially with
high-end hardware where users pay extra for higher performance.

# The Problem

PCI addresses (and hardware layout in general) must be fixed for our qemu
guests, since this is part of the machine state and must be stable for
live-migration suspend/resume, etc.

Currently, these addresses are hardcoded in perl hashes, with a long history of
devs (me included) piling new addresses on top, using whatever address was
free.

This led to a very convoluted and scattered assignment list, without any
underlying structure, which makes it harder the more new devices are added.
(Using a wrong address only surfaces when running some tests or starting a
guest with it)

# My Proposed Solution

So to fix the current situation I propose a plan in multiple phases, which
are now all part of this series, so i didn't mark it as RFC anymore.
(The rust part should be mostly fine now sans naming etc.)

## Phase 1 - untangling perl code to rust

To extend/switch between multiple PCI layouts, we first must untangle the
current situation and make it easier to see what is actually used.

I did this in rust, by using the type system to represent the PCI layout, and
inferring the addresses from the layout, not the other way around.

That makes it easier to see where there is actually free space and where we can
add new things.

It also highlights how messy the current layout is. (See the legacy layout in
the `pve-qemu-server-pci` crate)

I also introduced here a generic `HardwareConfig` struct that's intended to
hold information useful to more call sites in contrast to passing a lot of
arguments everywhere, similar to the Cfg2Cmd class, but usable in the rust
codebase.

## Phase 2 - extending pci layouts

This introduces a new layout ("grouped") that can be used in an opt-in manner,
and is vastly more logically constructed, which makes it easy to extend and
reason about.

This part entails restructuring the addr calls in qemu-server's perl code, so
that there is only a single entry point ("address" for example) instead of
splitting between 'print_pci_addr' and 'print_pcie_addr'.

This is folded into the commit to using the rust bindings, since it does not
make sense to recreate the old interface in rust, just to replace it with
the next patches anyway.

This now also creates the commandline for the q35 devices instead
of using 'readconfig' (guarded behind a new machine version).

## Phase 3 - more perl to rust code port

The HardwareConfig also contains a version comparison, so we can widely reuse
it in many methods, since we need it for the 'addr' interface anyway.

These changes are also folded for the same reason as above.

## Phase 4 - Host and NUMA layouts (the next part)

With all the pieces in place, we can then easily extend the new layout by PCI
switches that are needed for assigning to NUMA nodes and replicating the host
topology for passed through devices.

This is a separate setting on top of the grouped layout instead of more
layouts:

  machine: q35,pci-layout=grouped,pci-topology=<flat|numa|host>

* flat (default): every hostpci device with pcie=1 gets its own root port on
  the root bus
* numa: adds a pxb-pcie expander bridge per guest NUMA node and puts the root
  ports of that node's devices behind it
* host: additionally replicates the host PCIe switches that more than one
  passed through device is behind

The expander bridges go into slots of the grouped layout that were unused
before, so enabling a topology does not move any of the other devices.
qemu-server reads the NUMA node and the bridges of each device from sysfs, so
nothing has to be configured per device.

Since the placement depends on the hardware of the host, live migration is
prevented for all topologies except 'flat'.

The pve-manager patch only makes sure that the two new options don't get lost
when editing the machine in the UI, they are not exposed there (yet).

# Notes

* pve-qemu-server-rs is a new git repo, but the crate in it could
  easily be integrated somewhere else. Having this as a separate repo
  could replace 'qemu-server' at one point though.
* This is an RFC, so comments about the general design/plan are desired.
* Names and crate placement are not fixed, I know that some names I chose are
  not optimal, if you do have better names/places, please suggest them.
* Opted to do this in rust for now, since I think that is the best way forward.
  If the consensus is to do the layouting changes in perl, or keep some parts in
  perl, that's fine with me too and I'll change my efforts accordingly
* The repos depend on each other in the order pve-qemu-server-rs ->
  proxmox-perl-rs -> qemu-server. The parameters of the bindings are all
  required and unknown ones are rejected, so the qemu-server patches only
  work with bindings that are built from the matching state of the crate.
  E.g. the 'hostpci-topology' parameter is only passed with the last
  qemu-server patch, so with bindings built from the whole series, the
  earlier qemu-server patches don't work on their own.
* Neither 'pci-layout' nor 'pci-topology' is bound to a (new) machine
  version, both are opt-in via the machine property and work with every
  machine version. That means that all nodes in a cluster must know the
  options before they are used, since older nodes can't parse the machine
  property with them.
* The options are only checked when starting the guest, not when setting
  them, e.g. a non-flat topology with the legacy layout or without q35 only
  fails on start. Could be added later still though.
* For the 'numa' and 'host' topologies, the guest to host NUMA node mapping
  is taken from the 'numaN' entries. If there are none, we assume that guest
  node N is on host node N. Not sure if we should require an explicit
  binding here, opinions welcome. (Similar for devices where the host does
  not report a NUMA node, these currently count as node 0.)
* The bus numbers behind the expander bridges can shift when hostpci devices
  are added or removed. Since hostpci devices can't be hotplugged and the
  topologies prevent live migration, this only happens on a fresh start.
* Hostpci devices with pcie=1 now always get a root port on function 0 of
  their slot, also for existing machine versions, see the commit message of
  'pci: legacy: generate q35 chipset devices and hostpcie root ports'.
* @Fiona: i did not implement every change:
  - the checks in `PciAddress::new` and `AddressAssignment::new` are only
    `debug_assert!`s, to keep the constructors free of `Result`s
  - no additional devices on the root bus/no multiple instances of the
    builtin devices (as written in my replies, we can add more functions per
    slot or more bridges if we ever need to, and there is 'args')
  - unknown 'scsihw' and 'ostype' values are still not an error in the
    crate, since the types from the api-types crate accept them. Perl
    validates and defaults them before though.
  - the maximal configs are already tested by the 'full-*' cfg2cmd tests
    that are on master now

changes from v3:
- pve-qemu-server-rs:
* new patches for the 'numa' and 'host' topologies:
* `HardwareConfigParams` only takes explicit values and has no fallbacks
  anymore:
* the mainboard is no longer derived from the architecture, but must be
  given explicitly
* new `machine` module with `Machine`, `MachineVersion` and `Mainboard`
  instead of the version tuple in `HardwareConfig`
* group the scsi options of `HardwareConfig` in `ScsiOptions`
* add `AddressAssignment::new` that checks that multifunction is only set
  on function 0, and check the function number in `PciAddress::new`
* rename `FUNCTIONS_PER_DEVICE` to `MAX_FUNCTIONS_PER_DEVICE_SLOT`
* drop the outdated documentation about the fallbacks of
  `HardwareConfig::new`
* move the comment about the generic 'pve.' bus prefix to the patch that
  introduces the prefix
* add Fiona's R-b to 'pci: legacy: generate pci bridges' (only adapted to
  the changed structs)
* adapt tests

- proxmox-perl-rs:
* rebased on current master
* adapt to the changed parameters of the crate
* new patch: add `hostpcie_bus_devices` and an optional placements parameter
  for `addr`

- qemu-server:
* rebased on current master
* build the nested parameter hash for the HardwareConfig, derive the
  mainboard from the machine type and pass explicit defaults for 'scsihw'
  and 'ostype'
* adapt the expected output of the cfg2cmd tests that were added on master
  in the meantime
* drop the now unused `$winversion` parameter of `print_hostpci_devices`
* drop the virtiofs dir mapping mock, master has one now
* new patch: add the 'pci-topology' machine option

- pve-manager:
* new patch: don't lose the pci options when editing the machine

changes from v2:
- pve-qemu-server-rs:
* `HardwareConfig::new` now takes a `HardwareConfigParams` struct, which can
  be deserialized directly and rejects unknown keys
* the layout is deserialized directly into `PciLayout`, so an unknown layout
  is now an error instead of silently falling back to the legacy one
* hostpcie devices are passed as a list of objects (`{ index => N }`)
  instead of plain indices, so they can be extended later without changing
  the interface
* use an internal `Arch` enum, so the config can't hold an unknown
  architecture
* grouped layout: only set 'multifunction=on' on bridges that share their
  slot with other bridges (currently only the net bridges)
* the `Display` implementation of `Device` now prints the configuration ids
  (e.g. 'qga0', 'balloon0', 'virtioN') and is exhaustive
* the xhci controller has no index anymore (we can add more ports with hubs)
* document that `version_at_least` records the required pve version, and use
  relaxed ordering for it
* check the bridge count with `assert!` instead of `debug_assert!`
* drop the unused anyhow and log dependencies
* code style and doc fixes (`'static` layout lifetime, `match` in
  `map_legacy_pci_id`, comments)

- proxmox-perl-rs:
* `new` takes a single hash with the parameters instead of a list of
  positional arguments
* `addr` returns undef for devices that are not part of the layout instead
  of dying
* use 'this' for the object parameter and add the 'Class method'/'Method'
  doc prefixes
* doc fixes

- qemu-server:
* rebased on current master
* construct the HardwareConfig with a parameter hash, and pass the hostpcie
  devices as objects
* drop the now unused `PVE::QemuServer::Machine` import from USB.pm
* grouped layout tests: adapt to the bridges without 'multifunction=on'

0: https://bugzilla.proxmox.com/show_bug.cgi?id=7283


pve-qemu-server-rs:

Dominik Csapak (6):
  add pve-qemu-server-pci crate for guest PCI address generation
  pci: legacy: generate pci bridges
  pci: legacy: generate q35 chipset devices and hostpcie root ports
  pci: layout: add grouped PCI and PCIe layouts
  pci: bus: add buses for numa expander bridges and pcie switches
  pci: topology: place hostpcie devices like on the host


proxmox-perl-rs:

Dominik Csapak (2):
  pve: add bindings for `pve-qemu-server-pci` crate
  pve: pci: add host placements for the hostpci topology

 pve-rs/Cargo.toml                          |   1 +
 pve-rs/Makefile                            |   1 +
 pve-rs/src/bindings/mod.rs                 |   3 +
 pve-rs/src/bindings/pci/hardware_config.rs | 143 +++++++++++++++++++++
 pve-rs/src/bindings/pci/mod.rs             |   2 +
 5 files changed, 150 insertions(+)
 create mode 100644 pve-rs/src/bindings/pci/hardware_config.rs
 create mode 100644 pve-rs/src/bindings/pci/mod.rs


qemu-server:

Dominik Csapak (5):
  pci: use PVE::RS::PCI bindings for addresses and bus devices
  tree-wide: use HardwareConfig for the machine version checks
  cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1
  machine: add 'pci-layout' option
  machine: add 'pci-topology' option

 src/PVE/API2/Qemu.pm                          |   9 +-
 src/PVE/QemuMigrate.pm                        |   6 +
 src/PVE/QemuServer.pm                         | 264 +++++-----
 src/PVE/QemuServer/Drive.pm                   |   7 +-
 src/PVE/QemuServer/DriveDevice.pm             |  12 +-
 src/PVE/QemuServer/Machine.pm                 |  36 ++
 src/PVE/QemuServer/PCI.pm                     | 459 ++++++------------
 src/PVE/QemuServer/RNG.pm                     |   6 +-
 src/PVE/QemuServer/USB.pm                     |  62 ++-
 src/PVE/QemuServer/Virtiofs.pm                |   7 +-
 src/test/Makefile                             |   5 +-
 src/test/QemuServerTest/CommandLineMocks.pm   |  21 +
 src/test/cfg2cmd/audio-spice.conf.cmd         |  21 +-
 .../cfg2cmd/disk-frontend-properties.conf.cmd |  21 +-
 .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd  |  21 +-
 src/test/cfg2cmd/full-q35.conf.cmd            |  45 +-
 .../machine-power-state-override.conf.cmd     |  21 +-
 ...-grouped-layout-hostpci-host-topology.conf |  19 +
 ...uped-layout-hostpci-host-topology.conf.cmd |  57 +++
 src/test/cfg2cmd/q35-grouped-layout.conf      |  17 +
 src/test/cfg2cmd/q35-grouped-layout.conf.cmd  |  47 ++
 .../cfg2cmd/q35-grouped-virtiofs-viommu.conf  |  14 +
 .../q35-grouped-virtiofs-viommu.conf.cmd      |  46 ++
 src/test/cfg2cmd/q35-ide.conf.cmd             |  21 +-
 .../q35-linux-created-qemu-6.1.conf.cmd       |  21 +-
 .../cfg2cmd/q35-linux-hostpci-11.1+pve1.conf  |  21 +
 .../q35-linux-hostpci-11.1+pve1.conf.cmd      |  61 +++
 .../q35-linux-hostpci-driver-keep.conf.cmd    |  25 +-
 .../q35-linux-hostpci-mapping.conf.cmd        |  21 +-
 .../q35-linux-hostpci-multifunction.conf.cmd  |  21 +-
 .../q35-linux-hostpci-nvidia-vgpu.conf.cmd    |  21 +-
 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf |  16 +
 .../cfg2cmd/q35-linux-hostpci-wrong.conf.cmd  |  41 ++
 ...q35-linux-hostpci-x-pci-overrides.conf.cmd |  21 +-
 src/test/cfg2cmd/q35-linux-hostpci.conf.cmd   |  25 +-
 src/test/cfg2cmd/q35-linux-ivshmem.conf.cmd   |  21 +-
 src/test/cfg2cmd/q35-scsihw2.conf.cmd         |  21 +-
 src/test/cfg2cmd/q35-simple.conf.cmd          |  21 +-
 .../cfg2cmd/q35-viommu-intel-aw-bits.conf.cmd |  21 +-
 .../q35-viommu-intel-guest-phys-bits.conf.cmd |  21 +-
 src/test/cfg2cmd/q35-viommu-intel.conf.cmd    |  21 +-
 .../q35-viommu-virtio-aw-bits.conf.cmd        |  21 +-
 src/test/cfg2cmd/q35-viommu-virtio.conf.cmd   |  21 +-
 src/test/cfg2cmd/q35-win10-hostpci.conf.cmd   |   2 +-
 .../simple-grouped-virtiofs-viommu.conf       |  16 +
 .../simple-grouped-virtiofs-viommu.conf.cmd   |  45 ++
 src/test/cfg2cmd/simple-grouped.conf          |  15 +
 src/test/cfg2cmd/simple-grouped.conf.cmd      |  41 ++
 src/test/cfg2cmd/spice-win-grouped.conf       |  14 +
 src/test/cfg2cmd/spice-win-grouped.conf.cmd   |  48 ++
 src/test/cfg2cmd/tdx-attestation.conf.cmd     |  21 +-
 src/test/cfg2cmd/virtiofs-viommu.conf         |  16 +
 src/test/cfg2cmd/virtiofs-viommu.conf.cmd     |  40 ++
 src/test/run_pci_addr_checks.pl               | 141 ------
 54 files changed, 1401 insertions(+), 704 deletions(-)
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout-hostpci-host-topology.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf.cmd
 create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf.cmd
 create mode 100644 src/test/cfg2cmd/simple-grouped.conf
 create mode 100644 src/test/cfg2cmd/simple-grouped.conf.cmd
 create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf
 create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf.cmd
 create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf.cmd
 delete mode 100755 src/test/run_pci_addr_checks.pl


pve-manager:

Dominik Csapak (1):
  ui: qemu machine: don't lose pci options on edit cycle

 www/manager6/qemu/MachineEdit.js | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)


Summary over all repositories:
  60 files changed, 1575 insertions(+), 709 deletions(-)

-- 
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2026-11-13 14:43 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 14:39 [PATCH manager/proxmox-perl-rs/qemu-server/qemu-server-rs v4 00/14] pci-handling rewrite Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 01/14] add pve-qemu-server-pci crate for guest PCI address generation Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 02/14] pci: legacy: generate pci bridges Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 03/14] pci: legacy: generate q35 chipset devices and hostpcie root ports Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 04/14] pci: layout: add grouped PCI and PCIe layouts Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 05/14] pci: bus: add buses for numa expander bridges and pcie switches Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-qemu-server-rs v4 06/14] pci: topology: place hostpcie devices like on the host Dominik Csapak
2026-11-13 14:39 ` [PATCH proxmox-perl-rs v4 07/14] pve: add bindings for `pve-qemu-server-pci` crate Dominik Csapak
2026-11-13 14:39 ` [PATCH proxmox-perl-rs v4 08/14] pve: pci: add host placements for the hostpci topology Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 09/14] pci: use PVE::RS::PCI bindings for addresses and bus devices Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 10/14] tree-wide: use HardwareConfig for the machine version checks Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 11/14] cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1 Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 12/14] machine: add 'pci-layout' option Dominik Csapak
2026-11-13 14:39 ` [PATCH qemu-server v4 13/14] machine: add 'pci-topology' option Dominik Csapak
2026-11-13 14:39 ` [PATCH pve-manager v4 14/14] ui: qemu machine: don't lose pci options on edit cycle Dominik Csapak

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal