[RFC proxmox-perl-rs/qemu-server/qemu-server-rs v2 0/9] pci-handling rewrite (part 1/2/3)

[Dominik Csapak <d.csapak@proxmox.com>]

# Motivation

We want to extend the PCI layout mechanism to be easily extendable, especially
for NUMA awareness and replicating host-topology (e.g. see bug #7283 [0]).

This is necessary for some performance gains in guests, especially with
high-end hardware where users pay extra for higher performance.

# The Problem

PCI addresses (and hardware layout in general) must be fixed for our qemu
guests, since this is part of the machine state and must be stable for
live-migration suspend/resume, etc.

Currently, these addresses are hardcoded in perl hashes, with a long history of
devs (me included) piling new addresses on top, using whatever address was
free.

This led to a very convoluted and scattered assignment list, without any
underlying structure, which makes it harder the more new devices are added.
(Using a wrong address only surfaces when running some tests or starting a
guest with it)

# My Proposed Solution

So to fix the current situation I propose a plan in multiple phases:

## Phase 1 - untangling perl code to rust (this series)

To extend/switch between multiple PCI layouts, we first must untangle the
current situation and make it easier to see what is actually used.

I did this in rust, by using the type system to represent the PCI layout, and
inferring the addresses from the layout, not the other way around.

That makes it easier to see where there is actually free space and where we can
add new things.

It also highlights how messy the current layout is. (See the legacy layout in
the `pve-qemu-server-pci` crate)

I also introduced here a generic `HardwareConfig` struct that's intended to
hold information useful to more call sites in contrast to passing a lot of
arguments everywhere, similar to the Cfg2Cmd class, but usable in the rust
codebase.

## Phase 2 - extending pci layouts (this series)

This introduces a new layout ("grouped") that can be used in an opt-in manner,
and is vastly more logically constructed, which makes it easy to extend and
reason about.

This part entails restructuring the addr calls in qemu-server's perl code, so
that there is only a single entry point ("address" for example) instead of
splitting between 'print_pci_addr' and 'print_pcie_addr'.

This is folded into the commit to using the rust bindings, since it does not
make sense to recreate the old interface in rust, just to replace it with
the next patches anyway.

This now also creates the commandline for the q35 devices instead
of using 'readconfig' (guarded behind a new machine version).

## Phase 3 - more perl to rust code port (this series)

The HardwareConfig also contains a version comparison, so we can widely reuse
it in many methods, since we need it for the 'addr' interface anyway.

These changes are also folded for the same reason as above.

## Phase 4 - Host and NUMA layouts (the next part)

With all the pieces in place, we can then easily extend the new layout by PCI
switches that are needed for assigning to NUMA nodes and replicating the host
topology for passed through devices.

# Notes

* pve-qemu-server-rs is a new git repo, but the crate in it could
  easily be integrated somewhere else. Having this as a separate repo
  could replace 'qemu-server' at one point though.
* This is an RFC, so comments about the general design/plan are desired.
* Names and crate placement are not fixed, I know that some names I chose are
  not optimal, if you do have better names/places, please suggest them.
* I based this on my recent qemu-server hotplug v3 series [1], so keep that in
  mind when testing/applying
* Opted to do this in rust for now, since I think that is the best way forward.
  If the consensus is to do the layouting changes in perl, or keep some parts in
  perl, that's fine with me too and I'll change my efforts accordingly


changes from rfc v1:
* rebased on the hotplug v3[1] and cleanup[2] series
* split out the qemu-server cleanups/fixes into a separate series[2]
* this now also includes phase 2 and 3 of the plan

- pve-qemu-server-rs:
* renamed `Machine` to `HardwareConfig` and extended it (mainboard,
  layout, hostpcie devices, windows version)
* version_at_least now records the required +pveX version like the
  version guard in qemu-server, and treats a missing pve version as
  +pve0
* use a single `address` entry point instead of print_pci_addr and
  print_pcie_addr, the win7 quirk is now handled internally
* replaced get_pci_bridges with pci_bus_devices, which also returns the
  hostpcie root ports and the q35 chipset devices (from 11.1+pve1 on)
* add the root port on function 0 of used slots, otherwise the guest
  does not see the other functions
* renamed the 'v2' layout to 'grouped', removed the gaps between the
  bridges and moved the root ports
* take the device limits from pve-api-types instead of hardcoding them
* renamed some types and added a `LegacyRootPort` bus variant
* more tests

- proxmox-perl-rs:
* replaced the free functions and the `Machine` class with a single
  `PVE::RS::PCI::HardwareConfig` class

- qemu-server:
* removed the print_pci_addr/print_pcie_addr wrappers, construct the
  HardwareConfig once and pass it around instead of the arch
* root ports now come from the rust side, which fixes the invisible
  hostpcie devices without a root port on function 0
* use the HardwareConfig for the machine version checks
* generate the q35 chipset devices with 11.1+pve1 instead of using
  the readconfig file
* add the 'pci-layout' machine option to opt-in to the grouped layout

0: https://bugzilla.proxmox.com/show_bug.cgi?id=7283
1: https://lore.proxmox.com/pve-devel/20261007142145.984970-1-d.csapak@proxmox.com/
2: https://lore.proxmox.com/pve-devel/20261015075428.572882-1-d.csapak@proxmox.com/


pve-qemu-server-rs:

Dominik Csapak (4):
  add pve-qemu-server-pci crate for guest PCI address generation
  pci: legacy: generate pci bridges
  pci: legacy: generate q35 chipset devices and hostpcie root ports
  pci: layout: add grouped PCI and PCIe layouts


proxmox-perl-rs:

Dominik Csapak (1):
  pve: add bindings for `pve-qemu-server-pci` crate

 pve-rs/Cargo.toml                          |   1 +
 pve-rs/Makefile                            |   1 +
 pve-rs/src/bindings/mod.rs                 |   3 +
 pve-rs/src/bindings/pci/hardware_config.rs | 150 +++++++++++++++++++++
 pve-rs/src/bindings/pci/mod.rs             |   2 +
 5 files changed, 157 insertions(+)
 create mode 100644 pve-rs/src/bindings/pci/hardware_config.rs
 create mode 100644 pve-rs/src/bindings/pci/mod.rs


qemu-server:

Dominik Csapak (4):
  pci: use PVE::RS::PCI bindings for addresses and bus devices
  tree-wide: use HardwareConfig for the machine version checks
  cfg2cmd: q35: generate chipset devices with machine version 11.1+pve1
  machine: add 'pci-layout' option

 src/PVE/API2/Qemu.pm                          |   9 +-
 src/PVE/QemuServer.pm                         | 250 +++++++------
 src/PVE/QemuServer/Drive.pm                   |   7 +-
 src/PVE/QemuServer/DriveDevice.pm             |  12 +-
 src/PVE/QemuServer/Machine.pm                 |  16 +
 src/PVE/QemuServer/PCI.pm                     | 340 ++----------------
 src/PVE/QemuServer/RNG.pm                     |   6 +-
 src/PVE/QemuServer/USB.pm                     |  58 +--
 src/PVE/QemuServer/Virtiofs.pm                |   7 +-
 src/test/Makefile                             |   5 +-
 src/test/TestsCommon/CommandLineMocks.pm      |  27 ++
 .../cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd  |  21 +-
 src/test/cfg2cmd/q35-grouped-layout.conf      |  17 +
 src/test/cfg2cmd/q35-grouped-layout.conf.cmd  |  47 +++
 .../cfg2cmd/q35-grouped-virtiofs-viommu.conf  |  14 +
 .../q35-grouped-virtiofs-viommu.conf.cmd      |  46 +++
 src/test/cfg2cmd/q35-ide.conf.cmd             |  21 +-
 .../cfg2cmd/q35-linux-hostpci-11.1+pve1.conf  |  21 ++
 .../q35-linux-hostpci-11.1+pve1.conf.cmd      |  61 ++++
 .../q35-linux-hostpci-driver-keep.conf.cmd    |  25 +-
 .../q35-linux-hostpci-mapping.conf.cmd        |  21 +-
 .../q35-linux-hostpci-multifunction.conf.cmd  |  21 +-
 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf |  16 +
 .../cfg2cmd/q35-linux-hostpci-wrong.conf.cmd  |  41 +++
 ...q35-linux-hostpci-x-pci-overrides.conf.cmd |  21 +-
 src/test/cfg2cmd/q35-linux-hostpci.conf.cmd   |  25 +-
 src/test/cfg2cmd/q35-linux-ivshmem.conf.cmd   |  21 +-
 src/test/cfg2cmd/q35-scsihw2.conf.cmd         |  21 +-
 src/test/cfg2cmd/q35-simple.conf.cmd          |  21 +-
 .../cfg2cmd/q35-viommu-intel-aw-bits.conf.cmd |  21 +-
 .../q35-viommu-intel-guest-phys-bits.conf.cmd |  21 +-
 src/test/cfg2cmd/q35-viommu-intel.conf.cmd    |  21 +-
 .../q35-viommu-virtio-aw-bits.conf.cmd        |  21 +-
 src/test/cfg2cmd/q35-viommu-virtio.conf.cmd   |  21 +-
 src/test/cfg2cmd/q35-win10-hostpci.conf.cmd   |   2 +-
 .../simple-grouped-virtiofs-viommu.conf       |  16 +
 .../simple-grouped-virtiofs-viommu.conf.cmd   |  45 +++
 src/test/cfg2cmd/simple-grouped.conf          |  15 +
 src/test/cfg2cmd/simple-grouped.conf.cmd      |  41 +++
 src/test/cfg2cmd/spice-win-grouped.conf       |  14 +
 src/test/cfg2cmd/spice-win-grouped.conf.cmd   |  48 +++
 src/test/cfg2cmd/virtiofs-viommu.conf         |  16 +
 src/test/cfg2cmd/virtiofs-viommu.conf.cmd     |  40 +++
 src/test/run_pci_addr_checks.pl               | 141 --------
 44 files changed, 1039 insertions(+), 662 deletions(-)
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf
 create mode 100644 src/test/cfg2cmd/q35-grouped-layout.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/q35-grouped-virtiofs-viommu.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-11.1+pve1.conf.cmd
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf
 create mode 100644 src/test/cfg2cmd/q35-linux-hostpci-wrong.conf.cmd
 create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/simple-grouped-virtiofs-viommu.conf.cmd
 create mode 100644 src/test/cfg2cmd/simple-grouped.conf
 create mode 100644 src/test/cfg2cmd/simple-grouped.conf.cmd
 create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf
 create mode 100644 src/test/cfg2cmd/spice-win-grouped.conf.cmd
 create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf
 create mode 100644 src/test/cfg2cmd/virtiofs-viommu.conf.cmd
 delete mode 100755 src/test/run_pci_addr_checks.pl


Summary over all repositories:
  49 files changed, 1196 insertions(+), 662 deletions(-)

-- 
Generated by murpp 0.11.0