Re: [pve-devel] [PATCH many v3] add cluster-wide hardware device mapping

["DERUMIER, Alexandre" <Alexandre.DERUMIER@groupe-cyllene.com>]

Hi Dominik,

I will try to test it this week.

Le mardi 20 septembre 2022 à 14:50 +0200, Dominik Csapak a écrit :
> this series aims to add a cluster-wide device mapping for pci and usb
> devices.
> so that an admin can configure a device to be availble for migration
> and
> configuring for uses that are non-root
> 
> this version is mostly the same as v2, aside from some bugfixes,
> rebase
> and preventing from having mdev: 1 set for multifunction devices.
> i would appreciate if somebody could take a look at this series
> again ;) (below is the old cover letter + changelog)
> 
> built-in are some additional safety checks in contrast to current
> passthrough, e.g. if pci addresses shift, with the mapping
> we can detect that and prevent a vm to boot with the wrong device
> (in most cases, there are some edge cases when one has multiple
> of the same device, e.g. the same gpu, that we cannot detect)
> 
> new in this version is the ability to specify multiple devices for
> each host mapping, such that we can select the first free one on
> starting the vm (this fixes #3574). That makes using vGPUs and SR-IOV
> much more useful as a user does not have to hardcode the pci ids
> anymore
> 
> i left that feature seperated in a patch for pve-common(3/3) and in
> qemu-server ({12,13}/13) in the backend for easier review, but did
> not bother
> to do it for the gui (if we really don't want it, i can just send a
> different
> version for the ui)
> 
> also pve-common 1/3 and qemu-server 1-4/13 are general cleanups that
> would even make sense without the remaining patches
> (qemu-server 1/13 depends on pve-common 1/3)
> 
> changes from v2:
> * some bug fixes (e.g use of unitialized variable)
> * don't set mdev for multifunction devices
>   -> this should fix alexandres issue, since it's not possible
> anymore
>   to select a mediated device when having a multifunction device
>   selected
> 
> changes from v1:
> * dropped 'check_hw_perm' (just use 'check_full' now)
> * added some cleanups
> * renamed the buttons in the ui (hopefully better now)
> * added multi device mapping for each host
>   this includes a new 'multi pci' selector for that window, which
>   automatically adds entries for the whole slots which, when
> selected,
>   disabled the selection of the individual functions
> * fixed some issues (e.g. missing entries in the 'caps' object, wrong
>   usb config parsing, etc.)
> 
> changes from the rfc:
> * new cluster wide gui instead of node-local one (removed that, since
>   it's not necessary when we have a cluster-wide one)
> * uses json instead of a section config
> * api is quite different overall, i split the type into its own level
>   for configuring, similar to what we do in pbs
>   (e.g. /nodes/NODENAME/hardware/mapping/usb/)
> * fixed quite some bugs the rfc had
> * added patch for handling the gui with limited permissions better
> * added a 'comment' field for mappings
> 
> dependencies:
>     pve-common (1) breaks current qemu-server
>     pve-common (2,3) depends on pve-cluster
>     qemu-server (1-4) depends on pve-common (1)
>     qemu-server (5-11) depends on qemu-server(<5), pve-access-
> control,pve-common (2)
>     qemu-server (12,13) depends on qemu-server(<12), pve-common (3)
>     manager depends on qemu-server,pve-access-control,pve-common
> 
> pve-cluster:
> 
> Dominik Csapak (1):
>   add nodes/hardware-map.conf
> 
>  data/PVE/Cluster.pm | 1 +
>  data/src/status.c   | 1 +
>  2 files changed, 2 insertions(+)
> 
> pve-access-control:
> 
> Dominik Csapak (1):
>   PVE/AccessControl: add Hardware.* privileges and /hardware/ paths
> 
>  src/PVE/AccessControl.pm  | 13 +++++++++++++
>  src/PVE/RPCEnvironment.pm |  3 ++-
>  2 files changed, 15 insertions(+), 1 deletion(-)
> 
> pve-common:
> 
> Dominik Csapak (3):
>   SysFSTools: make mdev cleanup independent of pciid
>   add PVE/HardwareMap
>   HardwareMap: add support for multiple pci device paths per mapping
> 
>  src/Makefile           |   1 +
>  src/PVE/HardwareMap.pm | 378
> +++++++++++++++++++++++++++++++++++++++++
>  src/PVE/SysFSTools.pm  |   6 +-
>  3 files changed, 381 insertions(+), 4 deletions(-)
>  create mode 100644 src/PVE/HardwareMap.pm
> 
> qemu-server:
> 
> Dominik Csapak (13):
>   cleanup pci devices in more situations
>   PCI: make mediated device path independent of pci id
>   PCI: refactor print_pci_device
>   PCI: reuse parsed info from print_hostpci_devices
>   PVE/QemuServer: allow mapped usb devices in config
>   PVE/QemuServer: allow mapped pci deviced in config
>   PVE/API2/Qemu: add permission checks for mapped usb devices
>   PVE/API2/Qemu: add permission checks for mapped pci devices
>   PVE/QemuServer: extend 'check_local_resources' for mapped resources
>   PVE/API2/Qemu: migrate preconditions: use new check_local_resources
>     info
>   PVE/QemuMigrate: check for mapped resources on migration
>   fix #3574: enable multi pci device mapping from config
>   add tests for mapped pci devices
> 
>  PVE/API2/Qemu.pm                              | 109 +++++++++++--
>  PVE/QemuMigrate.pm                            |  13 +-
>  PVE/QemuServer.pm                             | 137 +++++++++++-----
>  PVE/QemuServer/PCI.pm                         | 149 +++++++++++++---
> --
>  PVE/QemuServer/USB.pm                         |  21 ++-
>  test/MigrationTest/Shared.pm                  |   7 +
>  test/cfg2cmd/q35-linux-hostpci-mapping.conf   |  17 ++
>  .../q35-linux-hostpci-mapping.conf.cmd        |  36 +++++
>  test/run_config2command_tests.pl              |  76 +++++++++
>  9 files changed, 473 insertions(+), 92 deletions(-)
>  create mode 100644 test/cfg2cmd/q35-linux-hostpci-mapping.conf
>  create mode 100644 test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd
> 
> pve-manager:
> 
> Dominik Csapak (13):
>   PVE/API2/Hardware: add Mapping.pm
>   PVE/API2/Cluster: add Hardware mapping list api call
>   ui: form/USBSelector: make it more flexible with nodename
>   ui: form: add PCIMapSelector
>   ui: form: add USBMapSelector
>   ui: qemu/PCIEdit: rework panel to add a mapped configuration
>   ui: qemu/USBEdit: add 'mapped' device case
>   ui: form: add MultiPCISelector
>   ui: add window/PCIEdit: edit window for pci mappings
>   ui: add window/USBEdit: edit window for usb mappings
>   ui: add dc/HardwareView: a CRUD interface for hardware mapping
>   ui: window/Migrate: allow mapped devices
>   ui: improve permission handling for hardware
> 
>  PVE/API2/Cluster.pm                   |   8 +
>  PVE/API2/Cluster/Hardware.pm          | 117 +++++
>  PVE/API2/Cluster/Makefile             |   1 +
>  PVE/API2/Hardware.pm                  |   6 +
>  PVE/API2/Hardware/Makefile            |   1 +
>  PVE/API2/Hardware/Mapping.pm          | 708
> ++++++++++++++++++++++++++
>  www/css/ext6-pve.css                  |   4 +
>  www/manager6/Makefile                 |   6 +
>  www/manager6/data/PermPathStore.js    |   1 +
>  www/manager6/dc/Config.js             |  18 +-
>  www/manager6/dc/HardwareView.js       | 324 ++++++++++++
>  www/manager6/form/MultiPCISelector.js | 289 +++++++++++
>  www/manager6/form/PCIMapSelector.js   | 102 ++++
>  www/manager6/form/PCISelector.js      |  18 +-
>  www/manager6/form/USBMapSelector.js   |  73 +++
>  www/manager6/form/USBSelector.js      |  33 +-
>  www/manager6/qemu/HardwareView.js     |  17 +-
>  www/manager6/qemu/PCIEdit.js          | 314 ++++++++----
>  www/manager6/qemu/USBEdit.js          |  36 +-
>  www/manager6/window/Migrate.js        |  37 +-
>  www/manager6/window/PCIEdit.js        | 283 ++++++++++
>  www/manager6/window/USBEdit.js        | 248 +++++++++
>  22 files changed, 2513 insertions(+), 131 deletions(-)
>  create mode 100644 PVE/API2/Cluster/Hardware.pm
>  create mode 100644 PVE/API2/Hardware/Mapping.pm
>  create mode 100644 www/manager6/dc/HardwareView.js
>  create mode 100644 www/manager6/form/MultiPCISelector.js
>  create mode 100644 www/manager6/form/PCIMapSelector.js
>  create mode 100644 www/manager6/form/USBMapSelector.js
>  create mode 100644 www/manager6/window/PCIEdit.js
>  create mode 100644 www/manager6/window/USBEdit.js
>