public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH many v2] add cluster-wide hardware device mapping
@ 2022-08-25  9:24 Dominik Csapak
  2022-08-25  9:24 ` [pve-devel] [PATCH cluster v2 1/1] add nodes/hardware-map.conf Dominik Csapak
                   ` (30 more replies)
  0 siblings, 31 replies; 42+ messages in thread
From: Dominik Csapak @ 2022-08-25  9:24 UTC (permalink / raw)
  To: pve-devel

this series aims to add a cluster-wide device mapping for pci and usb devices.
so that an admin can configure a device to be availble for migration and
configuring for uses that are non-root

built-in are some additional safety checks in contrast to current
passthrough, e.g. if pci addresses shift, with the mapping
we can detect that and prevent a vm to boot with the wrong device
(in most cases, there are some edge cases when one has multiple
of the same device, e.g. the same gpu, that we cannot detect)

new in this version is the ability to specify multiple devices for
each host mapping, such that we can select the first free one on
starting the vm (this fixes #3574). That makes using vGPUs and SR-IOV
much more useful as a user does not have to hardcode the pci ids anymore

i seperated that feature in a patch for pve-common(3/3) and in
qemu-server ({12,13}/13) in the backend for easier review, but did not bother
to do it for the gui (if we really don't want it, i can just send a different
version for the ui)

also pve-common 1/3 and qemu-server 1-4/13 are general cleanups that
would even make sense without the remaining patches
(qemu-server 1/13 depends on pve-common 1/3)

changes from v1:
* dropped 'check_hw_perm' (just use 'check_full' now)
* added some cleanups
* renamed the buttons in the ui (hopefully better now)
* added multi device mapping for each host
  this includes a new 'multi pci' selector for that window, which
  automatically adds entries for the whole slots which, when selected,
  disabled the selection of the individual functions
* fixed some issues (e.g. missing entries in the 'caps' object, wrong
  usb config parsing, etc.)

changes from the rfc:
* new cluster wide gui instead of node-local one (removed that, since
  it's not necessary when we have a cluster-wide one)
* uses json instead of a section config
* api is quite different overall, i split the type into its own level
  for configuring, similar to what we do in pbs
  (e.g. /nodes/NODENAME/hardware/mapping/usb/)
* fixed quite some bugs the rfc had
* added patch for handling the gui with limited permissions better
* added a 'comment' field for mappings

dependencies:
    pve-common (1) breaks current qemu-server
    pve-common (2,3) depends on pve-cluster
    qemu-server (1-4) depends on pve-common (1)
    qemu-server (5-11) depends on qemu-server(<5), pve-access-control,pve-common (2)
    qemu-server (12,13) depends on qemu-server(<12), pve-common (3)
    manager depends on qemu-server,pve-access-control,pve-common

pve-cluster:

Dominik Csapak (1):
  add nodes/hardware-map.conf

 data/PVE/Cluster.pm | 1 +
 data/src/status.c   | 1 +
 2 files changed, 2 insertions(+)

pve-access-control:

Dominik Csapak (1):
  PVE/AccessControl: add Hardware.* privileges and /hardware/ paths

 src/PVE/AccessControl.pm  | 13 +++++++++++++
 src/PVE/RPCEnvironment.pm |  3 ++-
 2 files changed, 15 insertions(+), 1 deletion(-)

pve-common:

Dominik Csapak (3):
  SysFSTools: make mdev cleanup independent of pciid
  add PVE/HardwareMap
  HardwareMap: add support for multiple pci device paths per mapping

 src/Makefile           |   1 +
 src/PVE/HardwareMap.pm | 376 +++++++++++++++++++++++++++++++++++++++++
 src/PVE/SysFSTools.pm  |   6 +-
 3 files changed, 379 insertions(+), 4 deletions(-)
 create mode 100644 src/PVE/HardwareMap.pm

qemu-server:

Dominik Csapak (13):
  cleanup pci devices in more situations
  PCI: make mediated device path independent of pci id
  PCI: refactor print_pci_device
  PCI: reuse parsed info from print_hostpci_devices
  PVE/QemuServer: allow mapped usb devices in config
  PVE/QemuServer: allow mapped pci deviced in config
  PVE/API2/Qemu: add permission checks for mapped usb devices
  PVE/API2/Qemu: add permission checks for mapped pci devices
  PVE/QemuServer: extend 'check_local_resources' for mapped resources
  PVE/API2/Qemu: migrate preconditions: use new check_local_resources
    info
  PVE/QemuMigrate: check for mapped resources on migration
  fix #3574: enable multi pci device mapping from config
  add tests for mapped pci devices

 PVE/API2/Qemu.pm                              | 109 +++++++++++--
 PVE/QemuMigrate.pm                            |  13 +-
 PVE/QemuServer.pm                             | 137 ++++++++++++-----
 PVE/QemuServer/PCI.pm                         | 145 +++++++++++++-----
 PVE/QemuServer/USB.pm                         |  21 ++-
 test/MigrationTest/Shared.pm                  |   7 +
 test/cfg2cmd/q35-linux-hostpci-mapping.conf   |  17 ++
 .../q35-linux-hostpci-mapping.conf.cmd        |  36 +++++
 test/run_config2command_tests.pl              |  76 +++++++++
 9 files changed, 469 insertions(+), 92 deletions(-)
 create mode 100644 test/cfg2cmd/q35-linux-hostpci-mapping.conf
 create mode 100644 test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd

pve-manager:

Dominik Csapak (13):
  PVE/API2/Hardware: add Mapping.pm
  PVE/API2/Cluster: add Hardware mapping list api call
  ui: form/USBSelector: make it more flexible with nodename
  ui: form: add PCIMapSelector
  ui: form: add USBMapSelector
  ui: qemu/PCIEdit: rework panel to add a mapped configuration
  ui: qemu/USBEdit: add 'mapped' device case
  ui: form: add MultiPCISelector
  ui: add window/PCIEdit: edit window for pci mappings
  ui: add window/USBEdit: edit window for usb mappings
  ui: add dc/HardwareView: a CRUD interface for hardware mapping
  ui: window/Migrate: allow mapped devices
  ui: improve permission handling for hardware

 PVE/API2/Cluster.pm                   |   8 +
 PVE/API2/Cluster/Hardware.pm          | 117 +++++
 PVE/API2/Cluster/Makefile             |   1 +
 PVE/API2/Hardware.pm                  |   6 +
 PVE/API2/Hardware/Makefile            |   1 +
 PVE/API2/Hardware/Mapping.pm          | 708 ++++++++++++++++++++++++++
 www/css/ext6-pve.css                  |   4 +
 www/manager6/Makefile                 |   6 +
 www/manager6/data/PermPathStore.js    |   1 +
 www/manager6/dc/Config.js             |  18 +-
 www/manager6/dc/HardwareView.js       | 324 ++++++++++++
 www/manager6/form/MultiPCISelector.js | 287 +++++++++++
 www/manager6/form/PCIMapSelector.js   | 102 ++++
 www/manager6/form/PCISelector.js      |  18 +-
 www/manager6/form/USBMapSelector.js   |  73 +++
 www/manager6/form/USBSelector.js      |  33 +-
 www/manager6/qemu/HardwareView.js     |  17 +-
 www/manager6/qemu/PCIEdit.js          | 314 ++++++++----
 www/manager6/qemu/USBEdit.js          |  36 +-
 www/manager6/window/Migrate.js        |  37 +-
 www/manager6/window/PCIEdit.js        | 287 +++++++++++
 www/manager6/window/USBEdit.js        | 248 +++++++++
 22 files changed, 2515 insertions(+), 131 deletions(-)
 create mode 100644 PVE/API2/Cluster/Hardware.pm
 create mode 100644 PVE/API2/Hardware/Mapping.pm
 create mode 100644 www/manager6/dc/HardwareView.js
 create mode 100644 www/manager6/form/MultiPCISelector.js
 create mode 100644 www/manager6/form/PCIMapSelector.js
 create mode 100644 www/manager6/form/USBMapSelector.js
 create mode 100644 www/manager6/window/PCIEdit.js
 create mode 100644 www/manager6/window/USBEdit.js

-- 
2.30.2





^ permalink raw reply	[flat|nested] 42+ messages in thread

end of thread, other threads:[~2022-09-02  9:03 UTC | newest]

Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-25  9:24 [pve-devel] [PATCH many v2] add cluster-wide hardware device mapping Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH cluster v2 1/1] add nodes/hardware-map.conf Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH access-control v2 1/1] PVE/AccessControl: add Hardware.* privileges and /hardware/ paths Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH common v2 1/3] SysFSTools: make mdev cleanup independent of pciid Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH common v2 2/3] add PVE/HardwareMap Dominik Csapak
2022-08-25 13:37   ` DERUMIER, Alexandre
2022-08-25  9:24 ` [pve-devel] [PATCH common v2 3/3] HardwareMap: add support for multiple pci device paths per mapping Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 01/13] cleanup pci devices in more situations Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 02/13] PCI: make mediated device path independent of pci id Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 03/13] PCI: refactor print_pci_device Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 04/13] PCI: reuse parsed info from print_hostpci_devices Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 05/13] PVE/QemuServer: allow mapped usb devices in config Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 06/13] PVE/QemuServer: allow mapped pci deviced " Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 07/13] PVE/API2/Qemu: add permission checks for mapped usb devices Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 08/13] PVE/API2/Qemu: add permission checks for mapped pci devices Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 09/13] PVE/QemuServer: extend 'check_local_resources' for mapped resources Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 10/13] PVE/API2/Qemu: migrate preconditions: use new check_local_resources info Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 11/13] PVE/QemuMigrate: check for mapped resources on migration Dominik Csapak
2022-08-25 18:00   ` DERUMIER, Alexandre
2022-08-26  6:33     ` Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 12/13] fix #3574: enable multi pci device mapping from config Dominik Csapak
2022-08-25 14:21   ` DERUMIER, Alexandre
2022-08-25 14:53     ` DERUMIER, Alexandre
2022-08-26  6:39       ` Dominik Csapak
2022-08-27 16:09         ` DERUMIER, Alexandre
2022-09-02  9:02           ` Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH qemu-server v2 13/13] add tests for mapped pci devices Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 01/13] PVE/API2/Hardware: add Mapping.pm Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 02/13] PVE/API2/Cluster: add Hardware mapping list api call Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 03/13] ui: form/USBSelector: make it more flexible with nodename Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 04/13] ui: form: add PCIMapSelector Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 05/13] ui: form: add USBMapSelector Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 06/13] ui: qemu/PCIEdit: rework panel to add a mapped configuration Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 07/13] ui: qemu/USBEdit: add 'mapped' device case Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 08/13] ui: form: add MultiPCISelector Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 09/13] ui: add window/PCIEdit: edit window for pci mappings Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 10/13] ui: add window/USBEdit: edit window for usb mappings Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 11/13] ui: add dc/HardwareView: a CRUD interface for hardware mapping Dominik Csapak
2022-08-25 15:37   ` DERUMIER, Alexandre
2022-08-26  6:48     ` Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 12/13] ui: window/Migrate: allow mapped devices Dominik Csapak
2022-08-25  9:24 ` [pve-devel] [PATCH manager v2 13/13] ui: improve permission handling for hardware Dominik Csapak

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal