[pve-devel] [PATCH storage/qemu-server/manager v7] implement ova/ovf import for file based storages

[Dominik Csapak <d.csapak@proxmox.com>]

This series enables importing ova/ovf from directory based storages,
inclusive upload/download via the webui (ova only).

It also improves the ovf importer by parsing the ostype, nics, bootorder
(and firmware from vmware exported files).

We now extract the images into either a given target storage or in the
import storage in the 'images' dir so accidentally left over images
are discoverable by the ui/cli.

This version includes a modified qemu-server patch of fabians hardening series:
https://lore.proxmox.com/pve-devel/20241104104221.228730-1-f.gruenbichler@proxmox.com/

I sent the qemu-server patch from fabian again but omitted some
problematic checks. I add them later with a check
against the import vtype again (last patch in qemu-server)

changes from v6:
* rebase on master (omit already applied patches)
* style/indentation fixes
* omit explicit check for .ova in upload path
* use assert_file_validity check for download_file_from_url
* only warn on whitespace in parent image (file_size_info) to preserve backward compatibility
  for possible existing cases
* fix cleanup on live-import  (added wrong variable to cleanup list)
* renamed 'import-extraction-storage' to 'import-working-storage'
* mad priv check for extraction storage simpler (combine branches)
* reworded some gettexts

changes from v5:
* removed leftover hunks in makefile
* moved ova checks to correct patch
* split up error messages for unexpected format
* remove unnecessary untaint
* reword error message
* reintroduce symlink check in ova/ovf check
* added sanity check for ovas after uploading/downloading
* added new patch for checking import vtypes
* fixed issue with files with absolute path

changes from v4:
* rebased on master/fabians series
* added the file_size_info check for untrusted images after extracting

changes from v3:
* fixed dependencies in control file
* removed unnecessary use statements
* removed unnecessary remove helper
* moved 'needs_extract' helper to qemu-server
* removed import storage param from PUT call
* check down/uploaded ova filename more strictly (same as listing)
* improved filepath checking in ovf
* forbid importing when extracted image references a base/backing file
* instead of trying to manually create a proper filename, use 'alloc' to
  create a small (1M) file with the same format and overwrite it with
  renaming. this also solves the cluster locking issue
* prefer using PVE::Storage functions instead of plugin methods in
  ova extraction code
* use $vollist for cleaning up extracted images in qemu-server and
  add manual cleanup for the success case

changes from v2:
* use better 'format' values for embedded images (e.g. ova+vmdk)
* use this format to decide if images should be extracted
* consistent use of the 'safe character' classes when listing
  and parsing
* also list vmdk/qcow2/raw images in content listing
  (this will be useful when we have a gui for the 'import-from'
  in the wizard/disk edit for vms)
* a few gui adaptions


changes from v1:
* move ovf code to GuestImport
* move extract/checking code to GuestImport
* don't return 'image' types from import volumes
* use allow 'safe' characters for filenames of ova/ovfs and inside
* check for non-regular files (e.g. symlinks) after extraction
* add new 'import-extraction-storage' for import
* rename panel in gui for directory storages
* typo fixes
* and probably more, see the individual patches for details

pve-storage:

Dominik Csapak (11):
  plugin: dir: implement import content type
  plugin: dir: handle ova files for import
  ovf: improve and simplify path checking code
  ovf: implement parsing the ostype
  ovf: implement parsing out firmware type
  ovf: implement rudimentary boot order
  ovf: implement parsing nics
  api: allow ova upload/download
  plugin: enable import for nfs/btrfs/cifs/cephfs/glusterfs
  add 'import' content type to 'check_volume_access'
  plugin: file_size_info: warn on parent images with unusual path

 src/PVE/API2/Storage/Status.pm                |  56 +++++-
 src/PVE/GuestImport.pm                        |  79 ++++++++
 src/PVE/GuestImport/OVF.pm                    | 187 ++++++++++++++++--
 src/PVE/Makefile                              |   1 +
 src/PVE/Storage.pm                            |  23 ++-
 src/PVE/Storage/BTRFSPlugin.pm                |   5 +
 src/PVE/Storage/CIFSPlugin.pm                 |   6 +-
 src/PVE/Storage/CephFSPlugin.pm               |   6 +-
 src/PVE/Storage/DirPlugin.pm                  |  52 ++++-
 src/PVE/Storage/GlusterfsPlugin.pm            |   6 +-
 src/PVE/Storage/NFSPlugin.pm                  |   6 +-
 src/PVE/Storage/Plugin.pm                     |  16 +-
 .../ovf_manifests/Win10-Liz_no_default_ns.ovf |   1 +
 src/test/parse_volname_test.pm                |  33 ++++
 src/test/path_to_volume_id_test.pm            |  21 ++
 src/test/run_ovf_tests.pl                     |  14 ++
 16 files changed, 480 insertions(+), 32 deletions(-)
 create mode 100644 src/PVE/GuestImport.pm

qemu-server:

Dominik Csapak (4):
  use OVF from Storage
  api: create: implement extracting disks when needed for import-from
  api: create: add 'import-working-storage' parameter
  api: check untrusted image files for import content type

Fabian Grünbichler (1):
  disk import: add additional safeguards for imported image files

 PVE/API2/Qemu.pm                              | 108 ++++++--
 PVE/CLI/qm.pm                                 |   4 +-
 PVE/QemuServer.pm                             |  12 +
 PVE/QemuServer/Helpers.pm                     |   5 +
 PVE/QemuServer/Makefile                       |   1 -
 PVE/QemuServer/OVF.pm                         | 242 ------------------
 debian/control                                |   2 -
 test/Makefile                                 |   5 +-
 test/ovf_manifests/Win10-Liz-disk1.vmdk       | Bin 65536 -> 0 bytes
 test/ovf_manifests/Win10-Liz.ovf              | 142 ----------
 .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 142 ----------
 test/ovf_manifests/Win_2008_R2_two-disks.ovf  | 145 -----------
 test/ovf_manifests/disk1.vmdk                 | Bin 65536 -> 0 bytes
 test/ovf_manifests/disk2.vmdk                 | Bin 65536 -> 0 bytes
 test/run_ovf_tests.pl                         |  71 -----
 15 files changed, 112 insertions(+), 767 deletions(-)
 delete mode 100644 PVE/QemuServer/OVF.pm
 delete mode 100644 test/ovf_manifests/Win10-Liz-disk1.vmdk
 delete mode 100755 test/ovf_manifests/Win10-Liz.ovf
 delete mode 100755 test/ovf_manifests/Win10-Liz_no_default_ns.ovf
 delete mode 100755 test/ovf_manifests/Win_2008_R2_two-disks.ovf
 delete mode 100644 test/ovf_manifests/disk1.vmdk
 delete mode 100644 test/ovf_manifests/disk2.vmdk
 delete mode 100755 test/run_ovf_tests.pl

pve-manager:

Dominik Csapak (9):
  ui: guest import: add ova-needs-extracting warning text
  ui: enable import content type for relevant storages
  ui: enable upload/download/remove buttons for 'import' type storages
  ui: disable 'import' button for non importable formats
  ui: import: improve rendering of volume names
  ui: guest import: add storage selector for ova extraction storage
  ui: guest import: change icon/text for non-esxi import storage
  ui: import: show size for dir-based storages
  ui: import: adapt live import help text to ova

 www/manager6/Utils.js                    |  9 +++++++-
 www/manager6/form/ContentTypeSelector.js |  2 +-
 www/manager6/storage/Browser.js          | 25 ++++++++++++++++-----
 www/manager6/storage/CephFSEdit.js       |  2 +-
 www/manager6/storage/GlusterFsEdit.js    |  2 +-
 www/manager6/window/GuestImport.js       | 28 +++++++++++++++++++++++-
 www/manager6/window/UploadToStorage.js   |  1 +
 7 files changed, 58 insertions(+), 11 deletions(-)

-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel