Re: [pve-devel] [PATCH storage/qemu-server/manager v7] implement ova/ovf import for file based storages

[Filip Schauer <f.schauer@proxmox.com>]

Hello,

I tried to quickly test the OVA import with a CentOS 7 OVA file that I
downloaded from:
https://dlconusc1.linuxvmimages.com/046389e06777452db2ccf9a32efa3760:dldatac/VirtualBox/C/7/CentOS_7.9.2009_VBM.zip

6785cb04dc5e7ab5a28ec139f631dfc8aeb8a6278c5fc66e1e7030d3f5a83b18 
/var/lib/vz/import/CentOS_7.9.2009_VBM_LinuxVMImages.COM.ova

After enabling the "import" content type on my local directory storage,
the ova file shows up in the import section of the storage.

Upon clicking "Import" in the web UI the "Import Guest" dialog opens
along with an error message stating:
XPath error : Undefined namespace prefix at 
/usr/share/perl5/PVE/GuestImport/OVF.pm line 239. (500)

Closing the error message also closes the import dialog, preventing me
from importing.

The same thing happens with this ArcoLinux OVA file:
https://master.dl.sourceforge.net/project/linuxovafiles/GUI/ArcoLinux-vm.ova?viasf=1 
to /var/lib/vz/import/ArcoLinux-vm.ova

2b353ac59598bac24d4cf0721e367935d00178a31d84e09c94425bdee3dab6a1 
/var/lib/vz/import/ArcoLinux-vm.ova

Tested-by: Filip Schauer <f.schauer@proxmox.com>

On 18/11/2024 16:29, Dominik Csapak wrote:
> This series enables importing ova/ovf from directory based storages,
> inclusive upload/download via the webui (ova only).
>
> It also improves the ovf importer by parsing the ostype, nics, bootorder
> (and firmware from vmware exported files).
>
> We now extract the images into either a given target storage or in the
> import storage in the 'images' dir so accidentally left over images
> are discoverable by the ui/cli.
>
> This version includes a modified qemu-server patch of fabians hardening series:
> https://lore.proxmox.com/pve-devel/20241104104221.228730-1-f.gruenbichler@proxmox.com/
>
> I sent the qemu-server patch from fabian again but omitted some
> problematic checks. I add them later with a check
> against the import vtype again (last patch in qemu-server)
>
> changes from v6:
> * rebase on master (omit already applied patches)
> * style/indentation fixes
> * omit explicit check for .ova in upload path
> * use assert_file_validity check for download_file_from_url
> * only warn on whitespace in parent image (file_size_info) to preserve backward compatibility
>    for possible existing cases
> * fix cleanup on live-import  (added wrong variable to cleanup list)
> * renamed 'import-extraction-storage' to 'import-working-storage'
> * mad priv check for extraction storage simpler (combine branches)
> * reworded some gettexts
>
> changes from v5:
> * removed leftover hunks in makefile
> * moved ova checks to correct patch
> * split up error messages for unexpected format
> * remove unnecessary untaint
> * reword error message
> * reintroduce symlink check in ova/ovf check
> * added sanity check for ovas after uploading/downloading
> * added new patch for checking import vtypes
> * fixed issue with files with absolute path
>
> changes from v4:
> * rebased on master/fabians series
> * added the file_size_info check for untrusted images after extracting
>
> changes from v3:
> * fixed dependencies in control file
> * removed unnecessary use statements
> * removed unnecessary remove helper
> * moved 'needs_extract' helper to qemu-server
> * removed import storage param from PUT call
> * check down/uploaded ova filename more strictly (same as listing)
> * improved filepath checking in ovf
> * forbid importing when extracted image references a base/backing file
> * instead of trying to manually create a proper filename, use 'alloc' to
>    create a small (1M) file with the same format and overwrite it with
>    renaming. this also solves the cluster locking issue
> * prefer using PVE::Storage functions instead of plugin methods in
>    ova extraction code
> * use $vollist for cleaning up extracted images in qemu-server and
>    add manual cleanup for the success case
>
> changes from v2:
> * use better 'format' values for embedded images (e.g. ova+vmdk)
> * use this format to decide if images should be extracted
> * consistent use of the 'safe character' classes when listing
>    and parsing
> * also list vmdk/qcow2/raw images in content listing
>    (this will be useful when we have a gui for the 'import-from'
>    in the wizard/disk edit for vms)
> * a few gui adaptions
>
>
> changes from v1:
> * move ovf code to GuestImport
> * move extract/checking code to GuestImport
> * don't return 'image' types from import volumes
> * use allow 'safe' characters for filenames of ova/ovfs and inside
> * check for non-regular files (e.g. symlinks) after extraction
> * add new 'import-extraction-storage' for import
> * rename panel in gui for directory storages
> * typo fixes
> * and probably more, see the individual patches for details
>
> pve-storage:
>
> Dominik Csapak (11):
>    plugin: dir: implement import content type
>    plugin: dir: handle ova files for import
>    ovf: improve and simplify path checking code
>    ovf: implement parsing the ostype
>    ovf: implement parsing out firmware type
>    ovf: implement rudimentary boot order
>    ovf: implement parsing nics
>    api: allow ova upload/download
>    plugin: enable import for nfs/btrfs/cifs/cephfs/glusterfs
>    add 'import' content type to 'check_volume_access'
>    plugin: file_size_info: warn on parent images with unusual path
>
>   src/PVE/API2/Storage/Status.pm                |  56 +++++-
>   src/PVE/GuestImport.pm                        |  79 ++++++++
>   src/PVE/GuestImport/OVF.pm                    | 187 ++++++++++++++++--
>   src/PVE/Makefile                              |   1 +
>   src/PVE/Storage.pm                            |  23 ++-
>   src/PVE/Storage/BTRFSPlugin.pm                |   5 +
>   src/PVE/Storage/CIFSPlugin.pm                 |   6 +-
>   src/PVE/Storage/CephFSPlugin.pm               |   6 +-
>   src/PVE/Storage/DirPlugin.pm                  |  52 ++++-
>   src/PVE/Storage/GlusterfsPlugin.pm            |   6 +-
>   src/PVE/Storage/NFSPlugin.pm                  |   6 +-
>   src/PVE/Storage/Plugin.pm                     |  16 +-
>   .../ovf_manifests/Win10-Liz_no_default_ns.ovf |   1 +
>   src/test/parse_volname_test.pm                |  33 ++++
>   src/test/path_to_volume_id_test.pm            |  21 ++
>   src/test/run_ovf_tests.pl                     |  14 ++
>   16 files changed, 480 insertions(+), 32 deletions(-)
>   create mode 100644 src/PVE/GuestImport.pm
>
> qemu-server:
>
> Dominik Csapak (4):
>    use OVF from Storage
>    api: create: implement extracting disks when needed for import-from
>    api: create: add 'import-working-storage' parameter
>    api: check untrusted image files for import content type
>
> Fabian Grünbichler (1):
>    disk import: add additional safeguards for imported image files
>
>   PVE/API2/Qemu.pm                              | 108 ++++++--
>   PVE/CLI/qm.pm                                 |   4 +-
>   PVE/QemuServer.pm                             |  12 +
>   PVE/QemuServer/Helpers.pm                     |   5 +
>   PVE/QemuServer/Makefile                       |   1 -
>   PVE/QemuServer/OVF.pm                         | 242 ------------------
>   debian/control                                |   2 -
>   test/Makefile                                 |   5 +-
>   test/ovf_manifests/Win10-Liz-disk1.vmdk       | Bin 65536 -> 0 bytes
>   test/ovf_manifests/Win10-Liz.ovf              | 142 ----------
>   .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 142 ----------
>   test/ovf_manifests/Win_2008_R2_two-disks.ovf  | 145 -----------
>   test/ovf_manifests/disk1.vmdk                 | Bin 65536 -> 0 bytes
>   test/ovf_manifests/disk2.vmdk                 | Bin 65536 -> 0 bytes
>   test/run_ovf_tests.pl                         |  71 -----
>   15 files changed, 112 insertions(+), 767 deletions(-)
>   delete mode 100644 PVE/QemuServer/OVF.pm
>   delete mode 100644 test/ovf_manifests/Win10-Liz-disk1.vmdk
>   delete mode 100755 test/ovf_manifests/Win10-Liz.ovf
>   delete mode 100755 test/ovf_manifests/Win10-Liz_no_default_ns.ovf
>   delete mode 100755 test/ovf_manifests/Win_2008_R2_two-disks.ovf
>   delete mode 100644 test/ovf_manifests/disk1.vmdk
>   delete mode 100644 test/ovf_manifests/disk2.vmdk
>   delete mode 100755 test/run_ovf_tests.pl
>
> pve-manager:
>
> Dominik Csapak (9):
>    ui: guest import: add ova-needs-extracting warning text
>    ui: enable import content type for relevant storages
>    ui: enable upload/download/remove buttons for 'import' type storages
>    ui: disable 'import' button for non importable formats
>    ui: import: improve rendering of volume names
>    ui: guest import: add storage selector for ova extraction storage
>    ui: guest import: change icon/text for non-esxi import storage
>    ui: import: show size for dir-based storages
>    ui: import: adapt live import help text to ova
>
>   www/manager6/Utils.js                    |  9 +++++++-
>   www/manager6/form/ContentTypeSelector.js |  2 +-
>   www/manager6/storage/Browser.js          | 25 ++++++++++++++++-----
>   www/manager6/storage/CephFSEdit.js       |  2 +-
>   www/manager6/storage/GlusterFsEdit.js    |  2 +-
>   www/manager6/window/GuestImport.js       | 28 +++++++++++++++++++++++-
>   www/manager6/window/UploadToStorage.js   |  1 +
>   7 files changed, 58 insertions(+), 11 deletions(-)
>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel