all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH storage/qemu-server/manager v6] implement ova/ovf import for file based storages
@ 2024-11-15 15:17 Dominik Csapak
  2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 01/12] copy OVF.pm from qemu-server Dominik Csapak
                   ` (30 more replies)
  0 siblings, 31 replies; 68+ messages in thread
From: Dominik Csapak @ 2024-11-15 15:17 UTC (permalink / raw)
  To: pve-devel

This series enables importing ova/ovf from directory based storages,
inclusive upload/download via the webui (ova only).

It also improves the ovf importer by parsing the ostype, nics, bootorder
(and firmware from vmware exported files).

I opted to move the OVF.pm to pve-storage, since there is no
real other place where we could put it. I put it in a new module
'GuestImport'

We now extract the images into either a given target storage or in the
import storage in the 'images' dir so accidentally left over images
are discoverable by the ui/cli.

This version is half rebased on fabians hardening series:
https://lore.proxmox.com/pve-devel/20241104104221.228730-1-f.gruenbichler@proxmox.com/

I sent the qemu-server patch from fabian again but omitted some
problematic checks. I add them later with a check
against the import vtype again (last patch in qemu-server)

changes from v5:
* removed leftover hunks in makefile
* moved ova checks to correct patch
* split up error messages for unexpected format
* remove unnecessary untaint
* reword error message
* reintroduce symlink check in ova/ovf check
* added sanity check for ovas after uploading/downloading
* added new patch for checking import vtypes
* fixed issue with files with absolute path

changes from v4:
* rebased on master/fabians series
* added the file_size_info check for untrusted images after extracting

changes from v3:
* fixed dependencies in control file
* removed unnecessary use statements
* removed unnecessary remove helper
* moved 'needs_extract' helper to qemu-server
* removed import storage param from PUT call
* check down/uploaded ova filename more strictly (same as listing)
* improved filepath checking in ovf
* forbid importing when extracted image references a base/backing file
* instead of trying to manually create a proper filename, use 'alloc' to
  create a small (1M) file with the same format and overwrite it with
  renaming. this also solves the cluster locking issue
* prefer using PVE::Storage functions instead of plugin methods in
  ova extraction code
* use $vollist for cleaning up extracted images in qemu-server and
  add manual cleanup for the success case

changes from v2:
* use better 'format' values for embedded images (e.g. ova+vmdk)
* use this format to decide if images should be extracted
* consistent use of the 'safe character' classes when listing
  and parsing
* also list vmdk/qcow2/raw images in content listing
  (this will be useful when we have a gui for the 'import-from'
  in the wizard/disk edit for vms)
* a few gui adaptions


changes from v1:
* move ovf code to GuestImport
* move extract/checking code to GuestImport
* don't return 'image' types from import volumes
* use allow 'safe' characters for filenames of ova/ovfs and inside
* check for non-regular files (e.g. symlinks) after extraction
* add new 'import-extraction-storage' for import
* rename panel in gui for directory storages
* typo fixes
* and probably more, see the individual patches for details

pve-storage:

Dominik Csapak (12):
  copy OVF.pm from qemu-server
  plugin: dir: implement import content type
  plugin: dir: handle ova files for import
  ovf: improve and simplify path checking code
  ovf: implement parsing the ostype
  ovf: implement parsing out firmware type
  ovf: implement rudimentary boot order
  ovf: implement parsing nics
  api: allow ova upload/download
  plugin: enable import for nfs/btrfs/cifs/cephfs/glusterfs
  add 'import' content type to 'check_volume_access'
  plugin: file_size_info: don't ignore base path with whitespace

 debian/control                                |   2 +
 src/PVE/API2/Storage/Status.pm                |  70 +++-
 src/PVE/GuestImport.pm                        |  79 ++++
 src/PVE/GuestImport/Makefile                  |   3 +
 src/PVE/GuestImport/OVF.pm                    | 386 ++++++++++++++++++
 src/PVE/Makefile                              |   2 +
 src/PVE/Storage.pm                            |  23 +-
 src/PVE/Storage/BTRFSPlugin.pm                |   5 +
 src/PVE/Storage/CIFSPlugin.pm                 |   6 +-
 src/PVE/Storage/CephFSPlugin.pm               |   6 +-
 src/PVE/Storage/DirPlugin.pm                  |  52 ++-
 src/PVE/Storage/GlusterfsPlugin.pm            |   6 +-
 src/PVE/Storage/NFSPlugin.pm                  |   6 +-
 src/PVE/Storage/Plugin.pm                     |  17 +-
 src/test/Makefile                             |   5 +-
 src/test/ovf_manifests/Win10-Liz-disk1.vmdk   | Bin 0 -> 65536 bytes
 src/test/ovf_manifests/Win10-Liz.ovf          | 142 +++++++
 .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 143 +++++++
 .../ovf_manifests/Win_2008_R2_two-disks.ovf   | 145 +++++++
 src/test/ovf_manifests/disk1.vmdk             | Bin 0 -> 65536 bytes
 src/test/ovf_manifests/disk2.vmdk             | Bin 0 -> 65536 bytes
 src/test/parse_volname_test.pm                |  33 ++
 src/test/path_to_volume_id_test.pm            |  21 +
 src/test/run_ovf_tests.pl                     |  85 ++++
 24 files changed, 1223 insertions(+), 14 deletions(-)
 create mode 100644 src/PVE/GuestImport.pm
 create mode 100644 src/PVE/GuestImport/Makefile
 create mode 100644 src/PVE/GuestImport/OVF.pm
 create mode 100644 src/test/ovf_manifests/Win10-Liz-disk1.vmdk
 create mode 100755 src/test/ovf_manifests/Win10-Liz.ovf
 create mode 100755 src/test/ovf_manifests/Win10-Liz_no_default_ns.ovf
 create mode 100755 src/test/ovf_manifests/Win_2008_R2_two-disks.ovf
 create mode 100644 src/test/ovf_manifests/disk1.vmdk
 create mode 100644 src/test/ovf_manifests/disk2.vmdk
 create mode 100755 src/test/run_ovf_tests.pl

qemu-server:

Dominik Csapak (5):
  api: delete unused OVF.pm
  use OVF from Storage
  api: create: implement extracting disks when needed for import-from
  api: create: add 'import-extraction-storage' parameter
  api: check untrusted image files for import content type

Fabian Grünbichler (1):
  disk import: add additional safeguards for imported image files

 PVE/API2/Qemu.pm                              | 106 ++++++--
 PVE/API2/Qemu/Makefile                        |   2 +-
 PVE/API2/Qemu/OVF.pm                          |  53 ----
 PVE/CLI/qm.pm                                 |   4 +-
 PVE/QemuServer.pm                             |  12 +
 PVE/QemuServer/Helpers.pm                     |   5 +
 PVE/QemuServer/Makefile                       |   1 -
 PVE/QemuServer/OVF.pm                         | 242 ------------------
 debian/control                                |   2 -
 test/Makefile                                 |   5 +-
 test/ovf_manifests/Win10-Liz-disk1.vmdk       | Bin 65536 -> 0 bytes
 test/ovf_manifests/Win10-Liz.ovf              | 142 ----------
 .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 142 ----------
 test/ovf_manifests/Win_2008_R2_two-disks.ovf  | 145 -----------
 test/ovf_manifests/disk1.vmdk                 | Bin 65536 -> 0 bytes
 test/ovf_manifests/disk2.vmdk                 | Bin 65536 -> 0 bytes
 test/run_ovf_tests.pl                         |  71 -----
 17 files changed, 112 insertions(+), 820 deletions(-)
 delete mode 100644 PVE/API2/Qemu/OVF.pm
 delete mode 100644 PVE/QemuServer/OVF.pm
 delete mode 100644 test/ovf_manifests/Win10-Liz-disk1.vmdk
 delete mode 100755 test/ovf_manifests/Win10-Liz.ovf
 delete mode 100755 test/ovf_manifests/Win10-Liz_no_default_ns.ovf
 delete mode 100755 test/ovf_manifests/Win_2008_R2_two-disks.ovf
 delete mode 100644 test/ovf_manifests/disk1.vmdk
 delete mode 100644 test/ovf_manifests/disk2.vmdk
 delete mode 100755 test/run_ovf_tests.pl

pve-manager:

Dominik Csapak (9):
  ui: fix special 'import' icon for non-esxi storages
  ui: guest import: add ova-needs-extracting warning text
  ui: enable import content type for relevant storages
  ui: enable upload/download/remove buttons for 'import' type storages
  ui: disable 'import' button for non importable formats
  ui: import: improve rendering of volume names
  ui: guest import: add storage selector for ova extraction storage
  ui: guest import: change icon/text for non-esxi import storage
  ui: import: show size for dir-based storages

 www/manager6/Utils.js                    | 11 +++++++++--
 www/manager6/form/ContentTypeSelector.js |  2 +-
 www/manager6/storage/Browser.js          | 25 ++++++++++++++++++------
 www/manager6/storage/CephFSEdit.js       |  2 +-
 www/manager6/storage/GlusterFsEdit.js    |  2 +-
 www/manager6/window/GuestImport.js       | 24 +++++++++++++++++++++++
 www/manager6/window/UploadToStorage.js   |  1 +
 7 files changed, 56 insertions(+), 11 deletions(-)

-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

^ permalink raw reply	[flat|nested] 68+ messages in thread

end of thread, other threads:[~2024-11-22 18:11 UTC | newest]

Thread overview: 68+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-11-15 15:17 [pve-devel] [PATCH storage/qemu-server/manager v6] implement ova/ovf import for file based storages Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 01/12] copy OVF.pm from qemu-server Dominik Csapak
2024-11-17 15:50   ` [pve-devel] applied: " Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 02/12] plugin: dir: implement import content type Dominik Csapak
2024-11-18 12:16   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 03/12] plugin: dir: handle ova files for import Dominik Csapak
2024-11-18 12:17   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 04/12] ovf: improve and simplify path checking code Dominik Csapak
2024-11-18 12:25   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 05/12] ovf: implement parsing the ostype Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 06/12] ovf: implement parsing out firmware type Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 07/12] ovf: implement rudimentary boot order Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 08/12] ovf: implement parsing nics Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 09/12] api: allow ova upload/download Dominik Csapak
2024-11-18 12:42   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 10/12] plugin: enable import for nfs/btrfs/cifs/cephfs/glusterfs Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 11/12] add 'import' content type to 'check_volume_access' Dominik Csapak
2024-11-18 12:58   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH storage v6 12/12] plugin: file_size_info: don't ignore base path with whitespace Dominik Csapak
2024-11-17 15:16   ` Thomas Lamprecht
2024-11-18  7:42     ` Dominik Csapak
2024-11-18  7:48       ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 1/6] disk import: add additional safeguards for imported image files Dominik Csapak
2024-11-18 13:08   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 2/6] api: delete unused OVF.pm Dominik Csapak
2024-11-17 15:18   ` [pve-devel] applied: " Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 3/6] use OVF from Storage Dominik Csapak
2024-11-17 17:42   ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 4/6] api: create: implement extracting disks when needed for import-from Dominik Csapak
2024-11-18 13:31   ` Fiona Ebner
2024-11-18 13:36     ` Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 5/6] api: create: add 'import-extraction-storage' parameter Dominik Csapak
2024-11-17 16:13   ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH qemu-server v6 6/6] api: check untrusted image files for import content type Dominik Csapak
2024-11-18 14:48   ` Fiona Ebner
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 1/9] ui: fix special 'import' icon for non-esxi storages Dominik Csapak
2024-11-17 16:21   ` [pve-devel] applied: " Thomas Lamprecht
2024-11-18  8:47     ` Dominik Csapak
2024-11-18  9:56       ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 2/9] ui: guest import: add ova-needs-extracting warning text Dominik Csapak
2024-11-17 16:29   ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 3/9] ui: enable import content type for relevant storages Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 4/9] ui: enable upload/download/remove buttons for 'import' type storages Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 5/9] ui: disable 'import' button for non importable formats Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 6/9] ui: import: improve rendering of volume names Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 7/9] ui: guest import: add storage selector for ova extraction storage Dominik Csapak
2024-11-17 16:31   ` Thomas Lamprecht
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 8/9] ui: guest import: change icon/text for non-esxi import storage Dominik Csapak
2024-11-15 15:17 ` [pve-devel] [PATCH manager v6 9/9] ui: import: show size for dir-based storages Dominik Csapak
2024-11-17 16:37 ` [pve-devel] [PATCH storage/qemu-server/manager v6] implement ova/ovf import for file based storages Thomas Lamprecht
2024-11-18 13:06 ` Lukas Wagner
2024-11-18 13:18   ` Dominik Csapak
2024-11-18 13:39     ` Lukas Wagner
2024-11-18 13:44       ` Dominik Csapak
2024-11-18 13:53         ` Dominik Csapak
2024-11-19  8:15           ` Lukas Wagner
2024-11-19  8:44             ` Dominik Csapak
2024-11-19  8:48             ` Thomas Lamprecht
2024-11-20 16:32               ` Gilberto Ferreira via pve-devel
2024-11-20 16:57                 ` Gilberto Ferreira via pve-devel
2024-11-21  8:24                   ` Dominik Csapak
2024-11-21 12:05                     ` Gilberto Ferreira via pve-devel
2024-11-21 12:23                       ` Gilberto Ferreira via pve-devel
2024-11-21 12:34                         ` Fabian Grünbichler
2024-11-22 18:10                           ` Gilberto Ferreira via pve-devel
2024-11-18 14:35 ` Daniel Herzig
2024-11-18 15:01   ` Daniel Herzig
2024-11-18 15:33 ` Dominik Csapak

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal