From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 29DEF1FF16B for ; Thu, 14 Nov 2024 10:34:21 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 21FE72C868; Thu, 14 Nov 2024 10:33:01 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Thu, 14 Nov 2024 10:32:01 +0100 Message-Id: <20241114093226.814530-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.236 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemu.pm, btrfsplugin.pm, glusterfsplugin.pm, nfsplugin.pm, dirplugin.pm, storage.pm, helpers.pm, plugin.pm, guestimport.pm, qemuserver.pm, ovf.pm, cephfsplugin.pm, proxmox.com, status.pm, qm.pm, cifsplugin.pm] URI_NOVOWEL 0.5 URI hostname has long non-vowel sequence Subject: [pve-devel] [PATCH storage/qemu-server/manager v5] implement ova/ovf import for file based storages X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" This series enables importing ova/ovf from directory based storages, inclusive upload/download via the webui (ova only). It also improves the ovf importer by parsing the ostype, nics, bootorder (and firmware from vmware exported files). I opted to move the OVF.pm to pve-storage, since there is no real other place where we could put it. I put it in a new module 'GuestImport' We now extract the images into either a given target storage or in the import storage in the 'images' dir so accidentally left over images are discoverable by the ui/cli. This version is rebased on fabians hardening series: https://lore.proxmox.com/pve-devel/20241104104221.228730-1-f.gruenbichler@proxmox.com/ So that series is a requirement for this changes from v4: * rebased on master/fabians series * added the file_size_info check for untrusted images after extracting changes from v3: * fixed dependencies in control file * removed unnecessary use statements * removed unnecessary remove helper * moved 'needs_extract' helper to qemu-server * removed import storage param from PUT call * check down/uploaded ova filename more strictly (same as listing) * improved filepath checking in ovf * forbid importing when extracted image references a base/backing file * instead of trying to manually create a proper filename, use 'alloc' to create a small (1M) file with the same format and overwrite it with renaming. this also solves the cluster locking issue * prefer using PVE::Storage functions instead of plugin methods in ova extraction code * use $vollist for cleaning up extracted images in qemu-server and add manual cleanup for the success case changes from v2: * use better 'format' values for embedded images (e.g. ova+vmdk) * use this format to decide if images should be extracted * consistent use of the 'safe character' classes when listing and parsing * also list vmdk/qcow2/raw images in content listing (this will be useful when we have a gui for the 'import-from' in the wizard/disk edit for vms) * a few gui adaptions changes from v1: * move ovf code to GuestImport * move extract/checking code to GuestImport * don't return 'image' types from import volumes * use allow 'safe' characters for filenames of ova/ovfs and inside * check for non-regular files (e.g. symlinks) after extraction * add new 'import-extraction-storage' for import * rename panel in gui for directory storages * typo fixes * and probably more, see the individual patches for details pve-storage: Dominik Csapak (12): copy OVF.pm from qemu-server plugin: dir: implement import content type plugin: dir: handle ova files for import ovf: improve and simplify path checking code ovf: implement parsing the ostype ovf: implement parsing out firmware type ovf: implement rudimentary boot order ovf: implement parsing nics api: allow ova upload/download plugin: enable import for nfs/btrfs/cifs/cephfs/glusterfs add 'import' content type to 'check_volume_access' plugin: file_size_info: don't ignore base path with whitespace debian/control | 2 + src/PVE/API2/Storage/Status.pm | 19 +- src/PVE/GuestImport.pm | 78 ++++ src/PVE/GuestImport/Makefile | 3 + src/PVE/GuestImport/OVF.pm | 383 ++++++++++++++++++ src/PVE/Makefile | 2 + src/PVE/Storage.pm | 23 +- src/PVE/Storage/BTRFSPlugin.pm | 5 + src/PVE/Storage/CIFSPlugin.pm | 6 +- src/PVE/Storage/CephFSPlugin.pm | 6 +- src/PVE/Storage/DirPlugin.pm | 52 ++- src/PVE/Storage/GlusterfsPlugin.pm | 6 +- src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NFSPlugin.pm | 6 +- src/PVE/Storage/Plugin.pm | 17 +- src/test/Makefile | 5 +- src/test/ovf_manifests/Win10-Liz-disk1.vmdk | Bin 0 -> 65536 bytes src/test/ovf_manifests/Win10-Liz.ovf | 142 +++++++ .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 143 +++++++ .../ovf_manifests/Win_2008_R2_two-disks.ovf | 145 +++++++ src/test/ovf_manifests/disk1.vmdk | Bin 0 -> 65536 bytes src/test/ovf_manifests/disk2.vmdk | Bin 0 -> 65536 bytes src/test/parse_volname_test.pm | 33 ++ src/test/path_to_volume_id_test.pm | 21 + src/test/run_ovf_tests.pl | 85 ++++ 25 files changed, 1170 insertions(+), 13 deletions(-) create mode 100644 src/PVE/GuestImport.pm create mode 100644 src/PVE/GuestImport/Makefile create mode 100644 src/PVE/GuestImport/OVF.pm create mode 100644 src/test/ovf_manifests/Win10-Liz-disk1.vmdk create mode 100755 src/test/ovf_manifests/Win10-Liz.ovf create mode 100755 src/test/ovf_manifests/Win10-Liz_no_default_ns.ovf create mode 100755 src/test/ovf_manifests/Win_2008_R2_two-disks.ovf create mode 100644 src/test/ovf_manifests/disk1.vmdk create mode 100644 src/test/ovf_manifests/disk2.vmdk create mode 100755 src/test/run_ovf_tests.pl qemu-server: Dominik Csapak (4): api: delete unused OVF.pm use OVF from Storage api: create: implement extracting disks when needed for import-from api: create: add 'import-extraction-storage' parameter PVE/API2/Qemu.pm | 86 ++++++- PVE/API2/Qemu/Makefile | 2 +- PVE/API2/Qemu/OVF.pm | 53 ---- PVE/CLI/qm.pm | 4 +- PVE/QemuServer.pm | 12 + PVE/QemuServer/Helpers.pm | 5 + PVE/QemuServer/Makefile | 1 - PVE/QemuServer/OVF.pm | 242 ------------------ debian/control | 2 - test/Makefile | 5 +- test/ovf_manifests/Win10-Liz-disk1.vmdk | Bin 65536 -> 0 bytes test/ovf_manifests/Win10-Liz.ovf | 142 ---------- .../ovf_manifests/Win10-Liz_no_default_ns.ovf | 142 ---------- test/ovf_manifests/Win_2008_R2_two-disks.ovf | 145 ----------- test/ovf_manifests/disk1.vmdk | Bin 65536 -> 0 bytes test/ovf_manifests/disk2.vmdk | Bin 65536 -> 0 bytes test/run_ovf_tests.pl | 71 ----- 17 files changed, 95 insertions(+), 817 deletions(-) delete mode 100644 PVE/API2/Qemu/OVF.pm delete mode 100644 PVE/QemuServer/OVF.pm delete mode 100644 test/ovf_manifests/Win10-Liz-disk1.vmdk delete mode 100755 test/ovf_manifests/Win10-Liz.ovf delete mode 100755 test/ovf_manifests/Win10-Liz_no_default_ns.ovf delete mode 100755 test/ovf_manifests/Win_2008_R2_two-disks.ovf delete mode 100644 test/ovf_manifests/disk1.vmdk delete mode 100644 test/ovf_manifests/disk2.vmdk delete mode 100755 test/run_ovf_tests.pl pve-manager: Dominik Csapak (9): ui: fix special 'import' icon for non-esxi storages ui: guest import: add ova-needs-extracting warning text ui: enable import content type for relevant storages ui: enable upload/download/remove buttons for 'import' type storages ui: disable 'import' button for non importable formats ui: import: improve rendering of volume names ui: guest import: add storage selector for ova extraction storage ui: guest import: change icon/text for non-esxi import storage ui: import: show size for dir-based storages www/manager6/Utils.js | 11 +++++++++-- www/manager6/form/ContentTypeSelector.js | 2 +- www/manager6/storage/Browser.js | 25 ++++++++++++++++++------ www/manager6/storage/CephFSEdit.js | 2 +- www/manager6/storage/GlusterFsEdit.js | 2 +- www/manager6/window/GuestImport.js | 24 +++++++++++++++++++++++ www/manager6/window/UploadToStorage.js | 1 + 7 files changed, 56 insertions(+), 11 deletions(-) -- 2.39.5 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel