[PATCH pve-storage, pve-manager v1 00/54] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types

[PATCH pve-storage, pve-manager v1 00/54] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types

[Max R. Carrara]

Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v1
===========================================================================

Basically what the title says. Implement subdirectory scanning for
directory-based storage types, which includes directories (duh), NFS,
CIFS, CephFS, and BTRFS. This fixes #2884 [1] and the more
narrowly-scoped #623 [2].

Note that this series is a complete refresh of N. Ullreich's original
series [3]. Only the general idea of how to implement this was kept
around [^1], but since a lot of the code the original series touched has
changed in the last three or so years, a lot of cleaning up needed to be
done before implementing subdir scanning was even sanely possible.

In particular, all parsing related to disk files corresponding to
certain volume types is done using a bunch of regexes [7] that are then
interpolated ad hoc at various locations [4][5][6], more than I could
list here.

Since interpolating in yet another regex (or possibly more) into every
single place where we do such parsing is brittle, error-prone, and also
unmaintainable in the long term, I decided to clean up the relevant code
in bite-sized pieces first.

In other words, patches #01 - #47 are just preparation for the actual
fixes #48 - #50. These prep patches are kept as contained as possible,
which is why you see a large number of patches overall. For example,
instead of refactoring everything related to one thing in one huge
patch, the changes are split into things like "break up this
if-elsif-chain", "move this block into this new helper", "adapt code
style here", and so on. Otherwise, it would be incredibly hard to follow
what's actually happening. Note that all this work ultimately results in
the original parsing regexes [7] being phased out in patch #53.

Along the way I also improve the existing tests, add new ones for
existing code, and also introduce a new module named
`PVE::Storage::Common:Parse`, whose purpose it is to collect all
parsing-related functionality for things regarding storage. (Besides,
parsing stuff should all be kept in one place anyways, so might as well
introduce a proper module for that.)

Finally, patch #54 makes the new 'max-scan-depth' available in the UI.
Note that no other UI-related changes are added otherwise, so the ISO /
CT template / snippets lists for storages have not changed. In the
future, we might want to make those prettier in some way as well, so
that it's easier to browse those lists (maybe something like a mini file
browser or something).

[^1]: That idea being to call the `get_subdir_files()` helper in
    `Plugin.pm` recursively.

Testing
=======

If anyone could give this series a spin, I'd be most grateful!

Here are some interesting things you could check out (non-exhaustive):
- Configuring the new 'max-scan-depth' property in the UI
- Adding subdirectories in your ISO, LXC template and snippets dirs
  (and populating those dirs afterwards)
- Checking whether the depth limit is honored
  --> 0 is the default, which retains the current behavior of not
      scanning through any subdirs
- Checking whether imports (.ova files etc) still work as expected
- ISO / CT template upload / deletion
- ...

Also note that I ran the tests in the repository for every single patch
that I added; if you want to do this for yourself as a sanity check, try
the following:

    git rebase -i --autostash --autosquash origin/master -x 'cd src && make test'

References
==========

[1]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884
[2]: https://bugzilla.proxmox.com/show_bug.cgi?id=623
[3]: https://lore.proxmox.com/pve-devel/20230721122314.80427-1-n.ullreich@proxmox.com/
[4]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/Storage/Plugin.pm;h=afd31414e94b17b4fc0582d0ab17db36fba19d8e;hb=refs/heads/master#l799
[5]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/API2/Storage/Status.pm;hb=e26583a0fe15742fc22de2dddc85baf0feb4809c#l593
[6]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/Storage.pm;hb=e26583a0fe15742fc22de2dddc85baf0feb4809c#l712
[7]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/Storage.pm;hb=e26583a0fe15742fc22de2dddc85baf0feb4809c#l116

Summary of Changes
==================

pve-storage:

Max R. Carrara (53):
  test: plugin tests: run tests with at most 4 jobs
  plugin, common: remove superfluous use of =pod command paragraph
  common: add POD headings for groups of helpers
  common: use Exporter module for PVE::Storage::Common
  plugin: make get_subdir_files a proper subroutine and update style
  plugin api: replace helpers w/ standalone subs, bump API version & age
  common: prevent autovivification in plugin_get_vtype_subdir helper
  plugin: break up needless if-elsif chain into separate if-blocks
  plugin: adapt get_subdir_files helper of list_volumes API method
  plugin: update code style of list_volumes plugin API method
  plugin: use closure for obtaining raw volume data in list_volumes
  plugin: use closure for inner loop logic in list_volumes
  storage: update code style in function path_to_volume_id
  storage: break up needless if-elsif chain in path_to_volume_id
  storage: heave vtype file path parsing logic inside loop into helper
  storage: clean up code that was moved into helper in path_to_volume_id
  api: status: move content type assert for up-/downloads into helper
  api: status: use helper from common module to get content directory
  api: status: move up-/download file path parsing code into helper
  api: status: simplify file content assertion logic for up-/download
  test: guest import: add tests for PVE::GuestImport
  tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0
  common: test: set up parser testing code, add tests for 'iso' vtype
  tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions
  tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions
  tree-wide: replace usages of inline regexes for snippets with parsers
  tree-wide: partially replace usages of regexes for 'import' vtype
  tree-wide: replace remaining usages of regexes for 'import' vtype
  plugin: simplify recently refactored logic in parse_volname method
  plugin: simplify recently refactored logic in get_subdir_files helper
  storage: simplify recently refactored logic in path_to_volume_id sub
  api: status: simplify recently added parsing helper for file transfers
  plugin: use parsing helper in parse_volume_id sub
  test: list volumes: reorganize and modernize test running code
  test: list volumes: fix broken test checking for vmlist modifications
  test: list volumes: introduce new format for test cases
  test: list volumes: remove legacy code and migrate cases to new format
  test: list volumes: document behavior wrt. undeclared content types
  plugin: correct comment in get_subdir_files helper
  test: parse volname: modernize code
  test: parse volname: adapt tests regarding 'import' volume type
  test: parse volname: move VM disk test creation into separate block
  test: parse volname: move backup file test creation into sep. block
  test: parse volname: parameterize test case creation for some vtypes
  test: volume id: modernize code
  test: volume id: rename 'volname' test case parameter to 'file'
  test: filesystem path: modernize code
  fix #2884: implement nested subdir scanning and support 'iso' vtype
  fix #2884: support nested subdir scanning for 'vztmpl' volume type
  fix #2884: support nested subdir scanning for 'snippets' vtype
  test: add more tests for 'import' vtype & guard against nested subdirs
  test: add tests guarding against subdir scanning for vtypes
  storage api: mark old public regexes for removal, bump APIVER & APIAGE

 ApiChangeLog                                |   38 +
 debian/control                              |    1 +
 src/PVE/API2/Storage/Status.pm              |  198 +-
 src/PVE/GuestImport.pm                      |   28 +-
 src/PVE/Makefile                            |    1 +
 src/PVE/Storage.pm                          |  109 +-
 src/PVE/Storage/BTRFSPlugin.pm              |   16 +-
 src/PVE/Storage/CephFSPlugin.pm             |    1 +
 src/PVE/Storage/Common.pm                   |   91 +-
 src/PVE/Storage/Common/Makefile             |    5 +
 src/PVE/Storage/Common/Parse.pm             |  482 +++++
 src/PVE/Storage/Common/test/Makefile        |    6 +
 src/PVE/Storage/Common/test/parser_tests.pl | 1130 ++++++++++
 src/PVE/Storage/Common/test/run_tests.pl    |   25 +
 src/PVE/Storage/DirPlugin.pm                |    1 +
 src/PVE/Storage/ESXiPlugin.pm               |    6 -
 src/PVE/Storage/Makefile                    |    4 +
 src/PVE/Storage/Plugin.pm                   |  362 ++--
 src/test/filesystem_path_test.pm            |  109 +-
 src/test/get_subdir_test.pm                 |   12 +-
 src/test/guest_import_test.pl               |  948 ++++++++
 src/test/list_volumes_test.pm               | 2163 +++++++++++++++----
 src/test/parse_volname_test.pm              |  696 ++++--
 src/test/path_to_volume_id_test.pm          |  201 +-
 src/test/run_plugin_tests.pl                |   18 +-
 src/test/run_volume_access_tests.pl         |    5 +-
 26 files changed, 5661 insertions(+), 995 deletions(-)
 create mode 100644 src/PVE/Storage/Common/Parse.pm
 create mode 100644 src/PVE/Storage/Common/test/Makefile
 create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
 create mode 100755 src/PVE/Storage/Common/test/run_tests.pl
 create mode 100755 src/test/guest_import_test.pl


pve-manager:

Max R. Carrara (1):
  fix #2884: ui: storage: add field for 'max-scan-depth' property

 www/manager6/storage/Base.js | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

--
2.47.3

[PATCH pve-storage v1 01/54] test: plugin tests: run tests with at most 4 jobs

[Max R. Carrara]

Speed up tests by allowing up to 4 jobs to run in parallel.

Base the amount of jobs on the output of `nproc` for environments with
less than 4 processing units available.

Sort tests alphabetically as well, just to make things neater.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/run_plugin_tests.pl | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 8bce9d3b..27c45b8c 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -8,14 +8,21 @@ $ENV{TZ} = 'UTC';

 use TAP::Harness;

-my $harness = TAP::Harness->new({ verbosity => -1 });
+my $MAX_JOBS = 4;
+
+my $nproc = eval { int(`nproc`) } || $MAX_JOBS;
+
+my $jobs = $nproc > $MAX_JOBS ? $MAX_JOBS : $nproc;
+
+my $harness = TAP::Harness->new({ verbosity => -1, jobs => $jobs });
+
 my $res = $harness->runtests(
     "archive_info_test.pm",
-    "parse_volname_test.pm",
-    "list_volumes_test.pm",
-    "path_to_volume_id_test.pm",
-    "get_subdir_test.pm",
     "filesystem_path_test.pm",
+    "get_subdir_test.pm",
+    "list_volumes_test.pm",
+    "parse_volname_test.pm",
+    "path_to_volume_id_test.pm",
     "prune_backups_test.pm",
 );

--
2.47.3

[PATCH pve-storage v1 02/54] plugin, common: remove superfluous use of =pod command paragraph

[Max R. Carrara]

Because a POD block may be started using any POD command paragraph
[0], using =pod here is not necessary. Therefore, remove these
occurrences.

In other words, =head1 and =head3 (for example) already begin a POD
block, so explicitly opening one with =pod is not needed.

This has the benefit of making the docstrings take up a little less
space overall.

[0]: https://perldoc.perl.org/perlpodspec#Pod-Commands

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 18 ------------------
 src/PVE/Storage/Plugin.pm |  6 ------
 2 files changed, 24 deletions(-)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 3932aeeb..c5f69694 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -11,8 +11,6 @@ use constant {
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
 };

-=pod
-
 =head1 NAME

 PVE::Storage::Common - Shared functions and utilities for storage plugins and storage operations
@@ -74,14 +72,10 @@ PVE::JSONSchema::register_standard_option(
     },
 );

-=pod
-
 =head1 FUNCTIONS

 =cut

-=pod
-
 =head3 align_size_up

     $aligned_size = align_size_up($size, $granularity)
@@ -102,8 +96,6 @@ sub align_size_up : prototype($$) {
     return $aligned_size;
 }

-=pod
-
 =head3 deallocate

     deallocate($file_handle, $offset, $length)
@@ -147,8 +139,6 @@ my sub run_qemu_img_json {
     return $json;
 }

-=pod
-
 =head3 qemu_img_create

     qemu_img_create($fmt, $size, $path, $options)
@@ -172,8 +162,6 @@ sub qemu_img_create {
     run_command($cmd, errmsg => "unable to create image");
 }

-=pod
-
 =head3 qemu_img_create_qcow2_backed

     qemu_img_create_qcow2_backed($path, $backing_path, $backing_format, $options)
@@ -209,8 +197,6 @@ sub qemu_img_create_qcow2_backed {
     run_command($cmd, errmsg => "unable to create image");
 }

-=pod
-
 =head3 qemu_img_info

     qemu_img_info($filename, $file_format, $timeout, $follow_backing_files)
@@ -231,8 +217,6 @@ sub qemu_img_info {
     return run_qemu_img_json($cmd, $timeout);
 }

-=pod
-
 =head3 qemu_img_measure

     qemu_img_measure($size, $fmt, $timeout, $options)
@@ -255,8 +239,6 @@ sub qemu_img_measure {
     return run_qemu_img_json($cmd, $timeout);
 }

-=pod
-
 =head3 qemu_img_resize

     qemu_img_resize($path, $format, $size, $preallocation, $timeout)
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index afd31414..84ac412a 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -2311,8 +2311,6 @@ sub rename_volume {
     return "${storeid}:${base}${target_vmid}/${target_volname}";
 }

-=pod
-
 =head3 rename_snapshot

     $plugin->rename_snapshot($scfg, $storeid, $volname, $source_snap, $target_snap)
@@ -2362,8 +2360,6 @@ my sub blockdev_options_nbd_unix {
     return $blockdev;
 }

-=pod
-
 =head3 qemu_blockdev_options

     $blockdev =
@@ -2535,8 +2531,6 @@ sub new_backup_provider {
     die "implement me if enabling the feature 'backup-provider' in plugindata()->{features}\n";
 }

-=pod
-
 =head3 volume_qemu_snapshot_method

     $method = $plugin->volume_qemu_snapshot_method($storeid, $scfg, $volname);
--
2.47.3

[PATCH pve-storage v1 03/54] common: add POD headings for groups of helpers

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index c5f69694..c031a643 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -76,6 +76,10 @@ PVE::JSONSchema::register_standard_option(

 =cut

+=head2 GENERAL HELPERS
+
+=cut
+
 =head3 align_size_up

     $aligned_size = align_size_up($size, $granularity)
@@ -117,6 +121,10 @@ sub deallocate : prototype($$$) {
     }
 }

+=head2 FUNCTIONS FOR QEMU IMAGE OPERATIONS
+
+=cut
+
 my sub run_qemu_img_json {
     my ($cmd, $timeout) = @_;
     my $json = '';
--
2.47.3

[PATCH pve-storage v1 04/54] common: use Exporter module for PVE::Storage::Common

[Max R. Carrara]

Expose the existing helpers in `PVE::Storage::Common` via the
`Exporter` module's `import()` function, allowing the helpers to be
imported as-is by other modules.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index c031a643..76784e9e 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -6,6 +6,19 @@ use PVE::JSONSchema;
 use PVE::Syscall;
 use PVE::Tools qw(run_command);

+use Exporter qw(import);
+
+our @EXPORT_OK = qw(
+    align_size_up
+    deallocate
+
+    qemu_img_create
+    qemu_img_create_qcow2_backed
+    qemu_img_info
+    qemu_img_measure
+    qemu_img_resize
+);
+
 use constant {
     FALLOC_FL_KEEP_SIZE => 0x01, # see linux/falloc.h
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
--
2.47.3

[PATCH pve-storage v1 05/54] plugin: make get_subdir_files a proper subroutine and update style

[Max R. Carrara]

This also renames a lot of variables so that they are in line with
the rest of code and also a bit more readable, as follows:

- $fn        -->  $filename
- $notes_fn  -->  $notes_filename
- $sid       -->  $storeid
- $tt        -->  $vtype

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 66 +++++++++++++++++++--------------------
 1 file changed, 33 insertions(+), 33 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 84ac412a..07233db3 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1669,69 +1669,69 @@ sub list_images {
     return $res;
 }

-# list templates ($tt = <iso|vztmpl|backup|snippets|import>)
-my $get_subdir_files = sub {
-    my ($sid, $path, $tt, $vmid) = @_;
+# $vtype = <iso|vztmpl|backup|snippets|import>
+my sub get_subdir_files {
+    my ($storeid, $path, $vtype, $vmid) = @_;

     my $res = [];

-    foreach my $fn (<$path/*>) {
-        my $st = File::stat::stat($fn);
+    for my $filename (<$path/*>) {
+        my $st = File::stat::stat($filename);

         next if (!$st || S_ISDIR($st->mode));

         my $info;

-        if ($tt eq 'iso') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+        if ($vtype eq 'iso') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;

-            $info = { volid => "$sid:iso/$1", format => 'iso' };
+            $info = { volid => "$storeid:iso/$1", format => 'iso' };

-        } elsif ($tt eq 'vztmpl') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+        } elsif ($vtype eq 'vztmpl') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;

-            $info = { volid => "$sid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };
+            $info = { volid => "$storeid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };

-        } elsif ($tt eq 'backup') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
-            my $original = $fn;
+        } elsif ($vtype eq 'backup') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+            my $original = $filename;
             my $format = $2;
-            $fn = $1;
+            $filename = $1;

             # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            next if defined($vmid) && $fn !~ m/\S+-$vmid-\S+/;
+            next if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;

-            $info = { volid => "$sid:backup/$fn", format => $format };
+            $info = { volid => "$storeid:backup/$filename", format => $format };

-            my $archive_info = eval { PVE::Storage::archive_info($fn) } // {};
+            my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};

             $info->{ctime} = $archive_info->{ctime} if defined($archive_info->{ctime});
             $info->{subtype} = $archive_info->{type} // 'unknown';

-            if (defined($vmid) || $fn =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
+            if (defined($vmid) || $filename =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
                 $info->{vmid} = $vmid // $1;
             }

-            my $notes_fn = $original . NOTES_EXT;
-            if (-f $notes_fn) {
-                my $notes = PVE::Tools::file_read_firstline($notes_fn);
+            my $notes_filename = $original . NOTES_EXT;
+            if (-f $notes_filename) {
+                my $notes = PVE::Tools::file_read_firstline($notes_filename);
                 $info->{notes} = eval { decode('UTF-8', $notes, 1) } // $notes
                     if defined($notes);
             }

             $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
-        } elsif ($tt eq 'snippets') {
+        } elsif ($vtype eq 'snippets') {

             $info = {
-                volid => "$sid:snippets/" . basename($fn),
+                volid => "$storeid:snippets/" . basename($filename),
                 format => 'snippet',
             };
-        } elsif ($tt eq 'import') {
+        } elsif ($vtype eq 'import') {
             next
-                if $fn !~
+                if $filename !~
                 m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;

-            $info = { volid => "$sid:import/$1", format => "$2" };
+            $info = { volid => "$storeid:import/$1", format => "$2" };
         }

         $info->{size} = $st->size;
@@ -1741,7 +1741,7 @@ my $get_subdir_files = sub {
     }

     return $res;
-};
+}

 # If attributes are set on a volume, they should be included in the result.
 # See get_volume_attribute for a list of possible attributes.
@@ -1759,15 +1759,15 @@ sub list_volumes {
             my $path = $class->get_subdir($scfg, $type);

             if ($type eq 'iso' && !defined($vmid)) {
-                $data = $get_subdir_files->($storeid, $path, 'iso');
+                $data = get_subdir_files($storeid, $path, 'iso');
             } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = $get_subdir_files->($storeid, $path, 'vztmpl');
+                $data = get_subdir_files($storeid, $path, 'vztmpl');
             } elsif ($type eq 'backup') {
-                $data = $get_subdir_files->($storeid, $path, 'backup', $vmid);
+                $data = get_subdir_files($storeid, $path, 'backup', $vmid);
             } elsif ($type eq 'snippets') {
-                $data = $get_subdir_files->($storeid, $path, 'snippets');
+                $data = get_subdir_files($storeid, $path, 'snippets');
             } elsif ($type eq 'import') {
-                $data = $get_subdir_files->($storeid, $path, 'import');
+                $data = get_subdir_files($storeid, $path, 'import');
             }
         }

--
2.47.3

[PATCH pve-storage v1 06/54] plugin api: replace helpers w/ standalone subs, bump API version & age

[Max R. Carrara]

Replace the `get_vtype_subdirs()` and `get_subdir()` helpers in
PVE::Storage::Plugin with standalone helper subroutines
`plugin_get_default_vtype_subdirs()` and `plugin_get_vtype_subdir()`
in PVE::Storage::Common.

This is mostly to prevent third-party plugins from relying on these
helpers as if they were part of the storage plugin API; the signature
of `get_subdir()` in particular looks as if it belongs to an API
method, even though it is only used as a helper in PVE::Storage,
PVE::Storage::Plugin, and PVE::Storage::BTRFSPlugin.

Therefore, make it explicit that these subroutines are really just
helpers by replacing them with slightly altered versions in
PVE::Storage::Common. Add docstrings for these replacements as well.

Additionally, note that PVE::Storage::ESXiPlugin is the only plugin
that explicitly overrides `get_subdir()`, treating it as if it were an
API method, but because that plugin is not filesystem-based (in the
sense that it does not declare the `path` property in its options),
calling `get_subdir()` would throw an exception anyhow. Also, it
should be mentioned that all methods that could call into
`get_subdir()` have been overridden inside ESXiPlugin anyway.

Therefore, the override of `get_subdir()` in PVE::Storage::ESXiPlugin
is superfluous, so remove it.

Additionally, replace all occurrences of `get_vtype_subdirs()` and
`get_subdir()` across the repository with their newly introduced
counterparts `plugin_get_default_vtype_subdirs()` and
`plugin_get_vtype_subdir()`. Adapt any comments and test names as well
in accordance with these changes.

Finally, increment APIAGE and APIVER in PVE::Storage and add a
corresponding entry to the API changelog.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                        | 22 ++++++++++++
 src/PVE/Storage.pm                  | 31 +++++++++--------
 src/PVE/Storage/BTRFSPlugin.pm      | 15 +++++----
 src/PVE/Storage/Common.pm           | 52 +++++++++++++++++++++++++++++
 src/PVE/Storage/ESXiPlugin.pm       |  6 ----
 src/PVE/Storage/Plugin.pm           | 47 ++++++++++----------------
 src/test/get_subdir_test.pm         | 12 ++++---
 src/test/parse_volname_test.pm      |  6 +++-
 src/test/path_to_volume_id_test.pm  |  6 +++-
 src/test/run_volume_access_tests.pl |  5 ++-
 10 files changed, 140 insertions(+), 62 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index de411927..d1d3e1c3 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -6,6 +6,28 @@ without breaking anything unaware of it.)

 Future changes should be documented in here.

+## Version 14:
+
+* Replace plugin helper `get_vtype_subdirs()` with standalone helper subroutine
+  `PVE::Storage::Common::plugin_get_default_vtype_subdirs()`
+
+  The `get_vtype_subdirs()` subroutine in `PVE::Storage::Plugin` was an
+  internal helper with a public signature. In order to make it clear that this
+  is a helper and not meant to be part of the storage plugin API, it is
+  replaced with the more explicit
+  `PVE::Storage::Common::plugin_get_default_vtype_subdirs()` subroutine.
+
+* Replace plugin method `get_subdir()` with standalone helper subroutine
+  `PVE::Storage::Common::plugin_get_vtype_subdir()`
+
+  The `get_subdir()` method in `PVE::Storage::Plugin` was an internal helper
+  method with a public signature. Moreover, its signature could mistakenly
+  suggest that this method was part of the storage plugin API, which it was not
+  originally meant to be.
+
+  In order to make it clear that this is just a helper, it is replaced with the
+  more explicit `PVE::Storage::Common::plugin_get_vtype_subdir()` subroutine.
+
 ## Version 13:

 * Add new parameter $hints to the `activate_volume()` and `map_volume()` plugin methods
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 6e87bac3..ef2e7887 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -22,6 +22,9 @@ use PVE::JSONSchema;
 use PVE::INotify;
 use PVE::RPCEnvironment;
 use PVE::SSHInfo;
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
 use PVE::RESTEnvironment qw(log_warn);

 use PVE::Storage::Plugin;
@@ -41,11 +44,11 @@ use PVE::Storage::BTRFSPlugin;
 use PVE::Storage::ESXiPlugin;

 # Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 13;
+use constant APIVER => 14;
 # Age is the number of versions we're backward compatible with.
 # This is like having 'current=APIVER' and age='APIAGE' in libtool,
 # see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 4;
+use constant APIAGE => 5;

 our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];

@@ -530,7 +533,7 @@ sub get_image_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    my $path = $plugin->get_subdir($scfg, 'images');
+    my $path = plugin_get_vtype_subdir($scfg, 'images');

     return $vmid ? "$path/$vmid" : $path;
 }
@@ -541,7 +544,7 @@ sub get_private_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    my $path = $plugin->get_subdir($scfg, 'rootdir');
+    my $path = plugin_get_vtype_subdir($scfg, 'rootdir');

     return $vmid ? "$path/$vmid" : $path;
 }
@@ -552,7 +555,7 @@ sub get_iso_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    return $plugin->get_subdir($scfg, 'iso');
+    return plugin_get_vtype_subdir($scfg, 'iso');
 }

 sub get_import_dir {
@@ -561,7 +564,7 @@ sub get_import_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    return $plugin->get_subdir($scfg, 'import');
+    return plugin_get_vtype_subdir($scfg, 'import');
 }

 sub get_vztmpl_dir {
@@ -570,7 +573,7 @@ sub get_vztmpl_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    return $plugin->get_subdir($scfg, 'vztmpl');
+    return plugin_get_vtype_subdir($scfg, 'vztmpl');
 }

 sub get_backup_dir {
@@ -579,7 +582,7 @@ sub get_backup_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-    return $plugin->get_subdir($scfg, 'backup');
+    return plugin_get_vtype_subdir($scfg, 'backup');
 }

 # library implementation
@@ -713,12 +716,12 @@ sub path_to_volume_id {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
-        my $imagedir = $plugin->get_subdir($scfg, 'images');
-        my $isodir = $plugin->get_subdir($scfg, 'iso');
-        my $tmpldir = $plugin->get_subdir($scfg, 'vztmpl');
-        my $backupdir = $plugin->get_subdir($scfg, 'backup');
-        my $snippetsdir = $plugin->get_subdir($scfg, 'snippets');
-        my $importdir = $plugin->get_subdir($scfg, 'import');
+        my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
+        my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
+        my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
+        my $backupdir = plugin_get_vtype_subdir($scfg, 'backup');
+        my $snippetsdir = plugin_get_vtype_subdir($scfg, 'snippets');
+        my $importdir = plugin_get_vtype_subdir($scfg, 'import');

         if ($path =~ m!^\Q$imagedir\E/(\d+)/([^/\s]+)$!) {
             my $vmid = $1;
diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index e68d2bf9..67b442f6 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -11,6 +11,9 @@ use File::Path qw(mkpath);
 use IO::Dir;
 use POSIX qw(EEXIST);

+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
 use PVE::Tools qw(run_command dir_glob_foreach);

 use PVE::Storage::DirPlugin;
@@ -188,7 +191,7 @@ sub filesystem_path {

     my ($vtype, $name, $vmid, undef, undef, $isBase, $format) = $class->parse_volname($volname);

-    my $path = $class->get_subdir($scfg, $vtype);
+    my $path = plugin_get_vtype_subdir($scfg, $vtype);

     $path .= "/$vmid" if $vtype eq 'images';

@@ -294,7 +297,7 @@ sub clone_image {
         return PVE::Storage::DirPlugin::clone_image(@_);
     }

-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";
     mkpath $imagedir;

@@ -327,7 +330,7 @@ sub alloc_image {

     # From Plugin.pm:

-    my $imagedir = $class->get_subdir($scfg, 'images') . "/$vmid";
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images') . "/$vmid";

     mkpath $imagedir;

@@ -643,7 +646,7 @@ sub volume_has_feature {

 sub list_images {
     my ($class, $storeid, $scfg, $vmid, $vollist, $cache) = @_;
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');

     my $res = [];

@@ -844,7 +847,7 @@ sub volume_import {
         $volname = $class->find_free_diskname($storeid, $scfg, $vmid, $volume_format, 1);
     }

-    my $imagedir = $class->get_subdir($scfg, $vtype);
+    my $imagedir = plugin_get_vtype_subdir($scfg, $vtype);
     $imagedir .= "/$vmid" if $vtype eq 'images';

     my $tmppath = "$imagedir/recv.$vmid.tmp";
@@ -975,7 +978,7 @@ sub rename_volume {
         if !$target_volname;
     $target_volname = "$target_vmid/$target_volname";

-    my $basedir = $class->get_subdir($scfg, 'images');
+    my $basedir = plugin_get_vtype_subdir($scfg, 'images');

     mkpath "${basedir}/${target_vmid}";
     my $source_dir = raw_name_to_dir($source_volname);
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 76784e9e..52bd627d 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -17,6 +17,9 @@ our @EXPORT_OK = qw(
     qemu_img_info
     qemu_img_measure
     qemu_img_resize
+
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
 );

 use constant {
@@ -24,6 +27,16 @@ use constant {
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
 };

+my $DEFAULT_VTYPE_SUBDIRS = {
+    images => 'images',
+    rootdir => 'private',
+    iso => 'template/iso',
+    vztmpl => 'template/cache',
+    backup => 'dump',
+    snippets => 'snippets',
+    import => 'import',
+};
+
 =head1 NAME

 PVE::Storage::Common - Shared functions and utilities for storage plugins and storage operations
@@ -283,4 +296,43 @@ sub qemu_img_resize {
     run_command($cmd, timeout => $timeout);
 }

+=head2 FUNCTIONS FOR STORAGE PLUGINS AND CONFIGURATIONS
+
+=cut
+
+=head3 plugin_get_default_vtype_subdirs
+
+    my $vtype_subdirs = plugin_get_default_vtype_subdirs()
+
+Returns a hashref containing the default sub-directories for every volume type.
+
+=cut
+
+sub plugin_get_default_vtype_subdirs : prototype() () {
+    return { $DEFAULT_VTYPE_SUBDIRS->%* }; # shallow copy
+}
+
+=head3 plugin_get_vtype_subdir
+
+    my $subdir = plugin_get_vtype_subdir($scfg, $vtype)
+
+Returns the sub-directory for the volume type C<$vtype> of a given storage
+configuration C<$scfg>.
+
+Raises an exception if the storage config has no C<path> attribute or
+if the given C<$vtype> does not exist.
+
+=cut
+
+sub plugin_get_vtype_subdir : prototype($$) ($scfg, $vtype) {
+    my $path = $scfg->{path};
+
+    die "storage definition has no path\n" if !$path;
+    die "unknown vtype '$vtype'\n" if !exists($DEFAULT_VTYPE_SUBDIRS->{$vtype});
+
+    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};
+
+    return "$path/$subdir";
+}
+
 1;
diff --git a/src/PVE/Storage/ESXiPlugin.pm b/src/PVE/Storage/ESXiPlugin.pm
index f89e427f..da5abb04 100644
--- a/src/PVE/Storage/ESXiPlugin.pm
+++ b/src/PVE/Storage/ESXiPlugin.pm
@@ -605,12 +605,6 @@ sub volume_has_feature {
     return undef;
 }

-sub get_subdir {
-    my ($class, $scfg, $vtype) = @_;
-
-    die "no subdirectories available for storage $class\n";
-}
-
 package PVE::Storage::ESXiPlugin::Manifest;

 use strict;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 07233db3..55268b29 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -15,7 +15,10 @@ use PVE::Tools qw(run_command);
 use PVE::JSONSchema qw(get_standard_option register_standard_option);
 use PVE::Cluster qw(cfs_register_file);

-use PVE::Storage::Common;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
+);

 use JSON;

@@ -836,31 +839,16 @@ sub parse_volname {
     die "unable to parse directory volume name '$volname'\n";
 }

-my $vtype_subdirs = {
-    images => 'images',
-    rootdir => 'private',
-    iso => 'template/iso',
-    vztmpl => 'template/cache',
-    backup => 'dump',
-    snippets => 'snippets',
-    import => 'import',
-};
-
+# FIXME: remove on the next APIAGE reset.
 sub get_vtype_subdirs {
-    return $vtype_subdirs;
+    return plugin_get_default_vtype_subdirs();
 }

+# FIXME: remove on the next APIAGE reset.
 sub get_subdir {
     my ($class, $scfg, $vtype) = @_;

-    my $path = $scfg->{path};
-
-    die "storage definition has no path\n" if !$path;
-    die "unknown vtype '$vtype'\n" if !exists($vtype_subdirs->{$vtype});
-
-    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $vtype_subdirs->{$vtype};
-
-    return "$path/$subdir";
+    return plugin_get_vtype_subdir($scfg, $vtype);
 }

 my sub get_snap_name {
@@ -885,7 +873,7 @@ sub filesystem_path {
     die "can't snapshot this image format\n"
         if defined($snapname) && $format !~ m/^(qcow2|qed)$/;

-    my $dir = $class->get_subdir($scfg, $vtype);
+    my $dir = plugin_get_vtype_subdir($scfg, $vtype);

     $dir .= "/$vmid" if $vtype eq 'images';

@@ -1016,7 +1004,7 @@ sub clone_image {

     die "clone_image only works on base images\n" if !$isBase;

-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";

     mkpath $imagedir;
@@ -1049,7 +1037,7 @@ sub clone_image {
 sub alloc_image {
     my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) = @_;

-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";

     mkpath $imagedir;
@@ -1608,7 +1596,7 @@ sub volume_has_feature {
 sub list_images {
     my ($class, $storeid, $scfg, $vmid, $vollist, $cache) = @_;

-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');

     my $format_info = $class->get_formats($scfg, $storeid);
     my $fmts = join('|', sort keys $format_info->{valid}->%*);
@@ -1756,7 +1744,7 @@ sub list_volumes {
         if ($type eq 'images' || $type eq 'rootdir') {
             $data = $class->list_images($storeid, $scfg, $vmid);
         } elsif ($scfg->{path}) {
-            my $path = $class->get_subdir($scfg, $type);
+            my $path = plugin_get_vtype_subdir($scfg, $type);

             if ($type eq 'iso' && !defined($vmid)) {
                 $data = get_subdir_files($storeid, $path, 'iso');
@@ -1909,13 +1897,14 @@ sub activate_storage {
             # FIXME The mkdir option is deprecated. Remove with PVE 9?
             && (!defined($scfg->{mkdir}) || $scfg->{mkdir})
         ) {
-            for my $vtype (sort keys %$vtype_subdirs) {
+            my $vtype_subdirs = plugin_get_default_vtype_subdirs();
+            for my $vtype (sort keys $vtype_subdirs->%*) {
                 # OpenVZMigrate uses backup (dump) dir
                 if (
                     defined($scfg->{content}->{$vtype})
                     || ($vtype eq 'backup' && defined($scfg->{content}->{'rootdir'}))
                 ) {
-                    my $subdir = $class->get_subdir($scfg, $vtype);
+                    my $subdir = plugin_get_vtype_subdir($scfg, $vtype);
                     mkpath $subdir if $subdir ne $path;
                 }
             }
@@ -1924,7 +1913,7 @@ sub activate_storage {
         # check that content dirs are pairwise inequal
         my $resolved_subdirs = {};
         for my $vtype (sort keys $scfg->{content}->%*) {
-            my $subdir = $class->get_subdir($scfg, $vtype);
+            my $subdir = plugin_get_vtype_subdir($scfg, $vtype);
             my $abs_subdir = abs_path($subdir);
             next if !defined($abs_subdir);

@@ -2294,7 +2283,7 @@ sub rename_volume {
     $target_volname = $class->find_free_diskname($storeid, $scfg, $target_vmid, $format, 1)
         if !$target_volname;

-    my $basedir = $class->get_subdir($scfg, 'images');
+    my $basedir = plugin_get_vtype_subdir($scfg, 'images');

     mkpath "${basedir}/${target_vmid}";

diff --git a/src/test/get_subdir_test.pm b/src/test/get_subdir_test.pm
index 5fb54458..843c8c39 100644
--- a/src/test/get_subdir_test.pm
+++ b/src/test/get_subdir_test.pm
@@ -5,16 +5,20 @@ use warnings;

 use lib qw(..);

+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
+);
 use PVE::Storage::Plugin;
 use Test::More;

 my $scfg_with_path = { path => '/some/path' };
-my $vtype_subdirs = PVE::Storage::Plugin::get_vtype_subdirs();
+my $vtype_subdirs = plugin_get_default_vtype_subdirs();

 # each test is comprised of the following array keys:
 # [0] => storage config; positive with path key
 # [1] => storage type;  see $vtype_subdirs
-# [2] => expected return from get_subdir
+# [2] => expected return from plugin_get_vtype_subdir
 my $tests = [
     # failed matches
     [$scfg_with_path, 'none', "unknown vtype 'none'\n"],
@@ -45,10 +49,10 @@ foreach my $tt (@$tests) {
     my ($scfg, $type, $expected) = @$tt;

     my $got;
-    eval { $got = PVE::Storage::Plugin->get_subdir($scfg, $type) };
+    eval { $got = plugin_get_vtype_subdir($scfg, $type) };
     $got = $@ if $@;

-    is($got, $expected, "get_subdir for $type") || diag(explain($got));
+    is($got, $expected, "plugin_get_vtype_subdir for $type") || diag(explain($got));
 }

 done_testing();
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 5c9478ac..2ff4c6d0 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -6,6 +6,9 @@ use warnings;
 use lib qw(..);

 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);
 use Test::More;

 my $vmid = 1234;
@@ -302,7 +305,8 @@ foreach my $virt (keys %$non_bkp_suffix) {
 plan tests => scalar @$tests + 1;

 my $seen_vtype;
-my $vtype_subdirs = { map { $_ => 1 } keys %{ PVE::Storage::Plugin::get_vtype_subdirs() } };
+my $vtype_subdirs =
+    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };

 foreach my $t (@$tests) {
     my $description = $t->{description};
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index dfa51e64..e7e36037 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -6,6 +6,9 @@ use warnings;
 use lib qw(..);

 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);

 use Test::More;

@@ -237,7 +240,8 @@ my @tests = (
 plan tests => scalar @tests + 1;

 my $seen_vtype;
-my $vtype_subdirs = { map { $_ => 1 } keys %{ PVE::Storage::Plugin::get_vtype_subdirs() } };
+my $vtype_subdirs =
+    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };

 foreach my $tt (@tests) {
     my $file = $tt->{volname};
diff --git a/src/test/run_volume_access_tests.pl b/src/test/run_volume_access_tests.pl
index 34487083..31bacaa9 100755
--- a/src/test/run_volume_access_tests.pl
+++ b/src/test/run_volume_access_tests.pl
@@ -10,6 +10,9 @@ use lib ('.', '..');

 use PVE::RPCEnvironment;
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);
 use PVE::Storage::Plugin;

 my $storage_cfg = <<'EOF';
@@ -64,7 +67,7 @@ $pve_cluster_module->mock(
 my $rpcenv = PVE::RPCEnvironment->init('pub');
 $rpcenv->init_request();

-my @types = sort keys PVE::Storage::Plugin::get_vtype_subdirs()->%*;
+my @types = sort keys plugin_get_default_vtype_subdirs()->%*;
 my $all_types = { map { $_ => 1 } @types };

 my @tests = (
--
2.47.3

[PATCH pve-storage v1 07/54] common: prevent autovivification in plugin_get_vtype_subdir helper

[Max R. Carrara]

When accessing a non-existing (or undefined) value in a hash(ref) and
using this value again for a hash lookup, i.e.
`$hashref->{foo}->{bar}` where `$hashref->{foo}` does not exist or is
`undef`, Perl will (un)conveniently instantiate a new hashref where
one was absent before. This is also called autovivification [0].

This means that if the config hash of directory-based storage plugin
that does *not* declare the 'content-dirs' property in its `options()`
is passed to `plugin_get_vtype_subdir()`, the 'content-dirs' key will
be instantiated as an empty hashref during the
`$scfg->{'content-dirs'}->{$vtype}` lookup.

Note that "directory-based storage plugin" means any storage plugin
that defines the 'path' property in its `options()` method.

While this is a minor issue, the `$scfg` hashref should IMO never be
modified unless necessary, or if the method actually is supposed to
modify it.

Therefore, prevent this autovivification [0] in
`plugin_get_vtype_subdir()`.

Additionally, use string concatenation instead of interpolation for
the returned subdirectory path.

[0]: https://perldoc.perl.org/perlref#Autovivification

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 52bd627d..abccf52b 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -330,9 +330,11 @@ sub plugin_get_vtype_subdir : prototype($$) ($scfg, $vtype) {
     die "storage definition has no path\n" if !$path;
     die "unknown vtype '$vtype'\n" if !exists($DEFAULT_VTYPE_SUBDIRS->{$vtype});

-    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};
+    # Intermediate step to prevent autovivification
+    my $content_dirs = $scfg->{'content-dirs'} // {};
+    my $subdir = $content_dirs->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};

-    return "$path/$subdir";
+    return $path . '/' . $subdir;
 }

 1;
--
2.47.3

[PATCH pve-storage v1 08/54] plugin: break up needless if-elsif chain into separate if-blocks

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 55268b29..2d76f452 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -808,29 +808,43 @@ sub parse_volname {
         my ($vmid, $name) = ($3, $4);
         my (undef, $format, $isBase) = parse_name_dir($name);
         return ('images', $name, $vmid, $basename, $basedvmid, $isBase, $format);
-    } elsif ($volname =~ m!^(\d+)/(\S+)$!) {
+    }
+
+    if ($volname =~ m!^(\d+)/(\S+)$!) {
         my ($vmid, $name) = ($1, $2);
         my (undef, $format, $isBase) = parse_name_dir($name);
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
-    } elsif ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
+    }
+
+    if ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
         return ('iso', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
+    }
+
+    if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
         return ('vztmpl', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
+    }
+
+    if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
         my $fn = $1;
         if ($fn =~ m/^vzdump-(openvz|lxc|qemu)-(\d+)-.+/) {
             return ('backup', $fn, $2, undef, undef, undef, 'raw');
         }
         return ('backup', $fn, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^snippets/([^/]+)$!) {
+    }
+
+    if ($volname =~ m!^snippets/([^/]+)$!) {
         return ('snippets', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~
+    }
+
+    if ($volname =~
         m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.ova\/${PVE::Storage::OVA_CONTENT_RE_1})$!
     ) {
         my $packed_image = $1;
         my $format = $2;
         return ('import', $packed_image, undef, undef, undef, undef, "ova+$format");
-    } elsif ($volname =~
+    }
+
+    if ($volname =~
         m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!
     ) {
         return ('import', $1, undef, undef, undef, undef, $2);
--
2.47.3

[PATCH pve-storage v1 09/54] plugin: adapt get_subdir_files helper of list_volumes API method

[Max R. Carrara]

Pull the body of the loop inside `get_subdir_files()` out into a
closure and break up the if-elsif chain into more readable if-clauses.

Strip the vtype subdirectory from the beginning of the file paths
being worked on inside that closure. This does not have an effect on
how any of the subsequent regexes match on the path / file name,
because they match at the end of a given path.

Additionally, take take the storage config hash `$scfg` instead of the
volume type subdir `$path` as parameter. Passing `$scfg` along
makes it possible to obtain the subdir for the given `$vtype` inside
the helper directly instead of passing both the vtype and its
associated subdirectory along.

All of these things make future changes easier to manage.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 95 ++++++++++++++++++++++++++-------------
 1 file changed, 64 insertions(+), 31 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 2d76f452..c4eb6f18 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1673,37 +1673,54 @@ sub list_images {

 # $vtype = <iso|vztmpl|backup|snippets|import>
 my sub get_subdir_files {
-    my ($storeid, $path, $vtype, $vmid) = @_;
+    my ($storeid, $scfg, $vtype, $vmid) = @_;
+
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);

     my $res = [];

-    for my $filename (<$path/*>) {
-        my $st = File::stat::stat($filename);
+    my $get_subdir_file_info = sub {
+        my ($path, $st) = @_;

-        next if (!$st || S_ISDIR($st->mode));
-
-        my $info;
+        # Strip the vtype subdir from beginning of the current path
+        # so that we don't have to take it into account when parsing the file name
+        my $filename = $path;
+        if ($filename !~ s!^\Q$vtype_subdir\E!!) {
+            return;
+        }

         if ($vtype eq 'iso') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+            return if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;

-            $info = { volid => "$storeid:iso/$1", format => 'iso' };
+            return {
+                volid => "$storeid:iso/$1",
+                format => 'iso',
+            };
+        }

-        } elsif ($vtype eq 'vztmpl') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+        if ($vtype eq 'vztmpl') {
+            return if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;

-            $info = { volid => "$storeid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };
+            return {
+                volid => "$storeid:vztmpl/$1",
+                format => $2 eq 'tar' ? $2 : "t$2",
+            };
+        }

-        } elsif ($vtype eq 'backup') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
-            my $original = $filename;
+        if ($vtype eq 'backup') {
+            return if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+
+            my $original = $path;
             my $format = $2;
             $filename = $1;

             # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            next if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
+            return if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;

-            $info = { volid => "$storeid:backup/$filename", format => $format };
+            my $info = {
+                volid => "$storeid:backup/$filename",
+                format => $format,
+            };

             my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};

@@ -1722,24 +1739,42 @@ my sub get_subdir_files {
             }

             $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
-        } elsif ($vtype eq 'snippets') {

-            $info = {
+            return $info;
+        }
+
+        if ($vtype eq 'snippets') {
+            return {
                 volid => "$storeid:snippets/" . basename($filename),
                 format => 'snippet',
             };
-        } elsif ($vtype eq 'import') {
-            next
+        }
+
+        if ($vtype eq 'import') {
+            return
                 if $filename !~
                 m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;

-            $info = { volid => "$storeid:import/$1", format => "$2" };
+            return {
+                volid => "$storeid:import/$1",
+                format => "$2",
+            };
         }

-        $info->{size} = $st->size;
-        $info->{ctime} //= $st->ctime;
+        return;
+    };

-        push @$res, $info;
+    for my $path (<$vtype_subdir/*>) {
+        my $st = File::stat::stat($path);
+
+        next if (!$st || S_ISDIR($st->mode));
+
+        if (defined(my $info = $get_subdir_file_info->($path, $st))) {
+            $info->{size} = $st->size;
+            $info->{ctime} //= $st->ctime;
+
+            push $res->@*, $info;
+        }
     }

     return $res;
@@ -1758,18 +1793,16 @@ sub list_volumes {
         if ($type eq 'images' || $type eq 'rootdir') {
             $data = $class->list_images($storeid, $scfg, $vmid);
         } elsif ($scfg->{path}) {
-            my $path = plugin_get_vtype_subdir($scfg, $type);
-
             if ($type eq 'iso' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $path, 'iso');
+                $data = get_subdir_files($storeid, $scfg, 'iso', undef);
             } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $path, 'vztmpl');
+                $data = get_subdir_files($storeid, $scfg, 'vztmpl', undef);
             } elsif ($type eq 'backup') {
-                $data = get_subdir_files($storeid, $path, 'backup', $vmid);
+                $data = get_subdir_files($storeid, $scfg, 'backup', $vmid);
             } elsif ($type eq 'snippets') {
-                $data = get_subdir_files($storeid, $path, 'snippets');
+                $data = get_subdir_files($storeid, $scfg, 'snippets', undef);
             } elsif ($type eq 'import') {
-                $data = get_subdir_files($storeid, $path, 'import');
+                $data = get_subdir_files($storeid, $scfg, 'import', undef);
             }
         }

--
2.47.3

[PATCH pve-storage v1 10/54] plugin: update code style of list_volumes plugin API method

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index c4eb6f18..d2e6e9df 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1787,7 +1787,7 @@ sub list_volumes {

     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
-    foreach my $type (@$content_types) {
+    for my $type ($content_types->@*) {
         my $data;

         if ($type eq 'images' || $type eq 'rootdir') {
@@ -1808,7 +1808,7 @@ sub list_volumes {

         next if !$data;

-        foreach my $item (@$data) {
+        for my $item ($data->@*) {
             if ($type eq 'images' || $type eq 'rootdir') {
                 my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
                 my $vmtype;
@@ -1825,7 +1825,7 @@ sub list_volumes {
                 $item->{content} = $type;
             }

-            push @$res, $item;
+            push $res->@*, $item;
         }
     }

--
2.47.3

[PATCH pve-storage v1 11/54] plugin: use closure for obtaining raw volume data in list_volumes

[Max R. Carrara]

To make the code in the `list_volumes()` API method a little more
readable and maintainable, break up the if-elsif chain that obtains
the raw volume data into a closure. The overall logic is kept the
same.

Also rename `$data` to `$raw_volumes` to make it a little clearer what
is actually being iterated over, since "data" does not really say
much.

Be a little more specific and use an un-definedness check on
`$raw_volumes` to continue with the iteration, instead of a plain
negation.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 51 ++++++++++++++++++++++++++-------------
 1 file changed, 34 insertions(+), 17 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index d2e6e9df..106cff6f 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1787,28 +1787,45 @@ sub list_volumes {

     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
-    for my $type ($content_types->@*) {
-        my $data;
+
+    my $get_raw_volumes_for_type = sub {
+        my ($type) = @_;

         if ($type eq 'images' || $type eq 'rootdir') {
-            $data = $class->list_images($storeid, $scfg, $vmid);
-        } elsif ($scfg->{path}) {
-            if ($type eq 'iso' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $scfg, 'iso', undef);
-            } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $scfg, 'vztmpl', undef);
-            } elsif ($type eq 'backup') {
-                $data = get_subdir_files($storeid, $scfg, 'backup', $vmid);
-            } elsif ($type eq 'snippets') {
-                $data = get_subdir_files($storeid, $scfg, 'snippets', undef);
-            } elsif ($type eq 'import') {
-                $data = get_subdir_files($storeid, $scfg, 'import', undef);
-            }
+            return $class->list_images($storeid, $scfg, $vmid);
         }

-        next if !$data;
+        return if !$scfg->{path};

-        for my $item ($data->@*) {
+        if ($type eq 'iso' && !defined($vmid)) {
+            return get_subdir_files($storeid, $scfg, 'iso', undef);
+        }
+
+        if ($type eq 'vztmpl' && !defined($vmid)) {
+            return get_subdir_files($storeid, $scfg, 'vztmpl', undef);
+        }
+
+        if ($type eq 'backup') {
+            return get_subdir_files($storeid, $scfg, 'backup', $vmid);
+        }
+
+        if ($type eq 'snippets') {
+            return get_subdir_files($storeid, $scfg, 'snippets', undef);
+        }
+
+        if ($type eq 'import') {
+            return get_subdir_files($storeid, $scfg, 'import', undef);
+        }
+
+        return;
+    };
+
+    for my $type ($content_types->@*) {
+        my $raw_volumes = $get_raw_volumes_for_type->($type);
+
+        next if !defined($raw_volumes);
+
+        for my $item ($raw_volumes->@*) {
             if ($type eq 'images' || $type eq 'rootdir') {
                 my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
                 my $vmtype;
--
2.47.3

[PATCH pve-storage v1 12/54] plugin: use closure for inner loop logic in list_volumes

[Max R. Carrara]

Factor most of the body of the inner loop of the `list_volumes()`
plugin API method into a closure and provide additional context for
why we have such an elaborate check for VM / CT disks in the first
place.

This has the added benefit that we now also do not modify the volume
hashes on the fly anymore when determining whether a volume's type
aligns with a guest's type. Small difference, but might make it a
little easier to adapt that code in the future, e.g. when we align
vtypes and content types.

Also, rename `$item` to `$volume` for clarity's sake.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 49 +++++++++++++++++++++++++--------------
 1 file changed, 32 insertions(+), 17 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 106cff6f..c96c1ed7 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1820,29 +1820,44 @@ sub list_volumes {
         return;
     };

+    my $get_content_type_for_volume = sub {
+        my ($volume, $type) = @_;
+
+        # It is currently impossible to determine whether a volume belongs to a
+        # VM or a CT based on the output of list_images alone.
+        # Therefore, we use the parsed ID ($volume->{vmid}) to look up the guest
+        # and make sure that the guest's type corresponds to the volume's type.
+        if ($type eq 'images' || $type eq 'rootdir') {
+            my $vminfo = $vmlist->{ids}->{ $volume->{vmid} };
+
+            my $vmtype;
+            if (defined($vminfo)) {
+                $vmtype = $vminfo->{type};
+            }
+
+            my $content_type = 'images';
+            if (defined($vmtype) && $vmtype eq 'lxc') {
+                $content_type = 'rootdir';
+            }
+
+            return if $content_type ne $type;
+        }
+
+        return $type;
+    };
+
     for my $type ($content_types->@*) {
         my $raw_volumes = $get_raw_volumes_for_type->($type);

         next if !defined($raw_volumes);

-        for my $item ($raw_volumes->@*) {
-            if ($type eq 'images' || $type eq 'rootdir') {
-                my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
-                my $vmtype;
-                if (defined($vminfo)) {
-                    $vmtype = $vminfo->{type};
-                }
-                if (defined($vmtype) && $vmtype eq 'lxc') {
-                    $item->{content} = 'rootdir';
-                } else {
-                    $item->{content} = 'images';
-                }
-                next if $type ne $item->{content};
-            } else {
-                $item->{content} = $type;
-            }
+        for my $volume ($raw_volumes->@*) {
+            my $content_type = $get_content_type_for_volume->($volume, $type);

-            push $res->@*, $item;
+            next if !defined($content_type);
+
+            $volume->{content} = $content_type;
+            push $res->@*, $volume;
         }
     }

--
2.47.3

[PATCH pve-storage v1 13/54] storage: update code style in function path_to_volume_id

[Max R. Carrara]

Also rename the initially parsed storeid from `$sid` to
`$parsed_storeid`.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index ef2e7887..70844602 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -696,9 +696,9 @@ sub path_to_volume_id {

     my $ids = $cfg->{ids};

-    my ($sid, $volname) = parse_volume_id($path, 1);
-    if ($sid) {
-        if (my $scfg = $ids->{$sid}) {
+    my ($parsed_storeid, $volname) = parse_volume_id($path, 1);
+    if ($parsed_storeid) {
+        if (my $scfg = $ids->{$parsed_storeid}) {
             if ($scfg->{path}) {
                 my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
                 my ($vtype, $name, $vmid) = $plugin->parse_volname($volname);
@@ -712,7 +712,7 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;

-    foreach my $sid (keys %$ids) {
+    for my $sid (keys $ids->%*) {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
@@ -728,7 +728,7 @@ sub path_to_volume_id {
             my $name = $2;

             my $vollist = $plugin->list_images($sid, $scfg, $vmid);
-            foreach my $info (@$vollist) {
+            for my $info ($vollist->@*) {
                 my ($storeid, $volname) = parse_volume_id($info->{volid});
                 my $volpath = $plugin->path($scfg, $volname, $storeid);
                 if ($volpath eq $path) {
--
2.47.3

[PATCH pve-storage v1 14/54] storage: break up needless if-elsif chain in path_to_volume_id

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 70844602..e9b0cb7e 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -735,19 +735,31 @@ sub path_to_volume_id {
                     return ('images', $info->{volid});
                 }
             }
-        } elsif ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
+
+            return ('');
+        }
+
+        if ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
             my $name = $1;
             return ('iso', "$sid:iso/$name");
-        } elsif ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
+        }
+
+        if ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
             my $name = $1;
             return ('vztmpl', "$sid:vztmpl/$name");
-        } elsif ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
+        }
+
+        if ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
             my $name = $1;
             return ('backup', "$sid:backup/$name");
-        } elsif ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
+        }
+
+        if ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
             my $name = $1;
             return ('snippets', "$sid:snippets/$name");
-        } elsif ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
+        }
+
+        if ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
             my $name = $1;
             return ('import', "$sid:import/$name");
         }
--
2.47.3

[PATCH pve-storage v1 15/54] storage: heave vtype file path parsing logic inside loop into helper

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index e9b0cb7e..2886ab9e 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -712,10 +712,9 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;

-    for my $sid (keys $ids->%*) {
-        my $scfg = $ids->{$sid};
-        next if !$scfg->{path};
-        my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+    my $parse_volid_from_file_path = sub {
+        my ($plugin, $sid, $scfg) = @_;
+
         my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
         my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
         my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
@@ -763,6 +762,17 @@ sub path_to_volume_id {
             my $name = $1;
             return ('import', "$sid:import/$name");
         }
+
+        return ('');
+    };
+
+    for my $sid (keys $ids->%*) {
+        my $scfg = $ids->{$sid};
+        next if !$scfg->{path};
+        my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+        my ($vtype, $volid) = $parse_volid_from_file_path->($plugin, $sid, $scfg);
+        return ($vtype, $volid) if $vtype;
     }

     # can't map path to volume id
--
2.47.3

[PATCH pve-storage v1 16/54] storage: clean up code that was moved into helper in path_to_volume_id

[Max R. Carrara]

In order to make the parsing logic in `path_to_volume_id()` both
simpler and more manageable for future changes, do the following:

1. Iterate over a list of vtypes that store their contents in
   subdirectories and fetch said subdirectories dynamically instead of
   hardcoding them.

2. Explicitly exclude the `rootdir` vtype in this list of vtypes, as
   its subdirectory is currently not used.

3. Strip the vtype subdirectory from the beginning of the path in
   order to make matching on the file name easier. Return early as a
   safeguard if this fails.

4. Adapt the original regexes in the parsing helper to not include the
   vtype subdir anymore, since this is now being stripped.

5. Instead of returning a 2-element array, return `undef` if the path
   fails to parse, and the volume ID (volid) alone if parsing
   succeeds. Since the caller is now passing along the vtype, it does
   not need to be returned anymore.

Note that the parsing logic of the `parse_volid_from_file_path()`
helper inside `path_to_volume_id()` now is very similar to the parsing
logic of the `get_subdir_file_info()` closure in the
`get_subdir_files()` helper inside `Plugin.pm`. This opens up further
simplification / code de-duplication in the future, so that our
parsing logic can eventually be kept in one place only.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 73 +++++++++++++++++++++++++++++++---------------
 1 file changed, 50 insertions(+), 23 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 2886ab9e..1b0569dd 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -23,6 +23,7 @@ use PVE::INotify;
 use PVE::RPCEnvironment;
 use PVE::SSHInfo;
 use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
 use PVE::RESTEnvironment qw(log_warn);
@@ -712,17 +713,31 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;

+    my $get_vtypes_to_check = sub {
+        my $default_vtype_subdirs = plugin_get_default_vtype_subdirs();
+
+        # TODO: vtype split: handle directory for 'rootdir' type.
+        # CTs currently use the same dir as VMs --> the dir for the 'images' type.
+        # Directory is therefore unused right now, so exclude it.
+        delete $default_vtype_subdirs->{rootdir};
+
+        return [sort keys $default_vtype_subdirs->%*];
+    };
+
     my $parse_volid_from_file_path = sub {
-        my ($plugin, $sid, $scfg) = @_;
+        my ($plugin, $sid, $scfg, $vtype) = @_;

-        my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
-        my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
-        my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
-        my $backupdir = plugin_get_vtype_subdir($scfg, 'backup');
-        my $snippetsdir = plugin_get_vtype_subdir($scfg, 'snippets');
-        my $importdir = plugin_get_vtype_subdir($scfg, 'import');
+        my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);

-        if ($path =~ m!^\Q$imagedir\E/(\d+)/([^/\s]+)$!) {
+        # Strip the vtype subdir from beginning of the current path
+        # so that we don't have to take it into account when parsing the file name
+        my $filename = $path;
+        if ($filename !~ s!^\Q$vtype_subdir\E!!) {
+            return;
+        }
+
+        if ($vtype eq 'images') {
+            return if $filename !~ m!/(\d+)/([^/\s]+)$!;
             my $vmid = $1;
             my $name = $2;

@@ -731,48 +746,60 @@ sub path_to_volume_id {
                 my ($storeid, $volname) = parse_volume_id($info->{volid});
                 my $volpath = $plugin->path($scfg, $volname, $storeid);
                 if ($volpath eq $path) {
-                    return ('images', $info->{volid});
+                    return $info->{volid};
                 }
             }

-            return ('');
+            return;
         }

-        if ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
+        if ($vtype eq 'iso') {
+            return if $filename !~ m!/([^/]+$ISO_EXT_RE_0)$!;
             my $name = $1;
-            return ('iso', "$sid:iso/$name");
+            return "$sid:iso/$name";
         }

-        if ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
+        if ($vtype eq 'vztmpl') {
+            return if $filename !~ m!/([^/]+$VZTMPL_EXT_RE_1)$!;
             my $name = $1;
-            return ('vztmpl', "$sid:vztmpl/$name");
+            return "$sid:vztmpl/$name";
         }

-        if ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
+        if ($vtype eq 'backup') {
+            return if $filename !~ m!/([^/]+$BACKUP_EXT_RE_2)$!;
             my $name = $1;
-            return ('backup', "$sid:backup/$name");
+            return "$sid:backup/$name";
         }

-        if ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
+        if ($vtype eq 'snippets') {
+            return if $filename !~ m!/([^/]+)$!;
             my $name = $1;
-            return ('snippets', "$sid:snippets/$name");
+            return "$sid:snippets/$name";
         }

-        if ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
+        if ($vtype eq 'import') {
+            return if $filename !~ m!/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!;
             my $name = $1;
-            return ('import', "$sid:import/$name");
+            return "$sid:import/$name";
         }

-        return ('');
+        return;
     };

+    my $vtypes_to_check = $get_vtypes_to_check->();
+
     for my $sid (keys $ids->%*) {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});

-        my ($vtype, $volid) = $parse_volid_from_file_path->($plugin, $sid, $scfg);
-        return ($vtype, $volid) if $vtype;
+        for my $vtype ($vtypes_to_check->@*) {
+            my $volid = $parse_volid_from_file_path->($plugin, $sid, $scfg, $vtype);
+
+            if (defined($volid)) {
+                return ($vtype, $volid);
+            }
+        }
     }

     # can't map path to volume id
--
2.47.3

[PATCH pve-storage v1 17/54] api: status: move content type assert for up-/downloads into helper

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 41 +++++++++++++++++++++-------------
 1 file changed, 26 insertions(+), 15 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 8225c3ad..b9e949ff 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -41,6 +41,29 @@ __PACKAGE__->register_method({
     path => '{storage}/file-restore',
 });

+# Content types that may currently be up- or downloaded via the API
+my @ALLOWED_FILE_TRANSFER_CONTENT_TYPES = ('iso', 'vztmpl', 'import');
+
+my sub assert_can_transfer_files : prototype($$$) {
+    my ($scfg, $storeid, $vtype) = @_;
+
+    if (!defined($scfg->{path})) {
+        die "unable to transfer files from / to storage type '$scfg->{type}'"
+            . " - not a file-based storage\n";
+    }
+
+    if (!$scfg->{content}->{$vtype}) {
+        die "storage '$storeid' is not configured for content type '$vtype'\n";
+    }
+
+    my $is_allowed_vtype = grep { $vtype } @ALLOWED_FILE_TRANSFER_CONTENT_TYPES;
+    if (!$is_allowed_vtype) {
+        raise_param_exc({ content => "content type '$vtype' not allowed for upload / download" });
+    }
+
+    return;
+}
+
 my sub assert_ova_contents {
     my ($file) = @_;

@@ -573,11 +596,10 @@ __PACKAGE__->register_method({
         my ($node, $storage) = $param->@{qw(node storage)};
         my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);

-        die "can't upload to storage type '$scfg->{type}'\n"
-            if !defined($scfg->{path});
-
         my $content = $param->{content};

+        assert_can_transfer_files($scfg, $storage, $content);
+
         my $tmpfilename = $param->{tmpfilename};
         die "missing temporary file name\n" if !$tmpfilename;

@@ -615,13 +637,8 @@ __PACKAGE__->register_method({
             }

             $path = PVE::Storage::get_import_dir($cfg, $storage);
-        } else {
-            raise_param_exc({ content => "upload content type '$content' not allowed" });
         }

-        die "storage '$storage' does not support '$content' content\n"
-            if !$scfg->{content}->{$content};
-
         my $dest = "$path/$filename";
         my $dirname = dirname($dest);

@@ -817,13 +834,9 @@ __PACKAGE__->register_method({
         my ($node, $storage, $compression) = $param->@{qw(node storage compression)};
         my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);

-        die "can't upload to storage type '$scfg->{type}', not a file based storage!\n"
-            if !defined($scfg->{path});
-
         my ($content, $url) = $param->@{ 'content', 'url' };

-        die "storage '$storage' is not configured for content-type '$content'\n"
-            if !$scfg->{content}->{$content};
+        assert_can_transfer_files($scfg, $storage, $content);

         my $filename = PVE::Storage::normalize_content_filename($param->{filename});

@@ -856,8 +869,6 @@ __PACKAGE__->register_method({
             }

             $path = PVE::Storage::get_import_dir($cfg, $storage);
-        } else {
-            raise_param_exc({ content => "upload content-type '$content' is not allowed" });
         }

         PVE::Storage::activate_storage($cfg, $storage);
--
2.47.3

[PATCH pve-storage v1 18/54] api: status: use helper from common module to get content directory

[Max R. Carrara]

In the subroutines for the upload / download_url API methods, use the
`plugin_get_vtype_subdir()` helper from `PVE::Storage::Common` to
obtain the sub-directory for the provided content type (vtype),
instead of doing that within the if-elsif chains.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index b9e949ff..742596d4 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -20,6 +20,9 @@ use PVE::API2::Storage::Content;
 use PVE::API2::Storage::FileRestore;
 use PVE::API2::Storage::PruneBackups;
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);

 use base qw(PVE::RESTHandler);

@@ -608,7 +611,6 @@ __PACKAGE__->register_method({

         my $filename = PVE::Storage::normalize_content_filename($param->{filename});

-        my $path;
         my $is_ova = 0;
         my $image_format;

@@ -616,12 +618,10 @@ __PACKAGE__->register_method({
             if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_iso_dir($cfg, $storage);
         } elsif ($content eq 'vztmpl') {
             if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_vztmpl_dir($cfg, $storage);
         } elsif ($content eq 'import') {
             if ($filename !~
                 m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
@@ -635,10 +635,10 @@ __PACKAGE__->register_method({
             } else {
                 $image_format = $format;
             }
-
-            $path = PVE::Storage::get_import_dir($cfg, $storage);
         }

+        my $path = plugin_get_vtype_subdir($scfg, $content);
+
         my $dest = "$path/$filename";
         my $dirname = dirname($dest);

@@ -840,7 +840,6 @@ __PACKAGE__->register_method({

         my $filename = PVE::Storage::normalize_content_filename($param->{filename});

-        my $path;
         my $is_ova = 0;
         my $image_format;

@@ -848,12 +847,10 @@ __PACKAGE__->register_method({
             if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_iso_dir($cfg, $storage);
         } elsif ($content eq 'vztmpl') {
             if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_vztmpl_dir($cfg, $storage);
         } elsif ($content eq 'import') {
             if ($filename !~
                 m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
@@ -867,10 +864,10 @@ __PACKAGE__->register_method({
             } else {
                 $image_format = $format;
             }
-
-            $path = PVE::Storage::get_import_dir($cfg, $storage);
         }

+        my $path = plugin_get_vtype_subdir($scfg, $content);
+
         PVE::Storage::activate_storage($cfg, $storage);
         File::Path::make_path($path);

--
2.47.3

[PATCH pve-storage v1 19/54] api: status: move up-/download file path parsing code into helper

[Max R. Carrara]

Create a separate helper for the common file path parsing parts of the
upload / download_url API methods that returns the extension of the
uploaded file. Break up the if-elsif chain along the way.

Keep the inner if-clauses for the 'import' vtype inline for now for
further simplifications later, but don't check whether we have the
'import' vtype before those anymore. This is fine because the 'ova'
file extension is not used for files of any other vtype.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 85 ++++++++++++++++++----------------
 1 file changed, 45 insertions(+), 40 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 742596d4..3a798899 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -67,6 +67,41 @@ my sub assert_can_transfer_files : prototype($$$) {
     return;
 }

+my sub parse_transferred_file_path_extension : prototype($$) {
+    my ($path, $vtype) = @_;
+
+    if ($vtype eq 'iso') {
+        if ($path !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    if ($vtype eq 'vztmpl') {
+        if ($path !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    if ($vtype eq 'import') {
+        if (
+            $path !~ m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
+        ) {
+            raise_param_exc({ filename => "invalid filename or wrong extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
+}
+
 my sub assert_ova_contents {
     my ($file) = @_;

@@ -614,27 +649,12 @@ __PACKAGE__->register_method({
         my $is_ova = 0;
         my $image_format;

-        if ($content eq 'iso') {
-            if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'vztmpl') {
-            if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'import') {
-            if ($filename !~
-                m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-            ) {
-                raise_param_exc({ filename => "invalid filename or wrong extension" });
-            }
-            my $format = $1;
+        my $extension = parse_transferred_file_path_extension($filename, $content);

-            if ($format eq 'ova') {
-                $is_ova = 1;
-            } else {
-                $image_format = $format;
-            }
+        if ($extension eq 'ova') {
+            $is_ova = 1;
+        } else {
+            $image_format = $extension;
         }

         my $path = plugin_get_vtype_subdir($scfg, $content);
@@ -843,27 +863,12 @@ __PACKAGE__->register_method({
         my $is_ova = 0;
         my $image_format;

-        if ($content eq 'iso') {
-            if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'vztmpl') {
-            if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'import') {
-            if ($filename !~
-                m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-            ) {
-                raise_param_exc({ filename => "invalid filename or wrong extension" });
-            }
-            my $format = $1;
+        my $extension = parse_transferred_file_path_extension($filename, $content);

-            if ($format eq 'ova') {
-                $is_ova = 1;
-            } else {
-                $image_format = $format;
-            }
+        if ($extension eq 'ova') {
+            $is_ova = 1;
+        } else {
+            $image_format = $extension;
         }

         my $path = plugin_get_vtype_subdir($scfg, $content);
--
2.47.3

[PATCH pve-storage v1 20/54] api: status: simplify file content assertion logic for up-/download

[Max R. Carrara]

Create another helper for the file content assertion logic in the
upload / download_url API methods. Absorb the `assert_ova_contents()`
helper along the way, since it does not need to be standalone anymore.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 89 ++++++++++++----------------------
 1 file changed, 32 insertions(+), 57 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 3a798899..2b0b121a 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -102,25 +102,40 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
 }

-my sub assert_ova_contents {
-    my ($file) = @_;
+my sub assert_file_transfer_contents_valid : prototype($$$) {
+    my ($path, $vtype, $extension) = @_;

-    # test if it's really a tar file with an ovf file inside
-    my $hasOvf = 0;
-    run_command(
-        ['tar', '-t', '-f', $file],
-        outfunc => sub {
-            my ($line) = @_;
+    if ($vtype eq 'iso') {
+        PVE::Storage::assert_iso_content($path);
+        return;
+    }

-            if ($line =~ m/\.ovf$/) {
-                $hasOvf = 1;
-            }
-        },
-    );
+    if ($vtype eq 'import') {
+        if ($extension eq 'ova') {
+            # test if it's really a tar file with an ovf file inside
+            my $hasOvf = 0;
+            run_command(
+                ['tar', '-t', '-f', $path],
+                outfunc => sub {
+                    my ($line) = @_;

-    die "ova archive has no .ovf file inside\n" if !$hasOvf;
+                    if ($line =~ m/\.ovf$/) {
+                        $hasOvf = 1;
+                    }
+                },
+            );

-    return 1;
+            die "ova archive has no .ovf file inside\n" if !$hasOvf;
+            return;
+        }
+
+        my $is_untrusted = 1;
+        my $timeout = 10;
+
+        PVE::Storage::file_size_info($path, $timeout, $extension, $is_untrusted);
+
+        return;
+    }
 }

 __PACKAGE__->register_method({
@@ -645,18 +660,7 @@ __PACKAGE__->register_method({
         die "temporary file '$tmpfilename' does not exist\n" if !defined($size);

         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
-
-        my $is_ova = 0;
-        my $image_format;
-
         my $extension = parse_transferred_file_path_extension($filename, $content);
-
-        if ($extension eq 'ova') {
-            $is_ova = 1;
-        } else {
-            $image_format = $extension;
-        }
-
         my $path = plugin_get_vtype_subdir($scfg, $content);

         my $dest = "$path/$filename";
@@ -725,16 +729,7 @@ __PACKAGE__->register_method({
                     }
                 }

-                if ($content eq 'iso') {
-                    PVE::Storage::assert_iso_content($tmpfilename);
-                }
-
-                if ($is_ova) {
-                    assert_ova_contents($tmpfilename);
-                } elsif (defined($image_format)) {
-                    # checks untrusted image
-                    PVE::Storage::file_size_info($tmpfilename, 10, $image_format, 1);
-                }
+                assert_file_transfer_contents_valid($tmpfilename, $content, $extension);
             };
             if (my $err = $@) {
                 # unlinks only the temporary file from the http server
@@ -859,18 +854,7 @@ __PACKAGE__->register_method({
         assert_can_transfer_files($scfg, $storage, $content);

         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
-
-        my $is_ova = 0;
-        my $image_format;
-
         my $extension = parse_transferred_file_path_extension($filename, $content);
-
-        if ($extension eq 'ova') {
-            $is_ova = 1;
-        } else {
-            $image_format = $extension;
-        }
-
         my $path = plugin_get_vtype_subdir($scfg, $content);

         PVE::Storage::activate_storage($cfg, $storage);
@@ -893,16 +877,7 @@ __PACKAGE__->register_method({
         $opts->{assert_file_validity} = sub {
             my ($tmp_path) = @_;

-            if ($content eq 'iso') {
-                PVE::Storage::assert_iso_content($tmp_path);
-            }
-
-            if ($is_ova) {
-                assert_ova_contents($tmp_path);
-            } elsif (defined($image_format)) {
-                # checks untrusted image
-                PVE::Storage::file_size_info($tmp_path, 10, $image_format, 1);
-            }
+            assert_file_transfer_contents_valid($tmp_path, $content, $extension);
         };

         my $worker = sub {
--
2.47.3

[PATCH pve-storage v1 21/54] test: guest import: add tests for PVE::GuestImport

[Max R. Carrara]

Set up a new test script for the `extract_disk_from_import_file()` sub
of `PVE::GuestImport` and run it as part of
`src/test/run_plugin_tests.pl`.

Make this possible by mocking cluster-related functionality that would
otherwise require to be run as root, namely reading, updating and
locking the storage config, as well as retrieving the list of
available VMs.

Then, make the test cases declarative in such a way that they are
mostly self-documenting regarding what they test. Document the keys
that the test declarations use.

Run each test in its clean environment, creating and removing the
entire directory structure before and after each test. Provide small
helpers specific to the testing code for creating arbitrary QEMU
images / .ova files and document them briefly.

Test for successful as well as failed imports.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/guest_import_test.pl | 948 ++++++++++++++++++++++++++++++++++
 src/test/run_plugin_tests.pl  |   1 +
 2 files changed, 949 insertions(+)
 create mode 100755 src/test/guest_import_test.pl

diff --git a/src/test/guest_import_test.pl b/src/test/guest_import_test.pl
new file mode 100755
index 00000000..04eec24d
--- /dev/null
+++ b/src/test/guest_import_test.pl
@@ -0,0 +1,948 @@
+#!/usr/bin/env perl
+
+use v5.36;
+
+use lib qw(..);
+
+use re qw(is_regexp);
+
+use Carp qw(confess);
+use File::Basename qw(dirname basename);
+use File::Path qw(make_path remove_tree);
+use File::Spec;
+use File::Temp;
+
+use PVE::Cluster;
+use PVE::GuestImport;
+use PVE::Storage;
+use PVE::Storage::Common qw(plugin_get_vtype_subdir);
+use PVE::Tools qw(run_command);
+
+use Test::More;
+use Test::MockModule;
+
+$ENV{TZ} = 'UTC';
+
+my $DEFAULT_SIZE = 4 * 1024 * 1024; # 4 MiB
+
+my $SOURCE_STOREID = 'source-store';
+my $SOURCE_STORAGE_PATH = File::Temp->newdir();
+
+my $TARGET_STOREID = 'target-store';
+my $TARGET_STORAGE_PATH = File::Temp->newdir();
+
+# Not actually set up on disk or elsewhere
+my $DUMMY_LVMTHIN_STOREID = 'dummy-lvmthin';
+
+my $STORAGE_CFG = <<~EOF;
+dir: $SOURCE_STOREID
+    shared 0
+    path $SOURCE_STORAGE_PATH
+    content images,iso,import
+
+dir: $TARGET_STOREID
+    shared 0
+    path $TARGET_STORAGE_PATH
+    content images,iso,import
+
+lvmthin: $DUMMY_LVMTHIN_STOREID
+    thinpool data
+    vgname pve
+    content images,rootdir
+EOF
+
+my $MOCKED_VMLIST = {
+    version => 1,
+    ids => {
+        1337 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 4,
+        },
+        1338 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 7,
+        },
+        1339 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 7,
+        },
+    },
+};
+
+my $mock_pve_cluster = Test::MockModule->new('PVE::Cluster')->redefine(
+    cfs_update => sub { },
+    get_config => sub($file) {
+        if ($file eq 'storage.cfg') {
+            return $STORAGE_CFG;
+        }
+
+        confess "mocked get_config() - file '$file' not handled";
+    },
+    get_vmlist => sub() { return $MOCKED_VMLIST; },
+);
+
+my $_LOCKDIR = File::Temp->newdir();
+my $mock_pve_storage_plugin = Test::MockModule->new('PVE::Storage::Plugin')->redefine(
+    cluster_lock_storage => sub($class, $storeid, $shared, $timeout, $func, @param) {
+        confess "\$shared = 1 is not handled" if $shared;
+
+        my $res = PVE::Tools::lock_file("$_LOCKDIR/pve-storage-test", $timeout, $func, @param);
+        die $@ if $@;
+        return $res;
+    },
+);
+
+=head3 create_test_image_file
+
+    my $file_path = "/foo/bar/disk.qcow2";
+    create_test_image_file($file_path, 'qcow2');
+
+Create a new QEMU image at C<$file_path> using the C<qemu-img> command,
+optionally with a certain C<$format>.
+
+The known formats are currently C<qcow2>, C<raw>, and C<vmdk>.
+
+=cut
+
+my sub create_test_image_file : prototype($;$) ($file_path, $format = undef) {
+    my $KNOWN_FORMATS = {
+        qcow2 => 1,
+        raw => 1,
+        vmdk => 1,
+    };
+
+    my $command = ['/usr/bin/qemu-img', 'create', $file_path, $DEFAULT_SIZE];
+
+    if (defined($format) && defined($KNOWN_FORMATS->{$format})) {
+        push($command->@*, '-f', $format);
+    }
+
+    eval { run_command($command, timeout => 10); };
+    confess $@ if $@;
+
+    return;
+}
+
+=head3 create_test_ova_file
+
+    my $file_path = "/srv/imports/some-import.ova";
+    my $content_file_name = "some-disk.qcow2";
+
+    create_test_ova_file($file_path, $content_file_name)
+
+Create a new C<.ova> file at the I<absolute> C<$file_path> and put a file named
+C<$content_file_name> inside using the C<tar> command. C<$content_file_name>
+must not have any directory components and is assumed to be in the same
+directory as C<$file_path>.
+
+=cut
+
+my sub create_test_ova_file : prototype($$) ($file_path, $content_file_name) {
+    my $file_path_dir = dirname($file_path);
+
+    confess "\$content_file_name ne basename(\$content_file_name)"
+        if $content_file_name ne basename($content_file_name);
+
+    my $tar_command = [
+        'tar',
+        '-c',
+        '--force-local',
+        '--no-same-owner',
+        '-f',
+        $file_path,
+        '-C',
+        $file_path_dir,
+        $content_file_name,
+    ];
+
+    eval { run_command($tar_command, timeout => 10); };
+    confess $@ if $@;
+
+    return;
+}
+
+=head2 TEST CASE FORMAT
+
+The parameters for individual tests are hashes with the following keys:
+
+    {
+        # Name of the test, also displayed on error.
+        description => "Importing a single disk, no existing disks (qcow2)",
+
+        # Definition of the source storage, i.e. the storage from which to import.
+        source => {
+            # The ID of the source storage.
+            storeid => $SOURCE_STOREID,
+
+            # A setup function used to set up the necessary volumes before the test cases are run.
+            # This is used to actually create the file(s) to be imported, for example.
+            'fn-setup-volumes' => sub($storeid) {
+                # [...]
+                return;
+            },
+        },
+
+        # Definition of the target storage, i.e. the storage which disk images are imported to.
+        target => {
+
+            # The ID of the target storage.
+            storeid => $TARGET_STOREID,
+
+            # Like for the source storage, this function can be used to set up any pre-existing
+            # volumes for the storage.
+            'fn-setup-volumes' => sub($storeid) {
+                # [...]
+                return;
+            },
+        },
+
+        # The cases being tested.
+        cases => [
+            {
+                # The name of the volume to import.
+                volname => 'import/some-import.ova/disk.qcow2',
+
+                # The ID of the guest to import the volume for.
+                vmid => 1337,
+
+                # The expected outcome when the invocation succeeds.
+                # May be a string or a regular expression.
+                # If this is a string, then the output has to be an exact match.
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+
+            {
+                volname => 'import/some-import.ova/disk-link.qcow2',
+                vmid => 1337,
+
+                # The expected outcome when the invocation fails.
+                # May be a string or a regular expression.
+                # If this is a string, then the output has to be an exact match.
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+
+=cut
+
+my $tests = [
+    {
+        description => "Importing a single disk, no existing disks (qcow2)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no existing disks (vmdk)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.vmdk';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'vmdk');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.vmdk',
+                vmid => 1338,
+                'expect-match' => "$TARGET_STOREID:1338/vm-1338-disk-0.vmdk",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no existing disks (raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.raw';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'raw');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.raw',
+                vmid => 1339,
+                'expect-match' => "$TARGET_STOREID:1339/vm-1339-disk-0.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid extension",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.txt';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.txt',
+                vmid => 1337,
+                'expect-fail' => qr/unable to parse directory volume name/,
+            },
+        ],
+    },
+    {
+        description => "Importing multiple disks, no existing disks (qcow2, vmdk, raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                {
+                    my $volume_file_name = 'some-qcow2-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.qcow2';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'qcow2');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-vmdk-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.vmdk';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'vmdk');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-raw-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.raw';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'raw');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+            {
+                volname => 'import/some-vmdk-import.ova/disk.vmdk',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-1.vmdk",
+            },
+            {
+                volname => 'import/some-raw-import.ova/disk.raw',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-2.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing multiple disks, existing disks (qcow2, vmdk, raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                {
+                    my $volume_file_name = 'some-qcow2-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.qcow2';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'qcow2');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-vmdk-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.vmdk';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'vmdk');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-raw-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.raw';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'raw');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $vmid = 1337;
+
+                PVE::Storage::vdisk_alloc(
+                    $cfg, $storeid, $vmid, 'qcow2', undef, $DEFAULT_SIZE,
+                );
+
+                PVE::Storage::vdisk_alloc(
+                    $cfg, $storeid, $vmid, 'qcow2', undef, $DEFAULT_SIZE,
+                );
+
+                return;
+            },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-2.qcow2",
+            },
+            {
+                volname => 'import/some-vmdk-import.ova/disk.vmdk',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-3.vmdk",
+            },
+            {
+                volname => 'import/some-raw-import.ova/disk.raw',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-4.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid volume name",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-🐧-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/unable to parse directory volume name/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid volume type",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $iso_dir = PVE::Storage::get_iso_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-linux-distro.iso';
+                my $volume_file_path = File::Spec->catfile($iso_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, undef);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'iso/some-linux-distro.iso',
+                vmid => 1337,
+                'expect-fail' => qr/only files with content type 'import' can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, not .ova file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-qcow2-import.ovf';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, undef);
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ovf',
+                vmid => 1337,
+                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no content referenced",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova',
+                vmid => 1337,
+                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid archive format for .ova",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, 'qcow2');
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/error during extraction/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, symlink instead of file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                my $symlink_file_name = 'disk-link.qcow2';
+                my $symlink_file_path = File::Spec->catfile($import_dir, $symlink_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                symlink($content_file_path, $symlink_file_path)
+                    or confess "creating symlink failed";
+
+                create_test_ova_file($volume_file_path, $symlink_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk-link.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, directory instead of file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                make_path($content_file_path, { verbose => 1, mode => 0750 });
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, file is not an image",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                eval { `touch $content_file_path` };
+                confess $@ if $@;
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/Could not open .* Image is not in qcow2 format/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, incompatible target storage type",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $DUMMY_LVMTHIN_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1339,
+                'expect-fail' => qr/has no path/,
+            },
+        ],
+    },
+];
+
+my sub setup_scfg_dir($scfg) {
+    confess "\$scfg has no 'path' key" if !$scfg->{path};
+
+    make_path($scfg->{path}, { verbose => 1, mode => 0700 });
+
+    my @vtype_subdirs = map { plugin_get_vtype_subdir($scfg, $_) } keys $scfg->{content}->%*;
+
+    make_path(@vtype_subdirs, { verbose => 1, mode => 0755 });
+
+    return;
+}
+
+my sub setup_test_env($test_def) {
+    my ($source, $target) = $test_def->@{qw(source target)};
+
+    my $setup_storage = sub($storage_def) {
+        my $cfg = PVE::Storage::config();
+
+        my $storeid = $storage_def->{storeid};
+        die "\$storeid = undef" if !defined($storeid);
+
+        my $scfg = PVE::Storage::storage_config($cfg, $storeid);
+
+        if ($scfg->{path}) {
+            setup_scfg_dir($scfg);
+        }
+
+        $storage_def->{'fn-setup-volumes'}->($storeid);
+    };
+
+    eval { $setup_storage->($source); };
+    if (my $err = $@) {
+        confess "failed to set up source storage: $err";
+    }
+
+    eval { $setup_storage->($target); };
+    if (my $err = $@) {
+        confess "failed to set up target storage: $err";
+    }
+
+    return;
+}
+
+my sub teardown_scfg_dir($scfg) {
+    confess "\$scfg has no 'path' key" if !$scfg->{path};
+
+    remove_tree($scfg->{path}, { verbose => 0 });
+
+    return;
+}
+
+my sub teardown_test_env($test_def) {
+    my ($source, $target) = $test_def->@{qw(source target)};
+
+    my $teardown_storage = sub($storage_def) {
+        my $cfg = PVE::Storage::config();
+
+        my $storeid = $storage_def->{storeid};
+        die "\$storeid = undef" if !defined($storeid);
+
+        my $scfg = PVE::Storage::storage_config($cfg, $storeid);
+        if ($scfg->{path}) {
+            teardown_scfg_dir($scfg);
+        }
+    };
+
+    eval { $teardown_storage->($target); };
+    if (my $err = $@) {
+        confess "failed to tear down target storage: $err";
+    }
+
+    eval { $teardown_storage->($source); };
+    if (my $err = $@) {
+        confess "failed to tear down source storage: $err";
+    }
+
+    return;
+}
+
+my sub run_test($test_def) {
+    my ($desc, $cases) = $test_def->@{qw(description cases)};
+
+    eval { setup_test_env($test_def); };
+    if (my $err = $@) {
+        teardown_test_env($test_def);
+
+        fail($desc);
+        diag("Failed to set up test environment: $err");
+    }
+
+    my $source_storeid = $test_def->{source}->{storeid};
+    my $target_storeid = $test_def->{target}->{storeid};
+
+    for my $case ($cases->@*) {
+        my ($volname, $vmid) = $case->@{qw(volname vmid)};
+
+        confess "($desc) case cannot have both 'expect-match' and 'expect-fail'"
+            if defined($case->{'expect-match'}) && defined($case->{'expect-fail'});
+
+        my $source_volid = $source_storeid . ':' . $volname;
+
+        my $case_desc = "$desc | $vmid | $volname";
+
+        my $result = eval {
+            return PVE::GuestImport::extract_disk_from_import_file(
+                $source_volid, $vmid, $target_storeid,
+            );
+        };
+        if (my $err = $@) {
+            my $re_expect_fail = $case->{'expect-fail'};
+            $re_expect_fail = qr/^\Q$re_expect_fail\E$/ if !is_regexp($re_expect_fail);
+
+            if (!ok($err =~ $re_expect_fail, $case_desc)) {
+                diag(explain($err));
+                diag("does not match");
+                diag(explain($case->{'expect-fail'}));
+            }
+
+            next;
+        }
+
+        my $re_expect_match = $case->{'expect-match'};
+        $re_expect_match = qr/^\Q$re_expect_match\E$/ if !is_regexp($re_expect_match);
+
+        if (!ok($result =~ $re_expect_match, $case_desc)) {
+            diag(explain($result));
+            diag("does not match");
+            diag(explain($case->{'expect-match'}));
+        }
+    }
+
+    teardown_test_env($test_def);
+
+    return;
+}
+
+my sub main() {
+    my $sum_cases = 0;
+    for my $test_def ($tests->@*) {
+        $sum_cases += scalar($test_def->{cases}->@*);
+    }
+
+    plan tests => $sum_cases;
+
+    for my $test_def ($tests->@*) {
+        run_test($test_def);
+    }
+
+    done_testing();
+
+    return;
+}
+
+main();
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 27c45b8c..7d3e2ce0 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -20,6 +20,7 @@ my $res = $harness->runtests(
     "archive_info_test.pm",
     "filesystem_path_test.pm",
     "get_subdir_test.pm",
+    "guest_import_test.pl",
     "list_volumes_test.pm",
     "parse_volname_test.pm",
     "path_to_volume_id_test.pm",
--
2.47.3

[PATCH pve-storage v1 22/54] tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0

[Max R. Carrara]

Introduce the `PVE::Storage::Common::Parse` module with the following
subroutines:

* `parse_path_as_volname_parts($path, $vtype)`

  Parses a given file path into several smaller parts that constitute
  a volume name for the given volume type.

* `parse_path_as_volname($path, $vtype)`

  Parses a given file path directly into a volume name.

* `parse_volname_as_parts($volname)`

  Parses an existing volume name into its constituent parts.

* `parse_path_as_volid_parts($storeid, $scfg, $path, $vtype)`

  Parses a given file path into several smaller parts that constitute
  a volume ID for the given storage ID, its configuration and volume
  type.

* `parse_path_as_volid($storeid, $scfg, $path, $vtype)`

  Like `parse_path_as_volid_parts()`, but parses the given path
  directly into a volume ID.

* `parse_volid_as_parts($volid)`

  Parses an existing volume ID into its storage ID and volume name
  parts.

As of this commit, these parsing functions only support the 'iso'
volume type, with the exception of `parse_volid_as_parts()`, which
does not depend on knowing the volume type.

Using the newly introduced parsing helpers, replace all occurrences of
the `PVE::Storage::ISO_EXT_RE_0` regex across the repository.

Keep the `ISO_EXT_RE_0` regex around for now, since its visibility is
public, which also means that it is part of the storage API (whether
that was intended or not). On an APIVER + APIAGE bump, this regex
should be marked for removal.

Additional Notes Regarding the new Parsers
==========================================

Support for other volume types will be added individually in future
commits.

The new *private* regex used for 'iso' vtype parsing is still matching
file extensions case-insensitively, but also matches the entire path
and file name portions of 'iso' file paths and volume names.

These parts are extracted using named regex groups, as that is much
easier to handle and keep track of mentally, even with smaller
regexes. These named groups are the "constituent parts" that are
returned by some of the new parser subroutines.

However, one important difference here is that named regex groups do
not support dashes `-` in their names, only underscores `_`. To keep
things consistent with our style (using dashes instead of underscores
in hash keys and the API), these named groups are formatted before
being returned—underscores are simply substituted with dashes.

Finally, all parser subroutines check whether a parent directory
reference (`..` or "double dots") is contained in the passed or
extracted file path, and return early if there is. This is an
additional safety measure that is intentionally introduced in this
commit to guard against any mishaps in the future as the parsers gain
more functionality, such as supporting nested directories inside
different volume types' subdirectories.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm  |   9 +-
 src/PVE/Storage.pm              |   7 +-
 src/PVE/Storage/Common.pm       |   2 +
 src/PVE/Storage/Common/Makefile |   1 +
 src/PVE/Storage/Common/Parse.pm | 330 ++++++++++++++++++++++++++++++++
 src/PVE/Storage/Plugin.pm       |  19 +-
 6 files changed, 358 insertions(+), 10 deletions(-)
 create mode 100644 src/PVE/Storage/Common/Parse.pm

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 2b0b121a..cf60c148 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -23,6 +23,9 @@ use PVE::Storage;
 use PVE::Storage::Common qw(
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_path_as_volname_parts
+);

 use base qw(PVE::RESTHandler);

@@ -71,11 +74,13 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     my ($path, $vtype) = @_;

     if ($vtype eq 'iso') {
-        if ($path !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
+        my $parts = parse_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "wrong file extension" });
         }

-        my $ext = $1;
+        my $ext = $parts->{ext};
         return $ext;
     }

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 1b0569dd..d7d683ad 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -26,6 +26,9 @@ use PVE::Storage::Common qw(
     plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_path_as_volid
+);
 use PVE::RESTEnvironment qw(log_warn);

 use PVE::Storage::Plugin;
@@ -754,9 +757,7 @@ sub path_to_volume_id {
         }

         if ($vtype eq 'iso') {
-            return if $filename !~ m!/([^/]+$ISO_EXT_RE_0)$!;
-            my $name = $1;
-            return "$sid:iso/$name";
+            return parse_path_as_volid($sid, $scfg, $path, $vtype);
         }

         if ($vtype eq 'vztmpl') {
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index abccf52b..49defb98 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -55,6 +55,8 @@ be grouped in a submodule can also be found here.

 =over

+=item * C<L<PVE::Storage::Common::Parse>>
+
 =back

 =head1 STANDARD OPTIONS FOR JSON SCHEMA
diff --git a/src/PVE/Storage/Common/Makefile b/src/PVE/Storage/Common/Makefile
index 0c4bba5b..0d9b1be1 100644
--- a/src/PVE/Storage/Common/Makefile
+++ b/src/PVE/Storage/Common/Makefile
@@ -1,4 +1,5 @@
 SOURCES = \
+	  Parse.pm			\


 .PHONY: install
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
new file mode 100644
index 00000000..bcc6f9fc
--- /dev/null
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -0,0 +1,330 @@
+package PVE::Storage::Common::Parse;
+
+use v5.36;
+
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
+
+use Exporter qw(import);
+
+our @EXPORT_OK = qw(
+    parse_path_as_volname_parts
+    parse_path_as_volname
+    parse_volname_as_parts
+
+    parse_path_as_volid_parts
+    parse_path_as_volid
+    parse_volid_as_parts
+);
+
+=head1 NAME
+
+C<PVE::Storage::Common::Parse> - Storage-related Parsing Functions
+
+=head1 DESCRIPTION
+
+This module contains various parsing functions for use within C<L<PVE::Storage>>,
+its submodules (including storage plugins) and other related modules.
+
+Parsing functions are categorized by their main purpose / area of application
+and may be further subdivided depending on what kind of data type they are
+primarily related to.
+
+=cut
+
+my $RE_PARENT_DIR = quotemeta('..');
+my $RE_CONTAINS_PARENT_DIR = qr!
+    ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
+    |
+    ( /$RE_PARENT_DIR/ )  # /../ --> Between two path components
+    |
+    ( /$RE_PARENT_DIR$ )  # /..  --> End of path
+!xn;
+
+my $RE_ISO_FILE_PATH = qr!
+    (?<path>
+        (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
+    )
+!xn;
+
+my $RE_FILE_PATH_FOR_VTYPE = {
+    iso => qr/^$RE_ISO_FILE_PATH$/,
+};
+
+my $RE_VOLNAME_FOR_VTYPE = {
+    iso => qr/^$RE_ISO_FILE_PATH$/,
+};
+
+my sub contains_parent_dir($path) {
+    return $path =~ $RE_CONTAINS_PARENT_DIR;
+}
+
+my sub strip_leading_path_separators($path) {
+    return $path =~ s!^/+!!r;
+}
+
+my sub strip_trailing_path_separators($path) {
+    return $path =~ s!/+$!!r;
+}
+
+my sub format_named_groups(%groups) {
+    my $result = {};
+
+    for my $old_key (keys %groups) {
+        my $new_key = $old_key =~ s/_/-/gr;
+        $result->{$new_key} = $groups{$old_key};
+    }
+
+    my @disk_path_components = ();
+
+    if (defined($result->{file})) {
+        $result->{file} = strip_leading_path_separators($result->{file});
+        push(@disk_path_components, $result->{file});
+    }
+
+    if (scalar(@disk_path_components)) {
+        $result->{'disk-path'} = join('/', @disk_path_components);
+    }
+
+    return $result;
+}
+
+my sub split_leading_dir_from_path($path, $directory) {
+    $directory = strip_trailing_path_separators($directory);
+
+    if ($path =~ m!^(?<leading_dir> \Q$directory\E ) / (?<remainder> .* ) $ !xn) {
+        return {
+            'leading-dir' => strip_trailing_path_separators($+{leading_dir}),
+            remainder => strip_leading_path_separators($+{remainder}),
+        };
+    }
+
+    return;
+}
+
+=head1 PARSERS RELATED TO VOLUMES
+
+The parsing functions in this section primarily deal with parsing data related
+to storage volumes, primarily C<L<volname>>s and C<L<volid>>s.
+
+=head2 VOLUME NAMES
+
+=cut
+
+=head3 parse_path_as_volname_parts
+
+Parses the given relative file path C<$path> according to the given C<$vtype>
+and returns a hashref containing the individually extracted parts that make up
+a C<volname> on success.
+
+On failure or when the provided C<$vtype> is not supported or does not exist,
+returns C<undef> instead.
+
+B<NOTE:> This function assumes that C<$path> is already relative to the
+directory that corresponds to the given C<$vtype>.
+
+For example, the path C</var/lib/vz/template/iso/custom-debian.iso> points to an
+ISO file on the default C<local> directory storage. In this case, the storage
+path is C</var/lib/vz>, the subdirectory for the C<$vtype = "iso"> is
+C<template/iso>, and C<$path> is C<custom-debian.iso>. To illustrate:
+
+    /var/lib/vz/template/iso/custom-debian.iso
+    └┬────────┘ └┬─────────┘ └┬──────────────┘
+     │           │            │
+     │           │           $path
+     │          subdir for
+     │          $vtype = "iso"
+     │
+     storage path
+
+If you want to parse an absolute path for an already known storage instead, see
+C<L<< parse_path_as_volid_parts()|/"parse_path_as_volid_parts" >>>.
+
+For a counterpart to this function, see
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>>.
+
+=cut
+
+sub parse_path_as_volname_parts : prototype($$) ($path, $vtype) {
+    return if contains_parent_dir($path);
+
+    # TODO: vtype split: Handle parsing for 'images' and 'rootdir' vtypes.
+
+    my $re_filepath = $RE_FILE_PATH_FOR_VTYPE->{$vtype};
+    return if !defined($re_filepath);
+
+    return if $path !~ $re_filepath;
+
+    my $parts = format_named_groups(%+);
+    $parts->{vtype} = $vtype;
+    $parts->{volname} = $vtype . '/' . $parts->{path};
+
+    return $parts;
+}
+
+=head3 parse_path_as_volname
+
+Like C<L<< parse_path_as_volname_parts()|/"parse_path_as_volname_parts" >>>, but
+instead of extracting the individual parts of the relative C<$path>, returns
+the correctly formatted C<volname> directly.
+
+For a counterpart to this function, see
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>>.
+
+=cut
+
+sub parse_path_as_volname : prototype($$) ($path, $vtype) {
+    return if contains_parent_dir($path);
+
+    # TODO: vtype split: Handle parsing for 'images' and 'rootdir' vtypes.
+
+    my $re_filepath = $RE_FILE_PATH_FOR_VTYPE->{$vtype};
+    return if !defined($re_filepath);
+
+    return if $path !~ $re_filepath;
+
+    return $vtype . '/' . $+{path};
+}
+
+=head3 parse_volname_as_parts
+
+Parses the provided C<$volname> and returns its constituent parts in a hashref
+upon success.
+
+Returns C<undef> if C<$volname> is prefixed with an unknown or unsupported
+C<vtype>, or if the path after the C<vtype> prefix cannot be parsed.
+
+This function can be seen as a counterpart to
+C<L<< parse_path_as_volname_parts()|/"parse_path_as_volname_parts" >>> and
+C<L<< parse_path_as_volname()|/"parse_path_as_volname" >>> and can be used to
+extract the information embedded within an already existing volume name, such
+as file extensions or the name of the file that a volume refers to.
+
+=cut
+
+sub parse_volname_as_parts : prototype($) ($volname) {
+    # TODO: vtype split: Handle volname for 'images' and 'rootdir' vtypes.
+    my ($vtype, $path) = split('/', $volname, 2);
+
+    # Either variable could be undef or an empty string here
+    return if !$vtype || !$path;
+
+    return if contains_parent_dir($path);
+
+    my $re_volname = $RE_VOLNAME_FOR_VTYPE->{$vtype};
+    return if !defined($re_volname);
+
+    return if $path !~ $re_volname;
+
+    my $parts = format_named_groups(%+);
+    $parts->{vtype} = $vtype;
+    $parts->{volname} = $volname;
+
+    return $parts;
+}
+
+=head2 VOLUME IDS
+
+=cut
+
+my $RE_VOLID = qr!
+    ^
+    (?<volid>
+        (?<storeid> (?i: [a-z][a-z0-9\-\_\.]*[a-z0-9] ) )
+        :    # separated by colon
+        (?<volname> .+)
+    )
+    $
+!xn;
+
+=head3 parse_path_as_volid_parts
+
+Parses the given absolute file path C<$path> according to the given C<$vtype>
+and returns a hashref containing the individually extracted parts that make up
+a C<$volid>, but only if C<$path> is valid for C<$storeid> and its config
+C<$scfg>.
+
+On failure or when the provided C<$vtype> is not supported or does not exist,
+returns C<undef> instead.
+
+B<NOTE:> Unlike C<L<< parse_path_as_volname_parts()|/"parse_path_as_volname_parts" >>>,
+this function requires that C<$path> is prefixed with the directory for the
+given C<$vtype>. To illustrate:
+
+    directory for $vtype = "iso"
+     │
+     │
+    ┌┴─────────────────────┐
+    │                      │
+    /var/lib/vz/template/iso/custom-debian.iso
+    │                                        │
+    └┬───────────────────────────────────────┘
+     │
+     │
+    $path
+
+If you are only interested in parsing the portion of the path belonging inside
+the C<vtype> directory independent of a C<$storeid>, see
+C<L<< parse_path_as_volname_parts()|/"parse_path_as_volname_parts" >>>.
+
+For a counterpart to this function, see
+C<L<< parse_volid_as_parts()|/"parse_volid_as_parts" >>>.
+
+=cut
+
+sub parse_path_as_volid_parts : prototype($$$$) ($storeid, $scfg, $path, $vtype) {
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    my $split_path_parts = split_leading_dir_from_path($path, $vtype_subdir);
+    return if !defined($split_path_parts);
+
+    my $volid_parts = parse_path_as_volname_parts($split_path_parts->{remainder}, $vtype);
+    return if !defined($volid_parts);
+
+    $volid_parts->{volid} = $storeid . ':' . $volid_parts->{volname};
+
+    return $volid_parts;
+}
+
+=head3 parse_path_as_volid
+
+Like C<L<< parse_path_as_volid_parts()|/"parse_path_as_volid_parts" >>>, but
+instead of extracting the individual parts of the C<$path>, returns the
+correctly formatted C<volid> directly.
+
+For a counterpart to this function, see
+C<L<< parse_volid_as_parts()|/"parse_volid_as_parts" >>>.
+
+=cut
+
+sub parse_path_as_volid : prototype($$$$) ($storeid, $scfg, $path, $vtype) {
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    my $split_path_parts = split_leading_dir_from_path($path, $vtype_subdir);
+    return if !defined($split_path_parts);
+
+    my $volname = parse_path_as_volname($split_path_parts->{remainder}, $vtype);
+    return if !defined($volname);
+
+    return $storeid . ':' . $volname;
+}
+
+=head3 parse_volid_as_parts
+
+Parses the provided C<$volid> and returns its C<storeid> and C<volname> parts
+in a hashref upon success and C<undef> otherwise.
+
+The C<volname> part of volumes that represent files can be further parsed into
+its constituent parts using the
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>> function.
+
+=cut
+
+sub parse_volid_as_parts : prototype($) ($volid) {
+    return if $volid !~ $RE_VOLID;
+
+    return format_named_groups(%+);
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index c96c1ed7..590ba3e0 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -19,6 +19,10 @@ use PVE::Storage::Common qw(
     plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_volname_as_parts
+    parse_path_as_volid_parts
+);

 use JSON;

@@ -816,8 +820,12 @@ sub parse_volname {
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
     }

-    if ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
-        return ('iso', $1, undef, undef, undef, undef, 'raw');
+    if (defined(my $parts = parse_volname_as_parts($volname))) {
+        my ($vtype, $volume_path) = $parts->@{qw(vtype path)};
+
+        if ($vtype eq 'iso') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }

     if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
@@ -1690,11 +1698,12 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'iso') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);

             return {
-                volid => "$storeid:iso/$1",
-                format => 'iso',
+                volid => $parts->{volid},
+                format => 'iso', # always 'iso' even if we have a file ending in .img
             };
         }

--
2.47.3

[PATCH pve-storage v1 23/54] common: test: set up parser testing code, add tests for 'iso' vtype

[Max R. Carrara]

Set up the necessary scaffolding for testing the new parser functions
in `PVE::Storage::Common::Parse` and add tests for the 'iso' vtype.

This also adds `libtest-differences-perl` to `Build-Depends`.

Since the volid parsing functions are built on top of the volname
parsing functions, their tests are built in a similar way; that is,
the volname parser test cases are taken and adapted for the volid
parsers.

This is primarily done to ensure consistency between the two
"families" of parsing functions; for example, the
`parse_path_as_volid_parts()` function should always return the same
hash as the `parse_path_as_volname_parts()` function, except that it
also contains a 'volid' key.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 debian/control                              |   1 +
 src/PVE/Makefile                            |   1 +
 src/PVE/Storage/Common/Makefile             |   4 +
 src/PVE/Storage/Common/test/Makefile        |   6 +
 src/PVE/Storage/Common/test/parser_tests.pl | 298 ++++++++++++++++++++
 src/PVE/Storage/Common/test/run_tests.pl    |  25 ++
 src/PVE/Storage/Makefile                    |   4 +
 7 files changed, 339 insertions(+)
 create mode 100644 src/PVE/Storage/Common/test/Makefile
 create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
 create mode 100755 src/PVE/Storage/Common/test/run_tests.pl

diff --git a/debian/control b/debian/control
index 0e0593af..ab30df06 100644
--- a/debian/control
+++ b/debian/control
@@ -10,6 +10,7 @@ Build-Depends: debhelper-compat (= 13),
                libpve-common-perl (>= 9.1.10),
                librados2-perl,
                libtest-mockmodule-perl,
+               libtest-differences-perl,
                libxml-libxml-perl,
                lintian,
                perl,
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 9e9f6aac..b8fc6325 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -14,6 +14,7 @@ install:

 .PHONY: test
 test:
+	$(MAKE) -C Storage test
 	$(MAKE) -C test test

 clean:
diff --git a/src/PVE/Storage/Common/Makefile b/src/PVE/Storage/Common/Makefile
index 0d9b1be1..7a13371d 100644
--- a/src/PVE/Storage/Common/Makefile
+++ b/src/PVE/Storage/Common/Makefile
@@ -5,3 +5,7 @@ SOURCES = \
 .PHONY: install
 install:
 	for i in ${SOURCES}; do install -D -m 0644 $$i ${DESTDIR}${PERLDIR}/PVE/Storage/Common/$$i; done
+
+.PHONY: test
+test:
+	make -C test test
diff --git a/src/PVE/Storage/Common/test/Makefile b/src/PVE/Storage/Common/test/Makefile
new file mode 100644
index 00000000..c4b2c0db
--- /dev/null
+++ b/src/PVE/Storage/Common/test/Makefile
@@ -0,0 +1,6 @@
+all: test
+
+.PHONY: test
+test: run_tests.pl
+	./run_tests.pl
+
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
new file mode 100755
index 00000000..0167ca74
--- /dev/null
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -0,0 +1,298 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use lib q(../../../..);
+
+use File::Temp;
+use Storable qw(dclone);
+
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
+use PVE::Storage::Common::Parse qw(
+    parse_path_as_volname_parts
+    parse_path_as_volname
+    parse_volname_as_parts
+
+    parse_path_as_volid_parts
+    parse_path_as_volid
+    parse_volid_as_parts
+);
+
+use Test::More qw(no_plan);
+use Test::Differences;
+
+my $volname_tests = [];
+
+my $volname_cases_iso_valid = [
+    # plain paths
+    {
+        path => 'custom-debian.iso',
+        expected => {
+            file => 'custom-debian.iso',
+            ext => 'iso',
+            'disk-path' => 'custom-debian.iso',
+            path => 'custom-debian.iso',
+            vtype => 'iso',
+            volname => 'iso/custom-debian.iso',
+        },
+    },
+    {
+        path => 'archlinux.img',
+        expected => {
+            file => 'archlinux.img',
+            ext => 'img',
+            'disk-path' => 'archlinux.img',
+            path => 'archlinux.img',
+            vtype => 'iso',
+            volname => 'iso/archlinux.img',
+        },
+    },
+
+    # case insensitive file extensions
+    {
+        path => 'Debian 13 Trixie.ISO',
+        expected => {
+            file => 'Debian 13 Trixie.ISO',
+            ext => 'ISO',
+            'disk-path' => 'Debian 13 Trixie.ISO',
+            path => 'Debian 13 Trixie.ISO',
+            vtype => 'iso',
+            volname => 'iso/Debian 13 Trixie.ISO',
+        },
+    },
+    {
+        path => 'Fedora.Img',
+        expected => {
+            file => 'Fedora.Img',
+            ext => 'Img',
+            'disk-path' => 'Fedora.Img',
+            path => 'Fedora.Img',
+            vtype => 'iso',
+            volname => 'iso/Fedora.Img',
+        },
+    },
+];
+
+my $volname_cases_iso_invalid = [
+    {
+        description => "Invalid file extension (iso)",
+        args => {
+            path => 'MacOS-kittycat.dmg',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+];
+
+my $cases_valid_all = [
+    $volname_cases_iso_valid,
+];
+
+my $cases_invalid_all = [
+    $volname_cases_iso_invalid,
+];
+
+{
+    for my $cases_list ($cases_valid_all->@*) {
+        for my $case ($cases_list->@*) {
+            my $path = $case->{path};
+            my $vtype = $case->{expected}->{vtype};
+
+            push(
+                $volname_tests->@*,
+                {
+                    description => "Valid path ($vtype) \"$path\"",
+                    args => {
+                        path => $path,
+                        vtype => $vtype,
+                    },
+                    expected => $case->{expected},
+                },
+            );
+        }
+    }
+
+    for my $cases_list ($cases_invalid_all->@*) {
+        push($volname_tests->@*, $cases_list->@*);
+    }
+}
+
+my sub run_volname_parsing_tests : prototype($) ($volname_tests) {
+    for my $case ($volname_tests->@*) {
+        subtest $case->{description} => sub() {
+            my ($path, $vtype) = $case->{args}->@{qw(path vtype)};
+
+            my $got_volname_parts = parse_path_as_volname_parts($path, $vtype);
+            my $got_volname = parse_path_as_volname($path, $vtype);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname_parts,
+                    $case->{expected},
+                    'parse_path_as_volname_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+
+                is(
+                    $got_volname,
+                    $case->{expected}->{volname},
+                    'parse_path_as_volname() returns expected volname',
+                );
+
+                my $got_volname_parts_from_volname = parse_volname_as_parts($got_volname);
+
+                eq_or_diff(
+                    $got_volname_parts_from_volname,
+                    $case->{expected},
+                    'parse_volname_as_parts() returns expected hashref from parsed volname',
+                );
+            } else {
+                is($got_volname_parts, undef, 'parse_path_as_volname_parts() returns undef');
+                is($got_volname, undef, 'parse_path_as_volname() returns undef');
+            }
+
+        };
+    }
+
+    return;
+}
+
+my $DEFAULT_STOREID = 'local';
+my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
+my $DEFAULT_SCFG = {
+    type => 'dir',
+    path => $DEFAULT_STORAGE_PATH,
+    shared => 0,
+    content => {
+        iso => 1,
+        rootdir => 1,
+        vztmpl => 1,
+        images => 1,
+        snippets => 1,
+        backup => 1,
+        import => 1,
+    },
+};
+
+my $ALT_STOREID = 'dir-store';
+my $ALT_STORAGE_PATH = File::Temp->newdir();
+my $ALT_SCFG = {
+    type => 'dir',
+    path => $ALT_STORAGE_PATH,
+    shared => 0,
+    content => { $DEFAULT_SCFG->{content}->%* },
+    'content-dirs' => {
+        iso => 'pve/volumes/iso',
+        vztmpl => 'pve/dumps/vz',
+        images => 'pve/guest/images',
+        snippets => 'pve/misc/snippets',
+        backup => 'pve/dumps/bak',
+        import => 'archive/import',
+    },
+};
+
+my sub format_volname_case_to_volid_case($storeid, $scfg, $volname_case) {
+    my $volid_case = dclone($volname_case);
+
+    my ($path, $vtype) = $volid_case->{args}->@{qw(path vtype)};
+
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    $volid_case->{args}->{path} = $vtype_subdir . '/' . $path;
+    $volid_case->{args}->{storeid} = $storeid;
+    $volid_case->{args}->{scfg} = $scfg;
+
+    if (defined($volid_case->{expected})) {
+        my $volname = $volid_case->{expected}->{volname};
+        $volid_case->{expected}->{volid} = $storeid . ':' . $volname;
+    }
+
+    return $volid_case;
+}
+
+my $volid_tests = [
+    {
+        description => "volid parsers build on volname parsers' behaviors (1)",
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $volname_tests->@*
+        ],
+    },
+    {
+        description => "volid parsers build on volname parsers' behaviors (2)",
+        storeid => $ALT_STOREID,
+        scfg => $ALT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($ALT_STOREID, $ALT_SCFG, $_) }
+                $volname_tests->@*
+        ],
+    },
+];
+
+my sub run_volid_parsing_tests : prototype($) ($volid_tests) {
+    for my $test ($volid_tests->@*) {
+        subtest $test->{description} => sub() {
+            for my $case ($test->{cases}->@*) {
+                my ($storeid, $scfg, $path, $vtype) =
+                    $case->{args}->@{qw(storeid scfg path vtype)};
+
+                my $got_volid_parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+                my $got_volid = parse_path_as_volid($storeid, $scfg, $path, $vtype);
+
+                note("Running test case based on:");
+                note($case->{description});
+
+                if (defined($case->{expected})) {
+                    eq_or_diff(
+                        $got_volid_parts,
+                        $case->{expected},
+                        'parse_path_as_volid_parts() returns expected hashref',
+                        { context => 50000 },
+                    );
+
+                    is(
+                        $got_volid,
+                        $case->{expected}->{volid},
+                        'parse_path_as_volid() returns expected volid',
+                    );
+
+                    my $expected_volid_parts = {
+                        volid => $case->{expected}->{volid},
+                        storeid => $case->{args}->{storeid},
+                        volname => $case->{expected}->{volname},
+                    };
+
+                    my $got_volid_parts_from_volid = parse_volid_as_parts($got_volid);
+
+                    eq_or_diff(
+                        $got_volid_parts_from_volid,
+                        $expected_volid_parts,
+                        'parse_volid_as_parts() returns expected hashref from parsed volid',
+                    );
+                } else {
+                    is($got_volid_parts, undef, 'parse_path_as_volid_parts() returns undef');
+                    is($got_volid, undef, 'parse_path_as_volid() returns undef');
+                }
+            }
+        };
+    }
+
+    return;
+}
+
+my sub main() {
+    unified_diff();
+
+    run_volname_parsing_tests($volname_tests);
+    run_volid_parsing_tests($volid_tests);
+
+    done_testing();
+
+    return;
+}
+
+main();
diff --git a/src/PVE/Storage/Common/test/run_tests.pl b/src/PVE/Storage/Common/test/run_tests.pl
new file mode 100755
index 00000000..ed8b1164
--- /dev/null
+++ b/src/PVE/Storage/Common/test/run_tests.pl
@@ -0,0 +1,25 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use TAP::Harness;
+
+my $MAX_JOBS = 4;
+
+my $nproc = eval { int(`nproc`) } || $MAX_JOBS;
+
+my $jobs = $nproc > $MAX_JOBS ? $MAX_JOBS : $nproc;
+
+my sub main() {
+    my $harness = TAP::Harness->new({ verbosity => -1, jobs => $jobs });
+
+    my $res = $harness->runtests(
+        "parser_tests.pl",
+    );
+
+    exit -1 if !$res || $res->{failed} || $res->{parse_errors};
+
+    return;
+}
+
+main();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a67dc25f..1c8d364b 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -21,3 +21,7 @@ install:
 	make -C Common install
 	for i in ${SOURCES}; do install -D -m 0644 $$i ${DESTDIR}${PERLDIR}/PVE/Storage/$$i; done
 	make -C LunCmd install
+
+.PHONY: test
+test:
+	make -C Common test
--
2.47.3

[PATCH pve-storage v1 24/54] tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions

[Max R. Carrara]

Add support for the 'vztmpl' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace usages of the
`PVE::Storage::VZTMPL_EXT_RE_1` regex across the repository.

As done with the `ISO_EXT_RE_0` regex, keep the `VZTMPL_EXT_RE_1`
regex around for now, since it's public and therefore also part of the
storage API. On an APIVER + APIAGE bump, this regex should be marked
for removal.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm              |   6 +-
 src/PVE/Storage.pm                          |   4 +-
 src/PVE/Storage/Common/Parse.pm             |  23 +++++
 src/PVE/Storage/Common/test/parser_tests.pl | 109 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  16 +--
 5 files changed, 147 insertions(+), 11 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index cf60c148..03929d26 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -85,11 +85,13 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     }

     if ($vtype eq 'vztmpl') {
-        if ($path !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
+        my $parts = parse_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "wrong file extension" });
         }

-        my $ext = $1;
+        my $ext = $parts->{ext};
         return $ext;
     }

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index d7d683ad..d5b0b637 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -761,9 +761,7 @@ sub path_to_volume_id {
         }

         if ($vtype eq 'vztmpl') {
-            return if $filename !~ m!/([^/]+$VZTMPL_EXT_RE_1)$!;
-            my $name = $1;
-            return "$sid:vztmpl/$name";
+            return parse_path_as_volid($sid, $scfg, $path, $vtype);
         }

         if ($vtype eq 'backup') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index bcc6f9fc..b18ef576 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -33,6 +33,14 @@ primarily related to.

 =cut

+my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');
+
+my sub join_to_re_alternations(@list) {
+    return join('|', map { quotemeta } @list);
+}
+
+my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSION_EXTENSIONS);
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -48,12 +56,27 @@ my $RE_ISO_FILE_PATH = qr!
     )
 !xn;

+my $RE_VZTMPL_FILE_PATH = qr!
+    (?<path>
+        (?<file>
+            [^/]+
+            \.
+            (?<ext>
+                (?<ext_archive> (?i: tar) )
+                ( \. (?<ext_compression> (?i: $RE_VZTMPL_COMPRESSION_EXTENSIONS)) )?
+            )
+        )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
+    vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
 };

 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
+    vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
 };

 my sub contains_parent_dir($path) {
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 0167ca74..31575b66 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -86,12 +86,121 @@ my $volname_cases_iso_invalid = [
     },
 ];

+my $volname_cases_vztmpl_valid = [
+    # plain paths
+    {
+        path => 'archlinux-base_20190924-1_amd64.tar',
+        expected => {
+            file => 'archlinux-base_20190924-1_amd64.tar',
+            ext => 'tar',
+            'ext-archive' => 'tar',
+            'disk-path' => 'archlinux-base_20190924-1_amd64.tar',
+            path => 'archlinux-base_20190924-1_amd64.tar',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar',
+        },
+    },
+    {
+        path => 'archlinux-base_20190924-1_amd64.tar.gz',
+        expected => {
+            file => 'archlinux-base_20190924-1_amd64.tar.gz',
+            ext => 'tar.gz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'gz',
+            'disk-path' => 'archlinux-base_20190924-1_amd64.tar.gz',
+            path => 'archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
+        },
+    },
+    {
+        path => 'alpine-3.10-default_20190626_amd64.tar.xz',
+        expected => {
+            file => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            ext => 'tar.xz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'xz',
+            'disk-path' => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            path => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
+        },
+    },
+    {
+        path => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+        expected => {
+            file => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            'disk-path' => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            path => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        },
+    },
+    {
+        path => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        expected => {
+            file => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            'disk-path' => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            path => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        },
+    },
+
+    # case-insensitive file extensions
+    {
+        path => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+        expected => {
+            file => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            ext => 'TAR.GZ',
+            'ext-archive' => 'TAR',
+            'ext-compression' => 'GZ',
+            'disk-path' => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            path => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+        },
+    },
+    {
+        path => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+        expected => {
+            file => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            ext => 'Tar.xZ',
+            'ext-archive' => 'Tar',
+            'ext-compression' => 'xZ',
+            'disk-path' => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            path => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/Alpine 3.10 default_20190626_amd64.Tar.xZ',
+        },
+    },
+];
+
+my $volname_cases_vztmpl_invalid = [
+    {
+        description => "Invalid file extension (vztmpl)",
+        args => {
+            path => 'archlinux-base_20190924-1_amd64.zip',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
+    $volname_cases_vztmpl_valid,
 ];

 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
+    $volname_cases_vztmpl_invalid,
 ];

 {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 590ba3e0..6bfa5c11 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -826,10 +826,10 @@ sub parse_volname {
         if ($vtype eq 'iso') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }

-    if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
-        return ('vztmpl', $1, undef, undef, undef, undef, 'raw');
+        if ($vtype eq 'vztmpl') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }

     if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
@@ -1708,11 +1708,15 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'vztmpl') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
+
+            my ($ext, $ext_compression) = $parts->@{qw(ext ext-compression)};
+            my $format = $ext eq 'tar' ? $ext : ('t' . $ext_compression);

             return {
-                volid => "$storeid:vztmpl/$1",
-                format => $2 eq 'tar' ? $2 : "t$2",
+                volid => $parts->{volid},
+                format => $format,
             };
         }

--
2.47.3

[PATCH pve-storage v1 25/54] tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions

[Max R. Carrara]

Add support for the 'backup' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace usages of the
`PVE::Storage::BACKUP_EXT_RE_2` regex across the repository.

Note that the new regex for parsing 'backup' vtype file paths is also
able to optionally extract the VMID and the type of guest from the
file name. This makes it possible to further simplify some parts of
the original logic.

As done with the `ISO_EXT_RE_0` and `VZTMPL_EXT_RE_1` regexes, keep
the `BACKUP_EXT_RE_2` regex around for now, since it's public and
therefore also part of the storage API. On an APIVER + APIAGE bump,
this regex should be marked for removal as well, like the others.

Additionally, the `COMPRESSOR_RE` constant in `PVE::Storage::Plugin`
is now unused. Together with its related `KNOWN_COMPRESSION_FORMATS`
constant, it should be phased out and replaced eventually. Add a TODO
comment for that.

In the meantime, keep track of the compression formats for specific
vtypes in the `::Common::Parse` module until we move them to a more
adequate place.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm                          |   4 +-
 src/PVE/Storage/Common/Parse.pm             |  39 +++++
 src/PVE/Storage/Common/test/parser_tests.pl | 181 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  37 ++--
 4 files changed, 240 insertions(+), 21 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index d5b0b637..ce6d2154 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -765,9 +765,7 @@ sub path_to_volume_id {
         }

         if ($vtype eq 'backup') {
-            return if $filename !~ m!/([^/]+$BACKUP_EXT_RE_2)$!;
-            my $name = $1;
-            return "$sid:backup/$name";
+            return parse_path_as_volid($sid, $scfg, $path, $vtype);
         }

         if ($vtype eq 'snippets') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index b18ef576..950f6b18 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -35,12 +35,16 @@ primarily related to.

 my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');

+my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }

 my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSION_EXTENSIONS);

+my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSION_EXTENSIONS);
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -50,6 +54,11 @@ my $RE_CONTAINS_PARENT_DIR = qr!
     ( /$RE_PARENT_DIR$ )  # /..  --> End of path
 !xn;

+# A vmid is an integer between 100 and 999_999_999
+# See: https://git.proxmox.com/?p=pve-common.git;a=blob;f=src/PVE/JSONSchema.pm;h=17e7126a6c42f8326a1da890680af10b6106d906;hb=refs/heads/master#l62
+# See also: https://git.proxmox.com/?p=pve-common.git;a=blob;f=src/PVE/JSONSchema.pm;h=17e7126a6c42f8326a1da890680af10b6106d906;hb=refs/heads/master#l304
+my $RE_VMID = qr![1-9][0-9]{2,8}!;
+
 my $RE_ISO_FILE_PATH = qr!
     (?<path>
         (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
@@ -69,14 +78,44 @@ my $RE_VZTMPL_FILE_PATH = qr!
     )
 !xn;

+my $RE_GUEST_VZDUMP_BACKUP_FILE_NAME = qr!
+    vzdump
+    - (?<type> openvz|lxc|qemu)
+    - (?<vmid> $RE_VMID)
+    - [^/]+
+!xn;
+
+my $RE_BACKUP_FILE_PATH = qr!
+    (?<path>
+        (?<file>
+            (
+                ($RE_GUEST_VZDUMP_BACKUP_FILE_NAME)
+                |
+                ([^/]+)
+            )
+            \.
+            (?<ext>
+                (?<ext_archive> tgz)
+                |
+                (
+                    (?<ext_archive> tar|vma)
+                    ( \. (?<ext_compression> $RE_BACKUP_COMPRESSION_EXTENSIONS) )?
+                )
+            )
+        )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
+    backup => qr/^$RE_BACKUP_FILE_PATH$/,
 };

 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
+    backup => qr/^$RE_BACKUP_FILE_PATH$/,
 };

 my sub contains_parent_dir($path) {
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 31575b66..96159608 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -193,14 +193,195 @@ my $volname_cases_vztmpl_invalid = [
     },
 ];

+my $volname_cases_backup_valid = [
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            ext => 'vma',
+            'ext-archive' => 'vma',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            ext => 'vma.gz',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'gz',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            path => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            ext => 'vma.lzo',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'lzo',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            ext => 'vma.zst',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'zst',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            ext => 'tar.lzo',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'lzo',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            path => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            ext => 'tar.gz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'gz',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            ext => 'tgz',
+            'ext-archive' => 'tgz',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            path => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+        },
+    },
+    {
+        path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+        expected => {
+            type => 'openvz',
+            vmid => '16112',
+            file => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            'disk-path' => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            vtype => 'backup',
+            volname => 'backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+        },
+    },
+];
+
+my $volname_cases_backup_invalid = [
+    {
+        description => "Invalid file extension (1) (backup)",
+        args => {
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.zip',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Invalid file extension (2) (backup)",
+        args => {
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Invalid file extension (3) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (1) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.Tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (2) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.tar.Gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
+    $volname_cases_backup_valid,
 ];

 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
+    $volname_cases_backup_invalid,
 ];

 {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 6bfa5c11..4d63e0a5 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -28,6 +28,7 @@ use JSON;

 use base qw(PVE::SectionConfig);

+# TODO: Phase out these two constants
 use constant KNOWN_COMPRESSION_FORMATS => ('gz', 'lzo', 'zst', 'bz2');
 use constant COMPRESSOR_RE => join('|', KNOWN_COMPRESSION_FORMATS);

@@ -830,14 +831,10 @@ sub parse_volname {
         if ($vtype eq 'vztmpl') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }

-    if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
-        my $fn = $1;
-        if ($fn =~ m/^vzdump-(openvz|lxc|qemu)-(\d+)-.+/) {
-            return ('backup', $fn, $2, undef, undef, undef, 'raw');
+        if ($vtype eq 'backup') {
+            return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
         }
-        return ('backup', $fn, undef, undef, undef, undef, 'raw');
     }

     if ($volname =~ m!^snippets/([^/]+)$!) {
@@ -1721,37 +1718,41 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'backup') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);

-            my $original = $path;
-            my $format = $2;
-            $filename = $1;
+            my $format = $parts->{ext};
+            my $volume_path = $parts->{path};

-            # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            return if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
+            # Check if parsed VMID matched provided VMID in order to avoid
+            # false positives (VMID in parent directory name)
+            my $parsed_vmid = $parts->{vmid};
+            if (defined($vmid) && defined($parsed_vmid)) {
+                return if $vmid ne $parsed_vmid;
+            }

             my $info = {
-                volid => "$storeid:backup/$filename",
+                volid => $parts->{volid},
                 format => $format,
             };

-            my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};
+            my $archive_info = eval { PVE::Storage::archive_info($volume_path) } // {};

             $info->{ctime} = $archive_info->{ctime} if defined($archive_info->{ctime});
             $info->{subtype} = $archive_info->{type} // 'unknown';

-            if (defined($vmid) || $filename =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
-                $info->{vmid} = $vmid // $1;
+            if (defined($vmid) || defined($parsed_vmid)) {
+                $info->{vmid} = $vmid // $parsed_vmid;
             }

-            my $notes_filename = $original . NOTES_EXT;
+            my $notes_filename = $path . NOTES_EXT;
             if (-f $notes_filename) {
                 my $notes = PVE::Tools::file_read_firstline($notes_filename);
                 $info->{notes} = eval { decode('UTF-8', $notes, 1) } // $notes
                     if defined($notes);
             }

-            $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
+            $info->{protected} = 1 if -e PVE::Storage::protection_file_path($path);

             return $info;
         }
--
2.47.3

[PATCH pve-storage v1 26/54] tree-wide: replace usages of inline regexes for snippets with parsers

[Max R. Carrara]

Add support for the 'snippets' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace parsing the a snippet
file's name via regexes or the `basename()` function.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm                          |  4 +---
 src/PVE/Storage/Common/Parse.pm             |  8 +++++++
 src/PVE/Storage/Common/test/parser_tests.pl | 25 +++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   | 11 +++++----
 4 files changed, 41 insertions(+), 7 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index ce6d2154..20b7bafa 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -769,9 +769,7 @@ sub path_to_volume_id {
         }

         if ($vtype eq 'snippets') {
-            return if $filename !~ m!/([^/]+)$!;
-            my $name = $1;
-            return "$sid:snippets/$name";
+            return parse_path_as_volid($sid, $scfg, $path, $vtype);
         }

         if ($vtype eq 'import') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 950f6b18..33220758 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -106,16 +106,24 @@ my $RE_BACKUP_FILE_PATH = qr!
     )
 !xn;

+my $RE_SNIPPETS_FILE_PATH = qr!
+    (?<path>
+        (?<file> [^/]+ )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
+    snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
 };

 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
+    snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
 };

 my sub contains_parent_dir($path) {
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 96159608..e0029fc6 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -372,10 +372,35 @@ my $volname_cases_backup_invalid = [
     },
 ];

+my $volname_cases_snippets_valid = [
+    # plain files
+    {
+        path => 'hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            'disk-path' => 'hookscript.pl',
+            path => 'hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/hookscript.pl',
+        },
+    },
+    {
+        path => 'userconfig.yaml',
+        expected => {
+            file => 'userconfig.yaml',
+            'disk-path' => 'userconfig.yaml',
+            path => 'userconfig.yaml',
+            vtype => 'snippets',
+            volname => 'snippets/userconfig.yaml',
+        },
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
     $volname_cases_backup_valid,
+    $volname_cases_snippets_valid,
 ];

 my $cases_invalid_all = [
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 4d63e0a5..b08e038b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -835,10 +835,10 @@ sub parse_volname {
         if ($vtype eq 'backup') {
             return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
         }
-    }

-    if ($volname =~ m!^snippets/([^/]+)$!) {
-        return ('snippets', $1, undef, undef, undef, undef, 'raw');
+        if ($vtype eq 'snippets') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }

     if ($volname =~
@@ -1758,8 +1758,11 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'snippets') {
+            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
+
             return {
-                volid => "$storeid:snippets/" . basename($filename),
+                volid => $parts->{volid},
                 format => 'snippet',
             };
         }
--
2.47.3

[PATCH pve-storage v1 27/54] tree-wide: partially replace usages of regexes for 'import' vtype

[Max R. Carrara]

Add partial support for the 'import' volume type in
`PVE::Storage::Common::Parse`. What remains unsupported right now is
parsing volume names for that type, as those require a little more
care.

Replace most usages of the `PVE::Storage::IMPORT_EXT_RE_1` regex with
parsing functions from `::Common::Parse`.

Replace the one remaining spot where we use the
`PVE::Storage::UPLOAD_IMPORT_EXT_RE_1` regex with a parsing function
as well. This regex should now be phased out in a future APIVER +
APIAGE bump, like the others in previous commits.

Note that the `UPLOAD_IMPORT_EXT_RE_1` regex only exists to handle the
special case of excluding .ovf files from the upload / download_url
API methods. Instead of adding a separate parser (or a flag etc.) to
handle this case, just parse the path of the up-/downloaded file
regardless. Then, raise a parameter exception when the `ovf` file
extension is encountered.

Also add a bunch of test cases that target the `import` volume type
for the `list_volumes()` plugin API method. For all of these tests,
sort the expected and resulting list items by their volume ID in order
to make the output comparison deterministic. Iterate over the declared
content / volume types of the mocked storage config instead of using a
fixed list of vtypes as well.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm  |  13 ++-
 src/PVE/Storage.pm              |   4 +-
 src/PVE/Storage/Common/Parse.pm |  13 +++
 src/PVE/Storage/Plugin.pm       |   9 +-
 src/test/list_volumes_test.pm   | 180 +++++++++++++++++++++++++++++++-
 5 files changed, 203 insertions(+), 16 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 03929d26..b578a1ed 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -96,13 +96,18 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     }

     if ($vtype eq 'import') {
-        if (
-            $path !~ m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-        ) {
+        my $parts = parse_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "invalid filename or wrong extension" });
         }

-        my $ext = $1;
+        my $ext = $parts->{ext};
+
+        if ($ext eq 'ovf') {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
         return $ext;
     }

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 20b7bafa..3a716894 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -773,9 +773,7 @@ sub path_to_volume_id {
         }

         if ($vtype eq 'import') {
-            return if $filename !~ m!/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!;
-            my $name = $1;
-            return "$sid:import/$name";
+            return parse_path_as_volid($sid, $scfg, $path, $vtype);
         }

         return;
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 33220758..bd7cec13 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -37,6 +37,8 @@ my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');

 my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');

+my @IMPORT_EXTENSIONS = ('ova', 'ovf', 'qcow2', 'raw', 'vmdk');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }
@@ -45,6 +47,10 @@ my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSI

 my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSION_EXTENSIONS);

+my $RE_IMPORT_EXTENSIONS = join_to_re_alternations(@IMPORT_EXTENSIONS);
+
+my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -112,11 +118,18 @@ my $RE_SNIPPETS_FILE_PATH = qr!
     )
 !xn;

+my $RE_IMPORT_FILE_PATH = qr!
+    (?<path>
+        (?<file> ($RE_SAFE_CHAR_CLASS)+ \. (?<ext> $RE_IMPORT_EXTENSIONS) )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
     snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
+    import => qr/^$RE_IMPORT_FILE_PATH$/,
 };

 my $RE_VOLNAME_FOR_VTYPE = {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index b08e038b..39e5cb0d 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1768,13 +1768,12 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'import') {
-            return
-                if $filename !~
-                m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;
+            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);

             return {
-                volid => "$storeid:import/$1",
-                format => "$2",
+                volid => $parts->{volid},
+                format => $parts->{ext},
             };
         }

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 08769027..5cb08880 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -72,6 +72,7 @@ my $scfg = {
         'images' => 1,
         'snippets' => 1,
         'backup' => 1,
+        'import' => 1,
     },
 };

@@ -462,6 +463,171 @@ my @tests = (
         ],
         expected => [], # returns empty list
     },
+    {
+        description => 'VMID: none, valid file names for import',
+        vmid => undef,
+        files => [
+            "$storage_dir/import/import.ova",
+            "$storage_dir/import/import.ovf",
+            "$storage_dir/import/some-disk.qcow2",
+            "$storage_dir/import/some-disk.vmdk",
+            "$storage_dir/import/some-raw-disk.raw",
+        ],
+        expected => [
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ovf',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ovf",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-disk.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'vmdk',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-disk.vmdk",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'raw',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-raw-disk.raw",
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching file paths for import',
+        vmid => undef,
+        files => [
+            # Malformed file names
+            "$storage_dir/import/import.ovff",
+            "$storage_dir/import/importova",
+            "$storage_dir/import/import.ov",
+            "$storage_dir/import/diskraw",
+            "$storage_dir/import/diskvmdk",
+            "$storage_dir/import/disk.invalid",
+            "$storage_dir/import/.ova",
+            "$storage_dir/import/.raw",
+            # Trailing whitespace must not be trimmed
+            "$storage_dir/import/import.ova\t",
+            "$storage_dir/import/disk.raw    ",
+            # Whitespace in file name
+            "$storage_dir/import/something I want to import.ova",
+            "$storage_dir/import/ .raw",
+            "$storage_dir/import/ disk .vmdk",
+            "$storage_dir/import/disk .qcow2",
+            "$storage_dir/import/ import.ova",
+            # Unsafe characters in file name
+            "$storage_dir/import/linux🐧-vm.ova",
+            "$storage_dir/import/🐪perl-playground🐪.ova",
+            "$storage_dir/import/fish_<><_<><_<><.ova",
+            $storage_dir . '/import/C:\\\\Windows\\Path.ova',
+            # Content inside .ova files may only be specified as part
+            # of volume names, and may never appear when looked up as
+            # a file path
+            "$storage_dir/import/import.ova/disk.qcow2",
+            "$storage_dir/import/import.ova/disk.raw",
+            "$storage_dir/import/import.ova/disk.vmdk",
+            "$storage_dir/import/import.ova/disk.invalid",
+        ],
+        expected => [], # returns empty list
+    },
+    {
+        description => 'VMID: none, weird but valid file names for import',
+        vmid => undef,
+        files => [
+            "$storage_dir/import/import.ova.ova",
+            "$storage_dir/import/import.ova.ova.ova",
+            "$storage_dir/import/import.ova.ova.ova.ova",
+            "$storage_dir/import/ova.ova",
+            "$storage_dir/import/ova.ovf",
+            "$storage_dir/import/ova.vmdk",
+            "$storage_dir/import/raw.raw.qcow2",
+            "$storage_dir/import/raw.raw.qcow2.import.qcow2",
+            "$storage_dir/import/raw.raw.raw.your-boat.ova",
+        ],
+        expected => [
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ovf',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.ovf",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'vmdk',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.vmdk",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.qcow2.import.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.raw.your-boat.ova",
+            },
+        ],
+    },
 );

 # provide static vmlist for tests
@@ -497,6 +663,10 @@ $mock_fsi->redefine(
     },
 );

+my sub cmp_volinfo_by_volid {
+    return $a->{volid} cmp $b->{volid};
+}
+
 my $plan = scalar @tests;
 plan tests => $plan + 1;

@@ -520,14 +690,14 @@ plan tests => $plan + 1;

 {
     my $sid = 'local';
-    my $types = ['rootdir', 'images', 'vztmpl', 'iso', 'backup', 'snippets'];
+    my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
     my @suffixes = ('qcow2', 'raw', 'vmdk', 'vhdx');

     # run through test cases
     foreach my $tt (@tests) {
         my $vmid = $tt->{vmid};
         my $files = $tt->{files};
-        my $expected = $tt->{expected};
+        my $expected = [sort cmp_volinfo_by_volid $tt->{expected}->@*];
         my $description = $tt->{description};
         my $parent = $tt->{parent};

@@ -550,8 +720,10 @@ plan tests => $plan + 1;
             }
         }

-        my $got;
-        eval { $got = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types) };
+        my $got = eval {
+            my $volume_list = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types);
+            return [sort cmp_volinfo_by_volid $volume_list->@*];
+        };
         $got = $@ if $@;

         is_deeply($got, $expected, $description) || diag(explain($got));
--
2.47.3

[PATCH pve-storage v1 28/54] tree-wide: replace remaining usages of regexes for 'import' vtype

[Max R. Carrara]

Implement the remaining pieces that were missing from fully supporting
the 'import' volume type in `PVE::Storage::Common::Parse`. In
particular, parsing volume names for '.ova' files that refer to
contents inside the '.ova' file is now fully supported.

Also add corresponding test cases to 'parser_tests.pl'.

Replace the remaining usages of the `PVE::Storage::IMPORT_EXT_RE_1`
and `PVE::Storage::OVA_CONTENT_RE_1` with parsing functions from
`::Common::Parse` across the repository.

Note that the logic in `PVE::Storage::Plugin::parse_volname` can now
be simplified a little further, since the `parse_volname_as_parts()`
parsing function also handles extracting the inner and outer file
extensions, amongst other parts.

Since the replaced regexes are now completely unused inside
`PVE::Storage` and family, they should be phased out with a future
APIVER + APIAGE bump, like the regexes in previous commits.

Finally, update the test cases in 'guest_import_test.pl' that match
for specific parsing-related exception messages.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/GuestImport.pm                      |  28 +-
 src/PVE/Storage/Common/Parse.pm             |  45 ++++
 src/PVE/Storage/Common/test/parser_tests.pl | 281 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  20 +-
 src/test/guest_import_test.pl               |   6 +-
 5 files changed, 350 insertions(+), 30 deletions(-)

diff --git a/src/PVE/GuestImport.pm b/src/PVE/GuestImport.pm
index 3d59dcd7..3be84fc9 100644
--- a/src/PVE/GuestImport.pm
+++ b/src/PVE/GuestImport.pm
@@ -6,6 +6,9 @@ use warnings;
 use File::Path;

 use PVE::Storage;
+use PVE::Storage::Common::Parse qw(
+    parse_volname_as_parts
+);
 use PVE::Tools qw(run_command);

 sub extract_disk_from_import_file {
@@ -13,31 +16,26 @@ sub extract_disk_from_import_file {

     my ($source_storeid, $volname) = PVE::Storage::parse_volume_id($volid);
     $target_storeid //= $source_storeid;
-    my $cfg = PVE::Storage::config();

-    my ($vtype, $name, undef, undef, undef, undef, $fmt) =
-        PVE::Storage::parse_volname($cfg, $volid);
+    my $parts = parse_volname_as_parts($volname);
+    die "cannot extract $volid - invalid volname $volname\n"
+        if !defined($parts);
+
+    my ($vtype, $outer_fmt) = $parts->@{qw(vtype ext)};

     die "only files with content type 'import' can be extracted\n"
         if $vtype ne 'import';

     die "only files from 'ova' format can be extracted\n"
-        if $fmt !~ m/^ova\+/;
+        if $outer_fmt ne 'ova';

-    # extract the inner file from the name
-    my $archive_volid;
-    my $inner_file;
-    my $inner_fmt;
-    if ($name =~ m!^(.*\.ova)/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+)$!) {
-        $archive_volid = "$source_storeid:import/$1";
-        $inner_file = $2;
-        ($inner_fmt) = $fmt =~ /^ova\+(.*)$/;
-    } else {
-        die "cannot extract $volid - invalid volname $volname\n";
-    }
+    my $archive_volid = $source_storeid . ':' . $vtype . '/' . $parts->{'disk-path'};
+    my $inner_file = $parts->{'content-file'};
+    my $inner_fmt = $parts->{'content-ext'};

     die "cannot determine format of '$volid'\n" if !$inner_fmt;

+    my $cfg = PVE::Storage::config();
     my $ova_path = PVE::Storage::path($cfg, $archive_volid);

     my $tmpdir = PVE::Storage::get_image_dir($cfg, $target_storeid, $vmid);
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index bd7cec13..9fb45b0c 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -39,6 +39,8 @@ my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');

 my @IMPORT_EXTENSIONS = ('ova', 'ovf', 'qcow2', 'raw', 'vmdk');

+my @OVA_CONTENT_EXTENSIONS = ('qcow2', 'raw', 'vmdk');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }
@@ -49,8 +51,12 @@ my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSI

 my $RE_IMPORT_EXTENSIONS = join_to_re_alternations(@IMPORT_EXTENSIONS);

+my $RE_OVA_CONTENT_EXTENSIONS = join_to_re_alternations(@OVA_CONTENT_EXTENSIONS);
+
 my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;

+my $RE_SAFE_CHAR_WITH_WHITESPACE_CLASS = qr/[ a-zA-Z0-9\-\.\+\=\_]/;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -124,6 +130,42 @@ my $RE_IMPORT_FILE_PATH = qr!
     )
 !xn;

+my $RE_OVA_CONTENT = qr!
+    (?<content_file>
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+ \. (?<content_ext> $RE_OVA_CONTENT_EXTENSIONS)
+    )
+!xn;
+
+# NOTE: Volume names with the 'import' vtype are treated differently when
+# they do not stem from a file path directly - see comments inline.
+my $RE_IMPORT_VOLNAME_OVA_FILE_WITH_CONTENT = qr!
+    (?<file>
+        # NOTE: Unlike in RE_IMPORT_FILE_PATH, we allow whitespace here
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+
+        \.
+        (?<ext> ova)
+    ) / (?<content> $RE_OVA_CONTENT)
+!xn;
+
+my $RE_IMPORT_VOLNAME_REGULAR_FILE = qr!
+    (?<file>
+        # NOTE: Unlike in RE_IMPORT_FILE_PATH, we allow whitespace here
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+
+        \.
+        (?<ext> $RE_IMPORT_EXTENSIONS)
+    )
+!xn;
+
+my $RE_IMPORT_VOLNAME = qr!
+    (?<path>
+        # NOTE: Order here matters - the ova+content regex is stricter,
+        # so try this branch here first
+        ($RE_IMPORT_VOLNAME_OVA_FILE_WITH_CONTENT)
+        |
+        ($RE_IMPORT_VOLNAME_REGULAR_FILE)
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
@@ -137,6 +179,9 @@ my $RE_VOLNAME_FOR_VTYPE = {
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
     snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
+
+    # special cases - not reusing file path regexes:
+    import => qr/^$RE_IMPORT_VOLNAME$/,
 };

 my sub contains_parent_dir($path) {
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index e0029fc6..b6e42307 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -396,17 +396,104 @@ my $volname_cases_snippets_valid = [
     },
 ];

+my $volname_cases_import_valid = [
+    {
+        path => 'import.ova',
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova',
+            vtype => 'import',
+            volname => 'import/import.ova',
+        },
+    },
+    {
+        path => 'import.ovf',
+        expected => {
+            file => 'import.ovf',
+            ext => 'ovf',
+            'disk-path' => 'import.ovf',
+            path => 'import.ovf',
+            vtype => 'import',
+            volname => 'import/import.ovf',
+        },
+    },
+    {
+        path => 'disk-0.qcow2',
+        expected => {
+            file => 'disk-0.qcow2',
+            ext => 'qcow2',
+            'disk-path' => 'disk-0.qcow2',
+            path => 'disk-0.qcow2',
+            vtype => 'import',
+            volname => 'import/disk-0.qcow2',
+        },
+    },
+    {
+        path => 'raw_disk_v2.5.raw',
+        expected => {
+            file => 'raw_disk_v2.5.raw',
+            ext => 'raw',
+            'disk-path' => 'raw_disk_v2.5.raw',
+            path => 'raw_disk_v2.5.raw',
+            vtype => 'import',
+            volname => 'import/raw_disk_v2.5.raw',
+        },
+    },
+    {
+        path => 'disk-1+data.vmdk',
+        expected => {
+            file => 'disk-1+data.vmdk',
+            ext => 'vmdk',
+            'disk-path' => 'disk-1+data.vmdk',
+            path => 'disk-1+data.vmdk',
+            vtype => 'import',
+            volname => 'import/disk-1+data.vmdk',
+        },
+    },
+];
+
+my $volname_cases_import_invalid = [
+    {
+        description => "Invalid file extension (import)",
+        args => {
+            path => 'import.zip',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (import)",
+        args => {
+            path => 'import.Ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Unsafe characters in file name (import)",
+        args => {
+            path => '🐪perl-playground🐪.ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
     $volname_cases_backup_valid,
     $volname_cases_snippets_valid,
+    $volname_cases_import_valid,
 ];

 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
     $volname_cases_backup_invalid,
+    $volname_cases_import_invalid,
 ];

 {
@@ -474,6 +561,178 @@ my sub run_volname_parsing_tests : prototype($) ($volname_tests) {
     return;
 }

+# NOTE: These run for parse_path_as_volname_parts() and parse_path_as_volname() only!
+my $cases_import_special_file_path = [
+    {
+        description => "Whitespace not allowed in volume file paths (import)",
+        args => {
+            path => 'Some Disk.qcow2',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (raw) (import)",
+        args => {
+            path => 'import.ova/disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (qcow2) (import)",
+        args => {
+            path => 'import.ova/disk.qcow2',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (vmdk) (import)",
+        args => {
+            path => 'import.ova/disk.vmdk',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+];
+
+# NOTE: These run for parse_volname_as_parts() only!
+my $cases_import_special_volname = [
+    {
+        description => "Whitespace allowed in volume names (import)",
+        args => {
+            volname => 'import/Some Disk.qcow2',
+        },
+        expected => {
+            file => 'Some Disk.qcow2',
+            ext => 'qcow2',
+            'disk-path' => 'Some Disk.qcow2',
+            path => 'Some Disk.qcow2',
+            vtype => 'import',
+            volname => 'import/Some Disk.qcow2',
+        },
+    },
+    {
+        description => "Volume name with OVA content (raw) (import)",
+        args => {
+            volname => 'import/import.ova/disk.raw',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.raw',
+            content => 'disk.raw',
+            'content-file' => 'disk.raw',
+            'content-ext' => 'raw',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.raw',
+        },
+    },
+    {
+        description => "Volume name with OVA content (qcow2) (import)",
+        args => {
+            volname => 'import/import.ova/disk.qcow2',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.qcow2',
+            content => 'disk.qcow2',
+            'content-file' => 'disk.qcow2',
+            'content-ext' => 'qcow2',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.qcow2',
+        },
+    },
+    {
+        description => "Volume name with OVA content (vmdk) (import)",
+        args => {
+            volname => 'import/import.ova/disk.vmdk',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.vmdk',
+            content => 'disk.vmdk',
+            'content-file' => 'disk.vmdk',
+            'content-ext' => 'vmdk',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.vmdk',
+        },
+    },
+    {
+        description => "Whitespace in volume name + OVA content (import)",
+        args => {
+            volname => 'import/Some Import.ova/disk.qcow2',
+        },
+        expected => {
+            file => 'Some Import.ova',
+            ext => 'ova',
+            'disk-path' => 'Some Import.ova',
+            path => 'Some Import.ova/disk.qcow2',
+            content => 'disk.qcow2',
+            'content-file' => 'disk.qcow2',
+            'content-ext' => 'qcow2',
+            vtype => 'import',
+            volname => 'import/Some Import.ova/disk.qcow2',
+        },
+    },
+];
+
+my sub run_special_import_volname_parsing_tests : prototype() () {
+    for my $case ($cases_import_special_file_path->@*) {
+        subtest $case->{description} => sub () {
+            my ($path, $vtype) = $case->{args}->@{qw(path vtype)};
+
+            my $got_volname_parts = parse_path_as_volname_parts($path, $vtype);
+            my $got_volname = parse_path_as_volname($path, $vtype);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname_parts,
+                    $case->{expected},
+                    'parse_path_as_volname_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+
+                is(
+                    $got_volname,
+                    $case->{expected}->{volname},
+                    'parse_path_as_volname() returns expected volname',
+                );
+            } else {
+                is($got_volname_parts, undef, 'parse_path_as_volname_parts() returns undef');
+                is($got_volname, undef, 'parse_path_as_volname() returns undef');
+            }
+        };
+    }
+
+    for my $case ($cases_import_special_volname->@*) {
+        subtest $case->{description} => sub () {
+            my ($volname) = $case->{args}->@{qw(volname)};
+
+            my $got_volname = parse_volname_as_parts($volname);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname,
+                    $case->{expected},
+                    'parse_volname_as_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+            } else {
+                is($got_volname, undef, 'parse_volname_as_parts() returns undef');
+            }
+        };
+    }
+
+    return;
+}
+
 my $DEFAULT_STOREID = 'local';
 my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
 my $DEFAULT_SCFG = {
@@ -546,6 +805,26 @@ my $volid_tests = [
                 $volname_tests->@*
         ],
     },
+    {
+        description =>
+            "volid parsers build on volname parsers' behaviors - 'import' vtype file paths (1)",
+        storedid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $cases_import_special_file_path->@*
+        ],
+    },
+    {
+        description =>
+            "volid parsers build on volname parsers' behaviors - 'import' vtype file paths (2)",
+        storedid => $ALT_STOREID,
+        scfg => $ALT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $cases_import_special_file_path->@*
+        ],
+    },
 ];

 my sub run_volid_parsing_tests : prototype($) ($volid_tests) {
@@ -603,6 +882,8 @@ my sub main() {
     unified_diff();

     run_volname_parsing_tests($volname_tests);
+    run_special_import_volname_parsing_tests();
+
     run_volid_parsing_tests($volid_tests);

     done_testing();
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 39e5cb0d..a3cb1343 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -839,20 +839,16 @@ sub parse_volname {
         if ($vtype eq 'snippets') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }

-    if ($volname =~
-        m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.ova\/${PVE::Storage::OVA_CONTENT_RE_1})$!
-    ) {
-        my $packed_image = $1;
-        my $format = $2;
-        return ('import', $packed_image, undef, undef, undef, undef, "ova+$format");
-    }
+        if ($vtype eq 'import') {
+            my $format = $parts->{ext};

-    if ($volname =~
-        m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!
-    ) {
-        return ('import', $1, undef, undef, undef, undef, $2);
+            if (defined(my $content_ext = $parts->{'content-ext'})) {
+                $format .= "+$content_ext";
+            }
+
+            return ($vtype, $volume_path, undef, undef, undef, undef, $format);
+        }
     }

     die "unable to parse directory volume name '$volname'\n";
diff --git a/src/test/guest_import_test.pl b/src/test/guest_import_test.pl
index 04eec24d..234013e3 100755
--- a/src/test/guest_import_test.pl
+++ b/src/test/guest_import_test.pl
@@ -357,7 +357,7 @@ my $tests = [
             {
                 volname => 'import/some-import.ova/disk.txt',
                 vmid => 1337,
-                'expect-fail' => qr/unable to parse directory volume name/,
+                'expect-fail' => qr/invalid volname/,
             },
         ],
     },
@@ -532,7 +532,7 @@ my $tests = [
             {
                 volname => 'import/some-🐧-import.ova/disk.qcow2',
                 vmid => 1337,
-                'expect-fail' => qr/unable to parse directory volume name/,
+                'expect-fail' => qr/invalid volname/,
             },
         ],
     },
@@ -619,7 +619,7 @@ my $tests = [
             {
                 volname => 'import/some-import.ova',
                 vmid => 1337,
-                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+                'expect-fail' => qr/cannot determine format of/,
             },
         ],
     },
--
2.47.3

[PATCH pve-storage v1 29/54] plugin: simplify recently refactored logic in parse_volname method

[Max R. Carrara]

Pull the if-clause with the definedness check out and make it a guard
clause, die-ing early if parsing fails.

Sum up the if-clauses of the `iso`, `vztmpl` and `snippets` volume
types, since their bodies have an identical pattern.

Keep the branches for the `backup` and `import` volume types separate.

Since the guard clause now already `die`s on a parsing failure, extend
the error message of the final `die` at the end of the method, stating
that the volume type is unhandled. This should never be hit unless we
introduce a new volume type and happen to forget to handle it in this
method.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 48 +++++++++++++++++----------------------
 1 file changed, 21 insertions(+), 27 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index a3cb1343..376e2515 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -821,37 +821,31 @@ sub parse_volname {
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
     }

-    if (defined(my $parts = parse_volname_as_parts($volname))) {
-        my ($vtype, $volume_path) = $parts->@{qw(vtype path)};
+    my $parts = parse_volname_as_parts($volname);
+    die "unable to parse directory volume name '$volname'\n"
+        if !defined($parts);

-        if ($vtype eq 'iso') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
+    my ($vtype, $volume_path) = $parts->@{qw(vtype path)};

-        if ($vtype eq 'vztmpl') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'backup') {
-            return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'snippets') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'import') {
-            my $format = $parts->{ext};
-
-            if (defined(my $content_ext = $parts->{'content-ext'})) {
-                $format .= "+$content_ext";
-            }
-
-            return ($vtype, $volume_path, undef, undef, undef, undef, $format);
-        }
+    if ($vtype eq 'iso' || $vtype eq 'vztmpl' || $vtype eq 'snippets') {
+        return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
     }

-    die "unable to parse directory volume name '$volname'\n";
+    if ($vtype eq 'backup') {
+        return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
+    }
+
+    if ($vtype eq 'import') {
+        my $format = $parts->{ext};
+
+        if (defined(my $content_ext = $parts->{'content-ext'})) {
+            $format .= "+$content_ext";
+        }
+
+        return ($vtype, $volume_path, undef, undef, undef, undef, $format);
+    }
+
+    die "unable to parse directory volume name '$volname' - unhandled volume type '$vtype'\n";
 }

 # FIXME: remove on the next APIAGE reset.
--
2.47.3

[PATCH pve-storage v1 30/54] plugin: simplify recently refactored logic in get_subdir_files helper

[Max R. Carrara]

Pull out the common pieces of the if-clauses that handle individual
volume types.

Do not strip the volume type subdirectory from the beginning of the
current path anymore, because that is handled by the
`parse_path_as_volid_parts()` parsing function, which left `$filename`
unused.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 23 ++---------------------
 1 file changed, 2 insertions(+), 21 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 376e2515..65e0869b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1677,17 +1677,10 @@ my sub get_subdir_files {
     my $get_subdir_file_info = sub {
         my ($path, $st) = @_;

-        # Strip the vtype subdir from beginning of the current path
-        # so that we don't have to take it into account when parsing the file name
-        my $filename = $path;
-        if ($filename !~ s!^\Q$vtype_subdir\E!!) {
-            return;
-        }
+        my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+        return if !defined($parts);

         if ($vtype eq 'iso') {
-            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => 'iso', # always 'iso' even if we have a file ending in .img
@@ -1695,9 +1688,6 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'vztmpl') {
-            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             my ($ext, $ext_compression) = $parts->@{qw(ext ext-compression)};
             my $format = $ext eq 'tar' ? $ext : ('t' . $ext_compression);

@@ -1708,9 +1698,6 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'backup') {
-            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             my $format = $parts->{ext};
             my $volume_path = $parts->{path};

@@ -1748,9 +1735,6 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'snippets') {
-            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => 'snippet',
@@ -1758,9 +1742,6 @@ my sub get_subdir_files {
         }

         if ($vtype eq 'import') {
-            my $parts = parse_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => $parts->{ext},
--
2.47.3

[PATCH pve-storage v1 31/54] storage: simplify recently refactored logic in path_to_volume_id sub

[Max R. Carrara]

Since the branches for all volume types except `images` do exactly the
same thing, just return the result of the `parse_path_as_volid()`
parsing function directly.

In order to avoid unnecessary work for the other volume types, strip
the volume type subdirectory from the given file path only for the
`images` volume type.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 40 ++++++++++------------------------------
 1 file changed, 10 insertions(+), 30 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 3a716894..199a6375 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -730,16 +730,16 @@ sub path_to_volume_id {
     my $parse_volid_from_file_path = sub {
         my ($plugin, $sid, $scfg, $vtype) = @_;

-        my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
-
-        # Strip the vtype subdir from beginning of the current path
-        # so that we don't have to take it into account when parsing the file name
-        my $filename = $path;
-        if ($filename !~ s!^\Q$vtype_subdir\E!!) {
-            return;
-        }
-
         if ($vtype eq 'images') {
+            my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+            # Strip the vtype subdir from beginning of the current path
+            # so that we don't have to take it into account when parsing the file name
+            my $filename = $path;
+            if ($filename !~ s!^\Q$vtype_subdir\E!!) {
+                return;
+            }
+
             return if $filename !~ m!/(\d+)/([^/\s]+)$!;
             my $vmid = $1;
             my $name = $2;
@@ -756,27 +756,7 @@ sub path_to_volume_id {
             return;
         }

-        if ($vtype eq 'iso') {
-            return parse_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'vztmpl') {
-            return parse_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'backup') {
-            return parse_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'snippets') {
-            return parse_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'import') {
-            return parse_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        return;
+        return parse_path_as_volid($sid, $scfg, $path, $vtype);
     };

     my $vtypes_to_check = $get_vtypes_to_check->();
--
2.47.3

[PATCH pve-storage v1 32/54] api: status: simplify recently added parsing helper for file transfers

[Max R. Carrara]

Since we already assert whether we can transfer files for a given
volume type using the `assert_can_transfer_files()` helper, and also
raise an exception if parsing fails, handling the parsing of each
volume type individually is not necessary anymore.

Therefore, pull out the common parts of the branches for individual
volume types.

Retain only the special case for the `import` type where we do not
accept `.ovf` files.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 40 ++++++----------------------------
 1 file changed, 7 insertions(+), 33 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index b578a1ed..bff59b66 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -73,45 +73,19 @@ my sub assert_can_transfer_files : prototype($$$) {
 my sub parse_transferred_file_path_extension : prototype($$) {
     my ($path, $vtype) = @_;

-    if ($vtype eq 'iso') {
-        my $parts = parse_path_as_volname_parts($path, $vtype);
+    my $parts = parse_path_as_volname_parts($path, $vtype);

-        if (!defined($parts)) {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        my $ext = $parts->{ext};
-        return $ext;
+    if (!defined($parts)) {
+        raise_param_exc({ filename => "invalid filename or wrong file extension" });
     }

-    if ($vtype eq 'vztmpl') {
-        my $parts = parse_path_as_volname_parts($path, $vtype);
+    my $ext = $parts->{ext};

-        if (!defined($parts)) {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        my $ext = $parts->{ext};
-        return $ext;
+    if ($vtype eq 'import' && $ext eq 'ovf') {
+        raise_param_exc({ filename => "wrong file extension" });
     }

-    if ($vtype eq 'import') {
-        my $parts = parse_path_as_volname_parts($path, $vtype);
-
-        if (!defined($parts)) {
-            raise_param_exc({ filename => "invalid filename or wrong extension" });
-        }
-
-        my $ext = $parts->{ext};
-
-        if ($ext eq 'ovf') {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        return $ext;
-    }
-
-    die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
+    return $ext;
 }

 my sub assert_file_transfer_contents_valid : prototype($$$) {
--
2.47.3

[PATCH pve-storage v1 33/54] plugin: use parsing helper in parse_volume_id sub

[Max R. Carrara]

In order to avoid having multiple subroutines do (sort of) the same
thing, use the `parse_volid_as_parts()` parsing function inside the
`parse_volume_id()` plugin API sub.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 65e0869b..3853682b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -22,6 +22,7 @@ use PVE::Storage::Common qw(
 use PVE::Storage::Common::Parse qw(
     parse_volname_as_parts
     parse_path_as_volid_parts
+    parse_volid_as_parts
 );

 use JSON;
@@ -428,8 +429,8 @@ PVE::JSONSchema::register_format('pve-volume-id', \&parse_volume_id);
 sub parse_volume_id {
     my ($volid, $noerr) = @_;

-    if ($volid =~ m/^([a-z][a-z0-9\-\_\.]*[a-z0-9]):(.+)$/i) {
-        return wantarray ? ($1, $2) : $1;
+    if (defined(my $parts = parse_volid_as_parts($volid))) {
+        return wantarray ? ($parts->{storeid}, $parts->{volname}) : $parts->{storeid};
     }
     return undef if $noerr;
     die "unable to parse volume ID '$volid'\n";
--
2.47.3

[PATCH pve-storage v1 34/54] test: list volumes: reorganize and modernize test running code

[Max R. Carrara]

Use the `use v5.36` pragma to enable subroutine signatures,
warnings and strictness by default.

Factor out the list of file suffixes for file formats that support
backing files into its own constant.

Introduce a `main()` subroutine.

Move the code that runs the existing tests for
`::Plugin->list_volumes()` into its own subroutine.

Move the remaining single test that checks whether the vmlist returned
by `PVE::Cluster::get_vmlist()` was modified into `main()`.

Move the call to `Test::More`'s `plan()` into `main()` as well.

Done as a preparational step for other changes.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 66 ++++++++++++++++++++---------------
 1 file changed, 38 insertions(+), 28 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 5cb08880..053ebbe3 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestListVolumes;

-use strict;
-use warnings;
+use v5.36;

 use lib qw(..);

@@ -13,7 +12,7 @@ use Test::More;
 use Test::MockModule;

 use Cwd;
-use File::Basename;
+use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
 use File::stat qw();
 use File::Temp;
@@ -76,6 +75,8 @@ my $scfg = {
     },
 };

+my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
+
 # The test cases are comprised of an arry of hashes with the following keys:
 # description => displayed on error by Test::More
 # vmid        => used for image matches by list_volume
@@ -667,31 +668,9 @@ my sub cmp_volinfo_by_volid {
     return $a->{volid} cmp $b->{volid};
 }

-my $plan = scalar @tests;
-plan tests => $plan + 1;
-
-{
-    # don't accidentally modify vmlist, see bug report
-    # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
-    my $scfg_with_type = { path => $storage_dir, type => 'dir' };
-    my $original_vmlist = { ids => {} };
-    my $tested_vmlist = dclone($original_vmlist);
-
-    PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
-
-    is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
-        || diag(
-            "Expected vmlist to remain\n",
-            explain($original_vmlist),
-            "but it turned to\n",
-            explain($tested_vmlist),
-        );
-}
-
-{
+my sub run_legacy_tests() {
     my $sid = 'local';
     my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
-    my @suffixes = ('qcow2', 'raw', 'vmdk', 'vhdx');

     # run through test cases
     foreach my $tt (@tests) {
@@ -704,7 +683,7 @@ plan tests => $plan + 1;
         # prepare environment
         my $num = 0; #parent disks
         for my $file (@$files) {
-            my ($name, $dir, $suffix) = fileparse($file, @suffixes);
+            my ($name, $dir, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);

             make_path($dir, { verbose => 1, mode => 0755 });

@@ -732,8 +711,39 @@ plan tests => $plan + 1;
         # we get wrong results from leftover files
         remove_tree($storage_dir, { verbose => 1 });
     }
+
+    return;
 }

-done_testing();
+sub main() {
+    my $plan = scalar @tests;
+    plan tests => $plan + 1;
+
+    {
+        # don't accidentally modify vmlist, see bug report
+        # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
+        my $scfg_with_type = { path => $storage_dir, type => 'dir' };
+        my $original_vmlist = { ids => {} };
+        my $tested_vmlist = dclone($original_vmlist);
+
+        PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
+
+        is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
+            || diag(
+                "Expected vmlist to remain\n",
+                explain($original_vmlist),
+                "but it turned to\n",
+                explain($tested_vmlist),
+            );
+    }
+
+    run_legacy_tests();
+
+    done_testing();
+
+    return;
+}
+
+main();

 1;
--
2.47.3

[PATCH pve-storage v1 35/54] test: list volumes: fix broken test checking for vmlist modifications

[Max R. Carrara]

This test never actually did anything, since it didn't use the mocked
vmlist at all. Therefore, fix it by comparing the vmlist with its
original after all tests have run.

Verified that this test catches modifications to the vmlist by
manually adding a line that modifies it in the `list_volumes()`
method, running the tests to see if this fails, and then removing that
line again.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 053ebbe3..2d941b60 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -719,26 +719,19 @@ sub main() {
     my $plan = scalar @tests;
     plan tests => $plan + 1;

-    {
-        # don't accidentally modify vmlist, see bug report
-        # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
-        my $scfg_with_type = { path => $storage_dir, type => 'dir' };
-        my $original_vmlist = { ids => {} };
-        my $tested_vmlist = dclone($original_vmlist);
-
-        PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
-
-        is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
-            || diag(
-                "Expected vmlist to remain\n",
-                explain($original_vmlist),
-                "but it turned to\n",
-                explain($tested_vmlist),
-            );
-    }
+    # Keep the original vmlist around in order to check whether it was modified
+    # after running all the tests. See:
+    # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
+    my $original_vmlist = dclone(PVE::Cluster::get_vmlist());

     run_legacy_tests();

+    my $vmlist = PVE::Cluster::get_vmlist();
+
+    is_deeply(
+        $vmlist, $original_vmlist, 'Result of PVE::Cluster::get_vmlist remains unmodified',
+    );
+
     done_testing();

     return;
--
2.47.3

[PATCH pve-storage v1 36/54] test: list volumes: introduce new format for test cases

[Max R. Carrara]

Introduce a new format for test cases in `list_volumes_test.pm`, which
allows specifying the `$storeid`, `$scfg` and the list of
content types / volume types for the given test.

At the same time, get rid of the `files` and `expected` lists in the
new format, which made it somewhat tedious to keep track of which file
belonged to which hashref in the expected output, or if it even was
supposed to show up in the output in the first place.

Get rid of the optional `parent` list for the same reason, too.

Replace these three lists with a single `cases` list that simply
contains hashes that match a given file path with the expected hash in
the final output. The `parent` may optionally be specified in the same
hash for a given file path as well, which makes the backing file logic
a little simpler.

Describe the new format in a small POD docstring.

Additionally, introduce replacements for the constants declared with
the `use constant` pragma.

Finally, migrate the first of the old test cases over to the new
format and run it through the new machinery added for the new test
case format.

The reason for introducing this new format here is to make it easier
to add more specific test cases. The "legacy" tests always pass all
volume types as parameter to `PVE::Storage::Plugin->list_volumes()`,
for example.

As an additional benefit, we could reuse / recycle the new testing
machinery added here for testing other storage (plugin) API methods as
well.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 399 +++++++++++++++++++++++++---------
 1 file changed, 300 insertions(+), 99 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 2d941b60..c83b782e 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -11,6 +11,7 @@ use PVE::Tools qw(run_command);
 use Test::More;
 use Test::MockModule;

+use Carp qw(confess);
 use Cwd;
 use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
@@ -22,6 +23,10 @@ use constant DEFAULT_SIZE => 131072; # 128 kiB
 use constant DEFAULT_USED => 262144; # 256 kiB
 use constant DEFAULT_CTIME => 1234567890;

+my $DEFAULT_SIZE = 128 * 1024; # 128 kiB
+my $DEFAULT_USED = 256 * 1024; # 256 kiB
+my $DEFAULT_CTIME = 1234567890;
+
 # get_vmlist() return values
 my $mocked_vmlist = {
     'version' => 1,
@@ -75,6 +80,23 @@ my $scfg = {
     },
 };

+my $DEFAULT_STOREID = 'local';
+my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
+my $DEFAULT_SCFG = {
+    type => 'dir',
+    path => $DEFAULT_STORAGE_PATH,
+    shared => 0,
+    content => {
+        iso => 1,
+        rootdir => 1,
+        vztmpl => 1,
+        images => 1,
+        snippets => 1,
+        backup => 1,
+        import => 1,
+    },
+};
+
 my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');

 # The test cases are comprised of an arry of hashes with the following keys:
@@ -84,103 +106,6 @@ my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
 # expected    => returned result hash
 #                (content, ctime, format, parent, size, used, vimd, volid)
 my @tests = (
-    {
-        description => 'VMID: 16110, VM, qcow2, backup, snippets',
-        vmid => '16110',
-        files => [
-            "$storage_dir/images/16110/vm-16110-disk-0.qcow2",
-            "$storage_dir/images/16110/vm-16110-disk-1.raw",
-            "$storage_dir/images/16110/vm-16110-disk-2.vmdk",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
-            "$storage_dir/snippets/userconfig.yaml",
-            "$storage_dir/snippets/hookscript.pl",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'raw',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-1.raw',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'vmdk',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-2.vmdk',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602700,
-                'format' => 'vma.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602765,
-                'format' => 'vma.lzo',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602835,
-                'format' => 'vma',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602835,
-                'format' => 'vma.zst',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/hookscript.pl',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/userconfig.yaml',
-            },
-        ],
-    },
     {
         description => 'VMID: 16112, lxc, raw, backup',
         vmid => '16112',
@@ -631,6 +556,189 @@ my @tests = (
     },
 );

+=head2 TEST CASE FORMAT
+
+The parameters for individual test cases are hashes with the following
+keys:
+
+    {
+        # Name of the test, also displayed on error.
+        description => '...',
+
+        # The storage ID for the test.
+        # Used as parameter when calling list_volumes.
+        storeid => 'some-storeid',
+
+        # The storage config hash for the test.
+        # Used as parameter when calling list_volumes.
+        scfg => {
+            type => 'dir',
+            path => '...',
+            shared => 0,
+            content => {
+                iso => 1,
+                # [...]
+            },
+            # [...]
+        },
+
+        # The VMID used as parameter when calling list_volumes.
+        # May be undef.
+        vmid => 1234,
+
+        # The list of volume types / content types used as parameter when
+        # calling list_volumes.
+        vtypes => [],
+
+        # List of hashes that associate a file with its expected hash in the
+        # output of list_volumes.
+        cases => [
+            {
+                # Absolute path to the file.
+                file => "/tmp/foobar/images/1234/vm-1234-disk-0.qcow2",
+
+                # Relative path to parent (backing file).
+                parent => '../ssss/base-4321-disk-0.qcow2',
+
+                # Expected hash in the output of list_volumes.
+                expected => {
+                    content => 'images',
+                    ctime => 1234567890, # inode change time in seconds since the epoc
+                    format => 'qcow2',
+                    parent => '../ssss/base-4321-disk-0.qcow2',
+                    size => 1337, # bytes!
+                    used => 1024, # bytes!
+                    vmid => 1234,
+                    volid => 'local:1234/vm-1234-disk-0.qcow2',
+                },
+            },
+        ],
+    }
+
+=cut
+
+my $test_param_list = [
+    {
+        description => 'VMID: 16110, VM, qcow2, backup, snippets',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16110,
+        vtypes => ['images', 'backup', 'snippets'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-0.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-1.raw",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-1.raw',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-2.vmdk",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-2.vmdk',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602700,
+                    format => 'vma.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602765,
+                    format => 'vma.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/userconfig.yaml",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/userconfig.yaml',
+                },
+            },
+        ],
+    },
+];
+
 # provide static vmlist for tests
 my $mock_cluster = Test::MockModule->new('PVE::Cluster', no_auto => 1);
 $mock_cluster->redefine(get_vmlist => sub { return $mocked_vmlist; });
@@ -715,15 +823,108 @@ my sub run_legacy_tests() {
     return;
 }

+my sub setup_test_env($test_params) {
+    my ($storeid, $scfg) = $test_params->@{qw(storeid scfg)};
+
+    make_path($scfg->{path}, { verbose => 1, mode => 0700 });
+
+    for my $case ($test_params->{cases}->@*) {
+        my ($file, $parent) = $case->@{qw(file parent)};
+
+        confess "\$file = undef" if !defined($file);
+
+        my ($file_name, $directory, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);
+
+        make_path($directory, { verbose => 1, mode => 0755 });
+
+        if ($file_name) {
+            # using qemu-img to also be able to represent the backing device
+            my $command = ['/usr/bin/qemu-img', 'create', "$file", $DEFAULT_SIZE];
+
+            push($command->@*, '-f', $suffix) if $suffix;
+            push($command->@*, '-u', '-b', $parent) if $parent;
+            push($command->@*, '-F', $suffix) if $parent && $suffix;
+
+            run_command($command, timeout => 10);
+        }
+    }
+
+    return;
+}
+
+my sub teardown_test_env($test_params) {
+    my $scfg = $test_params->{scfg};
+
+    remove_tree($scfg->{path}, { verbose => 1 });
+
+    return;
+}
+
+my sub run_test_for_params($test_params) {
+    eval { setup_test_env($test_params); };
+    if ($@) {
+        teardown_test_env($test_params);
+
+        fail($test_params->{description});
+        diag("Failed to set up test environment: $@");
+
+        return;
+    }
+
+    my ($storeid, $scfg, $vmid, $vtypes) = $test_params->@{qw(storeid scfg vmid vtypes)};
+
+    my $expected = [map { $_->{expected} // () } $test_params->{cases}->@*];
+    $expected = [sort cmp_volinfo_by_volid $expected->@*];
+
+    my $got = eval {
+        my $volume_list = PVE::Storage::Plugin->list_volumes($storeid, $scfg, $vmid, $vtypes);
+        return [sort cmp_volinfo_by_volid $volume_list->@*];
+    };
+    $got = $@ if $@;
+
+    my $desc_vmid = defined($vmid) ? $vmid : 'undef';
+    my $desc_vtypes = '[' . join(', ', $vtypes->@*) . ']';
+
+    my $description =
+        $test_params->{description} . " - \$vmid = $desc_vmid, \$vtypes = $desc_vtypes";
+
+    if (!is_deeply($got, $expected, $description)) {
+        diag("\$got = ", explain($got));
+        diag("\$expected = ", explain($expected));
+    }
+
+    teardown_test_env($test_params);
+
+    return;
+}
+
+my sub assert_test_params_keys_exist($test_params) {
+    my $desc = $test_params->{description};
+
+    confess "Test parameters without description"
+        if !$desc;
+
+    for my $key (qw(storeid scfg vmid vtypes cases)) {
+        confess "Key '$key' does not exist in '$desc'"
+            if !exists($test_params->{$key});
+    }
+
+    return;
+}
+
 sub main() {
-    my $plan = scalar @tests;
-    plan tests => $plan + 1;
+    plan tests => scalar(@tests) + scalar($test_param_list->@*) + 1;

     # Keep the original vmlist around in order to check whether it was modified
     # after running all the tests. See:
     # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
     my $original_vmlist = dclone(PVE::Cluster::get_vmlist());

+    for my $test_params ($test_param_list->@*) {
+        assert_test_params_keys_exist($test_params);
+        run_test_for_params($test_params);
+    }
+
     run_legacy_tests();

     my $vmlist = PVE::Cluster::get_vmlist();
--
2.47.3

[PATCH pve-storage v1 37/54] test: list volumes: remove legacy code and migrate cases to new format

[Max R. Carrara]

Remove all that remains of the old testing code, including the
constants defined with the `use constant` pragma and other old
variables.

Migrate all remaining test cases over to the new format.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 1185 ++++++++++++++++++---------------
 1 file changed, 654 insertions(+), 531 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index c83b782e..ca3bbac7 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -19,10 +19,6 @@ use File::stat qw();
 use File::Temp;
 use Storable qw(dclone);

-use constant DEFAULT_SIZE => 131072; # 128 kiB
-use constant DEFAULT_USED => 262144; # 256 kiB
-use constant DEFAULT_CTIME => 1234567890;
-
 my $DEFAULT_SIZE = 128 * 1024; # 128 kiB
 my $DEFAULT_USED = 256 * 1024; # 256 kiB
 my $DEFAULT_CTIME = 1234567890;
@@ -64,22 +60,6 @@ my $mocked_vmlist = {
     },
 };

-my $storage_dir = File::Temp->newdir();
-my $scfg = {
-    'type' => 'dir',
-    'path' => $storage_dir,
-    'shared' => 0,
-    'content' => {
-        'iso' => 1,
-        'rootdir' => 1,
-        'vztmpl' => 1,
-        'images' => 1,
-        'snippets' => 1,
-        'backup' => 1,
-        'import' => 1,
-    },
-};
-
 my $DEFAULT_STOREID = 'local';
 my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
 my $DEFAULT_SCFG = {
@@ -99,463 +79,6 @@ my $DEFAULT_SCFG = {

 my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');

-# The test cases are comprised of an arry of hashes with the following keys:
-# description => displayed on error by Test::More
-# vmid        => used for image matches by list_volume
-# files       => array of files for qemu-img to create
-# expected    => returned result hash
-#                (content, ctime, format, parent, size, used, vimd, volid)
-my @tests = (
-    {
-        description => 'VMID: 16112, lxc, raw, backup',
-        vmid => '16112',
-        files => [
-            "$storage_dir/images/16112/vm-16112-disk-0.raw",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
-        ],
-        expected => [
-            {
-                'content' => 'rootdir',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'raw',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16112',
-                'volid' => 'local:16112/vm-16112-disk-0.raw',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604370,
-                'format' => 'tar.lzo',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604970,
-                'format' => 'tar.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604970,
-                'format' => 'tar.zst',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585605570,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604370,
-                'format' => 'tar.bz2',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'openvz',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: 16114, VM, qcow2, linked clone',
-        vmid => '16114',
-        files => [
-            "$storage_dir/images/16114/vm-16114-disk-0.qcow2",
-            "$storage_dir/images/16114/vm-16114-disk-1.qcow2",
-        ],
-        parent => [
-            "../9004/base-9004-disk-0.qcow2", "../9004/base-9004-disk-1.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../9004/base-9004-disk-0.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16114',
-                'volid' => 'local:9004/base-9004-disk-0.qcow2/16114/vm-16114-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../9004/base-9004-disk-1.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16114',
-                'volid' => 'local:9004/base-9004-disk-1.qcow2/16114/vm-16114-disk-1.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: 9004, VM, template, qcow2',
-        vmid => '9004',
-        files => [
-            "$storage_dir/images/9004/base-9004-disk-0.qcow2",
-            "$storage_dir/images/9004/base-9004-disk-1.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '9004',
-                'volid' => 'local:9004/base-9004-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '9004',
-                'volid' => 'local:9004/base-9004-disk-1.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, templates, snippets, backup',
-        vmid => undef,
-        files => [
-            "$storage_dir/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
-            "$storage_dir/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
-            "$storage_dir/template/iso/archlinux-2020.02.01-x86_64.iso",
-            "$storage_dir/template/iso/debian-8.11.1-amd64-DVD-1.iso",
-            "$storage_dir/template/iso/debian-9.12.0-amd64-netinst.iso",
-            "$storage_dir/template/iso/proxmox-ve_6.1-1.iso",
-            "$storage_dir/template/cache/archlinux-base_20190924-1_amd64.tar.gz",
-            "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
-            "$storage_dir/template/cache/debian-11.0-standard_11.0-1_amd64.tar.bz2",
-            "$storage_dir/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
-            "$storage_dir/snippets/userconfig.yaml",
-            "$storage_dir/snippets/hookscript.pl",
-            "$storage_dir/private/1234/", # fileparse needs / at the end
-            "$storage_dir/private/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
-        ],
-        expected => [
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'txz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tbz2',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/archlinux-2020.02.01-x86_64.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/debian-8.11.1-amd64-DVD-1.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/debian-9.12.0-amd64-netinst.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/proxmox-ve_6.1-1.iso',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1580759863,
-                'format' => 'tar.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '19253',
-                'volid' => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1548098959,
-                'format' => 'tar',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '19254',
-                'volid' => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/hookscript.pl',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/userconfig.yaml',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, parent, non-matching',
-        # string instead of vmid in folder
-        #"$storage_dir/images/ssss/base-4321-disk-0.qcow2/1234/vm-1234-disk-0.qcow2",
-        vmid => undef,
-        files => [
-            "$storage_dir/images/1234/vm-1234-disk-0.qcow2",
-        ],
-        parent => [
-            "../ssss/base-4321-disk-0.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../ssss/base-4321-disk-0.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '1234',
-                'volid' => 'local:1234/vm-1234-disk-0.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, non-matching',
-        # failed matches
-        vmid => undef,
-        files => [
-            "$storage_dir/images/ssss/base-4321-disk-0.raw",
-            "$storage_dir/images/ssss/vm-1234-disk-0.qcow2",
-            "$storage_dir/template/iso/yet-again-a-installation-disk.dvd",
-            "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
-            "$storage_dir/private/subvol-19254-disk-0/19254",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
-        ],
-        expected => [], # returns empty list
-    },
-    {
-        description => 'VMID: none, valid file names for import',
-        vmid => undef,
-        files => [
-            "$storage_dir/import/import.ova",
-            "$storage_dir/import/import.ovf",
-            "$storage_dir/import/some-disk.qcow2",
-            "$storage_dir/import/some-disk.vmdk",
-            "$storage_dir/import/some-raw-disk.raw",
-        ],
-        expected => [
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ovf',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ovf",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-disk.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'vmdk',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-disk.vmdk",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'raw',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-raw-disk.raw",
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, non-matching file paths for import',
-        vmid => undef,
-        files => [
-            # Malformed file names
-            "$storage_dir/import/import.ovff",
-            "$storage_dir/import/importova",
-            "$storage_dir/import/import.ov",
-            "$storage_dir/import/diskraw",
-            "$storage_dir/import/diskvmdk",
-            "$storage_dir/import/disk.invalid",
-            "$storage_dir/import/.ova",
-            "$storage_dir/import/.raw",
-            # Trailing whitespace must not be trimmed
-            "$storage_dir/import/import.ova\t",
-            "$storage_dir/import/disk.raw    ",
-            # Whitespace in file name
-            "$storage_dir/import/something I want to import.ova",
-            "$storage_dir/import/ .raw",
-            "$storage_dir/import/ disk .vmdk",
-            "$storage_dir/import/disk .qcow2",
-            "$storage_dir/import/ import.ova",
-            # Unsafe characters in file name
-            "$storage_dir/import/linux🐧-vm.ova",
-            "$storage_dir/import/🐪perl-playground🐪.ova",
-            "$storage_dir/import/fish_<><_<><_<><.ova",
-            $storage_dir . '/import/C:\\\\Windows\\Path.ova',
-            # Content inside .ova files may only be specified as part
-            # of volume names, and may never appear when looked up as
-            # a file path
-            "$storage_dir/import/import.ova/disk.qcow2",
-            "$storage_dir/import/import.ova/disk.raw",
-            "$storage_dir/import/import.ova/disk.vmdk",
-            "$storage_dir/import/import.ova/disk.invalid",
-        ],
-        expected => [], # returns empty list
-    },
-    {
-        description => 'VMID: none, weird but valid file names for import',
-        vmid => undef,
-        files => [
-            "$storage_dir/import/import.ova.ova",
-            "$storage_dir/import/import.ova.ova.ova",
-            "$storage_dir/import/import.ova.ova.ova.ova",
-            "$storage_dir/import/ova.ova",
-            "$storage_dir/import/ova.ovf",
-            "$storage_dir/import/ova.vmdk",
-            "$storage_dir/import/raw.raw.qcow2",
-            "$storage_dir/import/raw.raw.qcow2.import.qcow2",
-            "$storage_dir/import/raw.raw.raw.your-boat.ova",
-        ],
-        expected => [
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ovf',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.ovf",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'vmdk',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.vmdk",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.qcow2.import.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.raw.your-boat.ova",
-            },
-        ],
-    },
-);
-
 =head2 TEST CASE FORMAT

 The parameters for individual test cases are hashes with the following
@@ -737,6 +260,655 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => 'VMID: 16112, lxc, raw, backup',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16112,
+        vtypes => ['rootdir', 'backup'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16112/vm-16112-disk-0.raw",
+                expected => {
+                    content => 'rootdir',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16112',
+                    volid => 'local:16112/vm-16112-disk-0.raw',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 16114, VM, qcow2, linked clone',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16114,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16114/vm-16114-disk-0.qcow2",
+                parent => '../9004/base-9004-disk-0.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../9004/base-9004-disk-0.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16114',
+                    volid => 'local:9004/base-9004-disk-0.qcow2/16114/vm-16114-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16114/vm-16114-disk-1.qcow2",
+                parent => '../9004/base-9004-disk-1.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../9004/base-9004-disk-1.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16114',
+                    volid => 'local:9004/base-9004-disk-1.qcow2/16114/vm-16114-disk-1.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 9004, VM, template, qcow2',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 9004,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/9004/base-9004-disk-0.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '9004',
+                    volid => 'local:9004/base-9004-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/9004/base-9004-disk-1.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '9004',
+                    volid => 'local:9004/base-9004-disk-1.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, templates, snippets, backup',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['vztmpl', 'iso', 'backup', 'snippets'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'txz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/archlinux-base_20190924-1_amd64.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-11.0-standard_11.0-1_amd64.tar.bz2",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tbz2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/archlinux-2020.02.01-x86_64.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/archlinux-2020.02.01-x86_64.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/debian-8.11.1-amd64-DVD-1.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/debian-8.11.1-amd64-DVD-1.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/debian-9.12.0-amd64-netinst.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/debian-9.12.0-amd64-netinst.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/proxmox-ve_6.1-1.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/proxmox-ve_6.1-1.iso',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => {
+                    content => 'backup',
+                    ctime => 1548098959,
+                    format => 'tar',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19254',
+                    volid => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/userconfig.yaml",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/userconfig.yaml',
+                },
+            },
+            {
+                # fileparse needs / at the end
+                file => "$DEFAULT_STORAGE_PATH/private/1234/",
+                expected => undef,
+            },
+            {
+                # fileparse needs / at the end
+                file => "$DEFAULT_STORAGE_PATH/private/1234/subvol-1234-disk-0.subvol/",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, parent, non-matching',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/1234/vm-1234-disk-0.qcow2",
+                parent => '../ssss/base-4321-disk-0.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../ssss/base-4321-disk-0.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '1234',
+                    volid => 'local:1234/vm-1234-disk-0.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => [sort keys $DEFAULT_SCFG->{content}->%*],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/ssss/base-4321-disk-0.raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/ssss/vm-1234-disk-0.qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/yet-again-a-installation-disk.dvd",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/private/subvol-19254-disk-0/19254",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, valid file names for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ovf",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ovf',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ovf',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-disk.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-disk.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-disk.vmdk",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-disk.vmdk',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-raw-disk.raw",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-raw-disk.raw',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching file paths for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            # Malformed file names
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ovff",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/importova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ov",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/diskraw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/diskvmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk.invalid",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/.raw",
+                expected => undef,
+            },
+
+            # Trailing whitespace must not be trimmed
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova\t",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk.raw    ",
+                expected => undef,
+            },
+
+            # Whitespace in file name
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/something I want to import.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ .raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ disk .vmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk .qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ import.ova",
+                expected => undef,
+            },
+
+            # Unsafe characters in file name
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/linux🐧-vm.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/🐪perl-playground🐪.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/fish_<><_<><_<><.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/C:\\\\Windows\\Path.ova",
+                expected => undef,
+            },
+
+            # Content inside .ova files may only be specified as part
+            # of volume names, and may never appear when looked up as
+            # a file path
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.vmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.invalid",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, weird but valid file names for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.ovf",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ovf',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.ovf',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.vmdk",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.vmdk',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.qcow2.import.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.qcow2.import.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.raw.your-boat.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.raw.your-boat.ova',
+                },
+            },
+        ],
+    },
 ];

 # provide static vmlist for tests
@@ -749,8 +921,8 @@ my $mock_stat = Test::MockModule->new('File::stat', no_auto => 1);
 $mock_stat->redefine(
     populate => sub {
         my (@st) = @_;
-        $st[7] = DEFAULT_SIZE;
-        $st[10] = DEFAULT_CTIME;
+        $st[7] = $DEFAULT_SIZE;
+        $st[10] = $DEFAULT_CTIME;

         my $result = $mock_stat->original('populate')->(@st);

@@ -765,8 +937,8 @@ $mock_fsi->redefine(
         my ($size, $format, $used, $parent, $ctime) =
             $mock_fsi->original('file_size_info')->(@_);

-        $size = DEFAULT_SIZE;
-        $used = DEFAULT_USED;
+        $size = $DEFAULT_SIZE;
+        $used = $DEFAULT_USED;

         return wantarray ? ($size, $format, $used, $parent, $ctime) : $size;
     },
@@ -776,53 +948,6 @@ my sub cmp_volinfo_by_volid {
     return $a->{volid} cmp $b->{volid};
 }

-my sub run_legacy_tests() {
-    my $sid = 'local';
-    my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
-
-    # run through test cases
-    foreach my $tt (@tests) {
-        my $vmid = $tt->{vmid};
-        my $files = $tt->{files};
-        my $expected = [sort cmp_volinfo_by_volid $tt->{expected}->@*];
-        my $description = $tt->{description};
-        my $parent = $tt->{parent};
-
-        # prepare environment
-        my $num = 0; #parent disks
-        for my $file (@$files) {
-            my ($name, $dir, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);
-
-            make_path($dir, { verbose => 1, mode => 0755 });
-
-            if ($name) {
-                # using qemu-img to also be able to represent the backing device
-                my @cmd = ('/usr/bin/qemu-img', 'create', "$file", DEFAULT_SIZE);
-                push @cmd, ('-f', $suffix) if $suffix;
-                push @cmd, ('-u', '-b', @$parent[$num]) if $parent;
-                push @cmd, ('-F', $suffix) if $parent && $suffix;
-                $num++;
-
-                run_command([@cmd]);
-            }
-        }
-
-        my $got = eval {
-            my $volume_list = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types);
-            return [sort cmp_volinfo_by_volid $volume_list->@*];
-        };
-        $got = $@ if $@;
-
-        is_deeply($got, $expected, $description) || diag(explain($got));
-
-        # clean up after each test case, otherwise
-        # we get wrong results from leftover files
-        remove_tree($storage_dir, { verbose => 1 });
-    }
-
-    return;
-}
-
 my sub setup_test_env($test_params) {
     my ($storeid, $scfg) = $test_params->@{qw(storeid scfg)};

@@ -913,7 +1038,7 @@ my sub assert_test_params_keys_exist($test_params) {
 }

 sub main() {
-    plan tests => scalar(@tests) + scalar($test_param_list->@*) + 1;
+    plan tests => scalar($test_param_list->@*) + 1;

     # Keep the original vmlist around in order to check whether it was modified
     # after running all the tests. See:
@@ -925,8 +1050,6 @@ sub main() {
         run_test_for_params($test_params);
     }

-    run_legacy_tests();
-
     my $vmlist = PVE::Cluster::get_vmlist();

     is_deeply(
--
2.47.3

[PATCH pve-storage v1 38/54] test: list volumes: document behavior wrt. undeclared content types

[Max R. Carrara]

Any content types / volume types that are passed to
`PVE::Storage::Plugin->list_volumes()` are still taken into account by
the method, even if the passed storage config does not declare them in
its `content` key. In other words, the `content` property is ignored
when listing volumes through `list_volumes()`.

Or to express this more explicitly, consider a directory storage with
`path` set to `/mnt/example` and `content` set to `iso,vztmpl`. Now,
if there are files in `/mnt/example/snippets` for some reason and the
`list_volumes()` method is called with `['snippets']` for the volume
type list parameter, then the volume info hashes for the files in
`/mnt/example/snippets` are included in the output.

Document this through a test case.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 109 ++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index ca3bbac7..455fb227 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -909,6 +909,115 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description =>
+            "VMID: none, volume info is still returned for content types that are not declared",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            content => {}, # note how no content types are declared here
+        },
+        vmid => undef,
+        vtypes => ['images', 'rootdir', 'vztmpl', 'iso', 'backup', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-0.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/1234/vm-1234-disk-0.qcow2",
+                parent => '../ssss/base-4321-disk-0.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../ssss/base-4321-disk-0.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '1234',
+                    volid => 'local:1234/vm-1234-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16112/vm-16112-disk-0.raw",
+                expected => {
+                    content => 'rootdir',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16112',
+                    volid => 'local:16112/vm-16112-disk-0.raw',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'txz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/archlinux-2020.02.01-x86_64.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/archlinux-2020.02.01-x86_64.iso',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova',
+                },
+            },
+        ],
+    },
 ];

 # provide static vmlist for tests
--
2.47.3

[PATCH pve-storage v1 39/54] plugin: correct comment in get_subdir_files helper

[Max R. Carrara]

The comment in the branch for 'backup' volume types in the
`get_subdir_files()` helper states that the condition below checks for
"false positives", apparently meaning that the VMID in the parent
directory might have been matched.

Challenge this claim by adding several new test cases that try to
cause the parser to return a wrong result.

As it turns out, the check the comment refers to is not (anymore)
there to check for "false positives" or VMIDs appearing in the parent
directory, but instead simply filters out backups that do not belong
to the provided VMID.

Note that if a backup has an arbitrary file name, that is, it's not
named something like "vzdump-qemu-1337-$TIMESTAMP.vma", that backup is
still returned in all cases.

Therefore, also include test cases for a plain "some-backup.tar.gz"
file and ensure that it is included in the expected output.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm     |   4 +-
 src/test/list_volumes_test.pm | 466 ++++++++++++++++++++++++++++++++++
 2 files changed, 468 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3853682b..101e0b6d 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1702,8 +1702,8 @@ my sub get_subdir_files {
             my $format = $parts->{ext};
             my $volume_path = $parts->{path};

-            # Check if parsed VMID matched provided VMID in order to avoid
-            # false positives (VMID in parent directory name)
+            # Check if parsed VMID matches provided VMID in order to avoid
+            # returning backups of other guests
             my $parsed_vmid = $parts->{vmid};
             if (defined($vmid) && defined($parsed_vmid)) {
                 return if $vmid ne $parsed_vmid;
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 455fb227..ce35c782 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -566,6 +566,284 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => 'VMID: none, backups of all guests',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602700,
+                    format => 'vma.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602765,
+                    format => 'vma.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => {
+                    content => 'backup',
+                    ctime => 1548098959,
+                    format => 'tar',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19254',
+                    volid => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
+                },
+            },
+            # Arbitrary backups are always included.
+            # Note that in this case, the 'vmid' key does not exist at all,
+            # instead of being set to undef.
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/some-backup.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'unknown',
+                    volid => 'local:backup/some-backup.tar.gz',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 16112, backups of specific guest',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16112,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => undef,
+            },
+            # Arbitrary backups are always included.
+            # In this case the 'vmid' also gets set to the provided one.
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/some-backup.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'unknown',
+                    vmid => 16112,
+                    volid => 'local:backup/some-backup.tar.gz',
+                },
+            },
+        ],
+    },
     {
         description => 'VMID: none, parent, non-matching',
         storeid => $DEFAULT_STOREID,
@@ -1020,6 +1298,194 @@ my $test_param_list = [
     },
 ];

+# Additional test cases that cannot be constructed within the list above
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in path of storage",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/dump/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir();
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in vtype subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in storage path and subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description =>
+            "VMID: 19253, file name with different VMID in storage path and subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 19253,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description =>
+            "VMID: none, file name in storage path and subdir of 'backup' vtype, no backups",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+                snippets => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 19253,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/snippets/hookscript.pl",
+                expected => undef,
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
 # provide static vmlist for tests
 my $mock_cluster = Test::MockModule->new('PVE::Cluster', no_auto => 1);
 $mock_cluster->redefine(get_vmlist => sub { return $mocked_vmlist; });
--
2.47.3

[PATCH pve-storage v1 40/54] test: parse volname: modernize code

[Max R. Carrara]

Modernize the existing code in `parse_volname_test.pm` while leaving
the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- fix a handful of typos
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 79 ++++++++++++++++++----------------
 1 file changed, 43 insertions(+), 36 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 2ff4c6d0..a03ac163 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestParseVolname;

-use strict;
-use warnings;
+use v5.36;

 use lib qw(..);

@@ -147,24 +146,24 @@ my $tests = [
         expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.qcow2',
         expected =>
             ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.vmdk',
         expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
     {
-        description => "Import, innner file of ova with whitespace in name",
+        description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.raw',
         expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
     },
@@ -214,7 +213,7 @@ my $tests = [

 # create more test cases for VM disk images matches
 my $disk_suffix = ['raw', 'qcow2', 'vmdk'];
-foreach my $s (@$disk_suffix) {
+for my $s ($disk_suffix->@*) {
     my @arr = (
         {
             description => "VM disk image, $s",
@@ -245,7 +244,7 @@ foreach my $s (@$disk_suffix) {
         },
     );

-    push @$tests, @arr;
+    push($tests->@*, @arr);
 }

 # create more test cases for backup files matches
@@ -255,9 +254,9 @@ my $bkp_suffix = {
     openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
 };

-foreach my $virt (keys %$bkp_suffix) {
+for my $virt (keys $bkp_suffix->%*) {
     my $suffix = $bkp_suffix->{$virt};
-    foreach my $s (@$suffix) {
+    for my $s ($suffix->@*) {
         my @arr = (
             {
                 description => "Backup archive, $virt, $s",
@@ -274,7 +273,7 @@ foreach my $virt (keys %$bkp_suffix) {
             },
         );

-        push @$tests, @arr;
+        push($tests->@*, @arr);
     }
 }

@@ -283,9 +282,9 @@ my $non_bkp_suffix = {
     qemu => ['vms.gz', 'vma.xz'],
     lxc => ['zip.gz', 'tgz.lzo'],
 };
-foreach my $virt (keys %$non_bkp_suffix) {
+for my $virt (keys $non_bkp_suffix->%*) {
     my $suffix = $non_bkp_suffix->{$virt};
-    foreach my $s (@$suffix) {
+    for my $s ($suffix->@*) {
         my @arr = (
             {
                 description => "Failed match: Backup archive, $virt, $s",
@@ -295,39 +294,47 @@ foreach my $virt (keys %$non_bkp_suffix) {
             },
         );

-        push @$tests, @arr;
+        push($tests->@*, @arr);
     }
 }

-#
-# run through test case array
-#
-plan tests => scalar @$tests + 1;
+my sub run_tests($tests) {
+    my $seen_vtype = {};
+    my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };

-my $seen_vtype;
-my $vtype_subdirs =
-    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
+    for my $t ($tests->@*) {
+        my $description = $t->{description};
+        my $volname = $t->{volname};
+        my $expected = $t->{expected};

-foreach my $t (@$tests) {
-    my $description = $t->{description};
-    my $volname = $t->{volname};
-    my $expected = $t->{expected};
+        my $got;
+        eval { $got = [PVE::Storage::Plugin->parse_volname($volname)] };
+        $got = $@ if $@;

-    my $got;
-    eval { $got = [PVE::Storage::Plugin->parse_volname($volname)] };
-    $got = $@ if $@;
+        is_deeply($got, $expected, $description);

-    is_deeply($got, $expected, $description);
+        $seen_vtype->{ $expected->[0] } = 1 if ref $expected eq 'ARRAY';
+    }

-    $seen_vtype->{ @$expected[0] } = 1 if ref $expected eq 'ARRAY';
+    # to check if all $vtype_subdirs are defined in path_to_volume_id
+    # or have a test
+    # FIXME re-enable after vtype split changes
+    #is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
+    is_deeply({}, {}, "vtype_subdir check");
+
+    return;
 }

-# to check if all $vtype_subdirs are defined in path_to_volume_id
-# or have a test
-# FIXME re-enable after vtype split changes
-#is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
-is_deeply({}, {}, "vtype_subdir check");
+my sub main() {
+    plan tests => scalar($tests->@*) + 1;

-done_testing();
+    run_tests($tests);
+
+    done_testing();
+
+    return;
+}
+
+main();

 1;
--
2.47.3

[PATCH pve-storage v1 41/54] test: parse volname: adapt tests regarding 'import' volume type

[Max R. Carrara]

Give the existing test cases unique 'description' keys and also
reorder them.

Add additional test cases for qcow2, vmdk and raw files.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index a03ac163..ea07c56b 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -146,27 +146,42 @@ my $tests = [
         expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
     },
     {
-        description => "Import, inner file of ova",
+        description => "Import, qcow2",
+        volname => 'import/import.qcow2',
+        expected => ['import', 'import.qcow2', undef, undef, undef, undef, 'qcow2'],
+    },
+    {
+        description => "Import, vmdk",
+        volname => 'import/import.vmdk',
+        expected => ['import', 'import.vmdk', undef, undef, undef, undef, 'vmdk'],
+    },
+    {
+        description => "Import, raw",
+        volname => 'import/import.raw',
+        expected => ['import', 'import.raw', undef, undef, undef, undef, 'raw'],
+    },
+    {
+        description => "Import, inner file of ova (qcow2)",
         volname => 'import/import.ova/disk.qcow2',
         expected =>
             ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
     },
     {
-        description => "Import, inner file of ova",
+        description => "Import, inner file of ova (vmdk)",
         volname => 'import/import.ova/disk.vmdk',
         expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
+    {
+        description => "Import, inner file of ova (raw)",
+        volname => 'import/import.ova/disk.raw',
+        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
+    },
     {
         description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
-    {
-        description => "Import, inner file of ova",
-        volname => 'import/import.ova/disk.raw',
-        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
-    },
     #
     # failed matches
     #
--
2.47.3

[PATCH pve-storage v1 42/54] test: parse volname: move VM disk test creation into separate block

[Max R. Carrara]

For future organizational purposes, move the loop that adds more
test cases for VM images into its own block.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 75 +++++++++++++++++++---------------
 1 file changed, 42 insertions(+), 33 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index ea07c56b..3777e6e8 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -226,40 +226,49 @@ my $tests = [
     },
 ];

-# create more test cases for VM disk images matches
-my $disk_suffix = ['raw', 'qcow2', 'vmdk'];
-for my $s ($disk_suffix->@*) {
-    my @arr = (
-        {
-            description => "VM disk image, $s",
-            volname => "$vmid/vm-$vmid-disk-1.$s",
-            expected => [
-                'images', "vm-$vmid-disk-1.$s", "$vmid", undef, undef, undef, "$s",
-            ],
-        },
-        {
-            description => "VM disk image, linked, $s",
-            volname => "$vmid/base-$vmid-disk-0.$s/$vmid/vm-$vmid-disk-0.$s",
-            expected => [
-                'images',
-                "vm-$vmid-disk-0.$s",
-                "$vmid",
-                "base-$vmid-disk-0.$s",
-                "$vmid",
-                undef,
-                "$s",
-            ],
-        },
-        {
-            description => "VM disk image, base, $s",
-            volname => "$vmid/base-$vmid-disk-0.$s",
-            expected => [
-                'images', "base-$vmid-disk-0.$s", "$vmid", undef, undef, 'base-', "$s",
-            ],
-        },
-    );
+# Additional test cases for VM disk images
+{
+    my $disk_suffixes = ['raw', 'qcow2', 'vmdk'];

-    push($tests->@*, @arr);
+    for my $suffix ($disk_suffixes->@*) {
+        my @extra_tests = (
+            {
+                description => "VM disk image, $suffix",
+                volname => "$vmid/vm-$vmid-disk-1.$suffix",
+                expected => [
+                    'images', "vm-$vmid-disk-1.$suffix", "$vmid", undef, undef, undef, $suffix,
+                ],
+            },
+            {
+                description => "VM disk image, linked, $suffix",
+                volname => "$vmid/base-$vmid-disk-0.$suffix/$vmid/vm-$vmid-disk-0.$suffix",
+                expected => [
+                    'images',
+                    "vm-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    "base-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    undef,
+                    $suffix,
+                ],
+            },
+            {
+                description => "VM disk image, base, $suffix",
+                volname => "$vmid/base-$vmid-disk-0.$suffix",
+                expected => [
+                    'images',
+                    "base-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    undef,
+                    undef,
+                    'base-',
+                    $suffix,
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
 }

 # create more test cases for backup files matches
--
2.47.3

[PATCH pve-storage v1 43/54] test: parse volname: move backup file test creation into sep. block

[Max R. Carrara]

Similar to the previous commit, move the code that adds additional
tests for backup file volnames into a separate block, for future
organizational purposes.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 88 +++++++++++++++++-----------------
 1 file changed, 45 insertions(+), 43 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 3777e6e8..fb300ad5 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -271,54 +271,56 @@ my $tests = [
     }
 }

-# create more test cases for backup files matches
-my $bkp_suffix = {
-    qemu => ['vma', 'vma.gz', 'vma.lzo', 'vma.zst'],
-    lxc => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst', 'tar.bz2'],
-    openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
-};
+# Additional tests for backup files
+{
+    my $bkp_suffixes = {
+        qemu => ['vma', 'vma.gz', 'vma.lzo', 'vma.zst'],
+        lxc => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst', 'tar.bz2'],
+        openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
+    };

-for my $virt (keys $bkp_suffix->%*) {
-    my $suffix = $bkp_suffix->{$virt};
-    for my $s ($suffix->@*) {
-        my @arr = (
-            {
-                description => "Backup archive, $virt, $s",
-                volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                expected => [
-                    'backup',
-                    "vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                    "$vmid",
-                    undef,
-                    undef,
-                    undef,
-                    'raw',
-                ],
-            },
-        );
+    for my $virt (keys $bkp_suffixes->%*) {
+        my $suffixes = $bkp_suffixes->{$virt};
+        for my $suffix ($suffixes->@*) {
+            my @extra_tests = (
+                {
+                    description => "Backup archive, $virt, $suffix",
+                    volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                    expected => [
+                        'backup',
+                        "vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                        "$vmid",
+                        undef,
+                        undef,
+                        undef,
+                        'raw',
+                    ],
+                },
+            );

-        push($tests->@*, @arr);
+            push($tests->@*, @extra_tests);
+        }
     }
-}

-# create more test cases for failed backup files matches
-my $non_bkp_suffix = {
-    qemu => ['vms.gz', 'vma.xz'],
-    lxc => ['zip.gz', 'tgz.lzo'],
-};
-for my $virt (keys $non_bkp_suffix->%*) {
-    my $suffix = $non_bkp_suffix->{$virt};
-    for my $s ($suffix->@*) {
-        my @arr = (
-            {
-                description => "Failed match: Backup archive, $virt, $s",
-                volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                expected =>
-                    "unable to parse directory volume name 'backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s'\n",
-            },
-        );
+    # Failed tests
+    my $non_bkp_suffixes = {
+        qemu => ['vms.gz', 'vma.xz'],
+        lxc => ['zip.gz', 'tgz.lzo'],
+    };
+    for my $virt (keys $non_bkp_suffixes->%*) {
+        my $suffixes = $non_bkp_suffixes->{$virt};
+        for my $suffix ($suffixes->@*) {
+            my @extra_tests = (
+                {
+                    description => "Failed match: Backup archive, $virt, $suffix",
+                    volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                    expected =>
+                        "unable to parse directory volume name 'backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix'\n",
+                },
+            );

-        push($tests->@*, @arr);
+            push($tests->@*, @extra_tests);
+        }
     }
 }

--
2.47.3

[PATCH pve-storage v1 44/54] test: parse volname: parameterize test case creation for some vtypes

[Max R. Carrara]

Currently, a lot of test cases are simply defined as part of the
overall list of tests. While this is completely fine in itself, it
does not scale if one wants to add cases that account for additional
parameters in the future.

Therefore, create a lot of existing test cases in blocks and push them
onto the list of test cases, instead of defining them up front.

Note that we already do this for test cases regarding VM disk / backup
file volume names.

This is done as a preparational step for future commits, primarily in
order to avoid writing a lot of test cases by hand.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 226 ++++++++++++++++-----------------
 1 file changed, 108 insertions(+), 118 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index fb300ad5..1dcec9d6 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -34,70 +34,6 @@ my $tests = [
         ],
     },
     #
-    # iso
-    #
-    {
-        description => 'ISO image, iso',
-        volname => 'iso/some-installation-disk.iso',
-        expected => ['iso', 'some-installation-disk.iso', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => 'ISO image, img',
-        volname => 'iso/some-other-installation-disk.img',
-        expected =>
-            ['iso', 'some-other-installation-disk.img', undef, undef, undef, undef, 'raw'],
-    },
-    #
-    # container templates
-    #
-    {
-        description => 'Container template tar',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar',
-        expected => [
-            'vztmpl', 'debian-10.0-standard_10.0-1_amd64.tar', undef, undef, undef, undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.gz',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.gz',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.xz',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.xz',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.bz2',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.bz2',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    #
     # container rootdir
     #
     {
@@ -120,62 +56,8 @@ my $tests = [
         ],
     },
     #
-    # Snippets
-    #
-    {
-        description => 'Snippets, yaml',
-        volname => 'snippets/userconfig.yaml',
-        expected => ['snippets', 'userconfig.yaml', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => 'Snippets, perl',
-        volname => 'snippets/hookscript.pl',
-        expected => ['snippets', 'hookscript.pl', undef, undef, undef, undef, 'raw'],
-    },
-    #
     # Import
     #
-    {
-        description => "Import, ova",
-        volname => 'import/import.ova',
-        expected => ['import', 'import.ova', undef, undef, undef, undef, 'ova'],
-    },
-    {
-        description => "Import, ovf",
-        volname => 'import/import.ovf',
-        expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
-    },
-    {
-        description => "Import, qcow2",
-        volname => 'import/import.qcow2',
-        expected => ['import', 'import.qcow2', undef, undef, undef, undef, 'qcow2'],
-    },
-    {
-        description => "Import, vmdk",
-        volname => 'import/import.vmdk',
-        expected => ['import', 'import.vmdk', undef, undef, undef, undef, 'vmdk'],
-    },
-    {
-        description => "Import, raw",
-        volname => 'import/import.raw',
-        expected => ['import', 'import.raw', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => "Import, inner file of ova (qcow2)",
-        volname => 'import/import.ova/disk.qcow2',
-        expected =>
-            ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
-    },
-    {
-        description => "Import, inner file of ova (vmdk)",
-        volname => 'import/import.ova/disk.vmdk',
-        expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
-    },
-    {
-        description => "Import, inner file of ova (raw)",
-        volname => 'import/import.ova/disk.raw',
-        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
-    },
     {
         description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
@@ -271,6 +153,48 @@ my $tests = [
     }
 }

+# Test cases for ISOs
+{
+    my $suffixes = ['iso', 'img'];
+    my $prefix = 'some-installation-disk';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "ISO image, $suffix",
+                volname => "iso/$file_name",
+                expected => ['iso', "$file_name", undef, undef, undef, undef, 'raw'],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for container templates
+{
+    my $suffixes = ['tar', 'tar.gz', 'tar.xz', 'tar.bz2'];
+    my $prefix = 'debian-10.0-standard_10.0-1_amd64';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "Container template, $suffix",
+                volname => "vztmpl/$file_name",
+                expected => [
+                    'vztmpl', "$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
 # Additional tests for backup files
 {
     my $bkp_suffixes = {
@@ -324,6 +248,72 @@ my $tests = [
     }
 }

+# Test cases for snippets
+{
+    my $file_names = ['userconfig.yaml', 'hookscript.pl'];
+
+    for my $file_name ($file_names->@*) {
+        my @extra_tests = (
+            {
+                description => "Snippets, $file_name",
+                volname => "snippets/$file_name",
+                expected => ['snippets', $file_name, undef, undef, undef, undef, 'raw'],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for import files
+{
+    my $suffixes = ['ova', 'ovf', 'qcow2', 'vmdk', 'raw'];
+    my $prefix = 'import-file';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "Import, $suffix",
+                volname => "import/$file_name",
+                expected => ['import', "$file_name", undef, undef, undef, undef, $suffix],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for OVA import files with content inside
+{
+    my $content_suffixes = ['qcow2', 'vmdk', 'raw'];
+    my $content_prefix = 'disk';
+    my $file_name = 'import.ova';
+
+    for my $content_suffix ($content_suffixes->@*) {
+        my $content_file_name = "$content_prefix.$content_suffix";
+
+        my @extra_tests = (
+            {
+                description => "Import, inner file of ova ($content_suffix)",
+                volname => "import/$file_name/$content_file_name",
+                expected => [
+                    'import',
+                    "$file_name/$content_file_name",
+                    undef,
+                    undef,
+                    undef,
+                    undef,
+                    "ova+$content_suffix",
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
 my sub run_tests($tests) {
     my $seen_vtype = {};
     my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };
--
2.47.3

[PATCH pve-storage v1 45/54] test: volume id: modernize code

[Max R. Carrara]

Modernize the existing code in `path_to_volume_id_test.pm` while
leaving the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
  - remove needless `diag(explain(...))` call
  - suppress warnings when calling `path_to_volume_id()` to make test
    output less noisy
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- capitalize constants
- move the comment regarding `File::Temp->newdir()` unlinking on exit
  to where the method is actually called
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/path_to_volume_id_test.pm | 149 ++++++++++++++++-------------
 1 file changed, 82 insertions(+), 67 deletions(-)

diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index e7e36037..72644482 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestPathToVolumeId;

-use strict;
-use warnings;
+use v5.36;

 use lib qw(..);

@@ -13,17 +12,17 @@ use PVE::Storage::Common qw(
 use Test::More;

 use Cwd;
-use File::Basename;
+use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
 use File::Temp;

-my $storage_dir = File::Temp->newdir();
-my $scfg = {
+my $DEFAULT_STORAGE_DIR = File::Temp->newdir(); # unlinks on exit
+my $DEFAULT_CFG = {
     'digest' => 'd29306346b8b25b90a4a96165f1e8f52d1af1eda',
     'ids' => {
         'local' => {
             'shared' => 0,
-            'path' => "$storage_dir",
+            'path' => "$DEFAULT_STORAGE_DIR",
             'type' => 'dir',
             'content' => {
                 'snippets' => 1,
@@ -44,24 +43,24 @@ my $scfg = {
 # description => to identify the test case
 # volname     => to create the test file
 # expected    => the result that path_to_volume_id should return
-my @tests = (
+my $tests = [
     {
         description => 'Image, qcow2',
-        volname => "$storage_dir/images/16110/vm-16110-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
         expected => [
             'images', 'local:16110/vm-16110-disk-0.qcow2',
         ],
     },
     {
         description => 'Image, raw',
-        volname => "$storage_dir/images/16112/vm-16112-disk-0.raw",
+        volname => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
         expected => [
             'images', 'local:16112/vm-16112-disk-0.raw',
         ],
     },
     {
         description => 'Image template, qcow2',
-        volname => "$storage_dir/images/9004/base-9004-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
         expected => [
             'images', 'local:9004/base-9004-disk-0.qcow2',
         ],
@@ -69,49 +68,49 @@ my @tests = (

     {
         description => 'Backup, vma.gz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
         ],
     },
     {
         description => 'Backup, vma.lzo',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
         ],
     },
     {
         description => 'Backup, vma',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
         ],
     },
     {
         description => 'Backup, tar.lzo',
-        volname => "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
         ],
     },
     {
         description => 'Backup, vma.zst',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
         ],
     },
     {
         description => 'Backup, tar.zst',
-        volname => "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst',
         ],
     },
     {
         description => 'Backup, tar.bz2',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
         expected => [
             'backup', 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
         ],
@@ -119,21 +118,23 @@ my @tests = (

     {
         description => 'ISO file',
-        volname => "$storage_dir/template/iso/yet-again-a-installation-disk.iso",
+        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
         expected => [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
     {
         description => 'CT template, tar.gz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
     {
         description => 'CT template, wrong ending, tar bz2',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
         ],
@@ -141,49 +142,50 @@ my @tests = (

     {
         description => 'Rootdir, folder subvol, legacy naming',
-        volname => "$storage_dir/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Rootdir, folder subvol',
-        volname => "$storage_dir/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Snippets, yaml',
-        volname => "$storage_dir/snippets/userconfig.yaml",
+        volname => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
         expected => [
             'snippets', 'local:snippets/userconfig.yaml',
         ],
     },
     {
         description => 'Snippets, hookscript',
-        volname => "$storage_dir/snippets/hookscript.pl",
+        volname => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
         expected => [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
     {
         description => 'CT template, tar.xz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
         ],
     },
     {
         description => 'Import, ova',
-        volname => "$storage_dir/import/import.ova",
+        volname => "$DEFAULT_STORAGE_DIR/import/import.ova",
         expected => [
             'import', 'local:import/import.ova',
         ],
     },
     {
         description => 'Import, ovf',
-        volname => "$storage_dir/import/import.ovf",
+        volname => "$DEFAULT_STORAGE_DIR/import/import.ovf",
         expected => [
             'import', 'local:import/import.ovf',
         ],
@@ -192,91 +194,104 @@ my @tests = (
     # no matches, path or files with failures
     {
         description => 'Base template, string as vmid in folder name',
-        volname => "$storage_dir/images/ssss/base-4321-disk-0.raw",
+        volname => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
         expected => [''],
     },
     {
         description => 'ISO file, wrong ending',
-        volname => "$storage_dir/template/iso/yet-again-a-installation-disk.dvd",
+        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
         expected => [''],
     },
     {
         description => 'CT template, wrong ending, zip.gz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, zip.gz',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, tgz.lzo',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
         expected => [''],
     },
     {
         description => 'Backup, wrong ending, qemu, vma.xz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, qemu, vms.gz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
     {
         description => 'Image, string as vmid in folder name',
-        volname => "$storage_dir/images/ssss/vm-1234-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
-        volname => "$storage_dir/import/test.foo",
+        volname => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
-);
+];

-plan tests => scalar @tests + 1;
+my sub run_tests($tests) {
+    my $seen_vtype;
+    my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };

-my $seen_vtype;
-my $vtype_subdirs =
-    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
+    for my $t ($tests->@*) {
+        my $file = $t->{volname};
+        my $expected = $t->{expected};
+        my $description = $t->{description};

-foreach my $tt (@tests) {
-    my $file = $tt->{volname};
-    my $expected = $tt->{expected};
-    my $description = $tt->{description};
+        # prepare environment
+        my ($name, $dir, $suffix) = fileparse($file);
+        make_path($dir, { verbose => 1, mode => 0755 });

-    # prepare environment
-    my ($name, $dir, $suffix) = fileparse($file);
-    make_path($dir, { verbose => 1, mode => 0755 });
+        if ($name) {
+            open(my $fh, ">>", "$file") || die "Error open file: $!";
+            close($fh);
+        }

-    if ($name) {
-        open(my $fh, ">>", "$file") || die "Error open file: $!";
-        close($fh);
+        my $got;
+        eval {
+            # Suppress warnings here to make output less noisy for certain tests
+            local $SIG{__WARN__} = sub { };
+
+            $got = [PVE::Storage::path_to_volume_id($DEFAULT_CFG, $file)];
+        };
+        $got = $@ if $@;
+
+        is_deeply($got, $expected, $description);
+
+        $seen_vtype->{ $expected->[0] } = 1
+            if ($expected->[0] ne '' && scalar($expected->@*) > 1);
     }

-    # run tests
-    my $got;
-    eval { $got = [PVE::Storage::path_to_volume_id($scfg, $file)] };
-    $got = $@ if $@;
+    # to check if all $vtype_subdirs are defined in path_to_volume_id
+    # or have a test
+    # FIXME re-enable after vtype split changes
+    #is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
+    is_deeply({}, {}, "vtype_subdir check");

-    is_deeply($got, $expected, $description) || diag(explain($got));
-
-    $seen_vtype->{ @$expected[0] } = 1
-        if (@$expected[0] ne '' && scalar @$expected > 1);
+    return;
 }

-# to check if all $vtype_subdirs are defined in path_to_volume_id
-# or have a test
-# FIXME re-enable after vtype split changes
-#is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
-is_deeply({}, {}, "vtype_subdir check");
+my sub main() {
+    plan tests => scalar($tests->@*) + 1;

-#cleanup
-# File::Temp unlinks tempdir on exit
+    run_tests($tests);

-done_testing();
+    done_testing();
+
+    return;
+}
+
+main();

 1;
--
2.47.3

[PATCH pve-storage v1 46/54] test: volume id: rename 'volname' test case parameter to 'file'

[Max R. Carrara]

... since it's for file paths, not volume names.

Improve the comment describing the test case format along the way.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/path_to_volume_id_test.pm | 72 ++++++++++++++----------------
 1 file changed, 34 insertions(+), 38 deletions(-)

diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 72644482..6c7d0d67 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -39,28 +39,28 @@ my $DEFAULT_CFG = {
     },
 };

-# the tests array consists of hashes with the following keys:
-# description => to identify the test case
-# volname     => to create the test file
-# expected    => the result that path_to_volume_id should return
+# Each test is comprised of the following keys:
+# description => to identify a single test
+# file        => the absolute path to the volume's file that should be parsed
+# expected    => expected output of path_to_volume_id(), as listref
 my $tests = [
     {
         description => 'Image, qcow2',
-        volname => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
         expected => [
             'images', 'local:16110/vm-16110-disk-0.qcow2',
         ],
     },
     {
         description => 'Image, raw',
-        volname => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
+        file => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
         expected => [
             'images', 'local:16112/vm-16112-disk-0.raw',
         ],
     },
     {
         description => 'Image template, qcow2',
-        volname => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
         expected => [
             'images', 'local:9004/base-9004-disk-0.qcow2',
         ],
@@ -68,49 +68,49 @@ my $tests = [

     {
         description => 'Backup, vma.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
         ],
     },
     {
         description => 'Backup, vma.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
         ],
     },
     {
         description => 'Backup, vma',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
         ],
     },
     {
         description => 'Backup, tar.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
         ],
     },
     {
         description => 'Backup, vma.zst',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
         ],
     },
     {
         description => 'Backup, tar.zst',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst',
         ],
     },
     {
         description => 'Backup, tar.bz2',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
         expected => [
             'backup', 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
         ],
@@ -118,23 +118,21 @@ my $tests = [

     {
         description => 'ISO file',
-        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
+        file => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
         expected => [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
     {
         description => 'CT template, tar.gz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
     {
         description => 'CT template, wrong ending, tar bz2',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
         ],
@@ -142,50 +140,49 @@ my $tests = [

     {
         description => 'Rootdir, folder subvol, legacy naming',
-        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        file => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Rootdir, folder subvol',
-        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        file => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Snippets, yaml',
-        volname => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
+        file => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
         expected => [
             'snippets', 'local:snippets/userconfig.yaml',
         ],
     },
     {
         description => 'Snippets, hookscript',
-        volname => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
+        file => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
         expected => [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
     {
         description => 'CT template, tar.xz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
         ],
     },
     {
         description => 'Import, ova',
-        volname => "$DEFAULT_STORAGE_DIR/import/import.ova",
+        file => "$DEFAULT_STORAGE_DIR/import/import.ova",
         expected => [
             'import', 'local:import/import.ova',
         ],
     },
     {
         description => 'Import, ovf',
-        volname => "$DEFAULT_STORAGE_DIR/import/import.ovf",
+        file => "$DEFAULT_STORAGE_DIR/import/import.ovf",
         expected => [
             'import', 'local:import/import.ovf',
         ],
@@ -194,48 +191,47 @@ my $tests = [
     # no matches, path or files with failures
     {
         description => 'Base template, string as vmid in folder name',
-        volname => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
+        file => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
         expected => [''],
     },
     {
         description => 'ISO file, wrong ending',
-        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
+        file => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
         expected => [''],
     },
     {
         description => 'CT template, wrong ending, zip.gz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, zip.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, tgz.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
         expected => [''],
     },
     {
         description => 'Backup, wrong ending, qemu, vma.xz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, qemu, vms.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
     {
         description => 'Image, string as vmid in folder name',
-        volname => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
-        volname => "$DEFAULT_STORAGE_DIR/import/test.foo",
+        file => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
 ];
@@ -245,7 +241,7 @@ my sub run_tests($tests) {
     my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };

     for my $t ($tests->@*) {
-        my $file = $t->{volname};
+        my $file = $t->{file};
         my $expected = $t->{expected};
         my $description = $t->{description};

--
2.47.3

[PATCH pve-storage v1 47/54] test: filesystem path: modernize code

[Max R. Carrara]

Modernize the existing code in `filesystem_path_test.pm` while leaving
the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
  - remove needless `diag(explain(...))` call
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- rename / capitalize the sole constant
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/filesystem_path_test.pm | 53 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 21 deletions(-)

diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index af52380c..5a715ce9 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -1,14 +1,14 @@
 package PVE::Storage::TestFilesystemPath;

-use strict;
-use warnings;
+use v5.36;

 use lib qw(..);

 use PVE::Storage;
+
 use Test::More;

-my $path = '/some/path';
+my $DEFAULT_STORAGE_DIR = '/some/path';

 # each array entry is a test that consists of the following keys:
 # volname  => image name that is passed to parse_volname
@@ -19,7 +19,7 @@ my $tests = [
         volname => '1234/vm-1234-disk-0.raw',
         snapname => undef,
         expected => [
-            "$path/images/1234/vm-1234-disk-0.raw", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.raw", '1234', 'images',
         ],
     },
     {
@@ -31,49 +31,60 @@ my $tests = [
         volname => '1234/vm-1234-disk-0.qcow2',
         snapname => undef,
         expected => [
-            "$path/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
         ],
     },
     {
         volname => '1234/vm-1234-disk-0.qcow2',
         snapname => 'my_snap',
         expected => [
-            "$path/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
         ],
     },
     {
         volname => 'iso/my-awesome-proxmox.iso',
         snapname => undef,
         expected => [
-            "$path/template/iso/my-awesome-proxmox.iso", undef, 'iso',
+            "$DEFAULT_STORAGE_DIR/template/iso/my-awesome-proxmox.iso", undef, 'iso',
         ],
     },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
         expected => [
-            "$path/dump/vzdump-qemu-1234-2020_03_30-21_12_40.vma", 1234, 'backup',
+            "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-1234-2020_03_30-21_12_40.vma", 1234,
+            'backup',
         ],
     },
 ];

-plan tests => scalar @$tests;
+my sub run_tests($tests) {
+    for my $t ($tests->@*) {
+        my $volname = $t->{volname};
+        my $snapname = $t->{snapname};
+        my $expected = $t->{expected};
+        my $scfg = { path => $DEFAULT_STORAGE_DIR };
+        my $got;

-foreach my $tt (@$tests) {
-    my $volname = $tt->{volname};
-    my $snapname = $tt->{snapname};
-    my $expected = $tt->{expected};
-    my $scfg = { path => $path };
-    my $got;
+        eval { $got = [PVE::Storage::Plugin->filesystem_path($scfg, $volname, $snapname)]; };
+        $got = $@ if $@;

-    eval { $got = [PVE::Storage::Plugin->filesystem_path($scfg, $volname, $snapname)]; };
-    $got = $@ if $@;
-
-    is_deeply($got, $expected, "wantarray: filesystem_path for $volname")
-        || diag(explain($got));
+        is_deeply($got, $expected, "wantarray: filesystem_path for $volname");
+    }

+    return;
 }

-done_testing();
+my sub main() {
+    plan tests => scalar($tests->@*);
+
+    run_tests($tests);
+
+    done_testing();
+
+    return;
+}
+
+main();

 1;
--
2.47.3

[PATCH pve-storage v1 48/54] fix #2884: implement nested subdir scanning and support 'iso' vtype

[Max R. Carrara]

Introduce a new 'max-scan-depth' property for directory-based
storages. This property makes it so that files for specific volume
types may also be put into (nested) subdirectories inside the volume
type's directory.

The maximum allowed depth is between 0 and 50, with 0 being the
default, which corresponds to the current behavior.

Support nested subdirectories for ISOs (the 'iso' vtype) only for now.

Achieve all of this by calling the `get_subdir_files()` helper in
`PVE::Storage::Plugin` recursively. Note that the default recursion
limit in Perl appears to be exactly 100 [0], way above the maximum
value of 50 for the 'max-scan-depth' property. Should we need deeper
nesting, we can always make the subroutine iterative instead of
recursive later.

Add additional test cases wherever applicable to account for nested
subdirectories, including cases that check whether the limit set by
the property is upheld, and also cases that check for the existence of
parent directory references ('..') in volume names.

[0]: https://perldoc.perl.org/perl5101delta#Deep-recursion-on-subroutine-%22%25s%22

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/BTRFSPlugin.pm              |   1 +
 src/PVE/Storage/CephFSPlugin.pm             |   1 +
 src/PVE/Storage/Common/Parse.pm             |  14 +++
 src/PVE/Storage/Common/test/parser_tests.pl |  50 ++++++++
 src/PVE/Storage/DirPlugin.pm                |   1 +
 src/PVE/Storage/Plugin.pm                   |  41 ++++++-
 src/test/filesystem_path_test.pm            |   8 ++
 src/test/list_volumes_test.pm               | 127 ++++++++++++++++++++
 src/test/parse_volname_test.pm              |  56 +++++++++
 src/test/path_to_volume_id_test.pm          |   7 ++
 10 files changed, 302 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index 67b442f6..0a29fd31 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -68,6 +68,7 @@ sub properties {
 sub options {
     return {
         path => { fixed => 1 },
+        'max-scan-depth' => { optional => 1 },
         nodes => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index fbc97113..b31f53d9 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -140,6 +140,7 @@ sub options {
     return {
         path => { fixed => 1 },
         'content-dirs' => { optional => 1 },
+        'max-scan-depth' => { optional => 1 },
         monhost => { optional => 1 },
         nodes => { optional => 1 },
         subdir => { optional => 1 },
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 9fb45b0c..f98b702d 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -57,6 +57,10 @@ my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;

 my $RE_SAFE_CHAR_WITH_WHITESPACE_CLASS = qr/[ a-zA-Z0-9\-\.\+\=\_]/;

+my $RE_DIRECTORY_COMPONENTS = qr!
+    ( ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+ / )*?  # NOTE: non-greedy matching
+!xn;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -73,6 +77,7 @@ my $RE_VMID = qr![1-9][0-9]{2,8}!;

 my $RE_ISO_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
     )
 !xn;
@@ -206,6 +211,15 @@ my sub format_named_groups(%groups) {

     my @disk_path_components = ();

+    if (defined($result->{dir})) {
+        if ($result->{dir} eq '') {
+            delete $result->{dir};
+        } else {
+            $result->{dir} = strip_trailing_path_separators($result->{dir});
+            push(@disk_path_components, $result->{dir});
+        }
+    }
+
     if (defined($result->{file})) {
         $result->{file} = strip_leading_path_separators($result->{file});
         push(@disk_path_components, $result->{file});
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index b6e42307..ad122cd2 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -73,6 +73,32 @@ my $volname_cases_iso_valid = [
             volname => 'iso/Fedora.Img',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/custom-debian.iso',
+        expected => {
+            file => 'custom-debian.iso',
+            ext => 'iso',
+            dir => 'subdir',
+            'disk-path' => 'subdir/custom-debian.iso',
+            path => 'subdir/custom-debian.iso',
+            vtype => 'iso',
+            volname => 'iso/subdir/custom-debian.iso',
+        },
+    },
+    {
+        path => 'deeply/nested/dir/hannah-montana-linux.IMG',
+        expected => {
+            file => 'hannah-montana-linux.IMG',
+            ext => 'IMG',
+            dir => 'deeply/nested/dir',
+            'disk-path' => 'deeply/nested/dir/hannah-montana-linux.IMG',
+            path => 'deeply/nested/dir/hannah-montana-linux.IMG',
+            vtype => 'iso',
+            volname => 'iso/deeply/nested/dir/hannah-montana-linux.IMG',
+        },
+    },
 ];

 my $volname_cases_iso_invalid = [
@@ -84,6 +110,30 @@ my $volname_cases_iso_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Parent dir reference in path (beginning) (iso)",
+        args => {
+            path => '../custom-debian.iso',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (iso)",
+        args => {
+            path => 'subdir/../custom-debian.iso',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (iso)",
+        args => {
+            path => 'subdir/custom-debian.iso/..',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
 ];

 my $volname_cases_vztmpl_valid = [
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 80c4a031..58b942e7 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -81,6 +81,7 @@ sub options {
     return {
         path => { fixed => 1 },
         'content-dirs' => { optional => 1 },
+        'max-scan-depth' => { optional => 1 },
         nodes => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 101e0b6d..0fbcbd4b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -233,6 +233,15 @@ my $defaultData = {
             format => "pve-dir-override-list",
             optional => 1,
         },
+        'max-scan-depth' => {
+            description => "Maximum depth of subdirectories to traverse when searching for"
+                . " ISOs in directories.",
+            type => 'integer',
+            default => 0,
+            minimum => 0,
+            maximum => 50,
+            optional => 1,
+        },
         options => {
             description => "NFS/CIFS mount options (see 'man nfs' or 'man mount.cifs')",
             type => 'string',
@@ -1669,10 +1678,16 @@ sub list_images {

 # $vtype = <iso|vztmpl|backup|snippets|import>
 my sub get_subdir_files {
-    my ($storeid, $scfg, $vtype, $vmid) = @_;
+    use feature 'current_sub'; # Needed for the __SUB__ token further below
+
+    my ($storeid, $scfg, $vtype, $vmid, $remaining_depth, $current_path) = @_;

     my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);

+    if (!defined($current_path)) {
+        $current_path = $vtype_subdir;
+    }
+
     my $res = [];

     my $get_subdir_file_info = sub {
@@ -1752,10 +1767,24 @@ my sub get_subdir_files {
         return;
     };

-    for my $path (<$vtype_subdir/*>) {
+    for my $path (<$current_path/*>) {
         my $st = File::stat::stat($path);

-        next if (!$st || S_ISDIR($st->mode));
+        next if !$st;
+
+        if (S_ISDIR($st->mode)) {
+            if (defined($remaining_depth) && $remaining_depth > 0) {
+                # Note: Have to call the subroutine via __SUB__->(...) because
+                # lexical subroutines otherwise cannot reference themselves
+                my $inner_res = __SUB__->(
+                    $storeid, $scfg, $vtype, $vmid, $remaining_depth - 1, $path,
+                );
+
+                push($res->@*, $inner_res->@*);
+            }
+
+            next;
+        }

         if (defined(my $info = $get_subdir_file_info->($path, $st))) {
             $info->{size} = $st->size;
@@ -1773,6 +1802,10 @@ my sub get_subdir_files {
 sub list_volumes {
     my ($class, $storeid, $scfg, $vmid, $content_types) = @_;

+    my $depth = $scfg->{'max-scan-depth'} // 0;
+    $depth = 0 if $depth < 0;
+    $depth = 50 if $depth > 50;
+
     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();

@@ -1786,7 +1819,7 @@ sub list_volumes {
         return if !$scfg->{path};

         if ($type eq 'iso' && !defined($vmid)) {
-            return get_subdir_files($storeid, $scfg, 'iso', undef);
+            return get_subdir_files($storeid, $scfg, 'iso', undef, $depth);
         }

         if ($type eq 'vztmpl' && !defined($vmid)) {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index 5a715ce9..26a74a0d 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -48,6 +48,14 @@ my $tests = [
             "$DEFAULT_STORAGE_DIR/template/iso/my-awesome-proxmox.iso", undef, 'iso',
         ],
     },
+    {
+        volname => 'iso/foo/bar/baz/my-awesome-proxmox.iso',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/iso/foo/bar/baz/my-awesome-proxmox.iso", undef,
+            'iso',
+        ],
+    },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index ce35c782..0daaba94 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1296,6 +1296,133 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => "VMID: none, no nested subdirectories when using defaults",
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => "VMID: none, nested subdirectories allowed, max-scan-depth = 1",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            'max-scan-depth' => 1,
+            content => {
+                iso => 1,
+                vztmpl => 1,
+                snippets => 1,
+                import => 1,
+            },
+        },
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/some-installer.iso',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => "VMID: none, nested subdirectories allowed, max-scan-depth = 5",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            'max-scan-depth' => 5,
+            content => {
+                iso => 1,
+                vztmpl => 1,
+                snippets => 1,
+                import => 1,
+            },
+        },
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/2/3/4/5/some-installer.iso',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/6/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
 ];

 # Additional test cases that cannot be constructed within the list above
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 1dcec9d6..24d1ed0e 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -167,10 +167,66 @@ my $tests = [
                 volname => "iso/$file_name",
                 expected => ['iso', "$file_name", undef, undef, undef, undef, 'raw'],
             },
+            {
+                description => "ISO image, $suffix, subdirectory",
+                volname => "iso/foo/$file_name",
+                expected => [
+                    'iso', "foo/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "ISO image, $suffix, nested subdirectories",
+                volname => "iso/foo/bar/baz/$file_name",
+                expected => [
+                    'iso', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "ISO image, $suffix, subdirectory with same name as file",
+                volname => "iso/$file_name/$file_name",
+                expected => [
+                    'iso', "$file_name/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
         );

         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.iso";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "ISO image, iso, parent directory reference before volume type prefix",
+                volname => "../iso/$file_name",
+                expected => "unable to parse directory volume name '../iso/$file_name'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference at beginning of volume path",
+                volname => "iso/../$file_name",
+                expected => "unable to parse directory volume name 'iso/../$file_name'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference at end of volume path",
+                volname => "iso/$file_name/..",
+                expected => "unable to parse directory volume name 'iso/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference between dir components of volume path",
+                volname => "iso/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'iso/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }

 # Test cases for container templates
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 6c7d0d67..bc87d289 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -123,6 +123,13 @@ my $tests = [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
+    {
+        description => 'ISO file, nested subdirectories',
+        file => "$DEFAULT_STORAGE_DIR/template/iso/foo/bar/hannah-montana-linux-installer.iso",
+        expected => [
+            'iso', 'local:iso/foo/bar/hannah-montana-linux-installer.iso',
+        ],
+    },
     {
         description => 'CT template, tar.gz',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
--
2.47.3

[PATCH pve-storage v1 49/54] fix #2884: support nested subdir scanning for 'vztmpl' volume type

[Max R. Carrara]

Add support for parsing nested subdirectories for the 'vztmpl' volume
type by adapting its corresponding regex used by parsing helpers.

Add additional test cases wherever applicable to account for nested
subdirectories for 'vztmpl' volumes, as done for the 'iso' volume
type.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  1 +
 src/PVE/Storage/Common/test/parser_tests.pl | 54 ++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  4 +-
 src/test/filesystem_path_test.pm            | 18 +++++
 src/test/list_volumes_test.pm               | 81 +++++++++++++++++++++
 src/test/parse_volname_test.pm              | 57 +++++++++++++++
 src/test/path_to_volume_id_test.pm          |  8 ++
 7 files changed, 221 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index f98b702d..92e5b0fd 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -84,6 +84,7 @@ my $RE_ISO_FILE_PATH = qr!

 my $RE_VZTMPL_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file>
             [^/]+
             \.
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index ad122cd2..c80d0d90 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -230,6 +230,36 @@ my $volname_cases_vztmpl_valid = [
             volname => 'vztmpl/Alpine 3.10 default_20190626_amd64.Tar.xZ',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        expected => {
+            file => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            dir => 'subdir',
+            'disk-path' => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+            path => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        },
+    },
+    {
+        path => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        expected => {
+            file => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            dir => 'deeply/nested/dir',
+            'disk-path' => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            path => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        },
+    },
 ];

 my $volname_cases_vztmpl_invalid = [
@@ -241,6 +271,30 @@ my $volname_cases_vztmpl_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Parent dir reference in path (beginning) (vztmpl)",
+        args => {
+            path => '../archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (vztmpl)",
+        args => {
+            path => 'subdir/../archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (vztmpl)",
+        args => {
+            path => 'subdir/archlinux-base_20190924-1_amd64.tar.gz/..',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
 ];

 my $volname_cases_backup_valid = [
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 0fbcbd4b..2df56e76 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -235,7 +235,7 @@ my $defaultData = {
         },
         'max-scan-depth' => {
             description => "Maximum depth of subdirectories to traverse when searching for"
-                . " ISOs in directories.",
+                . " ISOs and container templates in directories.",
             type => 'integer',
             default => 0,
             minimum => 0,
@@ -1823,7 +1823,7 @@ sub list_volumes {
         }

         if ($type eq 'vztmpl' && !defined($vmid)) {
-            return get_subdir_files($storeid, $scfg, 'vztmpl', undef);
+            return get_subdir_files($storeid, $scfg, 'vztmpl', undef, $depth);
         }

         if ($type eq 'backup') {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index 26a74a0d..b5c1ab33 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -56,6 +56,24 @@ my $tests = [
             'iso',
         ],
     },
+    {
+        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+            undef,
+            'vztmpl',
+        ],
+    },
+    {
+        volname => 'vztmpl/foo/bar/baz/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/cache/foo/bar/baz/debian-10.0-standard_10.0-1_amd64.tar.gz",
+            undef,
+            'vztmpl',
+        ],
+    },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 0daaba94..6bdfa37c 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1321,6 +1321,25 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1366,6 +1385,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1421,6 +1465,43 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/6/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/2/3/4/5/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/6/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 24d1ed0e..b90815c2 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -245,10 +245,67 @@ my $tests = [
                     'vztmpl', "$file_name", undef, undef, undef, undef, 'raw',
                 ],
             },
+            {
+                description => "Container template, $suffix, subdirectory",
+                volname => "vztmpl/foo/$file_name",
+                expected => [
+                    'vztmpl', "foo/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "Container template, $suffix, nested subdirectories",
+                volname => "vztmpl/foo/bar/baz/$file_name",
+                expected => [
+                    'vztmpl', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description =>
+                    "Container template, $suffix, subdirectory with same name as file",
+                volname => "vztmpl/$file_name/$file_name",
+                expected => [
+                    'vztmpl', "$file_name/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
         );

         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.tar.gz";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference before volume type prefix",
+                volname => "../vztmpl/$file_name",
+                expected => "unable to parse directory volume name '../vztmpl/$file_name'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference at beginning of volume path",
+                volname => "vztmpl/../$file_name",
+                expected => "unable to parse directory volume name 'vztmpl/../$file_name'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference at end of volume path",
+                volname => "vztmpl/$file_name/..",
+                expected => "unable to parse directory volume name 'vztmpl/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference between dir components of volume path",
+                volname => "vztmpl/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'vztmpl/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }

 # Additional tests for backup files
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index bc87d289..4dfc68e1 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -137,6 +137,14 @@ my $tests = [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
+    {
+        description => 'CT template, tar.gz, nested subdirectories',
+        file =>
+            "$DEFAULT_STORAGE_DIR/template/cache/foo/bar/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        expected => [
+            'vztmpl', 'local:vztmpl/foo/bar/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        ],
+    },
     {
         description => 'CT template, wrong ending, tar bz2',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
--
2.47.3

[PATCH pve-storage v1 50/54] fix #2884: support nested subdir scanning for 'snippets' vtype

[Max R. Carrara]

Add support for parsing nested subdirectories for the 'snippets'
volume type by adapting its corresponding regex used by parsing
helpers.

Add additional test cases wherever applicable to account for nested
subdirectories for 'snippets' volumes, as was done for the 'iso' and
'vztmpl' volume types.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  1 +
 src/PVE/Storage/Common/test/parser_tests.pl | 52 ++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  4 +-
 src/test/filesystem_path_test.pm            | 16 +++++
 src/test/list_volumes_test.pm               | 78 +++++++++++++++++++++
 src/test/parse_volname_test.pm              | 52 ++++++++++++++
 src/test/path_to_volume_id_test.pm          |  7 ++
 7 files changed, 208 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 92e5b0fd..55fbc129 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -126,6 +126,7 @@ my $RE_BACKUP_FILE_PATH = qr!

 my $RE_SNIPPETS_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file> [^/]+ )
     )
 !xn;
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index c80d0d90..38d5ebac 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -498,6 +498,57 @@ my $volname_cases_snippets_valid = [
             volname => 'snippets/userconfig.yaml',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            dir => 'subdir',
+            'disk-path' => 'subdir/hookscript.pl',
+            path => 'subdir/hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/subdir/hookscript.pl',
+        },
+    },
+    {
+        path => 'deeply/nested/subdir/hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            dir => 'deeply/nested/subdir',
+            'disk-path' => 'deeply/nested/subdir/hookscript.pl',
+            path => 'deeply/nested/subdir/hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/deeply/nested/subdir/hookscript.pl',
+        },
+    },
+];
+
+my $volname_cases_snippets_invalid = [
+    {
+        description => "Parent dir reference in path (beginning) (snippets)",
+        args => {
+            path => '../hookscript.pl',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (snippets)",
+        args => {
+            path => 'subdir/../hookscript.pl',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (snippets)",
+        args => {
+            path => 'subdir/hookscript.pl/..',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
 ];

 my $volname_cases_import_valid = [
@@ -597,6 +648,7 @@ my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
     $volname_cases_backup_invalid,
+    $volname_cases_snippets_invalid,
     $volname_cases_import_invalid,
 ];

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 2df56e76..3a3e1b62 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -235,7 +235,7 @@ my $defaultData = {
         },
         'max-scan-depth' => {
             description => "Maximum depth of subdirectories to traverse when searching for"
-                . " ISOs and container templates in directories.",
+                . " ISOs, container templates and snippets in directories.",
             type => 'integer',
             default => 0,
             minimum => 0,
@@ -1831,7 +1831,7 @@ sub list_volumes {
         }

         if ($type eq 'snippets') {
-            return get_subdir_files($storeid, $scfg, 'snippets', undef);
+            return get_subdir_files($storeid, $scfg, 'snippets', undef, $depth);
         }

         if ($type eq 'import') {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index b5c1ab33..ff6dffe1 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -82,6 +82,22 @@ my $tests = [
             'backup',
         ],
     },
+    {
+        volname => 'snippets/userconfig.yaml',
+        snapname => undef,
+        expected => ["$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml", undef, 'snippets'],
+    },
+    {
+        volname => 'snippets/hookscript.pl',
+        snapname => undef,
+        expected => ["$DEFAULT_STORAGE_DIR/snippets/hookscript.pl", undef, 'snippets'],
+    },
+    {
+        volname => 'snippets/foo/bar/baz/something.txt',
+        snapname => undef,
+        expected =>
+            ["$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", undef, 'snippets'],
+    },
 ];

 my sub run_tests($tests) {
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 6bdfa37c..edcf1ba1 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1340,6 +1340,24 @@ my $test_param_list = [
                     "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1410,6 +1428,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/some-hookscript.pl',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1502,6 +1545,41 @@ my $test_param_list = [
                     "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/6/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/2/3/4/5/some-hookscript.pl',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/6/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index b90815c2..0a425ee2 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -372,10 +372,62 @@ my $tests = [
                 volname => "snippets/$file_name",
                 expected => ['snippets', $file_name, undef, undef, undef, undef, 'raw'],
             },
+            {
+                description => "Snippets, $file_name, subdirectory",
+                volname => "snippets/foo/$file_name",
+                expected => ['snippets', "foo/$file_name", undef, undef, undef, undef, 'raw'],
+            },
+            {
+                description => "Snippets, $file_name, nested subdirectories",
+                volname => "snippets/foo/bar/baz/$file_name",
+                expected =>
+                    ['snippets', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw'],
+            },
+            {
+                description => "Snippets, $file_name, subdirectory with same name as file",
+                volname => "snippets/$file_name/$file_name",
+                expected =>
+                    ['snippets', "$file_name/$file_name", undef, undef, undef, undef, 'raw'],
+            },
         );

         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "some-file.txt";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference before volume type prefix",
+                volname => "../snippets/$file_name",
+                expected => "unable to parse directory volume name '../snippets/$file_name'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference at beginning of volume path",
+                volname => "snippets/../$file_name",
+                expected => "unable to parse directory volume name 'snippets/../$file_name'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference at end of volume path",
+                volname => "snippets/$file_name/..",
+                expected => "unable to parse directory volume name 'snippets/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference between dir components of volume path",
+                volname => "snippets/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'snippets/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }

 # Test cases for import files
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 4dfc68e1..d22f6272 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -181,6 +181,13 @@ my $tests = [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
+    {
+        description => 'Snippets, arbitrary file, nested subdirectories',
+        file => "$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt",
+        expected => [
+            'snippets', 'local:snippets/foo/bar/baz/something.txt',
+        ],
+    },
     {
         description => 'CT template, tar.xz',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
--
2.47.3

[PATCH pve-storage v1 51/54] test: add more tests for 'import' vtype & guard against nested subdirs

[Max R. Carrara]

Supporting nested subdirectories for the 'import' volume type is
currently not possible, because of a parsing ambiguity that would be
introduced. In particular, it would not be possible to differentiate
whether 'foo.ova' in 'foo.ova/bar.qcow2' refers to a directory ending
with '.ova' or an actual '.ova' file through parsing alone. Note this
down in a comment as well, so it doesn't get overlooked.

Therefore, instead of supporting nested subdirectories for the
'import' volume type, add tests that explicitly guard against
subdirectory scanning support.

Additionally, toss in a few extra tests for the `filesystem_path()`
plugin API method and also for the differences in whitespace handling
for file paths and volume names for the 'import' volume type.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |   7 ++
 src/PVE/Storage/Common/test/parser_tests.pl |  40 +++++++
 src/test/filesystem_path_test.pm            |  14 +++
 src/test/list_volumes_test.pm               |  65 +++++++++++
 src/test/parse_volname_test.pm              | 120 ++++++++++++++++++++
 src/test/path_to_volume_id_test.pm          |  13 +++
 6 files changed, 259 insertions(+)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 55fbc129..6bab7e28 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -131,6 +131,13 @@ my $RE_SNIPPETS_FILE_PATH = qr!
     )
 !xn;

+# NOTE: Supporting subdirectories would introduce a parsing ambiguity with
+# volume names; in particular, there would be no way to tell whether 'foo.ova'
+# in 'foo.ova/bar.qcow2' refers to a directory or a file just through parsing
+# alone.
+#
+# Therefore, neither RE_IMPORT_FILE_PATH nor RE_IMPORT_VOLNAME currently
+# accept subdirectory components.
 my $RE_IMPORT_FILE_PATH = qr!
     (?<path>
         (?<file> ($RE_SAFE_CHAR_CLASS)+ \. (?<ext> $RE_IMPORT_EXTENSIONS) )
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 38d5ebac..a788f2ab 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -634,6 +634,46 @@ my $volname_cases_import_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Subdirectory (import)",
+        args => {
+            path => 'subdir/disk.ovf',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Nested subdirectory (import)",
+        args => {
+            path => 'deeply/nested/subdir/disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (beginning) (import)",
+        args => {
+            path => '../disk.ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (import)",
+        args => {
+            path => 'subdir/../disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (import)",
+        args => {
+            path => 'subdir/disk.qcow2/..',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
 ];

 my $cases_valid_all = [
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index ff6dffe1..f06682a9 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -98,6 +98,20 @@ my $tests = [
         expected =>
             ["$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", undef, 'snippets'],
     },
+    {
+        volname => 'import/import.ova',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/import/import.ova", undef, 'import',
+        ],
+    },
+    {
+        volname => 'import/import.ovf',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/import/import.ovf", undef, 'import',
+        ],
+    },
 ];

 my sub run_tests($tests) {
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index edcf1ba1..b43b9cff 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1358,6 +1358,26 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed regardless
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed regardless
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/some-import.ova",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1453,6 +1473,26 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed + exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/some-import.ova",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1580,6 +1620,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/6/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Still not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed + exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/6/some-import.ova",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 0a425ee2..df639df3 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -64,6 +64,18 @@ my $tests = [
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
+    {
+        description => "Import, ova file on disk with whitespace in name",
+        volname => 'import/My neat import.ova/disk.vmdk',
+        expected =>
+            ['import', 'My neat import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
+    },
+    {
+        description => "Import, ova file on disk and inner file of ova with whitespace in name",
+        volname => 'import/My neat import.ova/OS disk.vmdk',
+        expected =>
+            ['import', 'My neat import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
+    },
     #
     # failed matches
     #
@@ -448,6 +460,57 @@ my $tests = [

         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.ova";
+
+        my @extra_failed_tests = (
+            {
+                description => "Import, ova, directory",
+                volname => "import/foo/$file_name",
+                expected => "unable to parse directory volume name 'import/foo/$file_name'\n",
+            },
+            {
+                description => "Import, ova, subdirectories",
+                volname => "import/foo/bar/baz/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/bar/baz/$file_name'\n",
+            },
+            {
+                description => "Import, ova, directory with same name as file",
+                volname => "import/$file_name/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$file_name'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference before volume type prefix",
+                volname => "../import/$file_name",
+                expected => "unable to parse directory volume name '../import/$file_name'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference at beginning of volume path",
+                volname => "import/../$file_name",
+                expected => "unable to parse directory volume name 'import/../$file_name'\n",
+            },
+            {
+                description => "Import, ova, parent directory reference at end of volume path",
+                volname => "import/$file_name/..",
+                expected => "unable to parse directory volume name 'import/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference between dir components of volume path",
+                volname => "import/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }

 # Test cases for OVA import files with content inside
@@ -477,6 +540,63 @@ my $tests = [

         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $content_file_name = "disk.qcow2";
+
+        my @extra_failed_tests = (
+            {
+                description => "Import, inner file of ova (qcow2), directory",
+                volname => "import/foo/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2), nested directories",
+                volname => "import/foo/bar/baz/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/bar/baz/$file_name/$content_file_name'\n",
+            },
+            {
+                description =>
+                    "Import, inner file of ova (qcow2), directory with same name as file",
+                volname => "import/$file_name/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference before volume type prefix",
+                volname => "../import/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name '../import/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at beginning of volume path",
+                volname => "import/../$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/../$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at end of disk path",
+                volname => "import/$file_name/../$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/../$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at end of volume path",
+                volname => "import/$file_name/$content_file_name/..",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$content_file_name/..'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }

 my sub run_tests($tests) {
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index d22f6272..3bfb37a6 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -256,6 +256,19 @@ my $tests = [
         file => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
+    {
+        # Allowed in volume names, but not in file paths
+        description => 'Import, ova, whitespace in file name',
+        file => "$DEFAULT_STORAGE_DIR/import/My Import.ova",
+        expected => [''],
+    },
+    {
+        # Supporting subdirectories for the 'import' volume type is
+        # currently not possible because it would introduce parsing ambiguities.
+        description => 'Import, ova, subdirectory',
+        file => "$DEFAULT_STORAGE_DIR/import/foo/import.ova",
+        expected => [''],
+    },
 ];

 my sub run_tests($tests) {
--
2.47.3

[PATCH pve-storage v1 52/54] test: add tests guarding against subdir scanning for vtypes

[Max R. Carrara]

Add tests that guard against implementing nested subdirectory scanning
for the 'backup' volume type in `parser_tests.pl`, and for the
'images' and 'backup' volume types in `path_to_volume_id_test.pm`.

For the 'backup' vtype, also add a comment above the parser's
corresponding regex that we do not allow nested subdirs there.

This is done mainly to prevent implementing subdir scanning blindly in
the future for these two volume types.

While adding support for subdir scanning for the 'backup' volume type
would probably be fairly trivial, there is also no (known) need for it
currently.

For the 'images' type, there currently is no plan or need to support
subdir scanning, and it would probably not make much sense anyway.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  1 +
 src/PVE/Storage/Common/test/parser_tests.pl | 40 +++++++++++++++++++++
 src/test/path_to_volume_id_test.pm          | 11 ++++++
 3 files changed, 52 insertions(+)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 6bab7e28..b33fe78e 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -103,6 +103,7 @@ my $RE_GUEST_VZDUMP_BACKUP_FILE_NAME = qr!
     - [^/]+
 !xn;

+# NOTE: Not supporting subdirectories here.
 my $RE_BACKUP_FILE_PATH = qr!
     (?<path>
         (?<file>
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index a788f2ab..882c0675 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -474,6 +474,46 @@ my $volname_cases_backup_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Subdirectory (backup)",
+        args => {
+            path => 'subdir/vzdump-qemu-16110-2020_03_30-21_12_40.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Nested subdirectory (backup)",
+        args => {
+            path => 'deeply/nested/subdir/vzdump-qemu-16110-2020_03_30-21_12_40.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (beginning) (backup)",
+        args => {
+            path => '../vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (backup)",
+        args => {
+            path => 'subdir/../vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (backup)",
+        args => {
+            path => 'subdir/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz/..',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
 ];

 my $volname_cases_snippets_valid = [
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 3bfb37a6..2eb17e9e 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -246,11 +246,22 @@ my $tests = [
         file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
+    {
+        description => 'Backup, nested subdirectories, vma.gz',
+        file =>
+            "$DEFAULT_STORAGE_DIR/dump/foo/bar/baz/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        expected => [''],
+    },
     {
         description => 'Image, string as vmid in folder name',
         file => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
+    {
+        description => 'Image, additional nested subdirectories before vmid',
+        file => "$DEFAULT_STORAGE_DIR/images/foo/bar/baz/1234/vm-1234-disk-0.qcow2",
+        expected => [''],
+    },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
         file => "$DEFAULT_STORAGE_DIR/import/test.foo",
--
2.47.3

[PATCH pve-storage v1 53/54] storage api: mark old public regexes for removal, bump APIVER & APIAGE

[Max R. Carrara]

Mark all unused regular expressions that were phased out during the
previous refactors and changes for removal on the next APIAGE reset.

Instead, the parsing functions in `PVE::Storage::Common::Parse` should
be used instead.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog              | 16 ++++++++++++++++
 src/PVE/Storage.pm        | 10 ++++++++--
 src/PVE/Storage/Plugin.pm |  1 +
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index d1d3e1c3..efcfef14 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -6,6 +6,22 @@ without breaking anything unaware of it.)

 Future changes should be documented in here.

+## Version 15:
+
+* Remove the following regular expressions:
+  * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
+  * `$PVE::Storage::VZTMPL_EXT_RE_1` (`vztmpl` volume type)
+  * `$PVE::Storage::BACKUP_EXT_RE_2` (`backup` volume type)
+  * `$PVE::Storage::IMPORT_EXT_RE_1` (`import` volume type)
+  * `$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1` (`import` volume type)
+  * `$PVE::Storage::OVA_CONTENT_RE_1` (`import` volume type)
+
+  These regular expressions were mostly used for parsing file paths and volume
+  names corresponding to their volume types, noted in parentheses above.
+
+  Instead, The parsing functions in `PVE::Storage::Common::Parse` should be
+  used.
+
 ## Version 14:

 * Replace plugin helper `get_vtype_subdirs()` with standalone helper subroutine
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 199a6375..3db34b68 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -48,11 +48,11 @@ use PVE::Storage::BTRFSPlugin;
 use PVE::Storage::ESXiPlugin;

 # Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 14;
+use constant APIVER => 15;
 # Age is the number of versions we're backward compatible with.
 # This is like having 'current=APIVER' and age='APIAGE' in libtool,
 # see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 5;
+use constant APIAGE => 6;

 our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];

@@ -120,19 +120,25 @@ PVE::Storage::Plugin->init();
 # the following REs indicate the number or capture groups via the trailing digit
 # CAUTION don't forget to update the digits accordingly after messing with the capture groups

+# FIXME: remove this regex on the next APIAGE reset.
 our $ISO_EXT_RE_0 = qr/\.(?:iso|img)/i;

+# FIXME: remove this regex on the next APIAGE reset.
 our $VZTMPL_EXT_RE_1 = qr/\.(?|(tar)(?!\.)|tar\.(gz|xz|zst|bz2))/i;

+# FIXME: remove this regex on the next APIAGE reset.
 our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;

+# FIXME: remove this regex on the next APIAGE reset.
 our $IMPORT_EXT_RE_1 = qr/\.(ova|ovf|qcow2|raw|vmdk)/;

+# FIXME: remove this regex on the next APIAGE reset.
 our $UPLOAD_IMPORT_EXT_RE_1 = qr/\.(ova|qcow2|raw|vmdk)/;

 our $SAFE_CHAR_CLASS_RE = qr/[a-zA-Z0-9\-\.\+\=\_]/;
 our $SAFE_CHAR_WITH_WHITESPACE_CLASS_RE = qr/[ a-zA-Z0-9\-\.\+\=\_]/;

+# FIXME: remove this regex on the next APIAGE reset.
 our $OVA_CONTENT_RE_1 = qr/${SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.(qcow2|raw|vmdk)/;

 # FIXME remove with PVE 9.0, add versioned breaks for pve-manager
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3a3e1b62..6e20c519 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -31,6 +31,7 @@ use base qw(PVE::SectionConfig);

 # TODO: Phase out these two constants
 use constant KNOWN_COMPRESSION_FORMATS => ('gz', 'lzo', 'zst', 'bz2');
+# FIXME: remove this regex on the next APIAGE reset.
 use constant COMPRESSOR_RE => join('|', KNOWN_COMPRESSION_FORMATS);

 use constant LOG_EXT => ".log";
--
2.47.3

[PATCH pve-manager v1 54/54] fix #2884: ui: storage: add field for 'max-scan-depth' property

[Max R. Carrara]

In the base storage panel, add a simple integer field in the advanced
section for storage types that use the new `max-scan-depth` property.

This means that the maximum directory scan depth can now be configured
for directories, NFS, CIFS, CephFS, and BTRFS.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 www/manager6/storage/Base.js | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index 8961460b..0cf90337 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -75,6 +75,20 @@ Ext.define('PVE.panel.StorageBase', {
             });
         }

+        const subdirScanningStorageTypes = ['dir', 'nfs', 'cifs', 'cephfs', 'btrfs'];
+
+        if (subdirScanningStorageTypes.includes(me.type)) {
+            addAdvancedWidget({
+                xtype: 'proxmoxintegerfield',
+                name: 'max-scan-depth',
+                fieldLabel: gettext('Maximum Directory Scan Depth'),
+                allowBlank: false,
+                value: 0,
+                minValue: 0,
+                maxValue: 50,
+            });
+        }
+
         const externalStorageManagedSnapshotSupport = ['dir', 'nfs', 'cifs', 'lvm'];

         if (externalStorageManagedSnapshotSupport.includes(me.type)) {
--
2.47.3