[PATCH manager/storage v2 00/50] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types

[PATCH manager/storage v2 00/50] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types

[Max R. Carrara]

Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v2
===========================================================================

Basically what the title says. Implement subdirectory scanning for
directory-based storage types, which includes directories (duh), NFS,
CIFS, CephFS, and BTRFS. This fixes #2884 [2884] and the more
narrowly-scoped #623 [623].

See the initial series's cover letter [init-cover] for the entire
context.

*Huge* thanks to @Wolfgang for the thorough review on v1, and also for
the off-list correspondence regarding this! :) This refresh incorporates
all of his feedback.

Changes Since v1
----------------

Since this series is quite massive, I decided to:
* track all changes here, not just the notable ones
* track changes to patches' index numbers
* add references to each suggestion relevant for the change, if
  applicable / mentioned over the mailing list

While this took a bit of extra time, it should hopefully alleviate some
tracking work on the maintainers' end.

- Clarify that `PVE::Storage::Common::plugin_get_default_vtype_subdirs()`
  returns an absolute path in its docstring (patch #6).

- Do not call if-elsif chains "needless" in the patches that break them
  up anymore (patch #8 and #14).

  Ref.: https://lore.proxmox.com/pve-devel/w44tdwzbsu3bh4xggcss6b3ukrv7imddbzackyf3qhsfv2j5mn@bogediza5n5g/

- Use a lookahead for a slash (`(?=/)`) after the `\E` in
  `get_subdir_files() in patch #9.

  Ref.: https://lore.proxmox.com/pve-devel/ex6gdtlicrwrxpw6bmz5awhw5bnnt4g5uokuajq4fn3b2kkkd6@mjkvz3oppvzi/

- Use a lookahead for a slash (`(?=/)`) after the `\E` in
  `path_to_volume_id()` in patch #16.

  Ref.: https://lore.proxmox.com/pve-devel/j5ygsulsykpaevszwpuv2726r7ogzz6wd2346telm7szlqmbwl@opm2banmchnp/

- Fix filtering using `grep` in patch #17 that made the entire check a
  no-op, which could have affected security.

  Ref.: https://lore.proxmox.com/pve-devel/hwb7vmxog4p3v3iqe74ptettlev3nk4vcgaqdrs67nnoohmh72@2wncblbtmcij/

- Do not use a hash in the `split_leading_dir_from_path()` helper in
  patch #22 anymore.

  Ref.: https://lore.proxmox.com/pve-devel/jfa5fql3svnils2dlklyz4vrgyo4jmn24tuv2wa7uu7lf2wgzb@7arsjxlzdivs/

- Rename the subs added in patch #22 to better distinguish between those
  that take a relative path and those take an absolute path.

  Ref.: https://lore.proxmox.com/pve-devel/jfa5fql3svnils2dlklyz4vrgyo4jmn24tuv2wa7uu7lf2wgzb@7arsjxlzdivs/

- Clarify in the docstrings of certain subs added in patch #22 that we
  return `undef` in scalar context, and an empty list in list context.

  Ref.: https://lore.proxmox.com/pve-devel/jfa5fql3svnils2dlklyz4vrgyo4jmn24tuv2wa7uu7lf2wgzb@7arsjxlzdivs/

- Improve the docstrings of the parsing subs in patch #22 by using
  examples for expected outputs and by making them a little less
  verbose.

- Document further special cases in every patch that introduces them
  -- patches #24, #25, #28, and also #44-46 (formerly patches #48-50
  in v1).

- Fold patches #29 - #32 of v1 into patch #29 of this series, since a
  separate patch for each site was a bit *too* granular.

- Do not replace string interpolation with concatenation in patch #7
  anymore.

  Ref.: https://lore.proxmox.com/pve-devel/ehmk4jjaekoqsxf2pqy6imbfouvaeuwdkzc2odvlskf72yl7yp@3lfm4ihnutiw/

- Do not return volumes of undeclared content types anymore instead of
  just documenting that behavior using test cases in patch #34 (formerly
  patch #38 in v1). Instead, return an empty list if the volumes of an
  undeclared content type are queried.

  Ref.: https://lore.proxmox.com/pve-devel/fgno5lzql7h2jrnyfcmn2usahi5e2chteiwfw543ak7mahglvq@ph2sigxrk2ty/

- Use greedy matching instead of non-greedy matching for the
  subdirectory component regex in patch #44 (formerly patch #48 in v1).

  Ref.: https://lore.proxmox.com/pve-devel/cvjtmro3odmdgh574wnfql3nlaezezukokbsuxdnmycadfcuah@4ckwm5hn2bci/

- Add API deprecations within the commits that actually deprecate things
  instead of bumping the APIVER + APIAGE twice throughout the series.

  This means that if e.g. one commit makes a regex in `PVE::Storage`
  obsolete, the deprecation notice in `ApiChangeLog` and TODO comment
  for that regex are added right away.

  Ref.: https://lore.proxmox.com/pve-devel/xapbgpwdqk7h6uebzkgxei7v7gamzbhyin2blbohd2bt4kyrr4@ogniag55fdai/

- Add patch #50 that replaces the usage of `$plugin->get_subdir()` with
  `PVE::Storage::Common::plugin_get_vtype_subdir()`, since the latter is
  now deprecated.

  Ref.: https://lore.proxmox.com/pve-devel/dzlh5oyu73idd2s6e3ovzrivdhhvxdmkiumwjvzq6nbjotidst@2i25ysrie7xt/

Additional Notes
----------------

Since this series also touches the compression extensions and moves them
to the new parsing module it introduces, I considered expanding upon
that, cleaning up our compression-related code as well.

However, given that this series is already large enough, I consider this
out of scope for now and will probably follow up on this in a separate
series.

Testing
-------

If anyone could give this series a spin, I'd be most grateful!

Here are some interesting things you could check out (non-exhaustive):
- Configuring the new 'max-scan-depth' property in the UI
- Adding subdirectories in your ISO, LXC template and snippets dirs
  (and populating those dirs afterwards)
- Checking whether the depth limit is honored
  --> 0 is the default, which retains the current behavior of not
      scanning through any subdirs
- Checking whether imports (.ova files etc) still work as expected
- ISO / CT template upload / deletion
- ...

Also note that I ran the tests in the repository for every single patch
that I added; if you want to do this for yourself as a sanity check, try
the following:

    git rebase -i --autostash --autosquash origin/master -x 'cd src && make test'

References
----------

[2884]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884
[623]: https://bugzilla.proxmox.com/show_bug.cgi?id=623
[init-cover]: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/

Summary of Changes
------------------


pve-storage:

Max R. Carrara (48):
  test: plugin tests: run tests with at most 4 jobs
  plugin, common: remove superfluous use of =pod command paragraph
  common: add POD headings for groups of helpers
  common: use Exporter module for PVE::Storage::Common
  plugin: make get_subdir_files a proper subroutine and update style
  plugin api: replace helpers w/ standalone subs, bump API version & age
  common: prevent autovivification in plugin_get_vtype_subdir helper
  plugin: break up if-elsif chain into separate if-blocks
  plugin: adapt get_subdir_files helper of list_volumes API method
  plugin: update code style of list_volumes plugin API method
  plugin: use closure for obtaining raw volume data in list_volumes
  plugin: use closure for inner loop logic in list_volumes
  storage: update code style in function path_to_volume_id
  storage: break up if-elsif chain in path_to_volume_id
  storage: heave vtype file path parsing logic inside loop into helper
  storage: clean up code that was moved into helper in path_to_volume_id
  api: status: move content type assert for up-/downloads into helper
  api: status: use helper from common module to get content directory
  api: status: move up-/download file path parsing code into helper
  api: status: simplify file content assertion logic for up-/download
  test: guest import: add tests for PVE::GuestImport
  tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0
  common: test: set up parser testing code, add tests for 'iso' vtype
  tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions
  tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions
  tree-wide: replace usages of inline regexes for snippets with parsers
  tree-wide: partially replace usages of regexes for 'import' vtype
  tree-wide: replace remaining usages of regexes for 'import' vtype
  tree-wide: simplify recently refactored parsing logic
  test: list volumes: reorganize and modernize test running code
  test: list volumes: fix broken test checking for vmlist modifications
  test: list volumes: introduce new format for test cases
  test: list volumes: remove legacy code and migrate cases to new format
  plugin: do not return volumes of undeclared content types anymore
  plugin: correct comment in get_subdir_files helper
  test: parse volname: modernize code
  test: parse volname: adapt tests regarding 'import' volume type
  test: parse volname: move VM disk test creation into separate block
  test: parse volname: move backup file test creation into sep. block
  test: parse volname: parameterize test case creation for some vtypes
  test: volume id: modernize code
  test: volume id: rename 'volname' test case parameter to 'file'
  test: filesystem path: modernize code
  fix #2884: implement nested subdir scanning and support 'iso' vtype
  fix #2884: support nested subdir scanning for 'vztmpl' volume type
  fix #2884: support nested subdir scanning for 'snippets' vtype
  test: add more tests for 'import' vtype & guard against nested subdirs
  test: add tests guarding against subdir scanning for vtypes

 ApiChangeLog                                |   45 +
 debian/control                              |    1 +
 src/PVE/API2/Storage/Status.pm              |  198 +-
 src/PVE/BackupProvider/Plugin/Base.pm       |    6 +-
 src/PVE/GuestImport.pm                      |   28 +-
 src/PVE/GuestImport/OVF.pm                  |   24 +-
 src/PVE/Makefile                            |    1 +
 src/PVE/Storage.pm                          |  118 +-
 src/PVE/Storage/BTRFSPlugin.pm              |   16 +-
 src/PVE/Storage/CephFSPlugin.pm             |    1 +
 src/PVE/Storage/Common.pm                   |   91 +-
 src/PVE/Storage/Common/Makefile             |    5 +
 src/PVE/Storage/Common/Parse.pm             |  674 ++++++
 src/PVE/Storage/Common/test/Makefile        |    6 +
 src/PVE/Storage/Common/test/parser_tests.pl | 1134 ++++++++++
 src/PVE/Storage/Common/test/run_tests.pl    |   25 +
 src/PVE/Storage/DirPlugin.pm                |    1 +
 src/PVE/Storage/ESXiPlugin.pm               |    6 -
 src/PVE/Storage/Makefile                    |    4 +
 src/PVE/Storage/Plugin.pm                   |  368 ++--
 src/test/filesystem_path_test.pm            |  109 +-
 src/test/get_subdir_test.pm                 |   12 +-
 src/test/guest_import_test.pl               |  948 +++++++++
 src/test/list_volumes_test.pm               | 2095 +++++++++++++++----
 src/test/parse_volname_test.pm              |  696 ++++--
 src/test/path_to_volume_id_test.pm          |  201 +-
 src/test/run_plugin_tests.pl                |   18 +-
 src/test/run_volume_access_tests.pl         |    5 +-
 28 files changed, 5833 insertions(+), 1003 deletions(-)
 create mode 100644 src/PVE/Storage/Common/Parse.pm
 create mode 100644 src/PVE/Storage/Common/test/Makefile
 create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
 create mode 100755 src/PVE/Storage/Common/test/run_tests.pl
 create mode 100755 src/test/guest_import_test.pl


pve-manager:

Max R. Carrara (2):
  fix #2884: ui: storage: add field for 'max-scan-depth' property
  pve8to9: use helper from common storage module to get vtype subdir

 PVE/CLI/pve8to9.pm           |  5 +++--
 www/manager6/storage/Base.js | 14 ++++++++++++++
 2 files changed, 17 insertions(+), 2 deletions(-)


Summary over all repositories:
  30 files changed, 5850 insertions(+), 1005 deletions(-)

-- 
Generated by murpp 0.12.0

[PATCH pve-storage v2 01/50] test: plugin tests: run tests with at most 4 jobs

[Max R. Carrara]

Speed up tests by allowing up to 4 jobs to run in parallel.

Base the amount of jobs on the output of `nproc` for environments with
less than 4 processing units available.

Sort tests alphabetically as well, just to make things neater.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/run_plugin_tests.pl | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 8bce9d3b..27c45b8c 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -8,14 +8,21 @@ $ENV{TZ} = 'UTC';
 
 use TAP::Harness;
 
-my $harness = TAP::Harness->new({ verbosity => -1 });
+my $MAX_JOBS = 4;
+
+my $nproc = eval { int(`nproc`) } || $MAX_JOBS;
+
+my $jobs = $nproc > $MAX_JOBS ? $MAX_JOBS : $nproc;
+
+my $harness = TAP::Harness->new({ verbosity => -1, jobs => $jobs });
+
 my $res = $harness->runtests(
     "archive_info_test.pm",
-    "parse_volname_test.pm",
-    "list_volumes_test.pm",
-    "path_to_volume_id_test.pm",
-    "get_subdir_test.pm",
     "filesystem_path_test.pm",
+    "get_subdir_test.pm",
+    "list_volumes_test.pm",
+    "parse_volname_test.pm",
+    "path_to_volume_id_test.pm",
     "prune_backups_test.pm",
 );
 
-- 
2.47.3

[PATCH pve-storage v2 02/50] plugin, common: remove superfluous use of =pod command paragraph

[Max R. Carrara]

Because a POD block may be started using any POD command paragraph
[0], using =pod here is not necessary. Therefore, remove these
occurrences.

In other words, =head1 and =head3 (for example) already begin a POD
block, so explicitly opening one with =pod is not needed.

This has the benefit of making the docstrings take up a little less
space overall.

[0]: https://perldoc.perl.org/perlpodspec#Pod-Commands

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 18 ------------------
 src/PVE/Storage/Plugin.pm |  6 ------
 2 files changed, 24 deletions(-)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 733d7fa4..bc6a4ff3 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -12,8 +12,6 @@ use constant {
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
 };
 
-=pod
-
 =head1 NAME
 
 PVE::Storage::Common - Shared functions and utilities for storage plugins and storage operations
@@ -75,14 +73,10 @@ PVE::JSONSchema::register_standard_option(
     },
 );
 
-=pod
-
 =head1 FUNCTIONS
 
 =cut
 
-=pod
-
 =head3 align_size_up
 
     $aligned_size = align_size_up($size, $granularity)
@@ -103,8 +97,6 @@ sub align_size_up : prototype($$) {
     return $aligned_size;
 }
 
-=pod
-
 =head3 deallocate
 
     deallocate($file_handle, $offset, $length)
@@ -148,8 +140,6 @@ my sub run_qemu_img_json {
     return $json;
 }
 
-=pod
-
 =head3 qemu_img_create
 
     qemu_img_create($fmt, $size, $path, $options)
@@ -174,8 +164,6 @@ sub qemu_img_create {
     run_command($cmd, errmsg => "unable to create image");
 }
 
-=pod
-
 =head3 qemu_img_create_qcow2_backed
 
     qemu_img_create_qcow2_backed($path, $backing_path, $backing_format, $options)
@@ -212,8 +200,6 @@ sub qemu_img_create_qcow2_backed {
     run_command($cmd, errmsg => "unable to create image");
 }
 
-=pod
-
 =head3 qemu_img_info
 
     qemu_img_info($filename, $file_format, $timeout, $follow_backing_files)
@@ -234,8 +220,6 @@ sub qemu_img_info {
     return run_qemu_img_json($cmd, $timeout);
 }
 
-=pod
-
 =head3 qemu_img_measure
 
     qemu_img_measure($size, $fmt, $timeout, $options)
@@ -260,8 +244,6 @@ sub qemu_img_measure {
     return run_qemu_img_json($cmd, $timeout);
 }
 
-=pod
-
 =head3 qemu_img_resize
 
     qemu_img_resize($path, $format, $size, $preallocation, $timeout)
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index c805a180..6394b6e0 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -2411,8 +2411,6 @@ sub rename_volume {
     return "${storeid}:${base}${target_vmid}/${target_volname}";
 }
 
-=pod
-
 =head3 rename_snapshot
 
     $plugin->rename_snapshot($scfg, $storeid, $volname, $source_snap, $target_snap)
@@ -2462,8 +2460,6 @@ my sub blockdev_options_nbd_unix {
     return $blockdev;
 }
 
-=pod
-
 =head3 qemu_blockdev_options
 
     $blockdev =
@@ -2636,8 +2632,6 @@ sub new_backup_provider {
     die "implement me if enabling the feature 'backup-provider' in plugindata()->{features}\n";
 }
 
-=pod
-
 =head3 volume_qemu_snapshot_method
 
     $method = $plugin->volume_qemu_snapshot_method($storeid, $scfg, $volname);
-- 
2.47.3

[PATCH pve-storage v2 03/50] common: add POD headings for groups of helpers

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index bc6a4ff3..befa3e7f 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -77,6 +77,10 @@ PVE::JSONSchema::register_standard_option(
 
 =cut
 
+=head2 GENERAL HELPERS
+
+=cut
+
 =head3 align_size_up
 
     $aligned_size = align_size_up($size, $granularity)
@@ -118,6 +122,10 @@ sub deallocate : prototype($$$) {
     }
 }
 
+=head2 FUNCTIONS FOR QEMU IMAGE OPERATIONS
+
+=cut
+
 my sub run_qemu_img_json {
     my ($cmd, $timeout) = @_;
     my $json = '';
-- 
2.47.3

[PATCH pve-storage v2 04/50] common: use Exporter module for PVE::Storage::Common

[Max R. Carrara]

Expose the existing helpers in `PVE::Storage::Common` via the
`Exporter` module's `import()` function, allowing the helpers to be
imported as-is by other modules.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index befa3e7f..49bcc67c 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -7,6 +7,19 @@ use PVE::RPCEnvironment;
 use PVE::Syscall;
 use PVE::Tools qw(run_command);
 
+use Exporter qw(import);
+
+our @EXPORT_OK = qw(
+    align_size_up
+    deallocate
+
+    qemu_img_create
+    qemu_img_create_qcow2_backed
+    qemu_img_info
+    qemu_img_measure
+    qemu_img_resize
+);
+
 use constant {
     FALLOC_FL_KEEP_SIZE => 0x01, # see linux/falloc.h
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
-- 
2.47.3

[PATCH pve-storage v2 05/50] plugin: make get_subdir_files a proper subroutine and update style

[Max R. Carrara]

This also renames a lot of variables so that they are in line with
the rest of code and also a bit more readable, as follows:

- $fn        -->  $filename
- $notes_fn  -->  $notes_filename
- $sid       -->  $storeid
- $tt        -->  $vtype

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 66 +++++++++++++++++++--------------------
 1 file changed, 33 insertions(+), 33 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 6394b6e0..bd59c024 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1695,69 +1695,69 @@ sub list_images {
     return $res;
 }
 
-# list templates ($tt = <iso|vztmpl|backup|snippets|import>)
-my $get_subdir_files = sub {
-    my ($sid, $path, $tt, $vmid) = @_;
+# $vtype = <iso|vztmpl|backup|snippets|import>
+my sub get_subdir_files {
+    my ($storeid, $path, $vtype, $vmid) = @_;
 
     my $res = [];
 
-    foreach my $fn (<$path/*>) {
-        my $st = File::stat::stat($fn);
+    for my $filename (<$path/*>) {
+        my $st = File::stat::stat($filename);
 
         next if (!$st || S_ISDIR($st->mode));
 
         my $info;
 
-        if ($tt eq 'iso') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+        if ($vtype eq 'iso') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
 
-            $info = { volid => "$sid:iso/$1", format => 'iso' };
+            $info = { volid => "$storeid:iso/$1", format => 'iso' };
 
-        } elsif ($tt eq 'vztmpl') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+        } elsif ($vtype eq 'vztmpl') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
 
-            $info = { volid => "$sid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };
+            $info = { volid => "$storeid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };
 
-        } elsif ($tt eq 'backup') {
-            next if $fn !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
-            my $original = $fn;
+        } elsif ($vtype eq 'backup') {
+            next if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+            my $original = $filename;
             my $format = $2;
-            $fn = $1;
+            $filename = $1;
 
             # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            next if defined($vmid) && $fn !~ m/\S+-$vmid-\S+/;
+            next if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
 
-            $info = { volid => "$sid:backup/$fn", format => $format };
+            $info = { volid => "$storeid:backup/$filename", format => $format };
 
-            my $archive_info = eval { PVE::Storage::archive_info($fn) } // {};
+            my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};
 
             $info->{ctime} = $archive_info->{ctime} if defined($archive_info->{ctime});
             $info->{subtype} = $archive_info->{type} // 'unknown';
 
-            if (defined($vmid) || $fn =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
+            if (defined($vmid) || $filename =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
                 $info->{vmid} = $vmid // $1;
             }
 
-            my $notes_fn = $original . NOTES_EXT;
-            if (-f $notes_fn) {
-                my $notes = PVE::Tools::file_read_firstline($notes_fn);
+            my $notes_filename = $original . NOTES_EXT;
+            if (-f $notes_filename) {
+                my $notes = PVE::Tools::file_read_firstline($notes_filename);
                 $info->{notes} = eval { decode('UTF-8', $notes, 1) } // $notes
                     if defined($notes);
             }
 
             $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
-        } elsif ($tt eq 'snippets') {
+        } elsif ($vtype eq 'snippets') {
 
             $info = {
-                volid => "$sid:snippets/" . basename($fn),
+                volid => "$storeid:snippets/" . basename($filename),
                 format => 'snippet',
             };
-        } elsif ($tt eq 'import') {
+        } elsif ($vtype eq 'import') {
             next
-                if $fn !~
+                if $filename !~
                 m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;
 
-            $info = { volid => "$sid:import/$1", format => "$2" };
+            $info = { volid => "$storeid:import/$1", format => "$2" };
         }
 
         $info->{size} = $st->size;
@@ -1767,7 +1767,7 @@ my $get_subdir_files = sub {
     }
 
     return $res;
-};
+}
 
 # If attributes are set on a volume, they should be included in the result.
 # See get_volume_attribute for a list of possible attributes.
@@ -1785,15 +1785,15 @@ sub list_volumes {
             my $path = $class->get_subdir($scfg, $type);
 
             if ($type eq 'iso' && !defined($vmid)) {
-                $data = $get_subdir_files->($storeid, $path, 'iso');
+                $data = get_subdir_files($storeid, $path, 'iso');
             } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = $get_subdir_files->($storeid, $path, 'vztmpl');
+                $data = get_subdir_files($storeid, $path, 'vztmpl');
             } elsif ($type eq 'backup') {
-                $data = $get_subdir_files->($storeid, $path, 'backup', $vmid);
+                $data = get_subdir_files($storeid, $path, 'backup', $vmid);
             } elsif ($type eq 'snippets') {
-                $data = $get_subdir_files->($storeid, $path, 'snippets');
+                $data = get_subdir_files($storeid, $path, 'snippets');
             } elsif ($type eq 'import') {
-                $data = $get_subdir_files->($storeid, $path, 'import');
+                $data = get_subdir_files($storeid, $path, 'import');
             }
         }
 
-- 
2.47.3

[PATCH pve-storage v2 06/50] plugin api: replace helpers w/ standalone subs, bump API version & age

[Max R. Carrara]

Replace the `get_vtype_subdirs()` and `get_subdir()` helpers in
PVE::Storage::Plugin with standalone helper subroutines
`plugin_get_default_vtype_subdirs()` and `plugin_get_vtype_subdir()`
in PVE::Storage::Common.

This is mostly to prevent third-party plugins from relying on these
helpers as if they were part of the storage plugin API; the signature
of `get_subdir()` in particular looks as if it belongs to an API
method, even though it is only used as a helper in PVE::Storage,
PVE::Storage::Plugin, and PVE::Storage::BTRFSPlugin.

Therefore, make it explicit that these subroutines are really just
helpers by replacing them with slightly altered versions in
PVE::Storage::Common. Add docstrings for these replacements as well.

Additionally, note that PVE::Storage::ESXiPlugin is the only plugin
that explicitly overrides `get_subdir()`, treating it as if it were an
API method, but because that plugin is not filesystem-based (in the
sense that it does not declare the `path` property in its options),
calling `get_subdir()` would throw an exception anyhow. Also, it
should be mentioned that all methods that could call into
`get_subdir()` have been overridden inside ESXiPlugin anyway.

Therefore, the override of `get_subdir()` in PVE::Storage::ESXiPlugin
is superfluous, so remove it.

Additionally, replace all occurrences of `get_vtype_subdirs()` and
`get_subdir()` across the repository with their newly introduced
counterparts `plugin_get_default_vtype_subdirs()` and
`plugin_get_vtype_subdir()`. Adapt any comments and test names as well
in accordance with these changes.

Finally, increment APIAGE and APIVER in PVE::Storage and add a
corresponding entry to the API changelog.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                        | 22 ++++++++++++
 src/PVE/Storage.pm                  | 31 +++++++++--------
 src/PVE/Storage/BTRFSPlugin.pm      | 15 +++++----
 src/PVE/Storage/Common.pm           | 52 +++++++++++++++++++++++++++++
 src/PVE/Storage/ESXiPlugin.pm       |  6 ----
 src/PVE/Storage/Plugin.pm           | 47 ++++++++++----------------
 src/test/get_subdir_test.pm         | 12 ++++---
 src/test/parse_volname_test.pm      |  6 +++-
 src/test/path_to_volume_id_test.pm  |  6 +++-
 src/test/run_volume_access_tests.pl |  5 ++-
 10 files changed, 140 insertions(+), 62 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index 80c5994e..aefc0ff8 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -6,6 +6,28 @@ without breaking anything unaware of it.)
 
 Future changes should be documented in here.
 
+## Version 16:
+
+* Replace plugin helper `get_vtype_subdirs()` with standalone helper subroutine
+  `PVE::Storage::Common::plugin_get_default_vtype_subdirs()`
+
+  The `get_vtype_subdirs()` subroutine in `PVE::Storage::Plugin` was an
+  internal helper with a public signature. In order to make it clear that this
+  is a helper and not meant to be part of the storage plugin API, it is
+  replaced with the more explicit
+  `PVE::Storage::Common::plugin_get_default_vtype_subdirs()` subroutine.
+
+* Replace plugin method `get_subdir()` with standalone helper subroutine
+  `PVE::Storage::Common::plugin_get_vtype_subdir()`
+
+  The `get_subdir()` method in `PVE::Storage::Plugin` was an internal helper
+  method with a public signature. Moreover, its signature could mistakenly
+  suggest that this method was part of the storage plugin API, which it was not
+  originally meant to be.
+
+  In order to make it clear that this is just a helper, it is replaced with the
+  more explicit `PVE::Storage::Common::plugin_get_vtype_subdir()` subroutine.
+
 ## Version 15:
 
 * Add new `$snapname` parameter to the `volume_resize()` plugin method
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 64ea9dad..ed87397e 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -22,6 +22,9 @@ use PVE::JSONSchema;
 use PVE::INotify;
 use PVE::RPCEnvironment;
 use PVE::SSHInfo;
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
 use PVE::RESTEnvironment qw(log_warn);
 
 use PVE::Storage::Plugin;
@@ -41,11 +44,11 @@ use PVE::Storage::BTRFSPlugin;
 use PVE::Storage::ESXiPlugin;
 
 # Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 15;
+use constant APIVER => 16;
 # Age is the number of versions we're backward compatible with.
 # This is like having 'current=APIVER' and age='APIAGE' in libtool,
 # see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 6;
+use constant APIAGE => 7;
 
 our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];
 
@@ -530,7 +533,7 @@ sub get_image_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    my $path = $plugin->get_subdir($scfg, 'images');
+    my $path = plugin_get_vtype_subdir($scfg, 'images');
 
     return $vmid ? "$path/$vmid" : $path;
 }
@@ -541,7 +544,7 @@ sub get_private_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    my $path = $plugin->get_subdir($scfg, 'rootdir');
+    my $path = plugin_get_vtype_subdir($scfg, 'rootdir');
 
     return $vmid ? "$path/$vmid" : $path;
 }
@@ -552,7 +555,7 @@ sub get_iso_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    return $plugin->get_subdir($scfg, 'iso');
+    return plugin_get_vtype_subdir($scfg, 'iso');
 }
 
 sub get_import_dir {
@@ -561,7 +564,7 @@ sub get_import_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    return $plugin->get_subdir($scfg, 'import');
+    return plugin_get_vtype_subdir($scfg, 'import');
 }
 
 sub get_vztmpl_dir {
@@ -570,7 +573,7 @@ sub get_vztmpl_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    return $plugin->get_subdir($scfg, 'vztmpl');
+    return plugin_get_vtype_subdir($scfg, 'vztmpl');
 }
 
 sub get_backup_dir {
@@ -579,7 +582,7 @@ sub get_backup_dir {
     my $scfg = storage_config($cfg, $storeid);
     my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-    return $plugin->get_subdir($scfg, 'backup');
+    return plugin_get_vtype_subdir($scfg, 'backup');
 }
 
 # library implementation
@@ -713,12 +716,12 @@ sub path_to_volume_id {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
-        my $imagedir = $plugin->get_subdir($scfg, 'images');
-        my $isodir = $plugin->get_subdir($scfg, 'iso');
-        my $tmpldir = $plugin->get_subdir($scfg, 'vztmpl');
-        my $backupdir = $plugin->get_subdir($scfg, 'backup');
-        my $snippetsdir = $plugin->get_subdir($scfg, 'snippets');
-        my $importdir = $plugin->get_subdir($scfg, 'import');
+        my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
+        my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
+        my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
+        my $backupdir = plugin_get_vtype_subdir($scfg, 'backup');
+        my $snippetsdir = plugin_get_vtype_subdir($scfg, 'snippets');
+        my $importdir = plugin_get_vtype_subdir($scfg, 'import');
 
         if ($path =~ m!^\Q$imagedir\E/(\d+)/([^/\s]+)$!) {
             my $vmid = $1;
diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index fb47aa0b..e75b4fe0 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -11,6 +11,9 @@ use File::Path qw(mkpath);
 use IO::Dir;
 use POSIX qw(EEXIST);
 
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
 use PVE::Tools qw(run_command dir_glob_foreach);
 
 use PVE::Storage::DirPlugin;
@@ -188,7 +191,7 @@ sub filesystem_path {
 
     my ($vtype, $name, $vmid, undef, undef, $isBase, $format) = $class->parse_volname($volname);
 
-    my $path = $class->get_subdir($scfg, $vtype);
+    my $path = plugin_get_vtype_subdir($scfg, $vtype);
 
     $path .= "/$vmid" if $vtype eq 'images';
 
@@ -294,7 +297,7 @@ sub clone_image {
         return PVE::Storage::DirPlugin::clone_image(@_);
     }
 
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";
     mkpath $imagedir;
 
@@ -327,7 +330,7 @@ sub alloc_image {
 
     # From Plugin.pm:
 
-    my $imagedir = $class->get_subdir($scfg, 'images') . "/$vmid";
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images') . "/$vmid";
 
     mkpath $imagedir;
 
@@ -644,7 +647,7 @@ sub volume_has_feature {
 
 sub list_images {
     my ($class, $storeid, $scfg, $vmid, $vollist, $cache) = @_;
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
 
     my $res = [];
 
@@ -845,7 +848,7 @@ sub volume_import {
         $volname = $class->find_free_diskname($storeid, $scfg, $vmid, $volume_format, 1);
     }
 
-    my $imagedir = $class->get_subdir($scfg, $vtype);
+    my $imagedir = plugin_get_vtype_subdir($scfg, $vtype);
     $imagedir .= "/$vmid" if $vtype eq 'images';
 
     my $tmppath = "$imagedir/recv.$vmid.tmp";
@@ -976,7 +979,7 @@ sub rename_volume {
         if !$target_volname;
     $target_volname = "$target_vmid/$target_volname";
 
-    my $basedir = $class->get_subdir($scfg, 'images');
+    my $basedir = plugin_get_vtype_subdir($scfg, 'images');
 
     mkpath "${basedir}/${target_vmid}";
     my $source_dir = raw_name_to_dir($source_volname);
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 49bcc67c..f5c08d07 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -18,6 +18,9 @@ our @EXPORT_OK = qw(
     qemu_img_info
     qemu_img_measure
     qemu_img_resize
+
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
 );
 
 use constant {
@@ -25,6 +28,16 @@ use constant {
     FALLOC_FL_PUNCH_HOLE => 0x02, # see linux/falloc.h
 };
 
+my $DEFAULT_VTYPE_SUBDIRS = {
+    images => 'images',
+    rootdir => 'private',
+    iso => 'template/iso',
+    vztmpl => 'template/cache',
+    backup => 'dump',
+    snippets => 'snippets',
+    import => 'import',
+};
+
 =head1 NAME
 
 PVE::Storage::Common - Shared functions and utilities for storage plugins and storage operations
@@ -290,4 +303,43 @@ sub qemu_img_resize {
     run_command($cmd, timeout => $timeout);
 }
 
+=head2 FUNCTIONS FOR STORAGE PLUGINS AND CONFIGURATIONS
+
+=cut
+
+=head3 plugin_get_default_vtype_subdirs
+
+    my $vtype_subdirs = plugin_get_default_vtype_subdirs()
+
+Returns a hashref containing the default sub-directories for every volume type.
+
+=cut
+
+sub plugin_get_default_vtype_subdirs : prototype() () {
+    return { $DEFAULT_VTYPE_SUBDIRS->%* }; # shallow copy
+}
+
+=head3 plugin_get_vtype_subdir
+
+    my $subdir = plugin_get_vtype_subdir($scfg, $vtype)
+
+Returns the sub-directory for the volume type C<$vtype> of a given storage
+configuration C<$scfg> as an absolute path.
+
+Raises an exception if the storage config has no C<path> attribute or
+if the given C<$vtype> does not exist.
+
+=cut
+
+sub plugin_get_vtype_subdir : prototype($$) ($scfg, $vtype) {
+    my $path = $scfg->{path};
+
+    die "storage definition has no path\n" if !$path;
+    die "unknown vtype '$vtype'\n" if !exists($DEFAULT_VTYPE_SUBDIRS->{$vtype});
+
+    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};
+
+    return "$path/$subdir";
+}
+
 1;
diff --git a/src/PVE/Storage/ESXiPlugin.pm b/src/PVE/Storage/ESXiPlugin.pm
index 19f23bb9..b334e2ac 100644
--- a/src/PVE/Storage/ESXiPlugin.pm
+++ b/src/PVE/Storage/ESXiPlugin.pm
@@ -605,12 +605,6 @@ sub volume_has_feature {
     return undef;
 }
 
-sub get_subdir {
-    my ($class, $scfg, $vtype) = @_;
-
-    die "no subdirectories available for storage $class\n";
-}
-
 package PVE::Storage::ESXiPlugin::Manifest;
 
 use strict;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index bd59c024..db74711c 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -15,7 +15,10 @@ use PVE::Tools qw(run_command);
 use PVE::JSONSchema qw(get_standard_option register_standard_option);
 use PVE::Cluster qw(cfs_register_file);
 
-use PVE::Storage::Common;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
+);
 
 use JSON;
 
@@ -841,31 +844,16 @@ sub parse_volname {
     die "unable to parse directory volume name '$volname'\n";
 }
 
-my $vtype_subdirs = {
-    images => 'images',
-    rootdir => 'private',
-    iso => 'template/iso',
-    vztmpl => 'template/cache',
-    backup => 'dump',
-    snippets => 'snippets',
-    import => 'import',
-};
-
+# FIXME: remove on the next APIAGE reset.
 sub get_vtype_subdirs {
-    return $vtype_subdirs;
+    return plugin_get_default_vtype_subdirs();
 }
 
+# FIXME: remove on the next APIAGE reset.
 sub get_subdir {
     my ($class, $scfg, $vtype) = @_;
 
-    my $path = $scfg->{path};
-
-    die "storage definition has no path\n" if !$path;
-    die "unknown vtype '$vtype'\n" if !exists($vtype_subdirs->{$vtype});
-
-    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $vtype_subdirs->{$vtype};
-
-    return "$path/$subdir";
+    return plugin_get_vtype_subdir($scfg, $vtype);
 }
 
 my sub get_snap_name {
@@ -890,7 +878,7 @@ sub filesystem_path {
     die "can't snapshot this image format\n"
         if defined($snapname) && $format !~ m/^(qcow2|qed)$/;
 
-    my $dir = $class->get_subdir($scfg, $vtype);
+    my $dir = plugin_get_vtype_subdir($scfg, $vtype);
 
     $dir .= "/$vmid" if $vtype eq 'images';
 
@@ -1021,7 +1009,7 @@ sub clone_image {
 
     die "clone_image only works on base images\n" if !$isBase;
 
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";
 
     mkpath $imagedir;
@@ -1054,7 +1042,7 @@ sub clone_image {
 sub alloc_image {
     my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) = @_;
 
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
     $imagedir .= "/$vmid";
 
     mkpath $imagedir;
@@ -1634,7 +1622,7 @@ sub volume_has_feature {
 sub list_images {
     my ($class, $storeid, $scfg, $vmid, $vollist, $cache) = @_;
 
-    my $imagedir = $class->get_subdir($scfg, 'images');
+    my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
 
     my $format_info = $class->get_formats($scfg, $storeid);
     my $fmts = join('|', sort keys $format_info->{valid}->%*);
@@ -1782,7 +1770,7 @@ sub list_volumes {
         if ($type eq 'images' || $type eq 'rootdir') {
             $data = $class->list_images($storeid, $scfg, $vmid);
         } elsif ($scfg->{path}) {
-            my $path = $class->get_subdir($scfg, $type);
+            my $path = plugin_get_vtype_subdir($scfg, $type);
 
             if ($type eq 'iso' && !defined($vmid)) {
                 $data = get_subdir_files($storeid, $path, 'iso');
@@ -1974,7 +1962,7 @@ sub activate_storage {
     my $try_create_subdir = sub {
         my ($vtype) = @_;
 
-        my $subdir = $class->get_subdir($scfg, $vtype);
+        my $subdir = plugin_get_vtype_subdir($scfg, $vtype);
 
         File::Path::make_path($subdir, { error => \my $errors });
         if (defined($errors) && scalar($errors->@*)) {
@@ -2007,7 +1995,8 @@ sub activate_storage {
         # FIXME The mkdir option is deprecated. Remove with PVE 9?
         && (!defined($scfg->{mkdir}) || $scfg->{mkdir})
     ) {
-        for my $vtype (sort keys %$vtype_subdirs) {
+        my $vtype_subdirs = plugin_get_default_vtype_subdirs();
+        for my $vtype (sort keys $vtype_subdirs->%*) {
             # OpenVZMigrate uses backup (dump) dir
             if (
                 defined($scfg->{content}->{$vtype})
@@ -2021,7 +2010,7 @@ sub activate_storage {
     # check that content dirs are pairwise inequal
     my $resolved_subdirs = {};
     for my $vtype (sort keys $scfg->{content}->%*) {
-        my $subdir = $class->get_subdir($scfg, $vtype);
+        my $subdir = plugin_get_vtype_subdir($scfg, $vtype);
         my $abs_subdir = abs_path($subdir);
         next if !defined($abs_subdir);
 
@@ -2394,7 +2383,7 @@ sub rename_volume {
     $target_volname = $class->find_free_diskname($storeid, $scfg, $target_vmid, $format, 1)
         if !$target_volname;
 
-    my $basedir = $class->get_subdir($scfg, 'images');
+    my $basedir = plugin_get_vtype_subdir($scfg, 'images');
 
     mkpath "${basedir}/${target_vmid}";
 
diff --git a/src/test/get_subdir_test.pm b/src/test/get_subdir_test.pm
index 5fb54458..843c8c39 100644
--- a/src/test/get_subdir_test.pm
+++ b/src/test/get_subdir_test.pm
@@ -5,16 +5,20 @@ use warnings;
 
 use lib qw(..);
 
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+    plugin_get_vtype_subdir
+);
 use PVE::Storage::Plugin;
 use Test::More;
 
 my $scfg_with_path = { path => '/some/path' };
-my $vtype_subdirs = PVE::Storage::Plugin::get_vtype_subdirs();
+my $vtype_subdirs = plugin_get_default_vtype_subdirs();
 
 # each test is comprised of the following array keys:
 # [0] => storage config; positive with path key
 # [1] => storage type;  see $vtype_subdirs
-# [2] => expected return from get_subdir
+# [2] => expected return from plugin_get_vtype_subdir
 my $tests = [
     # failed matches
     [$scfg_with_path, 'none', "unknown vtype 'none'\n"],
@@ -45,10 +49,10 @@ foreach my $tt (@$tests) {
     my ($scfg, $type, $expected) = @$tt;
 
     my $got;
-    eval { $got = PVE::Storage::Plugin->get_subdir($scfg, $type) };
+    eval { $got = plugin_get_vtype_subdir($scfg, $type) };
     $got = $@ if $@;
 
-    is($got, $expected, "get_subdir for $type") || diag(explain($got));
+    is($got, $expected, "plugin_get_vtype_subdir for $type") || diag(explain($got));
 }
 
 done_testing();
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 5c9478ac..2ff4c6d0 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -6,6 +6,9 @@ use warnings;
 use lib qw(..);
 
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);
 use Test::More;
 
 my $vmid = 1234;
@@ -302,7 +305,8 @@ foreach my $virt (keys %$non_bkp_suffix) {
 plan tests => scalar @$tests + 1;
 
 my $seen_vtype;
-my $vtype_subdirs = { map { $_ => 1 } keys %{ PVE::Storage::Plugin::get_vtype_subdirs() } };
+my $vtype_subdirs =
+    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
 
 foreach my $t (@$tests) {
     my $description = $t->{description};
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index dfa51e64..e7e36037 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -6,6 +6,9 @@ use warnings;
 use lib qw(..);
 
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);
 
 use Test::More;
 
@@ -237,7 +240,8 @@ my @tests = (
 plan tests => scalar @tests + 1;
 
 my $seen_vtype;
-my $vtype_subdirs = { map { $_ => 1 } keys %{ PVE::Storage::Plugin::get_vtype_subdirs() } };
+my $vtype_subdirs =
+    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
 
 foreach my $tt (@tests) {
     my $file = $tt->{volname};
diff --git a/src/test/run_volume_access_tests.pl b/src/test/run_volume_access_tests.pl
index 34487083..31bacaa9 100755
--- a/src/test/run_volume_access_tests.pl
+++ b/src/test/run_volume_access_tests.pl
@@ -10,6 +10,9 @@ use lib ('.', '..');
 
 use PVE::RPCEnvironment;
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
+);
 use PVE::Storage::Plugin;
 
 my $storage_cfg = <<'EOF';
@@ -64,7 +67,7 @@ $pve_cluster_module->mock(
 my $rpcenv = PVE::RPCEnvironment->init('pub');
 $rpcenv->init_request();
 
-my @types = sort keys PVE::Storage::Plugin::get_vtype_subdirs()->%*;
+my @types = sort keys plugin_get_default_vtype_subdirs()->%*;
 my $all_types = { map { $_ => 1 } @types };
 
 my @tests = (
-- 
2.47.3

[PATCH pve-storage v2 07/50] common: prevent autovivification in plugin_get_vtype_subdir helper

[Max R. Carrara]

When accessing a non-existing (or undefined) value in a hash(ref) and
using this value again for a hash lookup, i.e.
`$hashref->{foo}->{bar}` where `$hashref->{foo}` does not exist or is
`undef`, Perl will (un)conveniently instantiate a new hashref where
one was absent before. This is also called autovivification [0].

This means that if the config hash of directory-based storage plugin
that does *not* declare the 'content-dirs' property in its `options()`
is passed to `plugin_get_vtype_subdir()`, the 'content-dirs' key will
be instantiated as an empty hashref during the
`$scfg->{'content-dirs'}->{$vtype}` lookup.

Note that "directory-based storage plugin" means any storage plugin
that defines the 'path' property in its `options()` method.

While this is a minor issue, the `$scfg` hashref should IMO never be
modified unless necessary, or if the method actually is supposed to
modify it.

Therefore, prevent this autovivification [0] in
`plugin_get_vtype_subdir()`.

Additionally, use string concatenation instead of interpolation for
the returned subdirectory path.

[0]: https://perldoc.perl.org/perlref#Autovivification

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index f5c08d07..4b200d9d 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -337,7 +337,9 @@ sub plugin_get_vtype_subdir : prototype($$) ($scfg, $vtype) {
     die "storage definition has no path\n" if !$path;
     die "unknown vtype '$vtype'\n" if !exists($DEFAULT_VTYPE_SUBDIRS->{$vtype});
 
-    my $subdir = $scfg->{"content-dirs"}->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};
+    # Intermediate step to prevent autovivification
+    my $content_dirs = $scfg->{'content-dirs'} // {};
+    my $subdir = $content_dirs->{$vtype} // $DEFAULT_VTYPE_SUBDIRS->{$vtype};
 
     return "$path/$subdir";
 }
-- 
2.47.3

[PATCH pve-storage v2 08/50] plugin: break up if-elsif chain into separate if-blocks

[Max R. Carrara]

Done in order to make future changes a bit easier.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index db74711c..475214ec 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -813,29 +813,43 @@ sub parse_volname {
         my ($vmid, $name) = ($3, $4);
         my (undef, $format, $isBase) = parse_name_dir($name);
         return ('images', $name, $vmid, $basename, $basedvmid, $isBase, $format);
-    } elsif ($volname =~ m!^(\d+)/(\S+)$!) {
+    }
+
+    if ($volname =~ m!^(\d+)/(\S+)$!) {
         my ($vmid, $name) = ($1, $2);
         my (undef, $format, $isBase) = parse_name_dir($name);
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
-    } elsif ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
+    }
+
+    if ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
         return ('iso', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
+    }
+
+    if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
         return ('vztmpl', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
+    }
+
+    if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
         my $fn = $1;
         if ($fn =~ m/^vzdump-(openvz|lxc|qemu)-(\d+)-.+/) {
             return ('backup', $fn, $2, undef, undef, undef, 'raw');
         }
         return ('backup', $fn, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~ m!^snippets/([^/]+)$!) {
+    }
+
+    if ($volname =~ m!^snippets/([^/]+)$!) {
         return ('snippets', $1, undef, undef, undef, undef, 'raw');
-    } elsif ($volname =~
+    }
+
+    if ($volname =~
         m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.ova\/${PVE::Storage::OVA_CONTENT_RE_1})$!
     ) {
         my $packed_image = $1;
         my $format = $2;
         return ('import', $packed_image, undef, undef, undef, undef, "ova+$format");
-    } elsif ($volname =~
+    }
+
+    if ($volname =~
         m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!
     ) {
         return ('import', $1, undef, undef, undef, undef, $2);
-- 
2.47.3

[PATCH pve-storage v2 09/50] plugin: adapt get_subdir_files helper of list_volumes API method

[Max R. Carrara]

Pull the body of the loop inside `get_subdir_files()` out into a
closure and break up the if-elsif chain into more readable if-clauses.

Strip the vtype subdirectory from the beginning of the file paths
being worked on inside that closure. This does not have an effect on
how any of the subsequent regexes match on the path / file name,
because they match at the end of a given path.

Additionally, take take the storage config hash `$scfg` instead of the
volume type subdir `$path` as parameter. Passing `$scfg` along
makes it possible to obtain the subdir for the given `$vtype` inside
the helper directly instead of passing both the vtype and its
associated subdirectory along.

All of these things make future changes easier to manage.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 95 ++++++++++++++++++++++++++-------------
 1 file changed, 64 insertions(+), 31 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 475214ec..cbcb96b0 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1699,37 +1699,54 @@ sub list_images {
 
 # $vtype = <iso|vztmpl|backup|snippets|import>
 my sub get_subdir_files {
-    my ($storeid, $path, $vtype, $vmid) = @_;
+    my ($storeid, $scfg, $vtype, $vmid) = @_;
+
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
 
     my $res = [];
 
-    for my $filename (<$path/*>) {
-        my $st = File::stat::stat($filename);
+    my $get_subdir_file_info = sub {
+        my ($path, $st) = @_;
 
-        next if (!$st || S_ISDIR($st->mode));
-
-        my $info;
+        # Strip the vtype subdir from beginning of the current path
+        # so that we don't have to take it into account when parsing the file name
+        my $filename = $path;
+        if ($filename !~ s!^\Q$vtype_subdir\E(?=/)!!) {
+            return;
+        }
 
         if ($vtype eq 'iso') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+            return if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
 
-            $info = { volid => "$storeid:iso/$1", format => 'iso' };
+            return {
+                volid => "$storeid:iso/$1",
+                format => 'iso',
+            };
+        }
 
-        } elsif ($vtype eq 'vztmpl') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+        if ($vtype eq 'vztmpl') {
+            return if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
 
-            $info = { volid => "$storeid:vztmpl/$1", format => $2 eq 'tar' ? $2 : "t$2" };
+            return {
+                volid => "$storeid:vztmpl/$1",
+                format => $2 eq 'tar' ? $2 : "t$2",
+            };
+        }
 
-        } elsif ($vtype eq 'backup') {
-            next if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
-            my $original = $filename;
+        if ($vtype eq 'backup') {
+            return if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+
+            my $original = $path;
             my $format = $2;
             $filename = $1;
 
             # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            next if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
+            return if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
 
-            $info = { volid => "$storeid:backup/$filename", format => $format };
+            my $info = {
+                volid => "$storeid:backup/$filename",
+                format => $format,
+            };
 
             my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};
 
@@ -1748,24 +1765,42 @@ my sub get_subdir_files {
             }
 
             $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
-        } elsif ($vtype eq 'snippets') {
 
-            $info = {
+            return $info;
+        }
+
+        if ($vtype eq 'snippets') {
+            return {
                 volid => "$storeid:snippets/" . basename($filename),
                 format => 'snippet',
             };
-        } elsif ($vtype eq 'import') {
-            next
+        }
+
+        if ($vtype eq 'import') {
+            return
                 if $filename !~
                 m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;
 
-            $info = { volid => "$storeid:import/$1", format => "$2" };
+            return {
+                volid => "$storeid:import/$1",
+                format => "$2",
+            };
         }
 
-        $info->{size} = $st->size;
-        $info->{ctime} //= $st->ctime;
+        return;
+    };
 
-        push @$res, $info;
+    for my $path (<$vtype_subdir/*>) {
+        my $st = File::stat::stat($path);
+
+        next if (!$st || S_ISDIR($st->mode));
+
+        if (defined(my $info = $get_subdir_file_info->($path, $st))) {
+            $info->{size} = $st->size;
+            $info->{ctime} //= $st->ctime;
+
+            push $res->@*, $info;
+        }
     }
 
     return $res;
@@ -1784,18 +1819,16 @@ sub list_volumes {
         if ($type eq 'images' || $type eq 'rootdir') {
             $data = $class->list_images($storeid, $scfg, $vmid);
         } elsif ($scfg->{path}) {
-            my $path = plugin_get_vtype_subdir($scfg, $type);
-
             if ($type eq 'iso' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $path, 'iso');
+                $data = get_subdir_files($storeid, $scfg, 'iso', undef);
             } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $path, 'vztmpl');
+                $data = get_subdir_files($storeid, $scfg, 'vztmpl', undef);
             } elsif ($type eq 'backup') {
-                $data = get_subdir_files($storeid, $path, 'backup', $vmid);
+                $data = get_subdir_files($storeid, $scfg, 'backup', $vmid);
             } elsif ($type eq 'snippets') {
-                $data = get_subdir_files($storeid, $path, 'snippets');
+                $data = get_subdir_files($storeid, $scfg, 'snippets', undef);
             } elsif ($type eq 'import') {
-                $data = get_subdir_files($storeid, $path, 'import');
+                $data = get_subdir_files($storeid, $scfg, 'import', undef);
             }
         }
 
-- 
2.47.3

[PATCH pve-storage v2 10/50] plugin: update code style of list_volumes plugin API method

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index cbcb96b0..5ed5a919 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1813,7 +1813,7 @@ sub list_volumes {
 
     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
-    foreach my $type (@$content_types) {
+    for my $type ($content_types->@*) {
         my $data;
 
         if ($type eq 'images' || $type eq 'rootdir') {
@@ -1834,7 +1834,7 @@ sub list_volumes {
 
         next if !$data;
 
-        foreach my $item (@$data) {
+        for my $item ($data->@*) {
             if ($type eq 'images' || $type eq 'rootdir') {
                 my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
                 my $vmtype;
@@ -1851,7 +1851,7 @@ sub list_volumes {
                 $item->{content} = $type;
             }
 
-            push @$res, $item;
+            push $res->@*, $item;
         }
     }
 
-- 
2.47.3

[PATCH pve-storage v2 11/50] plugin: use closure for obtaining raw volume data in list_volumes

[Max R. Carrara]

To make the code in the `list_volumes()` API method a little more
readable and maintainable, break up the if-elsif chain that obtains
the raw volume data into a closure. The overall logic is kept the
same.

Also rename `$data` to `$raw_volumes` to make it a little clearer what
is actually being iterated over, since "data" does not really say
much.

Be a little more specific and use an un-definedness check on
`$raw_volumes` to continue with the iteration, instead of a plain
negation.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 51 ++++++++++++++++++++++++++-------------
 1 file changed, 34 insertions(+), 17 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 5ed5a919..99bc573d 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1813,28 +1813,45 @@ sub list_volumes {
 
     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
-    for my $type ($content_types->@*) {
-        my $data;
+
+    my $get_raw_volumes_for_type = sub {
+        my ($type) = @_;
 
         if ($type eq 'images' || $type eq 'rootdir') {
-            $data = $class->list_images($storeid, $scfg, $vmid);
-        } elsif ($scfg->{path}) {
-            if ($type eq 'iso' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $scfg, 'iso', undef);
-            } elsif ($type eq 'vztmpl' && !defined($vmid)) {
-                $data = get_subdir_files($storeid, $scfg, 'vztmpl', undef);
-            } elsif ($type eq 'backup') {
-                $data = get_subdir_files($storeid, $scfg, 'backup', $vmid);
-            } elsif ($type eq 'snippets') {
-                $data = get_subdir_files($storeid, $scfg, 'snippets', undef);
-            } elsif ($type eq 'import') {
-                $data = get_subdir_files($storeid, $scfg, 'import', undef);
-            }
+            return $class->list_images($storeid, $scfg, $vmid);
         }
 
-        next if !$data;
+        return if !$scfg->{path};
 
-        for my $item ($data->@*) {
+        if ($type eq 'iso' && !defined($vmid)) {
+            return get_subdir_files($storeid, $scfg, 'iso', undef);
+        }
+
+        if ($type eq 'vztmpl' && !defined($vmid)) {
+            return get_subdir_files($storeid, $scfg, 'vztmpl', undef);
+        }
+
+        if ($type eq 'backup') {
+            return get_subdir_files($storeid, $scfg, 'backup', $vmid);
+        }
+
+        if ($type eq 'snippets') {
+            return get_subdir_files($storeid, $scfg, 'snippets', undef);
+        }
+
+        if ($type eq 'import') {
+            return get_subdir_files($storeid, $scfg, 'import', undef);
+        }
+
+        return;
+    };
+
+    for my $type ($content_types->@*) {
+        my $raw_volumes = $get_raw_volumes_for_type->($type);
+
+        next if !defined($raw_volumes);
+
+        for my $item ($raw_volumes->@*) {
             if ($type eq 'images' || $type eq 'rootdir') {
                 my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
                 my $vmtype;
-- 
2.47.3

[PATCH pve-storage v2 12/50] plugin: use closure for inner loop logic in list_volumes

[Max R. Carrara]

Factor most of the body of the inner loop of the `list_volumes()`
plugin API method into a closure and provide additional context for
why we have such an elaborate check for VM / CT disks in the first
place.

This has the added benefit that we now also do not modify the volume
hashes on the fly anymore when determining whether a volume's type
aligns with a guest's type. Small difference, but might make it a
little easier to adapt that code in the future, e.g. when we align
vtypes and content types.

Also, rename `$item` to `$volume` for clarity's sake.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 49 +++++++++++++++++++++++++--------------
 1 file changed, 32 insertions(+), 17 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 99bc573d..6a113dcb 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1846,29 +1846,44 @@ sub list_volumes {
         return;
     };
 
+    my $get_content_type_for_volume = sub {
+        my ($volume, $type) = @_;
+
+        # It is currently impossible to determine whether a volume belongs to a
+        # VM or a CT based on the output of list_images alone.
+        # Therefore, we use the parsed ID ($volume->{vmid}) to look up the guest
+        # and make sure that the guest's type corresponds to the volume's type.
+        if ($type eq 'images' || $type eq 'rootdir') {
+            my $vminfo = $vmlist->{ids}->{ $volume->{vmid} };
+
+            my $vmtype;
+            if (defined($vminfo)) {
+                $vmtype = $vminfo->{type};
+            }
+
+            my $content_type = 'images';
+            if (defined($vmtype) && $vmtype eq 'lxc') {
+                $content_type = 'rootdir';
+            }
+
+            return if $content_type ne $type;
+        }
+
+        return $type;
+    };
+
     for my $type ($content_types->@*) {
         my $raw_volumes = $get_raw_volumes_for_type->($type);
 
         next if !defined($raw_volumes);
 
-        for my $item ($raw_volumes->@*) {
-            if ($type eq 'images' || $type eq 'rootdir') {
-                my $vminfo = $vmlist->{ids}->{ $item->{vmid} };
-                my $vmtype;
-                if (defined($vminfo)) {
-                    $vmtype = $vminfo->{type};
-                }
-                if (defined($vmtype) && $vmtype eq 'lxc') {
-                    $item->{content} = 'rootdir';
-                } else {
-                    $item->{content} = 'images';
-                }
-                next if $type ne $item->{content};
-            } else {
-                $item->{content} = $type;
-            }
+        for my $volume ($raw_volumes->@*) {
+            my $content_type = $get_content_type_for_volume->($volume, $type);
 
-            push $res->@*, $item;
+            next if !defined($content_type);
+
+            $volume->{content} = $content_type;
+            push $res->@*, $volume;
         }
     }
 
-- 
2.47.3

[PATCH pve-storage v2 13/50] storage: update code style in function path_to_volume_id

[Max R. Carrara]

Also rename the initially parsed storeid from `$sid` to
`$parsed_storeid`.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index ed87397e..589748d4 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -696,9 +696,9 @@ sub path_to_volume_id {
 
     my $ids = $cfg->{ids};
 
-    my ($sid, $volname) = parse_volume_id($path, 1);
-    if ($sid) {
-        if (my $scfg = $ids->{$sid}) {
+    my ($parsed_storeid, $volname) = parse_volume_id($path, 1);
+    if ($parsed_storeid) {
+        if (my $scfg = $ids->{$parsed_storeid}) {
             if ($scfg->{path}) {
                 my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
                 my ($vtype, $name, $vmid) = $plugin->parse_volname($volname);
@@ -712,7 +712,7 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;
 
-    foreach my $sid (keys %$ids) {
+    for my $sid (keys $ids->%*) {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
@@ -728,7 +728,7 @@ sub path_to_volume_id {
             my $name = $2;
 
             my $vollist = $plugin->list_images($sid, $scfg, $vmid);
-            foreach my $info (@$vollist) {
+            for my $info ($vollist->@*) {
                 my ($storeid, $volname) = parse_volume_id($info->{volid});
                 my $volpath = $plugin->path($scfg, $volname, $storeid);
                 if ($volpath eq $path) {
-- 
2.47.3

[PATCH pve-storage v2 14/50] storage: break up if-elsif chain in path_to_volume_id

[Max R. Carrara]

Done in order to make future changes a little easier.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 589748d4..7a3289f3 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -735,19 +735,31 @@ sub path_to_volume_id {
                     return ('images', $info->{volid});
                 }
             }
-        } elsif ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
+
+            return ('');
+        }
+
+        if ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
             my $name = $1;
             return ('iso', "$sid:iso/$name");
-        } elsif ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
+        }
+
+        if ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
             my $name = $1;
             return ('vztmpl', "$sid:vztmpl/$name");
-        } elsif ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
+        }
+
+        if ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
             my $name = $1;
             return ('backup', "$sid:backup/$name");
-        } elsif ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
+        }
+
+        if ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
             my $name = $1;
             return ('snippets', "$sid:snippets/$name");
-        } elsif ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
+        }
+
+        if ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
             my $name = $1;
             return ('import', "$sid:import/$name");
         }
-- 
2.47.3

[PATCH pve-storage v2 15/50] storage: heave vtype file path parsing logic inside loop into helper

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 7a3289f3..bb3e565d 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -712,10 +712,9 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;
 
-    for my $sid (keys $ids->%*) {
-        my $scfg = $ids->{$sid};
-        next if !$scfg->{path};
-        my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+    my $parse_volid_from_file_path = sub {
+        my ($plugin, $sid, $scfg) = @_;
+
         my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
         my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
         my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
@@ -763,6 +762,17 @@ sub path_to_volume_id {
             my $name = $1;
             return ('import', "$sid:import/$name");
         }
+
+        return ('');
+    };
+
+    for my $sid (keys $ids->%*) {
+        my $scfg = $ids->{$sid};
+        next if !$scfg->{path};
+        my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+        my ($vtype, $volid) = $parse_volid_from_file_path->($plugin, $sid, $scfg);
+        return ($vtype, $volid) if $vtype;
     }
 
     # can't map path to volume id
-- 
2.47.3

[PATCH pve-storage v2 16/50] storage: clean up code that was moved into helper in path_to_volume_id

[Max R. Carrara]

In order to make the parsing logic in `path_to_volume_id()` both
simpler and more manageable for future changes, do the following:

1. Iterate over a list of vtypes that store their contents in
   subdirectories and fetch said subdirectories dynamically instead of
   hardcoding them.

2. Explicitly exclude the `rootdir` vtype in this list of vtypes, as
   its subdirectory is currently not used.

3. Strip the vtype subdirectory from the beginning of the path in
   order to make matching on the file name easier. Return early as a
   safeguard if this fails.

4. Adapt the original regexes in the parsing helper to not include the
   vtype subdir anymore, since this is now being stripped.

5. Instead of returning a 2-element array, return `undef` if the path
   fails to parse, and the volume ID (volid) alone if parsing
   succeeds. Since the caller is now passing along the vtype, it does
   not need to be returned anymore.

Note that the parsing logic of the `parse_volid_from_file_path()`
helper inside `path_to_volume_id()` now is very similar to the parsing
logic of the `get_subdir_file_info()` closure in the
`get_subdir_files()` helper inside `Plugin.pm`. This opens up further
simplification / code de-duplication in the future, so that our
parsing logic can eventually be kept in one place only.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm | 73 +++++++++++++++++++++++++++++++---------------
 1 file changed, 50 insertions(+), 23 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index bb3e565d..485bd822 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -23,6 +23,7 @@ use PVE::INotify;
 use PVE::RPCEnvironment;
 use PVE::SSHInfo;
 use PVE::Storage::Common qw(
+    plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
 use PVE::RESTEnvironment qw(log_warn);
@@ -712,17 +713,31 @@ sub path_to_volume_id {
     # for example when nfs storage is not mounted
     $path = abs_path($path) || $path;
 
+    my $get_vtypes_to_check = sub {
+        my $default_vtype_subdirs = plugin_get_default_vtype_subdirs();
+
+        # TODO: vtype split: handle directory for 'rootdir' type.
+        # CTs currently use the same dir as VMs --> the dir for the 'images' type.
+        # Directory is therefore unused right now, so exclude it.
+        delete $default_vtype_subdirs->{rootdir};
+
+        return [sort keys $default_vtype_subdirs->%*];
+    };
+
     my $parse_volid_from_file_path = sub {
-        my ($plugin, $sid, $scfg) = @_;
+        my ($plugin, $sid, $scfg, $vtype) = @_;
 
-        my $imagedir = plugin_get_vtype_subdir($scfg, 'images');
-        my $isodir = plugin_get_vtype_subdir($scfg, 'iso');
-        my $tmpldir = plugin_get_vtype_subdir($scfg, 'vztmpl');
-        my $backupdir = plugin_get_vtype_subdir($scfg, 'backup');
-        my $snippetsdir = plugin_get_vtype_subdir($scfg, 'snippets');
-        my $importdir = plugin_get_vtype_subdir($scfg, 'import');
+        my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
 
-        if ($path =~ m!^\Q$imagedir\E/(\d+)/([^/\s]+)$!) {
+        # Strip the vtype subdir from beginning of the current path
+        # so that we don't have to take it into account when parsing the file name
+        my $filename = $path;
+        if ($filename !~ s!^\Q$vtype_subdir\E(?=/)!!) {
+            return;
+        }
+
+        if ($vtype eq 'images') {
+            return if $filename !~ m!/(\d+)/([^/\s]+)$!;
             my $vmid = $1;
             my $name = $2;
 
@@ -731,48 +746,60 @@ sub path_to_volume_id {
                 my ($storeid, $volname) = parse_volume_id($info->{volid});
                 my $volpath = $plugin->path($scfg, $volname, $storeid);
                 if ($volpath eq $path) {
-                    return ('images', $info->{volid});
+                    return $info->{volid};
                 }
             }
 
-            return ('');
+            return;
         }
 
-        if ($path =~ m!^\Q$isodir\E/([^/]+$ISO_EXT_RE_0)$!) {
+        if ($vtype eq 'iso') {
+            return if $filename !~ m!/([^/]+$ISO_EXT_RE_0)$!;
             my $name = $1;
-            return ('iso', "$sid:iso/$name");
+            return "$sid:iso/$name";
         }
 
-        if ($path =~ m!^\Q$tmpldir\E/([^/]+$VZTMPL_EXT_RE_1)$!) {
+        if ($vtype eq 'vztmpl') {
+            return if $filename !~ m!/([^/]+$VZTMPL_EXT_RE_1)$!;
             my $name = $1;
-            return ('vztmpl', "$sid:vztmpl/$name");
+            return "$sid:vztmpl/$name";
         }
 
-        if ($path =~ m!^\Q$backupdir\E/([^/]+$BACKUP_EXT_RE_2)$!) {
+        if ($vtype eq 'backup') {
+            return if $filename !~ m!/([^/]+$BACKUP_EXT_RE_2)$!;
             my $name = $1;
-            return ('backup', "$sid:backup/$name");
+            return "$sid:backup/$name";
         }
 
-        if ($path =~ m!^\Q$snippetsdir\E/([^/]+)$!) {
+        if ($vtype eq 'snippets') {
+            return if $filename !~ m!/([^/]+)$!;
             my $name = $1;
-            return ('snippets', "$sid:snippets/$name");
+            return "$sid:snippets/$name";
         }
 
-        if ($path =~ m!^\Q$importdir\E/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!) {
+        if ($vtype eq 'import') {
+            return if $filename !~ m!/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!;
             my $name = $1;
-            return ('import', "$sid:import/$name");
+            return "$sid:import/$name";
         }
 
-        return ('');
+        return;
     };
 
+    my $vtypes_to_check = $get_vtypes_to_check->();
+
     for my $sid (keys $ids->%*) {
         my $scfg = $ids->{$sid};
         next if !$scfg->{path};
         my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
 
-        my ($vtype, $volid) = $parse_volid_from_file_path->($plugin, $sid, $scfg);
-        return ($vtype, $volid) if $vtype;
+        for my $vtype ($vtypes_to_check->@*) {
+            my $volid = $parse_volid_from_file_path->($plugin, $sid, $scfg, $vtype);
+
+            if (defined($volid)) {
+                return ($vtype, $volid);
+            }
+        }
     }
 
     # can't map path to volume id
-- 
2.47.3

[PATCH pve-storage v2 17/50] api: status: move content type assert for up-/downloads into helper

[Max R. Carrara]

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 41 +++++++++++++++++++++-------------
 1 file changed, 26 insertions(+), 15 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 741d5144..40729465 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -43,6 +43,29 @@ __PACKAGE__->register_method({
     path => '{storage}/file-restore',
 });
 
+# Content types that may currently be up- or downloaded via the API
+my @ALLOWED_FILE_TRANSFER_CONTENT_TYPES = ('iso', 'vztmpl', 'import');
+
+my sub assert_can_transfer_files : prototype($$$) {
+    my ($scfg, $storeid, $vtype) = @_;
+
+    if (!defined($scfg->{path})) {
+        die "unable to transfer files from / to storage type '$scfg->{type}'"
+            . " - not a file-based storage\n";
+    }
+
+    if (!$scfg->{content}->{$vtype}) {
+        die "storage '$storeid' is not configured for content type '$vtype'\n";
+    }
+
+    my $is_allowed_vtype = grep { $_ eq $vtype } @ALLOWED_FILE_TRANSFER_CONTENT_TYPES;
+    if (!$is_allowed_vtype) {
+        raise_param_exc({ content => "content type '$vtype' not allowed for upload / download" });
+    }
+
+    return;
+}
+
 my sub assert_ova_contents {
     my ($file) = @_;
 
@@ -576,11 +599,10 @@ __PACKAGE__->register_method({
         my ($node, $storage) = $param->@{qw(node storage)};
         my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);
 
-        die "can't upload to storage type '$scfg->{type}'\n"
-            if !defined($scfg->{path});
-
         my $content = $param->{content};
 
+        assert_can_transfer_files($scfg, $storage, $content);
+
         my $tmpfilename = $param->{tmpfilename};
         die "missing temporary file name\n" if !$tmpfilename;
 
@@ -618,13 +640,8 @@ __PACKAGE__->register_method({
             }
 
             $path = PVE::Storage::get_import_dir($cfg, $storage);
-        } else {
-            raise_param_exc({ content => "upload content type '$content' not allowed" });
         }
 
-        die "storage '$storage' does not support '$content' content\n"
-            if !$scfg->{content}->{$content};
-
         my $dest = "$path/$filename";
         my $dirname = dirname($dest);
 
@@ -820,13 +837,9 @@ __PACKAGE__->register_method({
         my ($node, $storage, $compression) = $param->@{qw(node storage compression)};
         my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);
 
-        die "can't upload to storage type '$scfg->{type}', not a file based storage!\n"
-            if !defined($scfg->{path});
-
         my ($content, $url) = $param->@{ 'content', 'url' };
 
-        die "storage '$storage' is not configured for content-type '$content'\n"
-            if !$scfg->{content}->{$content};
+        assert_can_transfer_files($scfg, $storage, $content);
 
         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
 
@@ -859,8 +872,6 @@ __PACKAGE__->register_method({
             }
 
             $path = PVE::Storage::get_import_dir($cfg, $storage);
-        } else {
-            raise_param_exc({ content => "upload content-type '$content' is not allowed" });
         }
 
         PVE::Storage::activate_storage($cfg, $storage);
-- 
2.47.3

[PATCH pve-storage v2 18/50] api: status: use helper from common module to get content directory

[Max R. Carrara]

In the subroutines for the upload / download_url API methods, use the
`plugin_get_vtype_subdir()` helper from `PVE::Storage::Common` to
obtain the sub-directory for the provided content type (vtype),
instead of doing that within the if-elsif chains.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 40729465..09a37f4b 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -20,6 +20,9 @@ use PVE::API2::Storage::Content;
 use PVE::API2::Storage::FileRestore;
 use PVE::API2::Storage::PruneBackups;
 use PVE::Storage;
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
 
 use base qw(PVE::RESTHandler);
 
@@ -611,7 +614,6 @@ __PACKAGE__->register_method({
 
         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
 
-        my $path;
         my $is_ova = 0;
         my $image_format;
 
@@ -619,12 +621,10 @@ __PACKAGE__->register_method({
             if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_iso_dir($cfg, $storage);
         } elsif ($content eq 'vztmpl') {
             if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_vztmpl_dir($cfg, $storage);
         } elsif ($content eq 'import') {
             if ($filename !~
                 m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
@@ -638,10 +638,10 @@ __PACKAGE__->register_method({
             } else {
                 $image_format = $format;
             }
-
-            $path = PVE::Storage::get_import_dir($cfg, $storage);
         }
 
+        my $path = plugin_get_vtype_subdir($scfg, $content);
+
         my $dest = "$path/$filename";
         my $dirname = dirname($dest);
 
@@ -843,7 +843,6 @@ __PACKAGE__->register_method({
 
         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
 
-        my $path;
         my $is_ova = 0;
         my $image_format;
 
@@ -851,12 +850,10 @@ __PACKAGE__->register_method({
             if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_iso_dir($cfg, $storage);
         } elsif ($content eq 'vztmpl') {
             if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
                 raise_param_exc({ filename => "wrong file extension" });
             }
-            $path = PVE::Storage::get_vztmpl_dir($cfg, $storage);
         } elsif ($content eq 'import') {
             if ($filename !~
                 m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
@@ -870,10 +867,10 @@ __PACKAGE__->register_method({
             } else {
                 $image_format = $format;
             }
-
-            $path = PVE::Storage::get_import_dir($cfg, $storage);
         }
 
+        my $path = plugin_get_vtype_subdir($scfg, $content);
+
         PVE::Storage::activate_storage($cfg, $storage);
         File::Path::make_path($path);
 
-- 
2.47.3

[PATCH pve-storage v2 19/50] api: status: move up-/download file path parsing code into helper

[Max R. Carrara]

Create a separate helper for the common file path parsing parts of the
upload / download_url API methods that returns the extension of the
uploaded file. Break up the if-elsif chain along the way.

Keep the inner if-clauses for the 'import' vtype inline for now for
further simplifications later, but don't check whether we have the
'import' vtype before those anymore. This is fine because the 'ova'
file extension is not used for files of any other vtype.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 85 ++++++++++++++++++----------------
 1 file changed, 45 insertions(+), 40 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 09a37f4b..a9b4f798 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -69,6 +69,41 @@ my sub assert_can_transfer_files : prototype($$$) {
     return;
 }
 
+my sub parse_transferred_file_path_extension : prototype($$) {
+    my ($path, $vtype) = @_;
+
+    if ($vtype eq 'iso') {
+        if ($path !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    if ($vtype eq 'vztmpl') {
+        if ($path !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    if ($vtype eq 'import') {
+        if (
+            $path !~ m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
+        ) {
+            raise_param_exc({ filename => "invalid filename or wrong extension" });
+        }
+
+        my $ext = $1;
+        return $ext;
+    }
+
+    die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
+}
+
 my sub assert_ova_contents {
     my ($file) = @_;
 
@@ -617,27 +652,12 @@ __PACKAGE__->register_method({
         my $is_ova = 0;
         my $image_format;
 
-        if ($content eq 'iso') {
-            if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'vztmpl') {
-            if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'import') {
-            if ($filename !~
-                m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-            ) {
-                raise_param_exc({ filename => "invalid filename or wrong extension" });
-            }
-            my $format = $1;
+        my $extension = parse_transferred_file_path_extension($filename, $content);
 
-            if ($format eq 'ova') {
-                $is_ova = 1;
-            } else {
-                $image_format = $format;
-            }
+        if ($extension eq 'ova') {
+            $is_ova = 1;
+        } else {
+            $image_format = $extension;
         }
 
         my $path = plugin_get_vtype_subdir($scfg, $content);
@@ -846,27 +866,12 @@ __PACKAGE__->register_method({
         my $is_ova = 0;
         my $image_format;
 
-        if ($content eq 'iso') {
-            if ($filename !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'vztmpl') {
-            if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
-                raise_param_exc({ filename => "wrong file extension" });
-            }
-        } elsif ($content eq 'import') {
-            if ($filename !~
-                m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-            ) {
-                raise_param_exc({ filename => "invalid filename or wrong extension" });
-            }
-            my $format = $1;
+        my $extension = parse_transferred_file_path_extension($filename, $content);
 
-            if ($format eq 'ova') {
-                $is_ova = 1;
-            } else {
-                $image_format = $format;
-            }
+        if ($extension eq 'ova') {
+            $is_ova = 1;
+        } else {
+            $image_format = $extension;
         }
 
         my $path = plugin_get_vtype_subdir($scfg, $content);
-- 
2.47.3

[PATCH pve-storage v2 20/50] api: status: simplify file content assertion logic for up-/download

[Max R. Carrara]

Create another helper for the file content assertion logic in the
upload / download_url API methods. Absorb the `assert_ova_contents()`
helper along the way, since it does not need to be standalone anymore.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 89 ++++++++++++----------------------
 1 file changed, 32 insertions(+), 57 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index a9b4f798..45a79d56 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -104,25 +104,40 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
 }
 
-my sub assert_ova_contents {
-    my ($file) = @_;
+my sub assert_file_transfer_contents_valid : prototype($$$) {
+    my ($path, $vtype, $extension) = @_;
 
-    # test if it's really a tar file with an ovf file inside
-    my $hasOvf = 0;
-    run_command(
-        ['tar', '-t', '-f', $file],
-        outfunc => sub {
-            my ($line) = @_;
+    if ($vtype eq 'iso') {
+        PVE::Storage::assert_iso_content($path);
+        return;
+    }
 
-            if ($line =~ m/\.ovf$/) {
-                $hasOvf = 1;
-            }
-        },
-    );
+    if ($vtype eq 'import') {
+        if ($extension eq 'ova') {
+            # test if it's really a tar file with an ovf file inside
+            my $hasOvf = 0;
+            run_command(
+                ['tar', '-t', '-f', $path],
+                outfunc => sub {
+                    my ($line) = @_;
 
-    die "ova archive has no .ovf file inside\n" if !$hasOvf;
+                    if ($line =~ m/\.ovf$/) {
+                        $hasOvf = 1;
+                    }
+                },
+            );
 
-    return 1;
+            die "ova archive has no .ovf file inside\n" if !$hasOvf;
+            return;
+        }
+
+        my $is_untrusted = 1;
+        my $timeout = 10;
+
+        PVE::Storage::file_size_info($path, $timeout, $extension, $is_untrusted);
+
+        return;
+    }
 }
 
 __PACKAGE__->register_method({
@@ -648,18 +663,7 @@ __PACKAGE__->register_method({
         die "temporary file '$tmpfilename' does not exist\n" if !defined($size);
 
         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
-
-        my $is_ova = 0;
-        my $image_format;
-
         my $extension = parse_transferred_file_path_extension($filename, $content);
-
-        if ($extension eq 'ova') {
-            $is_ova = 1;
-        } else {
-            $image_format = $extension;
-        }
-
         my $path = plugin_get_vtype_subdir($scfg, $content);
 
         my $dest = "$path/$filename";
@@ -728,16 +732,7 @@ __PACKAGE__->register_method({
                     }
                 }
 
-                if ($content eq 'iso') {
-                    PVE::Storage::assert_iso_content($tmpfilename);
-                }
-
-                if ($is_ova) {
-                    assert_ova_contents($tmpfilename);
-                } elsif (defined($image_format)) {
-                    # checks untrusted image
-                    PVE::Storage::file_size_info($tmpfilename, 10, $image_format, 1);
-                }
+                assert_file_transfer_contents_valid($tmpfilename, $content, $extension);
             };
             if (my $err = $@) {
                 # unlinks only the temporary file from the http server
@@ -862,18 +857,7 @@ __PACKAGE__->register_method({
         assert_can_transfer_files($scfg, $storage, $content);
 
         my $filename = PVE::Storage::normalize_content_filename($param->{filename});
-
-        my $is_ova = 0;
-        my $image_format;
-
         my $extension = parse_transferred_file_path_extension($filename, $content);
-
-        if ($extension eq 'ova') {
-            $is_ova = 1;
-        } else {
-            $image_format = $extension;
-        }
-
         my $path = plugin_get_vtype_subdir($scfg, $content);
 
         PVE::Storage::activate_storage($cfg, $storage);
@@ -896,16 +880,7 @@ __PACKAGE__->register_method({
         $opts->{assert_file_validity} = sub {
             my ($tmp_path) = @_;
 
-            if ($content eq 'iso') {
-                PVE::Storage::assert_iso_content($tmp_path);
-            }
-
-            if ($is_ova) {
-                assert_ova_contents($tmp_path);
-            } elsif (defined($image_format)) {
-                # checks untrusted image
-                PVE::Storage::file_size_info($tmp_path, 10, $image_format, 1);
-            }
+            assert_file_transfer_contents_valid($tmp_path, $content, $extension);
         };
 
         my $worker = sub {
-- 
2.47.3

[PATCH pve-storage v2 21/50] test: guest import: add tests for PVE::GuestImport

[Max R. Carrara]

Set up a new test script for the `extract_disk_from_import_file()` sub
of `PVE::GuestImport` and run it as part of
`src/test/run_plugin_tests.pl`.

Make this possible by mocking cluster-related functionality that would
otherwise require to be run as root, namely reading, updating and
locking the storage config, as well as retrieving the list of
available VMs.

Then, make the test cases declarative in such a way that they are
mostly self-documenting regarding what they test. Document the keys
that the test declarations use.

Run each test in its clean environment, creating and removing the
entire directory structure before and after each test. Provide small
helpers specific to the testing code for creating arbitrary QEMU
images / .ova files and document them briefly.

Test for successful as well as failed imports.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/guest_import_test.pl | 948 ++++++++++++++++++++++++++++++++++
 src/test/run_plugin_tests.pl  |   1 +
 2 files changed, 949 insertions(+)
 create mode 100755 src/test/guest_import_test.pl

diff --git a/src/test/guest_import_test.pl b/src/test/guest_import_test.pl
new file mode 100755
index 00000000..04eec24d
--- /dev/null
+++ b/src/test/guest_import_test.pl
@@ -0,0 +1,948 @@
+#!/usr/bin/env perl
+
+use v5.36;
+
+use lib qw(..);
+
+use re qw(is_regexp);
+
+use Carp qw(confess);
+use File::Basename qw(dirname basename);
+use File::Path qw(make_path remove_tree);
+use File::Spec;
+use File::Temp;
+
+use PVE::Cluster;
+use PVE::GuestImport;
+use PVE::Storage;
+use PVE::Storage::Common qw(plugin_get_vtype_subdir);
+use PVE::Tools qw(run_command);
+
+use Test::More;
+use Test::MockModule;
+
+$ENV{TZ} = 'UTC';
+
+my $DEFAULT_SIZE = 4 * 1024 * 1024; # 4 MiB
+
+my $SOURCE_STOREID = 'source-store';
+my $SOURCE_STORAGE_PATH = File::Temp->newdir();
+
+my $TARGET_STOREID = 'target-store';
+my $TARGET_STORAGE_PATH = File::Temp->newdir();
+
+# Not actually set up on disk or elsewhere
+my $DUMMY_LVMTHIN_STOREID = 'dummy-lvmthin';
+
+my $STORAGE_CFG = <<~EOF;
+dir: $SOURCE_STOREID
+    shared 0
+    path $SOURCE_STORAGE_PATH
+    content images,iso,import
+
+dir: $TARGET_STOREID
+    shared 0
+    path $TARGET_STORAGE_PATH
+    content images,iso,import
+
+lvmthin: $DUMMY_LVMTHIN_STOREID
+    thinpool data
+    vgname pve
+    content images,rootdir
+EOF
+
+my $MOCKED_VMLIST = {
+    version => 1,
+    ids => {
+        1337 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 4,
+        },
+        1338 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 7,
+        },
+        1339 => {
+            node => 'node-0',
+            type => 'qemu',
+            version => 7,
+        },
+    },
+};
+
+my $mock_pve_cluster = Test::MockModule->new('PVE::Cluster')->redefine(
+    cfs_update => sub { },
+    get_config => sub($file) {
+        if ($file eq 'storage.cfg') {
+            return $STORAGE_CFG;
+        }
+
+        confess "mocked get_config() - file '$file' not handled";
+    },
+    get_vmlist => sub() { return $MOCKED_VMLIST; },
+);
+
+my $_LOCKDIR = File::Temp->newdir();
+my $mock_pve_storage_plugin = Test::MockModule->new('PVE::Storage::Plugin')->redefine(
+    cluster_lock_storage => sub($class, $storeid, $shared, $timeout, $func, @param) {
+        confess "\$shared = 1 is not handled" if $shared;
+
+        my $res = PVE::Tools::lock_file("$_LOCKDIR/pve-storage-test", $timeout, $func, @param);
+        die $@ if $@;
+        return $res;
+    },
+);
+
+=head3 create_test_image_file
+
+    my $file_path = "/foo/bar/disk.qcow2";
+    create_test_image_file($file_path, 'qcow2');
+
+Create a new QEMU image at C<$file_path> using the C<qemu-img> command,
+optionally with a certain C<$format>.
+
+The known formats are currently C<qcow2>, C<raw>, and C<vmdk>.
+
+=cut
+
+my sub create_test_image_file : prototype($;$) ($file_path, $format = undef) {
+    my $KNOWN_FORMATS = {
+        qcow2 => 1,
+        raw => 1,
+        vmdk => 1,
+    };
+
+    my $command = ['/usr/bin/qemu-img', 'create', $file_path, $DEFAULT_SIZE];
+
+    if (defined($format) && defined($KNOWN_FORMATS->{$format})) {
+        push($command->@*, '-f', $format);
+    }
+
+    eval { run_command($command, timeout => 10); };
+    confess $@ if $@;
+
+    return;
+}
+
+=head3 create_test_ova_file
+
+    my $file_path = "/srv/imports/some-import.ova";
+    my $content_file_name = "some-disk.qcow2";
+
+    create_test_ova_file($file_path, $content_file_name)
+
+Create a new C<.ova> file at the I<absolute> C<$file_path> and put a file named
+C<$content_file_name> inside using the C<tar> command. C<$content_file_name>
+must not have any directory components and is assumed to be in the same
+directory as C<$file_path>.
+
+=cut
+
+my sub create_test_ova_file : prototype($$) ($file_path, $content_file_name) {
+    my $file_path_dir = dirname($file_path);
+
+    confess "\$content_file_name ne basename(\$content_file_name)"
+        if $content_file_name ne basename($content_file_name);
+
+    my $tar_command = [
+        'tar',
+        '-c',
+        '--force-local',
+        '--no-same-owner',
+        '-f',
+        $file_path,
+        '-C',
+        $file_path_dir,
+        $content_file_name,
+    ];
+
+    eval { run_command($tar_command, timeout => 10); };
+    confess $@ if $@;
+
+    return;
+}
+
+=head2 TEST CASE FORMAT
+
+The parameters for individual tests are hashes with the following keys:
+
+    {
+        # Name of the test, also displayed on error.
+        description => "Importing a single disk, no existing disks (qcow2)",
+
+        # Definition of the source storage, i.e. the storage from which to import.
+        source => {
+            # The ID of the source storage.
+            storeid => $SOURCE_STOREID,
+
+            # A setup function used to set up the necessary volumes before the test cases are run.
+            # This is used to actually create the file(s) to be imported, for example.
+            'fn-setup-volumes' => sub($storeid) {
+                # [...]
+                return;
+            },
+        },
+
+        # Definition of the target storage, i.e. the storage which disk images are imported to.
+        target => {
+
+            # The ID of the target storage.
+            storeid => $TARGET_STOREID,
+
+            # Like for the source storage, this function can be used to set up any pre-existing
+            # volumes for the storage.
+            'fn-setup-volumes' => sub($storeid) {
+                # [...]
+                return;
+            },
+        },
+
+        # The cases being tested.
+        cases => [
+            {
+                # The name of the volume to import.
+                volname => 'import/some-import.ova/disk.qcow2',
+
+                # The ID of the guest to import the volume for.
+                vmid => 1337,
+
+                # The expected outcome when the invocation succeeds.
+                # May be a string or a regular expression.
+                # If this is a string, then the output has to be an exact match.
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+
+            {
+                volname => 'import/some-import.ova/disk-link.qcow2',
+                vmid => 1337,
+
+                # The expected outcome when the invocation fails.
+                # May be a string or a regular expression.
+                # If this is a string, then the output has to be an exact match.
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+
+=cut
+
+my $tests = [
+    {
+        description => "Importing a single disk, no existing disks (qcow2)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no existing disks (vmdk)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.vmdk';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'vmdk');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.vmdk',
+                vmid => 1338,
+                'expect-match' => "$TARGET_STOREID:1338/vm-1338-disk-0.vmdk",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no existing disks (raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.raw';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'raw');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.raw',
+                vmid => 1339,
+                'expect-match' => "$TARGET_STOREID:1339/vm-1339-disk-0.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid extension",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.txt';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.txt',
+                vmid => 1337,
+                'expect-fail' => qr/unable to parse directory volume name/,
+            },
+        ],
+    },
+    {
+        description => "Importing multiple disks, no existing disks (qcow2, vmdk, raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                {
+                    my $volume_file_name = 'some-qcow2-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.qcow2';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'qcow2');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-vmdk-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.vmdk';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'vmdk');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-raw-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.raw';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'raw');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-0.qcow2",
+            },
+            {
+                volname => 'import/some-vmdk-import.ova/disk.vmdk',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-1.vmdk",
+            },
+            {
+                volname => 'import/some-raw-import.ova/disk.raw',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-2.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing multiple disks, existing disks (qcow2, vmdk, raw)",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                {
+                    my $volume_file_name = 'some-qcow2-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.qcow2';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'qcow2');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-vmdk-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.vmdk';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'vmdk');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                {
+                    my $volume_file_name = 'some-raw-import.ova';
+                    my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                    my $content_file_name = 'disk.raw';
+                    my $content_file_path =
+                        File::Spec->catfile($import_dir, $content_file_name);
+
+                    create_test_image_file($content_file_path, 'raw');
+
+                    create_test_ova_file($volume_file_path, $content_file_name);
+                }
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $vmid = 1337;
+
+                PVE::Storage::vdisk_alloc(
+                    $cfg, $storeid, $vmid, 'qcow2', undef, $DEFAULT_SIZE,
+                );
+
+                PVE::Storage::vdisk_alloc(
+                    $cfg, $storeid, $vmid, 'qcow2', undef, $DEFAULT_SIZE,
+                );
+
+                return;
+            },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-2.qcow2",
+            },
+            {
+                volname => 'import/some-vmdk-import.ova/disk.vmdk',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-3.vmdk",
+            },
+            {
+                volname => 'import/some-raw-import.ova/disk.raw',
+                vmid => 1337,
+                'expect-match' => "$TARGET_STOREID:1337/vm-1337-disk-4.raw",
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid volume name",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-🐧-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/unable to parse directory volume name/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid volume type",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $iso_dir = PVE::Storage::get_iso_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-linux-distro.iso';
+                my $volume_file_path = File::Spec->catfile($iso_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, undef);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'iso/some-linux-distro.iso',
+                vmid => 1337,
+                'expect-fail' => qr/only files with content type 'import' can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, not .ova file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-qcow2-import.ovf';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, undef);
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-qcow2-import.ovf',
+                vmid => 1337,
+                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, no content referenced",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova',
+                vmid => 1337,
+                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, invalid archive format for .ova",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                create_test_image_file($volume_file_path, 'qcow2');
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/error during extraction/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, symlink instead of file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                my $symlink_file_name = 'disk-link.qcow2';
+                my $symlink_file_path = File::Spec->catfile($import_dir, $symlink_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                symlink($content_file_path, $symlink_file_path)
+                    or confess "creating symlink failed";
+
+                create_test_ova_file($volume_file_path, $symlink_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk-link.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, directory instead of file",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                make_path($content_file_path, { verbose => 1, mode => 0750 });
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/extracted file .* not a regular file/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, file is not an image",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                eval { `touch $content_file_path` };
+                confess $@ if $@;
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $TARGET_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1337,
+                'expect-fail' => qr/Could not open .* Image is not in qcow2 format/,
+            },
+        ],
+    },
+    {
+        description => "Importing a single disk, incompatible target storage type",
+        source => {
+            storeid => $SOURCE_STOREID,
+            'fn-setup-volumes' => sub($storeid) {
+                my $cfg = PVE::Storage::config();
+                my $import_dir = PVE::Storage::get_import_dir($cfg, $storeid);
+
+                my $volume_file_name = 'some-import.ova';
+                my $volume_file_path = File::Spec->catfile($import_dir, $volume_file_name);
+
+                my $content_file_name = 'disk.qcow2';
+                my $content_file_path = File::Spec->catfile($import_dir, $content_file_name);
+
+                create_test_image_file($content_file_path, 'qcow2');
+
+                create_test_ova_file($volume_file_path, $content_file_name);
+
+                return;
+            },
+        },
+        target => {
+            storeid => $DUMMY_LVMTHIN_STOREID,
+            'fn-setup-volumes' => sub($storeid) { return; },
+        },
+        cases => [
+            {
+                volname => 'import/some-import.ova/disk.qcow2',
+                vmid => 1339,
+                'expect-fail' => qr/has no path/,
+            },
+        ],
+    },
+];
+
+my sub setup_scfg_dir($scfg) {
+    confess "\$scfg has no 'path' key" if !$scfg->{path};
+
+    make_path($scfg->{path}, { verbose => 1, mode => 0700 });
+
+    my @vtype_subdirs = map { plugin_get_vtype_subdir($scfg, $_) } keys $scfg->{content}->%*;
+
+    make_path(@vtype_subdirs, { verbose => 1, mode => 0755 });
+
+    return;
+}
+
+my sub setup_test_env($test_def) {
+    my ($source, $target) = $test_def->@{qw(source target)};
+
+    my $setup_storage = sub($storage_def) {
+        my $cfg = PVE::Storage::config();
+
+        my $storeid = $storage_def->{storeid};
+        die "\$storeid = undef" if !defined($storeid);
+
+        my $scfg = PVE::Storage::storage_config($cfg, $storeid);
+
+        if ($scfg->{path}) {
+            setup_scfg_dir($scfg);
+        }
+
+        $storage_def->{'fn-setup-volumes'}->($storeid);
+    };
+
+    eval { $setup_storage->($source); };
+    if (my $err = $@) {
+        confess "failed to set up source storage: $err";
+    }
+
+    eval { $setup_storage->($target); };
+    if (my $err = $@) {
+        confess "failed to set up target storage: $err";
+    }
+
+    return;
+}
+
+my sub teardown_scfg_dir($scfg) {
+    confess "\$scfg has no 'path' key" if !$scfg->{path};
+
+    remove_tree($scfg->{path}, { verbose => 0 });
+
+    return;
+}
+
+my sub teardown_test_env($test_def) {
+    my ($source, $target) = $test_def->@{qw(source target)};
+
+    my $teardown_storage = sub($storage_def) {
+        my $cfg = PVE::Storage::config();
+
+        my $storeid = $storage_def->{storeid};
+        die "\$storeid = undef" if !defined($storeid);
+
+        my $scfg = PVE::Storage::storage_config($cfg, $storeid);
+        if ($scfg->{path}) {
+            teardown_scfg_dir($scfg);
+        }
+    };
+
+    eval { $teardown_storage->($target); };
+    if (my $err = $@) {
+        confess "failed to tear down target storage: $err";
+    }
+
+    eval { $teardown_storage->($source); };
+    if (my $err = $@) {
+        confess "failed to tear down source storage: $err";
+    }
+
+    return;
+}
+
+my sub run_test($test_def) {
+    my ($desc, $cases) = $test_def->@{qw(description cases)};
+
+    eval { setup_test_env($test_def); };
+    if (my $err = $@) {
+        teardown_test_env($test_def);
+
+        fail($desc);
+        diag("Failed to set up test environment: $err");
+    }
+
+    my $source_storeid = $test_def->{source}->{storeid};
+    my $target_storeid = $test_def->{target}->{storeid};
+
+    for my $case ($cases->@*) {
+        my ($volname, $vmid) = $case->@{qw(volname vmid)};
+
+        confess "($desc) case cannot have both 'expect-match' and 'expect-fail'"
+            if defined($case->{'expect-match'}) && defined($case->{'expect-fail'});
+
+        my $source_volid = $source_storeid . ':' . $volname;
+
+        my $case_desc = "$desc | $vmid | $volname";
+
+        my $result = eval {
+            return PVE::GuestImport::extract_disk_from_import_file(
+                $source_volid, $vmid, $target_storeid,
+            );
+        };
+        if (my $err = $@) {
+            my $re_expect_fail = $case->{'expect-fail'};
+            $re_expect_fail = qr/^\Q$re_expect_fail\E$/ if !is_regexp($re_expect_fail);
+
+            if (!ok($err =~ $re_expect_fail, $case_desc)) {
+                diag(explain($err));
+                diag("does not match");
+                diag(explain($case->{'expect-fail'}));
+            }
+
+            next;
+        }
+
+        my $re_expect_match = $case->{'expect-match'};
+        $re_expect_match = qr/^\Q$re_expect_match\E$/ if !is_regexp($re_expect_match);
+
+        if (!ok($result =~ $re_expect_match, $case_desc)) {
+            diag(explain($result));
+            diag("does not match");
+            diag(explain($case->{'expect-match'}));
+        }
+    }
+
+    teardown_test_env($test_def);
+
+    return;
+}
+
+my sub main() {
+    my $sum_cases = 0;
+    for my $test_def ($tests->@*) {
+        $sum_cases += scalar($test_def->{cases}->@*);
+    }
+
+    plan tests => $sum_cases;
+
+    for my $test_def ($tests->@*) {
+        run_test($test_def);
+    }
+
+    done_testing();
+
+    return;
+}
+
+main();
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 27c45b8c..7d3e2ce0 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -20,6 +20,7 @@ my $res = $harness->runtests(
     "archive_info_test.pm",
     "filesystem_path_test.pm",
     "get_subdir_test.pm",
+    "guest_import_test.pl",
     "list_volumes_test.pm",
     "parse_volname_test.pm",
     "path_to_volume_id_test.pm",
-- 
2.47.3

[PATCH pve-storage v2 22/50] tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0

[Max R. Carrara]

Introduce the `PVE::Storage::Common::Parse` module with the following
subroutines:

* `parse_rel_path_as_volname_parts($path, $vtype)`

  Parses a given file path into several smaller parts that constitute
  a volume name for the given volume type.

* `parse_rel_path_as_volname($path, $vtype)`

  Parses a given file path directly into a volume name.

* `parse_volname_as_parts($volname)`

  Parses an existing volume name into its constituent parts.

* `parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype)`

  Parses a given file path into several smaller parts that constitute
  a volume ID for the given storage ID, its configuration and volume
  type.

* `parse_abs_path_as_volid($storeid, $scfg, $path, $vtype)`

  Like `parse_abs_path_as_volid_parts()`, but parses the given path
  directly into a volume ID.

* `parse_volid_as_parts($volid)`

  Parses an existing volume ID into its storage ID and volume name
  parts.

As of this commit, these parsing functions only support the 'iso'
volume type, with the exception of `parse_volid_as_parts()`, which
does not depend on knowing the volume type.

Using the newly introduced parsing helpers, replace all occurrences of
the `PVE::Storage::ISO_EXT_RE_0` regex across the repository.

Since the `ISO_EXT_RE_0` regex is now completely unused, note its
removal in `ApiChangeLog` and add a FIXME to note that it should be
removed on the next APIAGE reset.

Additional Notes Regarding the new Parsers
==========================================

Support for other volume types will be added individually in future
commits.

The new *private* regex used for 'iso' vtype parsing is still matching
file extensions case-insensitively, but also matches the entire path
and file name portions of 'iso' file paths and volume names.

These parts are extracted using named regex groups, as that is much
easier to handle and keep track of mentally, even with smaller
regexes. These named groups are the "constituent parts" that are
returned by some of the new parser subroutines.

However, one important difference here is that named regex groups do
not support dashes `-` in their names, only underscores `_`. To keep
things consistent with our style (using dashes instead of underscores
in hash keys and the API), these named groups are formatted before
being returned—underscores are simply substituted with dashes.

Finally, all parser subroutines check whether a parent directory
reference (`..` or "double dots") is contained in the passed or
extracted file path, and return early if there is. This is an
additional safety measure that is intentionally introduced in this
commit to guard against any mishaps in the future as the parsers gain
more functionality, such as supporting nested directories inside
different volume types' subdirectories.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                    |   9 +
 src/PVE/API2/Storage/Status.pm  |   9 +-
 src/PVE/Storage.pm              |   8 +-
 src/PVE/Storage/Common.pm       |   2 +
 src/PVE/Storage/Common/Makefile |   1 +
 src/PVE/Storage/Common/Parse.pm | 389 ++++++++++++++++++++++++++++++++
 src/PVE/Storage/Plugin.pm       |  19 +-
 7 files changed, 427 insertions(+), 10 deletions(-)
 create mode 100644 src/PVE/Storage/Common/Parse.pm

diff --git a/ApiChangeLog b/ApiChangeLog
index aefc0ff8..7f79efb5 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -28,6 +28,15 @@ Future changes should be documented in here.
   In order to make it clear that this is just a helper, it is replaced with the
   more explicit `PVE::Storage::Common::plugin_get_vtype_subdir()` subroutine.
 
+* Remove the following regular expressions:
+  * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
+
+  These regular expressions were mostly used for parsing file paths and volume
+  names corresponding to their volume types, noted in parentheses above.
+
+  Instead, The parsing functions in `PVE::Storage::Common::Parse` should be
+  used.
+
 ## Version 15:
 
 * Add new `$snapname` parameter to the `volume_resize()` plugin method
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 45a79d56..310677d6 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -23,6 +23,9 @@ use PVE::Storage;
 use PVE::Storage::Common qw(
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_rel_path_as_volname_parts
+);
 
 use base qw(PVE::RESTHandler);
 
@@ -73,11 +76,13 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     my ($path, $vtype) = @_;
 
     if ($vtype eq 'iso') {
-        if ($path !~ m![^/]+$PVE::Storage::ISO_EXT_RE_0$!) {
+        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "wrong file extension" });
         }
 
-        my $ext = $1;
+        my $ext = $parts->{ext};
         return $ext;
     }
 
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 485bd822..5358be3f 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -26,6 +26,9 @@ use PVE::Storage::Common qw(
     plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_abs_path_as_volid
+);
 use PVE::RESTEnvironment qw(log_warn);
 
 use PVE::Storage::Plugin;
@@ -117,6 +120,7 @@ PVE::Storage::Plugin->init();
 # the following REs indicate the number or capture groups via the trailing digit
 # CAUTION don't forget to update the digits accordingly after messing with the capture groups
 
+# FIXME: remove this regex on the next APIAGE reset.
 our $ISO_EXT_RE_0 = qr/\.(?:iso|img)/i;
 
 our $VZTMPL_EXT_RE_1 = qr/\.(?|(tar)(?!\.)|tar\.(gz|xz|zst|bz2))/i;
@@ -754,9 +758,7 @@ sub path_to_volume_id {
         }
 
         if ($vtype eq 'iso') {
-            return if $filename !~ m!/([^/]+$ISO_EXT_RE_0)$!;
-            my $name = $1;
-            return "$sid:iso/$name";
+            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
         }
 
         if ($vtype eq 'vztmpl') {
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 4b200d9d..7efa279d 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -56,6 +56,8 @@ be grouped in a submodule can also be found here.
 
 =over
 
+=item * C<L<PVE::Storage::Common::Parse>>
+
 =back
 
 =head1 STANDARD OPTIONS FOR JSON SCHEMA
diff --git a/src/PVE/Storage/Common/Makefile b/src/PVE/Storage/Common/Makefile
index 0c4bba5b..0d9b1be1 100644
--- a/src/PVE/Storage/Common/Makefile
+++ b/src/PVE/Storage/Common/Makefile
@@ -1,4 +1,5 @@
 SOURCES = \
+	  Parse.pm			\
 
 
 .PHONY: install
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
new file mode 100644
index 00000000..1141a715
--- /dev/null
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -0,0 +1,389 @@
+package PVE::Storage::Common::Parse;
+
+use v5.36;
+
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
+
+use Exporter qw(import);
+
+our @EXPORT_OK = qw(
+    parse_rel_path_as_volname_parts
+    parse_rel_path_as_volname
+    parse_volname_as_parts
+
+    parse_abs_path_as_volid_parts
+    parse_abs_path_as_volid
+    parse_volid_as_parts
+);
+
+=head1 NAME
+
+C<PVE::Storage::Common::Parse> - Storage-related Parsing Functions
+
+=head1 DESCRIPTION
+
+This module contains various parsing functions for use within C<L<PVE::Storage>>,
+its submodules (including storage plugins) and other related modules.
+
+Parsing functions are categorized by their main purpose / area of application
+and may be further subdivided depending on what kind of data type they are
+primarily related to.
+
+=cut
+
+my $RE_PARENT_DIR = quotemeta('..');
+my $RE_CONTAINS_PARENT_DIR = qr!
+    ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
+    |
+    ( /$RE_PARENT_DIR/ )  # /../ --> Between two path components
+    |
+    ( /$RE_PARENT_DIR$ )  # /..  --> End of path
+!xn;
+
+my $RE_ISO_FILE_PATH = qr!
+    (?<path>
+        (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
+    )
+!xn;
+
+my $RE_FILE_PATH_FOR_VTYPE = {
+    iso => qr/^$RE_ISO_FILE_PATH$/,
+};
+
+my $RE_VOLNAME_FOR_VTYPE = {
+    iso => qr/^$RE_ISO_FILE_PATH$/,
+};
+
+my sub contains_parent_dir($path) {
+    return $path =~ $RE_CONTAINS_PARENT_DIR;
+}
+
+my sub strip_leading_path_separators($path) {
+    return $path =~ s!^/+!!r;
+}
+
+my sub strip_trailing_path_separators($path) {
+    return $path =~ s!/+$!!r;
+}
+
+my sub format_named_groups(%groups) {
+    my $result = {};
+
+    for my $old_key (keys %groups) {
+        my $new_key = $old_key =~ s/_/-/gr;
+        $result->{$new_key} = $groups{$old_key};
+    }
+
+    my @disk_path_components = ();
+
+    if (defined($result->{file})) {
+        $result->{file} = strip_leading_path_separators($result->{file});
+        push(@disk_path_components, $result->{file});
+    }
+
+    if (scalar(@disk_path_components)) {
+        $result->{'disk-path'} = join('/', @disk_path_components);
+    }
+
+    return $result;
+}
+
+my sub split_leading_dir_from_path($path, $directory) {
+    $directory = strip_trailing_path_separators($directory);
+
+    if ("$directory/" eq substr($path, 0, length($directory) + 1, '')) {
+        return ($directory, $path);
+    }
+
+    return;
+}
+
+=head1 PARSERS RELATED TO VOLUMES
+
+The parsing functions in this section primarily deal with parsing data related
+to storage volumes, primarily C<L<volname>>s and C<L<volid>>s.
+
+=head2 VOLUME NAMES
+
+All subroutines ending in C<_parts> in this section have a set of common
+"parts" that they return.
+
+These are:
+
+    {
+        # The name of the file the volume points to
+        file => 'custom-debian.iso',
+
+        # The extension (suffix) of the file the volume points to
+        ext => 'iso',
+
+        # The "real" path on disk
+        'disk-path' => 'custom-debian.iso',
+
+        # The "whole" path that the volume references,
+        path => 'custom-debian.iso',
+
+        # The volume's type
+        vtype => 'iso',
+
+        # The volname, which is "${vtype}/${path}"
+        volname => 'iso/custom-debian.iso',
+    }
+
+=cut
+
+=head3 parse_rel_path_as_volname_parts
+
+Parses the given relative file path C<$path> according to the given C<$vtype>,
+returning a hashref containing the parts that make up a C<volname> on success.
+
+For example, C<$path = "custom-debian.iso"> and C<$vtype = "iso"> are turned
+into:
+
+    {
+        file => 'custom-debian.iso',
+        ext => 'iso',
+        'disk-path' => 'custom-debian.iso',
+        path => 'custom-debian.iso',
+        vtype => 'iso',
+        volname => 'iso/custom-debian.iso',
+    }
+
+See L<VOLUME NAMES> for more details.
+
+On failure or when the provided C<$vtype> is not supported or does not exist,
+returns C<undef> in scalar context, and an empty list in list context.
+
+B<NOTE:> This function assumes that C<$path> is already relative to the
+directory that corresponds to the given C<$vtype>.
+
+If you want to parse an absolute path for an already known storage instead, see
+C<L<< parse_abs_path_as_volid_parts()|/"parse_abs_path_as_volid_parts" >>>.
+
+For a counterpart to this function, see
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>>.
+
+=cut
+
+sub parse_rel_path_as_volname_parts : prototype($$) ($path, $vtype) {
+    return if contains_parent_dir($path);
+
+    # TODO: vtype split: Handle parsing for 'images' and 'rootdir' vtypes.
+
+    my $re_filepath = $RE_FILE_PATH_FOR_VTYPE->{$vtype};
+    return if !defined($re_filepath);
+
+    return if $path !~ $re_filepath;
+
+    my $parts = format_named_groups(%+);
+    $parts->{vtype} = $vtype;
+    $parts->{volname} = $vtype . '/' . $parts->{path};
+
+    return $parts;
+}
+
+=head3 parse_rel_path_as_volname
+
+Like C<L<< parse_rel_path_as_volname_parts()|/"parse_rel_path_as_volname_parts" >>>, but
+instead of extracting the individual parts of the relative C<$path>, returns
+the correctly formatted C<volname> directly.
+
+For example, C<$path = "custom-debian.iso"> and C<$vtype = "iso"> are turned
+into C<"iso/custom-debian.iso">.
+
+For a counterpart to this function, see
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>>.
+
+=cut
+
+sub parse_rel_path_as_volname : prototype($$) ($path, $vtype) {
+    return if contains_parent_dir($path);
+
+    # TODO: vtype split: Handle parsing for 'images' and 'rootdir' vtypes.
+
+    my $re_filepath = $RE_FILE_PATH_FOR_VTYPE->{$vtype};
+    return if !defined($re_filepath);
+
+    return if $path !~ $re_filepath;
+
+    return $vtype . '/' . $+{path};
+}
+
+=head3 parse_volname_as_parts
+
+Parses the provided C<$volname> and returns its constituent parts in a hashref
+upon success.
+
+For example, C<$volname = "iso/custom-debian.iso"> is turned into:
+
+    {
+        file => 'custom-debian.iso',
+        ext => 'iso',
+        'disk-path' => 'custom-debian.iso',
+        path => 'custom-debian.iso',
+        vtype => 'iso',
+        volname => 'iso/custom-debian.iso',
+    }
+
+Returns C<undef> in scalar context and an empty list in list context, if
+C<$volname> is prefixed with an unknown or unsupported C<vtype>, or if the path
+after the C<vtype> prefix cannot be parsed.
+
+This function can be seen as a counterpart to
+C<L<< parse_rel_path_as_volname_parts()|/"parse_rel_path_as_volname_parts" >>> and
+C<L<< parse_rel_path_as_volname()|/"parse_rel_path_as_volname" >>> and can be used to
+extract the information embedded within an already existing volume name, such
+as file extensions or the name of the file that a volume refers to.
+
+=cut
+
+sub parse_volname_as_parts : prototype($) ($volname) {
+    # TODO: vtype split: Handle volname for 'images' and 'rootdir' vtypes.
+    my ($vtype, $path) = split('/', $volname, 2);
+
+    # Either variable could be undef or an empty string here
+    return if !$vtype || !$path;
+
+    return if contains_parent_dir($path);
+
+    my $re_volname = $RE_VOLNAME_FOR_VTYPE->{$vtype};
+    return if !defined($re_volname);
+
+    return if $path !~ $re_volname;
+
+    my $parts = format_named_groups(%+);
+    $parts->{vtype} = $vtype;
+    $parts->{volname} = $volname;
+
+    return $parts;
+}
+
+=head2 VOLUME IDS
+
+All subroutines ending in C<_parts> in this section have a set of common
+"parts" that they return. Most of there parts are shared with the common parts
+described in L<VOLUME NAMES>, with the exception of C<volid>.
+
+These are:
+
+    {
+        # See VOLUME NAMES
+        file => 'custom-debian.iso',
+        ext => 'iso',
+        'disk-path' => 'custom-debian.iso',
+        path => 'custom-debian.iso',
+        vtype => 'iso',
+        volname => 'iso/custom-debian.iso',
+
+        # Exclusive to volid parsing
+        volid => 'local:iso/custom-debian.iso',
+    }
+
+=cut
+
+my $RE_VOLID = qr!
+    ^
+    (?<volid>
+        (?<storeid> (?i: [a-z][a-z0-9\-\_\.]*[a-z0-9] ) )
+        :    # separated by colon
+        (?<volname> .+)
+    )
+    $
+!xn;
+
+=head3 parse_abs_path_as_volid_parts
+
+Parses the given absolute file path C<$path> according to the given C<$vtype>
+and returns a hashref containing the parts that make up a C<$volid> on success.
+C<$path> must exist for the passed C<$storeid> and its config C<$scfg>.
+
+For example, C<$path = "/var/lib/vz/template/iso/custom-debian.iso"> and
+C<$vtype = "iso"> on the C<local> directory storage are turned into:
+
+    {
+        # Identical to volume name parsing:
+        file => 'custom-debian.iso',
+        ext => 'iso',
+        'disk-path' => 'custom-debian.iso',
+        path => 'custom-debian.iso',
+        vtype => 'iso',
+        volname => 'iso/custom-debian.iso',
+
+        # Specific to volume ID parsers:
+        # "${storeid}:${volname}"
+        volid => 'local:iso/custom-debian.iso',
+    }
+
+See L<VOLUME IDS> for more details.
+
+On failure or when the provided C<$vtype> is not supported or does not exist,
+returns C<undef> in scalar context, and an empty list in list context.
+
+If you are only interested in parsing the portion of the path belonging inside
+the C<vtype> directory independent of a C<$storeid>, see
+C<L<< parse_rel_path_as_volname_parts()|/"parse_rel_path_as_volname_parts" >>>.
+
+For a counterpart to this function, see
+C<L<< parse_volid_as_parts()|/"parse_volid_as_parts" >>>.
+
+=cut
+
+sub parse_abs_path_as_volid_parts : prototype($$$$) ($storeid, $scfg, $path, $vtype) {
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    my ($leading_dir, $remainder) = split_leading_dir_from_path($path, $vtype_subdir);
+    return if !defined($leading_dir);
+
+    my $volid_parts = parse_rel_path_as_volname_parts($remainder, $vtype);
+    return if !defined($volid_parts);
+
+    $volid_parts->{volid} = $storeid . ':' . $volid_parts->{volname};
+
+    return $volid_parts;
+}
+
+=head3 parse_abs_path_as_volid
+
+Like C<L<< parse_abs_path_as_volid_parts()|/"parse_abs_path_as_volid_parts" >>>, but
+instead of extracting the individual parts of the C<$path>, returns the
+correctly formatted C<volid> directly.
+
+For a counterpart to this function, see
+C<L<< parse_volid_as_parts()|/"parse_volid_as_parts" >>>.
+
+=cut
+
+sub parse_abs_path_as_volid : prototype($$$$) ($storeid, $scfg, $path, $vtype) {
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    my ($leading_dir, $remainder) = split_leading_dir_from_path($path, $vtype_subdir);
+    return if !defined($leading_dir);
+
+    my $volname = parse_rel_path_as_volname($remainder, $vtype);
+    return if !defined($volname);
+
+    return $storeid . ':' . $volname;
+}
+
+=head3 parse_volid_as_parts
+
+Parses the provided C<$volid> and returns its C<storeid> and C<volname> parts
+in a hashref.
+
+On failure, returns C<undef> in scalar context, and an empty list in list context.
+
+The C<volname> part of volumes that represent files can be further parsed into
+its constituent parts using the
+C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>> function.
+
+=cut
+
+sub parse_volid_as_parts : prototype($) ($volid) {
+    return if $volid !~ $RE_VOLID;
+
+    return format_named_groups(%+);
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 6a113dcb..d4860344 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -19,6 +19,10 @@ use PVE::Storage::Common qw(
     plugin_get_default_vtype_subdirs
     plugin_get_vtype_subdir
 );
+use PVE::Storage::Common::Parse qw(
+    parse_volname_as_parts
+    parse_abs_path_as_volid_parts
+);
 
 use JSON;
 
@@ -821,8 +825,12 @@ sub parse_volname {
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
     }
 
-    if ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
-        return ('iso', $1, undef, undef, undef, undef, 'raw');
+    if (defined(my $parts = parse_volname_as_parts($volname))) {
+        my ($vtype, $volume_path) = $parts->@{qw(vtype path)};
+
+        if ($vtype eq 'iso') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }
 
     if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
@@ -1716,11 +1724,12 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'iso') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
+            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
 
             return {
-                volid => "$storeid:iso/$1",
-                format => 'iso',
+                volid => $parts->{volid},
+                format => 'iso', # always 'iso' even if we have a file ending in .img
             };
         }
 
-- 
2.47.3

[PATCH pve-storage v2 23/50] common: test: set up parser testing code, add tests for 'iso' vtype

[Max R. Carrara]

Set up the necessary scaffolding for testing the new parser functions
in `PVE::Storage::Common::Parse` and add tests for the 'iso' vtype.

This also adds `libtest-differences-perl` to `Build-Depends`.

Since the volid parsing functions are built on top of the volname
parsing functions, their tests are built in a similar way; that is,
the volname parser test cases are taken and adapted for the volid
parsers.

This is primarily done to ensure consistency between the two
"families" of parsing functions; for example, the
`parse_path_as_volid_parts()` function should always return the same
hash as the `parse_path_as_volname_parts()` function, except that it
also contains a 'volid' key.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 debian/control                              |   1 +
 src/PVE/Makefile                            |   1 +
 src/PVE/Storage/Common/Makefile             |   4 +
 src/PVE/Storage/Common/test/Makefile        |   6 +
 src/PVE/Storage/Common/test/parser_tests.pl | 301 ++++++++++++++++++++
 src/PVE/Storage/Common/test/run_tests.pl    |  25 ++
 src/PVE/Storage/Makefile                    |   4 +
 7 files changed, 342 insertions(+)
 create mode 100644 src/PVE/Storage/Common/test/Makefile
 create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
 create mode 100755 src/PVE/Storage/Common/test/run_tests.pl

diff --git a/debian/control b/debian/control
index 3ad25e7a..9b2b4f97 100644
--- a/debian/control
+++ b/debian/control
@@ -10,6 +10,7 @@ Build-Depends: debhelper-compat (= 13),
                libpve-common-perl (>= 9.1.12),
                librados2-perl,
                libtest-mockmodule-perl,
+               libtest-differences-perl,
                libxml-libxml-perl,
                lintian,
                perl,
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 9e9f6aac..b8fc6325 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -14,6 +14,7 @@ install:
 
 .PHONY: test
 test:
+	$(MAKE) -C Storage test
 	$(MAKE) -C test test
 
 clean:
diff --git a/src/PVE/Storage/Common/Makefile b/src/PVE/Storage/Common/Makefile
index 0d9b1be1..7a13371d 100644
--- a/src/PVE/Storage/Common/Makefile
+++ b/src/PVE/Storage/Common/Makefile
@@ -5,3 +5,7 @@ SOURCES = \
 .PHONY: install
 install:
 	for i in ${SOURCES}; do install -D -m 0644 $$i ${DESTDIR}${PERLDIR}/PVE/Storage/Common/$$i; done
+
+.PHONY: test
+test:
+	make -C test test
diff --git a/src/PVE/Storage/Common/test/Makefile b/src/PVE/Storage/Common/test/Makefile
new file mode 100644
index 00000000..c4b2c0db
--- /dev/null
+++ b/src/PVE/Storage/Common/test/Makefile
@@ -0,0 +1,6 @@
+all: test
+
+.PHONY: test
+test: run_tests.pl
+	./run_tests.pl
+
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
new file mode 100755
index 00000000..c19456de
--- /dev/null
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -0,0 +1,301 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use lib q(../../../..);
+
+use File::Temp;
+use Storable qw(dclone);
+
+use PVE::Storage::Common qw(
+    plugin_get_vtype_subdir
+);
+use PVE::Storage::Common::Parse qw(
+    parse_rel_path_as_volname_parts
+    parse_rel_path_as_volname
+    parse_volname_as_parts
+
+    parse_abs_path_as_volid_parts
+    parse_abs_path_as_volid
+    parse_volid_as_parts
+);
+
+use Test::More qw(no_plan);
+use Test::Differences;
+
+my $volname_tests = [];
+
+my $volname_cases_iso_valid = [
+    # plain paths
+    {
+        path => 'custom-debian.iso',
+        expected => {
+            file => 'custom-debian.iso',
+            ext => 'iso',
+            'disk-path' => 'custom-debian.iso',
+            path => 'custom-debian.iso',
+            vtype => 'iso',
+            volname => 'iso/custom-debian.iso',
+        },
+    },
+    {
+        path => 'archlinux.img',
+        expected => {
+            file => 'archlinux.img',
+            ext => 'img',
+            'disk-path' => 'archlinux.img',
+            path => 'archlinux.img',
+            vtype => 'iso',
+            volname => 'iso/archlinux.img',
+        },
+    },
+
+    # case insensitive file extensions
+    {
+        path => 'Debian 13 Trixie.ISO',
+        expected => {
+            file => 'Debian 13 Trixie.ISO',
+            ext => 'ISO',
+            'disk-path' => 'Debian 13 Trixie.ISO',
+            path => 'Debian 13 Trixie.ISO',
+            vtype => 'iso',
+            volname => 'iso/Debian 13 Trixie.ISO',
+        },
+    },
+    {
+        path => 'Fedora.Img',
+        expected => {
+            file => 'Fedora.Img',
+            ext => 'Img',
+            'disk-path' => 'Fedora.Img',
+            path => 'Fedora.Img',
+            vtype => 'iso',
+            volname => 'iso/Fedora.Img',
+        },
+    },
+];
+
+my $volname_cases_iso_invalid = [
+    {
+        description => "Invalid file extension (iso)",
+        args => {
+            path => 'MacOS-kittycat.dmg',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+];
+
+my $cases_valid_all = [
+    $volname_cases_iso_valid,
+];
+
+my $cases_invalid_all = [
+    $volname_cases_iso_invalid,
+];
+
+{
+    for my $cases_list ($cases_valid_all->@*) {
+        for my $case ($cases_list->@*) {
+            my $path = $case->{path};
+            my $vtype = $case->{expected}->{vtype};
+
+            push(
+                $volname_tests->@*,
+                {
+                    description => "Valid path ($vtype) \"$path\"",
+                    args => {
+                        path => $path,
+                        vtype => $vtype,
+                    },
+                    expected => $case->{expected},
+                },
+            );
+        }
+    }
+
+    for my $cases_list ($cases_invalid_all->@*) {
+        push($volname_tests->@*, $cases_list->@*);
+    }
+}
+
+my sub run_volname_parsing_tests : prototype($) ($volname_tests) {
+    for my $case ($volname_tests->@*) {
+        subtest $case->{description} => sub() {
+            my ($path, $vtype) = $case->{args}->@{qw(path vtype)};
+
+            my $got_volname_parts = parse_rel_path_as_volname_parts($path, $vtype);
+            my $got_volname = parse_rel_path_as_volname($path, $vtype);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname_parts,
+                    $case->{expected},
+                    'parse_rel_path_as_volname_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+
+                is(
+                    $got_volname,
+                    $case->{expected}->{volname},
+                    'parse_rel_path_as_volname() returns expected volname',
+                );
+
+                my $got_volname_parts_from_volname = parse_volname_as_parts($got_volname);
+
+                eq_or_diff(
+                    $got_volname_parts_from_volname,
+                    $case->{expected},
+                    'parse_volname_as_parts() returns expected hashref from parsed volname',
+                );
+            } else {
+                is($got_volname_parts, undef,
+                    'parse_rel_path_as_volname_parts() returns undef');
+                is($got_volname, undef, 'parse_rel_path_as_volname() returns undef');
+            }
+
+        };
+    }
+
+    return;
+}
+
+my $DEFAULT_STOREID = 'local';
+my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
+my $DEFAULT_SCFG = {
+    type => 'dir',
+    path => $DEFAULT_STORAGE_PATH,
+    shared => 0,
+    content => {
+        iso => 1,
+        rootdir => 1,
+        vztmpl => 1,
+        images => 1,
+        snippets => 1,
+        backup => 1,
+        import => 1,
+    },
+};
+
+my $ALT_STOREID = 'dir-store';
+my $ALT_STORAGE_PATH = File::Temp->newdir();
+my $ALT_SCFG = {
+    type => 'dir',
+    path => $ALT_STORAGE_PATH,
+    shared => 0,
+    content => { $DEFAULT_SCFG->{content}->%* },
+    'content-dirs' => {
+        iso => 'pve/volumes/iso',
+        vztmpl => 'pve/dumps/vz',
+        images => 'pve/guest/images',
+        snippets => 'pve/misc/snippets',
+        backup => 'pve/dumps/bak',
+        import => 'archive/import',
+    },
+};
+
+my sub format_volname_case_to_volid_case($storeid, $scfg, $volname_case) {
+    my $volid_case = dclone($volname_case);
+
+    my ($path, $vtype) = $volid_case->{args}->@{qw(path vtype)};
+
+    my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+    $volid_case->{args}->{path} = $vtype_subdir . '/' . $path;
+    $volid_case->{args}->{storeid} = $storeid;
+    $volid_case->{args}->{scfg} = $scfg;
+
+    if (defined($volid_case->{expected})) {
+        my $volname = $volid_case->{expected}->{volname};
+        $volid_case->{expected}->{volid} = $storeid . ':' . $volname;
+    }
+
+    return $volid_case;
+}
+
+my $volid_tests = [
+    {
+        description => "volid parsers build on volname parsers' behaviors (1)",
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $volname_tests->@*
+        ],
+    },
+    {
+        description => "volid parsers build on volname parsers' behaviors (2)",
+        storeid => $ALT_STOREID,
+        scfg => $ALT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($ALT_STOREID, $ALT_SCFG, $_) }
+                $volname_tests->@*
+        ],
+    },
+];
+
+my sub run_volid_parsing_tests : prototype($) ($volid_tests) {
+    for my $test ($volid_tests->@*) {
+        subtest $test->{description} => sub() {
+            for my $case ($test->{cases}->@*) {
+                my ($storeid, $scfg, $path, $vtype) =
+                    $case->{args}->@{qw(storeid scfg path vtype)};
+
+                my $got_volid_parts =
+                    parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+                my $got_volid = parse_abs_path_as_volid($storeid, $scfg, $path, $vtype);
+
+                note("Running test case based on:");
+                note($case->{description});
+
+                if (defined($case->{expected})) {
+                    eq_or_diff(
+                        $got_volid_parts,
+                        $case->{expected},
+                        'parse_abs_path_as_volid_parts() returns expected hashref',
+                        { context => 50000 },
+                    );
+
+                    is(
+                        $got_volid,
+                        $case->{expected}->{volid},
+                        'parse_abs_path_as_volid() returns expected volid',
+                    );
+
+                    my $expected_volid_parts = {
+                        volid => $case->{expected}->{volid},
+                        storeid => $case->{args}->{storeid},
+                        volname => $case->{expected}->{volname},
+                    };
+
+                    my $got_volid_parts_from_volid = parse_volid_as_parts($got_volid);
+
+                    eq_or_diff(
+                        $got_volid_parts_from_volid,
+                        $expected_volid_parts,
+                        'parse_volid_as_parts() returns expected hashref from parsed volid',
+                    );
+                } else {
+                    is($got_volid_parts, undef,
+                        'parse_abs_path_as_volid_parts() returns undef');
+                    is($got_volid, undef, 'parse_abs_path_as_volid() returns undef');
+                }
+            }
+        };
+    }
+
+    return;
+}
+
+my sub main() {
+    unified_diff();
+
+    run_volname_parsing_tests($volname_tests);
+    run_volid_parsing_tests($volid_tests);
+
+    done_testing();
+
+    return;
+}
+
+main();
diff --git a/src/PVE/Storage/Common/test/run_tests.pl b/src/PVE/Storage/Common/test/run_tests.pl
new file mode 100755
index 00000000..ed8b1164
--- /dev/null
+++ b/src/PVE/Storage/Common/test/run_tests.pl
@@ -0,0 +1,25 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use TAP::Harness;
+
+my $MAX_JOBS = 4;
+
+my $nproc = eval { int(`nproc`) } || $MAX_JOBS;
+
+my $jobs = $nproc > $MAX_JOBS ? $MAX_JOBS : $nproc;
+
+my sub main() {
+    my $harness = TAP::Harness->new({ verbosity => -1, jobs => $jobs });
+
+    my $res = $harness->runtests(
+        "parser_tests.pl",
+    );
+
+    exit -1 if !$res || $res->{failed} || $res->{parse_errors};
+
+    return;
+}
+
+main();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a67dc25f..1c8d364b 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -21,3 +21,7 @@ install:
 	make -C Common install
 	for i in ${SOURCES}; do install -D -m 0644 $$i ${DESTDIR}${PERLDIR}/PVE/Storage/$$i; done
 	make -C LunCmd install
+
+.PHONY: test
+test:
+	make -C Common test
-- 
2.47.3

[PATCH pve-storage v2 24/50] tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions

[Max R. Carrara]

Add support for the 'vztmpl' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace usages of the
`PVE::Storage::VZTMPL_EXT_RE_1` regex across the repository.

As done with the `ISO_EXT_RE_0` regex, note the removal of the
`VZTMPL_EXT_RE_1` regex in `ApiChangeLog`, since it's now completely
unused. Add a FIXME to note that it should be removed on the next
APIAGE reset.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                                |   1 +
 src/PVE/API2/Storage/Status.pm              |   6 +-
 src/PVE/Storage.pm                          |   5 +-
 src/PVE/Storage/Common/Parse.pm             |  54 +++++++++-
 src/PVE/Storage/Common/test/parser_tests.pl | 109 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  16 +--
 6 files changed, 179 insertions(+), 12 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index 7f79efb5..ffe3725c 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -30,6 +30,7 @@ Future changes should be documented in here.
 
 * Remove the following regular expressions:
   * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
+  * `$PVE::Storage::VZTMPL_EXT_RE_1` (`vztmpl` volume type)
 
   These regular expressions were mostly used for parsing file paths and volume
   names corresponding to their volume types, noted in parentheses above.
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 310677d6..60f08628 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -87,11 +87,13 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     }
 
     if ($vtype eq 'vztmpl') {
-        if ($path !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
+        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "wrong file extension" });
         }
 
-        my $ext = $1;
+        my $ext = $parts->{ext};
         return $ext;
     }
 
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 5358be3f..afe29178 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -123,6 +123,7 @@ PVE::Storage::Plugin->init();
 # FIXME: remove this regex on the next APIAGE reset.
 our $ISO_EXT_RE_0 = qr/\.(?:iso|img)/i;
 
+# FIXME: remove this regex on the next APIAGE reset.
 our $VZTMPL_EXT_RE_1 = qr/\.(?|(tar)(?!\.)|tar\.(gz|xz|zst|bz2))/i;
 
 our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
@@ -762,9 +763,7 @@ sub path_to_volume_id {
         }
 
         if ($vtype eq 'vztmpl') {
-            return if $filename !~ m!/([^/]+$VZTMPL_EXT_RE_1)$!;
-            my $name = $1;
-            return "$sid:vztmpl/$name";
+            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
         }
 
         if ($vtype eq 'backup') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 1141a715..207f287b 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -33,6 +33,14 @@ primarily related to.
 
 =cut
 
+my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');
+
+my sub join_to_re_alternations(@list) {
+    return join('|', map { quotemeta } @list);
+}
+
+my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSION_EXTENSIONS);
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -48,12 +56,27 @@ my $RE_ISO_FILE_PATH = qr!
     )
 !xn;
 
+my $RE_VZTMPL_FILE_PATH = qr!
+    (?<path>
+        (?<file>
+            [^/]+
+            \.
+            (?<ext>
+                (?<ext_archive> (?i: tar) )
+                ( \. (?<ext_compression> (?i: $RE_VZTMPL_COMPRESSION_EXTENSIONS)) )?
+            )
+        )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
+    vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
 };
 
 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
+    vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
 };
 
 my sub contains_parent_dir($path) {
@@ -132,6 +155,31 @@ These are:
         volname => 'iso/custom-debian.iso',
     }
 
+The following volume types have more specific parts in addition to the common
+ones:
+
+=over
+
+=item C<vztmpl>
+
+C<ext-archive>, C<ext-compression> (optional):
+
+    {
+        file => 'archlinux-base_20190924-1_amd64.tar.gz',
+        ext => 'tar.gz',
+
+        # Used to further differentiate 'ext'
+        'ext-archive' => 'tar',
+        'ext-compression' => 'gz',
+
+        'disk-path' => 'archlinux-base_20190924-1_amd64.tar.gz',
+        path => 'archlinux-base_20190924-1_amd64.tar.gz',
+        vtype => 'vztmpl',
+        volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
+    }
+
+=back
+
 =cut
 
 =head3 parse_rel_path_as_volname_parts
@@ -151,7 +199,7 @@ into:
         volname => 'iso/custom-debian.iso',
     }
 
-See L<VOLUME NAMES> for more details.
+See L<VOLUME NAMES> for more details and special cases.
 
 On failure or when the provided C<$vtype> is not supported or does not exist,
 returns C<undef> in scalar context, and an empty list in list context.
@@ -281,6 +329,10 @@ These are:
         volid => 'local:iso/custom-debian.iso',
     }
 
+The special cases described in L<VOLUME NAMES> also apply for the parser
+functions in this section. In fact, the volid parsing functions here build on
+the volname ones.
+
 =cut
 
 my $RE_VOLID = qr!
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index c19456de..1ea8df1a 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -86,12 +86,121 @@ my $volname_cases_iso_invalid = [
     },
 ];
 
+my $volname_cases_vztmpl_valid = [
+    # plain paths
+    {
+        path => 'archlinux-base_20190924-1_amd64.tar',
+        expected => {
+            file => 'archlinux-base_20190924-1_amd64.tar',
+            ext => 'tar',
+            'ext-archive' => 'tar',
+            'disk-path' => 'archlinux-base_20190924-1_amd64.tar',
+            path => 'archlinux-base_20190924-1_amd64.tar',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar',
+        },
+    },
+    {
+        path => 'archlinux-base_20190924-1_amd64.tar.gz',
+        expected => {
+            file => 'archlinux-base_20190924-1_amd64.tar.gz',
+            ext => 'tar.gz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'gz',
+            'disk-path' => 'archlinux-base_20190924-1_amd64.tar.gz',
+            path => 'archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
+        },
+    },
+    {
+        path => 'alpine-3.10-default_20190626_amd64.tar.xz',
+        expected => {
+            file => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            ext => 'tar.xz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'xz',
+            'disk-path' => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            path => 'alpine-3.10-default_20190626_amd64.tar.xz',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
+        },
+    },
+    {
+        path => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+        expected => {
+            file => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            'disk-path' => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            path => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        },
+    },
+    {
+        path => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        expected => {
+            file => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            'disk-path' => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            path => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        },
+    },
+
+    # case-insensitive file extensions
+    {
+        path => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+        expected => {
+            file => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            ext => 'TAR.GZ',
+            'ext-archive' => 'TAR',
+            'ext-compression' => 'GZ',
+            'disk-path' => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            path => 'ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/ARCHLINUX-BASE 20190924-1 AMD64.TAR.GZ',
+        },
+    },
+    {
+        path => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+        expected => {
+            file => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            ext => 'Tar.xZ',
+            'ext-archive' => 'Tar',
+            'ext-compression' => 'xZ',
+            'disk-path' => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            path => 'Alpine 3.10 default_20190626_amd64.Tar.xZ',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/Alpine 3.10 default_20190626_amd64.Tar.xZ',
+        },
+    },
+];
+
+my $volname_cases_vztmpl_invalid = [
+    {
+        description => "Invalid file extension (vztmpl)",
+        args => {
+            path => 'archlinux-base_20190924-1_amd64.zip',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
+    $volname_cases_vztmpl_valid,
 ];
 
 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
+    $volname_cases_vztmpl_invalid,
 ];
 
 {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index d4860344..b539ef78 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -831,10 +831,10 @@ sub parse_volname {
         if ($vtype eq 'iso') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }
 
-    if ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
-        return ('vztmpl', $1, undef, undef, undef, undef, 'raw');
+        if ($vtype eq 'vztmpl') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }
 
     if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
@@ -1734,11 +1734,15 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'vztmpl') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
+            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
+
+            my ($ext, $ext_compression) = $parts->@{qw(ext ext-compression)};
+            my $format = $ext eq 'tar' ? $ext : ('t' . $ext_compression);
 
             return {
-                volid => "$storeid:vztmpl/$1",
-                format => $2 eq 'tar' ? $2 : "t$2",
+                volid => $parts->{volid},
+                format => $format,
             };
         }
 
-- 
2.47.3

[PATCH pve-storage v2 25/50] tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions

[Max R. Carrara]

Add support for the 'backup' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace usages of the
`PVE::Storage::BACKUP_EXT_RE_2` regex across the repository.

Note that the new regex for parsing 'backup' vtype file paths is also
able to optionally extract the VMID and the type of guest from the
file name. This makes it possible to further simplify some parts of
the original logic.

As done with the `ISO_EXT_RE_0` and `VZTMPL_EXT_RE_1` regexes, note
the removal of the `BACKUP_EXT_RE_2` regex in `ApiChangeLog`, since
it's now completely unused. Add a FIXME to note that it should be
removed on the next APIAGE reset.

Additionally, the `COMPRESSOR_RE` constant in `PVE::Storage::Plugin`
is now unused. Together with its related `KNOWN_COMPRESSION_FORMATS`
constant, it should be phased out and replaced eventually. Add a TODO
comment for that.

In the meantime, keep track of the compression formats for specific
vtypes in the `::Common::Parse` module until we move them to a more
adequate place.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                                |   1 +
 src/PVE/Storage.pm                          |   5 +-
 src/PVE/Storage/Common/Parse.pm             |  63 +++++++
 src/PVE/Storage/Common/test/parser_tests.pl | 181 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  42 +++--
 5 files changed, 271 insertions(+), 21 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index ffe3725c..dc77bf41 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -31,6 +31,7 @@ Future changes should be documented in here.
 * Remove the following regular expressions:
   * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
   * `$PVE::Storage::VZTMPL_EXT_RE_1` (`vztmpl` volume type)
+  * `$PVE::Storage::BACKUP_EXT_RE_2` (`backup` volume type)
 
   These regular expressions were mostly used for parsing file paths and volume
   names corresponding to their volume types, noted in parentheses above.
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index afe29178..9b12ed8d 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -126,6 +126,7 @@ our $ISO_EXT_RE_0 = qr/\.(?:iso|img)/i;
 # FIXME: remove this regex on the next APIAGE reset.
 our $VZTMPL_EXT_RE_1 = qr/\.(?|(tar)(?!\.)|tar\.(gz|xz|zst|bz2))/i;
 
+# FIXME: remove this regex on the next APIAGE reset.
 our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
 
 our $IMPORT_EXT_RE_1 = qr/\.(ova|ovf|qcow2|raw|vmdk)/;
@@ -767,9 +768,7 @@ sub path_to_volume_id {
         }
 
         if ($vtype eq 'backup') {
-            return if $filename !~ m!/([^/]+$BACKUP_EXT_RE_2)$!;
-            my $name = $1;
-            return "$sid:backup/$name";
+            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
         }
 
         if ($vtype eq 'snippets') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 207f287b..d3667117 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -35,12 +35,16 @@ primarily related to.
 
 my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');
 
+my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }
 
 my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSION_EXTENSIONS);
 
+my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSION_EXTENSIONS);
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -50,6 +54,11 @@ my $RE_CONTAINS_PARENT_DIR = qr!
     ( /$RE_PARENT_DIR$ )  # /..  --> End of path
 !xn;
 
+# A vmid is an integer between 100 and 999_999_999
+# See: https://git.proxmox.com/?p=pve-common.git;a=blob;f=src/PVE/JSONSchema.pm;h=17e7126a6c42f8326a1da890680af10b6106d906;hb=refs/heads/master#l62
+# See also: https://git.proxmox.com/?p=pve-common.git;a=blob;f=src/PVE/JSONSchema.pm;h=17e7126a6c42f8326a1da890680af10b6106d906;hb=refs/heads/master#l304
+my $RE_VMID = qr![1-9][0-9]{2,8}!;
+
 my $RE_ISO_FILE_PATH = qr!
     (?<path>
         (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
@@ -69,14 +78,44 @@ my $RE_VZTMPL_FILE_PATH = qr!
     )
 !xn;
 
+my $RE_GUEST_VZDUMP_BACKUP_FILE_NAME = qr!
+    vzdump
+    - (?<type> openvz|lxc|qemu)
+    - (?<vmid> $RE_VMID)
+    - [^/]+
+!xn;
+
+my $RE_BACKUP_FILE_PATH = qr!
+    (?<path>
+        (?<file>
+            (
+                ($RE_GUEST_VZDUMP_BACKUP_FILE_NAME)
+                |
+                ([^/]+)
+            )
+            \.
+            (?<ext>
+                (?<ext_archive> tgz)
+                |
+                (
+                    (?<ext_archive> tar|vma)
+                    ( \. (?<ext_compression> $RE_BACKUP_COMPRESSION_EXTENSIONS) )?
+                )
+            )
+        )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
+    backup => qr/^$RE_BACKUP_FILE_PATH$/,
 };
 
 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
+    backup => qr/^$RE_BACKUP_FILE_PATH$/,
 };
 
 my sub contains_parent_dir($path) {
@@ -178,6 +217,30 @@ C<ext-archive>, C<ext-compression> (optional):
         volname => 'vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
     }
 
+=item C<backup>
+
+C<type> (optional), C<vmid> (optional), which are included in backups created
+by Proxmox VE, as well as C<ext-archive>, C<ext-compression> (optional):
+
+    {
+        file => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+
+        # Used to further differentiate 'file'
+        type => 'qemu',
+        vmid => '16110',
+
+        ext => 'vma.lzo',
+
+        # Used to further differentiate 'ext'
+        'ext-archive' => 'vma',
+        'ext-compression' => 'lzo',
+
+        'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        path => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        vtype => 'backup',
+        volname => 'backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+    }
+
 =back
 
 =cut
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 1ea8df1a..979758b5 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -193,14 +193,195 @@ my $volname_cases_vztmpl_invalid = [
     },
 ];
 
+my $volname_cases_backup_valid = [
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            ext => 'vma',
+            'ext-archive' => 'vma',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            ext => 'vma.gz',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'gz',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            path => 'vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            ext => 'vma.lzo',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'lzo',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+        },
+    },
+    {
+        path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+        expected => {
+            type => 'qemu',
+            vmid => '16110',
+            file => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            ext => 'vma.zst',
+            'ext-archive' => 'vma',
+            'ext-compression' => 'zst',
+            'disk-path' => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            path => 'vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+            vtype => 'backup',
+            volname => 'backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            ext => 'tar.lzo',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'lzo',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            path => 'vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            ext => 'tar.gz',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'gz',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            path => 'vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+        },
+    },
+    {
+        path => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+        expected => {
+            type => 'lxc',
+            vmid => '16112',
+            file => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            ext => 'tgz',
+            'ext-archive' => 'tgz',
+            'disk-path' => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            path => 'vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+            vtype => 'backup',
+            volname => 'backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+        },
+    },
+    {
+        path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+        expected => {
+            type => 'openvz',
+            vmid => '16112',
+            file => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            'disk-path' => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+            vtype => 'backup',
+            volname => 'backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+        },
+    },
+];
+
+my $volname_cases_backup_invalid = [
+    {
+        description => "Invalid file extension (1) (backup)",
+        args => {
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.tar.zip',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Invalid file extension (2) (backup)",
+        args => {
+            path => 'vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Invalid file extension (3) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (1) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.Tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (2) (backup)",
+        args => {
+            path => 'vzdump-qemu-16110-2020_03_30-21_12_40.tar.Gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
+    $volname_cases_backup_valid,
 ];
 
 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
+    $volname_cases_backup_invalid,
 ];
 
 {
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index b539ef78..3f1c436c 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -28,7 +28,9 @@ use JSON;
 
 use base qw(PVE::SectionConfig);
 
+# TODO: Phase out these two constants
 use constant KNOWN_COMPRESSION_FORMATS => ('gz', 'lzo', 'zst', 'bz2');
+# FIXME: remove this regex on the next APIAGE reset.
 use constant COMPRESSOR_RE => join('|', KNOWN_COMPRESSION_FORMATS);
 
 use constant LOG_EXT => ".log";
@@ -835,14 +837,10 @@ sub parse_volname {
         if ($vtype eq 'vztmpl') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }
 
-    if ($volname =~ m!^backup/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!) {
-        my $fn = $1;
-        if ($fn =~ m/^vzdump-(openvz|lxc|qemu)-(\d+)-.+/) {
-            return ('backup', $fn, $2, undef, undef, undef, 'raw');
+        if ($vtype eq 'backup') {
+            return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
         }
-        return ('backup', $fn, undef, undef, undef, undef, 'raw');
     }
 
     if ($volname =~ m!^snippets/([^/]+)$!) {
@@ -1747,37 +1745,45 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'backup') {
-            return if $filename !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
+            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
 
-            my $original = $path;
-            my $format = $2;
-            $filename = $1;
+            my $format = $parts->{ext};
+            my $volume_path = $parts->{path};
 
-            # only match for VMID now, to avoid false positives (VMID in parent directory name)
-            return if defined($vmid) && $filename !~ m/\S+-$vmid-\S+/;
+            # Check if parsed VMID matched provided VMID in order to avoid
+            # false positives (VMID in parent directory name)
+            my $parsed_vmid = $parts->{vmid};
+            if (defined($vmid)) {
+                if (defined($parsed_vmid)) {
+                    return if $vmid ne $parsed_vmid;
+                } else {
+                    return;
+                }
+            }
 
             my $info = {
-                volid => "$storeid:backup/$filename",
+                volid => $parts->{volid},
                 format => $format,
             };
 
-            my $archive_info = eval { PVE::Storage::archive_info($filename) } // {};
+            my $archive_info = eval { PVE::Storage::archive_info($volume_path) } // {};
 
             $info->{ctime} = $archive_info->{ctime} if defined($archive_info->{ctime});
             $info->{subtype} = $archive_info->{type} // 'unknown';
 
-            if (defined($vmid) || $filename =~ m!\-([1-9][0-9]{2,8})\-[^/]+\.${format}$!) {
-                $info->{vmid} = $vmid // $1;
+            if (defined($vmid) || defined($parsed_vmid)) {
+                $info->{vmid} = $vmid // $parsed_vmid;
             }
 
-            my $notes_filename = $original . NOTES_EXT;
+            my $notes_filename = $path . NOTES_EXT;
             if (-f $notes_filename) {
                 my $notes = PVE::Tools::file_read_firstline($notes_filename);
                 $info->{notes} = eval { decode('UTF-8', $notes, 1) } // $notes
                     if defined($notes);
             }
 
-            $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
+            $info->{protected} = 1 if -e PVE::Storage::protection_file_path($path);
 
             return $info;
         }
-- 
2.47.3

[PATCH pve-storage v2 26/50] tree-wide: replace usages of inline regexes for snippets with parsers

[Max R. Carrara]

Add support for the 'snippets' volume type in
`PVE::Storage::Common::Parse`.

Also add corresponding test cases.

Use the module's parsing functions to replace parsing the a snippet
file's name via regexes or the `basename()` function.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage.pm                          |  4 +---
 src/PVE/Storage/Common/Parse.pm             |  8 +++++++
 src/PVE/Storage/Common/test/parser_tests.pl | 25 +++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   | 11 +++++----
 4 files changed, 41 insertions(+), 7 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 9b12ed8d..675370a6 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -772,9 +772,7 @@ sub path_to_volume_id {
         }
 
         if ($vtype eq 'snippets') {
-            return if $filename !~ m!/([^/]+)$!;
-            my $name = $1;
-            return "$sid:snippets/$name";
+            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
         }
 
         if ($vtype eq 'import') {
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index d3667117..8e9769fb 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -106,16 +106,24 @@ my $RE_BACKUP_FILE_PATH = qr!
     )
 !xn;
 
+my $RE_SNIPPETS_FILE_PATH = qr!
+    (?<path>
+        (?<file> [^/]+ )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
+    snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
 };
 
 my $RE_VOLNAME_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
+    snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
 };
 
 my sub contains_parent_dir($path) {
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 979758b5..137b91b8 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -372,10 +372,35 @@ my $volname_cases_backup_invalid = [
     },
 ];
 
+my $volname_cases_snippets_valid = [
+    # plain files
+    {
+        path => 'hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            'disk-path' => 'hookscript.pl',
+            path => 'hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/hookscript.pl',
+        },
+    },
+    {
+        path => 'userconfig.yaml',
+        expected => {
+            file => 'userconfig.yaml',
+            'disk-path' => 'userconfig.yaml',
+            path => 'userconfig.yaml',
+            vtype => 'snippets',
+            volname => 'snippets/userconfig.yaml',
+        },
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
     $volname_cases_backup_valid,
+    $volname_cases_snippets_valid,
 ];
 
 my $cases_invalid_all = [
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3f1c436c..b327e4d2 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -841,10 +841,10 @@ sub parse_volname {
         if ($vtype eq 'backup') {
             return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
         }
-    }
 
-    if ($volname =~ m!^snippets/([^/]+)$!) {
-        return ('snippets', $1, undef, undef, undef, undef, 'raw');
+        if ($vtype eq 'snippets') {
+            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
+        }
     }
 
     if ($volname =~
@@ -1789,8 +1789,11 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'snippets') {
+            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
+
             return {
-                volid => "$storeid:snippets/" . basename($filename),
+                volid => $parts->{volid},
                 format => 'snippet',
             };
         }
-- 
2.47.3

[PATCH pve-storage v2 27/50] tree-wide: partially replace usages of regexes for 'import' vtype

[Max R. Carrara]

Add partial support for the 'import' volume type in
`PVE::Storage::Common::Parse`. What remains unsupported right now is
parsing volume names for that type, as those require a little more
care.

Replace most usages of the `PVE::Storage::IMPORT_EXT_RE_1` regex with
parsing functions from `::Common::Parse`.

Replace the one remaining spot where we use the
`PVE::Storage::UPLOAD_IMPORT_EXT_RE_1` regex with a parsing function
as well. Note the removal of this regex in `ApiChangeLog` and add a
corresponding FIXME comment for removing it on the next APIAGE reset.

It is important to note that the `UPLOAD_IMPORT_EXT_RE_1` regex only
exists to handle the special case of excluding .ovf files from the
upload / download_url API methods. Therefore, instead of adding a
separate parser (or a flag etc.) to handle this case, just parse the
path of the up-/downloaded file regardless. Then, raise a parameter
exception when the `ovf` file extension is encountered.

Also, since the `PVE::Storage::SAFE_CHAR_CLASS_RE` regex is now
unused, migrate it to `::Common::Parse` and allow it to be obtained by
using a new helper subroutine. Note this in `ApiChangeLog` as well,
and add a corresponding FIXME comment, too.

The main reason for moving `SAFE_CHAR_CLASS_RE` and making it private
is to discourage "shotgun parsing" throughout the repo and keeping
most of our parsing code in `::Common::Parse` -- just like the other
regexes that are removed.

Additionally, we avoid duplicating the regex that way. An alternative
would have been to import it from `PVE::Storage` and use it in
`::Common::Parse`, but that would have led to cyclic imports. That
approach would not have solved the shotgun parsing problem, though.

However, since that particular regex is still important for
third-party plugin authors (for backup provider plugins in particular)
and perhaps for some isolated parsing subs that don't *really* fit
into `::Common::Parse` in the future, allowing it to be obtained
through a subroutine makes its usage much more explicit. As a side
effect, putting the regex behind a subroutine prevents it from being
altered.

Finally, add a bunch of test cases that target the `import` volume
type for the `list_volumes()` plugin API method. For all of these
tests, sort the expected and resulting list items by their volume ID
in order to make the output comparison deterministic. Iterate over the
declared content / volume types of the mocked storage config instead
of using a fixed list of vtypes as well.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                          |   9 ++
 src/PVE/API2/Storage/Status.pm        |  13 +-
 src/PVE/BackupProvider/Plugin/Base.pm |   6 +-
 src/PVE/Storage.pm                    |   9 +-
 src/PVE/Storage/Common/Parse.pm       |  53 ++++++++
 src/PVE/Storage/Plugin.pm             |   9 +-
 src/test/list_volumes_test.pm         | 180 +++++++++++++++++++++++++-
 7 files changed, 259 insertions(+), 20 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index dc77bf41..b1b6b88e 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -32,6 +32,7 @@ Future changes should be documented in here.
   * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
   * `$PVE::Storage::VZTMPL_EXT_RE_1` (`vztmpl` volume type)
   * `$PVE::Storage::BACKUP_EXT_RE_2` (`backup` volume type)
+  * `$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1` (`import` volume type)
 
   These regular expressions were mostly used for parsing file paths and volume
   names corresponding to their volume types, noted in parentheses above.
@@ -39,6 +40,14 @@ Future changes should be documented in here.
   Instead, The parsing functions in `PVE::Storage::Common::Parse` should be
   used.
 
+* Migrate the following regular expressions:
+  * `$PVE::Storage::SAFE_CHAR_CLASS_RE`
+
+  These regular expressions are now defined as private constants in the
+  `PVE::Storage::Common::Parse` module and can be obtained by using the
+  corresponding subroutines in the *STANDALONE REGULAR EXPRESSIONS* section in
+  said module.
+
 ## Version 15:
 
 * Add new `$snapname` parameter to the `volume_resize()` plugin method
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 60f08628..d2451898 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -98,13 +98,18 @@ my sub parse_transferred_file_path_extension : prototype($$) {
     }
 
     if ($vtype eq 'import') {
-        if (
-            $path !~ m!${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1$!
-        ) {
+        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
+
+        if (!defined($parts)) {
             raise_param_exc({ filename => "invalid filename or wrong extension" });
         }
 
-        my $ext = $1;
+        my $ext = $parts->{ext};
+
+        if ($ext eq 'ovf') {
+            raise_param_exc({ filename => "wrong file extension" });
+        }
+
         return $ext;
     }
 
diff --git a/src/PVE/BackupProvider/Plugin/Base.pm b/src/PVE/BackupProvider/Plugin/Base.pm
index 24b1421e..addd1a55 100644
--- a/src/PVE/BackupProvider/Plugin/Base.pm
+++ b/src/PVE/BackupProvider/Plugin/Base.pm
@@ -247,9 +247,9 @@ determined for the backup task and returned to the caller via a hash reference:
     my $res = $backup_provider->backup_init($vmid, $vmtype, $start_time);
     my $archive_name = $res->{'archive-name'};
 
-The archive name must contain only characters from the
-C<$PVE::Storage::SAFE_CHAR_CLASS_RE> character class as well as forward slash
-C</> and colon C<:>.
+The archive name must contain only characters from the character class returned
+by C<L<< get_re_safe_char_class()|PVE::Storage::Common::Parse/"get_re_safe_char_class" >>>
+as well as forward slash C</> and colon C<:>.
 
 Use C<$self> to remember it for the C<backup_container()> or C<backup_vm()>
 method that will be called later.
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 675370a6..ce90f942 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -27,6 +27,7 @@ use PVE::Storage::Common qw(
     plugin_get_vtype_subdir
 );
 use PVE::Storage::Common::Parse qw(
+    get_re_safe_char_class
     parse_abs_path_as_volid
 );
 use PVE::RESTEnvironment qw(log_warn);
@@ -131,9 +132,11 @@ our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPR
 
 our $IMPORT_EXT_RE_1 = qr/\.(ova|ovf|qcow2|raw|vmdk)/;
 
+# FIXME: remove this regex on the next APIAGE reset.
 our $UPLOAD_IMPORT_EXT_RE_1 = qr/\.(ova|qcow2|raw|vmdk)/;
 
-our $SAFE_CHAR_CLASS_RE = qr/[a-zA-Z0-9\-\.\+\=\_]/;
+# FIXME: remove this regex on the next APIAGE reset.
+our $SAFE_CHAR_CLASS_RE = get_re_safe_char_class();
 our $SAFE_CHAR_WITH_WHITESPACE_CLASS_RE = qr/[ a-zA-Z0-9\-\.\+\=\_]/;
 
 our $OVA_CONTENT_RE_1 = qr/${SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.(qcow2|raw|vmdk)/;
@@ -776,9 +779,7 @@ sub path_to_volume_id {
         }
 
         if ($vtype eq 'import') {
-            return if $filename !~ m!/(${SAFE_CHAR_CLASS_RE}+${IMPORT_EXT_RE_1})$!;
-            my $name = $1;
-            return "$sid:import/$name";
+            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
         }
 
         return;
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 8e9769fb..2193aa65 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -9,6 +9,8 @@ use PVE::Storage::Common qw(
 use Exporter qw(import);
 
 our @EXPORT_OK = qw(
+    get_re_safe_char_class
+
     parse_rel_path_as_volname_parts
     parse_rel_path_as_volname
     parse_volname_as_parts
@@ -37,6 +39,8 @@ my @VZTMPL_COMPRESSION_EXTENSIONS = ('gz', 'xz', 'zst', 'bz2');
 
 my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');
 
+my @IMPORT_EXTENSIONS = ('ova', 'ovf', 'qcow2', 'raw', 'vmdk');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }
@@ -45,6 +49,10 @@ my $RE_VZTMPL_COMPRESSION_EXTENSIONS = join_to_re_alternations(@VZTMPL_COMPRESSI
 
 my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSION_EXTENSIONS);
 
+my $RE_IMPORT_EXTENSIONS = join_to_re_alternations(@IMPORT_EXTENSIONS);
+
+my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -112,11 +120,18 @@ my $RE_SNIPPETS_FILE_PATH = qr!
     )
 !xn;
 
+my $RE_IMPORT_FILE_PATH = qr!
+    (?<path>
+        (?<file> ($RE_SAFE_CHAR_CLASS)+ \. (?<ext> $RE_IMPORT_EXTENSIONS) )
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
     snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
+    import => qr/^$RE_IMPORT_FILE_PATH$/,
 };
 
 my $RE_VOLNAME_FOR_VTYPE = {
@@ -170,6 +185,44 @@ my sub split_leading_dir_from_path($path, $directory) {
     return;
 }
 
+=head1 STANDALONE REGULAR EXPRESSIONS
+
+Some of the regular expressions that this module defines can be obtained using
+the subroutines listed below.
+
+B<These subroutines are primarily intended to be used by third-party plugins.>
+
+Parsing code of inbuilt plugins and other modules in the C<L<PVE::Storage>>
+namespace should instead use the constants defined in
+C<L<PVE::Storage::Common::Parse>> directly, and keep the parsing code in there
+as well.
+
+=cut
+
+=head3 get_re_safe_char_class
+
+Returns a regex consisting of a single character class that matches any
+character that is safe to use in more critical places, such as untrusted file
+names.
+
+You would typically use this to obtain a constant that you can use in your own
+parsing code:
+
+
+    my $RE_SAFE_CHAR_CLASS = get_re_safe_char_class();
+
+    # [...]
+
+    my sub is_valid_file_name($file_name) {
+        return $file_name =~ m/${RE_SAFE_CHAR_CLASS}+ \. (json|toml)/xn;
+    }
+
+=cut
+
+sub get_re_safe_char_class() {
+    return $RE_SAFE_CHAR_CLASS;
+}
+
 =head1 PARSERS RELATED TO VOLUMES
 
 The parsing functions in this section primarily deal with parsing data related
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index b327e4d2..afddca73 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1799,13 +1799,12 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'import') {
-            return
-                if $filename !~
-                m!/(${PVE::Storage::SAFE_CHAR_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!i;
+            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+            return if !defined($parts);
 
             return {
-                volid => "$storeid:import/$1",
-                format => "$2",
+                volid => $parts->{volid},
+                format => $parts->{ext},
             };
         }
 
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 08769027..5cb08880 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -72,6 +72,7 @@ my $scfg = {
         'images' => 1,
         'snippets' => 1,
         'backup' => 1,
+        'import' => 1,
     },
 };
 
@@ -462,6 +463,171 @@ my @tests = (
         ],
         expected => [], # returns empty list
     },
+    {
+        description => 'VMID: none, valid file names for import',
+        vmid => undef,
+        files => [
+            "$storage_dir/import/import.ova",
+            "$storage_dir/import/import.ovf",
+            "$storage_dir/import/some-disk.qcow2",
+            "$storage_dir/import/some-disk.vmdk",
+            "$storage_dir/import/some-raw-disk.raw",
+        ],
+        expected => [
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ovf',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ovf",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-disk.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'vmdk',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-disk.vmdk",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'raw',
+                size => DEFAULT_SIZE,
+                volid => "local:import/some-raw-disk.raw",
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching file paths for import',
+        vmid => undef,
+        files => [
+            # Malformed file names
+            "$storage_dir/import/import.ovff",
+            "$storage_dir/import/importova",
+            "$storage_dir/import/import.ov",
+            "$storage_dir/import/diskraw",
+            "$storage_dir/import/diskvmdk",
+            "$storage_dir/import/disk.invalid",
+            "$storage_dir/import/.ova",
+            "$storage_dir/import/.raw",
+            # Trailing whitespace must not be trimmed
+            "$storage_dir/import/import.ova\t",
+            "$storage_dir/import/disk.raw    ",
+            # Whitespace in file name
+            "$storage_dir/import/something I want to import.ova",
+            "$storage_dir/import/ .raw",
+            "$storage_dir/import/ disk .vmdk",
+            "$storage_dir/import/disk .qcow2",
+            "$storage_dir/import/ import.ova",
+            # Unsafe characters in file name
+            "$storage_dir/import/linux🐧-vm.ova",
+            "$storage_dir/import/🐪perl-playground🐪.ova",
+            "$storage_dir/import/fish_<><_<><_<><.ova",
+            $storage_dir . '/import/C:\\\\Windows\\Path.ova',
+            # Content inside .ova files may only be specified as part
+            # of volume names, and may never appear when looked up as
+            # a file path
+            "$storage_dir/import/import.ova/disk.qcow2",
+            "$storage_dir/import/import.ova/disk.raw",
+            "$storage_dir/import/import.ova/disk.vmdk",
+            "$storage_dir/import/import.ova/disk.invalid",
+        ],
+        expected => [], # returns empty list
+    },
+    {
+        description => 'VMID: none, weird but valid file names for import',
+        vmid => undef,
+        files => [
+            "$storage_dir/import/import.ova.ova",
+            "$storage_dir/import/import.ova.ova.ova",
+            "$storage_dir/import/import.ova.ova.ova.ova",
+            "$storage_dir/import/ova.ova",
+            "$storage_dir/import/ova.ovf",
+            "$storage_dir/import/ova.vmdk",
+            "$storage_dir/import/raw.raw.qcow2",
+            "$storage_dir/import/raw.raw.qcow2.import.qcow2",
+            "$storage_dir/import/raw.raw.raw.your-boat.ova",
+        ],
+        expected => [
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/import.ova.ova.ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.ova",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ovf',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.ovf",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'vmdk',
+                size => DEFAULT_SIZE,
+                volid => "local:import/ova.vmdk",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'qcow2',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.qcow2.import.qcow2",
+            },
+            {
+                content => 'import',
+                ctime => DEFAULT_CTIME,
+                format => 'ova',
+                size => DEFAULT_SIZE,
+                volid => "local:import/raw.raw.raw.your-boat.ova",
+            },
+        ],
+    },
 );
 
 # provide static vmlist for tests
@@ -497,6 +663,10 @@ $mock_fsi->redefine(
     },
 );
 
+my sub cmp_volinfo_by_volid {
+    return $a->{volid} cmp $b->{volid};
+}
+
 my $plan = scalar @tests;
 plan tests => $plan + 1;
 
@@ -520,14 +690,14 @@ plan tests => $plan + 1;
 
 {
     my $sid = 'local';
-    my $types = ['rootdir', 'images', 'vztmpl', 'iso', 'backup', 'snippets'];
+    my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
     my @suffixes = ('qcow2', 'raw', 'vmdk', 'vhdx');
 
     # run through test cases
     foreach my $tt (@tests) {
         my $vmid = $tt->{vmid};
         my $files = $tt->{files};
-        my $expected = $tt->{expected};
+        my $expected = [sort cmp_volinfo_by_volid $tt->{expected}->@*];
         my $description = $tt->{description};
         my $parent = $tt->{parent};
 
@@ -550,8 +720,10 @@ plan tests => $plan + 1;
             }
         }
 
-        my $got;
-        eval { $got = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types) };
+        my $got = eval {
+            my $volume_list = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types);
+            return [sort cmp_volinfo_by_volid $volume_list->@*];
+        };
         $got = $@ if $@;
 
         is_deeply($got, $expected, $description) || diag(explain($got));
-- 
2.47.3

[PATCH pve-storage v2 28/50] tree-wide: replace remaining usages of regexes for 'import' vtype

[Max R. Carrara]

Implement the remaining pieces that were missing from fully supporting
the 'import' volume type in `PVE::Storage::Common::Parse`. In
particular, parsing volume names for '.ova' files that refer to
contents inside the '.ova' file is now fully supported.

Also add corresponding test cases to 'parser_tests.pl'.

Replace the remaining usages of the `PVE::Storage::IMPORT_EXT_RE_1`
and `PVE::Storage::OVA_CONTENT_RE_1` with parsing functions from
`::Common::Parse` across the repository. As before with other regexes,
mark their removal in `ApiChangeLog` and add corresponding FIXME
comments for a future APIAGE reset.

Note that the logic in `PVE::Storage::Plugin::parse_volname` can now
be simplified a little further, since the `parse_volname_as_parts()`
parsing function also handles extracting the inner and outer file
extensions, amongst other parts.

Since the only remaining usage of the
`PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE` regex is now in
`PVE::GuestImport::OVF`, handle it the same way as the removed
`SAFE_CHAR_CLASS_RE`. This means that we move it into
`::Common::Parse`, make it private, and only expose it through a
helper subroutine. Also note this in `ApiChangeLog` and add a
corresponding FIXME comment once again.

This allows us to stop using it in `PVE::GuestImport::OVF` as well,
where we instead add a small helper sub to aid us in parsing the
obtained file path.

Finally, update the test cases in 'guest_import_test.pl' that match
for specific parsing-related exception messages.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ApiChangeLog                                |   3 +
 src/PVE/GuestImport.pm                      |  28 +-
 src/PVE/GuestImport/OVF.pm                  |  24 +-
 src/PVE/Storage.pm                          |   7 +-
 src/PVE/Storage/Common/Parse.pm             |  81 ++++++
 src/PVE/Storage/Common/test/parser_tests.pl | 282 ++++++++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  20 +-
 src/test/guest_import_test.pl               |   6 +-
 8 files changed, 417 insertions(+), 34 deletions(-)

diff --git a/ApiChangeLog b/ApiChangeLog
index b1b6b88e..30dd0374 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -32,7 +32,9 @@ Future changes should be documented in here.
   * `$PVE::Storage::ISO_EXT_RE_0` (`iso` volume type)
   * `$PVE::Storage::VZTMPL_EXT_RE_1` (`vztmpl` volume type)
   * `$PVE::Storage::BACKUP_EXT_RE_2` (`backup` volume type)
+  * `$PVE::Storage::IMPORT_EXT_RE_1` (`import` volume type)
   * `$PVE::Storage::UPLOAD_IMPORT_EXT_RE_1` (`import` volume type)
+  * `$PVE::Storage::OVA_CONTENT_RE_1` (`import` volume type)
 
   These regular expressions were mostly used for parsing file paths and volume
   names corresponding to their volume types, noted in parentheses above.
@@ -42,6 +44,7 @@ Future changes should be documented in here.
 
 * Migrate the following regular expressions:
   * `$PVE::Storage::SAFE_CHAR_CLASS_RE`
+  * `$PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE`
 
   These regular expressions are now defined as private constants in the
   `PVE::Storage::Common::Parse` module and can be obtained by using the
diff --git a/src/PVE/GuestImport.pm b/src/PVE/GuestImport.pm
index 3d59dcd7..3be84fc9 100644
--- a/src/PVE/GuestImport.pm
+++ b/src/PVE/GuestImport.pm
@@ -6,6 +6,9 @@ use warnings;
 use File::Path;
 
 use PVE::Storage;
+use PVE::Storage::Common::Parse qw(
+    parse_volname_as_parts
+);
 use PVE::Tools qw(run_command);
 
 sub extract_disk_from_import_file {
@@ -13,31 +16,26 @@ sub extract_disk_from_import_file {
 
     my ($source_storeid, $volname) = PVE::Storage::parse_volume_id($volid);
     $target_storeid //= $source_storeid;
-    my $cfg = PVE::Storage::config();
 
-    my ($vtype, $name, undef, undef, undef, undef, $fmt) =
-        PVE::Storage::parse_volname($cfg, $volid);
+    my $parts = parse_volname_as_parts($volname);
+    die "cannot extract $volid - invalid volname $volname\n"
+        if !defined($parts);
+
+    my ($vtype, $outer_fmt) = $parts->@{qw(vtype ext)};
 
     die "only files with content type 'import' can be extracted\n"
         if $vtype ne 'import';
 
     die "only files from 'ova' format can be extracted\n"
-        if $fmt !~ m/^ova\+/;
+        if $outer_fmt ne 'ova';
 
-    # extract the inner file from the name
-    my $archive_volid;
-    my $inner_file;
-    my $inner_fmt;
-    if ($name =~ m!^(.*\.ova)/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+)$!) {
-        $archive_volid = "$source_storeid:import/$1";
-        $inner_file = $2;
-        ($inner_fmt) = $fmt =~ /^ova\+(.*)$/;
-    } else {
-        die "cannot extract $volid - invalid volname $volname\n";
-    }
+    my $archive_volid = $source_storeid . ':' . $vtype . '/' . $parts->{'disk-path'};
+    my $inner_file = $parts->{'content-file'};
+    my $inner_fmt = $parts->{'content-ext'};
 
     die "cannot determine format of '$volid'\n" if !$inner_fmt;
 
+    my $cfg = PVE::Storage::config();
     my $ova_path = PVE::Storage::path($cfg, $archive_volid);
 
     my $tmpdir = PVE::Storage::get_image_dir($cfg, $target_storeid, $vmid);
diff --git a/src/PVE/GuestImport/OVF.pm b/src/PVE/GuestImport/OVF.pm
index 8999cab4..ed76b8ff 100644
--- a/src/PVE/GuestImport/OVF.pm
+++ b/src/PVE/GuestImport/OVF.pm
@@ -12,6 +12,9 @@ use Cwd 'realpath';
 
 use PVE::Tools;
 use PVE::Storage;
+use PVE::Storage::Common::Parse qw(
+    get_re_safe_char_with_whitespace_class
+);
 use PVE::Seccomp qw(stmt /^BPF_/);
 
 # map OVF resources types to descriptive strings
@@ -184,6 +187,22 @@ my sub read_ovf_file {
     return $raw;
 }
 
+my $RE_SAFE_FILEPATH = qr/
+    ^ (?<filepath> (${get_re_safe_char_with_whitespace_class()}+) ) $
+/xn;
+
+my sub parse_filepath_from_xpath {
+    my ($filepath) = @_;
+
+    # untaint & check no sub/parent dirs
+    $filepath =~ $RE_SAFE_FILEPATH;
+    $filepath = $+{filepath};
+
+    return if !$filepath || $filepath eq '.' || $filepath eq '..';
+
+    return $filepath;
+}
+
 # returns two references, $qm which holds qm.conf style key/values, and \@disks
 my sub parse_ovf_do {
     my ($ovf, $ovf_data, $isOva, $debug) = @_;
@@ -357,9 +376,8 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType", $controller_id,
         }
         print "file path: $filepath\n" if $debug;
         my $original_filepath = $filepath;
-        ($filepath) = $filepath =~ m|^(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+)$|; # untaint & check no sub/parent dirs
-        die "referenced path '$original_filepath' is invalid\n"
-            if !$filepath || $filepath eq "." || $filepath eq "..";
+        $filepath = parse_filepath_from_xpath($filepath);
+        die "referenced path '$original_filepath' is invalid\n" if !$filepath;
 
         push @disks,
             {
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index ce90f942..98d5f64e 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -28,6 +28,7 @@ use PVE::Storage::Common qw(
 );
 use PVE::Storage::Common::Parse qw(
     get_re_safe_char_class
+    get_re_safe_char_with_whitespace_class
     parse_abs_path_as_volid
 );
 use PVE::RESTEnvironment qw(log_warn);
@@ -130,6 +131,7 @@ our $VZTMPL_EXT_RE_1 = qr/\.(?|(tar)(?!\.)|tar\.(gz|xz|zst|bz2))/i;
 # FIXME: remove this regex on the next APIAGE reset.
 our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
 
+# FIXME: remove this regex on the next APIAGE reset.
 our $IMPORT_EXT_RE_1 = qr/\.(ova|ovf|qcow2|raw|vmdk)/;
 
 # FIXME: remove this regex on the next APIAGE reset.
@@ -137,8 +139,11 @@ our $UPLOAD_IMPORT_EXT_RE_1 = qr/\.(ova|qcow2|raw|vmdk)/;
 
 # FIXME: remove this regex on the next APIAGE reset.
 our $SAFE_CHAR_CLASS_RE = get_re_safe_char_class();
-our $SAFE_CHAR_WITH_WHITESPACE_CLASS_RE = qr/[ a-zA-Z0-9\-\.\+\=\_]/;
 
+# FIXME: remove this regex on the next APIAGE reset.
+our $SAFE_CHAR_WITH_WHITESPACE_CLASS_RE = get_re_safe_char_with_whitespace_class();
+
+# FIXME: remove this regex on the next APIAGE reset.
 our $OVA_CONTENT_RE_1 = qr/${SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.(qcow2|raw|vmdk)/;
 
 # FIXME remove with PVE 9.0, add versioned breaks for pve-manager
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 2193aa65..296a6c4d 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -10,6 +10,7 @@ use Exporter qw(import);
 
 our @EXPORT_OK = qw(
     get_re_safe_char_class
+    get_re_safe_char_with_whitespace_class
 
     parse_rel_path_as_volname_parts
     parse_rel_path_as_volname
@@ -41,6 +42,8 @@ my @BACKUP_COMPRESSION_EXTENSIONS = ('gz', 'lzo', 'zst', 'bz2');
 
 my @IMPORT_EXTENSIONS = ('ova', 'ovf', 'qcow2', 'raw', 'vmdk');
 
+my @OVA_CONTENT_EXTENSIONS = ('qcow2', 'raw', 'vmdk');
+
 my sub join_to_re_alternations(@list) {
     return join('|', map { quotemeta } @list);
 }
@@ -51,8 +54,12 @@ my $RE_BACKUP_COMPRESSION_EXTENSIONS = join_to_re_alternations(@BACKUP_COMPRESSI
 
 my $RE_IMPORT_EXTENSIONS = join_to_re_alternations(@IMPORT_EXTENSIONS);
 
+my $RE_OVA_CONTENT_EXTENSIONS = join_to_re_alternations(@OVA_CONTENT_EXTENSIONS);
+
 my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;
 
+my $RE_SAFE_CHAR_WITH_WHITESPACE_CLASS = qr/[ a-zA-Z0-9\-\.\+\=\_]/;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -126,6 +133,42 @@ my $RE_IMPORT_FILE_PATH = qr!
     )
 !xn;
 
+my $RE_OVA_CONTENT = qr!
+    (?<content_file>
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+ \. (?<content_ext> $RE_OVA_CONTENT_EXTENSIONS)
+    )
+!xn;
+
+# NOTE: Volume names with the 'import' vtype are treated differently when
+# they do not stem from a file path directly - see comments inline.
+my $RE_IMPORT_VOLNAME_OVA_FILE_WITH_CONTENT = qr!
+    (?<file>
+        # NOTE: Unlike in RE_IMPORT_FILE_PATH, we allow whitespace here
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+
+        \.
+        (?<ext> ova)
+    ) / (?<content> $RE_OVA_CONTENT)
+!xn;
+
+my $RE_IMPORT_VOLNAME_REGULAR_FILE = qr!
+    (?<file>
+        # NOTE: Unlike in RE_IMPORT_FILE_PATH, we allow whitespace here
+        ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+
+        \.
+        (?<ext> $RE_IMPORT_EXTENSIONS)
+    )
+!xn;
+
+my $RE_IMPORT_VOLNAME = qr!
+    (?<path>
+        # NOTE: Order here matters - the ova+content regex is stricter,
+        # so try this branch here first
+        ($RE_IMPORT_VOLNAME_OVA_FILE_WITH_CONTENT)
+        |
+        ($RE_IMPORT_VOLNAME_REGULAR_FILE)
+    )
+!xn;
+
 my $RE_FILE_PATH_FOR_VTYPE = {
     iso => qr/^$RE_ISO_FILE_PATH$/,
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
@@ -139,6 +182,9 @@ my $RE_VOLNAME_FOR_VTYPE = {
     vztmpl => qr/^$RE_VZTMPL_FILE_PATH$/,
     backup => qr/^$RE_BACKUP_FILE_PATH$/,
     snippets => qr/^$RE_SNIPPETS_FILE_PATH$/,
+
+    # special cases - not reusing file path regexes:
+    import => qr/^$RE_IMPORT_VOLNAME$/,
 };
 
 my sub contains_parent_dir($path) {
@@ -223,6 +269,17 @@ sub get_re_safe_char_class() {
     return $RE_SAFE_CHAR_CLASS;
 }
 
+=head3 get_re_safe_char_with_whitespace_class
+
+Like C<L<< get_re_safe_char_class()|/"get_re_safe_char_class" >>>, but the
+returned character class also allows a space character (U+0020).
+
+=cut
+
+sub get_re_safe_char_with_whitespace_class() {
+    return $RE_SAFE_CHAR_WITH_WHITESPACE_CLASS;
+}
+
 =head1 PARSERS RELATED TO VOLUMES
 
 The parsing functions in this section primarily deal with parsing data related
@@ -302,6 +359,30 @@ by Proxmox VE, as well as C<ext-archive>, C<ext-compression> (optional):
         volname => 'backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
     }
 
+=item C<import>
+
+B<NOTE:> Only for C<L<< parse_volname_as_parts()|/"parse_volname_as_parts" >>>!
+
+C<import> volnames may additionally refer to the C<content> inside an OVA file,
+which is further differentiated into C<content-file> and C<content-ext>:
+
+    {
+        file => 'import.ova',
+        ext => 'ova',
+        'disk-path' => 'import.ova',
+        path => 'import.ova/disk.raw',
+
+        # Exclusive to 'import' volume names
+        content => 'disk.raw',
+
+        # Used to further differentiate 'content'
+        'content-file' => 'disk.raw',
+        'content-ext' => 'raw',
+
+        vtype => 'import',
+        volname => 'import/import.ova/disk.raw',
+    }
+
 =back
 
 =cut
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 137b91b8..2869d3f0 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -396,17 +396,104 @@ my $volname_cases_snippets_valid = [
     },
 ];
 
+my $volname_cases_import_valid = [
+    {
+        path => 'import.ova',
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova',
+            vtype => 'import',
+            volname => 'import/import.ova',
+        },
+    },
+    {
+        path => 'import.ovf',
+        expected => {
+            file => 'import.ovf',
+            ext => 'ovf',
+            'disk-path' => 'import.ovf',
+            path => 'import.ovf',
+            vtype => 'import',
+            volname => 'import/import.ovf',
+        },
+    },
+    {
+        path => 'disk-0.qcow2',
+        expected => {
+            file => 'disk-0.qcow2',
+            ext => 'qcow2',
+            'disk-path' => 'disk-0.qcow2',
+            path => 'disk-0.qcow2',
+            vtype => 'import',
+            volname => 'import/disk-0.qcow2',
+        },
+    },
+    {
+        path => 'raw_disk_v2.5.raw',
+        expected => {
+            file => 'raw_disk_v2.5.raw',
+            ext => 'raw',
+            'disk-path' => 'raw_disk_v2.5.raw',
+            path => 'raw_disk_v2.5.raw',
+            vtype => 'import',
+            volname => 'import/raw_disk_v2.5.raw',
+        },
+    },
+    {
+        path => 'disk-1+data.vmdk',
+        expected => {
+            file => 'disk-1+data.vmdk',
+            ext => 'vmdk',
+            'disk-path' => 'disk-1+data.vmdk',
+            path => 'disk-1+data.vmdk',
+            vtype => 'import',
+            volname => 'import/disk-1+data.vmdk',
+        },
+    },
+];
+
+my $volname_cases_import_invalid = [
+    {
+        description => "Invalid file extension (import)",
+        args => {
+            path => 'import.zip',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Uppercase letter in file extension (import)",
+        args => {
+            path => 'import.Ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Unsafe characters in file name (import)",
+        args => {
+            path => '🐪perl-playground🐪.ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+];
+
 my $cases_valid_all = [
     $volname_cases_iso_valid,
     $volname_cases_vztmpl_valid,
     $volname_cases_backup_valid,
     $volname_cases_snippets_valid,
+    $volname_cases_import_valid,
 ];
 
 my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
     $volname_cases_backup_invalid,
+    $volname_cases_import_invalid,
 ];
 
 {
@@ -475,6 +562,179 @@ my sub run_volname_parsing_tests : prototype($) ($volname_tests) {
     return;
 }
 
+# NOTE: These run for parse_rel_path_as_volname_parts() and parse_rel_path_as_volname() only!
+my $cases_import_special_file_path = [
+    {
+        description => "Whitespace not allowed in volume file paths (import)",
+        args => {
+            path => 'Some Disk.qcow2',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (raw) (import)",
+        args => {
+            path => 'import.ova/disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (qcow2) (import)",
+        args => {
+            path => 'import.ova/disk.qcow2',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Volume file path with disallowed OVA content (vmdk) (import)",
+        args => {
+            path => 'import.ova/disk.vmdk',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+];
+
+# NOTE: These run for parse_volname_as_parts() only!
+my $cases_import_special_volname = [
+    {
+        description => "Whitespace allowed in volume names (import)",
+        args => {
+            volname => 'import/Some Disk.qcow2',
+        },
+        expected => {
+            file => 'Some Disk.qcow2',
+            ext => 'qcow2',
+            'disk-path' => 'Some Disk.qcow2',
+            path => 'Some Disk.qcow2',
+            vtype => 'import',
+            volname => 'import/Some Disk.qcow2',
+        },
+    },
+    {
+        description => "Volume name with OVA content (raw) (import)",
+        args => {
+            volname => 'import/import.ova/disk.raw',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.raw',
+            content => 'disk.raw',
+            'content-file' => 'disk.raw',
+            'content-ext' => 'raw',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.raw',
+        },
+    },
+    {
+        description => "Volume name with OVA content (qcow2) (import)",
+        args => {
+            volname => 'import/import.ova/disk.qcow2',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.qcow2',
+            content => 'disk.qcow2',
+            'content-file' => 'disk.qcow2',
+            'content-ext' => 'qcow2',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.qcow2',
+        },
+    },
+    {
+        description => "Volume name with OVA content (vmdk) (import)",
+        args => {
+            volname => 'import/import.ova/disk.vmdk',
+        },
+        expected => {
+            file => 'import.ova',
+            ext => 'ova',
+            'disk-path' => 'import.ova',
+            path => 'import.ova/disk.vmdk',
+            content => 'disk.vmdk',
+            'content-file' => 'disk.vmdk',
+            'content-ext' => 'vmdk',
+            vtype => 'import',
+            volname => 'import/import.ova/disk.vmdk',
+        },
+    },
+    {
+        description => "Whitespace in volume name + OVA content (import)",
+        args => {
+            volname => 'import/Some Import.ova/disk.qcow2',
+        },
+        expected => {
+            file => 'Some Import.ova',
+            ext => 'ova',
+            'disk-path' => 'Some Import.ova',
+            path => 'Some Import.ova/disk.qcow2',
+            content => 'disk.qcow2',
+            'content-file' => 'disk.qcow2',
+            'content-ext' => 'qcow2',
+            vtype => 'import',
+            volname => 'import/Some Import.ova/disk.qcow2',
+        },
+    },
+];
+
+my sub run_special_import_volname_parsing_tests : prototype() () {
+    for my $case ($cases_import_special_file_path->@*) {
+        subtest $case->{description} => sub () {
+            my ($path, $vtype) = $case->{args}->@{qw(path vtype)};
+
+            my $got_volname_parts = parse_rel_path_as_volname_parts($path, $vtype);
+            my $got_volname = parse_rel_path_as_volname($path, $vtype);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname_parts,
+                    $case->{expected},
+                    'parse_rel_path_as_volname_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+
+                is(
+                    $got_volname,
+                    $case->{expected}->{volname},
+                    'parse_rel_path_as_volname() returns expected volname',
+                );
+            } else {
+                is($got_volname_parts, undef,
+                    'parse_rel_path_as_volname_parts() returns undef');
+                is($got_volname, undef, 'parse_rel_path_as_volname() returns undef');
+            }
+        };
+    }
+
+    for my $case ($cases_import_special_volname->@*) {
+        subtest $case->{description} => sub () {
+            my ($volname) = $case->{args}->@{qw(volname)};
+
+            my $got_volname = parse_volname_as_parts($volname);
+
+            if (defined($case->{expected})) {
+                eq_or_diff(
+                    $got_volname,
+                    $case->{expected},
+                    'parse_volname_as_parts() returns expected hashref',
+                    { context => 50000 },
+                );
+            } else {
+                is($got_volname, undef, 'parse_volname_as_parts() returns undef');
+            }
+        };
+    }
+
+    return;
+}
+
 my $DEFAULT_STOREID = 'local';
 my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
 my $DEFAULT_SCFG = {
@@ -547,6 +807,26 @@ my $volid_tests = [
                 $volname_tests->@*
         ],
     },
+    {
+        description =>
+            "volid parsers build on volname parsers' behaviors - 'import' vtype file paths (1)",
+        storedid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $cases_import_special_file_path->@*
+        ],
+    },
+    {
+        description =>
+            "volid parsers build on volname parsers' behaviors - 'import' vtype file paths (2)",
+        storedid => $ALT_STOREID,
+        scfg => $ALT_SCFG,
+        cases => [
+            map { format_volname_case_to_volid_case($DEFAULT_STOREID, $DEFAULT_SCFG, $_) }
+                $cases_import_special_file_path->@*
+        ],
+    },
 ];
 
 my sub run_volid_parsing_tests : prototype($) ($volid_tests) {
@@ -606,6 +886,8 @@ my sub main() {
     unified_diff();
 
     run_volname_parsing_tests($volname_tests);
+    run_special_import_volname_parsing_tests();
+
     run_volid_parsing_tests($volid_tests);
 
     done_testing();
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index afddca73..45c0b838 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -845,20 +845,16 @@ sub parse_volname {
         if ($vtype eq 'snippets') {
             return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
         }
-    }
 
-    if ($volname =~
-        m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+\.ova\/${PVE::Storage::OVA_CONTENT_RE_1})$!
-    ) {
-        my $packed_image = $1;
-        my $format = $2;
-        return ('import', $packed_image, undef, undef, undef, undef, "ova+$format");
-    }
+        if ($vtype eq 'import') {
+            my $format = $parts->{ext};
 
-    if ($volname =~
-        m!^import/(${PVE::Storage::SAFE_CHAR_WITH_WHITESPACE_CLASS_RE}+$PVE::Storage::IMPORT_EXT_RE_1)$!
-    ) {
-        return ('import', $1, undef, undef, undef, undef, $2);
+            if (defined(my $content_ext = $parts->{'content-ext'})) {
+                $format .= "+$content_ext";
+            }
+
+            return ($vtype, $volume_path, undef, undef, undef, undef, $format);
+        }
     }
 
     die "unable to parse directory volume name '$volname'\n";
diff --git a/src/test/guest_import_test.pl b/src/test/guest_import_test.pl
index 04eec24d..234013e3 100755
--- a/src/test/guest_import_test.pl
+++ b/src/test/guest_import_test.pl
@@ -357,7 +357,7 @@ my $tests = [
             {
                 volname => 'import/some-import.ova/disk.txt',
                 vmid => 1337,
-                'expect-fail' => qr/unable to parse directory volume name/,
+                'expect-fail' => qr/invalid volname/,
             },
         ],
     },
@@ -532,7 +532,7 @@ my $tests = [
             {
                 volname => 'import/some-🐧-import.ova/disk.qcow2',
                 vmid => 1337,
-                'expect-fail' => qr/unable to parse directory volume name/,
+                'expect-fail' => qr/invalid volname/,
             },
         ],
     },
@@ -619,7 +619,7 @@ my $tests = [
             {
                 volname => 'import/some-import.ova',
                 vmid => 1337,
-                'expect-fail' => qr/only files from 'ova' format can be extracted/,
+                'expect-fail' => qr/cannot determine format of/,
             },
         ],
     },
-- 
2.47.3

[PATCH pve-storage v2 29/50] tree-wide: simplify recently refactored parsing logic

[Max R. Carrara]

PVE::API2::Storage::Status:

In the parsing helper for file transfers, it is not necessary to
handle the parsing of each volume type individually anymore, since we
use `assert_can_transfer_files()` beforehand and also raise an
exception if parsing fails in general.

Therefore, pull out the common parts of the branches for individual
volume types. Retain only the special case for the `import `type where
we do not accept `.ovf` files.

PVE::Storage:

Merge the branches in `path_to_volume_id()` for all volume types,
except `images`, since they all do the same thing now. Strip the
volume type subdirectory from the given file path only for the
`images` volume type.

PVE::Storage::Plugin:

Condense all of the if-clauses as much as possible in the
`parse_volname()` and `get_subdir_files()` subs.

In `parse_volname()`, extend the error message of the final `die` at
the end of the method, stating that the volume type is unhandled.

The guard clause after the `parse_volname_as_parts()` call already
`die`s on parsing failure, so that final `die` should never be hit
unless we introduce a new volume type and happen to forget to handle
it in this method.

In `get_subdir_files(), do not strip the volume type subdirectory from
the beginning of the current path anymore, because that is handled by
the `parse_path_as_volid_parts()` parsing function, which left
`$filename` unused.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 40 ++++--------------
 src/PVE/Storage.pm             | 40 +++++-------------
 src/PVE/Storage/Plugin.pm      | 76 ++++++++++++----------------------
 3 files changed, 43 insertions(+), 113 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index d2451898..2fd23555 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -75,45 +75,19 @@ my sub assert_can_transfer_files : prototype($$$) {
 my sub parse_transferred_file_path_extension : prototype($$) {
     my ($path, $vtype) = @_;
 
-    if ($vtype eq 'iso') {
-        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
+    my $parts = parse_rel_path_as_volname_parts($path, $vtype);
 
-        if (!defined($parts)) {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        my $ext = $parts->{ext};
-        return $ext;
+    if (!defined($parts)) {
+        raise_param_exc({ filename => "invalid filename or wrong file extension" });
     }
 
-    if ($vtype eq 'vztmpl') {
-        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
+    my $ext = $parts->{ext};
 
-        if (!defined($parts)) {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        my $ext = $parts->{ext};
-        return $ext;
+    if ($vtype eq 'import' && $ext eq 'ovf') {
+        raise_param_exc({ filename => "wrong file extension" });
     }
 
-    if ($vtype eq 'import') {
-        my $parts = parse_rel_path_as_volname_parts($path, $vtype);
-
-        if (!defined($parts)) {
-            raise_param_exc({ filename => "invalid filename or wrong extension" });
-        }
-
-        my $ext = $parts->{ext};
-
-        if ($ext eq 'ovf') {
-            raise_param_exc({ filename => "wrong file extension" });
-        }
-
-        return $ext;
-    }
-
-    die "upload / download: failed to parse '$path' - unhandled content type '$vtype'\n";
+    return $ext;
 }
 
 my sub assert_file_transfer_contents_valid : prototype($$$) {
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 98d5f64e..f48b47eb 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -741,16 +741,16 @@ sub path_to_volume_id {
     my $parse_volid_from_file_path = sub {
         my ($plugin, $sid, $scfg, $vtype) = @_;
 
-        my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
-
-        # Strip the vtype subdir from beginning of the current path
-        # so that we don't have to take it into account when parsing the file name
-        my $filename = $path;
-        if ($filename !~ s!^\Q$vtype_subdir\E(?=/)!!) {
-            return;
-        }
-
         if ($vtype eq 'images') {
+            my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
+
+            # Strip the vtype subdir from beginning of the current path
+            # so that we don't have to take it into account when parsing the file name
+            my $filename = $path;
+            if ($filename !~ s!^\Q$vtype_subdir\E(?=/)!!) {
+                return;
+            }
+
             return if $filename !~ m!/(\d+)/([^/\s]+)$!;
             my $vmid = $1;
             my $name = $2;
@@ -767,27 +767,7 @@ sub path_to_volume_id {
             return;
         }
 
-        if ($vtype eq 'iso') {
-            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'vztmpl') {
-            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'backup') {
-            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'snippets') {
-            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        if ($vtype eq 'import') {
-            return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
-        }
-
-        return;
+        return parse_abs_path_as_volid($sid, $scfg, $path, $vtype);
     };
 
     my $vtypes_to_check = $get_vtypes_to_check->();
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 45c0b838..def9e495 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -22,6 +22,7 @@ use PVE::Storage::Common qw(
 use PVE::Storage::Common::Parse qw(
     parse_volname_as_parts
     parse_abs_path_as_volid_parts
+    parse_volid_as_parts
 );
 
 use JSON;
@@ -434,8 +435,8 @@ PVE::JSONSchema::register_format('pve-volume-id', \&parse_volume_id);
 sub parse_volume_id {
     my ($volid, $noerr) = @_;
 
-    if ($volid =~ m/^([a-z][a-z0-9\-\_\.]*[a-z0-9]):(.+)$/i) {
-        return wantarray ? ($1, $2) : $1;
+    if (defined(my $parts = parse_volid_as_parts($volid))) {
+        return wantarray ? ($parts->{storeid}, $parts->{volname}) : $parts->{storeid};
     }
     return undef if $noerr;
     die "unable to parse volume ID '$volid'\n";
@@ -827,37 +828,31 @@ sub parse_volname {
         return ('images', $name, $vmid, undef, undef, $isBase, $format);
     }
 
-    if (defined(my $parts = parse_volname_as_parts($volname))) {
-        my ($vtype, $volume_path) = $parts->@{qw(vtype path)};
+    my $parts = parse_volname_as_parts($volname);
+    die "unable to parse directory volume name '$volname'\n"
+        if !defined($parts);
 
-        if ($vtype eq 'iso') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
+    my ($vtype, $volume_path) = $parts->@{qw(vtype path)};
 
-        if ($vtype eq 'vztmpl') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'backup') {
-            return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'snippets') {
-            return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
-        }
-
-        if ($vtype eq 'import') {
-            my $format = $parts->{ext};
-
-            if (defined(my $content_ext = $parts->{'content-ext'})) {
-                $format .= "+$content_ext";
-            }
-
-            return ($vtype, $volume_path, undef, undef, undef, undef, $format);
-        }
+    if ($vtype eq 'iso' || $vtype eq 'vztmpl' || $vtype eq 'snippets') {
+        return ($vtype, $volume_path, undef, undef, undef, undef, 'raw');
     }
 
-    die "unable to parse directory volume name '$volname'\n";
+    if ($vtype eq 'backup') {
+        return ($vtype, $volume_path, $parts->{vmid}, undef, undef, undef, 'raw');
+    }
+
+    if ($vtype eq 'import') {
+        my $format = $parts->{ext};
+
+        if (defined(my $content_ext = $parts->{'content-ext'})) {
+            $format .= "+$content_ext";
+        }
+
+        return ($vtype, $volume_path, undef, undef, undef, undef, $format);
+    }
+
+    die "unable to parse directory volume name '$volname' - unhandled volume type '$vtype'\n";
 }
 
 # FIXME: remove on the next APIAGE reset.
@@ -1710,17 +1705,10 @@ my sub get_subdir_files {
     my $get_subdir_file_info = sub {
         my ($path, $st) = @_;
 
-        # Strip the vtype subdir from beginning of the current path
-        # so that we don't have to take it into account when parsing the file name
-        my $filename = $path;
-        if ($filename !~ s!^\Q$vtype_subdir\E(?=/)!!) {
-            return;
-        }
+        my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
+        return if !defined($parts);
 
         if ($vtype eq 'iso') {
-            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => 'iso', # always 'iso' even if we have a file ending in .img
@@ -1728,9 +1716,6 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'vztmpl') {
-            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             my ($ext, $ext_compression) = $parts->@{qw(ext ext-compression)};
             my $format = $ext eq 'tar' ? $ext : ('t' . $ext_compression);
 
@@ -1741,9 +1726,6 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'backup') {
-            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             my $format = $parts->{ext};
             my $volume_path = $parts->{path};
 
@@ -1785,9 +1767,6 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'snippets') {
-            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => 'snippet',
@@ -1795,9 +1774,6 @@ my sub get_subdir_files {
         }
 
         if ($vtype eq 'import') {
-            my $parts = parse_abs_path_as_volid_parts($storeid, $scfg, $path, $vtype);
-            return if !defined($parts);
-
             return {
                 volid => $parts->{volid},
                 format => $parts->{ext},
-- 
2.47.3

[PATCH pve-storage v2 30/50] test: list volumes: reorganize and modernize test running code

[Max R. Carrara]

Use the `use v5.36` pragma to enable subroutine signatures,
warnings and strictness by default.

Factor out the list of file suffixes for file formats that support
backing files into its own constant.

Introduce a `main()` subroutine.

Move the code that runs the existing tests for
`::Plugin->list_volumes()` into its own subroutine.

Move the remaining single test that checks whether the vmlist returned
by `PVE::Cluster::get_vmlist()` was modified into `main()`.

Move the call to `Test::More`'s `plan()` into `main()` as well.

Done as a preparational step for other changes.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 66 ++++++++++++++++++++---------------
 1 file changed, 38 insertions(+), 28 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 5cb08880..053ebbe3 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestListVolumes;
 
-use strict;
-use warnings;
+use v5.36;
 
 use lib qw(..);
 
@@ -13,7 +12,7 @@ use Test::More;
 use Test::MockModule;
 
 use Cwd;
-use File::Basename;
+use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
 use File::stat qw();
 use File::Temp;
@@ -76,6 +75,8 @@ my $scfg = {
     },
 };
 
+my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
+
 # The test cases are comprised of an arry of hashes with the following keys:
 # description => displayed on error by Test::More
 # vmid        => used for image matches by list_volume
@@ -667,31 +668,9 @@ my sub cmp_volinfo_by_volid {
     return $a->{volid} cmp $b->{volid};
 }
 
-my $plan = scalar @tests;
-plan tests => $plan + 1;
-
-{
-    # don't accidentally modify vmlist, see bug report
-    # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
-    my $scfg_with_type = { path => $storage_dir, type => 'dir' };
-    my $original_vmlist = { ids => {} };
-    my $tested_vmlist = dclone($original_vmlist);
-
-    PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
-
-    is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
-        || diag(
-            "Expected vmlist to remain\n",
-            explain($original_vmlist),
-            "but it turned to\n",
-            explain($tested_vmlist),
-        );
-}
-
-{
+my sub run_legacy_tests() {
     my $sid = 'local';
     my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
-    my @suffixes = ('qcow2', 'raw', 'vmdk', 'vhdx');
 
     # run through test cases
     foreach my $tt (@tests) {
@@ -704,7 +683,7 @@ plan tests => $plan + 1;
         # prepare environment
         my $num = 0; #parent disks
         for my $file (@$files) {
-            my ($name, $dir, $suffix) = fileparse($file, @suffixes);
+            my ($name, $dir, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);
 
             make_path($dir, { verbose => 1, mode => 0755 });
 
@@ -732,8 +711,39 @@ plan tests => $plan + 1;
         # we get wrong results from leftover files
         remove_tree($storage_dir, { verbose => 1 });
     }
+
+    return;
 }
 
-done_testing();
+sub main() {
+    my $plan = scalar @tests;
+    plan tests => $plan + 1;
+
+    {
+        # don't accidentally modify vmlist, see bug report
+        # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
+        my $scfg_with_type = { path => $storage_dir, type => 'dir' };
+        my $original_vmlist = { ids => {} };
+        my $tested_vmlist = dclone($original_vmlist);
+
+        PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
+
+        is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
+            || diag(
+                "Expected vmlist to remain\n",
+                explain($original_vmlist),
+                "but it turned to\n",
+                explain($tested_vmlist),
+            );
+    }
+
+    run_legacy_tests();
+
+    done_testing();
+
+    return;
+}
+
+main();
 
 1;
-- 
2.47.3

[PATCH pve-storage v2 31/50] test: list volumes: fix broken test checking for vmlist modifications

[Max R. Carrara]

This test never actually did anything, since it didn't use the mocked
vmlist at all. Therefore, fix it by comparing the vmlist with its
original after all tests have run.

Verified that this test catches modifications to the vmlist by
manually adding a line that modifies it in the `list_volumes()`
method, running the tests to see if this fails, and then removing that
line again.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 053ebbe3..2d941b60 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -719,26 +719,19 @@ sub main() {
     my $plan = scalar @tests;
     plan tests => $plan + 1;
 
-    {
-        # don't accidentally modify vmlist, see bug report
-        # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
-        my $scfg_with_type = { path => $storage_dir, type => 'dir' };
-        my $original_vmlist = { ids => {} };
-        my $tested_vmlist = dclone($original_vmlist);
-
-        PVE::Storage::Plugin->list_volumes('sid', $scfg_with_type, undef, ['images']);
-
-        is_deeply($tested_vmlist, $original_vmlist, 'PVE::Cluster::vmlist remains unmodified')
-            || diag(
-                "Expected vmlist to remain\n",
-                explain($original_vmlist),
-                "but it turned to\n",
-                explain($tested_vmlist),
-            );
-    }
+    # Keep the original vmlist around in order to check whether it was modified
+    # after running all the tests. See:
+    # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
+    my $original_vmlist = dclone(PVE::Cluster::get_vmlist());
 
     run_legacy_tests();
 
+    my $vmlist = PVE::Cluster::get_vmlist();
+
+    is_deeply(
+        $vmlist, $original_vmlist, 'Result of PVE::Cluster::get_vmlist remains unmodified',
+    );
+
     done_testing();
 
     return;
-- 
2.47.3

[PATCH pve-storage v2 32/50] test: list volumes: introduce new format for test cases

[Max R. Carrara]

Introduce a new format for test cases in `list_volumes_test.pm`, which
allows specifying the `$storeid`, `$scfg` and the list of
content types / volume types for the given test.

At the same time, get rid of the `files` and `expected` lists in the
new format, which made it somewhat tedious to keep track of which file
belonged to which hashref in the expected output, or if it even was
supposed to show up in the output in the first place.

Get rid of the optional `parent` list for the same reason, too.

Replace these three lists with a single `cases` list that simply
contains hashes that match a given file path with the expected hash in
the final output. The `parent` may optionally be specified in the same
hash for a given file path as well, which makes the backing file logic
a little simpler.

Describe the new format in a small POD docstring.

Additionally, introduce replacements for the constants declared with
the `use constant` pragma.

Finally, migrate the first of the old test cases over to the new
format and run it through the new machinery added for the new test
case format.

The reason for introducing this new format here is to make it easier
to add more specific test cases. The "legacy" tests always pass all
volume types as parameter to `PVE::Storage::Plugin->list_volumes()`,
for example.

As an additional benefit, we could reuse / recycle the new testing
machinery added here for testing other storage (plugin) API methods as
well.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 399 +++++++++++++++++++++++++---------
 1 file changed, 300 insertions(+), 99 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 2d941b60..c83b782e 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -11,6 +11,7 @@ use PVE::Tools qw(run_command);
 use Test::More;
 use Test::MockModule;
 
+use Carp qw(confess);
 use Cwd;
 use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
@@ -22,6 +23,10 @@ use constant DEFAULT_SIZE => 131072; # 128 kiB
 use constant DEFAULT_USED => 262144; # 256 kiB
 use constant DEFAULT_CTIME => 1234567890;
 
+my $DEFAULT_SIZE = 128 * 1024; # 128 kiB
+my $DEFAULT_USED = 256 * 1024; # 256 kiB
+my $DEFAULT_CTIME = 1234567890;
+
 # get_vmlist() return values
 my $mocked_vmlist = {
     'version' => 1,
@@ -75,6 +80,23 @@ my $scfg = {
     },
 };
 
+my $DEFAULT_STOREID = 'local';
+my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
+my $DEFAULT_SCFG = {
+    type => 'dir',
+    path => $DEFAULT_STORAGE_PATH,
+    shared => 0,
+    content => {
+        iso => 1,
+        rootdir => 1,
+        vztmpl => 1,
+        images => 1,
+        snippets => 1,
+        backup => 1,
+        import => 1,
+    },
+};
+
 my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
 
 # The test cases are comprised of an arry of hashes with the following keys:
@@ -84,103 +106,6 @@ my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
 # expected    => returned result hash
 #                (content, ctime, format, parent, size, used, vimd, volid)
 my @tests = (
-    {
-        description => 'VMID: 16110, VM, qcow2, backup, snippets',
-        vmid => '16110',
-        files => [
-            "$storage_dir/images/16110/vm-16110-disk-0.qcow2",
-            "$storage_dir/images/16110/vm-16110-disk-1.raw",
-            "$storage_dir/images/16110/vm-16110-disk-2.vmdk",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
-            "$storage_dir/snippets/userconfig.yaml",
-            "$storage_dir/snippets/hookscript.pl",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'raw',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-1.raw',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'vmdk',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16110',
-                'volid' => 'local:16110/vm-16110-disk-2.vmdk',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602700,
-                'format' => 'vma.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602765,
-                'format' => 'vma.lzo',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602835,
-                'format' => 'vma',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585602835,
-                'format' => 'vma.zst',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'qemu',
-                'vmid' => '16110',
-                'volid' => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/hookscript.pl',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/userconfig.yaml',
-            },
-        ],
-    },
     {
         description => 'VMID: 16112, lxc, raw, backup',
         vmid => '16112',
@@ -631,6 +556,189 @@ my @tests = (
     },
 );
 
+=head2 TEST CASE FORMAT
+
+The parameters for individual test cases are hashes with the following
+keys:
+
+    {
+        # Name of the test, also displayed on error.
+        description => '...',
+
+        # The storage ID for the test.
+        # Used as parameter when calling list_volumes.
+        storeid => 'some-storeid',
+
+        # The storage config hash for the test.
+        # Used as parameter when calling list_volumes.
+        scfg => {
+            type => 'dir',
+            path => '...',
+            shared => 0,
+            content => {
+                iso => 1,
+                # [...]
+            },
+            # [...]
+        },
+
+        # The VMID used as parameter when calling list_volumes.
+        # May be undef.
+        vmid => 1234,
+
+        # The list of volume types / content types used as parameter when
+        # calling list_volumes.
+        vtypes => [],
+
+        # List of hashes that associate a file with its expected hash in the
+        # output of list_volumes.
+        cases => [
+            {
+                # Absolute path to the file.
+                file => "/tmp/foobar/images/1234/vm-1234-disk-0.qcow2",
+
+                # Relative path to parent (backing file).
+                parent => '../ssss/base-4321-disk-0.qcow2',
+
+                # Expected hash in the output of list_volumes.
+                expected => {
+                    content => 'images',
+                    ctime => 1234567890, # inode change time in seconds since the epoc
+                    format => 'qcow2',
+                    parent => '../ssss/base-4321-disk-0.qcow2',
+                    size => 1337, # bytes!
+                    used => 1024, # bytes!
+                    vmid => 1234,
+                    volid => 'local:1234/vm-1234-disk-0.qcow2',
+                },
+            },
+        ],
+    }
+
+=cut
+
+my $test_param_list = [
+    {
+        description => 'VMID: 16110, VM, qcow2, backup, snippets',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16110,
+        vtypes => ['images', 'backup', 'snippets'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-0.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-1.raw",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-1.raw',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-2.vmdk",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16110',
+                    volid => 'local:16110/vm-16110-disk-2.vmdk',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602700,
+                    format => 'vma.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602765,
+                    format => 'vma.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/userconfig.yaml",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/userconfig.yaml',
+                },
+            },
+        ],
+    },
+];
+
 # provide static vmlist for tests
 my $mock_cluster = Test::MockModule->new('PVE::Cluster', no_auto => 1);
 $mock_cluster->redefine(get_vmlist => sub { return $mocked_vmlist; });
@@ -715,15 +823,108 @@ my sub run_legacy_tests() {
     return;
 }
 
+my sub setup_test_env($test_params) {
+    my ($storeid, $scfg) = $test_params->@{qw(storeid scfg)};
+
+    make_path($scfg->{path}, { verbose => 1, mode => 0700 });
+
+    for my $case ($test_params->{cases}->@*) {
+        my ($file, $parent) = $case->@{qw(file parent)};
+
+        confess "\$file = undef" if !defined($file);
+
+        my ($file_name, $directory, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);
+
+        make_path($directory, { verbose => 1, mode => 0755 });
+
+        if ($file_name) {
+            # using qemu-img to also be able to represent the backing device
+            my $command = ['/usr/bin/qemu-img', 'create', "$file", $DEFAULT_SIZE];
+
+            push($command->@*, '-f', $suffix) if $suffix;
+            push($command->@*, '-u', '-b', $parent) if $parent;
+            push($command->@*, '-F', $suffix) if $parent && $suffix;
+
+            run_command($command, timeout => 10);
+        }
+    }
+
+    return;
+}
+
+my sub teardown_test_env($test_params) {
+    my $scfg = $test_params->{scfg};
+
+    remove_tree($scfg->{path}, { verbose => 1 });
+
+    return;
+}
+
+my sub run_test_for_params($test_params) {
+    eval { setup_test_env($test_params); };
+    if ($@) {
+        teardown_test_env($test_params);
+
+        fail($test_params->{description});
+        diag("Failed to set up test environment: $@");
+
+        return;
+    }
+
+    my ($storeid, $scfg, $vmid, $vtypes) = $test_params->@{qw(storeid scfg vmid vtypes)};
+
+    my $expected = [map { $_->{expected} // () } $test_params->{cases}->@*];
+    $expected = [sort cmp_volinfo_by_volid $expected->@*];
+
+    my $got = eval {
+        my $volume_list = PVE::Storage::Plugin->list_volumes($storeid, $scfg, $vmid, $vtypes);
+        return [sort cmp_volinfo_by_volid $volume_list->@*];
+    };
+    $got = $@ if $@;
+
+    my $desc_vmid = defined($vmid) ? $vmid : 'undef';
+    my $desc_vtypes = '[' . join(', ', $vtypes->@*) . ']';
+
+    my $description =
+        $test_params->{description} . " - \$vmid = $desc_vmid, \$vtypes = $desc_vtypes";
+
+    if (!is_deeply($got, $expected, $description)) {
+        diag("\$got = ", explain($got));
+        diag("\$expected = ", explain($expected));
+    }
+
+    teardown_test_env($test_params);
+
+    return;
+}
+
+my sub assert_test_params_keys_exist($test_params) {
+    my $desc = $test_params->{description};
+
+    confess "Test parameters without description"
+        if !$desc;
+
+    for my $key (qw(storeid scfg vmid vtypes cases)) {
+        confess "Key '$key' does not exist in '$desc'"
+            if !exists($test_params->{$key});
+    }
+
+    return;
+}
+
 sub main() {
-    my $plan = scalar @tests;
-    plan tests => $plan + 1;
+    plan tests => scalar(@tests) + scalar($test_param_list->@*) + 1;
 
     # Keep the original vmlist around in order to check whether it was modified
     # after running all the tests. See:
     # https://pve.proxmox.com/pipermail/pve-devel/2020-January/041096.html
     my $original_vmlist = dclone(PVE::Cluster::get_vmlist());
 
+    for my $test_params ($test_param_list->@*) {
+        assert_test_params_keys_exist($test_params);
+        run_test_for_params($test_params);
+    }
+
     run_legacy_tests();
 
     my $vmlist = PVE::Cluster::get_vmlist();
-- 
2.47.3

[PATCH pve-storage v2 33/50] test: list volumes: remove legacy code and migrate cases to new format

[Max R. Carrara]

Remove all that remains of the old testing code, including the
constants defined with the `use constant` pragma and other old
variables.

Migrate all remaining test cases over to the new format.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/list_volumes_test.pm | 1185 ++++++++++++++++++---------------
 1 file changed, 654 insertions(+), 531 deletions(-)

diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index c83b782e..ca3bbac7 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -19,10 +19,6 @@ use File::stat qw();
 use File::Temp;
 use Storable qw(dclone);
 
-use constant DEFAULT_SIZE => 131072; # 128 kiB
-use constant DEFAULT_USED => 262144; # 256 kiB
-use constant DEFAULT_CTIME => 1234567890;
-
 my $DEFAULT_SIZE = 128 * 1024; # 128 kiB
 my $DEFAULT_USED = 256 * 1024; # 256 kiB
 my $DEFAULT_CTIME = 1234567890;
@@ -64,22 +60,6 @@ my $mocked_vmlist = {
     },
 };
 
-my $storage_dir = File::Temp->newdir();
-my $scfg = {
-    'type' => 'dir',
-    'path' => $storage_dir,
-    'shared' => 0,
-    'content' => {
-        'iso' => 1,
-        'rootdir' => 1,
-        'vztmpl' => 1,
-        'images' => 1,
-        'snippets' => 1,
-        'backup' => 1,
-        'import' => 1,
-    },
-};
-
 my $DEFAULT_STOREID = 'local';
 my $DEFAULT_STORAGE_PATH = File::Temp->newdir();
 my $DEFAULT_SCFG = {
@@ -99,463 +79,6 @@ my $DEFAULT_SCFG = {
 
 my @BACKING_FILE_SUFFIXES = ('qcow2', 'raw', 'vmdk', 'vhdx');
 
-# The test cases are comprised of an arry of hashes with the following keys:
-# description => displayed on error by Test::More
-# vmid        => used for image matches by list_volume
-# files       => array of files for qemu-img to create
-# expected    => returned result hash
-#                (content, ctime, format, parent, size, used, vimd, volid)
-my @tests = (
-    {
-        description => 'VMID: 16112, lxc, raw, backup',
-        vmid => '16112',
-        files => [
-            "$storage_dir/images/16112/vm-16112-disk-0.raw",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
-            "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
-        ],
-        expected => [
-            {
-                'content' => 'rootdir',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'raw',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16112',
-                'volid' => 'local:16112/vm-16112-disk-0.raw',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604370,
-                'format' => 'tar.lzo',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604970,
-                'format' => 'tar.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604970,
-                'format' => 'tar.zst',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585605570,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1585604370,
-                'format' => 'tar.bz2',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'openvz',
-                'vmid' => '16112',
-                'volid' => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: 16114, VM, qcow2, linked clone',
-        vmid => '16114',
-        files => [
-            "$storage_dir/images/16114/vm-16114-disk-0.qcow2",
-            "$storage_dir/images/16114/vm-16114-disk-1.qcow2",
-        ],
-        parent => [
-            "../9004/base-9004-disk-0.qcow2", "../9004/base-9004-disk-1.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../9004/base-9004-disk-0.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16114',
-                'volid' => 'local:9004/base-9004-disk-0.qcow2/16114/vm-16114-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../9004/base-9004-disk-1.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '16114',
-                'volid' => 'local:9004/base-9004-disk-1.qcow2/16114/vm-16114-disk-1.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: 9004, VM, template, qcow2',
-        vmid => '9004',
-        files => [
-            "$storage_dir/images/9004/base-9004-disk-0.qcow2",
-            "$storage_dir/images/9004/base-9004-disk-1.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '9004',
-                'volid' => 'local:9004/base-9004-disk-0.qcow2',
-            },
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => undef,
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '9004',
-                'volid' => 'local:9004/base-9004-disk-1.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, templates, snippets, backup',
-        vmid => undef,
-        files => [
-            "$storage_dir/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
-            "$storage_dir/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
-            "$storage_dir/template/iso/archlinux-2020.02.01-x86_64.iso",
-            "$storage_dir/template/iso/debian-8.11.1-amd64-DVD-1.iso",
-            "$storage_dir/template/iso/debian-9.12.0-amd64-netinst.iso",
-            "$storage_dir/template/iso/proxmox-ve_6.1-1.iso",
-            "$storage_dir/template/cache/archlinux-base_20190924-1_amd64.tar.gz",
-            "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
-            "$storage_dir/template/cache/debian-11.0-standard_11.0-1_amd64.tar.bz2",
-            "$storage_dir/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
-            "$storage_dir/snippets/userconfig.yaml",
-            "$storage_dir/snippets/hookscript.pl",
-            "$storage_dir/private/1234/", # fileparse needs / at the end
-            "$storage_dir/private/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
-        ],
-        expected => [
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'txz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tgz',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
-            },
-            {
-                'content' => 'vztmpl',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'tbz2',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/archlinux-2020.02.01-x86_64.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/debian-8.11.1-amd64-DVD-1.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/debian-9.12.0-amd64-netinst.iso',
-            },
-            {
-                'content' => 'iso',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'iso',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:iso/proxmox-ve_6.1-1.iso',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1580759863,
-                'format' => 'tar.gz',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '19253',
-                'volid' => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
-            },
-            {
-                'content' => 'backup',
-                'ctime' => 1548098959,
-                'format' => 'tar',
-                'size' => DEFAULT_SIZE,
-                'subtype' => 'lxc',
-                'vmid' => '19254',
-                'volid' => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/hookscript.pl',
-            },
-            {
-                'content' => 'snippets',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'snippet',
-                'size' => DEFAULT_SIZE,
-                'volid' => 'local:snippets/userconfig.yaml',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, parent, non-matching',
-        # string instead of vmid in folder
-        #"$storage_dir/images/ssss/base-4321-disk-0.qcow2/1234/vm-1234-disk-0.qcow2",
-        vmid => undef,
-        files => [
-            "$storage_dir/images/1234/vm-1234-disk-0.qcow2",
-        ],
-        parent => [
-            "../ssss/base-4321-disk-0.qcow2",
-        ],
-        expected => [
-            {
-                'content' => 'images',
-                'ctime' => DEFAULT_CTIME,
-                'format' => 'qcow2',
-                'parent' => '../ssss/base-4321-disk-0.qcow2',
-                'size' => DEFAULT_SIZE,
-                'used' => DEFAULT_USED,
-                'vmid' => '1234',
-                'volid' => 'local:1234/vm-1234-disk-0.qcow2',
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, non-matching',
-        # failed matches
-        vmid => undef,
-        files => [
-            "$storage_dir/images/ssss/base-4321-disk-0.raw",
-            "$storage_dir/images/ssss/vm-1234-disk-0.qcow2",
-            "$storage_dir/template/iso/yet-again-a-installation-disk.dvd",
-            "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
-            "$storage_dir/private/subvol-19254-disk-0/19254",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
-            "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
-            "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
-        ],
-        expected => [], # returns empty list
-    },
-    {
-        description => 'VMID: none, valid file names for import',
-        vmid => undef,
-        files => [
-            "$storage_dir/import/import.ova",
-            "$storage_dir/import/import.ovf",
-            "$storage_dir/import/some-disk.qcow2",
-            "$storage_dir/import/some-disk.vmdk",
-            "$storage_dir/import/some-raw-disk.raw",
-        ],
-        expected => [
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ovf',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ovf",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-disk.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'vmdk',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-disk.vmdk",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'raw',
-                size => DEFAULT_SIZE,
-                volid => "local:import/some-raw-disk.raw",
-            },
-        ],
-    },
-    {
-        description => 'VMID: none, non-matching file paths for import',
-        vmid => undef,
-        files => [
-            # Malformed file names
-            "$storage_dir/import/import.ovff",
-            "$storage_dir/import/importova",
-            "$storage_dir/import/import.ov",
-            "$storage_dir/import/diskraw",
-            "$storage_dir/import/diskvmdk",
-            "$storage_dir/import/disk.invalid",
-            "$storage_dir/import/.ova",
-            "$storage_dir/import/.raw",
-            # Trailing whitespace must not be trimmed
-            "$storage_dir/import/import.ova\t",
-            "$storage_dir/import/disk.raw    ",
-            # Whitespace in file name
-            "$storage_dir/import/something I want to import.ova",
-            "$storage_dir/import/ .raw",
-            "$storage_dir/import/ disk .vmdk",
-            "$storage_dir/import/disk .qcow2",
-            "$storage_dir/import/ import.ova",
-            # Unsafe characters in file name
-            "$storage_dir/import/linux🐧-vm.ova",
-            "$storage_dir/import/🐪perl-playground🐪.ova",
-            "$storage_dir/import/fish_<><_<><_<><.ova",
-            $storage_dir . '/import/C:\\\\Windows\\Path.ova',
-            # Content inside .ova files may only be specified as part
-            # of volume names, and may never appear when looked up as
-            # a file path
-            "$storage_dir/import/import.ova/disk.qcow2",
-            "$storage_dir/import/import.ova/disk.raw",
-            "$storage_dir/import/import.ova/disk.vmdk",
-            "$storage_dir/import/import.ova/disk.invalid",
-        ],
-        expected => [], # returns empty list
-    },
-    {
-        description => 'VMID: none, weird but valid file names for import',
-        vmid => undef,
-        files => [
-            "$storage_dir/import/import.ova.ova",
-            "$storage_dir/import/import.ova.ova.ova",
-            "$storage_dir/import/import.ova.ova.ova.ova",
-            "$storage_dir/import/ova.ova",
-            "$storage_dir/import/ova.ovf",
-            "$storage_dir/import/ova.vmdk",
-            "$storage_dir/import/raw.raw.qcow2",
-            "$storage_dir/import/raw.raw.qcow2.import.qcow2",
-            "$storage_dir/import/raw.raw.raw.your-boat.ova",
-        ],
-        expected => [
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/import.ova.ova.ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.ova",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ovf',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.ovf",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'vmdk',
-                size => DEFAULT_SIZE,
-                volid => "local:import/ova.vmdk",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'qcow2',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.qcow2.import.qcow2",
-            },
-            {
-                content => 'import',
-                ctime => DEFAULT_CTIME,
-                format => 'ova',
-                size => DEFAULT_SIZE,
-                volid => "local:import/raw.raw.raw.your-boat.ova",
-            },
-        ],
-    },
-);
-
 =head2 TEST CASE FORMAT
 
 The parameters for individual test cases are hashes with the following
@@ -737,6 +260,655 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => 'VMID: 16112, lxc, raw, backup',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16112,
+        vtypes => ['rootdir', 'backup'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16112/vm-16112-disk-0.raw",
+                expected => {
+                    content => 'rootdir',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16112',
+                    volid => 'local:16112/vm-16112-disk-0.raw',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 16114, VM, qcow2, linked clone',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16114,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16114/vm-16114-disk-0.qcow2",
+                parent => '../9004/base-9004-disk-0.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../9004/base-9004-disk-0.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16114',
+                    volid => 'local:9004/base-9004-disk-0.qcow2/16114/vm-16114-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16114/vm-16114-disk-1.qcow2",
+                parent => '../9004/base-9004-disk-1.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../9004/base-9004-disk-1.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '16114',
+                    volid => 'local:9004/base-9004-disk-1.qcow2/16114/vm-16114-disk-1.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 9004, VM, template, qcow2',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 9004,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/9004/base-9004-disk-0.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '9004',
+                    volid => 'local:9004/base-9004-disk-0.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/9004/base-9004-disk-1.qcow2",
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => undef,
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '9004',
+                    volid => 'local:9004/base-9004-disk-1.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, templates, snippets, backup',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['vztmpl', 'iso', 'backup', 'snippets'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'txz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/alpine-3.10-default_20190626_amd64.tar.xz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/archlinux-base_20190924-1_amd64.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/archlinux-base_20190924-1_amd64.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-11.0-standard_11.0-1_amd64.tar.bz2",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tbz2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/archlinux-2020.02.01-x86_64.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/archlinux-2020.02.01-x86_64.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/debian-8.11.1-amd64-DVD-1.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/debian-8.11.1-amd64-DVD-1.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/debian-9.12.0-amd64-netinst.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/debian-9.12.0-amd64-netinst.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/proxmox-ve_6.1-1.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/proxmox-ve_6.1-1.iso',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => {
+                    content => 'backup',
+                    ctime => 1548098959,
+                    format => 'tar',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19254',
+                    volid => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/userconfig.yaml",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/userconfig.yaml',
+                },
+            },
+            {
+                # fileparse needs / at the end
+                file => "$DEFAULT_STORAGE_PATH/private/1234/",
+                expected => undef,
+            },
+            {
+                # fileparse needs / at the end
+                file => "$DEFAULT_STORAGE_PATH/private/1234/subvol-1234-disk-0.subvol/",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, parent, non-matching',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['images'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/1234/vm-1234-disk-0.qcow2",
+                parent => '../ssss/base-4321-disk-0.qcow2',
+                expected => {
+                    content => 'images',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    parent => '../ssss/base-4321-disk-0.qcow2',
+                    size => $DEFAULT_SIZE,
+                    used => $DEFAULT_USED,
+                    vmid => '1234',
+                    volid => 'local:1234/vm-1234-disk-0.qcow2',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => [sort keys $DEFAULT_SCFG->{content}->%*],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/ssss/base-4321-disk-0.raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/ssss/vm-1234-disk-0.qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/yet-again-a-installation-disk.dvd",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/private/subvol-19254-disk-0/19254",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, valid file names for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ovf",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ovf',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ovf',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-disk.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-disk.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-disk.vmdk",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-disk.vmdk',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-raw-disk.raw",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'raw',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-raw-disk.raw',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, non-matching file paths for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            # Malformed file names
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ovff",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/importova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ov",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/diskraw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/diskvmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk.invalid",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/.raw",
+                expected => undef,
+            },
+
+            # Trailing whitespace must not be trimmed
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova\t",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk.raw    ",
+                expected => undef,
+            },
+
+            # Whitespace in file name
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/something I want to import.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ .raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ disk .vmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/disk .qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ import.ova",
+                expected => undef,
+            },
+
+            # Unsafe characters in file name
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/linux🐧-vm.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/🐪perl-playground🐪.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/fish_<><_<><_<><.ova",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/C:\\\\Windows\\Path.ova",
+                expected => undef,
+            },
+
+            # Content inside .ova files may only be specified as part
+            # of volume names, and may never appear when looked up as
+            # a file path
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.raw",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.vmdk",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova/disk.invalid",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => 'VMID: none, weird but valid file names for import',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova.ova.ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/import.ova.ova.ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.ova',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.ovf",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ovf',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.ovf',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/ova.vmdk",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'vmdk',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/ova.vmdk',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.qcow2.import.qcow2",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'qcow2',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.qcow2.import.qcow2',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/raw.raw.raw.your-boat.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/raw.raw.raw.your-boat.ova',
+                },
+            },
+        ],
+    },
 ];
 
 # provide static vmlist for tests
@@ -749,8 +921,8 @@ my $mock_stat = Test::MockModule->new('File::stat', no_auto => 1);
 $mock_stat->redefine(
     populate => sub {
         my (@st) = @_;
-        $st[7] = DEFAULT_SIZE;
-        $st[10] = DEFAULT_CTIME;
+        $st[7] = $DEFAULT_SIZE;
+        $st[10] = $DEFAULT_CTIME;
 
         my $result = $mock_stat->original('populate')->(@st);
 
@@ -765,8 +937,8 @@ $mock_fsi->redefine(
         my ($size, $format, $used, $parent, $ctime) =
             $mock_fsi->original('file_size_info')->(@_);
 
-        $size = DEFAULT_SIZE;
-        $used = DEFAULT_USED;
+        $size = $DEFAULT_SIZE;
+        $used = $DEFAULT_USED;
 
         return wantarray ? ($size, $format, $used, $parent, $ctime) : $size;
     },
@@ -776,53 +948,6 @@ my sub cmp_volinfo_by_volid {
     return $a->{volid} cmp $b->{volid};
 }
 
-my sub run_legacy_tests() {
-    my $sid = 'local';
-    my $types = [grep { $scfg->{content}->{$_} } keys $scfg->{content}->%*];
-
-    # run through test cases
-    foreach my $tt (@tests) {
-        my $vmid = $tt->{vmid};
-        my $files = $tt->{files};
-        my $expected = [sort cmp_volinfo_by_volid $tt->{expected}->@*];
-        my $description = $tt->{description};
-        my $parent = $tt->{parent};
-
-        # prepare environment
-        my $num = 0; #parent disks
-        for my $file (@$files) {
-            my ($name, $dir, $suffix) = fileparse($file, @BACKING_FILE_SUFFIXES);
-
-            make_path($dir, { verbose => 1, mode => 0755 });
-
-            if ($name) {
-                # using qemu-img to also be able to represent the backing device
-                my @cmd = ('/usr/bin/qemu-img', 'create', "$file", DEFAULT_SIZE);
-                push @cmd, ('-f', $suffix) if $suffix;
-                push @cmd, ('-u', '-b', @$parent[$num]) if $parent;
-                push @cmd, ('-F', $suffix) if $parent && $suffix;
-                $num++;
-
-                run_command([@cmd]);
-            }
-        }
-
-        my $got = eval {
-            my $volume_list = PVE::Storage::Plugin->list_volumes($sid, $scfg, $vmid, $types);
-            return [sort cmp_volinfo_by_volid $volume_list->@*];
-        };
-        $got = $@ if $@;
-
-        is_deeply($got, $expected, $description) || diag(explain($got));
-
-        # clean up after each test case, otherwise
-        # we get wrong results from leftover files
-        remove_tree($storage_dir, { verbose => 1 });
-    }
-
-    return;
-}
-
 my sub setup_test_env($test_params) {
     my ($storeid, $scfg) = $test_params->@{qw(storeid scfg)};
 
@@ -913,7 +1038,7 @@ my sub assert_test_params_keys_exist($test_params) {
 }
 
 sub main() {
-    plan tests => scalar(@tests) + scalar($test_param_list->@*) + 1;
+    plan tests => scalar($test_param_list->@*) + 1;
 
     # Keep the original vmlist around in order to check whether it was modified
     # after running all the tests. See:
@@ -925,8 +1050,6 @@ sub main() {
         run_test_for_params($test_params);
     }
 
-    run_legacy_tests();
-
     my $vmlist = PVE::Cluster::get_vmlist();
 
     is_deeply(
-- 
2.47.3

[PATCH pve-storage v2 34/50] plugin: do not return volumes of undeclared content types anymore

[Max R. Carrara]

Currently, any content types / volume types that are passed to
`PVE::Storage::Plugin->list_volumes()` are still taken into account by
the method, even if the passed storage config does not declare them in
its `content` key. In other words, the `content` property is ignored
when listing volumes through `list_volumes()`.

Or to express this more explicitly, consider a directory storage with
`path` set to `/mnt/example` and `content` set to `iso,vztmpl`. Now,
if there are files in `/mnt/example/snippets` for some reason and the
`list_volumes()` method is called with `['snippets']` for the volume
type list parameter, then the volume info hashes for the files in
`/mnt/example/snippets` are included in the output.

There is not really any use case for this behavior, and to my
knowledge, it is also not something that we rely on.

Therefore, instead of returning volumes for undeclared content types,
return an empty list for that type. Document this new behavior through
a test case.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm     |  4 +++
 src/test/list_volumes_test.pm | 50 +++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index def9e495..0047bc5b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1810,6 +1810,10 @@ sub list_volumes {
     my $get_raw_volumes_for_type = sub {
         my ($type) = @_;
 
+        if (!$scfg->{content}->{$type}) {
+            return [];
+        }
+
         if ($type eq 'images' || $type eq 'rootdir') {
             return $class->list_images($storeid, $scfg, $vmid);
         }
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index ca3bbac7..1b9be6c5 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -909,6 +909,56 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description =>
+            "VMID: none, volume info is still returned for content types that are not declared",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            content => {}, # note how no content types are declared here
+        },
+        vmid => undef,
+        vtypes => ['images', 'rootdir', 'vztmpl', 'iso', 'backup', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16110/vm-16110-disk-0.qcow2",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/1234/vm-1234-disk-0.qcow2",
+                parent => '../ssss/base-4321-disk-0.qcow2',
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/images/16112/vm-16112-disk-0.raw",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/alpine-3.10-default_20190626_amd64.tar.xz",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/archlinux-2020.02.01-x86_64.iso",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/hookscript.pl",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/import.ova",
+                expected => undef,
+            },
+        ],
+    },
 ];
 
 # provide static vmlist for tests
-- 
2.47.3

[PATCH pve-storage v2 35/50] plugin: correct comment in get_subdir_files helper

[Max R. Carrara]

The comment in the branch for 'backup' volume types in the
`get_subdir_files()` helper states that the condition below checks for
"false positives", apparently meaning that the VMID in the parent
directory might have been matched.

Challenge this claim by adding several new test cases that try to
cause the parser to return a wrong result.

As it turns out, the check the comment refers to is not (anymore)
there to check for "false positives" or VMIDs appearing in the parent
directory, but instead simply filters out backups that do not belong
to the provided VMID.

Note that if a backup has an arbitrary file name, that is, it's not
named something like "vzdump-qemu-1337-$TIMESTAMP.vma", that backup is
still returned when *no* `$vmid` is given. If it is given however, it
has to match the parsed `$vmid`.

Therefore, also include test cases for a plain "some-backup.tar.gz"
file that account for these things.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Plugin.pm     |   4 +-
 src/test/list_volumes_test.pm | 457 ++++++++++++++++++++++++++++++++++
 2 files changed, 459 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 0047bc5b..e2f24480 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1729,8 +1729,8 @@ my sub get_subdir_files {
             my $format = $parts->{ext};
             my $volume_path = $parts->{path};
 
-            # Check if parsed VMID matched provided VMID in order to avoid
-            # false positives (VMID in parent directory name)
+            # Check if parsed VMID matches provided VMID in order to avoid
+            # returning backups of other guests
             my $parsed_vmid = $parts->{vmid};
             if (defined($vmid)) {
                 if (defined($parsed_vmid)) {
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 1b9be6c5..53ce5116 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -566,6 +566,275 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => 'VMID: none, backups of all guests',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602700,
+                    format => 'vma.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602765,
+                    format => 'vma.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => {
+                    content => 'backup',
+                    ctime => 1548098959,
+                    format => 'tar',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19254',
+                    volid => 'local:backup/vzdump-lxc-19254-2019_01_21-19_29_19.tar',
+                },
+            },
+            # Arbitrary backups are only included if no $vmid is provided.
+            # Note that in this case, the 'vmid' key does not exist at all,
+            # instead of being set to undef.
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/some-backup.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'unknown',
+                    volid => 'local:backup/some-backup.tar.gz',
+                },
+            },
+        ],
+    },
+    {
+        description => 'VMID: 16112, backups of specific guest',
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => 16112,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.lzo',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604970,
+                    format => 'tar.zst',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_49_30.tar.zst',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-16112-2020_03_30-21_59_30.tgz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585605570,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-lxc-16112-2020_03_30-21_59_30.tgz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585604370,
+                    format => 'tar.bz2',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'openvz',
+                    vmid => '16112',
+                    volid => 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/vzdump-lxc-19254-2019_01_21-19_29_19.tar",
+                expected => undef,
+            },
+            # Arbitrary backups are only included if no $vmid is provided.
+            {
+                file => "$DEFAULT_STORAGE_PATH/dump/some-backup.tar.gz",
+                expected => undef,
+            },
+        ],
+    },
     {
         description => 'VMID: none, parent, non-matching',
         storeid => $DEFAULT_STOREID,
@@ -961,6 +1230,194 @@ my $test_param_list = [
     },
 ];
 
+# Additional test cases that cannot be constructed within the list above
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in path of storage",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/dump/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir();
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in vtype subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description => "VMID: 16110, file name in storage path and subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 16110,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/$file_name",
+                expected => {
+                    content => 'backup',
+                    ctime => 1585602835,
+                    format => 'vma',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'qemu',
+                    vmid => '16110',
+                    volid => "local:backup/$file_name",
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description =>
+            "VMID: 19253, file name with different VMID in storage path and subdir of 'backup' vtype",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 19253,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/$file_name/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz",
+                expected => {
+                    content => 'backup',
+                    ctime => 1580759863,
+                    format => 'tar.gz',
+                    size => $DEFAULT_SIZE,
+                    subtype => 'lxc',
+                    vmid => '19253',
+                    volid => 'local:backup/vzdump-lxc-19253-2020_02_03-19_57_43.tar.gz',
+                },
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
+{
+    my $file_name = "vzdump-qemu-16110-2020_03_30-21_13_55.vma";
+    my $storage_path = File::Temp->newdir() . '/' . $file_name;
+
+    my $backup_vmid_test_params = {
+        description =>
+            "VMID: none, file name in storage path and subdir of 'backup' vtype, no backups",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $storage_path,
+            shared => 0,
+            content => {
+                backup => 1,
+                snippets => 1,
+            },
+            'content-dirs' => {
+                backup => $file_name,
+            },
+        },
+        vmid => 19253,
+        vtypes => ['backup'],
+        cases => [
+            {
+                file => "$storage_path/snippets/hookscript.pl",
+                expected => undef,
+            },
+        ],
+    };
+
+    push($test_param_list->@*, $backup_vmid_test_params);
+}
+
 # provide static vmlist for tests
 my $mock_cluster = Test::MockModule->new('PVE::Cluster', no_auto => 1);
 $mock_cluster->redefine(get_vmlist => sub { return $mocked_vmlist; });
-- 
2.47.3

[PATCH pve-storage v2 36/50] test: parse volname: modernize code

[Max R. Carrara]

Modernize the existing code in `parse_volname_test.pm` while leaving
the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- fix a handful of typos
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 79 ++++++++++++++++++----------------
 1 file changed, 43 insertions(+), 36 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 2ff4c6d0..a03ac163 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestParseVolname;
 
-use strict;
-use warnings;
+use v5.36;
 
 use lib qw(..);
 
@@ -147,24 +146,24 @@ my $tests = [
         expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.qcow2',
         expected =>
             ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.vmdk',
         expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
     {
-        description => "Import, innner file of ova with whitespace in name",
+        description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
     {
-        description => "Import, innner file of ova",
+        description => "Import, inner file of ova",
         volname => 'import/import.ova/disk.raw',
         expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
     },
@@ -214,7 +213,7 @@ my $tests = [
 
 # create more test cases for VM disk images matches
 my $disk_suffix = ['raw', 'qcow2', 'vmdk'];
-foreach my $s (@$disk_suffix) {
+for my $s ($disk_suffix->@*) {
     my @arr = (
         {
             description => "VM disk image, $s",
@@ -245,7 +244,7 @@ foreach my $s (@$disk_suffix) {
         },
     );
 
-    push @$tests, @arr;
+    push($tests->@*, @arr);
 }
 
 # create more test cases for backup files matches
@@ -255,9 +254,9 @@ my $bkp_suffix = {
     openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
 };
 
-foreach my $virt (keys %$bkp_suffix) {
+for my $virt (keys $bkp_suffix->%*) {
     my $suffix = $bkp_suffix->{$virt};
-    foreach my $s (@$suffix) {
+    for my $s ($suffix->@*) {
         my @arr = (
             {
                 description => "Backup archive, $virt, $s",
@@ -274,7 +273,7 @@ foreach my $virt (keys %$bkp_suffix) {
             },
         );
 
-        push @$tests, @arr;
+        push($tests->@*, @arr);
     }
 }
 
@@ -283,9 +282,9 @@ my $non_bkp_suffix = {
     qemu => ['vms.gz', 'vma.xz'],
     lxc => ['zip.gz', 'tgz.lzo'],
 };
-foreach my $virt (keys %$non_bkp_suffix) {
+for my $virt (keys $non_bkp_suffix->%*) {
     my $suffix = $non_bkp_suffix->{$virt};
-    foreach my $s (@$suffix) {
+    for my $s ($suffix->@*) {
         my @arr = (
             {
                 description => "Failed match: Backup archive, $virt, $s",
@@ -295,39 +294,47 @@ foreach my $virt (keys %$non_bkp_suffix) {
             },
         );
 
-        push @$tests, @arr;
+        push($tests->@*, @arr);
     }
 }
 
-#
-# run through test case array
-#
-plan tests => scalar @$tests + 1;
+my sub run_tests($tests) {
+    my $seen_vtype = {};
+    my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };
 
-my $seen_vtype;
-my $vtype_subdirs =
-    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
+    for my $t ($tests->@*) {
+        my $description = $t->{description};
+        my $volname = $t->{volname};
+        my $expected = $t->{expected};
 
-foreach my $t (@$tests) {
-    my $description = $t->{description};
-    my $volname = $t->{volname};
-    my $expected = $t->{expected};
+        my $got;
+        eval { $got = [PVE::Storage::Plugin->parse_volname($volname)] };
+        $got = $@ if $@;
 
-    my $got;
-    eval { $got = [PVE::Storage::Plugin->parse_volname($volname)] };
-    $got = $@ if $@;
+        is_deeply($got, $expected, $description);
 
-    is_deeply($got, $expected, $description);
+        $seen_vtype->{ $expected->[0] } = 1 if ref $expected eq 'ARRAY';
+    }
 
-    $seen_vtype->{ @$expected[0] } = 1 if ref $expected eq 'ARRAY';
+    # to check if all $vtype_subdirs are defined in path_to_volume_id
+    # or have a test
+    # FIXME re-enable after vtype split changes
+    #is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
+    is_deeply({}, {}, "vtype_subdir check");
+
+    return;
 }
 
-# to check if all $vtype_subdirs are defined in path_to_volume_id
-# or have a test
-# FIXME re-enable after vtype split changes
-#is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
-is_deeply({}, {}, "vtype_subdir check");
+my sub main() {
+    plan tests => scalar($tests->@*) + 1;
 
-done_testing();
+    run_tests($tests);
+
+    done_testing();
+
+    return;
+}
+
+main();
 
 1;
-- 
2.47.3

[PATCH pve-storage v2 37/50] test: parse volname: adapt tests regarding 'import' volume type

[Max R. Carrara]

Give the existing test cases unique 'description' keys and also
reorder them.

Add additional test cases for qcow2, vmdk and raw files.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index a03ac163..ea07c56b 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -146,27 +146,42 @@ my $tests = [
         expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
     },
     {
-        description => "Import, inner file of ova",
+        description => "Import, qcow2",
+        volname => 'import/import.qcow2',
+        expected => ['import', 'import.qcow2', undef, undef, undef, undef, 'qcow2'],
+    },
+    {
+        description => "Import, vmdk",
+        volname => 'import/import.vmdk',
+        expected => ['import', 'import.vmdk', undef, undef, undef, undef, 'vmdk'],
+    },
+    {
+        description => "Import, raw",
+        volname => 'import/import.raw',
+        expected => ['import', 'import.raw', undef, undef, undef, undef, 'raw'],
+    },
+    {
+        description => "Import, inner file of ova (qcow2)",
         volname => 'import/import.ova/disk.qcow2',
         expected =>
             ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
     },
     {
-        description => "Import, inner file of ova",
+        description => "Import, inner file of ova (vmdk)",
         volname => 'import/import.ova/disk.vmdk',
         expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
+    {
+        description => "Import, inner file of ova (raw)",
+        volname => 'import/import.ova/disk.raw',
+        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
+    },
     {
         description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
-    {
-        description => "Import, inner file of ova",
-        volname => 'import/import.ova/disk.raw',
-        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
-    },
     #
     # failed matches
     #
-- 
2.47.3

[PATCH pve-storage v2 38/50] test: parse volname: move VM disk test creation into separate block

[Max R. Carrara]

For future organizational purposes, move the loop that adds more
test cases for VM images into its own block.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 75 +++++++++++++++++++---------------
 1 file changed, 42 insertions(+), 33 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index ea07c56b..3777e6e8 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -226,40 +226,49 @@ my $tests = [
     },
 ];
 
-# create more test cases for VM disk images matches
-my $disk_suffix = ['raw', 'qcow2', 'vmdk'];
-for my $s ($disk_suffix->@*) {
-    my @arr = (
-        {
-            description => "VM disk image, $s",
-            volname => "$vmid/vm-$vmid-disk-1.$s",
-            expected => [
-                'images', "vm-$vmid-disk-1.$s", "$vmid", undef, undef, undef, "$s",
-            ],
-        },
-        {
-            description => "VM disk image, linked, $s",
-            volname => "$vmid/base-$vmid-disk-0.$s/$vmid/vm-$vmid-disk-0.$s",
-            expected => [
-                'images',
-                "vm-$vmid-disk-0.$s",
-                "$vmid",
-                "base-$vmid-disk-0.$s",
-                "$vmid",
-                undef,
-                "$s",
-            ],
-        },
-        {
-            description => "VM disk image, base, $s",
-            volname => "$vmid/base-$vmid-disk-0.$s",
-            expected => [
-                'images', "base-$vmid-disk-0.$s", "$vmid", undef, undef, 'base-', "$s",
-            ],
-        },
-    );
+# Additional test cases for VM disk images
+{
+    my $disk_suffixes = ['raw', 'qcow2', 'vmdk'];
 
-    push($tests->@*, @arr);
+    for my $suffix ($disk_suffixes->@*) {
+        my @extra_tests = (
+            {
+                description => "VM disk image, $suffix",
+                volname => "$vmid/vm-$vmid-disk-1.$suffix",
+                expected => [
+                    'images', "vm-$vmid-disk-1.$suffix", "$vmid", undef, undef, undef, $suffix,
+                ],
+            },
+            {
+                description => "VM disk image, linked, $suffix",
+                volname => "$vmid/base-$vmid-disk-0.$suffix/$vmid/vm-$vmid-disk-0.$suffix",
+                expected => [
+                    'images',
+                    "vm-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    "base-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    undef,
+                    $suffix,
+                ],
+            },
+            {
+                description => "VM disk image, base, $suffix",
+                volname => "$vmid/base-$vmid-disk-0.$suffix",
+                expected => [
+                    'images',
+                    "base-$vmid-disk-0.$suffix",
+                    "$vmid",
+                    undef,
+                    undef,
+                    'base-',
+                    $suffix,
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
 }
 
 # create more test cases for backup files matches
-- 
2.47.3

[PATCH pve-storage v2 39/50] test: parse volname: move backup file test creation into sep. block

[Max R. Carrara]

Similar to the previous commit, move the code that adds additional
tests for backup file volnames into a separate block, for future
organizational purposes.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 88 +++++++++++++++++-----------------
 1 file changed, 45 insertions(+), 43 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 3777e6e8..fb300ad5 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -271,54 +271,56 @@ my $tests = [
     }
 }
 
-# create more test cases for backup files matches
-my $bkp_suffix = {
-    qemu => ['vma', 'vma.gz', 'vma.lzo', 'vma.zst'],
-    lxc => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst', 'tar.bz2'],
-    openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
-};
+# Additional tests for backup files
+{
+    my $bkp_suffixes = {
+        qemu => ['vma', 'vma.gz', 'vma.lzo', 'vma.zst'],
+        lxc => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst', 'tar.bz2'],
+        openvz => ['tar', 'tgz', 'tar.gz', 'tar.lzo', 'tar.zst'],
+    };
 
-for my $virt (keys $bkp_suffix->%*) {
-    my $suffix = $bkp_suffix->{$virt};
-    for my $s ($suffix->@*) {
-        my @arr = (
-            {
-                description => "Backup archive, $virt, $s",
-                volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                expected => [
-                    'backup',
-                    "vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                    "$vmid",
-                    undef,
-                    undef,
-                    undef,
-                    'raw',
-                ],
-            },
-        );
+    for my $virt (keys $bkp_suffixes->%*) {
+        my $suffixes = $bkp_suffixes->{$virt};
+        for my $suffix ($suffixes->@*) {
+            my @extra_tests = (
+                {
+                    description => "Backup archive, $virt, $suffix",
+                    volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                    expected => [
+                        'backup',
+                        "vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                        "$vmid",
+                        undef,
+                        undef,
+                        undef,
+                        'raw',
+                    ],
+                },
+            );
 
-        push($tests->@*, @arr);
+            push($tests->@*, @extra_tests);
+        }
     }
-}
 
-# create more test cases for failed backup files matches
-my $non_bkp_suffix = {
-    qemu => ['vms.gz', 'vma.xz'],
-    lxc => ['zip.gz', 'tgz.lzo'],
-};
-for my $virt (keys $non_bkp_suffix->%*) {
-    my $suffix = $non_bkp_suffix->{$virt};
-    for my $s ($suffix->@*) {
-        my @arr = (
-            {
-                description => "Failed match: Backup archive, $virt, $s",
-                volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s",
-                expected =>
-                    "unable to parse directory volume name 'backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$s'\n",
-            },
-        );
+    # Failed tests
+    my $non_bkp_suffixes = {
+        qemu => ['vms.gz', 'vma.xz'],
+        lxc => ['zip.gz', 'tgz.lzo'],
+    };
+    for my $virt (keys $non_bkp_suffixes->%*) {
+        my $suffixes = $non_bkp_suffixes->{$virt};
+        for my $suffix ($suffixes->@*) {
+            my @extra_tests = (
+                {
+                    description => "Failed match: Backup archive, $virt, $suffix",
+                    volname => "backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix",
+                    expected =>
+                        "unable to parse directory volume name 'backup/vzdump-$virt-$vmid-2020_03_30-21_12_40.$suffix'\n",
+                },
+            );
 
-        push($tests->@*, @arr);
+            push($tests->@*, @extra_tests);
+        }
     }
 }
 
-- 
2.47.3

[PATCH pve-storage v2 40/50] test: parse volname: parameterize test case creation for some vtypes

[Max R. Carrara]

Currently, a lot of test cases are simply defined as part of the
overall list of tests. While this is completely fine in itself, it
does not scale if one wants to add cases that account for additional
parameters in the future.

Therefore, create a lot of existing test cases in blocks and push them
onto the list of test cases, instead of defining them up front.

Note that we already do this for test cases regarding VM disk / backup
file volume names.

This is done as a preparational step for future commits, primarily in
order to avoid writing a lot of test cases by hand.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/parse_volname_test.pm | 226 ++++++++++++++++-----------------
 1 file changed, 108 insertions(+), 118 deletions(-)

diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index fb300ad5..1dcec9d6 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -34,70 +34,6 @@ my $tests = [
         ],
     },
     #
-    # iso
-    #
-    {
-        description => 'ISO image, iso',
-        volname => 'iso/some-installation-disk.iso',
-        expected => ['iso', 'some-installation-disk.iso', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => 'ISO image, img',
-        volname => 'iso/some-other-installation-disk.img',
-        expected =>
-            ['iso', 'some-other-installation-disk.img', undef, undef, undef, undef, 'raw'],
-    },
-    #
-    # container templates
-    #
-    {
-        description => 'Container template tar',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar',
-        expected => [
-            'vztmpl', 'debian-10.0-standard_10.0-1_amd64.tar', undef, undef, undef, undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.gz',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.gz',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.xz',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.xz',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    {
-        description => 'Container template tar.bz2',
-        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
-        expected => [
-            'vztmpl',
-            'debian-10.0-standard_10.0-1_amd64.tar.bz2',
-            undef,
-            undef,
-            undef,
-            undef,
-            'raw',
-        ],
-    },
-    #
     # container rootdir
     #
     {
@@ -120,62 +56,8 @@ my $tests = [
         ],
     },
     #
-    # Snippets
-    #
-    {
-        description => 'Snippets, yaml',
-        volname => 'snippets/userconfig.yaml',
-        expected => ['snippets', 'userconfig.yaml', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => 'Snippets, perl',
-        volname => 'snippets/hookscript.pl',
-        expected => ['snippets', 'hookscript.pl', undef, undef, undef, undef, 'raw'],
-    },
-    #
     # Import
     #
-    {
-        description => "Import, ova",
-        volname => 'import/import.ova',
-        expected => ['import', 'import.ova', undef, undef, undef, undef, 'ova'],
-    },
-    {
-        description => "Import, ovf",
-        volname => 'import/import.ovf',
-        expected => ['import', 'import.ovf', undef, undef, undef, undef, 'ovf'],
-    },
-    {
-        description => "Import, qcow2",
-        volname => 'import/import.qcow2',
-        expected => ['import', 'import.qcow2', undef, undef, undef, undef, 'qcow2'],
-    },
-    {
-        description => "Import, vmdk",
-        volname => 'import/import.vmdk',
-        expected => ['import', 'import.vmdk', undef, undef, undef, undef, 'vmdk'],
-    },
-    {
-        description => "Import, raw",
-        volname => 'import/import.raw',
-        expected => ['import', 'import.raw', undef, undef, undef, undef, 'raw'],
-    },
-    {
-        description => "Import, inner file of ova (qcow2)",
-        volname => 'import/import.ova/disk.qcow2',
-        expected =>
-            ['import', 'import.ova/disk.qcow2', undef, undef, undef, undef, 'ova+qcow2'],
-    },
-    {
-        description => "Import, inner file of ova (vmdk)",
-        volname => 'import/import.ova/disk.vmdk',
-        expected => ['import', 'import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
-    },
-    {
-        description => "Import, inner file of ova (raw)",
-        volname => 'import/import.ova/disk.raw',
-        expected => ['import', 'import.ova/disk.raw', undef, undef, undef, undef, 'ova+raw'],
-    },
     {
         description => "Import, inner file of ova with whitespace in name",
         volname => 'import/import.ova/OS disk.vmdk',
@@ -271,6 +153,48 @@ my $tests = [
     }
 }
 
+# Test cases for ISOs
+{
+    my $suffixes = ['iso', 'img'];
+    my $prefix = 'some-installation-disk';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "ISO image, $suffix",
+                volname => "iso/$file_name",
+                expected => ['iso', "$file_name", undef, undef, undef, undef, 'raw'],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for container templates
+{
+    my $suffixes = ['tar', 'tar.gz', 'tar.xz', 'tar.bz2'];
+    my $prefix = 'debian-10.0-standard_10.0-1_amd64';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "Container template, $suffix",
+                volname => "vztmpl/$file_name",
+                expected => [
+                    'vztmpl', "$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
 # Additional tests for backup files
 {
     my $bkp_suffixes = {
@@ -324,6 +248,72 @@ my $tests = [
     }
 }
 
+# Test cases for snippets
+{
+    my $file_names = ['userconfig.yaml', 'hookscript.pl'];
+
+    for my $file_name ($file_names->@*) {
+        my @extra_tests = (
+            {
+                description => "Snippets, $file_name",
+                volname => "snippets/$file_name",
+                expected => ['snippets', $file_name, undef, undef, undef, undef, 'raw'],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for import files
+{
+    my $suffixes = ['ova', 'ovf', 'qcow2', 'vmdk', 'raw'];
+    my $prefix = 'import-file';
+
+    for my $suffix ($suffixes->@*) {
+        my $file_name = "$prefix.$suffix";
+
+        my @extra_tests = (
+            {
+                description => "Import, $suffix",
+                volname => "import/$file_name",
+                expected => ['import', "$file_name", undef, undef, undef, undef, $suffix],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
+# Test cases for OVA import files with content inside
+{
+    my $content_suffixes = ['qcow2', 'vmdk', 'raw'];
+    my $content_prefix = 'disk';
+    my $file_name = 'import.ova';
+
+    for my $content_suffix ($content_suffixes->@*) {
+        my $content_file_name = "$content_prefix.$content_suffix";
+
+        my @extra_tests = (
+            {
+                description => "Import, inner file of ova ($content_suffix)",
+                volname => "import/$file_name/$content_file_name",
+                expected => [
+                    'import',
+                    "$file_name/$content_file_name",
+                    undef,
+                    undef,
+                    undef,
+                    undef,
+                    "ova+$content_suffix",
+                ],
+            },
+        );
+
+        push($tests->@*, @extra_tests);
+    }
+}
+
 my sub run_tests($tests) {
     my $seen_vtype = {};
     my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };
-- 
2.47.3

[PATCH pve-storage v2 41/50] test: volume id: modernize code

[Max R. Carrara]

Modernize the existing code in `path_to_volume_id_test.pm` while
leaving the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
  - remove needless `diag(explain(...))` call
  - suppress warnings when calling `path_to_volume_id()` to make test
    output less noisy
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- capitalize constants
- move the comment regarding `File::Temp->newdir()` unlinking on exit
  to where the method is actually called
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/path_to_volume_id_test.pm | 149 ++++++++++++++++-------------
 1 file changed, 82 insertions(+), 67 deletions(-)

diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index e7e36037..72644482 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -1,7 +1,6 @@
 package PVE::Storage::TestPathToVolumeId;
 
-use strict;
-use warnings;
+use v5.36;
 
 use lib qw(..);
 
@@ -13,17 +12,17 @@ use PVE::Storage::Common qw(
 use Test::More;
 
 use Cwd;
-use File::Basename;
+use File::Basename qw(fileparse);
 use File::Path qw(make_path remove_tree);
 use File::Temp;
 
-my $storage_dir = File::Temp->newdir();
-my $scfg = {
+my $DEFAULT_STORAGE_DIR = File::Temp->newdir(); # unlinks on exit
+my $DEFAULT_CFG = {
     'digest' => 'd29306346b8b25b90a4a96165f1e8f52d1af1eda',
     'ids' => {
         'local' => {
             'shared' => 0,
-            'path' => "$storage_dir",
+            'path' => "$DEFAULT_STORAGE_DIR",
             'type' => 'dir',
             'content' => {
                 'snippets' => 1,
@@ -44,24 +43,24 @@ my $scfg = {
 # description => to identify the test case
 # volname     => to create the test file
 # expected    => the result that path_to_volume_id should return
-my @tests = (
+my $tests = [
     {
         description => 'Image, qcow2',
-        volname => "$storage_dir/images/16110/vm-16110-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
         expected => [
             'images', 'local:16110/vm-16110-disk-0.qcow2',
         ],
     },
     {
         description => 'Image, raw',
-        volname => "$storage_dir/images/16112/vm-16112-disk-0.raw",
+        volname => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
         expected => [
             'images', 'local:16112/vm-16112-disk-0.raw',
         ],
     },
     {
         description => 'Image template, qcow2',
-        volname => "$storage_dir/images/9004/base-9004-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
         expected => [
             'images', 'local:9004/base-9004-disk-0.qcow2',
         ],
@@ -69,49 +68,49 @@ my @tests = (
 
     {
         description => 'Backup, vma.gz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
         ],
     },
     {
         description => 'Backup, vma.lzo',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
         ],
     },
     {
         description => 'Backup, vma',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
         ],
     },
     {
         description => 'Backup, tar.lzo',
-        volname => "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
         ],
     },
     {
         description => 'Backup, vma.zst',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
         ],
     },
     {
         description => 'Backup, tar.zst',
-        volname => "$storage_dir/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst',
         ],
     },
     {
         description => 'Backup, tar.bz2',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
         expected => [
             'backup', 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
         ],
@@ -119,21 +118,23 @@ my @tests = (
 
     {
         description => 'ISO file',
-        volname => "$storage_dir/template/iso/yet-again-a-installation-disk.iso",
+        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
         expected => [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
     {
         description => 'CT template, tar.gz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
     {
         description => 'CT template, wrong ending, tar bz2',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
         ],
@@ -141,49 +142,50 @@ my @tests = (
 
     {
         description => 'Rootdir, folder subvol, legacy naming',
-        volname => "$storage_dir/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Rootdir, folder subvol',
-        volname => "$storage_dir/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Snippets, yaml',
-        volname => "$storage_dir/snippets/userconfig.yaml",
+        volname => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
         expected => [
             'snippets', 'local:snippets/userconfig.yaml',
         ],
     },
     {
         description => 'Snippets, hookscript',
-        volname => "$storage_dir/snippets/hookscript.pl",
+        volname => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
         expected => [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
     {
         description => 'CT template, tar.xz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
         ],
     },
     {
         description => 'Import, ova',
-        volname => "$storage_dir/import/import.ova",
+        volname => "$DEFAULT_STORAGE_DIR/import/import.ova",
         expected => [
             'import', 'local:import/import.ova',
         ],
     },
     {
         description => 'Import, ovf',
-        volname => "$storage_dir/import/import.ovf",
+        volname => "$DEFAULT_STORAGE_DIR/import/import.ovf",
         expected => [
             'import', 'local:import/import.ovf',
         ],
@@ -192,91 +194,104 @@ my @tests = (
     # no matches, path or files with failures
     {
         description => 'Base template, string as vmid in folder name',
-        volname => "$storage_dir/images/ssss/base-4321-disk-0.raw",
+        volname => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
         expected => [''],
     },
     {
         description => 'ISO file, wrong ending',
-        volname => "$storage_dir/template/iso/yet-again-a-installation-disk.dvd",
+        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
         expected => [''],
     },
     {
         description => 'CT template, wrong ending, zip.gz',
-        volname => "$storage_dir/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+        volname =>
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, zip.gz',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, tgz.lzo',
-        volname => "$storage_dir/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
         expected => [''],
     },
     {
         description => 'Backup, wrong ending, qemu, vma.xz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, qemu, vms.gz',
-        volname => "$storage_dir/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
     {
         description => 'Image, string as vmid in folder name',
-        volname => "$storage_dir/images/ssss/vm-1234-disk-0.qcow2",
+        volname => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
-        volname => "$storage_dir/import/test.foo",
+        volname => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
-);
+];
 
-plan tests => scalar @tests + 1;
+my sub run_tests($tests) {
+    my $seen_vtype;
+    my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };
 
-my $seen_vtype;
-my $vtype_subdirs =
-    { map { $_ => 1 } keys %{ plugin_get_default_vtype_subdirs() } };
+    for my $t ($tests->@*) {
+        my $file = $t->{volname};
+        my $expected = $t->{expected};
+        my $description = $t->{description};
 
-foreach my $tt (@tests) {
-    my $file = $tt->{volname};
-    my $expected = $tt->{expected};
-    my $description = $tt->{description};
+        # prepare environment
+        my ($name, $dir, $suffix) = fileparse($file);
+        make_path($dir, { verbose => 1, mode => 0755 });
 
-    # prepare environment
-    my ($name, $dir, $suffix) = fileparse($file);
-    make_path($dir, { verbose => 1, mode => 0755 });
+        if ($name) {
+            open(my $fh, ">>", "$file") || die "Error open file: $!";
+            close($fh);
+        }
 
-    if ($name) {
-        open(my $fh, ">>", "$file") || die "Error open file: $!";
-        close($fh);
+        my $got;
+        eval {
+            # Suppress warnings here to make output less noisy for certain tests
+            local $SIG{__WARN__} = sub { };
+
+            $got = [PVE::Storage::path_to_volume_id($DEFAULT_CFG, $file)];
+        };
+        $got = $@ if $@;
+
+        is_deeply($got, $expected, $description);
+
+        $seen_vtype->{ $expected->[0] } = 1
+            if ($expected->[0] ne '' && scalar($expected->@*) > 1);
     }
 
-    # run tests
-    my $got;
-    eval { $got = [PVE::Storage::path_to_volume_id($scfg, $file)] };
-    $got = $@ if $@;
+    # to check if all $vtype_subdirs are defined in path_to_volume_id
+    # or have a test
+    # FIXME re-enable after vtype split changes
+    #is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
+    is_deeply({}, {}, "vtype_subdir check");
 
-    is_deeply($got, $expected, $description) || diag(explain($got));
-
-    $seen_vtype->{ @$expected[0] } = 1
-        if (@$expected[0] ne '' && scalar @$expected > 1);
+    return;
 }
 
-# to check if all $vtype_subdirs are defined in path_to_volume_id
-# or have a test
-# FIXME re-enable after vtype split changes
-#is_deeply($seen_vtype, $vtype_subdirs, "vtype_subdir check");
-is_deeply({}, {}, "vtype_subdir check");
+my sub main() {
+    plan tests => scalar($tests->@*) + 1;
 
-#cleanup
-# File::Temp unlinks tempdir on exit
+    run_tests($tests);
 
-done_testing();
+    done_testing();
+
+    return;
+}
+
+main();
 
 1;
-- 
2.47.3

[PATCH pve-storage v2 42/50] test: volume id: rename 'volname' test case parameter to 'file'

[Max R. Carrara]

... since it's for file paths, not volume names.

Improve the comment describing the test case format along the way.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/path_to_volume_id_test.pm | 72 ++++++++++++++----------------
 1 file changed, 34 insertions(+), 38 deletions(-)

diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 72644482..6c7d0d67 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -39,28 +39,28 @@ my $DEFAULT_CFG = {
     },
 };
 
-# the tests array consists of hashes with the following keys:
-# description => to identify the test case
-# volname     => to create the test file
-# expected    => the result that path_to_volume_id should return
+# Each test is comprised of the following keys:
+# description => to identify a single test
+# file        => the absolute path to the volume's file that should be parsed
+# expected    => expected output of path_to_volume_id(), as listref
 my $tests = [
     {
         description => 'Image, qcow2',
-        volname => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/16110/vm-16110-disk-0.qcow2",
         expected => [
             'images', 'local:16110/vm-16110-disk-0.qcow2',
         ],
     },
     {
         description => 'Image, raw',
-        volname => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
+        file => "$DEFAULT_STORAGE_DIR/images/16112/vm-16112-disk-0.raw",
         expected => [
             'images', 'local:16112/vm-16112-disk-0.raw',
         ],
     },
     {
         description => 'Image template, qcow2',
-        volname => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/9004/base-9004-disk-0.qcow2",
         expected => [
             'images', 'local:9004/base-9004-disk-0.qcow2',
         ],
@@ -68,49 +68,49 @@ my $tests = [
 
     {
         description => 'Backup, vma.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz',
         ],
     },
     {
         description => 'Backup, vma.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_12_45.vma.lzo',
         ],
     },
     {
         description => 'Backup, vma',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma',
         ],
     },
     {
         description => 'Backup, tar.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.lzo',
         ],
     },
     {
         description => 'Backup, vma.zst',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst",
         expected => [
             'backup', 'local:backup/vzdump-qemu-16110-2020_03_30-21_13_55.vma.zst',
         ],
     },
     {
         description => 'Backup, tar.zst',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst",
         expected => [
             'backup', 'local:backup/vzdump-lxc-16112-2020_03_30-21_39_30.tar.zst',
         ],
     },
     {
         description => 'Backup, tar.bz2',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2",
         expected => [
             'backup', 'local:backup/vzdump-openvz-16112-2020_03_30-21_39_30.tar.bz2',
         ],
@@ -118,23 +118,21 @@ my $tests = [
 
     {
         description => 'ISO file',
-        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
+        file => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.iso",
         expected => [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
     {
         description => 'CT template, tar.gz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
     {
         description => 'CT template, wrong ending, tar bz2',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.bz2',
         ],
@@ -142,50 +140,49 @@ my $tests = [
 
     {
         description => 'Rootdir, folder subvol, legacy naming',
-        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        file => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Rootdir, folder subvol',
-        volname => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
+        file => "$DEFAULT_STORAGE_DIR/images/1234/subvol-1234-disk-0.subvol/", # fileparse needs / at the end
         expected => [
             'images', 'local:1234/subvol-1234-disk-0.subvol',
         ],
     },
     {
         description => 'Snippets, yaml',
-        volname => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
+        file => "$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml",
         expected => [
             'snippets', 'local:snippets/userconfig.yaml',
         ],
     },
     {
         description => 'Snippets, hookscript',
-        volname => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
+        file => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl",
         expected => [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
     {
         description => 'CT template, tar.xz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
         expected => [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
         ],
     },
     {
         description => 'Import, ova',
-        volname => "$DEFAULT_STORAGE_DIR/import/import.ova",
+        file => "$DEFAULT_STORAGE_DIR/import/import.ova",
         expected => [
             'import', 'local:import/import.ova',
         ],
     },
     {
         description => 'Import, ovf',
-        volname => "$DEFAULT_STORAGE_DIR/import/import.ovf",
+        file => "$DEFAULT_STORAGE_DIR/import/import.ovf",
         expected => [
             'import', 'local:import/import.ovf',
         ],
@@ -194,48 +191,47 @@ my $tests = [
     # no matches, path or files with failures
     {
         description => 'Base template, string as vmid in folder name',
-        volname => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
+        file => "$DEFAULT_STORAGE_DIR/images/ssss/base-4321-disk-0.raw",
         expected => [''],
     },
     {
         description => 'ISO file, wrong ending',
-        volname => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
+        file => "$DEFAULT_STORAGE_DIR/template/iso/yet-again-a-installation-disk.dvd",
         expected => [''],
     },
     {
         description => 'CT template, wrong ending, zip.gz',
-        volname =>
-            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
+        file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, zip.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.zip.gz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, openvz, tgz.lzo',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-openvz-16112-2020_03_30-21_39_30.tgz.lzo",
         expected => [''],
     },
     {
         description => 'Backup, wrong ending, qemu, vma.xz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vma.xz",
         expected => [''],
     },
     {
         description => 'Backup, wrong format, qemu, vms.gz',
-        volname => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
+        file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
     {
         description => 'Image, string as vmid in folder name',
-        volname => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
+        file => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
-        volname => "$DEFAULT_STORAGE_DIR/import/test.foo",
+        file => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
 ];
@@ -245,7 +241,7 @@ my sub run_tests($tests) {
     my $vtype_subdirs = { map { $_ => 1 } keys plugin_get_default_vtype_subdirs()->%* };
 
     for my $t ($tests->@*) {
-        my $file = $t->{volname};
+        my $file = $t->{file};
         my $expected = $t->{expected};
         my $description = $t->{description};
 
-- 
2.47.3

[PATCH pve-storage v2 43/50] test: filesystem path: modernize code

[Max R. Carrara]

Modernize the existing code in `filesystem_path_test.pm` while leaving
the tests unchanged otherwise.

In particular,
- move the test execution code into its own subroutine
  - remove needless `diag(explain(...))` call
- add and call a `main()` subroutine
- declare `use v5.36;` instead of `use strict;` and `use warnings;`
- rename / capitalize the sole constant
- adapt the code style to fit our more modern style guide

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/test/filesystem_path_test.pm | 53 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 21 deletions(-)

diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index af52380c..5a715ce9 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -1,14 +1,14 @@
 package PVE::Storage::TestFilesystemPath;
 
-use strict;
-use warnings;
+use v5.36;
 
 use lib qw(..);
 
 use PVE::Storage;
+
 use Test::More;
 
-my $path = '/some/path';
+my $DEFAULT_STORAGE_DIR = '/some/path';
 
 # each array entry is a test that consists of the following keys:
 # volname  => image name that is passed to parse_volname
@@ -19,7 +19,7 @@ my $tests = [
         volname => '1234/vm-1234-disk-0.raw',
         snapname => undef,
         expected => [
-            "$path/images/1234/vm-1234-disk-0.raw", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.raw", '1234', 'images',
         ],
     },
     {
@@ -31,49 +31,60 @@ my $tests = [
         volname => '1234/vm-1234-disk-0.qcow2',
         snapname => undef,
         expected => [
-            "$path/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
         ],
     },
     {
         volname => '1234/vm-1234-disk-0.qcow2',
         snapname => 'my_snap',
         expected => [
-            "$path/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
+            "$DEFAULT_STORAGE_DIR/images/1234/vm-1234-disk-0.qcow2", '1234', 'images',
         ],
     },
     {
         volname => 'iso/my-awesome-proxmox.iso',
         snapname => undef,
         expected => [
-            "$path/template/iso/my-awesome-proxmox.iso", undef, 'iso',
+            "$DEFAULT_STORAGE_DIR/template/iso/my-awesome-proxmox.iso", undef, 'iso',
         ],
     },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
         expected => [
-            "$path/dump/vzdump-qemu-1234-2020_03_30-21_12_40.vma", 1234, 'backup',
+            "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-1234-2020_03_30-21_12_40.vma", 1234,
+            'backup',
         ],
     },
 ];
 
-plan tests => scalar @$tests;
+my sub run_tests($tests) {
+    for my $t ($tests->@*) {
+        my $volname = $t->{volname};
+        my $snapname = $t->{snapname};
+        my $expected = $t->{expected};
+        my $scfg = { path => $DEFAULT_STORAGE_DIR };
+        my $got;
 
-foreach my $tt (@$tests) {
-    my $volname = $tt->{volname};
-    my $snapname = $tt->{snapname};
-    my $expected = $tt->{expected};
-    my $scfg = { path => $path };
-    my $got;
+        eval { $got = [PVE::Storage::Plugin->filesystem_path($scfg, $volname, $snapname)]; };
+        $got = $@ if $@;
 
-    eval { $got = [PVE::Storage::Plugin->filesystem_path($scfg, $volname, $snapname)]; };
-    $got = $@ if $@;
-
-    is_deeply($got, $expected, "wantarray: filesystem_path for $volname")
-        || diag(explain($got));
+        is_deeply($got, $expected, "wantarray: filesystem_path for $volname");
+    }
 
+    return;
 }
 
-done_testing();
+my sub main() {
+    plan tests => scalar($tests->@*);
+
+    run_tests($tests);
+
+    done_testing();
+
+    return;
+}
+
+main();
 
 1;
-- 
2.47.3

[PATCH pve-storage v2 44/50] fix #2884: implement nested subdir scanning and support 'iso' vtype

[Max R. Carrara]

Introduce a new 'max-scan-depth' property for directory-based
storages. This property makes it so that files for specific volume
types may also be put into (nested) subdirectories inside the volume
type's directory.

The maximum allowed depth is between 0 and 50, with 0 being the
default, which corresponds to the current behavior.

Support nested subdirectories for ISOs (the 'iso' vtype) only for now.

Achieve all of this by calling the `get_subdir_files()` helper in
`PVE::Storage::Plugin` recursively. Note that the default recursion
limit in Perl appears to be exactly 100 [0], way above the maximum
value of 50 for the 'max-scan-depth' property. Should we need deeper
nesting, we can always make the subroutine iterative instead of
recursive later.

Add additional test cases wherever applicable to account for nested
subdirectories, including cases that check whether the limit set by
the property is upheld, and also cases that check for the existence of
parent directory references ('..') in volume names.

[0]: https://perldoc.perl.org/perl5101delta#Deep-recursion-on-subroutine-%22%25s%22

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/BTRFSPlugin.pm              |   1 +
 src/PVE/Storage/CephFSPlugin.pm             |   1 +
 src/PVE/Storage/Common/Parse.pm             |  18 +++
 src/PVE/Storage/Common/test/parser_tests.pl |  50 ++++++++
 src/PVE/Storage/DirPlugin.pm                |   1 +
 src/PVE/Storage/Plugin.pm                   |  39 +++++-
 src/test/filesystem_path_test.pm            |   8 ++
 src/test/list_volumes_test.pm               | 127 ++++++++++++++++++++
 src/test/parse_volname_test.pm              |  56 +++++++++
 src/test/path_to_volume_id_test.pm          |   7 ++
 10 files changed, 304 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index e75b4fe0..b5b41dbe 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -68,6 +68,7 @@ sub properties {
 sub options {
     return {
         path => { fixed => 1 },
+        'max-scan-depth' => { optional => 1 },
         nodes => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index fbc97113..b31f53d9 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -140,6 +140,7 @@ sub options {
     return {
         path => { fixed => 1 },
         'content-dirs' => { optional => 1 },
+        'max-scan-depth' => { optional => 1 },
         monhost => { optional => 1 },
         nodes => { optional => 1 },
         subdir => { optional => 1 },
diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 296a6c4d..1d021afa 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -60,6 +60,10 @@ my $RE_SAFE_CHAR_CLASS = qr/[a-zA-Z0-9\-\.\+\=\_]/;
 
 my $RE_SAFE_CHAR_WITH_WHITESPACE_CLASS = qr/[ a-zA-Z0-9\-\.\+\=\_]/;
 
+my $RE_DIRECTORY_COMPONENTS = qr!
+    ( ($RE_SAFE_CHAR_WITH_WHITESPACE_CLASS)+ / )*
+!xn;
+
 my $RE_PARENT_DIR = quotemeta('..');
 my $RE_CONTAINS_PARENT_DIR = qr!
     ( ^$RE_PARENT_DIR/ )  #  ../ --> Beginning of path
@@ -76,6 +80,7 @@ my $RE_VMID = qr![1-9][0-9]{2,8}!;
 
 my $RE_ISO_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file> [^/]+ \. (?<ext> (?i: iso|img) ) )
     )
 !xn;
@@ -209,6 +214,15 @@ my sub format_named_groups(%groups) {
 
     my @disk_path_components = ();
 
+    if (defined($result->{dir})) {
+        if ($result->{dir} eq '') {
+            delete $result->{dir};
+        } else {
+            $result->{dir} = strip_trailing_path_separators($result->{dir});
+            push(@disk_path_components, $result->{dir});
+        }
+    }
+
     if (defined($result->{file})) {
         $result->{file} = strip_leading_path_separators($result->{file});
         push(@disk_path_components, $result->{file});
@@ -312,6 +326,10 @@ These are:
         volname => 'iso/custom-debian.iso',
     }
 
+Volume types that support subdirectories in their C<path> part also have a
+C<dir> part if a subdirectory is specified. This is currently limited to the
+C<iso> volume type.
+
 The following volume types have more specific parts in addition to the common
 ones:
 
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 2869d3f0..204002d0 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -73,6 +73,32 @@ my $volname_cases_iso_valid = [
             volname => 'iso/Fedora.Img',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/custom-debian.iso',
+        expected => {
+            file => 'custom-debian.iso',
+            ext => 'iso',
+            dir => 'subdir',
+            'disk-path' => 'subdir/custom-debian.iso',
+            path => 'subdir/custom-debian.iso',
+            vtype => 'iso',
+            volname => 'iso/subdir/custom-debian.iso',
+        },
+    },
+    {
+        path => 'deeply/nested/dir/hannah-montana-linux.IMG',
+        expected => {
+            file => 'hannah-montana-linux.IMG',
+            ext => 'IMG',
+            dir => 'deeply/nested/dir',
+            'disk-path' => 'deeply/nested/dir/hannah-montana-linux.IMG',
+            path => 'deeply/nested/dir/hannah-montana-linux.IMG',
+            vtype => 'iso',
+            volname => 'iso/deeply/nested/dir/hannah-montana-linux.IMG',
+        },
+    },
 ];
 
 my $volname_cases_iso_invalid = [
@@ -84,6 +110,30 @@ my $volname_cases_iso_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Parent dir reference in path (beginning) (iso)",
+        args => {
+            path => '../custom-debian.iso',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (iso)",
+        args => {
+            path => 'subdir/../custom-debian.iso',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (iso)",
+        args => {
+            path => 'subdir/custom-debian.iso/..',
+            vtype => 'iso',
+        },
+        expected => undef,
+    },
 ];
 
 my $volname_cases_vztmpl_valid = [
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 80c4a031..58b942e7 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -81,6 +81,7 @@ sub options {
     return {
         path => { fixed => 1 },
         'content-dirs' => { optional => 1 },
+        'max-scan-depth' => { optional => 1 },
         nodes => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index e2f24480..e96ab2d5 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -234,6 +234,15 @@ my $defaultData = {
             format => "pve-dir-override-list",
             optional => 1,
         },
+        'max-scan-depth' => {
+            description => "Maximum depth of subdirectories to traverse when searching for"
+                . " ISOs in directories.",
+            type => 'integer',
+            default => 0,
+            minimum => 0,
+            maximum => 50,
+            optional => 1,
+        },
         options => {
             description => "NFS/CIFS mount options (see 'man nfs' or 'man mount.cifs')",
             type => 'string',
@@ -1696,10 +1705,16 @@ sub list_images {
 
 # $vtype = <iso|vztmpl|backup|snippets|import>
 my sub get_subdir_files {
-    my ($storeid, $scfg, $vtype, $vmid) = @_;
+    use feature 'current_sub'; # Needed for the __SUB__ token further below
+
+    my ($storeid, $scfg, $vtype, $vmid, $remaining_depth, $current_path) = @_;
 
     my $vtype_subdir = plugin_get_vtype_subdir($scfg, $vtype);
 
+    if (!defined($current_path)) {
+        $current_path = $vtype_subdir;
+    }
+
     my $res = [];
 
     my $get_subdir_file_info = sub {
@@ -1783,10 +1798,22 @@ my sub get_subdir_files {
         return;
     };
 
-    for my $path (<$vtype_subdir/*>) {
+    for my $path (<$current_path/*>) {
         my $st = File::stat::stat($path);
 
-        next if (!$st || S_ISDIR($st->mode));
+        next if !$st;
+
+        if (S_ISDIR($st->mode)) {
+            if (defined($remaining_depth) && $remaining_depth > 0) {
+                my $inner_res = __SUB__->(
+                    $storeid, $scfg, $vtype, $vmid, $remaining_depth - 1, $path,
+                );
+
+                push($res->@*, $inner_res->@*);
+            }
+
+            next;
+        }
 
         if (defined(my $info = $get_subdir_file_info->($path, $st))) {
             $info->{size} = $st->size;
@@ -1804,6 +1831,10 @@ my sub get_subdir_files {
 sub list_volumes {
     my ($class, $storeid, $scfg, $vmid, $content_types) = @_;
 
+    my $depth = $scfg->{'max-scan-depth'} // 0;
+    $depth = 0 if $depth < 0;
+    $depth = 50 if $depth > 50;
+
     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
 
@@ -1821,7 +1852,7 @@ sub list_volumes {
         return if !$scfg->{path};
 
         if ($type eq 'iso' && !defined($vmid)) {
-            return get_subdir_files($storeid, $scfg, 'iso', undef);
+            return get_subdir_files($storeid, $scfg, 'iso', undef, $depth);
         }
 
         if ($type eq 'vztmpl' && !defined($vmid)) {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index 5a715ce9..26a74a0d 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -48,6 +48,14 @@ my $tests = [
             "$DEFAULT_STORAGE_DIR/template/iso/my-awesome-proxmox.iso", undef, 'iso',
         ],
     },
+    {
+        volname => 'iso/foo/bar/baz/my-awesome-proxmox.iso',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/iso/foo/bar/baz/my-awesome-proxmox.iso", undef,
+            'iso',
+        ],
+    },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 53ce5116..b1f72091 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1228,6 +1228,133 @@ my $test_param_list = [
             },
         ],
     },
+    {
+        description => "VMID: none, no nested subdirectories when using defaults",
+        storeid => $DEFAULT_STOREID,
+        scfg => $DEFAULT_SCFG,
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => "VMID: none, nested subdirectories allowed, max-scan-depth = 1",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            'max-scan-depth' => 1,
+            content => {
+                iso => 1,
+                vztmpl => 1,
+                snippets => 1,
+                import => 1,
+            },
+        },
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/some-installer.iso',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
+    {
+        description => "VMID: none, nested subdirectories allowed, max-scan-depth = 5",
+        storeid => $DEFAULT_STOREID,
+        scfg => {
+            type => 'dir',
+            path => $DEFAULT_STORAGE_PATH,
+            shared => 0,
+            'max-scan-depth' => 5,
+            content => {
+                iso => 1,
+                vztmpl => 1,
+                snippets => 1,
+                import => 1,
+            },
+        },
+        vmid => undef,
+        vtypes => ['iso', 'vztmpl', 'snippets', 'import'],
+        cases => [
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/some-installer.iso',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
+                expected => {
+                    content => 'iso',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'iso',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:iso/1/2/3/4/5/some-installer.iso',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/6/some-installer.iso",
+                expected => undef,
+            },
+        ],
+    },
 ];
 
 # Additional test cases that cannot be constructed within the list above
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 1dcec9d6..24d1ed0e 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -167,10 +167,66 @@ my $tests = [
                 volname => "iso/$file_name",
                 expected => ['iso', "$file_name", undef, undef, undef, undef, 'raw'],
             },
+            {
+                description => "ISO image, $suffix, subdirectory",
+                volname => "iso/foo/$file_name",
+                expected => [
+                    'iso', "foo/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "ISO image, $suffix, nested subdirectories",
+                volname => "iso/foo/bar/baz/$file_name",
+                expected => [
+                    'iso', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "ISO image, $suffix, subdirectory with same name as file",
+                volname => "iso/$file_name/$file_name",
+                expected => [
+                    'iso', "$file_name/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
         );
 
         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.iso";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "ISO image, iso, parent directory reference before volume type prefix",
+                volname => "../iso/$file_name",
+                expected => "unable to parse directory volume name '../iso/$file_name'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference at beginning of volume path",
+                volname => "iso/../$file_name",
+                expected => "unable to parse directory volume name 'iso/../$file_name'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference at end of volume path",
+                volname => "iso/$file_name/..",
+                expected => "unable to parse directory volume name 'iso/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "ISO image, iso, parent directory reference between dir components of volume path",
+                volname => "iso/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'iso/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }
 
 # Test cases for container templates
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 6c7d0d67..bc87d289 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -123,6 +123,13 @@ my $tests = [
             'iso', 'local:iso/yet-again-a-installation-disk.iso',
         ],
     },
+    {
+        description => 'ISO file, nested subdirectories',
+        file => "$DEFAULT_STORAGE_DIR/template/iso/foo/bar/hannah-montana-linux-installer.iso",
+        expected => [
+            'iso', 'local:iso/foo/bar/hannah-montana-linux-installer.iso',
+        ],
+    },
     {
         description => 'CT template, tar.gz',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
-- 
2.47.3

[PATCH pve-storage v2 45/50] fix #2884: support nested subdir scanning for 'vztmpl' volume type

[Max R. Carrara]

Add support for parsing nested subdirectories for the 'vztmpl' volume
type by adapting its corresponding regex used by parsing helpers.

Add additional test cases wherever applicable to account for nested
subdirectories for 'vztmpl' volumes, as done for the 'iso' volume
type.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  3 +-
 src/PVE/Storage/Common/test/parser_tests.pl | 54 ++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  4 +-
 src/test/filesystem_path_test.pm            | 18 +++++
 src/test/list_volumes_test.pm               | 81 +++++++++++++++++++++
 src/test/parse_volname_test.pm              | 57 +++++++++++++++
 src/test/path_to_volume_id_test.pm          |  8 ++
 7 files changed, 222 insertions(+), 3 deletions(-)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 1d021afa..86787e75 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -87,6 +87,7 @@ my $RE_ISO_FILE_PATH = qr!
 
 my $RE_VZTMPL_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file>
             [^/]+
             \.
@@ -328,7 +329,7 @@ These are:
 
 Volume types that support subdirectories in their C<path> part also have a
 C<dir> part if a subdirectory is specified. This is currently limited to the
-C<iso> volume type.
+C<iso> and C<vztmpl> volume types.
 
 The following volume types have more specific parts in addition to the common
 ones:
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 204002d0..7508d0e5 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -230,6 +230,36 @@ my $volname_cases_vztmpl_valid = [
             volname => 'vztmpl/Alpine 3.10 default_20190626_amd64.Tar.xZ',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        expected => {
+            file => 'debian-10.0-standard_10.0-1_amd64.tar.zst',
+            ext => 'tar.zst',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'zst',
+            dir => 'subdir',
+            'disk-path' => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+            path => 'subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/subdir/debian-10.0-standard_10.0-1_amd64.tar.zst',
+        },
+    },
+    {
+        path => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        expected => {
+            file => 'debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            ext => 'tar.bz2',
+            'ext-archive' => 'tar',
+            'ext-compression' => 'bz2',
+            dir => 'deeply/nested/dir',
+            'disk-path' => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            path => 'deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+            vtype => 'vztmpl',
+            volname => 'vztmpl/deeply/nested/dir/debian-11.0-standard_11.0-1_amd64.tar.bz2',
+        },
+    },
 ];
 
 my $volname_cases_vztmpl_invalid = [
@@ -241,6 +271,30 @@ my $volname_cases_vztmpl_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Parent dir reference in path (beginning) (vztmpl)",
+        args => {
+            path => '../archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (vztmpl)",
+        args => {
+            path => 'subdir/../archlinux-base_20190924-1_amd64.tar.gz',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (vztmpl)",
+        args => {
+            path => 'subdir/archlinux-base_20190924-1_amd64.tar.gz/..',
+            vtype => 'vztmpl',
+        },
+        expected => undef,
+    },
 ];
 
 my $volname_cases_backup_valid = [
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index e96ab2d5..2e22e6b7 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -236,7 +236,7 @@ my $defaultData = {
         },
         'max-scan-depth' => {
             description => "Maximum depth of subdirectories to traverse when searching for"
-                . " ISOs in directories.",
+                . " ISOs and container templates in directories.",
             type => 'integer',
             default => 0,
             minimum => 0,
@@ -1856,7 +1856,7 @@ sub list_volumes {
         }
 
         if ($type eq 'vztmpl' && !defined($vmid)) {
-            return get_subdir_files($storeid, $scfg, 'vztmpl', undef);
+            return get_subdir_files($storeid, $scfg, 'vztmpl', undef, $depth);
         }
 
         if ($type eq 'backup') {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index 26a74a0d..b5c1ab33 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -56,6 +56,24 @@ my $tests = [
             'iso',
         ],
     },
+    {
+        volname => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz",
+            undef,
+            'vztmpl',
+        ],
+    },
+    {
+        volname => 'vztmpl/foo/bar/baz/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/template/cache/foo/bar/baz/debian-10.0-standard_10.0-1_amd64.tar.gz",
+            undef,
+            'vztmpl',
+        ],
+    },
     {
         volname => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
         snapname => undef,
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index b1f72091..4dcadb87 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1253,6 +1253,25 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => undef,
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1298,6 +1317,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1353,6 +1397,43 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/iso/1/2/3/4/5/6/some-installer.iso",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/template/cache/1/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
+                expected => {
+                    content => 'vztmpl',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'tgz',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:vztmpl/1/2/3/4/5/some-lxc-template.tar.gz',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file =>
+                    "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/6/some-lxc-template.tar.gz",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 24d1ed0e..b90815c2 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -245,10 +245,67 @@ my $tests = [
                     'vztmpl', "$file_name", undef, undef, undef, undef, 'raw',
                 ],
             },
+            {
+                description => "Container template, $suffix, subdirectory",
+                volname => "vztmpl/foo/$file_name",
+                expected => [
+                    'vztmpl', "foo/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description => "Container template, $suffix, nested subdirectories",
+                volname => "vztmpl/foo/bar/baz/$file_name",
+                expected => [
+                    'vztmpl', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
+            {
+                description =>
+                    "Container template, $suffix, subdirectory with same name as file",
+                volname => "vztmpl/$file_name/$file_name",
+                expected => [
+                    'vztmpl', "$file_name/$file_name", undef, undef, undef, undef, 'raw',
+                ],
+            },
         );
 
         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.tar.gz";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference before volume type prefix",
+                volname => "../vztmpl/$file_name",
+                expected => "unable to parse directory volume name '../vztmpl/$file_name'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference at beginning of volume path",
+                volname => "vztmpl/../$file_name",
+                expected => "unable to parse directory volume name 'vztmpl/../$file_name'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference at end of volume path",
+                volname => "vztmpl/$file_name/..",
+                expected => "unable to parse directory volume name 'vztmpl/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Container template, tar.gz, parent directory reference between dir components of volume path",
+                volname => "vztmpl/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'vztmpl/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }
 
 # Additional tests for backup files
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index bc87d289..4dfc68e1 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -137,6 +137,14 @@ my $tests = [
             'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
         ],
     },
+    {
+        description => 'CT template, tar.gz, nested subdirectories',
+        file =>
+            "$DEFAULT_STORAGE_DIR/template/cache/foo/bar/debian-10.0-standard_10.0-1_amd64.tar.gz",
+        expected => [
+            'vztmpl', 'local:vztmpl/foo/bar/debian-10.0-standard_10.0-1_amd64.tar.gz',
+        ],
+    },
     {
         description => 'CT template, wrong ending, tar bz2',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.bz2",
-- 
2.47.3

[PATCH pve-storage v2 46/50] fix #2884: support nested subdir scanning for 'snippets' vtype

[Max R. Carrara]

Add support for parsing nested subdirectories for the 'snippets'
volume type by adapting its corresponding regex used by parsing
helpers.

Add additional test cases wherever applicable to account for nested
subdirectories for 'snippets' volumes, as was done for the 'iso' and
'vztmpl' volume types.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  3 +-
 src/PVE/Storage/Common/test/parser_tests.pl | 52 ++++++++++++++
 src/PVE/Storage/Plugin.pm                   |  4 +-
 src/test/filesystem_path_test.pm            | 16 +++++
 src/test/list_volumes_test.pm               | 78 +++++++++++++++++++++
 src/test/parse_volname_test.pm              | 52 ++++++++++++++
 src/test/path_to_volume_id_test.pm          |  7 ++
 7 files changed, 209 insertions(+), 3 deletions(-)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index 86787e75..e665f081 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -129,6 +129,7 @@ my $RE_BACKUP_FILE_PATH = qr!
 
 my $RE_SNIPPETS_FILE_PATH = qr!
     (?<path>
+        (?<dir> $RE_DIRECTORY_COMPONENTS )?
         (?<file> [^/]+ )
     )
 !xn;
@@ -329,7 +330,7 @@ These are:
 
 Volume types that support subdirectories in their C<path> part also have a
 C<dir> part if a subdirectory is specified. This is currently limited to the
-C<iso> and C<vztmpl> volume types.
+C<iso>, C<vztmpl> and C<snippets> volume types.
 
 The following volume types have more specific parts in addition to the common
 ones:
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 7508d0e5..a5402390 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -498,6 +498,57 @@ my $volname_cases_snippets_valid = [
             volname => 'snippets/userconfig.yaml',
         },
     },
+
+    # subdirectories
+    {
+        path => 'subdir/hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            dir => 'subdir',
+            'disk-path' => 'subdir/hookscript.pl',
+            path => 'subdir/hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/subdir/hookscript.pl',
+        },
+    },
+    {
+        path => 'deeply/nested/subdir/hookscript.pl',
+        expected => {
+            file => 'hookscript.pl',
+            dir => 'deeply/nested/subdir',
+            'disk-path' => 'deeply/nested/subdir/hookscript.pl',
+            path => 'deeply/nested/subdir/hookscript.pl',
+            vtype => 'snippets',
+            volname => 'snippets/deeply/nested/subdir/hookscript.pl',
+        },
+    },
+];
+
+my $volname_cases_snippets_invalid = [
+    {
+        description => "Parent dir reference in path (beginning) (snippets)",
+        args => {
+            path => '../hookscript.pl',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (snippets)",
+        args => {
+            path => 'subdir/../hookscript.pl',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (snippets)",
+        args => {
+            path => 'subdir/hookscript.pl/..',
+            vtype => 'snippets',
+        },
+        expected => undef,
+    },
 ];
 
 my $volname_cases_import_valid = [
@@ -597,6 +648,7 @@ my $cases_invalid_all = [
     $volname_cases_iso_invalid,
     $volname_cases_vztmpl_invalid,
     $volname_cases_backup_invalid,
+    $volname_cases_snippets_invalid,
     $volname_cases_import_invalid,
 ];
 
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 2e22e6b7..d011417e 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -236,7 +236,7 @@ my $defaultData = {
         },
         'max-scan-depth' => {
             description => "Maximum depth of subdirectories to traverse when searching for"
-                . " ISOs and container templates in directories.",
+                . " ISOs, container templates and snippets in directories.",
             type => 'integer',
             default => 0,
             minimum => 0,
@@ -1864,7 +1864,7 @@ sub list_volumes {
         }
 
         if ($type eq 'snippets') {
-            return get_subdir_files($storeid, $scfg, 'snippets', undef);
+            return get_subdir_files($storeid, $scfg, 'snippets', undef, $depth);
         }
 
         if ($type eq 'import') {
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index b5c1ab33..ff6dffe1 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -82,6 +82,22 @@ my $tests = [
             'backup',
         ],
     },
+    {
+        volname => 'snippets/userconfig.yaml',
+        snapname => undef,
+        expected => ["$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml", undef, 'snippets'],
+    },
+    {
+        volname => 'snippets/hookscript.pl',
+        snapname => undef,
+        expected => ["$DEFAULT_STORAGE_DIR/snippets/hookscript.pl", undef, 'snippets'],
+    },
+    {
+        volname => 'snippets/foo/bar/baz/something.txt',
+        snapname => undef,
+        expected =>
+            ["$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", undef, 'snippets'],
+    },
 ];
 
 my sub run_tests($tests) {
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index 4dcadb87..d0b054c9 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1272,6 +1272,24 @@ my $test_param_list = [
                     "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => undef,
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1342,6 +1360,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/some-hookscript.pl',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1434,6 +1477,41 @@ my $test_param_list = [
                     "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/6/some-lxc-template.tar.gz",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/some-hookscript.pl',
+                },
+            },
+            {
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
+                expected => {
+                    content => 'snippets',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'snippet',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:snippets/1/2/3/4/5/some-hookscript.pl',
+                },
+            },
+            {
+                # Exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/6/some-hookscript.pl",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index b90815c2..0a425ee2 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -372,10 +372,62 @@ my $tests = [
                 volname => "snippets/$file_name",
                 expected => ['snippets', $file_name, undef, undef, undef, undef, 'raw'],
             },
+            {
+                description => "Snippets, $file_name, subdirectory",
+                volname => "snippets/foo/$file_name",
+                expected => ['snippets', "foo/$file_name", undef, undef, undef, undef, 'raw'],
+            },
+            {
+                description => "Snippets, $file_name, nested subdirectories",
+                volname => "snippets/foo/bar/baz/$file_name",
+                expected =>
+                    ['snippets', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw'],
+            },
+            {
+                description => "Snippets, $file_name, subdirectory with same name as file",
+                volname => "snippets/$file_name/$file_name",
+                expected =>
+                    ['snippets', "$file_name/$file_name", undef, undef, undef, undef, 'raw'],
+            },
         );
 
         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "some-file.txt";
+
+        my @extra_failed_tests = (
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference before volume type prefix",
+                volname => "../snippets/$file_name",
+                expected => "unable to parse directory volume name '../snippets/$file_name'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference at beginning of volume path",
+                volname => "snippets/../$file_name",
+                expected => "unable to parse directory volume name 'snippets/../$file_name'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference at end of volume path",
+                volname => "snippets/$file_name/..",
+                expected => "unable to parse directory volume name 'snippets/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Snippets, $file_name, parent directory reference between dir components of volume path",
+                volname => "snippets/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'snippets/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }
 
 # Test cases for import files
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 4dfc68e1..d22f6272 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -181,6 +181,13 @@ my $tests = [
             'snippets', 'local:snippets/hookscript.pl',
         ],
     },
+    {
+        description => 'Snippets, arbitrary file, nested subdirectories',
+        file => "$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt",
+        expected => [
+            'snippets', 'local:snippets/foo/bar/baz/something.txt',
+        ],
+    },
     {
         description => 'CT template, tar.xz',
         file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz",
-- 
2.47.3

[PATCH pve-storage v2 47/50] test: add more tests for 'import' vtype & guard against nested subdirs

[Max R. Carrara]

Supporting nested subdirectories for the 'import' volume type is
currently not possible, because of a parsing ambiguity that would be
introduced. In particular, it would not be possible to differentiate
whether 'foo.ova' in 'foo.ova/bar.qcow2' refers to a directory ending
with '.ova' or an actual '.ova' file through parsing alone. Note this
down in a comment as well, so it doesn't get overlooked.

Therefore, instead of supporting nested subdirectories for the
'import' volume type, add tests that explicitly guard against
subdirectory scanning support.

Additionally, toss in a few extra tests for the `filesystem_path()`
plugin API method and also for the differences in whitespace handling
for file paths and volume names for the 'import' volume type.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |   7 ++
 src/PVE/Storage/Common/test/parser_tests.pl |  40 +++++++
 src/test/filesystem_path_test.pm            |  14 +++
 src/test/list_volumes_test.pm               |  65 +++++++++++
 src/test/parse_volname_test.pm              | 120 ++++++++++++++++++++
 src/test/path_to_volume_id_test.pm          |  13 +++
 6 files changed, 259 insertions(+)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index e665f081..e64ac719 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -134,6 +134,13 @@ my $RE_SNIPPETS_FILE_PATH = qr!
     )
 !xn;
 
+# NOTE: Supporting subdirectories would introduce a parsing ambiguity with
+# volume names; in particular, there would be no way to tell whether 'foo.ova'
+# in 'foo.ova/bar.qcow2' refers to a directory or a file just through parsing
+# alone.
+#
+# Therefore, neither RE_IMPORT_FILE_PATH nor RE_IMPORT_VOLNAME currently
+# accept subdirectory components.
 my $RE_IMPORT_FILE_PATH = qr!
     (?<path>
         (?<file> ($RE_SAFE_CHAR_CLASS)+ \. (?<ext> $RE_IMPORT_EXTENSIONS) )
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index a5402390..01daf1b7 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -634,6 +634,46 @@ my $volname_cases_import_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Subdirectory (import)",
+        args => {
+            path => 'subdir/disk.ovf',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Nested subdirectory (import)",
+        args => {
+            path => 'deeply/nested/subdir/disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (beginning) (import)",
+        args => {
+            path => '../disk.ova',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (import)",
+        args => {
+            path => 'subdir/../disk.raw',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (import)",
+        args => {
+            path => 'subdir/disk.qcow2/..',
+            vtype => 'import',
+        },
+        expected => undef,
+    },
 ];
 
 my $cases_valid_all = [
diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index ff6dffe1..f06682a9 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -98,6 +98,20 @@ my $tests = [
         expected =>
             ["$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", undef, 'snippets'],
     },
+    {
+        volname => 'import/import.ova',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/import/import.ova", undef, 'import',
+        ],
+    },
+    {
+        volname => 'import/import.ovf',
+        snapname => undef,
+        expected => [
+            "$DEFAULT_STORAGE_DIR/import/import.ovf", undef, 'import',
+        ],
+    },
 ];
 
 my sub run_tests($tests) {
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index d0b054c9..c185fdee 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -1290,6 +1290,26 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed regardless
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed regardless
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/some-import.ova",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1385,6 +1405,26 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed + exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/some-import.ova",
+                expected => undef,
+            },
         ],
     },
     {
@@ -1512,6 +1552,31 @@ my $test_param_list = [
                 file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/6/some-hookscript.pl",
                 expected => undef,
             },
+            {
+                file => "$DEFAULT_STORAGE_PATH/import/some-import.ova",
+                expected => {
+                    content => 'import',
+                    ctime => $DEFAULT_CTIME,
+                    format => 'ova',
+                    size => $DEFAULT_SIZE,
+                    volid => 'local:import/some-import.ova',
+                },
+            },
+            {
+                # Not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Still not allowed
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/some-import.ova",
+                expected => undef,
+            },
+            {
+                # Not allowed + exceeds max-scan-depth
+                file => "$DEFAULT_STORAGE_PATH/import/1/2/3/4/5/6/some-import.ova",
+                expected => undef,
+            },
         ],
     },
 ];
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index 0a425ee2..df639df3 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -64,6 +64,18 @@ my $tests = [
         expected =>
             ['import', 'import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
     },
+    {
+        description => "Import, ova file on disk with whitespace in name",
+        volname => 'import/My neat import.ova/disk.vmdk',
+        expected =>
+            ['import', 'My neat import.ova/disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
+    },
+    {
+        description => "Import, ova file on disk and inner file of ova with whitespace in name",
+        volname => 'import/My neat import.ova/OS disk.vmdk',
+        expected =>
+            ['import', 'My neat import.ova/OS disk.vmdk', undef, undef, undef, undef, 'ova+vmdk'],
+    },
     #
     # failed matches
     #
@@ -448,6 +460,57 @@ my $tests = [
 
         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $file_name = "$prefix.ova";
+
+        my @extra_failed_tests = (
+            {
+                description => "Import, ova, directory",
+                volname => "import/foo/$file_name",
+                expected => "unable to parse directory volume name 'import/foo/$file_name'\n",
+            },
+            {
+                description => "Import, ova, subdirectories",
+                volname => "import/foo/bar/baz/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/bar/baz/$file_name'\n",
+            },
+            {
+                description => "Import, ova, directory with same name as file",
+                volname => "import/$file_name/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$file_name'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference before volume type prefix",
+                volname => "../import/$file_name",
+                expected => "unable to parse directory volume name '../import/$file_name'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference at beginning of volume path",
+                volname => "import/../$file_name",
+                expected => "unable to parse directory volume name 'import/../$file_name'\n",
+            },
+            {
+                description => "Import, ova, parent directory reference at end of volume path",
+                volname => "import/$file_name/..",
+                expected => "unable to parse directory volume name 'import/$file_name/..'\n",
+            },
+            {
+                description =>
+                    "Import, ova, parent directory reference between dir components of volume path",
+                volname => "import/foo/../bar/$file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/../bar/$file_name'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }
 
 # Test cases for OVA import files with content inside
@@ -477,6 +540,63 @@ my $tests = [
 
         push($tests->@*, @extra_tests);
     }
+
+    # Failed tests
+    {
+        my $content_file_name = "disk.qcow2";
+
+        my @extra_failed_tests = (
+            {
+                description => "Import, inner file of ova (qcow2), directory",
+                volname => "import/foo/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2), nested directories",
+                volname => "import/foo/bar/baz/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/foo/bar/baz/$file_name/$content_file_name'\n",
+            },
+            {
+                description =>
+                    "Import, inner file of ova (qcow2), directory with same name as file",
+                volname => "import/$file_name/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference before volume type prefix",
+                volname => "../import/$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name '../import/$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at beginning of volume path",
+                volname => "import/../$file_name/$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/../$file_name/$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at end of disk path",
+                volname => "import/$file_name/../$content_file_name",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/../$content_file_name'\n",
+            },
+            {
+                description => "Import, inner file of ova (qcow2),"
+                    . " parent directory reference at end of volume path",
+                volname => "import/$file_name/$content_file_name/..",
+                expected =>
+                    "unable to parse directory volume name 'import/$file_name/$content_file_name/..'\n",
+            },
+        );
+
+        push($tests->@*, @extra_failed_tests);
+    }
 }
 
 my sub run_tests($tests) {
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index d22f6272..3bfb37a6 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -256,6 +256,19 @@ my $tests = [
         file => "$DEFAULT_STORAGE_DIR/import/test.foo",
         expected => [''],
     },
+    {
+        # Allowed in volume names, but not in file paths
+        description => 'Import, ova, whitespace in file name',
+        file => "$DEFAULT_STORAGE_DIR/import/My Import.ova",
+        expected => [''],
+    },
+    {
+        # Supporting subdirectories for the 'import' volume type is
+        # currently not possible because it would introduce parsing ambiguities.
+        description => 'Import, ova, subdirectory',
+        file => "$DEFAULT_STORAGE_DIR/import/foo/import.ova",
+        expected => [''],
+    },
 ];
 
 my sub run_tests($tests) {
-- 
2.47.3

[PATCH pve-storage v2 48/50] test: add tests guarding against subdir scanning for vtypes

[Max R. Carrara]

Add tests that guard against implementing nested subdirectory scanning
for the 'backup' volume type in `parser_tests.pl`, and for the
'images' and 'backup' volume types in `path_to_volume_id_test.pm`.

For the 'backup' vtype, also add a comment above the parser's
corresponding regex that we do not allow nested subdirs there.

This is done mainly to prevent implementing subdir scanning blindly in
the future for these two volume types.

While adding support for subdir scanning for the 'backup' volume type
would probably be fairly trivial, there is also no (known) need for it
currently.

For the 'images' type, there currently is no plan or need to support
subdir scanning, and it would probably not make much sense anyway.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 src/PVE/Storage/Common/Parse.pm             |  1 +
 src/PVE/Storage/Common/test/parser_tests.pl | 40 +++++++++++++++++++++
 src/test/path_to_volume_id_test.pm          | 11 ++++++
 3 files changed, 52 insertions(+)

diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm
index e64ac719..807ff901 100644
--- a/src/PVE/Storage/Common/Parse.pm
+++ b/src/PVE/Storage/Common/Parse.pm
@@ -106,6 +106,7 @@ my $RE_GUEST_VZDUMP_BACKUP_FILE_NAME = qr!
     - [^/]+
 !xn;
 
+# NOTE: Not supporting subdirectories here.
 my $RE_BACKUP_FILE_PATH = qr!
     (?<path>
         (?<file>
diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl
index 01daf1b7..d1c7c838 100755
--- a/src/PVE/Storage/Common/test/parser_tests.pl
+++ b/src/PVE/Storage/Common/test/parser_tests.pl
@@ -474,6 +474,46 @@ my $volname_cases_backup_invalid = [
         },
         expected => undef,
     },
+    {
+        description => "Subdirectory (backup)",
+        args => {
+            path => 'subdir/vzdump-qemu-16110-2020_03_30-21_12_40.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Nested subdirectory (backup)",
+        args => {
+            path => 'deeply/nested/subdir/vzdump-qemu-16110-2020_03_30-21_12_40.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (beginning) (backup)",
+        args => {
+            path => '../vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (middle) (backup)",
+        args => {
+            path => 'subdir/../vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
+    {
+        description => "Parent dir reference in path (end) (backup)",
+        args => {
+            path => 'subdir/vzdump-lxc-16112-2020_03_30-21_49_30.tar.gz/..',
+            vtype => 'backup',
+        },
+        expected => undef,
+    },
 ];
 
 my $volname_cases_snippets_valid = [
diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm
index 3bfb37a6..2eb17e9e 100644
--- a/src/test/path_to_volume_id_test.pm
+++ b/src/test/path_to_volume_id_test.pm
@@ -246,11 +246,22 @@ my $tests = [
         file => "$DEFAULT_STORAGE_DIR/dump/vzdump-qemu-16110-2020_03_30-21_12_40.vms.gz",
         expected => [''],
     },
+    {
+        description => 'Backup, nested subdirectories, vma.gz',
+        file =>
+            "$DEFAULT_STORAGE_DIR/dump/foo/bar/baz/vzdump-qemu-16110-2020_03_30-21_11_40.vma.gz",
+        expected => [''],
+    },
     {
         description => 'Image, string as vmid in folder name',
         file => "$DEFAULT_STORAGE_DIR/images/ssss/vm-1234-disk-0.qcow2",
         expected => [''],
     },
+    {
+        description => 'Image, additional nested subdirectories before vmid',
+        file => "$DEFAULT_STORAGE_DIR/images/foo/bar/baz/1234/vm-1234-disk-0.qcow2",
+        expected => [''],
+    },
     {
         description => 'Import, non ova/ovf/disk image in import dir',
         file => "$DEFAULT_STORAGE_DIR/import/test.foo",
-- 
2.47.3

[PATCH pve-manager v2 49/50] fix #2884: ui: storage: add field for 'max-scan-depth' property

[Max R. Carrara]

In the base storage panel, add a simple integer field in the advanced
section for storage types that use the new `max-scan-depth` property.

This means that the maximum directory scan depth can now be configured
for directories, NFS, CIFS, CephFS, and BTRFS.

Originally-by: Noel Ullreich <n.ullreich@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 www/manager6/storage/Base.js | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index 8961460b..0cf90337 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -75,6 +75,20 @@ Ext.define('PVE.panel.StorageBase', {
             });
         }
 
+        const subdirScanningStorageTypes = ['dir', 'nfs', 'cifs', 'cephfs', 'btrfs'];
+
+        if (subdirScanningStorageTypes.includes(me.type)) {
+            addAdvancedWidget({
+                xtype: 'proxmoxintegerfield',
+                name: 'max-scan-depth',
+                fieldLabel: gettext('Maximum Directory Scan Depth'),
+                allowBlank: false,
+                value: 0,
+                minValue: 0,
+                maxValue: 50,
+            });
+        }
+
         const externalStorageManagedSnapshotSupport = ['dir', 'nfs', 'cifs', 'lvm'];
 
         if (externalStorageManagedSnapshotSupport.includes(me.type)) {
-- 
2.47.3

[PATCH pve-manager v2 50/50] pve8to9: use helper from common storage module to get vtype subdir

[Max R. Carrara]

Use the `plugin_get_vtype_subdir()` helper from `PVE::Storage::Common`
to obtain the sub-directory for the provided content type (vtype),
instead of the now deprecated `get_subdir()` method.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/CLI/pve8to9.pm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/PVE/CLI/pve8to9.pm b/PVE/CLI/pve8to9.pm
index 97d564bd..a139eeca 100644
--- a/PVE/CLI/pve8to9.pm
+++ b/PVE/CLI/pve8to9.pm
@@ -23,6 +23,7 @@ use PVE::Network;
 use PVE::NodeConfig;
 use PVE::RPCEnvironment;
 use PVE::Storage;
+use PVE::Storage::Common;
 use PVE::Storage::Plugin;
 use PVE::Tools qw(run_command split_list file_get_contents trim);
 use PVE::QemuConfig;
@@ -1269,9 +1270,9 @@ sub check_storage_content_dirs {
         }
 
         my $resolved_subdirs = {};
-        my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
         for my $vtype (keys $scfg->{content}->%*) {
-            my $abs_subdir = Cwd::abs_path($plugin->get_subdir($scfg, $vtype));
+            my $abs_subdir =
+                Cwd::abs_path(PVE::Storage::Common::plugin_get_vtype_subdir($scfg, $vtype));
             next if !defined($abs_subdir);
             push $resolved_subdirs->{$abs_subdir}->@*, $vtype;
         }
-- 
2.47.3