From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 550831FF0C0 for ; Wed, 18 Nov 2026 10:14:31 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8FA8621597; Wed, 18 Nov 2026 10:14:24 +0100 (CET) From: "Max R. Carrara" To: pve-devel@lists.proxmox.com Subject: [PATCH pve-storage v3 46/51] fix #2884: support nested subdir scanning for 'snippets' vtype Date: Wed, 18 Nov 2026 10:14:08 +0100 Message-ID: <20261118091416.27183-2-m.carrara@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118091416.27183-1-m.carrara@proxmox.com> References: <20261117203953.913035-1-m.carrara@proxmox.com> <20261118091416.27183-1-m.carrara@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794993259093 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.237 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LDH3UYUZFM4JYKQ73325QIXZUQIQP7UE X-Message-ID-Hash: LDH3UYUZFM4JYKQ73325QIXZUQIQP7UE X-MailFrom: m.carrara@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Wolfgang Bumiller X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add support for parsing nested subdirectories for the 'snippets' volume type by adapting its corresponding regex used by parsing helpers. Add additional test cases wherever applicable to account for nested subdirectories for 'snippets' volumes, as was done for the 'iso' and 'vztmpl' volume types. Originally-by: Noel Ullreich Signed-off-by: Max R. Carrara --- src/PVE/Storage/Common/Parse.pm | 3 +- src/PVE/Storage/Common/test/parser_tests.pl | 52 ++++++++++++++ src/PVE/Storage/Plugin.pm | 4 +- src/test/filesystem_path_test.pm | 16 +++++ src/test/list_volumes_test.pm | 78 +++++++++++++++++++++ src/test/parse_volname_test.pm | 57 +++++++++++++++ src/test/path_to_volume_id_test.pm | 7 ++ 7 files changed, 214 insertions(+), 3 deletions(-) diff --git a/src/PVE/Storage/Common/Parse.pm b/src/PVE/Storage/Common/Parse.pm index 4137d09d..8ec2236d 100644 --- a/src/PVE/Storage/Common/Parse.pm +++ b/src/PVE/Storage/Common/Parse.pm @@ -140,10 +140,11 @@ my $RE_BACKUP_FILE_PATH = qr! ) !xn; my $RE_SNIPPETS_FILE_PATH = qr! (? + (? $RE_DIRECTORY_COMPONENTS )? (? [^/]+ ) ) !xn; my $RE_IMPORT_FILE_PATH = qr! @@ -346,11 +347,11 @@ These are: volname => 'iso/custom-debian.iso', } Volume types that support subdirectories in their C part also have a C part if a subdirectory is specified. This is currently limited to the -C and C volume types. +C, C and C volume types. The following volume types have more specific parts in addition to the common ones: =over diff --git a/src/PVE/Storage/Common/test/parser_tests.pl b/src/PVE/Storage/Common/test/parser_tests.pl index 20c57885..44aa7e88 100755 --- a/src/PVE/Storage/Common/test/parser_tests.pl +++ b/src/PVE/Storage/Common/test/parser_tests.pl @@ -590,10 +590,61 @@ my $volname_cases_snippets_valid = [ path => 'userconfig.yaml', vtype => 'snippets', volname => 'snippets/userconfig.yaml', }, }, + + # subdirectories + { + path => 'subdir/hookscript.pl', + expected => { + file => 'hookscript.pl', + dir => 'subdir', + 'disk-path' => 'subdir/hookscript.pl', + path => 'subdir/hookscript.pl', + vtype => 'snippets', + volname => 'snippets/subdir/hookscript.pl', + }, + }, + { + path => 'deeply/nested/subdir/hookscript.pl', + expected => { + file => 'hookscript.pl', + dir => 'deeply/nested/subdir', + 'disk-path' => 'deeply/nested/subdir/hookscript.pl', + path => 'deeply/nested/subdir/hookscript.pl', + vtype => 'snippets', + volname => 'snippets/deeply/nested/subdir/hookscript.pl', + }, + }, +]; + +my $volname_cases_snippets_invalid = [ + { + description => "Parent dir reference in path (beginning) (snippets)", + args => { + path => '../hookscript.pl', + vtype => 'snippets', + }, + expected => undef, + }, + { + description => "Parent dir reference in path (middle) (snippets)", + args => { + path => 'subdir/../hookscript.pl', + vtype => 'snippets', + }, + expected => undef, + }, + { + description => "Parent dir reference in path (end) (snippets)", + args => { + path => 'subdir/hookscript.pl/..', + vtype => 'snippets', + }, + expected => undef, + }, ]; my $volname_cases_import_valid = [ { path => 'import.ova', @@ -716,10 +767,11 @@ my $cases_valid_all = [ my $cases_invalid_all = [ $volname_cases_iso_invalid, $volname_cases_vztmpl_invalid, $volname_cases_backup_invalid, + $volname_cases_snippets_invalid, $volname_cases_import_invalid, $volname_cases_newline_invalid, ]; { diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index c932ec16..9c57cf37 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -274,11 +274,11 @@ my $defaultData = { optional => 1, }, 'max-scan-depth' => { title => "Maximum Directory Scan Depth", description => "Maximum depth of subdirectories to traverse when searching for" - . " ISOs and container templates in directories.", + . " ISOs, container templates and snippets in directories.", type => 'integer', default => 0, minimum => 0, maximum => 50, optional => 1, @@ -2119,11 +2119,11 @@ sub list_volumes { if ($type eq 'backup') { return get_subdir_files($class, $storeid, $scfg, 'backup', $vmid); } if ($type eq 'snippets') { - return get_subdir_files($class, $storeid, $scfg, 'snippets', undef); + return get_subdir_files($class, $storeid, $scfg, 'snippets', undef, $depth); } if ($type eq 'import') { return get_subdir_files($class, $storeid, $scfg, 'import', undef); } diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm index a511f4ef..06789187 100644 --- a/src/test/filesystem_path_test.pm +++ b/src/test/filesystem_path_test.pm @@ -116,10 +116,26 @@ my $tests = [ snapname => undef, expected => [ "$DEFAULT_STORAGE_DIR/private/subvol-1234-disk-0", 1234, 'rootdir', ], }, + { + volname => 'snippets/userconfig.yaml', + snapname => undef, + expected => ["$DEFAULT_STORAGE_DIR/snippets/userconfig.yaml", undef, 'snippets'], + }, + { + volname => 'snippets/hookscript.pl', + snapname => undef, + expected => ["$DEFAULT_STORAGE_DIR/snippets/hookscript.pl", undef, 'snippets'], + }, + { + volname => 'snippets/foo/bar/baz/something.txt', + snapname => undef, + expected => + ["$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", undef, 'snippets'], + }, ]; my sub run_tests($tests) { for my $t ($tests->@*) { my $volname = $t->{volname}; diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm index 6d8d4faf..c93b1634 100644 --- a/src/test/list_volumes_test.pm +++ b/src/test/list_volumes_test.pm @@ -1406,10 +1406,28 @@ my $test_param_list = [ { file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/some-lxc-template.tar.gz", expected => undef, }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/some-hookscript.pl', + }, + }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl", + expected => undef, + }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl", + expected => undef, + }, ], }, { description => "VMID: none, nested subdirectories allowed, max-scan-depth = 1", storeid => $DEFAULT_STOREID, @@ -1476,10 +1494,35 @@ my $test_param_list = [ { # Exceeds max-scan-depth file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/some-lxc-template.tar.gz", expected => undef, }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/some-hookscript.pl', + }, + }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/1/some-hookscript.pl', + }, + }, + { + # Exceeds max-scan-depth + file => "$DEFAULT_STORAGE_PATH/snippets/1/2/some-hookscript.pl", + expected => undef, + }, ], }, { description => "VMID: none, nested subdirectories allowed, max-scan-depth = 5", storeid => $DEFAULT_STOREID, @@ -1568,10 +1611,45 @@ my $test_param_list = [ # Exceeds max-scan-depth file => "$DEFAULT_STORAGE_PATH/template/cache/1/2/3/4/5/6/some-lxc-template.tar.gz", expected => undef, }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/some-hookscript.pl', + }, + }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/1/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/1/some-hookscript.pl', + }, + }, + { + file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/some-hookscript.pl", + expected => { + content => 'snippets', + ctime => $DEFAULT_CTIME, + format => 'snippet', + size => $DEFAULT_SIZE, + volid => 'local:snippets/1/2/3/4/5/some-hookscript.pl', + }, + }, + { + # Exceeds max-scan-depth + file => "$DEFAULT_STORAGE_PATH/snippets/1/2/3/4/5/6/some-hookscript.pl", + expected => undef, + }, ], }, { description => "VMID: none, max-scan-depth = 2, nested content dir of snippets", storeid => $DEFAULT_STOREID, diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm index 37f2f705..f2aa0344 100644 --- a/src/test/parse_volname_test.pm +++ b/src/test/parse_volname_test.pm @@ -694,14 +694,71 @@ for my $plugin_name (qw(ZFSPlugin ZFSPoolPlugin)) { { description => "Snippets, $file_name", volname => "snippets/$file_name", expected => ['snippets', $file_name, undef, undef, undef, undef, 'raw'], }, + { + description => "Snippets, $file_name, subdirectory", + volname => "snippets/foo/$file_name", + expected => ['snippets', "foo/$file_name", undef, undef, undef, undef, 'raw'], + }, + { + description => "Snippets, $file_name, nested subdirectories", + volname => "snippets/foo/bar/baz/$file_name", + expected => + ['snippets', "foo/bar/baz/$file_name", undef, undef, undef, undef, 'raw'], + }, + { + description => "Snippets, $file_name, subdirectory with same name as file", + volname => "snippets/$file_name/$file_name", + expected => + ['snippets', "$file_name/$file_name", undef, undef, undef, undef, 'raw'], + }, ); push($tests->@*, @extra_tests); } + + # Expected to fail + { + my $file_name = "some-file.txt"; + + my @extra_failed_tests = ( + { + description => + "Snippets, $file_name, parent directory reference before volume type prefix", + volname => "../snippets/$file_name", + expected => "unable to parse directory volume name '../snippets/$file_name'\n", + }, + { + description => + "Snippets, $file_name, parent directory reference at beginning of volume path", + volname => "snippets/../$file_name", + expected => "unable to parse directory volume name 'snippets/../$file_name'\n", + }, + { + description => "Snippets, parent directory reference as entire path", + volname => "snippets/..", + expected => "unable to parse directory volume name 'snippets/..'\n", + }, + { + description => + "Snippets, $file_name, parent directory reference at end of volume path", + volname => "snippets/$file_name/..", + expected => "unable to parse directory volume name 'snippets/$file_name/..'\n", + }, + { + description => + "Snippets, $file_name, parent directory reference between dir components of volume path", + volname => "snippets/foo/../bar/$file_name", + expected => + "unable to parse directory volume name 'snippets/foo/../bar/$file_name'\n", + }, + ); + + push($tests->@*, @extra_failed_tests); + } } # Test cases for import files { my $suffixes = ['ova', 'ovf', 'qcow2', 'vmdk', 'raw']; diff --git a/src/test/path_to_volume_id_test.pm b/src/test/path_to_volume_id_test.pm index 2f45c9b4..dcd05111 100644 --- a/src/test/path_to_volume_id_test.pm +++ b/src/test/path_to_volume_id_test.pm @@ -179,10 +179,17 @@ my $tests = [ file => "$DEFAULT_STORAGE_DIR/snippets/hookscript.pl", expected => [ 'snippets', 'local:snippets/hookscript.pl', ], }, + { + description => 'Snippets, arbitrary file, nested subdirectories', + file => "$DEFAULT_STORAGE_DIR/snippets/foo/bar/baz/something.txt", + expected => [ + 'snippets', 'local:snippets/foo/bar/baz/something.txt', + ], + }, { description => 'CT template, tar.xz', file => "$DEFAULT_STORAGE_DIR/template/cache/debian-10.0-standard_10.0-1_amd64.tar.xz", expected => [ 'vztmpl', 'local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz', -- 2.47.3