[pve-devel] [PATCH pve-storage/pve-manager v3 0/4] fix #623: show isos/vztmpl/snippets in subdirs

[pve-devel] [PATCH pve-storage/pve-manager v3 0/4] fix #623: show isos/vztmpl/snippets in subdirs

[Noel Ullreich]

This patch fixes #623, allowing isos/vztmpl/snippets in subdirectories.
This feature is opt-in and can be set from the API, web interface or
with `pvesm`.

I addressed the security concerns raised by Fabian, now parent
directories in the path (i.e. `/my/path/../somewhere/`) are forbidded.
I have kept the permission to use symlinks, however, if this is a
security issue, symlinks can easily be forbidden as well. This,
however, would be a breaking change.

parts of the tests as well as the regex for checking, if a `/../` is in
the path have been taken and/or adapted from an older patch that was
never merged:
https://lists.proxmox.com/pipermail/pve-devel/2020-May/043622.html

This is a complete rework from v1, so I don't see a point in writing
what the differences are. It's all different.

----
changes from v2:
* rebased so that applying with new structure in pve-storage works 
(/PVE was moved to /src/PVE/)
* fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)

Noel Ullreich (4):

pve-storage:
  recursively go through subdirs to find files
  add `subdir-depth` option to filesystems
  update test for recursive subdir search

 src/PVE/Storage.pm                 |  7 +++
 src/PVE/Storage/CIFSPlugin.pm      |  1 +
 src/PVE/Storage/CephFSPlugin.pm    |  1 +
 src/PVE/Storage/DirPlugin.pm       |  1 +
 src/PVE/Storage/GlusterfsPlugin.pm |  1 +
 src/PVE/Storage/NFSPlugin.pm       |  1 +
 src/PVE/Storage/Plugin.pm          | 63 +++++++++++++++++----------
 src/test/filesystem_path_test.pm   | 18 ++++++++
 src/test/list_volumes_test.pm      | 68 ++++++++++++++++++++++++++++++
 src/test/parse_volname_test.pm     | 40 ++++++++++++++++++
 10 files changed, 179 insertions(+), 22 deletions(-)

pve-manager:
 www/manager6/storage/Base.js | 11 +++++++++++
 1 file changed, 11 insertions(+)
-- 
2.30.2

[pve-devel] [PATCH pve-storage v3 1/3] recursively go through subdirs to find files

[Noel Ullreich]

This patch allows `get_subdir_files` to recursively call itself, so that
subdirectories of set depth can be searched. We allow searching for
isos, vztmpl and snippets but not backups.

As a security measure, when parsing a given path, parent
directories (`/../`) are forbidden.

The feature is opt-in, i.e. the searchdepth is 0 by default. It can be
changed via the API, the web interface and `pvesm` (see the other
patches).

Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
---
changes from v2:
* fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)

 src/PVE/Storage.pm        |  7 +++++
 src/PVE/Storage/Plugin.pm | 56 ++++++++++++++++++++++++---------------
 2 files changed, 41 insertions(+), 22 deletions(-)

diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index b99ed35..250a892 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -113,6 +113,13 @@ our $VZTMPL_EXT_RE_1 = qr/\.tar\.(gz|xz|zst)/i;
 
 our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
 
+# '..' is forbidden at the beginning, between two '/' and at the end
+my $dots = quotemeta('..');
+my $beginning = qr!^$dots/!;
+my $between = qr!/$dots/!;
+my $end = qr!/$dots$!;
+our $forbidden_double_dots_re = qr!(?:$beginning|$between|$end)!;
+
 # FIXME remove with PVE 8.0, add versioned breaks for pve-manager
 our $vztmpl_extension_re = $VZTMPL_EXT_RE_1;
 
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index ab6b675..87a08c3 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -621,6 +621,8 @@ sub parse_name_dir {
 sub parse_volname {
     my ($class, $volname) = @_;
 
+    die "volname must not contain parent directories '/../'\n" if $volname =~ $PVE::Storage::forbidden_double_dots_re;
+
     if ($volname =~ m!^(\d+)/(\S+)/(\d+)/(\S+)$!) {
 	my ($basedvmid, $basename) = ($1, $2);
 	parse_name_dir($basename);
@@ -631,9 +633,9 @@ sub parse_volname {
 	my ($vmid, $name) = ($1, $2);
 	my (undef, $format, $isBase) = parse_name_dir($name);
 	return ('images', $name, $vmid, undef, undef, $isBase, $format);
-    } elsif ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
+    } elsif ($volname =~ m!^iso/((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
 	return ('iso', $1);
-    } elsif ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
+    } elsif ($volname =~ m!^vztmpl/((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
 	return ('vztmpl', $1);
     } elsif ($volname =~ m!^rootdir/(\d+)$!) {
 	return ('rootdir', $1, $1);
@@ -643,7 +645,7 @@ sub parse_volname {
 	    return ('backup', $fn, $2);
 	}
 	return ('backup', $fn);
-    } elsif ($volname =~ m!^snippets/([^/]+)$!) {
+    } elsif ($volname =~ m!^snippets/((?:[0-9A-z\_\-\.]+\/)*[^\/]+)$!) {
 	return ('snippets', $1);
     }
 
@@ -1212,28 +1214,33 @@ sub list_images {
 }
 
 # list templates ($tt = <iso|vztmpl|backup|snippets>)
-my $get_subdir_files = sub {
-    my ($sid, $path, $tt, $vmid) = @_;
+sub get_subdir_files {
+    my ($sid, $path, $tt, $scfg, $vmid, $remaining_depth) = @_;
+    my $storage_path = $scfg->{path};
+    my $content_dir = $scfg->{"content-dirs"}->{$tt} // $vtype_subdirs->{$tt};
 
     my $res = [];
 
     foreach my $fn (<$path/*>) {
-	my $st = File::stat::stat($fn);
+	my $st = File::stat::lstat($fn);
+
+	next if (!$st);
 
-	next if (!$st || S_ISDIR($st->mode));
+	if (S_ISDIR($st->mode)) {
+	    if ($remaining_depth) {
+		push @$res, get_subdir_files($sid, $fn, $tt, $scfg, $vmid, $remaining_depth-1);
+	    }
+	    next;
+	}
 
 	my $info;
 
 	if ($tt eq 'iso') {
-	    next if $fn !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
-
+	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::ISO_EXT_RE_0)/;
 	    $info = { volid => "$sid:iso/$1", format => 'iso' };
-
 	} elsif ($tt eq 'vztmpl') {
-	    next if $fn !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
-
+	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::VZTMPL_EXT_RE_1)/;
 	    $info = { volid => "$sid:vztmpl/$1", format => "t$2" };
-
 	} elsif ($tt eq 'backup') {
 	    next if $fn !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
 	    my $original = $fn;
@@ -1262,9 +1269,9 @@ my $get_subdir_files = sub {
 
 	    $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
 	} elsif ($tt eq 'snippets') {
-
+	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*.+)/;
 	    $info = {
-		volid => "$sid:snippets/". basename($fn),
+		volid => "$sid:snippets/$1", #basename($fn),
 		format => 'snippet',
 	    };
 	}
@@ -1274,14 +1281,18 @@ my $get_subdir_files = sub {
 
 	push @$res, $info;
     }
-
     return $res;
 };
 
+sub flatten {
+    map { ref eq 'ARRAY' ? flatten(@{$_}) : $_ } @_;
+}
+
 # If attributes are set on a volume, they should be included in the result.
 # See get_volume_attribute for a list of possible attributes.
 sub list_volumes {
     my ($class, $storeid, $scfg, $vmid, $content_types) = @_;
+    my $max_depth = $scfg->{'subdir-depth'} // 0;
 
     my $res = [];
     my $vmlist = PVE::Cluster::get_vmlist();
@@ -1294,17 +1305,19 @@ sub list_volumes {
 	    my $path = $class->get_subdir($scfg, $type);
 
 	    if ($type eq 'iso' && !defined($vmid)) {
-		$data = $get_subdir_files->($storeid, $path, 'iso');
+		$data = get_subdir_files($storeid, $path, 'iso', $scfg, undef, $max_depth);
 	    } elsif ($type eq 'vztmpl'&& !defined($vmid)) {
-		$data = $get_subdir_files->($storeid, $path, 'vztmpl');
+		$data = get_subdir_files($storeid, $path , 'vztmpl', $scfg, undef, $max_depth);
 	    } elsif ($type eq 'backup') {
-		$data = $get_subdir_files->($storeid, $path, 'backup', $vmid);
+		$data = get_subdir_files($storeid, $path, 'backup', $scfg, $vmid, $max_depth);
 	    } elsif ($type eq 'snippets') {
-		$data = $get_subdir_files->($storeid, $path, 'snippets');
+		$data = get_subdir_files($storeid, $path, 'snippets', $scfg, undef, $max_depth);
 	    }
 	}
 
-	next if !$data;
+	$data = [flatten($data)];
+
+	next if !@$data[0];
 
 	foreach my $item (@$data) {
 	    if ($type eq 'images' || $type eq 'rootdir') {
@@ -1322,7 +1335,6 @@ sub list_volumes {
 	    } else {
 		$item->{content} = $type;
 	    }
-
 	    push @$res, $item;
 	}
     }
-- 
2.30.2

[pve-devel] [PATCH pve-storage v3 2/3] add `subdir-depth` option to filesystems

[Noel Ullreich]

Add the `subdir-depth` to the filesystems that can hold
isos/vztmpl/snippets.

Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
---
 src/PVE/Storage/CIFSPlugin.pm      | 1 +
 src/PVE/Storage/CephFSPlugin.pm    | 1 +
 src/PVE/Storage/DirPlugin.pm       | 1 +
 src/PVE/Storage/GlusterfsPlugin.pm | 1 +
 src/PVE/Storage/NFSPlugin.pm       | 1 +
 src/PVE/Storage/Plugin.pm          | 7 +++++++
 6 files changed, 12 insertions(+)

diff --git a/src/PVE/Storage/CIFSPlugin.pm b/src/PVE/Storage/CIFSPlugin.pm
index 71b85aa..1ac8fcf 100644
--- a/src/PVE/Storage/CIFSPlugin.pm
+++ b/src/PVE/Storage/CIFSPlugin.pm
@@ -155,6 +155,7 @@ sub options {
 	'create-subdirs' => { optional => 1 },
 	bwlimit => { optional => 1 },
 	preallocation => { optional => 1 },
+	'subdir-depth' => { optional => 1},
 	options => { optional => 1 },
     };
 }
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index 8aad518..7f0f5b1 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -156,6 +156,7 @@ sub options {
 	'prune-backups' => { optional => 1 },
 	'max-protected-backups' => { optional => 1 },
 	'fs-name' => { optional => 1 },
+	'subdir-depth' => { optional => 1},
     };
 }
 
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 2efa8d5..5f3739b 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -80,6 +80,7 @@ sub options {
 	is_mountpoint => { optional => 1 },
 	bwlimit => { optional => 1 },
 	preallocation => { optional => 1 },
+	'subdir-depth' => { optional => 1},
    };
 }
 
diff --git a/src/PVE/Storage/GlusterfsPlugin.pm b/src/PVE/Storage/GlusterfsPlugin.pm
index 2b7f9e1..e151209 100644
--- a/src/PVE/Storage/GlusterfsPlugin.pm
+++ b/src/PVE/Storage/GlusterfsPlugin.pm
@@ -141,6 +141,7 @@ sub options {
 	'create-subdirs' => { optional => 1 },
 	bwlimit => { optional => 1 },
 	preallocation => { optional => 1 },
+	'subdir-depth' => { optional => 1},
     };
 }
 
diff --git a/src/PVE/Storage/NFSPlugin.pm b/src/PVE/Storage/NFSPlugin.pm
index f2e4c0d..dd88fe3 100644
--- a/src/PVE/Storage/NFSPlugin.pm
+++ b/src/PVE/Storage/NFSPlugin.pm
@@ -91,6 +91,7 @@ sub options {
 	'create-subdirs' => { optional => 1 },
 	bwlimit => { optional => 1 },
 	preallocation => { optional => 1 },
+	'subdir-depth' => { optional => 1},
     };
 }
 
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 87a08c3..f289af0 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -192,6 +192,13 @@ my $defaultData = {
 	    type => "string", format => "pve-dir-override-list",
 	    optional => 1,
 	},
+	'subdir-depth' => {
+	    description => "Maximal depth of subdirectories to look though when
+	    searching for isos/container templates/snippets in a directory",
+	    type => 'integer',
+	    default => 0,
+	    optional =>	1,
+	},
 	options => {
 	    description => "NFS/CIFS mount options (see 'man nfs' or 'man mount.cifs')",
 	    type => 'string',
-- 
2.30.2

[pve-devel] [PATCH pve-storage v3 3/3] update test for recursive subdir search

[Noel Ullreich]

Tests now also test if traversing subdirectories works. That means
checking if:
 - parent directories in paths are caught
 - checking that nested files are found
 - checking that files below the maximum depth set are not found

Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
---
 src/test/filesystem_path_test.pm | 18 +++++++++
 src/test/list_volumes_test.pm    | 68 ++++++++++++++++++++++++++++++++
 src/test/parse_volname_test.pm   | 40 +++++++++++++++++++
 3 files changed, 126 insertions(+)

diff --git a/src/test/filesystem_path_test.pm b/src/test/filesystem_path_test.pm
index c1b6d90..35a1d4e 100644
--- a/src/test/filesystem_path_test.pm
+++ b/src/test/filesystem_path_test.pm
@@ -56,6 +56,24 @@ my $tests = [
 	    'iso'
 	],
     },
+    {
+	volname  => 'iso/a/b/c/my-awesome-proxmox.iso',
+	snapname => undef,
+	expected => [
+	    "$path/template/iso/a/b/c/my-awesome-proxmox.iso",
+	    undef,
+	    'iso'
+	],
+    },
+    {
+	volname  => 'vztmpl/a/b/c/my-awesome-template.tar.gz',
+	snapname => undef,
+	expected => [
+	    "$path/template/cache/a/b/c/my-awesome-template.tar.gz",
+	    undef,
+	    'vztmpl'
+	],
+    },
     {
 	volname  => "backup/vzdump-qemu-1234-2020_03_30-21_12_40.vma",
 	snapname => undef,
diff --git a/src/test/list_volumes_test.pm b/src/test/list_volumes_test.pm
index d155cb9..523639b 100644
--- a/src/test/list_volumes_test.pm
+++ b/src/test/list_volumes_test.pm
@@ -74,6 +74,7 @@ my $scfg = {
 	'snippets' => 1,
 	'backup'   => 1,
     },
+    'subdir-depth' => 10,
 };
 
 # The test cases are comprised of an arry of hashes with the following keys:
@@ -448,6 +449,73 @@ my @tests = (
 	],
 	expected => [], # returns empty list
     },
+    {
+	descripton => 'VMID: none, subdirectory',
+	# find files in subdirectories, test if maximum depth is working
+	vmid => undef,
+	files => [
+	    "$storage_dir/template/cache/dir1/dir2/uwuntu-3.10-default_20190626_amd64.tar.xz",
+	    "$storage_dir/template/iso/a/b/c/installation-disk.iso",
+	    "$storage_dir/template/iso/a/b/c/d.d/yet-again-an-installation-disk.iso",
+	    "$storage_dir/template/iso/a/b/template/iso/c/d/another-installation-disk.iso",
+	    "$storage_dir/template/iso/1/2/3/4/5/6/7/8/9/10//wunkus.iso", # test depth
+	    "$storage_dir/template/iso/1/2/3/4/5/6/7/8/9/10/11/wunkus.iso", # this should not match
+	    "$storage_dir/template/iso/a/../somedir/little.iso",
+	    "$storage_dir/template/iso/a/../../somedir/little.iso", # this should not match
+	    "$storage_dir/snippets/a/b/c/d/userconfig.yaml",
+	],
+	expected => [
+	    {
+		'content' => 'vztmpl',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'txz',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:vztmpl/dir1/dir2/uwuntu-3.10-default_20190626_amd64.tar.xz',
+	    },
+	    {
+		'content' => 'iso',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'iso',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:iso/1/2/3/4/5/6/7/8/9/10/wunkus.iso',
+	    },
+	    {
+		'content' => 'iso',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'iso',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:iso/a/b/c/d.d/yet-again-an-installation-disk.iso',
+	    },
+	    {
+		'content' => 'iso',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'iso',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:iso/a/b/c/installation-disk.iso',
+	    },
+	    {
+		'content' => 'iso',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'iso',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:iso/a/b/template/iso/c/d/another-installation-disk.iso',
+	    },
+	    {
+		'content' => 'iso',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'iso',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:iso/somedir/little.iso',
+	    },
+	    {
+		'content' => 'snippets',
+		'ctime'   => DEFAULT_CTIME,
+		'format'  => 'snippet',
+		'size'    => DEFAULT_SIZE,
+		'volid'   => 'local:snippets/a/b/c/d/userconfig.yaml',
+	    },
+	],
+    }
 );
 
 
diff --git a/src/test/parse_volname_test.pm b/src/test/parse_volname_test.pm
index d6ac885..737ae9e 100644
--- a/src/test/parse_volname_test.pm
+++ b/src/test/parse_volname_test.pm
@@ -31,11 +31,26 @@ my $tests = [
 	volname     => 'iso/some-installation-disk.iso',
 	expected    => ['iso', 'some-installation-disk.iso'],
     },
+    {
+	description => 'ISO image, iso, nested',
+	volname     => 'iso/a/b/c/some-installation-disk.iso',
+	expected    => ['iso', 'a/b/c/some-installation-disk.iso'],
+    },
     {
 	description => 'ISO image, img',
 	volname     => 'iso/some-other-installation-disk.img',
 	expected    => ['iso', 'some-other-installation-disk.img'],
     },
+    {
+	description => 'ISO image, img, nested',
+	volname     => 'iso/a/b/c/some-other-installation-disk.img',
+	expected    => ['iso', 'a/b/c/some-other-installation-disk.img'],
+    },
+    {
+	description => 'ISO image, img, nested with dots',
+	volname     => 'iso/a/b.b/c/some-other-installation-disk.img',
+	expected    => ['iso', 'a/b.b/c/some-other-installation-disk.img'],
+    },
     #
     # container templates
     #
@@ -44,11 +59,21 @@ my $tests = [
 	volname     => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz',
 	expected    => ['vztmpl', 'debian-10.0-standard_10.0-1_amd64.tar.gz'],
     },
+    {
+	description => 'Container template tar.gz nested',
+	volname     => 'vztmpl/a/b/c/debian-10.0-standard_10.0-1_amd64.tar.gz',
+	expected    => ['vztmpl', 'a/b/c/debian-10.0-standard_10.0-1_amd64.tar.gz'],
+    },
     {
 	description => 'Container template tar.xz',
 	volname     => 'vztmpl/debian-10.0-standard_10.0-1_amd64.tar.xz',
 	expected    => ['vztmpl', 'debian-10.0-standard_10.0-1_amd64.tar.xz'],
     },
+    {
+	description => 'Container template tar.xz nested',
+	volname     => 'vztmpl/a/b/c/debian-10.0-standard_10.0-1_amd64.tar.xz',
+	expected    => ['vztmpl', 'a/b/c/debian-10.0-standard_10.0-1_amd64.tar.xz'],
+    },
     #
     # container rootdir
     #
@@ -123,6 +148,21 @@ my $tests = [
 	volname     => "$vmid/base-$vmid-disk-0.qcow2/ssss/vm-$vmid-disk-0.qcow2",
 	expected    => "unable to parse volume filename 'base-$vmid-disk-0.qcow2/ssss/vm-$vmid-disk-0.qcow2'\n",
     },
+    {
+	description => 'Failed match: ISO image, forbidden double dots between two /',
+	volname     => 'iso/a/../c/some-installation-disk.iso',
+	expected    => "volname must not contain parent directories '/../'\n",
+    },
+    {
+	description => 'Failed match: ISO image, forbidden double dots in the beginning',
+	volname     => '../iso/some-installation-disk.iso',
+	expected    => "volname must not contain parent directories '/../'\n",
+    },
+    {
+	description => 'Failed match: ISO image, forbidden double dots in the end',
+	volname     => 'iso/some-installation-disk.iso/..',
+	expected    => "volname must not contain parent directories '/../'\n",
+    },
 ];
 
 # create more test cases for VM disk images matches
-- 
2.30.2

[pve-devel] [PATCH pve-manager v3]ui: add field to set subdir-depth in web interface

[Noel Ullreich]

When adding or editing a storage device in Datacenter->Storage in the
web interface, the subdirectory depth can be set in the advanced menu.

Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
---
 www/manager6/storage/Base.js | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index c8c735f31..c6844426d 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -48,6 +48,7 @@ Ext.define('PVE.panel.StorageBase', {
 		name: 'enable',
 		checked: true,
 		uncheckedValue: 0,
+		minValue: 0,
 		fieldLabel: gettext('Enable'),
 	    },
 	);
@@ -63,13 +64,23 @@ Ext.define('PVE.panel.StorageBase', {
 		deleteEmpty: !me.isCreate,
 		value: '__default__',
 	    };
+	    const recursionDepth = {
+		xtype: 'proxmoxintegerfield',
+		name: 'subdir-depth',
+		fieldLabel: gettext('Subdirectory Depth'),
+		allowBlank: false,
+		value: 0,
+		minValue: 0,
+	    };
 
 	    me.advancedColumn1 = me.advancedColumn1 || [];
 	    me.advancedColumn2 = me.advancedColumn2 || [];
 	    if (me.advancedColumn2.length < me.advancedColumn1.length) {
 		me.advancedColumn2.unshift(preallocSelector);
+		me.advancedColumn2.unshift(recursionDepth);
 	    } else {
 		me.advancedColumn1.unshift(preallocSelector);
+		me.advancedColumn1.unshift(recursionDepth);
 	    }
 	}
 
-- 
2.30.2

Re: [pve-devel] [PATCH pve-storage/pve-manager v3 0/4] fix #623: show isos/vztmpl/snippets in subdirs

[Markus Frank]

Works fine.
Seems to be a good addition for organizing ISOs, vztmpl, etc.

You could add a selector for a directory or a "destination path"
text-input to the file upload as a follow-up patch.

Tested-by: Markus Frank <m.frank@proxmox.com>

On 6/15/23 14:03, Noel Ullreich wrote:
> This patch fixes #623, allowing isos/vztmpl/snippets in subdirectories.
> This feature is opt-in and can be set from the API, web interface or
> with `pvesm`.
> 
> I addressed the security concerns raised by Fabian, now parent
> directories in the path (i.e. `/my/path/../somewhere/`) are forbidded.
> I have kept the permission to use symlinks, however, if this is a
> security issue, symlinks can easily be forbidden as well. This,
> however, would be a breaking change.
> 
> parts of the tests as well as the regex for checking, if a `/../` is in
> the path have been taken and/or adapted from an older patch that was
> never merged:
> https://lists.proxmox.com/pipermail/pve-devel/2020-May/043622.html
> 
> This is a complete rework from v1, so I don't see a point in writing
> what the differences are. It's all different.
> 
> ----
> changes from v2:
> * rebased so that applying with new structure in pve-storage works
> (/PVE was moved to /src/PVE/)
> * fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)
> 
> Noel Ullreich (4):
> 
> pve-storage:
>    recursively go through subdirs to find files
>    add `subdir-depth` option to filesystems
>    update test for recursive subdir search
> 
>   src/PVE/Storage.pm                 |  7 +++
>   src/PVE/Storage/CIFSPlugin.pm      |  1 +
>   src/PVE/Storage/CephFSPlugin.pm    |  1 +
>   src/PVE/Storage/DirPlugin.pm       |  1 +
>   src/PVE/Storage/GlusterfsPlugin.pm |  1 +
>   src/PVE/Storage/NFSPlugin.pm       |  1 +
>   src/PVE/Storage/Plugin.pm          | 63 +++++++++++++++++----------
>   src/test/filesystem_path_test.pm   | 18 ++++++++
>   src/test/list_volumes_test.pm      | 68 ++++++++++++++++++++++++++++++
>   src/test/parse_volname_test.pm     | 40 ++++++++++++++++++
>   10 files changed, 179 insertions(+), 22 deletions(-)
> 
> pve-manager:
>   www/manager6/storage/Base.js | 11 +++++++++++
>   1 file changed, 11 insertions(+)

Re: [pve-devel] [PATCH pve-storage/pve-manager v3 0/4] fix #623: show isos/vztmpl/snippets in subdirs

[Noel Ullreich]

Ping. Still applies

With me leaving at the end of the month, I want to make sure there is 
plenty of time for my outstanding patches to be reviewed, updated, and 
applied.

On 15-06-2023 14:03, Noel Ullreich wrote:
> This patch fixes #623, allowing isos/vztmpl/snippets in subdirectories.
> This feature is opt-in and can be set from the API, web interface or
> with `pvesm`.
>
> I addressed the security concerns raised by Fabian, now parent
> directories in the path (i.e. `/my/path/../somewhere/`) are forbidded.
> I have kept the permission to use symlinks, however, if this is a
> security issue, symlinks can easily be forbidden as well. This,
> however, would be a breaking change.
>
> parts of the tests as well as the regex for checking, if a `/../` is in
> the path have been taken and/or adapted from an older patch that was
> never merged:
> https://lists.proxmox.com/pipermail/pve-devel/2020-May/043622.html
>
> This is a complete rework from v1, so I don't see a point in writing
> what the differences are. It's all different.
>
> ----
> changes from v2:
> * rebased so that applying with new structure in pve-storage works
> (/PVE was moved to /src/PVE/)
> * fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)
>
> Noel Ullreich (4):
>
> pve-storage:
>    recursively go through subdirs to find files
>    add `subdir-depth` option to filesystems
>    update test for recursive subdir search
>
>   src/PVE/Storage.pm                 |  7 +++
>   src/PVE/Storage/CIFSPlugin.pm      |  1 +
>   src/PVE/Storage/CephFSPlugin.pm    |  1 +
>   src/PVE/Storage/DirPlugin.pm       |  1 +
>   src/PVE/Storage/GlusterfsPlugin.pm |  1 +
>   src/PVE/Storage/NFSPlugin.pm       |  1 +
>   src/PVE/Storage/Plugin.pm          | 63 +++++++++++++++++----------
>   src/test/filesystem_path_test.pm   | 18 ++++++++
>   src/test/list_volumes_test.pm      | 68 ++++++++++++++++++++++++++++++
>   src/test/parse_volname_test.pm     | 40 ++++++++++++++++++
>   10 files changed, 179 insertions(+), 22 deletions(-)
>
> pve-manager:
>   www/manager6/storage/Base.js | 11 +++++++++++
>   1 file changed, 11 insertions(+)

Re: [pve-devel] [PATCH pve-storage v3 1/3] recursively go through subdirs to find files

[Fabian Grünbichler]

On June 15, 2023 2:03 pm, Noel Ullreich wrote:
> This patch allows `get_subdir_files` to recursively call itself, so that
> subdirectories of set depth can be searched. We allow searching for
> isos, vztmpl and snippets but not backups.
> 
> As a security measure, when parsing a given path, parent
> directories (`/../`) are forbidden.
> 
> The feature is opt-in, i.e. the searchdepth is 0 by default. It can be
> changed via the API, the web interface and `pvesm` (see the other
> patches).
> 
> Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
> ---
> changes from v2:
> * fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)
> 
>  src/PVE/Storage.pm        |  7 +++++
>  src/PVE/Storage/Plugin.pm | 56 ++++++++++++++++++++++++---------------
>  2 files changed, 41 insertions(+), 22 deletions(-)
> 
> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
> index b99ed35..250a892 100755
> --- a/src/PVE/Storage.pm
> +++ b/src/PVE/Storage.pm
> @@ -113,6 +113,13 @@ our $VZTMPL_EXT_RE_1 = qr/\.tar\.(gz|xz|zst)/i;
>  
>  our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
>  
> +# '..' is forbidden at the beginning, between two '/' and at the end
> +my $dots = quotemeta('..');
> +my $beginning = qr!^$dots/!;
> +my $between = qr!/$dots/!;
> +my $end = qr!/$dots$!;
> +our $forbidden_double_dots_re = qr!(?:$beginning|$between|$end)!;
> +
>  # FIXME remove with PVE 8.0, add versioned breaks for pve-manager
>  our $vztmpl_extension_re = $VZTMPL_EXT_RE_1;
>  
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index ab6b675..87a08c3 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -621,6 +621,8 @@ sub parse_name_dir {
>  sub parse_volname {
>      my ($class, $volname) = @_;
>  
> +    die "volname must not contain parent directories '/../'\n" if $volname =~ $PVE::Storage::forbidden_double_dots_re;
> +
>      if ($volname =~ m!^(\d+)/(\S+)/(\d+)/(\S+)$!) {
>  	my ($basedvmid, $basename) = ($1, $2);
>  	parse_name_dir($basename);
> @@ -631,9 +633,9 @@ sub parse_volname {
>  	my ($vmid, $name) = ($1, $2);
>  	my (undef, $format, $isBase) = parse_name_dir($name);
>  	return ('images', $name, $vmid, undef, undef, $isBase, $format);
> -    } elsif ($volname =~ m!^iso/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!) {
> +    } elsif ($volname =~ m!^iso/((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::ISO_EXT_RE_0)$!) {

nit: this regex part:

(?:[0-9A-z\_\-\.]+\/)*[^\/]+

matching the intermediate sub-dirs and the non-extension part of the
file name is repeated a few times in this patch, it might be a good idea
to unify that (maybe with groups matching various parts?)

>  	return ('iso', $1);
> -    } elsif ($volname =~ m!^vztmpl/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
> +    } elsif ($volname =~ m!^vztmpl/((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!) {
>  	return ('vztmpl', $1);
>      } elsif ($volname =~ m!^rootdir/(\d+)$!) {
>  	return ('rootdir', $1, $1);
> @@ -643,7 +645,7 @@ sub parse_volname {
>  	    return ('backup', $fn, $2);
>  	}
>  	return ('backup', $fn);
> -    } elsif ($volname =~ m!^snippets/([^/]+)$!) {
> +    } elsif ($volname =~ m!^snippets/((?:[0-9A-z\_\-\.]+\/)*[^\/]+)$!) {
>  	return ('snippets', $1);
>      }
>  
> @@ -1212,28 +1214,33 @@ sub list_images {
>  }
>  
>  # list templates ($tt = <iso|vztmpl|backup|snippets>)
> -my $get_subdir_files = sub {
> -    my ($sid, $path, $tt, $vmid) = @_;
> +sub get_subdir_files {
> +    my ($sid, $path, $tt, $scfg, $vmid, $remaining_depth) = @_;
> +    my $storage_path = $scfg->{path};
> +    my $content_dir = $scfg->{"content-dirs"}->{$tt} // $vtype_subdirs->{$tt};
>  
>      my $res = [];
>  
>      foreach my $fn (<$path/*>) {
> -	my $st = File::stat::stat($fn);
> +	my $st = File::stat::lstat($fn);
> +
> +	next if (!$st);
>  
> -	next if (!$st || S_ISDIR($st->mode));
> +	if (S_ISDIR($st->mode)) {
> +	    if ($remaining_depth) {
> +		push @$res, get_subdir_files($sid, $fn, $tt, $scfg, $vmid, $remaining_depth-1);
> +	    }
> +	    next;
> +	}
>  
>  	my $info;
>  
>  	if ($tt eq 'iso') {
> -	    next if $fn !~ m!/([^/]+$PVE::Storage::ISO_EXT_RE_0)$!i;
> -
> +	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::ISO_EXT_RE_0)/;
>  	    $info = { volid => "$sid:iso/$1", format => 'iso' };
> -
>  	} elsif ($tt eq 'vztmpl') {
> -	    next if $fn !~ m!/([^/]+$PVE::Storage::VZTMPL_EXT_RE_1)$!;
> -
> +	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*[^\/]+$PVE::Storage::VZTMPL_EXT_RE_1)/;
>  	    $info = { volid => "$sid:vztmpl/$1", format => "t$2" };
> -
>  	} elsif ($tt eq 'backup') {
>  	    next if $fn !~ m!/([^/]+$PVE::Storage::BACKUP_EXT_RE_2)$!;
>  	    my $original = $fn;
> @@ -1262,9 +1269,9 @@ my $get_subdir_files = sub {
>  
>  	    $info->{protected} = 1 if -e PVE::Storage::protection_file_path($original);
>  	} elsif ($tt eq 'snippets') {
> -
> +	    next if $fn !~ m/(?:^$storage_path\/$content_dir\/)((?:[0-9A-z\_\-\.]+\/)*.+)/;
>  	    $info = {
> -		volid => "$sid:snippets/". basename($fn),
> +		volid => "$sid:snippets/$1", #basename($fn),
>  		format => 'snippet',
>  	    };
>  	}
> @@ -1274,14 +1281,18 @@ my $get_subdir_files = sub {
>  
>  	push @$res, $info;
>      }
> -
>      return $res;
>  };
>  
> +sub flatten {
> +    map { ref eq 'ARRAY' ? flatten(@{$_}) : $_ } @_;
> +}
> +
>  # If attributes are set on a volume, they should be included in the result.
>  # See get_volume_attribute for a list of possible attributes.
>  sub list_volumes {
>      my ($class, $storeid, $scfg, $vmid, $content_types) = @_;
> +    my $max_depth = $scfg->{'subdir-depth'} // 0;
>  
>      my $res = [];
>      my $vmlist = PVE::Cluster::get_vmlist();
> @@ -1294,17 +1305,19 @@ sub list_volumes {
>  	    my $path = $class->get_subdir($scfg, $type);
>  
>  	    if ($type eq 'iso' && !defined($vmid)) {
> -		$data = $get_subdir_files->($storeid, $path, 'iso');
> +		$data = get_subdir_files($storeid, $path, 'iso', $scfg, undef, $max_depth);
>  	    } elsif ($type eq 'vztmpl'&& !defined($vmid)) {
> -		$data = $get_subdir_files->($storeid, $path, 'vztmpl');
> +		$data = get_subdir_files($storeid, $path , 'vztmpl', $scfg, undef, $max_depth);
>  	    } elsif ($type eq 'backup') {
> -		$data = $get_subdir_files->($storeid, $path, 'backup', $vmid);
> +		$data = get_subdir_files($storeid, $path, 'backup', $scfg, $vmid, $max_depth);
>  	    } elsif ($type eq 'snippets') {
> -		$data = $get_subdir_files->($storeid, $path, 'snippets');
> +		$data = get_subdir_files($storeid, $path, 'snippets', $scfg, undef, $max_depth);
>  	    }
>  	}
>  
> -	next if !$data;
> +	$data = [flatten($data)];
> +
> +	next if !@$data[0];
>  
>  	foreach my $item (@$data) {
>  	    if ($type eq 'images' || $type eq 'rootdir') {
> @@ -1322,7 +1335,6 @@ sub list_volumes {
>  	    } else {
>  		$item->{content} = $type;
>  	    }
> -
>  	    push @$res, $item;
>  	}
>      }
> -- 
> 2.30.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> 

Re: [pve-devel] [PATCH pve-storage/pve-manager v3 0/4] fix #623: show isos/vztmpl/snippets in subdirs

[Fabian Grünbichler]

On June 15, 2023 2:03 pm, Noel Ullreich wrote:
> This patch fixes #623, allowing isos/vztmpl/snippets in subdirectories.
> This feature is opt-in and can be set from the API, web interface or
> with `pvesm`.
> 
> I addressed the security concerns raised by Fabian, now parent
> directories in the path (i.e. `/my/path/../somewhere/`) are forbidded.
> I have kept the permission to use symlinks, however, if this is a
> security issue, symlinks can easily be forbidden as well. This,
> however, would be a breaking change.

w.r.t. the symlinks:

symlinks are (still) allowed for the files themselves, which is okay.
what is a bit strange is that the "size" of a symlinked iso is that of
the symlink, not of the target, i.e., it depends on the name length
instead of the content size ;)

symlinks are not allowed (or rather, ignored) for the intermediate
components, which I guess would be one of the main use cases for
symlinks in the first place? having to link each file separately seems
tedious..

I tried to think about possible "bad" scenarios with symlinked subdirs,
but all of them are applicable to symlinked files as well and either
- require direct write access to the storage directory hierarchy to
  allow the creation of "dangerous" symlinks (not exposed over the API)
- an attacker-controlled host-mounted subvol that is mounted below the
  iso/template/.. content dir (which is actually a variant of the above
  I guess)

with the size and dir parts addressed, and the small nit I noted inline
with patch #1, consider this

Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

unless somebody comes up with a symlink-related attack scenario that
would be exploitable on a regular PVE setup which I missed, of course ;)

> parts of the tests as well as the regex for checking, if a `/../` is in
> the path have been taken and/or adapted from an older patch that was
> never merged:
> https://lists.proxmox.com/pipermail/pve-devel/2020-May/043622.html
> 
> This is a complete rework from v1, so I don't see a point in writing
> what the differences are. It's all different.
> 
> ----
> changes from v2:
> * rebased so that applying with new structure in pve-storage works 
> (/PVE was moved to /src/PVE/)
> * fixed the path of the volid for snippets in Pluggin.pm (thanks @Markus)
> 
> Noel Ullreich (4):
> 
> pve-storage:
>   recursively go through subdirs to find files
>   add `subdir-depth` option to filesystems
>   update test for recursive subdir search
> 
>  src/PVE/Storage.pm                 |  7 +++
>  src/PVE/Storage/CIFSPlugin.pm      |  1 +
>  src/PVE/Storage/CephFSPlugin.pm    |  1 +
>  src/PVE/Storage/DirPlugin.pm       |  1 +
>  src/PVE/Storage/GlusterfsPlugin.pm |  1 +
>  src/PVE/Storage/NFSPlugin.pm       |  1 +
>  src/PVE/Storage/Plugin.pm          | 63 +++++++++++++++++----------
>  src/test/filesystem_path_test.pm   | 18 ++++++++
>  src/test/list_volumes_test.pm      | 68 ++++++++++++++++++++++++++++++
>  src/test/parse_volname_test.pm     | 40 ++++++++++++++++++
>  10 files changed, 179 insertions(+), 22 deletions(-)
> 
> pve-manager:
>  www/manager6/storage/Base.js | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> -- 
> 2.30.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> 

Re: [pve-devel] [PATCH pve-manager v3]ui: add field to set subdir-depth in web interface

[Thomas Lamprecht]

Am 15/06/2023 um 14:03 schrieb Noel Ullreich:
> When adding or editing a storage device in Datacenter->Storage in the
> web interface, the subdirectory depth can be set in the advanced menu.
> 
> Signed-off-by: Noel Ullreich <n.ullreich@proxmox.com>
> ---
>  www/manager6/storage/Base.js | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
> index c8c735f31..c6844426d 100644
> --- a/www/manager6/storage/Base.js
> +++ b/www/manager6/storage/Base.js
> @@ -48,6 +48,7 @@ Ext.define('PVE.panel.StorageBase', {
>  		name: 'enable',
>  		checked: true,
>  		uncheckedValue: 0,
> +		minValue: 0,
>  		fieldLabel: gettext('Enable'),
>  	    },
>  	);
> @@ -63,13 +64,23 @@ Ext.define('PVE.panel.StorageBase', {
>  		deleteEmpty: !me.isCreate,
>  		value: '__default__',
>  	    };
> +	    const recursionDepth = {
> +		xtype: 'proxmoxintegerfield',
> +		name: 'subdir-depth',
> +		fieldLabel: gettext('Subdirectory Depth'),

Would just use "Scan Depth" as label, and maybe something like that might make
more sense too for the parameter name "scan-depth" or "subdir-scan-depth", as
"subdir-depth" isn't _that_ telling on its own..