[pve-devel] [PATCH manager/storage v6 0/3] fix #4849: allow download of compressed ISOs

[pve-devel] [PATCH manager/storage v6 0/3] fix #4849: allow download of compressed ISOs

[Philipp Hufnagl]

Many web pages offer the download of the disk images compressed. This
patch allows the download of archives (like .gz), automatically detects
the format and decompresses it

Changes since v5:
  * split manager patch into frontend/backend
  * remove not needed regex group

Changes since v4:
  * add commit messages
  * fix nit

Changes since v3:
  * generate compression regex from compression list
  * fix logic errors

Changes since v2:
  * move compression code to the download function in common
  * minor code improvements


Changes since v1:
  * Improve code quality as suggested by feedback

Philipp Hufnagl (2):
  fix #4849: api: download to storage: automatically dectect and
    configure compression
  fix #4849: ui: download to storage: automatically dectect and
    configure compression

 PVE/API2/Nodes.pm                           | 20 ++++++++++++++++++++
 www/manager6/Makefile                       |  1 +
 www/manager6/form/DecompressionSelector.js  | 13 +++++++++++++
 www/manager6/window/DownloadUrlToStorage.js | 17 +++++++++++++++++
 4 files changed, 51 insertions(+)
 create mode 100644 www/manager6/form/DecompressionSelector.js

Philipp Hufnagl (1):
  fix #4849: download-url: allow download and decompression of
    compressed ISOs

 src/PVE/API2/Storage/Status.pm | 14 +++++++++++++-
 src/PVE/Storage.pm             |  6 ++++++
 src/PVE/Storage/Plugin.pm      |  3 ++-
 3 files changed, 21 insertions(+), 2 deletions(-)

-- 
2.39.2

[pve-devel] [PATCH manager v6 1/2] fix #4849: api: download to storage: automatically dectect and configure compression

[Philipp Hufnagl]

extends the query_url_metadata callback with the functionallity to
detect used compressions. If a compression is used it tells the ui which
one

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
---
 PVE/API2/Nodes.pm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 5a148d1d..66a8bb0b 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -1564,6 +1564,12 @@ __PACKAGE__->register_method({
 		type => 'boolean',
 		optional => 1,
 		default => 1,
+	    },
+	    'detect-compression' => {
+		description => "If true an auto detect of used compression will be attempted",
+		type => 'boolean',
+		optional => 1,
+		default => 0,
 	    }
 	},
     },
@@ -1583,6 +1589,11 @@ __PACKAGE__->register_method({
 		type => 'string',
 		optional => 1,
 	    },
+	    compression => {
+		type => 'string',
+		enum => $PVE::Storage::Plugin::KNOWN_COMPRESSION_FORMATS,
+		optional => 1,
+	    },
 	},
     },
     code => sub {
@@ -1606,6 +1617,8 @@ __PACKAGE__->register_method({
 	    );
 	}
 
+	my $detect_compression = $param->{'detect-compression'};
+
 	my $req = HTTP::Request->new(HEAD => $url);
 	my $res = $ua->request($req);
 
@@ -1615,6 +1628,7 @@ __PACKAGE__->register_method({
 	my $disposition = $res->header("Content-Disposition");
 	my $type = $res->header("Content-Type");
 
+	my $compression;
 	my $filename;
 
 	if ($disposition && ($disposition =~ m/filename="([^"]*)"/ || $disposition =~ m/filename=([^;]*)/)) {
@@ -1628,10 +1642,16 @@ __PACKAGE__->register_method({
 	    $type = $1;
 	}
 
+	if ($detect_compression && $filename =~ m!^(.+)\.(${\PVE::Storage::Plugin::COMPRESSOR_RE})$!) {
+	    $filename = $1;
+	    $compression = $2;
+	}
+
 	my $ret = {};
 	$ret->{filename} = $filename if $filename;
 	$ret->{size} = $size + 0 if $size;
 	$ret->{mimetype} = $type if $type;
+	$ret->{compression} = $compression if $compression;
 
 	return $ret;
     }});
-- 
2.39.2

Re: [pve-devel] [PATCH manager v6 1/2] fix #4849: api: download to storage: automatically dectect and configure compression

[Dominik Csapak]

looks mostly good, one high level comment, and a few nits inline:

do we really need the 'detect-compression' parameter?

when wouldn't we want that? also the gui always enables that for isos anyway?
if there is a good reason, that would be nice to have in a commit message

if we can always use the detection, we don't have to give the parameter in the gui
and save a few lines and an additional api parameter we have to bring with us for a long time ;)

On 8/14/23 16:42, Philipp Hufnagl wrote:
> extends the query_url_metadata callback with the functionallity to
> detect used compressions. If a compression is used it tells the ui which
> one
> 
> Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
> ---
>   PVE/API2/Nodes.pm | 20 ++++++++++++++++++++
>   1 file changed, 20 insertions(+)
> 
> diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
> index 5a148d1d..66a8bb0b 100644
> --- a/PVE/API2/Nodes.pm
> +++ b/PVE/API2/Nodes.pm
> @@ -1564,6 +1564,12 @@ __PACKAGE__->register_method({
>   		type => 'boolean',
>   		optional => 1,
>   		default => 1,
> +	    },
> +	    'detect-compression' => {
> +		description => "If true an auto detect of used compression will be attempted",
> +		type => 'boolean',
> +		optional => 1,
> +		default => 0,
>   	    }
>   	},
>       },
> @@ -1583,6 +1589,11 @@ __PACKAGE__->register_method({
>   		type => 'string',
>   		optional => 1,
>   	    },
> +	    compression => {
> +		type => 'string',
> +		enum => $PVE::Storage::Plugin::KNOWN_COMPRESSION_FORMATS,

if you use this, you have to import it with a 'use' statement at the top of the package
in this case it works because the storage part includes it already, but should that change
it would not work anymore

> +		optional => 1,
> +	    },
>   	},
>       },
>       code => sub {
> @@ -1606,6 +1617,8 @@ __PACKAGE__->register_method({
>   	    );
>   	}
>   
> +	my $detect_compression = $param->{'detect-compression'};
> +

if we decide to leave the parameter in, i'd prefer to set the default here:

my $detect_compression = $param->{'detect-compression'} // 0;


(it works fine without that, but now i don't have to guess what the default should be without 
looking at the parameter description)

>   	my $req = HTTP::Request->new(HEAD => $url);
>   	my $res = $ua->request($req);
>   
> @@ -1615,6 +1628,7 @@ __PACKAGE__->register_method({
>   	my $disposition = $res->header("Content-Disposition");
>   	my $type = $res->header("Content-Type");
>   
> +	my $compression;
>   	my $filename;
>   
>   	if ($disposition && ($disposition =~ m/filename="([^"]*)"/ || $disposition =~ m/filename=([^;]*)/)) {
> @@ -1628,10 +1642,16 @@ __PACKAGE__->register_method({
>   	    $type = $1;
>   	}
>   
> +	if ($detect_compression && $filename =~ m!^(.+)\.(${\PVE::Storage::Plugin::COMPRESSOR_RE})$!) {
> +	    $filename = $1;
> +	    $compression = $2;
> +	}
> +
>   	my $ret = {};
>   	$ret->{filename} = $filename if $filename;
>   	$ret->{size} = $size + 0 if $size;
>   	$ret->{mimetype} = $type if $type;
> +	$ret->{compression} = $compression if $compression;
>   
>   	return $ret;
>       }});

[pve-devel] [PATCH manager v6 2/2] fix #4849: ui: download to storage: automatically dectect and configure compression

[Philipp Hufnagl]

extends the download iso prompt with a "compression algorithm" drop down
under advanced. User can configure there if a decompression algorithm
should be used from the storage backend. The compression algorithm will
be automatically guessed when calling query_url_metadata

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
---
 www/manager6/Makefile                       |  1 +
 www/manager6/form/DecompressionSelector.js  | 13 +++++++++++++
 www/manager6/window/DownloadUrlToStorage.js | 17 +++++++++++++++++
 3 files changed, 31 insertions(+)
 create mode 100644 www/manager6/form/DecompressionSelector.js

diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 7ec9d7a5..42a27548 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -34,6 +34,7 @@ JSSRC= 							\
 	form/ContentTypeSelector.js			\
 	form/ControllerSelector.js			\
 	form/DayOfWeekSelector.js			\
+	form/DecompressionSelector.js       \
 	form/DiskFormatSelector.js			\
 	form/DiskStorageSelector.js			\
 	form/EmailNotificationSelector.js		\
diff --git a/www/manager6/form/DecompressionSelector.js b/www/manager6/form/DecompressionSelector.js
new file mode 100644
index 00000000..abd19316
--- /dev/null
+++ b/www/manager6/form/DecompressionSelector.js
@@ -0,0 +1,13 @@
+Ext.define('PVE.form.DecompressionSelector', {
+    extend: 'Proxmox.form.KVComboBox',
+    alias: ['widget.pveDecompressionSelector'],
+    config: {
+	deleteEmpty: false,
+    },
+    comboItems: [
+		['__default__', Proxmox.Utils.NoneText],
+		['lzo', 'LZO'],
+		['gz', 'GZIP'],
+		['zst', 'ZSTD'],
+    ],
+});
diff --git a/www/manager6/window/DownloadUrlToStorage.js b/www/manager6/window/DownloadUrlToStorage.js
index 90320da4..559a1c05 100644
--- a/www/manager6/window/DownloadUrlToStorage.js
+++ b/www/manager6/window/DownloadUrlToStorage.js
@@ -49,6 +49,9 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
 	    vm.set('size', '-');
 	    vm.set('mimetype', '-');
 	},
+	decompressionPossible: function() {
+	    return this.view.content === 'iso';
+	},
 
 	urlCheck: function(field) {
 	    let me = this;
@@ -66,6 +69,7 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
 		params: {
 		    url: queryParam.url,
 		    'verify-certificates': queryParam['verify-certificates'],
+		    'detect-compression': me.decompressionPossible() ? 1 : 0,
 		},
 		waitMsgTarget: view,
 		failure: res => {
@@ -84,6 +88,7 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
 			filename: data.filename || "",
 			size: (data.size && Proxmox.Utils.format_size(data.size)) || gettext("Unknown"),
 			mimetype: data.mimetype || gettext("Unknown"),
+			compression: data.compression || '__default__',
 		    });
 		},
 	    });
@@ -223,6 +228,18 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
 	if (!me.storage) {
 	    throw "no storage ID specified";
 	}
+	if (me.content === 'iso') {
+	    me.items[0].advancedColumn2.push(
+
+		{
+		    xtype: 'pveDecompressionSelector',
+		    name: 'compression',
+		    fieldLabel: gettext('Decompression algorithm'),
+		    allowBlank: true,
+		    hasNoneOption: true,
+		    value: '__default__',
+		});
+	}
 
 	me.callParent();
     },
-- 
2.39.2

Re: [pve-devel] [PATCH manager v6 2/2] fix #4849: ui: download to storage: automatically dectect and configure compression

[Dominik Csapak]

a few comments inline

On 8/14/23 16:42, Philipp Hufnagl wrote:
> extends the download iso prompt with a "compression algorithm" drop down
> under advanced. User can configure there if a decompression algorithm
> should be used from the storage backend. The compression algorithm will
> be automatically guessed when calling query_url_metadata
> 
> Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
> ---
>   www/manager6/Makefile                       |  1 +
>   www/manager6/form/DecompressionSelector.js  | 13 +++++++++++++
>   www/manager6/window/DownloadUrlToStorage.js | 17 +++++++++++++++++
>   3 files changed, 31 insertions(+)
>   create mode 100644 www/manager6/form/DecompressionSelector.js
> 
> diff --git a/www/manager6/Makefile b/www/manager6/Makefile
> index 7ec9d7a5..42a27548 100644
> --- a/www/manager6/Makefile
> +++ b/www/manager6/Makefile
> @@ -34,6 +34,7 @@ JSSRC= 							\
>   	form/ContentTypeSelector.js			\
>   	form/ControllerSelector.js			\
>   	form/DayOfWeekSelector.js			\
> +	form/DecompressionSelector.js       \
>   	form/DiskFormatSelector.js			\
>   	form/DiskStorageSelector.js			\
>   	form/EmailNotificationSelector.js		\
> diff --git a/www/manager6/form/DecompressionSelector.js b/www/manager6/form/DecompressionSelector.js
> new file mode 100644
> index 00000000..abd19316
> --- /dev/null
> +++ b/www/manager6/form/DecompressionSelector.js
> @@ -0,0 +1,13 @@
> +Ext.define('PVE.form.DecompressionSelector', {
> +    extend: 'Proxmox.form.KVComboBox',
> +    alias: ['widget.pveDecompressionSelector'],
> +    config: {
> +	deleteEmpty: false,
> +    },
> +    comboItems: [
> +		['__default__', Proxmox.Utils.NoneText],
> +		['lzo', 'LZO'],
> +		['gz', 'GZIP'],
> +		['zst', 'ZSTD'],
> +    ],
> +});
> diff --git a/www/manager6/window/DownloadUrlToStorage.js b/www/manager6/window/DownloadUrlToStorage.js
> index 90320da4..559a1c05 100644
> --- a/www/manager6/window/DownloadUrlToStorage.js
> +++ b/www/manager6/window/DownloadUrlToStorage.js
> @@ -49,6 +49,9 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
>   	    vm.set('size', '-');
>   	    vm.set('mimetype', '-');
>   	},
> +	decompressionPossible: function() {
> +	    return this.view.content === 'iso';
> +	},

this is a oneline that's only used one time, i'd rather implement it where we need it
also 'this.getView()' is preferred to 'this.view', but in this case it does not matter because...

>   
>   	urlCheck: function(field) {
>   	    let me = this;
> @@ -66,6 +69,7 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
>   		params: {
>   		    url: queryParam.url,
>   		    'verify-certificates': queryParam['verify-certificates'],
> +		    'detect-compression': me.decompressionPossible() ? 1 : 0,

if you do it here, there is already a 'view' variable, so you could do

'detect-compression': view.content === 'iso' ? 1 : 0,

(also is it even necessary to cast to 1/0? i'd hoped that we can simply have true/false
here too, so it'd simplify to

'detect-compression': view.content === 'iso'


also, see my notes on the other patch if we even need this

>   		},
>   		waitMsgTarget: view,
>   		failure: res => {
> @@ -84,6 +88,7 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
>   			filename: data.filename || "",
>   			size: (data.size && Proxmox.Utils.format_size(data.size)) || gettext("Unknown"),
>   			mimetype: data.mimetype || gettext("Unknown"),
> +			compression: data.compression || '__default__',
>   		    });
>   		},
>   	    });
> @@ -223,6 +228,18 @@ Ext.define('PVE.window.DownloadUrlToStorage', {
>   	if (!me.storage) {
>   	    throw "no storage ID specified";
>   	}
> +	if (me.content === 'iso') {
> +	    me.items[0].advancedColumn2.push(
> +
> +		{
> +		    xtype: 'pveDecompressionSelector',
> +		    name: 'compression',
> +		    fieldLabel: gettext('Decompression algorithm'),
> +		    allowBlank: true,
> +		    hasNoneOption: true,
> +		    value: '__default__',
> +		});
> +	}

while this works here (by accident?), this pattern is dangerous, because
you modify not the instance items, but the variable of the class itself
(that's not copied before callParent)


so it would be better to always add the field in the 'items' property,
but use the cbind variables to hide/disable it when necessary

>   
>   	me.callParent();
>       },

[pve-devel] [PATCH storage v6 1/1] fix #4849: download-url: allow download and decompression of compressed ISOs

[Philipp Hufnagl]

adds information for how to decompress isos.

generates the compressor regex from a list of comression formats (to
avoid redundancy)
extends the download_url wtih the functionality to handley compression
for images

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
---
 src/PVE/API2/Storage/Status.pm | 14 +++++++++++++-
 src/PVE/Storage.pm             |  6 ++++++
 src/PVE/Storage/Plugin.pm      |  3 ++-
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index 2aaeff6..7875530 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -578,6 +578,12 @@ __PACKAGE__->register_method({
 		requires => 'checksum-algorithm',
 		optional => 1,
 	    },
+	    compression => {
+		description => "Decompress the downloaded file using specified compression algorithm",
+		type => 'string',
+		enum => $PVE::Storage::Plugin::KNOWN_COMPRESSION_FORMATS,
+		optional => 1,
+	    },
 	    'checksum-algorithm' => {
 		description => "The algorithm to calculate the checksum of the file.",
 		type => 'string',
@@ -604,7 +610,7 @@ __PACKAGE__->register_method({
 
 	my $cfg = PVE::Storage::config();
 
-	my ($node, $storage) = $param->@{'node', 'storage'};
+	my ($node, $storage, $compression) = $param->@{'node', 'storage','compression'};
 	my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);
 
 	die "can't upload to storage type '$scfg->{type}', not a file based storage!\n"
@@ -649,6 +655,12 @@ __PACKAGE__->register_method({
 	}
 
 	my $worker = sub {
+	    if ($compression) {
+		die "decompression not supported for $content\n" if $content ne 'iso';
+		my $info = PVE::Storage::decompressor_info('iso', $compression);
+		die "no decompression method found\n" if (! $info->{decompressor});
+		$opts->{decompression_command} = $info->{decompressor};
+	    }
 	    PVE::Tools::download_file_from_url("$path/$filename", $url, $opts);
 	};
 
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index a4d85e1..cb70113 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -1531,6 +1531,12 @@ sub decompressor_info {
 	    lzo => ['lzop', '-d', '-c'],
 	    zst => ['zstd', '-q', '-d', '-c'],
 	},
+	iso => {
+	    # zstd seem to be able to handle .gzip fine. Therefore we dont need additional other tool
+	    gz => ['zcat'],
+	    lzo => ['lzop', '-d', '-c'],
+	    zst => ['zstd', '-q', '-d', '-c'],
+	},
     };
 
     die "ERROR: archive format not defined\n"
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 74d1987..71fcff0 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -19,7 +19,8 @@ use JSON;
 
 use base qw(PVE::SectionConfig);
 
-use constant COMPRESSOR_RE => 'gz|lzo|zst';
+use constant KNOWN_COMPRESSION_FORMATS =>  ( 'gz', 'lzo', 'zst');
+use constant COMPRESSOR_RE => join( '|', KNOWN_COMPRESSION_FORMATS);
 
 use constant LOG_EXT => ".log";
 use constant NOTES_EXT => ".notes";
-- 
2.39.2

[pve-devel] applied: [PATCH storage v6 1/1] fix #4849: download-url: allow download and decompression of compressed ISOs

[Wolfgang Bumiller]

this one's applied