Re: [pve-devel] [PATCH V2 storage 1/1] fix #4849: download-url: allow download and decompression of compressed ISOs

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

On July 27, 2023 5:28 pm, Philipp Hufnagl wrote:
> Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
> ---
>  src/PVE/API2/Storage/Status.pm | 22 +++++++++++++++++++---
>  src/PVE/Storage.pm             |  6 ++++++
>  2 files changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
> index 2aaeff6..a087c53 100644
> --- a/src/PVE/API2/Storage/Status.pm
> +++ b/src/PVE/API2/Storage/Status.pm
> @@ -23,6 +23,8 @@ use PVE::Storage;
>  
>  use base qw(PVE::RESTHandler);
>  
> +our $KNOWN_COMPRESSION_FORMATS = ['zst', 'gz', 'lzo'];
> +
>  __PACKAGE__->register_method ({
>      subclass => "PVE::API2::Storage::PruneBackups",
>      path => '{storage}/prunebackups',
> @@ -578,6 +580,12 @@ __PACKAGE__->register_method({
>  		requires => 'checksum-algorithm',
>  		optional => 1,
>  	    },
> +	    compression => {
> +		description => "Decompress the downloaded file using following compression algorithm",

nit: s/following/specified or "this"

> +		type => 'string',
> +		enum => $KNOWN_COMPRESSION_FORMATS,
> +		optional => 1,
> +	    },
>  	    'checksum-algorithm' => {
>  		description => "The algorithm to calculate the checksum of the file.",
>  		type => 'string',
> @@ -604,7 +612,7 @@ __PACKAGE__->register_method({
>  
>  	my $cfg = PVE::Storage::config();
>  
> -	my ($node, $storage) = $param->@{'node', 'storage'};
> +	my ($node, $storage, $compression) = $param->@{'node', 'storage','compression'};
>  	my $scfg = PVE::Storage::storage_check_enabled($cfg, $storage, $node);
>  
>  	die "can't upload to storage type '$scfg->{type}', not a file based storage!\n"
> @@ -627,6 +635,7 @@ __PACKAGE__->register_method({
>  	    if ($filename !~ m![^/]+$PVE::Storage::VZTMPL_EXT_RE_1$!) {
>  		raise_param_exc({ filename => "wrong file extension" });
>  	    }
> +            die "decompression not supported for vztempl" if $compression;

'vztmpl', or CT template, but I would switch this check around -> only
allow iso. that way, if we ever add new content types, we only expose
them to decompression by adding them, instead of automatically. in
general, it is usually a better approach to specify what you accept, and
not what you reject, unless you want to automatically accept all future
additions, which is rarely the case ;)

>  	    $path = PVE::Storage::get_vztmpl_dir($cfg, $storage);
>  	} else {
>  	    raise_param_exc({ content => "upload content-type '$content' is not allowed" });
> @@ -642,14 +651,21 @@ __PACKAGE__->register_method({
>  	    http_proxy => $dccfg->{http_proxy},
>  	};
>  
> -	my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
> +	my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm' };
>  	if ($checksum) {
>  	    $opts->{"${checksum_algorithm}sum"} = $checksum;
>  	    $opts->{hash_required} = 1;
>  	}
>  
>  	my $worker = sub {
> -	    PVE::Tools::download_file_from_url("$path/$filename", $url, $opts);
> +	    my $save_to = "$path/$filename";
> +	    die "refusing to override existing file $save_to \n" if -e $save_to ;
> +	    $save_to .= ".$compression" if $compression;

see below - this is not needed (here) if the decompressor is switched to
STDOUT.. the idea of moving the decompression into
download_file_from_url was that the caller need not be concerned with
*how* the decompression is handled - just provide the command to do the
decompression ;)

> +	    if ($compression) {
> +                my $info = PVE::Storage::decompressor_info('iso', $compression);
> +                $opts->{decompression_command} = $info->{decompressor};

nit: indentation?

> +	    }
> +	    PVE::Tools::download_file_from_url($save_to, $url, $opts);
>  	};
>  
>  	my $worker_id = PVE::Tools::encode_text($filename); # must not pass : or the like as w-ID
> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
> index a4d85e1..9dbf38f 100755
> --- a/src/PVE/Storage.pm
> +++ b/src/PVE/Storage.pm
> @@ -1531,6 +1531,12 @@ sub decompressor_info {
>  	    lzo => ['lzop', '-d', '-c'],
>  	    zst => ['zstd', '-q', '-d', '-c'],
>  	},
> +	iso => {
> +            # zstd seem to be able to handle .gzip fine. Therefore we dont need additional other tool

nit: indentation of comment

> +	    gz =>  ['zstd', '-q', '-d'],
> +	    zst => ['zstd', '-q', '-d'],
> +	    lzo => ['lzop', '-q', '-d'],

I'd still like these switched to the STDOUT variants like with vma, see
previous review and comment at the pve-common patch.

> +	},
>      };
>  
>      die "ERROR: archive format not defined\n"
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
>