From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [V3 PATCH common 1/1] fix #4849: download file from url: add opt parameter for a decompression command
Date: Mon, 31 Jul 2023 15:42:43 +0200 [thread overview]
Message-ID: <1690809730.b23ewyo656.astroid@yuna.none> (raw)
In-Reply-To: <20230731083924.18234-2-p.hufnagl@proxmox.com>
On July 31, 2023 10:39 am, Philipp Hufnagl wrote:
> Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
> ---
> src/PVE/Tools.pm | 59 +++++++++++++++++++++++++++++++-----------------
> 1 file changed, 38 insertions(+), 21 deletions(-)
>
> diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
> index 9ffac12..ab97129 100644
> --- a/src/PVE/Tools.pm
> +++ b/src/PVE/Tools.pm
[snipped lots of unrelated changes here!]
> @@ -2013,10 +2013,13 @@ sub download_file_from_url {
> }
> }
>
> - my $tmpdest = "$dest.tmp.$$";
> + my $tmp_download = "$dest.tmp_dwnl.$$";
> + my $tmp_decomp = "$dest.tmp.dcom.$$";
nit: the naming scheme should be unified ;)
> eval {
> local $SIG{INT} = sub {
> - unlink $tmpdest or warn "could not cleanup temporary file: $!";
> + unlink $tmp_download or warn "could not cleanup temporary file: $!";
> + die "got interrupted by signal\n";
this die here
> + unlink $tmp_decomp or warn "could not cleanup temporary file: $!";
means this part never gets to run. also, this part should be guarded by
an if, since for the uncompressed case, $tmp_decomp never gets created
in the first place (see below).
> die "got interrupted by signal\n";
> };
>
> @@ -2029,7 +2032,7 @@ sub download_file_from_url {
> $ENV{https_proxy} = $opts->{https_proxy};
> }
>
> - my $cmd = ['wget', '--progress=dot:giga', '-O', $tmpdest, $url];
> + my $cmd = ['wget', '--progress=dot:giga', '-O', $tmp_download, $url];
>
> if (!($opts->{verify_certificates} // 1)) { # default to true
> push @$cmd, '--no-check-certificate';
> @@ -2041,7 +2044,7 @@ sub download_file_from_url {
> if ($checksum_algorithm) {
> print "calculating checksum...";
>
> - my $checksum_got = get_file_hash($checksum_algorithm, $tmpdest);
> + my $checksum_got = get_file_hash($checksum_algorithm, $tmp_download);
>
> if (lc($checksum_got) eq lc($checksum_expected)) {
> print "OK, checksum verified\n";
> @@ -2051,10 +2054,24 @@ sub download_file_from_url {
> }
> }
>
> - rename($tmpdest, $dest) or die "unable to rename temporary file: $!\n";
> + if (my $cmd = $opts->{decompression_command}) {
nit: indentation (starting) here is wrong
> + push @$cmd, $tmp_download;
> +
> +
nit: extra blank line?
> + my $fh;
> + if (!open($fh, ">", "$tmp_decomp")) {
> + die "cant open temporary file $tmp_decomp for decompresson: $!\n";
> + }
> + print "decompressing $tmp_download to $tmp_decomp\n";
> + eval { run_command($cmd, output => '>&'.fileno($fh)); };
> + my $rerr = $@;
nit: $rerr should just be $err
> + unlink $tmp_download;
> + die "$rerr\n" if $rerr;
> + }
> + rename($tmp_decomp, $dest) or die "unable to rename temporary file: $!\n";
this needs to be properly handling the no-decompression case - did you
test that? because for me it's broken:
$ pveam download ext4 rockylinux-9-default_20221109_amd64.tar.xz
downloading http://download.proxmox.com/images/system/rockylinux-9-default_20221109_amd64.tar.xz to /mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz
--2023-07-31 15:29:34-- http://download.proxmox.com/images/system/rockylinux-9-default_20221109_amd64.tar.xz
Resolving download.proxmox.com (download.proxmox.com)... 212.224.123.70, 2a01:7e0:0:424::249
Connecting to download.proxmox.com (download.proxmox.com)|212.224.123.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 102704656 (98M) [application/octet-stream]
Saving to: '/mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888'
0K ........ ........ ........ ........ 32% 54.8M 1s
32768K ........ ........ ........ ........ 65% 45.3M 1s
65536K ........ ........ ........ ........ 98% 45.1M 0s
98304K . 100% 46.8M=2.0s
2023-07-31 15:29:36 (47.9 MB/s) - '/mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888' saved [102704656/102704656]
calculating checksum...OK, checksum verified
could not cleanup temporary file: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 2073.
unable to rename temporary file: No such file or directory
> };
> if (my $err = $@) {
> - unlink $tmpdest or warn "could not cleanup temporary file: $!";
> + unlink $tmp_decomp or warn "could not cleanup temporary file: $!";
same here - this should cleanup $tmp_download if uncompressed, or
$tmp_decomp if decompression is enabled..
also, pre-existing (see output above), missing newline in error message.
after the above 'pveam' invocation:
$ ls -lh /mnt/pve/ext4/template/cache
total 98M
-rw-r--r-- 1 root root 98M Nov 9 2022 rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888
> die $err;
> }
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
next prev parent reply other threads:[~2023-07-31 13:43 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-31 8:39 [pve-devel] [V3 PATCH storage/manager/common 0/3]fix #4849: allow download of compressed ISOs Philipp Hufnagl
2023-07-31 8:39 ` [pve-devel] [V3 PATCH common 1/1] fix #4849: download file from url: add opt parameter for a decompression command Philipp Hufnagl
2023-07-31 13:42 ` Fabian Grünbichler [this message]
2023-07-31 8:39 ` [pve-devel] [V3 PATCH manager 1/1] fix #4849: download to storage: automatically dectect and configure compression Philipp Hufnagl
2023-07-31 8:39 ` [pve-devel] [V3 PATCH storage 1/1] fix #4849: download-url: allow download and decompression of compressed ISOs Philipp Hufnagl
2023-07-31 13:42 ` Fabian Grünbichler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1690809730.b23ewyo656.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox