From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 9211E8D59 for ; Mon, 31 Jul 2023 15:43:22 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 527163C6A for ; Mon, 31 Jul 2023 15:42:52 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 31 Jul 2023 15:42:51 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 236F842D1C for ; Mon, 31 Jul 2023 15:42:51 +0200 (CEST) Date: Mon, 31 Jul 2023 15:42:43 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE development discussion References: <20230731083924.18234-1-p.hufnagl@proxmox.com> <20230731083924.18234-2-p.hufnagl@proxmox.com> In-Reply-To: <20230731083924.18234-2-p.hufnagl@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1690809730.b23ewyo656.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.069 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] [V3 PATCH common 1/1] fix #4849: download file from url: add opt parameter for a decompression command X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jul 2023 13:43:22 -0000 On July 31, 2023 10:39 am, Philipp Hufnagl wrote: > Signed-off-by: Philipp Hufnagl > --- > src/PVE/Tools.pm | 59 +++++++++++++++++++++++++++++++----------------- > 1 file changed, 38 insertions(+), 21 deletions(-) >=20 > diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm > index 9ffac12..ab97129 100644 > --- a/src/PVE/Tools.pm > +++ b/src/PVE/Tools.pm [snipped lots of unrelated changes here!] > @@ -2013,10 +2013,13 @@ sub download_file_from_url { > } > } > =20 > - my $tmpdest =3D "$dest.tmp.$$"; > + my $tmp_download =3D "$dest.tmp_dwnl.$$"; > + my $tmp_decomp =3D "$dest.tmp.dcom.$$"; nit: the naming scheme should be unified ;) > eval { > local $SIG{INT} =3D sub { > - unlink $tmpdest or warn "could not cleanup temporary file: $!"; > + unlink $tmp_download or warn "could not cleanup temporary file: $!"= ; > + die "got interrupted by signal\n"; this die here > + unlink $tmp_decomp or warn "could not cleanup temporary file: $!"; means this part never gets to run. also, this part should be guarded by an if, since for the uncompressed case, $tmp_decomp never gets created in the first place (see below). > die "got interrupted by signal\n"; > }; > =20 > @@ -2029,7 +2032,7 @@ sub download_file_from_url { > $ENV{https_proxy} =3D $opts->{https_proxy}; > } > =20 > - my $cmd =3D ['wget', '--progress=3Ddot:giga', '-O', $tmpdest, $url]= ; > + my $cmd =3D ['wget', '--progress=3Ddot:giga', '-O', $tmp_download, = $url]; > =20 > if (!($opts->{verify_certificates} // 1)) { # default to true > push @$cmd, '--no-check-certificate'; > @@ -2041,7 +2044,7 @@ sub download_file_from_url { > if ($checksum_algorithm) { > print "calculating checksum..."; > =20 > - my $checksum_got =3D get_file_hash($checksum_algorithm, $tmpdest); > + my $checksum_got =3D get_file_hash($checksum_algorithm, $tmp_downlo= ad); > =20 > if (lc($checksum_got) eq lc($checksum_expected)) { > print "OK, checksum verified\n"; > @@ -2051,10 +2054,24 @@ sub download_file_from_url { > } > } > =20 > - rename($tmpdest, $dest) or die "unable to rename temporary file: $!\n"; > + if (my $cmd =3D $opts->{decompression_command}) { nit: indentation (starting) here is wrong > + push @$cmd, $tmp_download; > + > + nit: extra blank line? > + my $fh; > + if (!open($fh, ">", "$tmp_decomp")) { > + die "cant open temporary file $tmp_decomp for decompresson: $!\n"; > + } > + print "decompressing $tmp_download to $tmp_decomp\n"; > + eval { run_command($cmd, output =3D> '>&'.fileno($fh)); }; > + my $rerr =3D $@; nit: $rerr should just be $err > + unlink $tmp_download; > + die "$rerr\n" if $rerr; > + } > + rename($tmp_decomp, $dest) or die "unable to rename temporary file: $!\= n"; this needs to be properly handling the no-decompression case - did you test that? because for me it's broken: $ pveam download ext4 rockylinux-9-default_20221109_amd64.tar.xz downloading http://download.proxmox.com/images/system/rockylinux-9-default_= 20221109_amd64.tar.xz to /mnt/pve/ext4/template/cache/rockylinux-9-default_= 20221109_amd64.tar.xz --2023-07-31 15:29:34-- http://download.proxmox.com/images/system/rockylin= ux-9-default_20221109_amd64.tar.xz Resolving download.proxmox.com (download.proxmox.com)... 212.224.123.70, 2a= 01:7e0:0:424::249 Connecting to download.proxmox.com (download.proxmox.com)|212.224.123.70|:8= 0... connected. HTTP request sent, awaiting response... 200 OK Length: 102704656 (98M) [application/octet-stream] Saving to: '/mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd6= 4.tar.xz.tmp_dwnl.665888' 0K ........ ........ ........ ........ 32% 54.8M 1s 32768K ........ ........ ........ ........ 65% 45.3M 1s 65536K ........ ........ ........ ........ 98% 45.1M 0s 98304K . 100% 46.8M=3D2.0s 2023-07-31 15:29:36 (47.9 MB/s) - '/mnt/pve/ext4/template/cache/rockylinux-= 9-default_20221109_amd64.tar.xz.tmp_dwnl.665888' saved [102704656/102704656= ] calculating checksum...OK, checksum verified could not cleanup temporary file: No such file or directory at /usr/share/p= erl5/PVE/Tools.pm line 2073. unable to rename temporary file: No such file or directory > }; > if (my $err =3D $@) { > - unlink $tmpdest or warn "could not cleanup temporary file: $!"; > + unlink $tmp_decomp or warn "could not cleanup temporary file: $!"; same here - this should cleanup $tmp_download if uncompressed, or $tmp_decomp if decompression is enabled.. also, pre-existing (see output above), missing newline in error message. after the above 'pveam' invocation: $ ls -lh /mnt/pve/ext4/template/cache total 98M -rw-r--r-- 1 root root 98M Nov 9 2022 rockylinux-9-default_20221109_amd6= 4.tar.xz.tmp_dwnl.665888 > die $err; > } > =20 > --=20 > 2.39.2 >=20 >=20 >=20 > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >=20 >=20 >=20