From: Oguz Bektas <o.bektas@proxmox.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH v4 common 2/7] tools: add download_file_from_url
Date: Thu, 6 May 2021 12:04:43 +0200 [thread overview]
Message-ID: <20210506100443.GA12590@gaia.proxmox.com> (raw)
In-Reply-To: <20210506091105.40976-2-l.stechauner@proxmox.com>
hi,
see inline for some small suggestions :)
On Thu, May 06, 2021 at 11:11:00AM +0200, Lorenz Stechauner wrote:
> code is based on
> manager:PVE/API2/Nodes.pm:aplinfo
>
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
> src/PVE/Tools.pm | 123 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 123 insertions(+)
>
> diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
> index 16ae3d2..c751426 100644
> --- a/src/PVE/Tools.pm
> +++ b/src/PVE/Tools.pm
> @@ -1829,4 +1829,127 @@ sub safe_compare {
> return $cmp->($left, $right);
> }
>
> +
> +# opts
> +# -> hash_required
> +# -> http_proxy
> +# -> verify_certificates
> +# -> sha(1|224|256|384|512)sum
> +# -> md5sum
> +sub download_file_from_url {
> + my ($dest, $url, $opts) = @_;
> +
> + my $tmpdest = "$dest.tmp.$$";
> +
> + my $worker = sub {
> + my $upid = shift;
> +
> + print "donwloading $url to $dest\n";
small typo here
> +
> + eval {
> + if (-f $dest) {
> + print "calculating checksum of existing file...\n";
> + my ($correct, $hash, $expected) = check_file_hash($opts, $dest, 1);
> +
> + if ($hash && $correct) {
> + print "file already exists - no need to download\n";
> + return;
> + } else {
> + print "mismatch, downloading\n";
> + }
> + }
> +
> + my @cmd = ('/usr/bin/wget', '--progress=dot:mega', '-O', $tmpdest, $url);
> +
> + local %ENV;
> + if ($opts->{http_proxy}) {
> + $ENV{http_proxy} = $opts->{http_proxy};
might be worth it to also add https_proxy here
> + }
> +
> + if (defined($opts->{verify_certificates}) && $opts->{verify_certificates} == 0) {
> + push @cmd, '--no-check-certificate';
> + }
> +
> + if (system(@cmd) != 0) {
> + die "download failed - $!\n";
> + }
we don't use 'system' for executing commands (especially when a command
parameter is supplied by a user!). see the 'run_command' helper in
pve-common (which also does shellquoting)
> +
> + print "trying to calculate checksum...\n";
> +
> + my ($correct, $hash, $expected) = check_file_hash($opts, $tmpdest, !$opts->{hash_required});
is it necessary to call check_file_hash unless the option hash_required
is passed?
> +
> + die "could not calculate checksum\n" if ($opts->{hash_required} && !$hash);
> +
> + if ($hash) {
> + if ($correct) {
> + print "checksum verified\n";
> + } else {
> + die "wrong checksum: $hash != $expected\n";
> + }
> + } else {
> + print "no checksum for verification specified\n";
> + }
> +
> + if (!rename($tmpdest, $dest)) {
> + die "unable to save file - $!\n";
> + }
> + };
> + my $err = $@;
> +
> + unlink $tmpdest;
> +
> + if ($err) {
> + print "\n";
> + die $err;
> + }
> +
> + print "download finished\n";
> + };
> +
> + my $rpcenv = PVE::RPCEnvironment::get();
> + my $user = $rpcenv->get_user();
> +
> + (my $filename = $dest) =~ s!.*/([^/]*)$!\1!;
> +
> + return $rpcenv->fork_worker('download', $filename, $user, $worker);
> +}
> +
> +sub check_file_hash {
> + my ($checksums, $filename, $noerr) = @_;
> +
> + my $digest;
> + my $expected;
> +
> + eval {
> + open(my $fh, '<', $filename) or die "Can't open '$filename': $!";
> + binmode($fh);
> + if (defined($checksums->{sha512sum})) {
> + $expected = $checksums->{sha512sum};
> + $digest = Digest::SHA->new(512)->addfile($fh)->hexdigest;
> + } elsif (defined($checksums->{sha384sum})) {
> + $expected = $checksums->{sha384sum};
> + $digest = Digest::SHA->new(384)->addfile($fh)->hexdigest;
> + } elsif (defined($checksums->{sha256sum})) {
> + $expected = $checksums->{sha256sum};
> + $digest = Digest::SHA->new(256)->addfile($fh)->hexdigest;
> + } elsif (defined($checksums->{sha224sum})) {
> + $expected = $checksums->{sha224sum};
> + $digest = Digest::SHA->new(224)->addfile($fh)->hexdigest;
> + } elsif (defined($checksums->{sha1sum})) {
> + $expected = $checksums->{sha1sum};
> + $digest = Digest::SHA->new(1)->addfile($fh)->hexdigest;
> + } elsif (defined($checksums->{md5sum})) {
> + $expected = $checksums->{md5sum};
> + $digest = Digest::MD5->new->addfile($fh)->hexdigest;
hmm not necessary but maybe you could also do something like this (not
tested):
...
my $sha_algorithms = ('1', '224', '256', '384', '512');
foreach my $algorithm (@$sha_algorithms) {
if (defined($checksums->{"sha$algorithm"})) {
$expected = $checksums->{"sha$algorithm"};
$digest = Digest::SHA->new($algorithm)->addfile($fh)->hexdigest;
}
}
to avoid having a lot of if/elsif clauses (md5 would probably have another
clause but 2 is better than 5-6).
> + } else {
> + die "no expected checksum defined";
> + }
> + close($fh);
> + };
> +
> + die "checking hash failed - $@\n" if $@ && !$noerr;
> +
> + return (($digest ? lc($digest) eq lc($expected) : 0), $digest, $expected);
> +}
> +
> 1;
> --
> 2.20.1
>
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
next prev parent reply other threads:[~2021-05-06 10:04 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-03 10:20 [pve-devel] [PATCH-SERIES v2 manager/storage 0/2] fix #1710: add retrieve method for Lorenz Stechauner
2021-05-03 10:20 ` [pve-devel] [PATCH v2 storage 1/2] fix #1710: add retrieve method for storage Lorenz Stechauner
2021-05-03 10:21 ` [pve-devel] [PATCH v2 manager 2/2] fix #1710: add retrieve from url button " Lorenz Stechauner
2021-05-04 8:55 ` [pve-devel] [PATCH-SERIES v3 manager/storage 0/2] " Lorenz Stechauner
2021-05-04 8:56 ` [pve-devel] [PATCH v3 storage 1/2] fix #1710: add retrieve method " Lorenz Stechauner
2021-05-04 9:31 ` Thomas Lamprecht
2021-05-04 8:57 ` [pve-devel] [PATCH v3 manager 2/2] fix #1710: add retrieve from url button " Lorenz Stechauner
2021-05-04 9:47 ` Thomas Lamprecht
2021-05-06 9:10 ` [pve-devel] [PATCH-SERIES v4 manager/common/storage 0/7] fix #1710: add download from url button Lorenz Stechauner
2021-05-06 9:10 ` [pve-devel] [PATCH v4 manager 1/7] api: nodes: add query_url_metadata method Lorenz Stechauner
2021-05-06 9:11 ` [pve-devel] [PATCH v4 common 2/7] tools: add download_file_from_url Lorenz Stechauner
2021-05-06 10:04 ` Oguz Bektas [this message]
2021-05-06 12:15 ` Thomas Lamprecht
2021-05-06 12:17 ` Thomas Lamprecht
2021-05-06 9:11 ` [pve-devel] [PATCH v4 manager 3/7] api: nodes: refactor aplinfo to use common download function Lorenz Stechauner
2021-05-06 9:11 ` [pve-devel] [PATCH v4 storage 4/7] status: add download_url method Lorenz Stechauner
2021-05-06 9:23 ` [pve-devel] [PATCH v5 storage] " Lorenz Stechauner
2021-05-06 9:11 ` [pve-devel] [PATCH v4 manager 5/7] ui: add HashAlgorithmSelector Lorenz Stechauner
2021-05-06 9:11 ` [pve-devel] [PATCH v4 manager 6/7] ui: Utils: change download task format Lorenz Stechauner
2021-05-06 9:11 ` [pve-devel] [PATCH v4 manager 7/7] fix #1710: ui: storage: add download from url button Lorenz Stechauner
2021-05-06 13:15 ` [pve-devel] [PATCH v4 manager 1/7] api: nodes: add query_url_metadata method Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210506100443.GA12590@gaia.proxmox.com \
--to=o.bektas@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal