From: Lorenz Stechauner <l.stechauner@proxmox.com>
To: Lorenz Stechauner <l.stechauner@proxmox.com>,
Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH v7 common 1/1] tools: add download_file_from_url
Date: Tue, 15 Jun 2021 09:58:00 +0200 [thread overview]
Message-ID: <f50afecf-cf12-8195-1f4c-8a91060be217@proxmox.com> (raw)
In-Reply-To: <20210614090557.33455-2-l.stechauner@proxmox.com>
adds a common function to download arbitrary files from urls.
security notice: this function does not perform any permission checking.
the calling function and/or api endpoint has to make sure, that only
authorized users may use this function.
caution: This function is able to download files from internal networks
(which would not be visible/accessible from outside).
On 14.06.21 11:05, Lorenz Stechauner wrote:
> code is based on
> manager:PVE/API2/Nodes.pm:aplinfo
>
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
> src/PVE/Tools.pm | 124 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 124 insertions(+)
>
> diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
> index 16ae3d2..7b82e00 100644
> --- a/src/PVE/Tools.pm
> +++ b/src/PVE/Tools.pm
> @@ -1829,4 +1829,128 @@ sub safe_compare {
> return $cmp->($left, $right);
> }
>
> +
> +# opts
> +# -> hash_required
> +# if 1, at least one checksum has to be specified otherwise an error will be thrown
> +# -> http_proxy
> +# -> https_proxy
> +# -> verify_certificates
> +# -> sha(1|224|256|384|512)sum
> +# -> md5sum
> +sub download_file_from_url {
> + my ($dest, $url, $opts) = @_;
> +
> + my $tmpdest = "$dest.tmp.$$";
> +
> + my $algorithm;
> + my $expected;
> +
> + for ('sha512', 'sha384', 'sha256', 'sha224', 'sha1', 'md5') {
> + if (defined($opts->{"${_}sum"})) {
> + $algorithm = $_;
> + $expected = $opts->{"${_}sum"};
> + last;
> + }
> + }
> +
> + die "checksum required but not specified\n" if ($opts->{hash_required} && !$algorithm);
> +
> + my $worker = sub {
> + my $upid = shift;
> +
> + print "downloading $url to $dest\n";
> +
> + eval {
> + if (-f $dest && $algorithm) {
> + print "calculating checksum of existing file...\n";
> + my $correct = check_file_hash($algorithm, $expected, $dest);
> +
> + if ($correct) {
> + print "file already exists, no need to download\n";
> + return;
> + } else {
> + print "mismatch, downloading\n";
> + }
> + }
> +
> + my @cmd = ('/usr/bin/wget', '--progress=dot:mega', '-O', $tmpdest, $url);
> +
> + local %ENV;
> + if ($opts->{http_proxy}) {
> + $ENV{http_proxy} = $opts->{http_proxy};
> + }
> + if ($opts->{https_proxy}) {
> + $ENV{https_proxy} = $opts->{https_proxy};
> + }
> +
> + my $verify = $opts->{verify_certificates} // 1;
> + if (!$verify) {
> + push @cmd, '--no-check-certificate';
> + }
> +
> + if (run_command([[@cmd]]) != 0) {
> + die "download failed: $!\n";
> + }
> +
> + if ($algorithm) {
> + print "calculating checksum...\n";
> +
> + my $correct = check_file_hash($algorithm, $expected, $tmpdest);
> +
> + if ($correct) {
> + print "checksum verified\n";
> + } else {
> + die "checksum mismatch\n";
> + }
> + } else {
> + print "no checksum for verification specified\n";
> + }
> +
> + if (!rename($tmpdest, $dest)) {
> + die "unable to save file: $!\n";
> + }
> + };
> + my $err = $@;
> +
> + unlink $tmpdest;
> +
> + if ($err) {
> + print "\n";
> + die $err;
> + }
> +
> + print "download finished\n";
> + };
> +
> + my $rpcenv = PVE::RPCEnvironment::get();
> + my $user = $rpcenv->get_user();
> +
> + (my $filename = $dest) =~ s!.*/([^/]*)$!$1!;
> +
> + return $rpcenv->fork_worker('download', $filename, $user, $worker);
> +}
> +
> +sub check_file_hash {
> + my ($algorithm, $expected, $filename) = @_;
> +
> + my $algorithm_map = {
> + 'md5' => sub { Digest::MD5->new },
> + 'sha1' => sub { Digest::SHA->new(1) },
> + 'sha224' => sub { Digest::SHA->new(224) },
> + 'sha256' => sub { Digest::SHA->new(256) },
> + 'sha384' => sub { Digest::SHA->new(384) },
> + 'sha512' => sub { Digest::SHA->new(512) },
> + };
> +
> + my $digester = $algorithm_map->{$algorithm}->() or die "unknown algorithm '$algorithm'\n";
> +
> + open(my $fh, '<', $filename) or die "unable to open '$filename': $!\n";
> + binmode($fh);
> +
> + my $digest = $digester->addfile($fh)->hexdigest;
> +
> + return lc($digest) eq lc($expected);
> +}
> +
> 1;
next prev parent reply other threads:[~2021-06-15 7:58 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-14 9:05 [pve-devel] [PATCH-SERIES v7 manager/common/storage] fix #1710: add downlaod from url button Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 common 1/1] tools: add download_file_from_url Lorenz Stechauner
2021-06-15 7:58 ` Lorenz Stechauner [this message]
2021-06-15 12:25 ` [pve-devel] applied: " Thomas Lamprecht
2021-06-14 9:05 ` [pve-devel] [PATCH v7 storage 1/1] status: add download_url method Lorenz Stechauner
2021-06-15 7:58 ` Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 manager 1/5] api: nodes: add query_url_metadata method Lorenz Stechauner
2021-06-15 7:58 ` Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 manager 2/5] api: nodes: refactor aplinfo to use common download function Lorenz Stechauner
2021-06-15 7:57 ` Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 manager 3/5] ui: add HashAlgorithmSelector Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 manager 4/5] ui: Utils: change download task format Lorenz Stechauner
2021-06-14 9:05 ` [pve-devel] [PATCH v7 manager 5/5] fix #1710: ui: storage: add download from url button Lorenz Stechauner
2021-06-15 7:58 ` Lorenz Stechauner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f50afecf-cf12-8195-1f4c-8a91060be217@proxmox.com \
--to=l.stechauner@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox