From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 64CA39E9C6 for ; Wed, 7 Jun 2023 08:54:46 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4C5223D849 for ; Wed, 7 Jun 2023 08:54:46 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 7 Jun 2023 08:54:45 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E4C8F40903 for ; Wed, 7 Jun 2023 08:54:44 +0200 (CEST) Date: Wed, 7 Jun 2023 08:54:44 +0200 From: Wolfgang Bumiller To: Christoph Heiss Cc: pve-devel@lists.proxmox.com Message-ID: References: <20230330092519.93868-1-c.heiss@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20230330092519.93868-1-c.heiss@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.021 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] [PATCH manager] pvesh: decode streamed responses X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2023 06:54:46 -0000 looks mostly fine, I'd like some minor changes: On Thu, Mar 30, 2023 at 11:25:20AM +0200, Christoph Heiss wrote: > This allows to use `pvesh` on endpoints like /nodes/{node}/journal, > which return streamed (and possibly gzip'd) responses. > > Currently, e.g. `pvesh get /nodes/localhost/journal --lastentries 10` > fails with: > > gzip: stdout: Broken pipe > got hash object, but result schema specified array! > > Using e.g. `--output-format yaml` resulted in: > > --- > download: > content-encoding: gzip > content-type: application/json > fh: &1 !!perl/ref > =: *1 > stream: 1 > > gzip: stdout: Broken pipe > Failed to write > > This is due the API call returning a "download" object (as seen above), > which contains (among some other things) a file handle to read the > response from. > > With this patch, the response from such endpoints is now correctly read > and displayed. Only handles combinations of `Content-Encoding` == 'gzip' > and either 'text/plain' or 'application/json' for `Content-Type`. > > This tries to mimic the behavior of the API server implementation when > encountering `download` objects. > > Tested this with all four output formats 'text', 'json', 'json-pretty' > and 'yaml', as well as "cross-node" in a local test cluster. > > Signed-off-by: Christoph Heiss > --- > As far as I could see (aka. grep for it), the only two endpoints which > implement this are /nodes/{node}/journal and > /nodes/{node}/tasks/{upid}/log, latter one only with `--download 1` > set. > > PVE/CLI/pvesh.pm | 35 +++++++++++++++++++++++++++++++++++ > 1 file changed, 35 insertions(+) > > diff --git a/PVE/CLI/pvesh.pm b/PVE/CLI/pvesh.pm > index 9acf292a..764d47a2 100755 > --- a/PVE/CLI/pvesh.pm > +++ b/PVE/CLI/pvesh.pm > @@ -15,6 +15,7 @@ use PVE::CLIHandler; > use PVE::API2Tools; > use PVE::API2; > use JSON; > +use IO::Uncompress::Gunzip qw(gunzip); > > use base qw(PVE::CLIHandler); > > @@ -281,6 +282,37 @@ my $cond_add_standard_output_properties = sub { > return PVE::RESTHandler::add_standard_output_properties($props, $keys); > }; > > +my $handle_streamed_response = sub { > + my ($download) = @_; > + my ($fh, $path, $encoding, $type) = > + $download->@{'fh', 'path', 'content-encoding', 'content-type'}; > + > + die "{download} returned but neither fh nor path given\n" > + if !defined($fh) and !defined($path); ^ style nit: use && here please > + > + if (defined($path)) { > + open($fh, '<', $path) > + or die "open stream path '$path' for reading failed: $!\n"; > + } > + > + local $/; > + my $data = <$fh>; > + > + if (defined($encoding)) { > + die "unknown 'content-encoding' $encoding\n" if $encoding ne 'gzip'; > + my $out; > + gunzip(\$data => \$out); > + $data = $out; > + } > + > + if (defined($type) && not $type =~ qw!^text/plain!) { style: `$type !~ …` instead of 'not $type =~ …' > + die "unknown 'content-type' $type\n" if not $type =~ qw!^application/json!; Would be nice to move the $type check above the part doing the gunzip() to avoid unnecessary work. > + $data = decode_json($data)->{data}; > + } > + > + return $data; > +}; > + > sub call_api_method { > my ($cmd, $param) = @_; > > @@ -310,6 +342,9 @@ sub call_api_method { > } > > $data = $handler->handle($info, $param); > + > + $data = &$handle_streamed_response($data->{download}) > + if ref($data) eq 'HASH' && ref($data->{download}) eq 'HASH'; > } > > return if $opt_nooutput || $stdopts->{quiet}; > -- > 2.39.2