From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BBA4C9B872 for ; Thu, 19 Oct 2023 09:28:56 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 95792168E4 for ; Thu, 19 Oct 2023 09:28:26 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 19 Oct 2023 09:28:25 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 9EB0242E61 for ; Thu, 19 Oct 2023 09:28:25 +0200 (CEST) Date: Thu, 19 Oct 2023 09:28:24 +0200 From: Christoph Heiss Cc: Proxmox VE development discussion Message-ID: References: <20230607141511.560351-1-c.heiss@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230607141511.560351-1-c.heiss@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.685 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment MISSING_HEADERS 1.021 Missing To: header POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH manager v2] pvesh: decode streamed responses X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Oct 2023 07:28:56 -0000 Ping, still applies. On Wed, Jun 07, 2023 at 04:15:11PM +0200, Christoph Heiss wrote: > > This allows to use `pvesh` on endpoints like /nodes/{node}/journal, > which return streamed (and possibly gzip'd) responses. > > Currently, e.g. `pvesh get /nodes/localhost/journal --lastentries 10` > fails with: > > gzip: stdout: Broken pipe > got hash object, but result schema specified array! > > Using e.g. `--output-format yaml` resulted in: > > --- > download: > content-encoding: gzip > content-type: application/json > fh: &1 !!perl/ref > =: *1 > stream: 1 > > gzip: stdout: Broken pipe > Failed to write > > This is due the API call returning a "download" object (as seen above), > which contains (among some other things) a file handle to read the > response from. > > With this patch, the response from such endpoints is now correctly read > and displayed. Only handles combinations of `Content-Encoding` == 'gzip' > and either 'text/plain' or 'application/json' for `Content-Type`. > > This tries to mimic the behavior of the API server implementation when > encountering `download` objects. > > Tested this with all four output formats 'text', 'json', 'json-pretty' > and 'yaml', as well as "cross-node" in a local test cluster. > > Signed-off-by: Christoph Heiss > --- > Changes v1 -> v2: > * Fix some style nits > * Move content-{encoding,type} checks before actual decoding > > As far as I could see (aka. grep for it), the only two endpoints which > implement this are /nodes/{node}/journal and > /nodes/{node}/tasks/{upid}/log, latter one only with `--download 1` > set. > > PVE/CLI/pvesh.pm | 39 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 39 insertions(+) > > diff --git a/PVE/CLI/pvesh.pm b/PVE/CLI/pvesh.pm > index 9acf292a..c0c8187b 100755 > --- a/PVE/CLI/pvesh.pm > +++ b/PVE/CLI/pvesh.pm > @@ -15,6 +15,7 @@ use PVE::CLIHandler; > use PVE::API2Tools; > use PVE::API2; > use JSON; > +use IO::Uncompress::Gunzip qw(gunzip); > > use base qw(PVE::CLIHandler); > > @@ -281,6 +282,41 @@ my $cond_add_standard_output_properties = sub { > return PVE::RESTHandler::add_standard_output_properties($props, $keys); > }; > > +my $handle_streamed_response = sub { > + my ($download) = @_; > + my ($fh, $path, $encoding, $type) = > + $download->@{'fh', 'path', 'content-encoding', 'content-type'}; > + > + die "{download} returned but neither fh nor path given\n" > + if !defined($fh) && !defined($path); > + > + die "unknown 'content-encoding' $encoding\n" > + if defined($encoding) && $encoding ne 'gzip'; > + > + die "unknown 'content-type' $type\n" > + if defined($type) && $type !~ qw!^(text/plain)|(application/json)$!; > + > + if (defined($path)) { > + open($fh, '<', $path) > + or die "open stream path '$path' for reading failed: $!\n"; > + } > + > + local $/; > + my $data = <$fh>; > + > + if (defined($encoding)) { > + my $out; > + gunzip(\$data => \$out); > + $data = $out; > + } > + > + if (defined($type) && $type eq 'application/json') { > + $data = decode_json($data)->{data}; > + } > + > + return $data; > +}; > + > sub call_api_method { > my ($cmd, $param) = @_; > > @@ -310,6 +346,9 @@ sub call_api_method { > } > > $data = $handler->handle($info, $param); > + > + $data = &$handle_streamed_response($data->{download}) > + if ref($data) eq 'HASH' && ref($data->{download}) eq 'HASH'; > } > > return if $opt_nooutput || $stdopts->{quiet}; > -- > 2.40.1 > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > >