From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A3C139EF6D for ; Wed, 7 Jun 2023 16:15:16 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 8DC091A9D2 for ; Wed, 7 Jun 2023 16:15:16 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 7 Jun 2023 16:15:15 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 46B3541F2E for ; Wed, 7 Jun 2023 16:15:15 +0200 (CEST) From: Christoph Heiss To: pve-devel@lists.proxmox.com Date: Wed, 7 Jun 2023 16:15:11 +0200 Message-Id: <20230607141511.560351-1-c.heiss@proxmox.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.232 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH manager v2] pvesh: decode streamed responses X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2023 14:15:16 -0000 This allows to use `pvesh` on endpoints like /nodes/{node}/journal, which return streamed (and possibly gzip'd) responses. Currently, e.g. `pvesh get /nodes/localhost/journal --lastentries 10` fails with: gzip: stdout: Broken pipe got hash object, but result schema specified array! Using e.g. `--output-format yaml` resulted in: --- download: content-encoding: gzip content-type: application/json fh: &1 !!perl/ref =: *1 stream: 1 gzip: stdout: Broken pipe Failed to write This is due the API call returning a "download" object (as seen above), which contains (among some other things) a file handle to read the response from. With this patch, the response from such endpoints is now correctly read and displayed. Only handles combinations of `Content-Encoding` == 'gzip' and either 'text/plain' or 'application/json' for `Content-Type`. This tries to mimic the behavior of the API server implementation when encountering `download` objects. Tested this with all four output formats 'text', 'json', 'json-pretty' and 'yaml', as well as "cross-node" in a local test cluster. Signed-off-by: Christoph Heiss --- Changes v1 -> v2: * Fix some style nits * Move content-{encoding,type} checks before actual decoding As far as I could see (aka. grep for it), the only two endpoints which implement this are /nodes/{node}/journal and /nodes/{node}/tasks/{upid}/log, latter one only with `--download 1` set. PVE/CLI/pvesh.pm | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/PVE/CLI/pvesh.pm b/PVE/CLI/pvesh.pm index 9acf292a..c0c8187b 100755 --- a/PVE/CLI/pvesh.pm +++ b/PVE/CLI/pvesh.pm @@ -15,6 +15,7 @@ use PVE::CLIHandler; use PVE::API2Tools; use PVE::API2; use JSON; +use IO::Uncompress::Gunzip qw(gunzip); use base qw(PVE::CLIHandler); @@ -281,6 +282,41 @@ my $cond_add_standard_output_properties = sub { return PVE::RESTHandler::add_standard_output_properties($props, $keys); }; +my $handle_streamed_response = sub { + my ($download) = @_; + my ($fh, $path, $encoding, $type) = + $download->@{'fh', 'path', 'content-encoding', 'content-type'}; + + die "{download} returned but neither fh nor path given\n" + if !defined($fh) && !defined($path); + + die "unknown 'content-encoding' $encoding\n" + if defined($encoding) && $encoding ne 'gzip'; + + die "unknown 'content-type' $type\n" + if defined($type) && $type !~ qw!^(text/plain)|(application/json)$!; + + if (defined($path)) { + open($fh, '<', $path) + or die "open stream path '$path' for reading failed: $!\n"; + } + + local $/; + my $data = <$fh>; + + if (defined($encoding)) { + my $out; + gunzip(\$data => \$out); + $data = $out; + } + + if (defined($type) && $type eq 'application/json') { + $data = decode_json($data)->{data}; + } + + return $data; +}; + sub call_api_method { my ($cmd, $param) = @_; @@ -310,6 +346,9 @@ sub call_api_method { } $data = $handler->handle($info, $param); + + $data = &$handle_streamed_response($data->{download}) + if ref($data) eq 'HASH' && ref($data->{download}) eq 'HASH'; } return if $opt_nooutput || $stdopts->{quiet}; -- 2.40.1