From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 475749210F for ; Tue, 28 Mar 2023 15:01:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 30BA21C768 for ; Tue, 28 Mar 2023 15:01:43 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 28 Mar 2023 15:01:42 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E3D464708F for ; Tue, 28 Mar 2023 15:01:41 +0200 (CEST) From: Christoph Heiss To: pve-devel@lists.proxmox.com Date: Tue, 28 Mar 2023 15:01:39 +0200 Message-Id: <20230328130139.510340-1-c.heiss@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.266 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH manager] pvesh: decode streamed responses properly X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Mar 2023 13:01:43 -0000 This allows to use `pvesh` on endpoints like /nodes/{node}/journal, which return streamed (and possibly gzip'd) responses. Currently, e.g. `pvesh get /nodes/localhost/journal --lastentries 10` fails with: gzip: stdout: Broken pipe got hash object, but result schema specified array! (.. and similar messages for other output formats.) This is due the API call returning a "download" object, which contains (among some other things) a file handle to read the response from. With this patch, the response from such endpoints is now correctly read/displayed. Only handles combinations of `Content-Encoding` == 'gzip' and either 'text/plain' or 'application/json' as `Content-Type`. As far as I could see (aka. grep for it), only these combinations are used currently. Tested this with all four output formats 'text', 'json', 'json-pretty' and 'yaml', as well as "cross-node" in a local test cluster. Signed-off-by: Christoph Heiss --- PVE/CLI/pvesh.pm | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/PVE/CLI/pvesh.pm b/PVE/CLI/pvesh.pm index 9acf292a..85cdaf8e 100755 --- a/PVE/CLI/pvesh.pm +++ b/PVE/CLI/pvesh.pm @@ -15,6 +15,7 @@ use PVE::CLIHandler; use PVE::API2Tools; use PVE::API2; use JSON; +use IO::Uncompress::Gunzip qw(gunzip); use base qw(PVE::CLIHandler); @@ -281,6 +282,28 @@ my $cond_add_standard_output_properties = sub { return PVE::RESTHandler::add_standard_output_properties($props, $keys); }; +my $handle_streamed_response = sub { + my ($download) = @_; + my ($fh, $encoding, $type) = $download->@{'fh', 'content-encoding', 'content-type'}; + + local $/; + my $data = <$fh>; + + if (defined($encoding)) { + die "unknown 'content-encoding' $encoding\n" if $encoding ne 'gzip'; + my $out; + gunzip(\$data => \$out); + $data = $out; + } + + if (defined($type) && not $type =~ qw!^text/plain!) { + die "unknown 'content-type' $type\n" if not $type =~ qw!^application/json!; + $data = decode_json($data)->{data}; + } + + return $data; +}; + sub call_api_method { my ($cmd, $param) = @_; @@ -310,6 +333,9 @@ sub call_api_method { } $data = $handler->handle($info, $param); + + $data = &$handle_streamed_response($data->{download}) + if ref($data) eq 'HASH' && defined($data->{download}); } return if $opt_nooutput || $stdopts->{quiet}; -- 2.39.2