From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <d.csapak@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 79CCA81755
for <pve-devel@lists.proxmox.com>; Wed, 24 Nov 2021 15:48:20 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 7074521754
for <pve-devel@lists.proxmox.com>; Wed, 24 Nov 2021 15:47:50 +0100 (CET)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id 807C221746
for <pve-devel@lists.proxmox.com>; Wed, 24 Nov 2021 15:47:49 +0100 (CET)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 5A22E447C8
for <pve-devel@lists.proxmox.com>; Wed, 24 Nov 2021 15:47:49 +0100 (CET)
From: Dominik Csapak <d.csapak@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Wed, 24 Nov 2021 15:47:48 +0100
Message-Id: <20211124144748.68687-4-d.csapak@proxmox.com>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20211124144748.68687-1-d.csapak@proxmox.com>
References: <20211124144748.68687-1-d.csapak@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.201 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pve-devel] [PATCH manager 1/1] api: journal: stream the journal
data to the client
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 24 Nov 2021 14:48:20 -0000
instead of accumulating the whole output of 'mini-journalreader' in
the api call (this can be quite big), use the download mechanic of the
http-server to stream the output to the client.
we lose some error handling possibilities, but we do not have
to allocate anything here, and since perl does not free memory after
allocating[0] this is our desired behaviour.
to keep api compatiblitiy, we need to give the journalreader the '-j'
flag to let it output json.
also tell the http server that the encoding is gzip and pipe
the output through it.
0: https://perldoc.perl.org/perlfaq3#How-can-I-free-an-array-or-hash-so-my-program-shrinks?
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
PVE/API2/Nodes.pm | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 565cbccc..d57a1937 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -819,19 +819,25 @@ __PACKAGE__->register_method({
my $rpcenv = PVE::RPCEnvironment::get();
my $user = $rpcenv->get_user();
- my $cmd = ["/usr/bin/mini-journalreader"];
+ my $cmd = ["/usr/bin/mini-journalreader", "-j"];
push @$cmd, '-n', $param->{lastentries} if $param->{lastentries};
push @$cmd, '-b', $param->{since} if $param->{since};
push @$cmd, '-e', $param->{until} if $param->{until};
- push @$cmd, '-f', $param->{startcursor} if $param->{startcursor};
- push @$cmd, '-t', $param->{endcursor} if $param->{endcursor};
+ push @$cmd, '-f', PVE::Tools::shellquote($param->{startcursor}) if $param->{startcursor};
+ push @$cmd, '-t', PVE::Tools::shellquote($param->{endcursor}) if $param->{endcursor};
+ push @$cmd, ' | gzip ';
- my $lines = [];
- my $parser = sub { push @$lines, shift };
+ open(my $fh, "-|", join(' ', @$cmd))
+ or die "could not start mini-journalreader";
- PVE::Tools::run_command($cmd, outfunc => $parser);
-
- return $lines;
+ return {
+ download => {
+ fh => $fh,
+ stream => 1,
+ 'content-type' => 'application/json',
+ 'content-encoding' => 'gzip',
+ },
+ },
}});
my $sslcert;
--
2.30.2