From: Dominik Csapak <d.csapak@proxmox.com>
To: pmg-devel@lists.proxmox.com
Subject: [pmg-devel] [PATCH pmg-api] api: journal: stream the journal data to the client
Date: Wed, 24 Nov 2021 15:48:52 +0100 [thread overview]
Message-ID: <20211124144852.72471-1-d.csapak@proxmox.com> (raw)
instead of accumulating the whole output of 'mini-journalreader' in
the api call (this can be quite big), use the download mechanic of the
http-server to stream the output to the client.
we lose some error handling possibilities, but we do not have
to allocate anything here, and since perl does not free memory after
allocating[0] this is our desired behaviour.
to keep api compatiblitiy, we need to give the journalreader the '-j'
flag to let it output json.
also tell the http server that the encoding is gzip and pipe
the output through it.
0: https://perldoc.perl.org/perlfaq3#How-can-I-free-an-array-or-hash-so-my-program-shrinks?
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
this is basically a 1:1 copy of the changes in pve-manager[0]
obviously needs the changes from http-server and mini-journalreader
from[1]
0: https://lists.proxmox.com/pipermail/pve-devel/2021-November/051072.html
1: https://lists.proxmox.com/pipermail/pve-devel/2021-November/051070.html
src/PMG/API2/Nodes.pm | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/src/PMG/API2/Nodes.pm b/src/PMG/API2/Nodes.pm
index daa90c4..472a5f3 100644
--- a/src/PMG/API2/Nodes.pm
+++ b/src/PMG/API2/Nodes.pm
@@ -334,19 +334,25 @@ __PACKAGE__->register_method({
code => sub {
my ($param) = @_;
- my $cmd = ["/usr/bin/mini-journalreader"];
+ my $cmd = ["/usr/bin/mini-journalreader", "-j"];
push @$cmd, '-n', $param->{lastentries} if $param->{lastentries};
push @$cmd, '-b', $param->{since} if $param->{since};
push @$cmd, '-e', $param->{until} if $param->{until};
- push @$cmd, '-f', $param->{startcursor} if $param->{startcursor};
- push @$cmd, '-t', $param->{endcursor} if $param->{endcursor};
+ push @$cmd, '-f', PVE::Tools::shellquote($param->{startcursor}) if $param->{startcursor};
+ push @$cmd, '-t', PVE::Tools::shellquote($param->{endcursor}) if $param->{endcursor};
+ push @$cmd, ' | gzip ';
- my $lines = [];
- my $parser = sub { push @$lines, shift };
+ open(my $fh, "-|", join(' ', @$cmd))
+ or die "could not start mini-journalreader";
- PVE::Tools::run_command($cmd, outfunc => $parser);
-
- return $lines;
+ return {
+ download => {
+ fh => $fh,
+ stream => 1,
+ 'content-type' => 'application/json',
+ 'content-encoding' => 'gzip',
+ },
+ },
}});
my $shell_cmd_map = {
--
2.30.2
next reply other threads:[~2021-11-24 14:48 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-24 14:48 Dominik Csapak [this message]
2021-11-24 17:58 ` [pmg-devel] applied: " Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211124144852.72471-1-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pmg-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox