From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1152081773 for ; Wed, 24 Nov 2021 15:48:55 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0D07321934 for ; Wed, 24 Nov 2021 15:48:55 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id EA23D2192A for ; Wed, 24 Nov 2021 15:48:53 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C2C3645CD3 for ; Wed, 24 Nov 2021 15:48:53 +0100 (CET) From: Dominik Csapak To: pmg-devel@lists.proxmox.com Date: Wed, 24 Nov 2021 15:48:52 +0100 Message-Id: <20211124144852.72471-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.201 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pmg-devel] [PATCH pmg-api] api: journal: stream the journal data to the client X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Nov 2021 14:48:55 -0000 instead of accumulating the whole output of 'mini-journalreader' in the api call (this can be quite big), use the download mechanic of the http-server to stream the output to the client. we lose some error handling possibilities, but we do not have to allocate anything here, and since perl does not free memory after allocating[0] this is our desired behaviour. to keep api compatiblitiy, we need to give the journalreader the '-j' flag to let it output json. also tell the http server that the encoding is gzip and pipe the output through it. 0: https://perldoc.perl.org/perlfaq3#How-can-I-free-an-array-or-hash-so-my-program-shrinks? Signed-off-by: Dominik Csapak --- this is basically a 1:1 copy of the changes in pve-manager[0] obviously needs the changes from http-server and mini-journalreader from[1] 0: https://lists.proxmox.com/pipermail/pve-devel/2021-November/051072.html 1: https://lists.proxmox.com/pipermail/pve-devel/2021-November/051070.html src/PMG/API2/Nodes.pm | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/src/PMG/API2/Nodes.pm b/src/PMG/API2/Nodes.pm index daa90c4..472a5f3 100644 --- a/src/PMG/API2/Nodes.pm +++ b/src/PMG/API2/Nodes.pm @@ -334,19 +334,25 @@ __PACKAGE__->register_method({ code => sub { my ($param) = @_; - my $cmd = ["/usr/bin/mini-journalreader"]; + my $cmd = ["/usr/bin/mini-journalreader", "-j"]; push @$cmd, '-n', $param->{lastentries} if $param->{lastentries}; push @$cmd, '-b', $param->{since} if $param->{since}; push @$cmd, '-e', $param->{until} if $param->{until}; - push @$cmd, '-f', $param->{startcursor} if $param->{startcursor}; - push @$cmd, '-t', $param->{endcursor} if $param->{endcursor}; + push @$cmd, '-f', PVE::Tools::shellquote($param->{startcursor}) if $param->{startcursor}; + push @$cmd, '-t', PVE::Tools::shellquote($param->{endcursor}) if $param->{endcursor}; + push @$cmd, ' | gzip '; - my $lines = []; - my $parser = sub { push @$lines, shift }; + open(my $fh, "-|", join(' ', @$cmd)) + or die "could not start mini-journalreader"; - PVE::Tools::run_command($cmd, outfunc => $parser); - - return $lines; + return { + download => { + fh => $fh, + stream => 1, + 'content-type' => 'application/json', + 'content-encoding' => 'gzip', + }, + }, }}); my $shell_cmd_map = { -- 2.30.2