* [pve-devel] [PATCH common v2] run_command: improve performance for logging and long lines
@ 2020-07-30 9:04 Dominik Csapak
2020-08-19 7:00 ` [pve-devel] applied: " Thomas Lamprecht
0 siblings, 1 reply; 2+ messages in thread
From: Dominik Csapak @ 2020-07-30 9:04 UTC (permalink / raw)
To: pve-devel
to call out/err/logfunc with each line, we search for a newline and call
outfunc/logfunc with everything before that
since we do a select/read (with 4096 size) in a loop, this means
that if we have very long lines, we search for a newline in an
ever growing buffer (for which we know does not contain a newline)
so instead, only search the new data for newlines
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from v1:
* keep the substitution instead of a match, making the diff a little smaller
this fixes a bug in the v1, when there were multiple lines in one read
src/PVE/Tools.pm | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
index 4399b2f..f013fb7 100644
--- a/src/PVE/Tools.pm
+++ b/src/PVE/Tools.pm
@@ -496,12 +496,13 @@ sub run_command {
if ($h eq $reader) {
if ($outfunc || $logfunc) {
eval {
- $outlog .= $buf;
- while ($outlog =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
- my $line = $1;
+ while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//) {
+ my $line = $outlog . $1;
+ $outlog = '';
&$outfunc($line) if $outfunc;
&$logfunc($line) if $logfunc;
}
+ $outlog .= $buf;
};
my $err = $@;
if ($err) {
@@ -516,12 +517,13 @@ sub run_command {
} elsif ($h eq $error) {
if ($errfunc || $logfunc) {
eval {
- $errlog .= $buf;
- while ($errlog =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
- my $line = $1;
+ while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
+ my $line = $errlog . $1;
+ $errlog = '';
&$errfunc($line) if $errfunc;
&$logfunc($line) if $logfunc;
}
+ $errlog .= $buf;
};
my $err = $@;
if ($err) {
--
2.20.1
^ permalink raw reply [flat|nested] 2+ messages in thread
* [pve-devel] applied: [PATCH common v2] run_command: improve performance for logging and long lines
2020-07-30 9:04 [pve-devel] [PATCH common v2] run_command: improve performance for logging and long lines Dominik Csapak
@ 2020-08-19 7:00 ` Thomas Lamprecht
0 siblings, 0 replies; 2+ messages in thread
From: Thomas Lamprecht @ 2020-08-19 7:00 UTC (permalink / raw)
To: Proxmox VE development discussion, Dominik Csapak
On 30.07.20 11:04, Dominik Csapak wrote:
> to call out/err/logfunc with each line, we search for a newline and call
> outfunc/logfunc with everything before that
>
> since we do a select/read (with 4096 size) in a loop, this means
> that if we have very long lines, we search for a newline in an
> ever growing buffer (for which we know does not contain a newline)
>
> so instead, only search the new data for newlines
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> changes from v1:
> * keep the substitution instead of a match, making the diff a little smaller
> this fixes a bug in the v1, when there were multiple lines in one read
> src/PVE/Tools.pm | 14 ++++++++------
> 1 file changed, 8 insertions(+), 6 deletions(-)
>
>
applied, thanks! With the followup below, doing two things both already present
before your optimization, so just FYI:
* non-capturing group for things we do not use
* fix matching of \r\n sequence, as this was non-greedy and so a \r\n was matched
as two lines, one with \r and one with \n. But, the regex clearly indicates that
this wasn't intended.
Also, I made the change to not use the s modifier also for the $h eq $error case,
for consistency (it does not matters much, we do not use . here)
diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
index d9c69e3..f9270d9 100644
--- a/src/PVE/Tools.pm
+++ b/src/PVE/Tools.pm
@@ -497,7 +497,7 @@ sub run_command {
if ($h eq $reader) {
if ($outfunc || $logfunc) {
eval {
- while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//) {
+ while ($buf =~ s/^([^\010\r\n]*)(?:\n|(?:\010)+|\r\n?)//) {
my $line = $outlog . $1;
$outlog = '';
&$outfunc($line) if $outfunc;
@@ -518,7 +518,7 @@ sub run_command {
} elsif ($h eq $error) {
if ($errfunc || $logfunc) {
eval {
- while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
+ while ($buf =~ s/^([^\010\r\n]*)(?:\n|(?:\010)+|\r\n?)//) {
my $line = $errlog . $1;
$errlog = '';
&$errfunc($line) if $errfunc;
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-08-19 7:01 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-30 9:04 [pve-devel] [PATCH common v2] run_command: improve performance for logging and long lines Dominik Csapak
2020-08-19 7:00 ` [pve-devel] applied: " Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox