[pve-devel] [PATCH common v2] run_command: improve performance for logging and long lines

[pve-devel] [PATCH common v2] run_command: improve performance for logging and long lines

[Dominik Csapak]

to call out/err/logfunc with each line, we search for a newline and call
outfunc/logfunc with everything before that

since we do a select/read (with 4096 size) in a loop, this means
that if we have very long lines, we search for a newline in an
ever growing buffer (for which we know does not contain a newline)

so instead, only search the new data for newlines

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from v1:
* keep the substitution instead of a match, making the diff a little smaller
  this fixes a bug in the v1, when there were multiple lines in one read
 src/PVE/Tools.pm | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
index 4399b2f..f013fb7 100644
--- a/src/PVE/Tools.pm
+++ b/src/PVE/Tools.pm
@@ -496,12 +496,13 @@ sub run_command {
 		if ($h eq $reader) {
 		    if ($outfunc || $logfunc) {
 			eval {
-			    $outlog .= $buf;
-			    while ($outlog =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
-				my $line = $1;
+			    while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//) {
+				my $line = $outlog . $1;
+				$outlog = '';
 				&$outfunc($line) if $outfunc;
 				&$logfunc($line) if $logfunc;
 			    }
+			    $outlog .= $buf;
 			};
 			my $err = $@;
 			if ($err) {
@@ -516,12 +517,13 @@ sub run_command {
 		} elsif ($h eq $error) {
 		    if ($errfunc || $logfunc) {
 			eval {
-			    $errlog .= $buf;
-			    while ($errlog =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
-				my $line = $1;
+			    while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
+				my $line = $errlog . $1;
+				$errlog = '';
 				&$errfunc($line) if $errfunc;
 				&$logfunc($line) if $logfunc;
 			    }
+			    $errlog .= $buf;
 			};
 			my $err = $@;
 			if ($err) {
-- 
2.20.1

[pve-devel] applied: [PATCH common v2] run_command: improve performance for logging and long lines

[Thomas Lamprecht]

On 30.07.20 11:04, Dominik Csapak wrote:
> to call out/err/logfunc with each line, we search for a newline and call
> outfunc/logfunc with everything before that
> 
> since we do a select/read (with 4096 size) in a loop, this means
> that if we have very long lines, we search for a newline in an
> ever growing buffer (for which we know does not contain a newline)
> 
> so instead, only search the new data for newlines
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> changes from v1:
> * keep the substitution instead of a match, making the diff a little smaller
>   this fixes a bug in the v1, when there were multiple lines in one read
>  src/PVE/Tools.pm | 14 ++++++++------
>  1 file changed, 8 insertions(+), 6 deletions(-)
> 
>

applied, thanks! With the followup below, doing two things both already present
before your optimization, so just FYI:
* non-capturing group for things we do not use
* fix matching of \r\n sequence, as this was non-greedy and so a \r\n was matched
  as two lines, one with \r and one with \n. But, the regex clearly indicates that
  this wasn't intended.

Also, I made the change to not use the s modifier also for the $h eq $error case,
for consistency (it does not matters much, we do not use . here)

diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
index d9c69e3..f9270d9 100644
--- a/src/PVE/Tools.pm
+++ b/src/PVE/Tools.pm
@@ -497,7 +497,7 @@ sub run_command {
                if ($h eq $reader) {
                    if ($outfunc || $logfunc) {
                        eval {
-                           while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//) {
+                           while ($buf =~ s/^([^\010\r\n]*)(?:\n|(?:\010)+|\r\n?)//) {
                                my $line = $outlog . $1;
                                $outlog = '';
                                &$outfunc($line) if $outfunc;
@@ -518,7 +518,7 @@ sub run_command {
                } elsif ($h eq $error) {
                    if ($errfunc || $logfunc) {
                        eval {
-                           while ($buf =~ s/^([^\010\r\n]*)(\r|\n|(\010)+|\r\n)//s) {
+                           while ($buf =~ s/^([^\010\r\n]*)(?:\n|(?:\010)+|\r\n?)//) {
                                my $line = $errlog . $1;
                                $errlog = '';
                                &$errfunc($line) if $errfunc;