From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E8C0089E1 for ; Wed, 16 Nov 2022 12:03:19 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C9D4D1E0C2 for ; Wed, 16 Nov 2022 12:03:19 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 16 Nov 2022 12:03:19 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0C7BC44B9A for ; Wed, 16 Nov 2022 12:03:18 +0100 (CET) From: Leo Nunner To: pve-devel@lists.proxmox.com Date: Wed, 16 Nov 2022 12:02:33 +0100 Message-Id: <20221116110233.54160-1-l.nunner@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: =?UTF-8?Q?0=0A=09?=AWL -0.111 Adjusted score from AWL reputation of From: =?UTF-8?Q?address=0A=09?=BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict =?UTF-8?Q?Alignment=0A=09?=SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF =?UTF-8?Q?Record=0A=09?=SPF_PASS -0.001 SPF: sender matches SPF =?UTF-8?Q?record=0A=09?=URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [lvmplugin.pm, plugin.pm, storage.pm] Subject: [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2022 11:03:20 -0000 dd supports a 'status' flag, which enables it to show the copied bytes, duration, and the transfer rate, which then get printed to stderr. Signed-off-by: Leo Nunner --- Changes from v1: - Add rate-limit for dd output with the intervals suggested by Thomas PVE/Storage.pm | 23 ++++++++++++++++++----- PVE/Storage/LVMPlugin.pm | 2 +- PVE/Storage/Plugin.pm | 4 ++-- 3 files changed, 21 insertions(+), 8 deletions(-) diff --git a/PVE/Storage.pm b/PVE/Storage.pm index c21b85e..dfceca9 100755 --- a/PVE/Storage.pm +++ b/PVE/Storage.pm @@ -821,12 +821,25 @@ sub storage_migrate { my $new_volid; my $pattern = volume_imported_message(undef, 1); - my $match_volid_and_log = sub { + # Matches new volid and rate-limits dd output + my $match_and_log = sub { my $line = shift; + my $show = 1; + + # rate-limit dd logs + if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) { + if ($1 < 60) { # if < 60s, print every 3s + $show = !($1 % 3); + } elsif($1 < 600) { # if < 10mins, print every 10s + $show = !($1 % 10); + } else { # else, print every 30s + $show = !($1 % 30); + } + } $new_volid = $1 if ($line =~ $pattern); - if ($logfunc) { + if ($logfunc && $show) { chomp($line); $logfunc->($line); } @@ -855,7 +868,7 @@ sub storage_migrate { # we won't be reading from the socket shutdown($socket, 0); - eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $logfunc); }; + eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $match_and_log); }; my $send_error = $@; # don't close the connection entirely otherwise the receiving end @@ -864,7 +877,7 @@ sub storage_migrate { # wait for the remote process to finish while (my $line = <$info>) { - $match_volid_and_log->("[$target_sshinfo->{name}] $line"); + $match_and_log->("[$target_sshinfo->{name}] $line"); } # now close the socket @@ -877,7 +890,7 @@ sub storage_migrate { die $send_error if $send_error; } else { push @$cmds, $recv; - run_command($cmds, logfunc => $match_volid_and_log); + run_command($cmds, logfunc => $match_and_log); } die "unable to get ID of the migrated volume\n" diff --git a/PVE/Storage/LVMPlugin.pm b/PVE/Storage/LVMPlugin.pm index a706e0c..4b951e7 100644 --- a/PVE/Storage/LVMPlugin.pm +++ b/PVE/Storage/LVMPlugin.pm @@ -645,7 +645,7 @@ sub volume_export { $size = int($1); }); PVE::Storage::Plugin::write_common_header($fh, $size); - run_command(['dd', "if=$file", "bs=64k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=64k", "status=progress"], output => '>&'.fileno($fh)); } sub volume_import_formats { diff --git a/PVE/Storage/Plugin.pm b/PVE/Storage/Plugin.pm index 8a41df1..e35fa97 100644 --- a/PVE/Storage/Plugin.pm +++ b/PVE/Storage/Plugin.pm @@ -1496,7 +1496,7 @@ sub volume_export { goto unsupported if $with_snapshots || $file_format eq 'subvol'; write_common_header($fh, $size); if ($file_format eq 'raw') { - run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh)); } else { run_command(['qemu-img', 'convert', '-f', $file_format, '-O', 'raw', $file, '/dev/stdout'], output => '>&'.fileno($fh)); @@ -1506,7 +1506,7 @@ sub volume_export { my $data_format = $1; goto unsupported if !$with_snapshots || $file_format ne $data_format; write_common_header($fh, $size); - run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh)); return; } elsif ($format eq 'tar+size') { goto unsupported if $file_format ne 'subvol'; -- 2.30.2