From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id B2D2E8797 for ; Thu, 31 Aug 2023 12:42:06 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 95EB98B5C for ; Thu, 31 Aug 2023 12:42:06 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 31 Aug 2023 12:42:05 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DF86D47AFB for ; Thu, 31 Aug 2023 12:42:04 +0200 (CEST) From: Leo Nunner To: pve-devel@lists.proxmox.com Date: Thu, 31 Aug 2023 12:41:59 +0200 Message-Id: <20230831104159.119407-1-l.nunner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.100 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [storage.pm, plugin.pm, lvmplugin.pm] Subject: [pve-devel] [PATCH v3 storage] fix #3004: show progress of offline migration in task log X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Aug 2023 10:42:06 -0000 dd supports a 'status' flag, which enables it to show the copied bytes, duration, and the transfer rate, which then get printed to stderr. Signed-off-by: Leo Nunner --- Changes since v2: - Refactor to new repo structure - Implement style changes suggested for v2 src/PVE/Storage.pm | 17 +++++++++++++++-- src/PVE/Storage/LVMPlugin.pm | 2 +- src/PVE/Storage/Plugin.pm | 4 ++-- 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index 71fd4be..7990e0e 100755 --- a/src/PVE/Storage.pm +++ b/src/PVE/Storage.pm @@ -821,12 +821,25 @@ sub storage_migrate { my $new_volid; my $pattern = volume_imported_message(undef, 1); + # Matches new volid and rate-limits dd output my $match_volid_and_log = sub { my $line = shift; + my $show = 1; + + # rate-limit dd logs + if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) { + if ($1 < 60) { # if < 60s, print every 3s + $show = !($1 % 3); + } elsif ($1 < 600) { # if < 10mins, print every 10s + $show = !($1 % 10); + } else { # else, print every 30s + $show = !($1 % 30); + } + } $new_volid = $1 if ($line =~ $pattern); - if ($logfunc) { + if ($logfunc && $show) { chomp($line); $logfunc->($line); } @@ -855,7 +868,7 @@ sub storage_migrate { # we won't be reading from the socket shutdown($socket, 0); - eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $logfunc); }; + eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $match_volid_and_log); }; my $send_error = $@; # don't close the connection entirely otherwise the receiving end diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index a706e0c..4b951e7 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -645,7 +645,7 @@ sub volume_export { $size = int($1); }); PVE::Storage::Plugin::write_common_header($fh, $size); - run_command(['dd', "if=$file", "bs=64k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=64k", "status=progress"], output => '>&'.fileno($fh)); } sub volume_import_formats { diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index 0f0fde4..815773b 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -1577,7 +1577,7 @@ sub volume_export { goto unsupported if $with_snapshots || $file_format eq 'subvol'; write_common_header($fh, $size); if ($file_format eq 'raw') { - run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh)); } else { run_command(['qemu-img', 'convert', '-f', $file_format, '-O', 'raw', $file, '/dev/stdout'], output => '>&'.fileno($fh)); @@ -1587,7 +1587,7 @@ sub volume_export { my $data_format = $1; goto unsupported if !$with_snapshots || $file_format ne $data_format; write_common_header($fh, $size); - run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh)); + run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh)); return; } elsif ($format eq 'tar+size') { goto unsupported if $file_format ne 'subvol'; -- 2.39.2