* [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log
@ 2022-11-16 11:02 Leo Nunner
2023-08-30 8:22 ` Fiona Ebner
0 siblings, 1 reply; 3+ messages in thread
From: Leo Nunner @ 2022-11-16 11:02 UTC (permalink / raw)
To: pve-devel
dd supports a 'status' flag, which enables it to show the copied bytes,
duration, and the transfer rate, which then get printed to stderr.
Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
---
Changes from v1:
- Add rate-limit for dd output with the intervals suggested by
Thomas
PVE/Storage.pm | 23 ++++++++++++++++++-----
PVE/Storage/LVMPlugin.pm | 2 +-
PVE/Storage/Plugin.pm | 4 ++--
3 files changed, 21 insertions(+), 8 deletions(-)
diff --git a/PVE/Storage.pm b/PVE/Storage.pm
index c21b85e..dfceca9 100755
--- a/PVE/Storage.pm
+++ b/PVE/Storage.pm
@@ -821,12 +821,25 @@ sub storage_migrate {
my $new_volid;
my $pattern = volume_imported_message(undef, 1);
- my $match_volid_and_log = sub {
+ # Matches new volid and rate-limits dd output
+ my $match_and_log = sub {
my $line = shift;
+ my $show = 1;
+
+ # rate-limit dd logs
+ if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) {
+ if ($1 < 60) { # if < 60s, print every 3s
+ $show = !($1 % 3);
+ } elsif($1 < 600) { # if < 10mins, print every 10s
+ $show = !($1 % 10);
+ } else { # else, print every 30s
+ $show = !($1 % 30);
+ }
+ }
$new_volid = $1 if ($line =~ $pattern);
- if ($logfunc) {
+ if ($logfunc && $show) {
chomp($line);
$logfunc->($line);
}
@@ -855,7 +868,7 @@ sub storage_migrate {
# we won't be reading from the socket
shutdown($socket, 0);
- eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $logfunc); };
+ eval { run_command($cmds, output => '>&'.fileno($socket), errfunc => $match_and_log); };
my $send_error = $@;
# don't close the connection entirely otherwise the receiving end
@@ -864,7 +877,7 @@ sub storage_migrate {
# wait for the remote process to finish
while (my $line = <$info>) {
- $match_volid_and_log->("[$target_sshinfo->{name}] $line");
+ $match_and_log->("[$target_sshinfo->{name}] $line");
}
# now close the socket
@@ -877,7 +890,7 @@ sub storage_migrate {
die $send_error if $send_error;
} else {
push @$cmds, $recv;
- run_command($cmds, logfunc => $match_volid_and_log);
+ run_command($cmds, logfunc => $match_and_log);
}
die "unable to get ID of the migrated volume\n"
diff --git a/PVE/Storage/LVMPlugin.pm b/PVE/Storage/LVMPlugin.pm
index a706e0c..4b951e7 100644
--- a/PVE/Storage/LVMPlugin.pm
+++ b/PVE/Storage/LVMPlugin.pm
@@ -645,7 +645,7 @@ sub volume_export {
$size = int($1);
});
PVE::Storage::Plugin::write_common_header($fh, $size);
- run_command(['dd', "if=$file", "bs=64k"], output => '>&'.fileno($fh));
+ run_command(['dd', "if=$file", "bs=64k", "status=progress"], output => '>&'.fileno($fh));
}
sub volume_import_formats {
diff --git a/PVE/Storage/Plugin.pm b/PVE/Storage/Plugin.pm
index 8a41df1..e35fa97 100644
--- a/PVE/Storage/Plugin.pm
+++ b/PVE/Storage/Plugin.pm
@@ -1496,7 +1496,7 @@ sub volume_export {
goto unsupported if $with_snapshots || $file_format eq 'subvol';
write_common_header($fh, $size);
if ($file_format eq 'raw') {
- run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh));
+ run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh));
} else {
run_command(['qemu-img', 'convert', '-f', $file_format, '-O', 'raw', $file, '/dev/stdout'],
output => '>&'.fileno($fh));
@@ -1506,7 +1506,7 @@ sub volume_export {
my $data_format = $1;
goto unsupported if !$with_snapshots || $file_format ne $data_format;
write_common_header($fh, $size);
- run_command(['dd', "if=$file", "bs=4k"], output => '>&'.fileno($fh));
+ run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh));
return;
} elsif ($format eq 'tar+size') {
goto unsupported if $file_format ne 'subvol';
--
2.30.2
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log
2022-11-16 11:02 [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log Leo Nunner
@ 2023-08-30 8:22 ` Fiona Ebner
2023-08-31 9:00 ` Leo Nunner
0 siblings, 1 reply; 3+ messages in thread
From: Fiona Ebner @ 2023-08-30 8:22 UTC (permalink / raw)
To: Proxmox VE development discussion, Leo Nunner
Sorry about the late review!
Am 16.11.22 um 12:02 schrieb Leo Nunner:
> --- a/PVE/Storage.pm
> +++ b/PVE/Storage.pm
Needs a rebase, because files got moved to src/
> @@ -821,12 +821,25 @@ sub storage_migrate {
>
> my $new_volid;
> my $pattern = volume_imported_message(undef, 1);
> - my $match_volid_and_log = sub {
> + # Matches new volid and rate-limits dd output
> + my $match_and_log = sub {
Why rename the function?
> my $line = shift;
> + my $show = 1;
> +
> + # rate-limit dd logs
> + if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) {
> + if ($1 < 60) { # if < 60s, print every 3s
> + $show = !($1 % 3);
> + } elsif($1 < 600) { # if < 10mins, print every 10s
Style nit: missing space after elsif
> + $show = !($1 % 10);
> + } else { # else, print every 30s
> + $show = !($1 % 30);
> + }
> + }
Upon completion something strange happens: There's a duplicate final log
with fractional seconds and duplicate records in/out.
> 2023-08-30 09:57:05 Formatting '/mnt/pve/dir/images/105/vm-105-disk-0.raw', fmt=raw size=1073741824 preallocation=off
> 2023-08-30 09:57:08 1050218496 bytes (1.1 GB, 1002 MiB) copied, 3 s, 350 MB/s
> 2023-08-30 09:57:08 262144+0 records in
> 2023-08-30 09:57:08 262144+0 records out
> 2023-08-30 09:57:08 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.06228 s, 351 MB/s
> 2023-08-30 09:57:08 10458+11861 records in
> 2023-08-30 09:57:08 10458+11861 records out
> 2023-08-30 09:57:08 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.75082 s, 390 MB/s
> 2023-08-30 09:57:08 successfully imported 'dir:105/vm-105-disk-0.raw'
> 2023-08-30 10:00:38 Logical volume "vm-105-disk-0" created.
> 2023-08-30 10:00:42 873070592 bytes (873 MB, 833 MiB) copied, 3 s, 291 MB/s
> 2023-08-30 10:00:45 1944322048 bytes (1.9 GB, 1.8 GiB) copied, 6 s, 324 MB/s
> 2023-08-30 10:00:48 2976448512 bytes (3.0 GB, 2.8 GiB) copied, 9 s, 331 MB/s
> 2023-08-30 10:00:51 3998810112 bytes (4.0 GB, 3.7 GiB) copied, 12 s, 333 MB/s
> 2023-08-30 10:00:51 65536+0 records in
> 2023-08-30 10:00:51 65536+0 records out
> 2023-08-30 10:00:51 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.1171 s, 327 MB/s
> 2023-08-30 10:00:53 44465+42142 records in
> 2023-08-30 10:00:53 44465+42142 records out
> 2023-08-30 10:00:53 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 14.4084 s, 298 MB/s
> 2023-08-30 10:00:53 successfully imported 'lvmthin:vm-105-disk-0'
I think it's because we get it once from the source and once from the
target as better seen when using insecure migration:
> 2023-08-30 10:10:07 262144+0 records in
> 2023-08-30 10:10:07 262144+0 records out
> 2023-08-30 10:10:07 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.43834 s, 747 MB/s
> 2023-08-30 10:10:07 [pve8a2] Formatting '/mnt/pve/dir/images/105/vm-105-disk-0.raw', fmt=raw size=1073741824 preallocation=off
> 2023-08-30 10:10:07 [pve8a2] 6997+25632 records in
> 2023-08-30 10:10:07 [pve8a2] 6997+25632 records out
> 2023-08-30 10:10:07 [pve8a2] 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.41876 s, 757 MB/s
> 2023-08-30 10:10:07 [pve8a2] successfully imported 'dir:105/vm-105-disk-0.raw'
> 2023-08-30 10:10:07 volume 'dir:105/vm-105-disk-0.raw' is 'dir:105/vm-105-disk-0.raw' on the target
For insecure migration, we know which logs originate from source and
which from the target, so we could avoid the confusing duplicates. Maybe
there is a not too-involved way for secure migration too?
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log
2023-08-30 8:22 ` Fiona Ebner
@ 2023-08-31 9:00 ` Leo Nunner
0 siblings, 0 replies; 3+ messages in thread
From: Leo Nunner @ 2023-08-31 9:00 UTC (permalink / raw)
To: Proxmox VE development discussion
Thanks for the review!
On 2023-08-30 10:22, Fiona Ebner wrote:
> Sorry about the late review!
>
> Am 16.11.22 um 12:02 schrieb Leo Nunner:
>> --- a/PVE/Storage.pm
>> +++ b/PVE/Storage.pm
> Needs a rebase, because files got moved to src/
>
>> @@ -821,12 +821,25 @@ sub storage_migrate {
>>
>> my $new_volid;
>> my $pattern = volume_imported_message(undef, 1);
>> - my $match_volid_and_log = sub {
>> + # Matches new volid and rate-limits dd output
>> + my $match_and_log = sub {
> Why rename the function?
>
>> my $line = shift;
>> + my $show = 1;
>> +
>> + # rate-limit dd logs
>> + if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) {
>> + if ($1 < 60) { # if < 60s, print every 3s
>> + $show = !($1 % 3);
>> + } elsif($1 < 600) { # if < 10mins, print every 10s
> Style nit: missing space after elsif
>
>> + $show = !($1 % 10);
>> + } else { # else, print every 30s
>> + $show = !($1 % 30);
>> + }
>> + }
ack, I'll fix those things in the v2.
> Upon completion something strange happens: There's a duplicate final log
> with fractional seconds and duplicate records in/out.
The duplicate logs seem to happen already without my patch:
> 2023-08-31 10:38:54 found local disk
> 'cluster-dir:100/vm-100-disk-0.raw' (attached)
> 2023-08-31 10:38:54 copying local disk images
> 2023-08-31 10:38:56 Formatting
> '/mnt/dir/images/100/vm-100-disk-0.raw', fmt=raw size=2147483648
> preallocation=off
> 2023-08-31 10:39:02 524288+0 records in
> 2023-08-31 10:39:02 524288+0 records out
> 2023-08-31 10:39:02 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.52443
> s, 329 MB/s
> 2023-08-31 10:39:02 16749+32053 records in
> 2023-08-31 10:39:02 16749+32053 records out
> 2023-08-31 10:39:02 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.22392
> s, 345 MB/s
> 2023-08-31 10:39:02 successfully imported
> 'cluster-dir:100/vm-100-disk-0.raw'
> 2023-08-31 10:39:02 volume 'cluster-dir:100/vm-100-disk-0.raw' is
> 'cluster-dir:100/vm-100-disk-0.raw' on the target
> 2023-08-31 10:39:03 migration finished successfully (duration 00:00:09)
> TASK OK
So this might be something that could be fixed in a separate, future patch.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-08-31 9:00 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-16 11:02 [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log Leo Nunner
2023-08-30 8:22 ` Fiona Ebner
2023-08-31 9:00 ` Leo Nunner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal