From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D0D8DA817 for ; Wed, 30 Aug 2023 10:22:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B1C8B109E0 for ; Wed, 30 Aug 2023 10:22:14 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 30 Aug 2023 10:22:10 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C58BB47703 for ; Wed, 30 Aug 2023 10:22:09 +0200 (CEST) Message-ID: Date: Wed, 30 Aug 2023 10:22:08 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Content-Language: en-US To: Proxmox VE development discussion , Leo Nunner References: <20221116110233.54160-1-l.nunner@proxmox.com> From: Fiona Ebner In-Reply-To: <20221116110233.54160-1-l.nunner@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.547 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -1.242 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [storage.pm] Subject: Re: [pve-devel] [PATCH v2 storage] fix #3004: show progress of offline migration in task log X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Aug 2023 08:22:44 -0000 Sorry about the late review! Am 16.11.22 um 12:02 schrieb Leo Nunner: > --- a/PVE/Storage.pm > +++ b/PVE/Storage.pm Needs a rebase, because files got moved to src/ > @@ -821,12 +821,25 @@ sub storage_migrate { > > my $new_volid; > my $pattern = volume_imported_message(undef, 1); > - my $match_volid_and_log = sub { > + # Matches new volid and rate-limits dd output > + my $match_and_log = sub { Why rename the function? > my $line = shift; > + my $show = 1; > + > + # rate-limit dd logs > + if ($line =~ /(?:\d+ bytes)(?:.+?copied, )(\d+) s/) { > + if ($1 < 60) { # if < 60s, print every 3s > + $show = !($1 % 3); > + } elsif($1 < 600) { # if < 10mins, print every 10s Style nit: missing space after elsif > + $show = !($1 % 10); > + } else { # else, print every 30s > + $show = !($1 % 30); > + } > + } Upon completion something strange happens: There's a duplicate final log with fractional seconds and duplicate records in/out. > 2023-08-30 09:57:05 Formatting '/mnt/pve/dir/images/105/vm-105-disk-0.raw', fmt=raw size=1073741824 preallocation=off > 2023-08-30 09:57:08 1050218496 bytes (1.1 GB, 1002 MiB) copied, 3 s, 350 MB/s > 2023-08-30 09:57:08 262144+0 records in > 2023-08-30 09:57:08 262144+0 records out > 2023-08-30 09:57:08 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.06228 s, 351 MB/s > 2023-08-30 09:57:08 10458+11861 records in > 2023-08-30 09:57:08 10458+11861 records out > 2023-08-30 09:57:08 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.75082 s, 390 MB/s > 2023-08-30 09:57:08 successfully imported 'dir:105/vm-105-disk-0.raw' > 2023-08-30 10:00:38 Logical volume "vm-105-disk-0" created. > 2023-08-30 10:00:42 873070592 bytes (873 MB, 833 MiB) copied, 3 s, 291 MB/s > 2023-08-30 10:00:45 1944322048 bytes (1.9 GB, 1.8 GiB) copied, 6 s, 324 MB/s > 2023-08-30 10:00:48 2976448512 bytes (3.0 GB, 2.8 GiB) copied, 9 s, 331 MB/s > 2023-08-30 10:00:51 3998810112 bytes (4.0 GB, 3.7 GiB) copied, 12 s, 333 MB/s > 2023-08-30 10:00:51 65536+0 records in > 2023-08-30 10:00:51 65536+0 records out > 2023-08-30 10:00:51 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.1171 s, 327 MB/s > 2023-08-30 10:00:53 44465+42142 records in > 2023-08-30 10:00:53 44465+42142 records out > 2023-08-30 10:00:53 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 14.4084 s, 298 MB/s > 2023-08-30 10:00:53 successfully imported 'lvmthin:vm-105-disk-0' I think it's because we get it once from the source and once from the target as better seen when using insecure migration: > 2023-08-30 10:10:07 262144+0 records in > 2023-08-30 10:10:07 262144+0 records out > 2023-08-30 10:10:07 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.43834 s, 747 MB/s > 2023-08-30 10:10:07 [pve8a2] Formatting '/mnt/pve/dir/images/105/vm-105-disk-0.raw', fmt=raw size=1073741824 preallocation=off > 2023-08-30 10:10:07 [pve8a2] 6997+25632 records in > 2023-08-30 10:10:07 [pve8a2] 6997+25632 records out > 2023-08-30 10:10:07 [pve8a2] 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.41876 s, 757 MB/s > 2023-08-30 10:10:07 [pve8a2] successfully imported 'dir:105/vm-105-disk-0.raw' > 2023-08-30 10:10:07 volume 'dir:105/vm-105-disk-0.raw' is 'dir:105/vm-105-disk-0.raw' on the target For insecure migration, we know which logs originate from source and which from the target, so we could avoid the confusing duplicates. Maybe there is a not too-involved way for secure migration too?