From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 284A568B5F for ; Wed, 10 Feb 2021 13:06:43 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1538C83E9 for ; Wed, 10 Feb 2021 13:06:13 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 3E47483D7 for ; Wed, 10 Feb 2021 13:06:09 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0776F46200; Wed, 10 Feb 2021 13:06:09 +0100 (CET) To: pve-devel@lists.proxmox.com, Jochen Plumeyer References: <00da01d6f563$8448f6b0$8cdae410$@uhlmann-ludewig.de> From: Fabian Ebner Message-ID: <1c96f370-e415-ff15-a092-3327f5c3957c@proxmox.com> Date: Wed, 10 Feb 2021 13:06:02 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <00da01d6f563$8448f6b0$8cdae410$@uhlmann-ludewig.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.132 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.265 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] PATCH pve-zsync compress option for gz-compressed sync transfers X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2021 12:06:43 -0000 Hi, first of all, sorry for the late reply. It seems that long lines got warped and the patch therefore does not apply. Please use 'git format-patch' and/or 'git send-email'. Is there a specific reason why you opted for gzip with the default options instead of e.g. zstd? ZFS itself can produce compressed streams if the datasets are compressed, and if we ever add that in the future, the option names might get confusing. Maybe there's a more descriptive name than just 'compress'? One more comment inline: Am 28.01.21 um 11:51 schrieb Jochen Plumeyer: > Hi everybody, > > according to my measurements, gz-compressing zfs syncs can reduce about 70% > of data traffic during pve-zsync operation. > > Hence, the option "--compress" has been added in this patch. > If enabled, the existing traffic is directed through " | gzip -c " --SYNC > STREAM--> "gzip -d |" pipe commands. > > Testing has been done, today we will deploy it in production. > > Merging would be appreciated. > > Cheers, > > Jochen Plumeyer > > diff --git a/pve-zsync b/pve-zsync > index 5c95955..57f722e 100755 > --- a/pve-zsync > +++ b/pve-zsync > @@ -64,6 +64,7 @@ if (defined($command) && $command ne 'help' && $command ne > 'printpod') { > check_bin ('zfs'); > check_bin ('ssh'); > check_bin ('scp'); > + check_bin ('gzip'); This check should only be executed if the compress option is set. > } > > $SIG{TERM} = $SIG{QUIT} = $SIG{PIPE} = $SIG{HUP} = $SIG{KILL} = $SIG{INT} = > sub { > @@ -229,6 +230,7 @@ sub parse_argv { > dest => undef, > source => undef, > verbose => undef, > + compress => undef, > limit => undef, > maxsnap => undef, > name => undef, > @@ -245,6 +247,7 @@ sub parse_argv { > 'dest=s' => \$param->{dest}, > 'source=s' => \$param->{source}, > 'verbose' => \$param->{verbose}, > + 'compress' => \$param->{compress}, > 'limit=i' => \$param->{limit}, > 'maxsnap=i' => \$param->{maxsnap}, > 'name=s' => \$param->{name}, > @@ -446,6 +449,7 @@ sub format_job { > $text .= " --limit $job->{limit}" if $job->{limit}; > $text .= " --method $job->{method}"; > $text .= " --verbose" if $job->{verbose}; > + $text .= " --compress" if $job->{compress}; > $text .= " --source-user $job->{source_user}"; > $text .= " --dest-user $job->{dest_user}"; > $text .= " --properties" if $job->{properties}; > @@ -967,6 +971,8 @@ sub send_image { > } > push @$cmd, '--', "$source->{all}\@$source->{new_snap}"; > > + push @$cmd, \'|', 'gzip', '-c' if $param->{compress}; > + > if ($param->{limit}){ > my $bwl = $param->{limit}*1024; > push @$cmd, \'|', 'cstream', '-t', $bwl; > @@ -977,8 +983,10 @@ sub send_image { > > push @$cmd, \'|'; > push @$cmd, 'ssh', '-o', 'BatchMode=yes', > "$param->{dest_user}\@$dest->{ip}", '--' if $dest->{ip}; > + push @$cmd, \'"', 'gzip', '-d', \'|' if $param->{compress}; > push @$cmd, 'zfs', 'recv', '-F', '--'; > push @$cmd, "$target"; > + push @$cmd, \'"' if $param->{compress}; > > eval { > run_cmd($cmd) > @@ -1172,6 +1180,10 @@ $PROGNAME sync -dest -source > [OPTIONS]\n > > print out the sync progress. > > + -compress boolean > + > + use gzip compression for sync transfer > + > -properties boolean > > Include the dataset's properties in the stream. > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > >