From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 884A96BDC8 for ; Thu, 28 Jan 2021 11:52:06 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 71A2F2C61E for ; Thu, 28 Jan 2021 11:51:36 +0100 (CET) Received: from p02.hosting.plutex.de (p02.hosting.plutex.de [IPv6:2a02:16d0:0:200::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 4A8C82C60F for ; Thu, 28 Jan 2021 11:51:35 +0100 (CET) Received: from TS12 (unknown [109.69.64.221]) by p02.hosting.plutex.de (Postfix) with ESMTPSA id 55FA01723210 for ; Thu, 28 Jan 2021 11:51:22 +0100 (CET) From: "Jochen Plumeyer" To: Date: Thu, 28 Jan 2021 11:51:22 +0100 Message-ID: <00da01d6f563$8448f6b0$8cdae410$@uhlmann-ludewig.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: de Thread-Index: Adb1Ymq2H6KvkaCiRQaQvAuaBvmoFA== X-PPP-Message-ID: <161183108257.39438.14109570011590346580@p02.hosting.plutex.de> X-PPP-Vhost: uhlmann-ludewig.de X-SPAM-LEVEL: Spam detection results: 1 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record Subject: [pve-devel] PATCH pve-zsync compress option for gz-compressed sync transfers X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 10:52:06 -0000 Hi everybody, according to my measurements, gz-compressing zfs syncs can reduce about 70% of data traffic during pve-zsync operation. Hence, the option "--compress" has been added in this patch. If enabled, the existing traffic is directed through " | gzip -c " --SYNC STREAM--> "gzip -d |" pipe commands. Testing has been done, today we will deploy it in production. Merging would be appreciated. Cheers, Jochen Plumeyer diff --git a/pve-zsync b/pve-zsync index 5c95955..57f722e 100755 --- a/pve-zsync +++ b/pve-zsync @@ -64,6 +64,7 @@ if (defined($command) && $command ne 'help' && $command ne 'printpod') { check_bin ('zfs'); check_bin ('ssh'); check_bin ('scp'); + check_bin ('gzip'); } $SIG{TERM} = $SIG{QUIT} = $SIG{PIPE} = $SIG{HUP} = $SIG{KILL} = $SIG{INT} = sub { @@ -229,6 +230,7 @@ sub parse_argv { dest => undef, source => undef, verbose => undef, + compress => undef, limit => undef, maxsnap => undef, name => undef, @@ -245,6 +247,7 @@ sub parse_argv { 'dest=s' => \$param->{dest}, 'source=s' => \$param->{source}, 'verbose' => \$param->{verbose}, + 'compress' => \$param->{compress}, 'limit=i' => \$param->{limit}, 'maxsnap=i' => \$param->{maxsnap}, 'name=s' => \$param->{name}, @@ -446,6 +449,7 @@ sub format_job { $text .= " --limit $job->{limit}" if $job->{limit}; $text .= " --method $job->{method}"; $text .= " --verbose" if $job->{verbose}; + $text .= " --compress" if $job->{compress}; $text .= " --source-user $job->{source_user}"; $text .= " --dest-user $job->{dest_user}"; $text .= " --properties" if $job->{properties}; @@ -967,6 +971,8 @@ sub send_image { } push @$cmd, '--', "$source->{all}\@$source->{new_snap}"; + push @$cmd, \'|', 'gzip', '-c' if $param->{compress}; + if ($param->{limit}){ my $bwl = $param->{limit}*1024; push @$cmd, \'|', 'cstream', '-t', $bwl; @@ -977,8 +983,10 @@ sub send_image { push @$cmd, \'|'; push @$cmd, 'ssh', '-o', 'BatchMode=yes', "$param->{dest_user}\@$dest->{ip}", '--' if $dest->{ip}; + push @$cmd, \'"', 'gzip', '-d', \'|' if $param->{compress}; push @$cmd, 'zfs', 'recv', '-F', '--'; push @$cmd, "$target"; + push @$cmd, \'"' if $param->{compress}; eval { run_cmd($cmd) @@ -1172,6 +1180,10 @@ $PROGNAME sync -dest -source [OPTIONS]\n print out the sync progress. + -compress boolean + + use gzip compression for sync transfer + -properties boolean Include the dataset's properties in the stream.