From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id EB65862816 for ; Wed, 30 Sep 2020 16:16:10 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E898E1A2F6 for ; Wed, 30 Sep 2020 16:16:10 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id CE4D41A2D3 for ; Wed, 30 Sep 2020 16:16:08 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 9295445A45 for ; Wed, 30 Sep 2020 16:16:08 +0200 (CEST) From: Stefan Reiter To: pbs-devel@lists.proxmox.com Date: Wed, 30 Sep 2020 16:15:56 +0200 Message-Id: <20200930141601.27233-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.047 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [pull.rs, environment.rs, verify.rs] Subject: [pbs-devel] [PATCH v2 0/5] backup validation improvements X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Sep 2020 14:16:11 -0000 First patch skips server-side validation (existance check) of last snapshot's chunks if the base was verified successfully at least 7 days ago. With verify scheduling this means that users can avoid the overhead during backups by doing weekly verifications. The latter 4 patches (marked RFC, not sure if the idea is that sound) make the validation happen in a seperate thread. This requires some modifications to ParallelHandler and carries the tradeoff that we have to verify *all* chunks of the last backup, since when the thread starts we don't know yet which ones the client will upload and overwrite anyway. v2: * use proxmox::tools::time in patch 1 * others unchanged proxmox-backup: Stefan Reiter (5): backup: don't validate chunk existance if base was recently verified ParallelHandler add unbounded mode ParallelHandler: add check_abort function and handle errors during join ParallelHandler: exit early if this or other thread aborted backup: validate chunk existance in background src/api2/backup/environment.rs | 175 ++++++++++++++++++++++++--------- src/backup/verify.rs | 3 +- src/client/pull.rs | 3 +- src/tools/parallel_handler.rs | 68 ++++++++++--- 4 files changed, 185 insertions(+), 64 deletions(-) -- 2.20.1