From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 05E116029A for ; Tue, 8 Sep 2020 12:52:48 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id EDDC38FEC for ; Tue, 8 Sep 2020 12:52:17 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 0DAD28FDF for ; Tue, 8 Sep 2020 12:52:17 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C94F944AA0 for ; Tue, 8 Sep 2020 12:52:16 +0200 (CEST) Date: Tue, 8 Sep 2020 12:51:32 +0200 (CEST) From: Dietmar Maurer To: Proxmox Backup Server development discussion , Stefan Reiter Message-ID: <2026998658.958.1599562293031@webmail.proxmox.com> In-Reply-To: <20200907153036.9324-1-s.reiter@proxmox.com> References: <20200907153036.9324-1-s.reiter@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Normal X-Mailer: Open-Xchange Mailer v7.10.3-Rev21 X-Originating-Client: open-xchange-appsuite X-SPAM-LEVEL: Spam detection results: 0 AWL 0.086 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [environment.rs, mod.rs, proxmox.com, verify.rs, datastore.rs] Subject: Re: [pbs-devel] [PATCH v2 0/5] Improve corrupt chunk handling X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 10:52:48 -0000 applied 0, 1,2,3 and 5 patch 4/5 needs some improvements. > On 09/07/2020 5:30 PM Stefan Reiter wrote: > > > Verify will now rename chunks it detects as corrupted, so future backups will be > forced to write them. The next GC will then clean these ".bad" files up, since > it has to scan each chunk directory anyway. > > In case the last backup uses some of these chunks, but is not the one that > failed verification, the client may still omit these chunks, which could lead to > a broken backup. Patch 4 detects these cases by checking all referenced chunks > for existance (which certainly adds a bit of overhead, especially to otherwise > minimal dirty-bitmap backups). > > Additionally, the last patch makes sure all chunks have their atime updated, > even if they won't be written (when they already exist), to eliminate a race > with GC where the chunk might be missing after a successful backup. > > Also, friendly ping on: > https://lists.proxmox.com/pipermail/pbs-devel/2020-September/000479.html > This series makes the most sense with that patch already applied, the feedback > from Dominik is addressed here. > > v2: > * address Thomas' feedback (as well as Dietmar's comment) > * add patch 5 > > > proxmox-backup: Stefan Reiter (5): > verify: fix log units > verify: rename corrupted chunks with .bad extension > gc: remove .bad files on garbage collect > backup: check all referenced chunks actually exist > backup: touch all chunks, even if they exist > > src/api2/backup/environment.rs | 21 +++++++++- > src/api2/types/mod.rs | 3 ++ > src/backup/chunk_store.rs | 73 ++++++++++++++++++++++++++++------ > src/backup/datastore.rs | 5 ++- > src/backup/verify.rs | 34 +++++++++++++++- > 5 files changed, 120 insertions(+), 16 deletions(-) > > -- > 2.20.1 > > > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel