From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A1D2560CCE for ; Thu, 3 Sep 2020 16:17:13 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 976A71A111 for ; Thu, 3 Sep 2020 16:17:13 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 56EFB1A0F9 for ; Thu, 3 Sep 2020 16:17:12 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 034F1449FE for ; Thu, 3 Sep 2020 16:17:12 +0200 (CEST) From: Stefan Reiter To: pbs-devel@lists.proxmox.com Date: Thu, 3 Sep 2020 16:17:05 +0200 Message-Id: <20200903141705.6344-5-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200903141705.6344-1-s.reiter@proxmox.com> References: <20200903141705.6344-1-s.reiter@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.046 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [environment.rs] Subject: [pbs-devel] [PATCH proxmox-backup 4/4] backup: check all referenced chunks actually exist X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Sep 2020 14:17:13 -0000 A client can omit uploading chunks in the "known_chunks" list, those then also won't be written on the server side. Check all those chunks mentioned in the index but not uploaded for existance and report an error if they don't exist instead of marking a potentially broken backup as "successful". Signed-off-by: Stefan Reiter --- src/api2/backup/environment.rs | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs index 973563d3..df22b1d6 100644 --- a/src/api2/backup/environment.rs +++ b/src/api2/backup/environment.rs @@ -1,6 +1,6 @@ use anyhow::{bail, format_err, Error}; use std::sync::{Arc, Mutex}; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use ::serde::{Serialize}; use serde_json::{json, Value}; @@ -73,6 +73,7 @@ struct SharedBackupState { dynamic_writers: HashMap, fixed_writers: HashMap, known_chunks: HashMap<[u8;32], u32>, + touched_chunks: HashSet<[u8;32]>, backup_size: u64, // sums up size of all files backup_stat: UploadStatistic, } @@ -126,6 +127,7 @@ impl BackupEnvironment { dynamic_writers: HashMap::new(), fixed_writers: HashMap::new(), known_chunks: HashMap::new(), + touched_chunks: HashSet::new(), backup_size: 0, backup_stat: UploadStatistic::new(), }; @@ -196,6 +198,7 @@ impl BackupEnvironment { // register chunk state.known_chunks.insert(digest, size); + state.touched_chunks.insert(digest); Ok(()) } @@ -229,6 +232,7 @@ impl BackupEnvironment { // register chunk state.known_chunks.insert(digest, size); + state.touched_chunks.insert(digest); Ok(()) } @@ -490,6 +494,21 @@ impl BackupEnvironment { } } + // make sure all chunks that were referenced actually exist + for (digest, _) in state.known_chunks.iter() { + // if they were uploaded just now they have already been touched + if state.touched_chunks.contains(digest) { + continue; + } + + if !self.datastore.chunk_path(digest).0.exists() { + bail!( + "chunk '{}' was attempted to be reused but doesn't exist", + digest_to_hex(digest) + ); + } + } + // marks the backup as successful state.finished = true; -- 2.20.1