From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <s.reiter@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id C49B761C03
for <pbs-devel@lists.proxmox.com>; Mon, 7 Sep 2020 17:31:17 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 68926D668
for <pbs-devel@lists.proxmox.com>; Mon, 7 Sep 2020 17:30:47 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[212.186.127.180])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id E6398D5F7
for <pbs-devel@lists.proxmox.com>; Mon, 7 Sep 2020 17:30:44 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id AD7EC44A8B
for <pbs-devel@lists.proxmox.com>; Mon, 7 Sep 2020 17:30:44 +0200 (CEST)
From: Stefan Reiter <s.reiter@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Mon, 7 Sep 2020 17:30:35 +0200
Message-Id: <20200907153036.9324-5-s.reiter@proxmox.com>
X-Mailer: git-send-email 2.20.1
In-Reply-To: <20200907153036.9324-1-s.reiter@proxmox.com>
References: <20200907153036.9324-1-s.reiter@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.053 Adjusted score from AWL reputation of From: address
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/,
medium trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] [PATCH v2 proxmox-backup 4/5] backup: check all
referenced chunks actually exist
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Mon, 07 Sep 2020 15:31:17 -0000
A client can omit uploading chunks in the "known_chunks" list, those
then also won't be written on the server side. Check all those chunks
mentioned in the index but not uploaded for existance and report an
error if they don't exist instead of marking a potentially broken backup
as "successful".
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---
src/api2/backup/environment.rs | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 973563d3..df22b1d6 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -1,6 +1,6 @@
use anyhow::{bail, format_err, Error};
use std::sync::{Arc, Mutex};
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use ::serde::{Serialize};
use serde_json::{json, Value};
@@ -73,6 +73,7 @@ struct SharedBackupState {
dynamic_writers: HashMap<usize, DynamicWriterState>,
fixed_writers: HashMap<usize, FixedWriterState>,
known_chunks: HashMap<[u8;32], u32>,
+ touched_chunks: HashSet<[u8;32]>,
backup_size: u64, // sums up size of all files
backup_stat: UploadStatistic,
}
@@ -126,6 +127,7 @@ impl BackupEnvironment {
dynamic_writers: HashMap::new(),
fixed_writers: HashMap::new(),
known_chunks: HashMap::new(),
+ touched_chunks: HashSet::new(),
backup_size: 0,
backup_stat: UploadStatistic::new(),
};
@@ -196,6 +198,7 @@ impl BackupEnvironment {
// register chunk
state.known_chunks.insert(digest, size);
+ state.touched_chunks.insert(digest);
Ok(())
}
@@ -229,6 +232,7 @@ impl BackupEnvironment {
// register chunk
state.known_chunks.insert(digest, size);
+ state.touched_chunks.insert(digest);
Ok(())
}
@@ -490,6 +494,21 @@ impl BackupEnvironment {
}
}
+ // make sure all chunks that were referenced actually exist
+ for (digest, _) in state.known_chunks.iter() {
+ // if they were uploaded just now they have already been touched
+ if state.touched_chunks.contains(digest) {
+ continue;
+ }
+
+ if !self.datastore.chunk_path(digest).0.exists() {
+ bail!(
+ "chunk '{}' was attempted to be reused but doesn't exist",
+ digest_to_hex(digest)
+ );
+ }
+ }
+
// marks the backup as successful
state.finished = true;
--
2.20.1