From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors
Date: Mon, 16 Nov 2026 10:54:33 +0100 [thread overview]
Message-ID: <20261116095434.110541-12-c.ebner@proxmox.com> (raw)
In-Reply-To: <20261116095434.110541-1-c.ebner@proxmox.com>
Fetching an object for verification might fail when sending the
request to the S3 API or while receiving the response. Both are
currently handled as hard errors, leading to the manifest state being
set to verify failed. This however does not reflect the actual state,
as a subsequent verification can work just fine.
Therefore, distinguish between permanent, transient and verification
task related errors (abort/shutdown).
If only transient errors, but no permanent errors occur, strip any
verify okay state since then it is not possible to determine if the
snapshot is still fine. That is best represented by not having a
verify state, as consequence sync jobs with verified-only filtering
will then exclude such snapshots as well. Independent of the error
variant, always keep a previously set verify error state in order to
avoid allowing backups to reuse this snapshot as reference. This
covers both the new state as well as the manifest state prior to
verification. On task related errors, the verify state is not
altered at all.
Errors of the chunk listing, parallel chunk readers or decoders also
leave the verification incomplete, but are only considered after all
chunks already sent have been checked, so they never hide permanent
errors.
Avoid manifest updates if not needed, since these cause additional
requests on S3 backed datastores and might fail as well.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
src/backup/verify.rs | 252 ++++++++++++++++++++++++++++++++++---------
1 file changed, 201 insertions(+), 51 deletions(-)
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index d2325ce9a..624c9a9aa 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -1,10 +1,10 @@
use pbs_config::BackupLockGuard;
use std::collections::HashSet;
-use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Instant;
-use anyhow::{Error, bail};
+use anyhow::{Error, format_err};
use tracing::{error, info, warn};
use proxmox_parallel_handler::{ParallelHandler, SendHandle};
@@ -38,6 +38,7 @@ struct IndexVerifyState {
read_bytes: AtomicU64,
decoded_bytes: AtomicU64,
errors: AtomicUsize,
+ fetch_errors: AtomicUsize,
datastore: Arc<DataStore>,
corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
@@ -54,6 +55,7 @@ impl IndexVerifyState {
read_bytes: AtomicU64::new(0),
decoded_bytes: AtomicU64::new(0),
errors: AtomicUsize::new(0),
+ fetch_errors: AtomicUsize::new(0),
datastore: Arc::clone(datastore),
corrupt_chunks: Arc::clone(corrupt_chunks),
verified_chunks: Arc::clone(verified_chunks),
@@ -62,6 +64,17 @@ impl IndexVerifyState {
}
}
+/// Outcome of a failed archive verification, deciding how the snapshot's verify state is updated.
+enum VerificationError {
+ /// The archive is known to be bad, for example because of corrupt or missing chunks.
+ Permanent(Error),
+ /// The archive could not be fully verified, for example because chunks could not be fetched
+ /// from the S3 backend. Its state is unknown.
+ Transient(Error),
+ /// The task was aborted or the daemon is shutting down, unrelated to the archive itself.
+ Task(Error),
+}
+
impl VerifyWorker {
/// Creates a new VerifyWorker for a given task worker and datastore.
pub fn new(
@@ -94,27 +107,38 @@ impl VerifyWorker {
})
}
- fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
- let blob = backup_dir.load_blob(info.filename.as_ref())?;
+ fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
+ let blob = backup_dir
+ .load_blob(info.filename.as_ref())
+ .map_err(VerificationError::Permanent)?;
let raw_size = blob.raw_size();
if raw_size != info.size {
- bail!("wrong size ({} != {})", info.size, raw_size);
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong size ({} != {})",
+ info.size,
+ raw_size
+ )));
}
let csum = openssl::sha::sha256(blob.raw_data());
if csum != info.csum {
- bail!("wrong index checksum");
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong index checksum"
+ )));
}
- match blob.crypt_mode()? {
+ match blob.crypt_mode().map_err(VerificationError::Permanent)? {
CryptMode::Encrypt => Ok(()),
CryptMode::None => {
// digest already verified above
- blob.decode(None, None)?;
+ blob.decode(None, None)
+ .map_err(VerificationError::Permanent)?;
Ok(())
}
- CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
+ CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!(
+ "Invalid CryptMode for blob"
+ ))),
}
}
@@ -122,7 +146,7 @@ impl VerifyWorker {
&self,
index: Box<dyn IndexFile + Send>,
crypt_mode: CryptMode,
- ) -> Result<(), Error> {
+ ) -> Result<(), VerificationError> {
let verify_state = Arc::new(IndexVerifyState::new(
&self.datastore,
&self.corrupt_chunks,
@@ -179,17 +203,34 @@ impl VerifyWorker {
}
};
+ // Track if the task was aborted or shutdown, so only that is treated as task error and any
+ // other error of the chunk ordering leaves the archive unverified.
+ let task_aborted_or_shutdown = AtomicBool::new(false);
let check_abort = |pos: usize| -> Result<(), Error> {
if pos & 1023 == 0 {
- self.worker.check_abort()?;
- self.worker.fail_on_shutdown()?;
+ let result = self
+ .worker
+ .check_abort()
+ .and_then(|_| self.worker.fail_on_shutdown());
+ task_aborted_or_shutdown.store(result.is_err(), Ordering::Release);
+ result?;
}
Ok(())
};
- let chunk_list = self
- .datastore
- .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
+ let (chunk_list, chunk_list_error) =
+ match self
+ .datastore
+ .get_chunks_in_order(&*index, skip_chunk, check_abort)
+ {
+ Ok(chunk_list) => (chunk_list, None),
+ Err(err) => {
+ if task_aborted_or_shutdown.load(Ordering::Acquire) {
+ return Err(VerificationError::Task(err));
+ }
+ (Vec::new(), Some(err))
+ }
+ };
let reader_pool = ParallelHandler::new("read chunks", self.read_threads, {
let decoder_pool = decoder_pool.channel();
@@ -205,9 +246,13 @@ impl VerifyWorker {
)
}
});
+
+ let mut chunk_send_error: Option<Error> = None;
for (pos, _) in chunk_list {
- self.worker.check_abort()?;
- self.worker.fail_on_shutdown()?;
+ self.worker.check_abort().map_err(VerificationError::Task)?;
+ self.worker
+ .fail_on_shutdown()
+ .map_err(VerificationError::Task)?;
let info = index.chunk_info(pos).unwrap();
@@ -216,11 +261,15 @@ impl VerifyWorker {
continue; // already verified or marked corrupt
}
- reader_pool.send(info)?;
+ if let Err(err) = reader_pool.send(info) {
+ chunk_send_error = Some(err.into());
+ break;
+ }
}
- reader_pool.complete()?;
- decoder_pool.complete()?;
+ // always wait for all workers, so all errors of chunks already sent are accounted for
+ let reader_result = reader_pool.complete();
+ let decoder_result = decoder_pool.complete();
let elapsed = verify_state.start_time.elapsed().as_secs_f64();
@@ -234,13 +283,46 @@ impl VerifyWorker {
let decode_speed = decoded_bytes_mib / elapsed;
let error_count = verify_state.errors.load(Ordering::SeqCst);
+ let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst);
info!(
- " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
+ " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)"
);
- if verify_state.errors.load(Ordering::SeqCst) > 0 {
- bail!("chunks could not be verified");
+ if error_count > 0 {
+ return Err(VerificationError::Permanent(format_err!(
+ "chunks could not be verified"
+ )));
+ }
+
+ if let Some(err) = chunk_list_error {
+ return Err(VerificationError::Transient(
+ err.context("incomplete verification because of chunk list error"),
+ ));
+ }
+
+ if let Some(err) = chunk_send_error {
+ return Err(VerificationError::Transient(
+ err.context("incomplete verification because of chunk send error"),
+ ));
+ }
+
+ if let Err(err) = reader_result {
+ return Err(VerificationError::Transient(Error::from(err).context(
+ "incomplete verification because of error in reader pool",
+ )));
+ }
+
+ if let Err(err) = decoder_result {
+ return Err(VerificationError::Transient(Error::from(err).context(
+ "incomplete verification because of error in decoder pool",
+ )));
+ }
+
+ if fetch_error_count > 0 {
+ return Err(VerificationError::Transient(format_err!(
+ "chunks could not be fetched"
+ )));
}
Ok(())
@@ -298,7 +380,7 @@ impl VerifyWorker {
),
),
Err(err) => {
- verify_state.errors.fetch_add(1, Ordering::SeqCst);
+ verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
error!("can't verify chunk, load failed - {err}");
}
}
@@ -320,37 +402,63 @@ impl VerifyWorker {
}
}
- fn verify_fixed_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+ fn verify_fixed_index(
+ &self,
+ backup_dir: &BackupDir,
+ info: &FileInfo,
+ ) -> Result<(), VerificationError> {
let mut path = backup_dir.relative_path();
path.push(info.filename.as_ref());
- let index = self.datastore.open_fixed_reader(&path)?;
+ let index = self
+ .datastore
+ .open_fixed_reader(&path)
+ .map_err(VerificationError::Permanent)?;
let (csum, size) = index.compute_csum();
if size != info.size {
- bail!("wrong size ({} != {})", info.size, size);
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong size ({} != {})",
+ info.size,
+ size
+ )));
}
if csum != info.csum {
- bail!("wrong index checksum");
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong index checksum"
+ )));
}
self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
}
- fn verify_dynamic_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+ fn verify_dynamic_index(
+ &self,
+ backup_dir: &BackupDir,
+ info: &FileInfo,
+ ) -> Result<(), VerificationError> {
let mut path = backup_dir.relative_path();
path.push(info.filename.as_ref());
- let index = self.datastore.open_dynamic_reader(&path)?;
+ let index = self
+ .datastore
+ .open_dynamic_reader(&path)
+ .map_err(VerificationError::Permanent)?;
let (csum, size) = index.compute_csum();
if size != info.size {
- bail!("wrong size ({} != {})", info.size, size);
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong size ({} != {})",
+ info.size,
+ size
+ )));
}
if csum != info.csum {
- bail!("wrong index checksum");
+ return Err(VerificationError::Permanent(format_err!(
+ "wrong index checksum"
+ )));
}
self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
@@ -426,7 +534,7 @@ impl VerifyWorker {
let mut error_count = 0;
- let mut verify_result = VerifyState::Ok;
+ let mut verify_result = Some(VerifyState::Ok);
for info in manifest.files() {
info!(" check {}", info.filename);
let result = match info.filename.archive_type() {
@@ -438,30 +546,72 @@ impl VerifyWorker {
self.worker.check_abort()?;
self.worker.fail_on_shutdown()?;
- if let Err(err) = result {
- info!(
- "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
- file_name = info.filename,
- );
- error_count += 1;
- verify_result = VerifyState::Failed;
+ match result {
+ Ok(_) => (),
+ // leave the verify state untouched, the outcome is not related to the snapshot
+ Err(VerificationError::Task(err)) => return Err(err),
+ Err(VerificationError::Permanent(err)) => {
+ error_count += 1;
+ verify_result = Some(VerifyState::Failed);
+
+ info!(
+ "verify {datastore_name}:{backup_dir_name}/{file_name} failed with permanent error: {err}",
+ file_name = info.filename,
+ );
+ }
+ Err(VerificationError::Transient(err)) => {
+ error_count += 1;
+ // strip any VerifyState::Ok when updating the manifest
+ if let Some(VerifyState::Ok) = verify_result {
+ verify_result = None;
+ }
+
+ info!(
+ "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
+ file_name = info.filename,
+ );
+ }
}
}
- let verify_state = SnapshotVerifyState {
- state: verify_result,
- upid,
+ let do_update_manifest = match verify_result {
+ Some(_) => true,
+ None => match manifest.verify_state() {
+ Ok(Some(SnapshotVerifyState {
+ state: VerifyState::Ok,
+ ..
+ })) => true,
+ // keep failed state (and its original UPID) or unverified state untouched
+ Ok(Some(SnapshotVerifyState {
+ state: VerifyState::Failed,
+ ..
+ }))
+ | Ok(None) => false,
+ Err(err) => {
+ info!(
+ "failed to parse verify state for {datastore_name}:{backup_dir_name}: {err}"
+ );
+ // set to failed since verify state parsing failed
+ verify_result = Some(VerifyState::Failed);
+ true
+ }
+ },
};
- if let Err(err) = {
- let verify_state = serde_json::to_value(verify_state)?;
- backup_dir.update_manifest(&self.datastore.backend()?, |manifest| {
- manifest.unprotected["verify_state"] = verify_state;
+ let verify_state = verify_result.map(|state| SnapshotVerifyState { state, upid });
+ if do_update_manifest {
+ if let Err(err) = backup_dir.update_manifest(&self.backend, |manifest| {
+ if let Some(verify_state) = verify_state {
+ let verify_state = serde_json::to_value(verify_state)?;
+ manifest.unprotected["verify_state"] = verify_state;
+ } else if let Some(unprotected) = manifest.unprotected.as_object_mut() {
+ unprotected.remove("verify_state");
+ }
Ok(())
- })
- } {
- info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
- return Ok(false);
+ }) {
+ info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
+ return Ok(false);
+ }
}
Ok(error_count == 0)
--
2.47.3
next prev parent reply other threads:[~2026-11-16 9:55 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-16 9:54 ` Christian Ebner [this message]
2026-11-16 9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116095434.110541-12-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.