From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 8AFC61FF0B7 for ; Fri, 13 Nov 2026 15:22:31 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E732C216D0; Fri, 13 Nov 2026 15:22:25 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup v4 10/11] fix #7606: verify: s3: handle transient chunk load errors Date: Fri, 13 Nov 2026 15:21:47 +0100 Message-ID: <20261113142148.569471-11-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261113142148.569471-1-c.ebner@proxmox.com> References: <20261113142148.569471-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794579714144 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.390 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 4PUIHPMXKT26U67DKRTIJMYQKDOXRYSN X-Message-ID-Hash: 4PUIHPMXKT26U67DKRTIJMYQKDOXRYSN X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Fetching an object for verification might fail when sending the request to the S3 API or while receiving the response. Both are currently handled as hard errors, leading to the manifest state being set to verify failed. This however does not reflect the actual state, as a subsequent verification can work just fine. Therefore, distinguish between permanent, transient and verification task related errors (abort/shutdown). If only transient errors, but no permanent errors occur, strip any verify okay state since then it is not possible to determine if the snapshot is still fine. That is best represented by not having a verify state, as consequence sync jobs with verified-only filtering will then exclude such snapshots as well. Independent of the error variant, always keep a previously set verify error state in order to avoid allowing backups to reuse this snapshot as reference. This covers both the new state as well as the manifest state prior to verification. On task related errors, the verify state is not altered at all. Errors of the chunk listing, parallel chunk readers or decoders also leave the verification incomplete, but are only considered after all chunks already sent have been checked, so they never hide permanent errors. Avoid manifest updates if not needed, since these cause additional requests on S3 backed datastores and might fail as well. Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606 Signed-off-by: Christian Ebner --- src/backup/verify.rs | 252 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 201 insertions(+), 51 deletions(-) diff --git a/src/backup/verify.rs b/src/backup/verify.rs index d2325ce9a..624c9a9aa 100644 --- a/src/backup/verify.rs +++ b/src/backup/verify.rs @@ -1,10 +1,10 @@ use pbs_config::BackupLockGuard; use std::collections::HashSet; -use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Instant; -use anyhow::{Error, bail}; +use anyhow::{Error, format_err}; use tracing::{error, info, warn}; use proxmox_parallel_handler::{ParallelHandler, SendHandle}; @@ -38,6 +38,7 @@ struct IndexVerifyState { read_bytes: AtomicU64, decoded_bytes: AtomicU64, errors: AtomicUsize, + fetch_errors: AtomicUsize, datastore: Arc, corrupt_chunks: Arc>>, verified_chunks: Arc>>, @@ -54,6 +55,7 @@ impl IndexVerifyState { read_bytes: AtomicU64::new(0), decoded_bytes: AtomicU64::new(0), errors: AtomicUsize::new(0), + fetch_errors: AtomicUsize::new(0), datastore: Arc::clone(datastore), corrupt_chunks: Arc::clone(corrupt_chunks), verified_chunks: Arc::clone(verified_chunks), @@ -62,6 +64,17 @@ impl IndexVerifyState { } } +/// Outcome of a failed archive verification, deciding how the snapshot's verify state is updated. +enum VerificationError { + /// The archive is known to be bad, for example because of corrupt or missing chunks. + Permanent(Error), + /// The archive could not be fully verified, for example because chunks could not be fetched + /// from the S3 backend. Its state is unknown. + Transient(Error), + /// The task was aborted or the daemon is shutting down, unrelated to the archive itself. + Task(Error), +} + impl VerifyWorker { /// Creates a new VerifyWorker for a given task worker and datastore. pub fn new( @@ -94,27 +107,38 @@ impl VerifyWorker { }) } - fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> { - let blob = backup_dir.load_blob(info.filename.as_ref())?; + fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> { + let blob = backup_dir + .load_blob(info.filename.as_ref()) + .map_err(VerificationError::Permanent)?; let raw_size = blob.raw_size(); if raw_size != info.size { - bail!("wrong size ({} != {})", info.size, raw_size); + return Err(VerificationError::Permanent(format_err!( + "wrong size ({} != {})", + info.size, + raw_size + ))); } let csum = openssl::sha::sha256(blob.raw_data()); if csum != info.csum { - bail!("wrong index checksum"); + return Err(VerificationError::Permanent(format_err!( + "wrong index checksum" + ))); } - match blob.crypt_mode()? { + match blob.crypt_mode().map_err(VerificationError::Permanent)? { CryptMode::Encrypt => Ok(()), CryptMode::None => { // digest already verified above - blob.decode(None, None)?; + blob.decode(None, None) + .map_err(VerificationError::Permanent)?; Ok(()) } - CryptMode::SignOnly => bail!("Invalid CryptMode for blob"), + CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!( + "Invalid CryptMode for blob" + ))), } } @@ -122,7 +146,7 @@ impl VerifyWorker { &self, index: Box, crypt_mode: CryptMode, - ) -> Result<(), Error> { + ) -> Result<(), VerificationError> { let verify_state = Arc::new(IndexVerifyState::new( &self.datastore, &self.corrupt_chunks, @@ -179,17 +203,34 @@ impl VerifyWorker { } }; + // Track if the task was aborted or shutdown, so only that is treated as task error and any + // other error of the chunk ordering leaves the archive unverified. + let task_aborted_or_shutdown = AtomicBool::new(false); let check_abort = |pos: usize| -> Result<(), Error> { if pos & 1023 == 0 { - self.worker.check_abort()?; - self.worker.fail_on_shutdown()?; + let result = self + .worker + .check_abort() + .and_then(|_| self.worker.fail_on_shutdown()); + task_aborted_or_shutdown.store(result.is_err(), Ordering::Release); + result?; } Ok(()) }; - let chunk_list = self - .datastore - .get_chunks_in_order(&*index, skip_chunk, check_abort)?; + let (chunk_list, chunk_list_error) = + match self + .datastore + .get_chunks_in_order(&*index, skip_chunk, check_abort) + { + Ok(chunk_list) => (chunk_list, None), + Err(err) => { + if task_aborted_or_shutdown.load(Ordering::Acquire) { + return Err(VerificationError::Task(err)); + } + (Vec::new(), Some(err)) + } + }; let reader_pool = ParallelHandler::new("read chunks", self.read_threads, { let decoder_pool = decoder_pool.channel(); @@ -205,9 +246,13 @@ impl VerifyWorker { ) } }); + + let mut chunk_send_error: Option = None; for (pos, _) in chunk_list { - self.worker.check_abort()?; - self.worker.fail_on_shutdown()?; + self.worker.check_abort().map_err(VerificationError::Task)?; + self.worker + .fail_on_shutdown() + .map_err(VerificationError::Task)?; let info = index.chunk_info(pos).unwrap(); @@ -216,11 +261,15 @@ impl VerifyWorker { continue; // already verified or marked corrupt } - reader_pool.send(info)?; + if let Err(err) = reader_pool.send(info) { + chunk_send_error = Some(err.into()); + break; + } } - reader_pool.complete()?; - decoder_pool.complete()?; + // always wait for all workers, so all errors of chunks already sent are accounted for + let reader_result = reader_pool.complete(); + let decoder_result = decoder_pool.complete(); let elapsed = verify_state.start_time.elapsed().as_secs_f64(); @@ -234,13 +283,46 @@ impl VerifyWorker { let decode_speed = decoded_bytes_mib / elapsed; let error_count = verify_state.errors.load(Ordering::SeqCst); + let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst); info!( - " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)" + " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)" ); - if verify_state.errors.load(Ordering::SeqCst) > 0 { - bail!("chunks could not be verified"); + if error_count > 0 { + return Err(VerificationError::Permanent(format_err!( + "chunks could not be verified" + ))); + } + + if let Some(err) = chunk_list_error { + return Err(VerificationError::Transient( + err.context("incomplete verification because of chunk list error"), + )); + } + + if let Some(err) = chunk_send_error { + return Err(VerificationError::Transient( + err.context("incomplete verification because of chunk send error"), + )); + } + + if let Err(err) = reader_result { + return Err(VerificationError::Transient(Error::from(err).context( + "incomplete verification because of error in reader pool", + ))); + } + + if let Err(err) = decoder_result { + return Err(VerificationError::Transient(Error::from(err).context( + "incomplete verification because of error in decoder pool", + ))); + } + + if fetch_error_count > 0 { + return Err(VerificationError::Transient(format_err!( + "chunks could not be fetched" + ))); } Ok(()) @@ -298,7 +380,7 @@ impl VerifyWorker { ), ), Err(err) => { - verify_state.errors.fetch_add(1, Ordering::SeqCst); + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst); error!("can't verify chunk, load failed - {err}"); } } @@ -320,37 +402,63 @@ impl VerifyWorker { } } - fn verify_fixed_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> { + fn verify_fixed_index( + &self, + backup_dir: &BackupDir, + info: &FileInfo, + ) -> Result<(), VerificationError> { let mut path = backup_dir.relative_path(); path.push(info.filename.as_ref()); - let index = self.datastore.open_fixed_reader(&path)?; + let index = self + .datastore + .open_fixed_reader(&path) + .map_err(VerificationError::Permanent)?; let (csum, size) = index.compute_csum(); if size != info.size { - bail!("wrong size ({} != {})", info.size, size); + return Err(VerificationError::Permanent(format_err!( + "wrong size ({} != {})", + info.size, + size + ))); } if csum != info.csum { - bail!("wrong index checksum"); + return Err(VerificationError::Permanent(format_err!( + "wrong index checksum" + ))); } self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode()) } - fn verify_dynamic_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> { + fn verify_dynamic_index( + &self, + backup_dir: &BackupDir, + info: &FileInfo, + ) -> Result<(), VerificationError> { let mut path = backup_dir.relative_path(); path.push(info.filename.as_ref()); - let index = self.datastore.open_dynamic_reader(&path)?; + let index = self + .datastore + .open_dynamic_reader(&path) + .map_err(VerificationError::Permanent)?; let (csum, size) = index.compute_csum(); if size != info.size { - bail!("wrong size ({} != {})", info.size, size); + return Err(VerificationError::Permanent(format_err!( + "wrong size ({} != {})", + info.size, + size + ))); } if csum != info.csum { - bail!("wrong index checksum"); + return Err(VerificationError::Permanent(format_err!( + "wrong index checksum" + ))); } self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode()) @@ -426,7 +534,7 @@ impl VerifyWorker { let mut error_count = 0; - let mut verify_result = VerifyState::Ok; + let mut verify_result = Some(VerifyState::Ok); for info in manifest.files() { info!(" check {}", info.filename); let result = match info.filename.archive_type() { @@ -438,30 +546,72 @@ impl VerifyWorker { self.worker.check_abort()?; self.worker.fail_on_shutdown()?; - if let Err(err) = result { - info!( - "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}", - file_name = info.filename, - ); - error_count += 1; - verify_result = VerifyState::Failed; + match result { + Ok(_) => (), + // leave the verify state untouched, the outcome is not related to the snapshot + Err(VerificationError::Task(err)) => return Err(err), + Err(VerificationError::Permanent(err)) => { + error_count += 1; + verify_result = Some(VerifyState::Failed); + + info!( + "verify {datastore_name}:{backup_dir_name}/{file_name} failed with permanent error: {err}", + file_name = info.filename, + ); + } + Err(VerificationError::Transient(err)) => { + error_count += 1; + // strip any VerifyState::Ok when updating the manifest + if let Some(VerifyState::Ok) = verify_result { + verify_result = None; + } + + info!( + "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}", + file_name = info.filename, + ); + } } } - let verify_state = SnapshotVerifyState { - state: verify_result, - upid, + let do_update_manifest = match verify_result { + Some(_) => true, + None => match manifest.verify_state() { + Ok(Some(SnapshotVerifyState { + state: VerifyState::Ok, + .. + })) => true, + // keep failed state (and its original UPID) or unverified state untouched + Ok(Some(SnapshotVerifyState { + state: VerifyState::Failed, + .. + })) + | Ok(None) => false, + Err(err) => { + info!( + "failed to parse verify state for {datastore_name}:{backup_dir_name}: {err}" + ); + // set to failed since verify state parsing failed + verify_result = Some(VerifyState::Failed); + true + } + }, }; - if let Err(err) = { - let verify_state = serde_json::to_value(verify_state)?; - backup_dir.update_manifest(&self.datastore.backend()?, |manifest| { - manifest.unprotected["verify_state"] = verify_state; + let verify_state = verify_result.map(|state| SnapshotVerifyState { state, upid }); + if do_update_manifest { + if let Err(err) = backup_dir.update_manifest(&self.backend, |manifest| { + if let Some(verify_state) = verify_state { + let verify_state = serde_json::to_value(verify_state)?; + manifest.unprotected["verify_state"] = verify_state; + } else if let Some(unprotected) = manifest.unprotected.as_object_mut() { + unprotected.remove("verify_state"); + } Ok(()) - }) - } { - info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}"); - return Ok(false); + }) { + info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}"); + return Ok(false); + } } Ok(error_count == 0) -- 2.47.3