From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors
Date: Thu, 12 Nov 2026 10:39:08 +0100 [thread overview]
Message-ID: <1794475595.m77v4wiujn.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-4-c.ebner@proxmox.com>
On November 10, 2026 4:37 pm, Christian Ebner wrote:
> Fetching an object for verification might fail when sending the
> request to the S3 API or while receiving the response. Both are
> currently handled as hard errors, leading to the manifest state being
> set to verify failed. This however does not reflect the actual state,
> as a subsequent verification can work just fine.
>
> Therefore, distinguish and propagate these error kinds by enum
> variants, adapting return types and call sites accordingly.
>
> Handle the variants respectively when setting/updating the verify
> state for the manifest: if only transient errors, but no permanent
> errors occur, strip any verify okay state since then it is not
> possible to determine if the snapshot is still fine. That is best
> represented by not having a verify state, as consequence sync jobs
> with verified-only filtering will then exclude such snapshots as
> well.
>
> Independent of the error variant, always keep a previously set verify
> error state in order to avoid allowing backups to reuse this snapshot
> as reference. This covers both the new state as well as the manifest
> state prior to verification.
>
> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> src/backup/verify.rs | 190 ++++++++++++++++++++++++++++++++-----------
> 1 file changed, 143 insertions(+), 47 deletions(-)
>
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index 4d497c219..670a899ca 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -4,7 +4,7 @@ use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
> use std::sync::{Arc, Mutex};
> use std::time::Instant;
>
> -use anyhow::{Error, bail};
> +use anyhow::{Error, format_err};
> use http_body_util::BodyExt;
> use tracing::{error, info, warn};
>
> @@ -39,6 +39,7 @@ struct IndexVerifyState {
> read_bytes: AtomicU64,
> decoded_bytes: AtomicU64,
> errors: AtomicUsize,
> + fetch_errors: AtomicUsize,
> datastore: Arc<DataStore>,
> corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
> verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
> @@ -55,6 +56,7 @@ impl IndexVerifyState {
> read_bytes: AtomicU64::new(0),
> decoded_bytes: AtomicU64::new(0),
> errors: AtomicUsize::new(0),
> + fetch_errors: AtomicUsize::new(0),
> datastore: Arc::clone(datastore),
> corrupt_chunks: Arc::clone(corrupt_chunks),
> verified_chunks: Arc::clone(verified_chunks),
> @@ -63,6 +65,11 @@ impl IndexVerifyState {
> }
> }
>
> +enum VerificationError {
> + Transient(Error),
> + Permanent(Error),
> +}
> +
> impl VerifyWorker {
> /// Creates a new VerifyWorker for a given task worker and datastore.
> pub fn new(
> @@ -95,27 +102,38 @@ impl VerifyWorker {
> })
> }
>
> - fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
> - let blob = backup_dir.load_blob(info.filename.as_ref())?;
> + fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
> + let blob = backup_dir
> + .load_blob(info.filename.as_ref())
> + .map_err(VerificationError::Permanent)?;
>
> let raw_size = blob.raw_size();
> if raw_size != info.size {
> - bail!("wrong size ({} != {})", info.size, raw_size);
> + return Err(VerificationError::Permanent(format_err!(
> + "wrong size ({} != {})",
> + info.size,
> + raw_size
> + )));
> }
>
> let csum = openssl::sha::sha256(blob.raw_data());
> if csum != info.csum {
> - bail!("wrong index checksum");
> + return Err(VerificationError::Permanent(format_err!(
> + "wrong index checksum"
> + )));
> }
>
> - match blob.crypt_mode()? {
> + match blob.crypt_mode().map_err(VerificationError::Permanent)? {
> CryptMode::Encrypt => Ok(()),
> CryptMode::None => {
> // digest already verified above
> - blob.decode(None, None)?;
> + blob.decode(None, None)
> + .map_err(VerificationError::Permanent)?;
> Ok(())
> }
> - CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
> + CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!(
> + "Invalid CryptMode for blob"
> + ))),
> }
> }
>
> @@ -123,7 +141,7 @@ impl VerifyWorker {
> &self,
> index: Box<dyn IndexFile + Send>,
> crypt_mode: CryptMode,
> - ) -> Result<(), Error> {
> + ) -> Result<(), VerificationError> {
> let verify_state = Arc::new(IndexVerifyState::new(
> &self.datastore,
> &self.corrupt_chunks,
> @@ -190,7 +208,8 @@ impl VerifyWorker {
>
> let chunk_list = self
> .datastore
> - .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
> + .get_chunks_in_order(&*index, skip_chunk, check_abort)
> + .map_err(VerificationError::Transient)?;
this is a bit dangerous IMHO, it relies on get_chunks_in_order to only
return an error on worker aborts (which is the case at the moment).
would need at least a comment there that documents this behaviour and
verify's reliance on it?
also see next comment..
>
> let reader_pool = ParallelHandler::new("read chunks", self.read_threads, {
> let decoder_pool = decoder_pool.channel();
> @@ -207,8 +226,12 @@ impl VerifyWorker {
> }
> });
> for (pos, _) in chunk_list {
> - self.worker.check_abort()?;
> - self.worker.fail_on_shutdown()?;
> + self.worker
> + .check_abort()
> + .map_err(VerificationError::Transient)?;
> + self.worker
> + .fail_on_shutdown()
> + .map_err(VerificationError::Transient)?;
should these be a third category that leaves the verification state
untouched? if the verification aborts cause of these, it's not related
to the verification itself at all?
>
> let info = index.chunk_info(pos).unwrap();
>
> @@ -217,11 +240,17 @@ impl VerifyWorker {
> continue; // already verified or marked corrupt
> }
>
> - reader_pool.send(info)?;
> + reader_pool
> + .send(info)
> + .map_err(|err| VerificationError::Transient(err.into()))?;
this needs to check for permanent errors first, else it could hide a
permanent error that has occured already? or it could be treated like
worker aborts, I haven't checked the full implications here..
> }
>
> - reader_pool.complete()?;
> - decoder_pool.complete()?;
> + reader_pool
> + .complete()
> + .map_err(|err| VerificationError::Transient(err.into()))?;
> + decoder_pool
> + .complete()
> + .map_err(|err| VerificationError::Transient(err.into()))?;
same for these two
>
> let elapsed = verify_state.start_time.elapsed().as_secs_f64();
>
> @@ -235,13 +264,22 @@ impl VerifyWorker {
> let decode_speed = decoded_bytes_mib / elapsed;
>
> let error_count = verify_state.errors.load(Ordering::SeqCst);
> + let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst);
>
> info!(
> - " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
> + " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)"
> );
>
> - if verify_state.errors.load(Ordering::SeqCst) > 0 {
> - bail!("chunks could not be verified");
> + if error_count > 0 {
> + return Err(VerificationError::Permanent(format_err!(
> + "chunks could not be verified"
> + )));
> + }
like is done here?
> +
> + if fetch_error_count > 0 {
> + return Err(VerificationError::Transient(format_err!(
> + "chunks could not be fetched"
> + )));
> }
>
> Ok(())
> @@ -297,7 +335,7 @@ impl VerifyWorker {
> }
> }
> Err(err) => {
> - verify_state.errors.fetch_add(1, Ordering::SeqCst);
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> error!("can't verify chunk, load failed - {err}");
> }
> }
> @@ -311,7 +349,7 @@ impl VerifyWorker {
> ),
> ),
> Err(err) => {
> - verify_state.errors.fetch_add(1, Ordering::SeqCst);
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> error!("can't verify chunk, load failed - {err}");
> }
> }
> [..]
next prev parent reply other threads:[~2026-11-12 9:39 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler [this message]
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler
2026-11-12 10:09 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1794475595.m77v4wiujn.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.