From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors
Date: Thu, 12 Nov 2026 10:39:08 +0100 [thread overview]
Message-ID: <1794475595.m77v4wiujn.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-4-c.ebner@proxmox.com>
On November 10, 2026 4:37 pm, Christian Ebner wrote:
> Fetching an object for verification might fail when sending the
> request to the S3 API or while receiving the response. Both are
> currently handled as hard errors, leading to the manifest state being
> set to verify failed. This however does not reflect the actual state,
> as a subsequent verification can work just fine.
>
> Therefore, distinguish and propagate these error kinds by enum
> variants, adapting return types and call sites accordingly.
>
> Handle the variants respectively when setting/updating the verify
> state for the manifest: if only transient errors, but no permanent
> errors occur, strip any verify okay state since then it is not
> possible to determine if the snapshot is still fine. That is best
> represented by not having a verify state, as consequence sync jobs
> with verified-only filtering will then exclude such snapshots as
> well.
>
> Independent of the error variant, always keep a previously set verify
> error state in order to avoid allowing backups to reuse this snapshot
> as reference. This covers both the new state as well as the manifest
> state prior to verification.
>
> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> src/backup/verify.rs | 190 ++++++++++++++++++++++++++++++++-----------
> 1 file changed, 143 insertions(+), 47 deletions(-)
>
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index 4d497c219..670a899ca 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -4,7 +4,7 @@ use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
> use std::sync::{Arc, Mutex};
> use std::time::Instant;
>
> -use anyhow::{Error, bail};
> +use anyhow::{Error, format_err};
> use http_body_util::BodyExt;
> use tracing::{error, info, warn};
>
> @@ -39,6 +39,7 @@ struct IndexVerifyState {
> read_bytes: AtomicU64,
> decoded_bytes: AtomicU64,
> errors: AtomicUsize,
> + fetch_errors: AtomicUsize,
> datastore: Arc<DataStore>,
> corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
> verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
> @@ -55,6 +56,7 @@ impl IndexVerifyState {
> read_bytes: AtomicU64::new(0),
> decoded_bytes: AtomicU64::new(0),
> errors: AtomicUsize::new(0),
> + fetch_errors: AtomicUsize::new(0),
> datastore: Arc::clone(datastore),
> corrupt_chunks: Arc::clone(corrupt_chunks),
> verified_chunks: Arc::clone(verified_chunks),
> @@ -63,6 +65,11 @@ impl IndexVerifyState {
> }
> }
>
> +enum VerificationError {
> + Transient(Error),
> + Permanent(Error),
> +}
> +
> impl VerifyWorker {
> /// Creates a new VerifyWorker for a given task worker and datastore.
> pub fn new(
> @@ -95,27 +102,38 @@ impl VerifyWorker {
> })
> }
>
> - fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
> - let blob = backup_dir.load_blob(info.filename.as_ref())?;
> + fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
> + let blob = backup_dir
> + .load_blob(info.filename.as_ref())
> + .map_err(VerificationError::Permanent)?;
>
> let raw_size = blob.raw_size();
> if raw_size != info.size {
> - bail!("wrong size ({} != {})", info.size, raw_size);
> + return Err(VerificationError::Permanent(format_err!(
> + "wrong size ({} != {})",
> + info.size,
> + raw_size
> + )));
> }
>
> let csum = openssl::sha::sha256(blob.raw_data());
> if csum != info.csum {
> - bail!("wrong index checksum");
> + return Err(VerificationError::Permanent(format_err!(
> + "wrong index checksum"
> + )));
> }
>
> - match blob.crypt_mode()? {
> + match blob.crypt_mode().map_err(VerificationError::Permanent)? {
> CryptMode::Encrypt => Ok(()),
> CryptMode::None => {
> // digest already verified above
> - blob.decode(None, None)?;
> + blob.decode(None, None)
> + .map_err(VerificationError::Permanent)?;
> Ok(())
> }
> - CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
> + CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!(
> + "Invalid CryptMode for blob"
> + ))),
> }
> }
>
> @@ -123,7 +141,7 @@ impl VerifyWorker {
> &self,
> index: Box<dyn IndexFile + Send>,
> crypt_mode: CryptMode,
> - ) -> Result<(), Error> {
> + ) -> Result<(), VerificationError> {
> let verify_state = Arc::new(IndexVerifyState::new(
> &self.datastore,
> &self.corrupt_chunks,
> @@ -190,7 +208,8 @@ impl VerifyWorker {
>
> let chunk_list = self
> .datastore
> - .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
> + .get_chunks_in_order(&*index, skip_chunk, check_abort)
> + .map_err(VerificationError::Transient)?;
this is a bit dangerous IMHO, it relies on get_chunks_in_order to only
return an error on worker aborts (which is the case at the moment).
would need at least a comment there that documents this behaviour and
verify's reliance on it?
also see next comment..
>
> let reader_pool = ParallelHandler::new("read chunks", self.read_threads, {
> let decoder_pool = decoder_pool.channel();
> @@ -207,8 +226,12 @@ impl VerifyWorker {
> }
> });
> for (pos, _) in chunk_list {
> - self.worker.check_abort()?;
> - self.worker.fail_on_shutdown()?;
> + self.worker
> + .check_abort()
> + .map_err(VerificationError::Transient)?;
> + self.worker
> + .fail_on_shutdown()
> + .map_err(VerificationError::Transient)?;
should these be a third category that leaves the verification state
untouched? if the verification aborts cause of these, it's not related
to the verification itself at all?
>
> let info = index.chunk_info(pos).unwrap();
>
> @@ -217,11 +240,17 @@ impl VerifyWorker {
> continue; // already verified or marked corrupt
> }
>
> - reader_pool.send(info)?;
> + reader_pool
> + .send(info)
> + .map_err(|err| VerificationError::Transient(err.into()))?;
this needs to check for permanent errors first, else it could hide a
permanent error that has occured already? or it could be treated like
worker aborts, I haven't checked the full implications here..
> }
>
> - reader_pool.complete()?;
> - decoder_pool.complete()?;
> + reader_pool
> + .complete()
> + .map_err(|err| VerificationError::Transient(err.into()))?;
> + decoder_pool
> + .complete()
> + .map_err(|err| VerificationError::Transient(err.into()))?;
same for these two
>
> let elapsed = verify_state.start_time.elapsed().as_secs_f64();
>
> @@ -235,13 +264,22 @@ impl VerifyWorker {
> let decode_speed = decoded_bytes_mib / elapsed;
>
> let error_count = verify_state.errors.load(Ordering::SeqCst);
> + let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst);
>
> info!(
> - " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
> + " verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)"
> );
>
> - if verify_state.errors.load(Ordering::SeqCst) > 0 {
> - bail!("chunks could not be verified");
> + if error_count > 0 {
> + return Err(VerificationError::Permanent(format_err!(
> + "chunks could not be verified"
> + )));
> + }
like is done here?
> +
> + if fetch_error_count > 0 {
> + return Err(VerificationError::Transient(format_err!(
> + "chunks could not be fetched"
> + )));
> }
>
> Ok(())
> @@ -297,7 +335,7 @@ impl VerifyWorker {
> }
> }
> Err(err) => {
> - verify_state.errors.fetch_add(1, Ordering::SeqCst);
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> error!("can't verify chunk, load failed - {err}");
> }
> }
> @@ -311,7 +349,7 @@ impl VerifyWorker {
> ),
> ),
> Err(err) => {
> - verify_state.errors.fetch_add(1, Ordering::SeqCst);
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> error!("can't verify chunk, load failed - {err}");
> }
> }
> [..]
next prev parent reply other threads:[~2026-11-12 9:39 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler [this message]
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler
2026-11-12 10:09 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1794475595.m77v4wiujn.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox