public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>,
	pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors
Date: Thu, 12 Nov 2026 11:06:33 +0100	[thread overview]
Message-ID: <bc6e26b0-d596-44a5-afeb-67b31ef49cfa@proxmox.com> (raw)
In-Reply-To: <1794475595.m77v4wiujn.astroid@yuna.none>

On 11/12/26 10:39 AM, Fabian Grünbichler wrote:
> On November 10, 2026 4:37 pm, Christian Ebner wrote:
>> Fetching an object for verification might fail when sending the
>> request to the S3 API or while receiving the response. Both are
>> currently handled as hard errors, leading to the manifest state being
>> set to verify failed. This however does not reflect the actual state,
>> as a subsequent verification can work just fine.
>>
>> Therefore, distinguish and propagate these error kinds by enum
>> variants, adapting return types and call sites accordingly.
>>
>> Handle the variants respectively when setting/updating the verify
>> state for the manifest: if only transient errors, but no permanent
>> errors occur, strip any verify okay state since then it is not
>> possible to determine if the snapshot is still fine. That is best
>> represented by not having a verify state, as consequence sync jobs
>> with verified-only filtering will then exclude such snapshots as
>> well.
>>
>> Independent of the error variant, always keep a previously set verify
>> error state in order to avoid allowing backups to reuse this snapshot
>> as reference. This covers both the new state as well as the manifest
>> state prior to verification.
>>
>> Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>>   src/backup/verify.rs | 190 ++++++++++++++++++++++++++++++++-----------
>>   1 file changed, 143 insertions(+), 47 deletions(-)
>>
>> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
>> index 4d497c219..670a899ca 100644
>> --- a/src/backup/verify.rs
>> +++ b/src/backup/verify.rs
>> @@ -4,7 +4,7 @@ use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
>>   use std::sync::{Arc, Mutex};
>>   use std::time::Instant;
>>   
>> -use anyhow::{Error, bail};
>> +use anyhow::{Error, format_err};
>>   use http_body_util::BodyExt;
>>   use tracing::{error, info, warn};
>>   
>> @@ -39,6 +39,7 @@ struct IndexVerifyState {
>>       read_bytes: AtomicU64,
>>       decoded_bytes: AtomicU64,
>>       errors: AtomicUsize,
>> +    fetch_errors: AtomicUsize,
>>       datastore: Arc<DataStore>,
>>       corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
>>       verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
>> @@ -55,6 +56,7 @@ impl IndexVerifyState {
>>               read_bytes: AtomicU64::new(0),
>>               decoded_bytes: AtomicU64::new(0),
>>               errors: AtomicUsize::new(0),
>> +            fetch_errors: AtomicUsize::new(0),
>>               datastore: Arc::clone(datastore),
>>               corrupt_chunks: Arc::clone(corrupt_chunks),
>>               verified_chunks: Arc::clone(verified_chunks),
>> @@ -63,6 +65,11 @@ impl IndexVerifyState {
>>       }
>>   }
>>   
>> +enum VerificationError {
>> +    Transient(Error),
>> +    Permanent(Error),
>> +}
>> +
>>   impl VerifyWorker {
>>       /// Creates a new VerifyWorker for a given task worker and datastore.
>>       pub fn new(
>> @@ -95,27 +102,38 @@ impl VerifyWorker {
>>           })
>>       }
>>   
>> -    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
>> -        let blob = backup_dir.load_blob(info.filename.as_ref())?;
>> +    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
>> +        let blob = backup_dir
>> +            .load_blob(info.filename.as_ref())
>> +            .map_err(VerificationError::Permanent)?;
>>   
>>           let raw_size = blob.raw_size();
>>           if raw_size != info.size {
>> -            bail!("wrong size ({} != {})", info.size, raw_size);
>> +            return Err(VerificationError::Permanent(format_err!(
>> +                "wrong size ({} != {})",
>> +                info.size,
>> +                raw_size
>> +            )));
>>           }
>>   
>>           let csum = openssl::sha::sha256(blob.raw_data());
>>           if csum != info.csum {
>> -            bail!("wrong index checksum");
>> +            return Err(VerificationError::Permanent(format_err!(
>> +                "wrong index checksum"
>> +            )));
>>           }
>>   
>> -        match blob.crypt_mode()? {
>> +        match blob.crypt_mode().map_err(VerificationError::Permanent)? {
>>               CryptMode::Encrypt => Ok(()),
>>               CryptMode::None => {
>>                   // digest already verified above
>> -                blob.decode(None, None)?;
>> +                blob.decode(None, None)
>> +                    .map_err(VerificationError::Permanent)?;
>>                   Ok(())
>>               }
>> -            CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
>> +            CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!(
>> +                "Invalid CryptMode for blob"
>> +            ))),
>>           }
>>       }
>>   
>> @@ -123,7 +141,7 @@ impl VerifyWorker {
>>           &self,
>>           index: Box<dyn IndexFile + Send>,
>>           crypt_mode: CryptMode,
>> -    ) -> Result<(), Error> {
>> +    ) -> Result<(), VerificationError> {
>>           let verify_state = Arc::new(IndexVerifyState::new(
>>               &self.datastore,
>>               &self.corrupt_chunks,
>> @@ -190,7 +208,8 @@ impl VerifyWorker {
>>   
>>           let chunk_list = self
>>               .datastore
>> -            .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
>> +            .get_chunks_in_order(&*index, skip_chunk, check_abort)
>> +            .map_err(VerificationError::Transient)?;
> 
> this is a bit dangerous IMHO, it relies on get_chunks_in_order to only
> return an error on worker aborts (which is the case at the moment).
> would need at least a comment there that documents this behaviour and
> verify's reliance on it?
> 
> also see next comment..
> 
>>   
>>           let reader_pool = ParallelHandler::new("read chunks", self.read_threads, {
>>               let decoder_pool = decoder_pool.channel();
>> @@ -207,8 +226,12 @@ impl VerifyWorker {
>>               }
>>           });
>>           for (pos, _) in chunk_list {
>> -            self.worker.check_abort()?;
>> -            self.worker.fail_on_shutdown()?;
>> +            self.worker
>> +                .check_abort()
>> +                .map_err(VerificationError::Transient)?;
>> +            self.worker
>> +                .fail_on_shutdown()
>> +                .map_err(VerificationError::Transient)?;
> 
> should these be a third category that leaves the verification state
> untouched? if the verification aborts cause of these, it's not related
> to the verification itself at all?

Yeah, makes sense to have VerificationError::Task to correctly 
distinguish these as well. Will adapt for v3.

> 
>>   
>>               let info = index.chunk_info(pos).unwrap();
>>   
>> @@ -217,11 +240,17 @@ impl VerifyWorker {
>>                   continue; // already verified or marked corrupt
>>               }
>>   
>> -            reader_pool.send(info)?;
>> +            reader_pool
>> +                .send(info)
>> +                .map_err(|err| VerificationError::Transient(err.into()))?;
> 
> this needs to check for permanent errors first, else it could hide a
> permanent error that has occured already? or it could be treated like
> worker aborts, I haven't checked the full implications here..

good catch! the send error should never hide permanent errors

> 
>>           }
>>   
>> -        reader_pool.complete()?;
>> -        decoder_pool.complete()?;
>> +        reader_pool
>> +            .complete()
>> +            .map_err(|err| VerificationError::Transient(err.into()))?;
>> +        decoder_pool
>> +            .complete()
>> +            .map_err(|err| VerificationError::Transient(err.into()))?;
> 
> same for these two
> 
>>   
>>           let elapsed = verify_state.start_time.elapsed().as_secs_f64();
>>   
>> @@ -235,13 +264,22 @@ impl VerifyWorker {
>>           let decode_speed = decoded_bytes_mib / elapsed;
>>   
>>           let error_count = verify_state.errors.load(Ordering::SeqCst);
>> +        let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst);
>>   
>>           info!(
>> -            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
>> +            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)"
>>           );
>>   
>> -        if verify_state.errors.load(Ordering::SeqCst) > 0 {
>> -            bail!("chunks could not be verified");
>> +        if error_count > 0 {
>> +            return Err(VerificationError::Permanent(format_err!(
>> +                "chunks could not be verified"
>> +            )));
>> +        }
> 
> like is done here?
> 
>> +
>> +        if fetch_error_count > 0 {
>> +            return Err(VerificationError::Transient(format_err!(
>> +                "chunks could not be fetched"
>> +            )));
>>           }
>>   
>>           Ok(())
>> @@ -297,7 +335,7 @@ impl VerifyWorker {
>>                                   }
>>                               }
>>                               Err(err) => {
>> -                                verify_state.errors.fetch_add(1, Ordering::SeqCst);
>> +                                verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>>                                   error!("can't verify chunk, load failed - {err}");
>>                               }
>>                           }
>> @@ -311,7 +349,7 @@ impl VerifyWorker {
>>                           ),
>>                       ),
>>                       Err(err) => {
>> -                        verify_state.errors.fetch_add(1, Ordering::SeqCst);
>> +                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>>                           error!("can't verify chunk, load failed - {err}");
>>                       }
>>                   }
> 
>> [..]





  reply	other threads:[~2026-11-12 10:06 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11  0:29   ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12  9:39   ` Fabian Grünbichler
2026-11-12 10:06     ` Christian Ebner [this message]
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12  9:48   ` Fabian Grünbichler
2026-11-12 10:09     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12  9:41   ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=bc6e26b0-d596-44a5-afeb-67b31ef49cfa@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=f.gruenbichler@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal