public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>,
	pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors
Date: Thu, 12 Nov 2026 11:09:10 +0100	[thread overview]
Message-ID: <c4886f17-e1b0-4dde-aadf-d5557ded508f@proxmox.com> (raw)
In-Reply-To: <1794476595.f1ixh0zxb1.astroid@yuna.none>

On 11/12/26 10:48 AM, Fabian Grünbichler wrote:
> On November 10, 2026 4:37 pm, Christian Ebner wrote:
>> If collecting the response body for the S3 object fetching requests
>> fail, retry with exponential backoff time and only consider this
>> as transient fetch error on retry exhaustion. Sending the requests
>> is already retried by the s3-client, so not to be repeated.
>>
>> This avoids that an otherwise long running verification which already
>> downloaded a lot of chunks needs to be redone due to transient
>> connection issues.
> 
> not super happy about backoff handling in two places now.. could we move
> this into proxmox_s3 by adding a `get_object_content` helper there?
> 
> except for the "does S3 work" checks, we always immediately collect the
> object response anyway on the PBS side, and I guess if we want to retry
> collecting when verifying, we also want when restoring/.. ?

Yeah, agreed: best to move and integrate into proxmox-s3-client crate 
and adapt all object fetching related call sites to use that as well.

>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>>   src/backup/verify.rs | 102 ++++++++++++++++++++++++++-----------------
>>   1 file changed, 63 insertions(+), 39 deletions(-)
>>
>> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
>> index 670a899ca..edfb52a5d 100644
>> --- a/src/backup/verify.rs
>> +++ b/src/backup/verify.rs
>> @@ -2,7 +2,7 @@ use pbs_config::BackupLockGuard;
>>   use std::collections::HashSet;
>>   use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
>>   use std::sync::{Arc, Mutex};
>> -use std::time::Instant;
>> +use std::time::{Duration, Instant};
>>   
>>   use anyhow::{Error, format_err};
>>   use http_body_util::BodyExt;
>> @@ -309,48 +309,72 @@ impl VerifyWorker {
>>               },
>>               DatastoreBackend::S3(s3_client) => {
>>                   let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
>> -                match proxmox_async::runtime::block_on(s3_client.get_object(object_key)) {
>> -                    Ok(Some(response)) => {
>> -                        match proxmox_async::runtime::block_on(response.content.collect()) {
>> -                            Ok(raw_chunk) => {
>> -                                match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
>> -                                    Ok(chunk) => {
>> -                                        let size = info.size();
>> -                                        verify_state
>> -                                            .read_bytes
>> -                                            .fetch_add(chunk.raw_size(), Ordering::SeqCst);
>> -                                        decoder_pool.send((chunk, info.digest, size))?;
>> -                                        verify_state
>> -                                            .decoded_bytes
>> -                                            .fetch_add(size, Ordering::SeqCst);
>> -                                    }
>> -                                    Err(err) => Self::add_corrupt_chunk(
>> -                                        verify_state,
>> -                                        info.digest,
>> -                                        &format!(
>> -                                            "can't verify chunk with digest {} - {err}",
>> -                                            hex::encode(info.digest)
>> +
>> +                for retry in 0..3 {
>> +                    match proxmox_async::runtime::block_on(s3_client.get_object(object_key.clone()))
>> +                    {
>> +                        Ok(Some(response)) => {
>> +                            match proxmox_async::runtime::block_on(response.content.collect()) {
>> +                                Ok(raw_chunk) => {
>> +                                    match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
>> +                                        Ok(chunk) => {
>> +                                            let size = info.size();
>> +                                            verify_state
>> +                                                .read_bytes
>> +                                                .fetch_add(chunk.raw_size(), Ordering::SeqCst);
>> +                                            decoder_pool.send((chunk, info.digest, size))?;
>> +                                            verify_state
>> +                                                .decoded_bytes
>> +                                                .fetch_add(size, Ordering::SeqCst);
>> +                                        }
>> +                                        Err(err) => Self::add_corrupt_chunk(
>> +                                            verify_state,
>> +                                            info.digest,
>> +                                            &format!(
>> +                                                "can't verify chunk with digest {} - {err}",
>> +                                                hex::encode(info.digest)
>> +                                            ),
>>                                           ),
>> -                                    ),
>> +                                    }
>> +                                    break;
>> +                                }
>> +                                Err(err) => {
>> +                                    if retry < 2 {
>> +                                        warn!(
>> +                                            "retry reading chunk with digest {}, load failed - {err}",
>> +                                            hex::encode(info.digest),
>> +                                        );
>> +                                        let backoff = Duration::from_secs(3_u64.pow(retry));
>> +                                        std::thread::sleep(backoff);
>> +                                    } else {
>> +                                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> +                                        error!(
>> +                                            "can't verify chunk with digest {}, load failed - {err}",
>> +                                            hex::encode(info.digest),
>> +                                        );
>> +                                    }
>>                                   }
>> -                            }
>> -                            Err(err) => {
>> -                                verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> -                                error!("can't verify chunk, load failed - {err}");
>>                               }
>>                           }
>> -                    }
>> -                    Ok(None) => Self::add_corrupt_chunk(
>> -                        verify_state,
>> -                        info.digest,
>> -                        &format!(
>> -                            "can't verify missing chunk with digest {}",
>> -                            hex::encode(info.digest)
>> -                        ),
>> -                    ),
>> -                    Err(err) => {
>> -                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> -                        error!("can't verify chunk, load failed - {err}");
>> +                        Ok(None) => {
>> +                            Self::add_corrupt_chunk(
>> +                                verify_state,
>> +                                info.digest,
>> +                                &format!(
>> +                                    "can't verify missing chunk with digest {}",
>> +                                    hex::encode(info.digest)
>> +                                ),
>> +                            );
>> +                            break;
>> +                        }
>> +                        Err(err) => {
>> +                            verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> +                            error!(
>> +                                "can't verify chunk with digest {}, load failed - {err}",
>> +                                hex::encode(info.digest),
>> +                            );
>> +                            break;
>> +                        }
>>                       }
>>                   }
>>               }
>> -- 
>> 2.47.3
>>
>>
>>
>>
>>
>>





  reply	other threads:[~2026-11-12 10:09 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11  0:29   ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12  9:39   ` Fabian Grünbichler
2026-11-12 10:06     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12  9:48   ` Fabian Grünbichler
2026-11-12 10:09     ` Christian Ebner [this message]
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12  9:41   ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c4886f17-e1b0-4dde-aadf-d5557ded508f@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=f.gruenbichler@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal