public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors
Date: Thu, 12 Nov 2026 10:48:54 +0100	[thread overview]
Message-ID: <1794476595.f1ixh0zxb1.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-5-c.ebner@proxmox.com>

On November 10, 2026 4:37 pm, Christian Ebner wrote:
> If collecting the response body for the S3 object fetching requests
> fail, retry with exponential backoff time and only consider this
> as transient fetch error on retry exhaustion. Sending the requests
> is already retried by the s3-client, so not to be repeated.
> 
> This avoids that an otherwise long running verification which already
> downloaded a lot of chunks needs to be redone due to transient
> connection issues.

not super happy about backoff handling in two places now.. could we move
this into proxmox_s3 by adding a `get_object_content` helper there?

except for the "does S3 work" checks, we always immediately collect the
object response anyway on the PBS side, and I guess if we want to retry
collecting when verifying, we also want when restoring/.. ?

> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  src/backup/verify.rs | 102 ++++++++++++++++++++++++++-----------------
>  1 file changed, 63 insertions(+), 39 deletions(-)
> 
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index 670a899ca..edfb52a5d 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -2,7 +2,7 @@ use pbs_config::BackupLockGuard;
>  use std::collections::HashSet;
>  use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
>  use std::sync::{Arc, Mutex};
> -use std::time::Instant;
> +use std::time::{Duration, Instant};
>  
>  use anyhow::{Error, format_err};
>  use http_body_util::BodyExt;
> @@ -309,48 +309,72 @@ impl VerifyWorker {
>              },
>              DatastoreBackend::S3(s3_client) => {
>                  let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
> -                match proxmox_async::runtime::block_on(s3_client.get_object(object_key)) {
> -                    Ok(Some(response)) => {
> -                        match proxmox_async::runtime::block_on(response.content.collect()) {
> -                            Ok(raw_chunk) => {
> -                                match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
> -                                    Ok(chunk) => {
> -                                        let size = info.size();
> -                                        verify_state
> -                                            .read_bytes
> -                                            .fetch_add(chunk.raw_size(), Ordering::SeqCst);
> -                                        decoder_pool.send((chunk, info.digest, size))?;
> -                                        verify_state
> -                                            .decoded_bytes
> -                                            .fetch_add(size, Ordering::SeqCst);
> -                                    }
> -                                    Err(err) => Self::add_corrupt_chunk(
> -                                        verify_state,
> -                                        info.digest,
> -                                        &format!(
> -                                            "can't verify chunk with digest {} - {err}",
> -                                            hex::encode(info.digest)
> +
> +                for retry in 0..3 {
> +                    match proxmox_async::runtime::block_on(s3_client.get_object(object_key.clone()))
> +                    {
> +                        Ok(Some(response)) => {
> +                            match proxmox_async::runtime::block_on(response.content.collect()) {
> +                                Ok(raw_chunk) => {
> +                                    match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
> +                                        Ok(chunk) => {
> +                                            let size = info.size();
> +                                            verify_state
> +                                                .read_bytes
> +                                                .fetch_add(chunk.raw_size(), Ordering::SeqCst);
> +                                            decoder_pool.send((chunk, info.digest, size))?;
> +                                            verify_state
> +                                                .decoded_bytes
> +                                                .fetch_add(size, Ordering::SeqCst);
> +                                        }
> +                                        Err(err) => Self::add_corrupt_chunk(
> +                                            verify_state,
> +                                            info.digest,
> +                                            &format!(
> +                                                "can't verify chunk with digest {} - {err}",
> +                                                hex::encode(info.digest)
> +                                            ),
>                                          ),
> -                                    ),
> +                                    }
> +                                    break;
> +                                }
> +                                Err(err) => {
> +                                    if retry < 2 {
> +                                        warn!(
> +                                            "retry reading chunk with digest {}, load failed - {err}",
> +                                            hex::encode(info.digest),
> +                                        );
> +                                        let backoff = Duration::from_secs(3_u64.pow(retry));
> +                                        std::thread::sleep(backoff);
> +                                    } else {
> +                                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> +                                        error!(
> +                                            "can't verify chunk with digest {}, load failed - {err}",
> +                                            hex::encode(info.digest),
> +                                        );
> +                                    }
>                                  }
> -                            }
> -                            Err(err) => {
> -                                verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> -                                error!("can't verify chunk, load failed - {err}");
>                              }
>                          }
> -                    }
> -                    Ok(None) => Self::add_corrupt_chunk(
> -                        verify_state,
> -                        info.digest,
> -                        &format!(
> -                            "can't verify missing chunk with digest {}",
> -                            hex::encode(info.digest)
> -                        ),
> -                    ),
> -                    Err(err) => {
> -                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> -                        error!("can't verify chunk, load failed - {err}");
> +                        Ok(None) => {
> +                            Self::add_corrupt_chunk(
> +                                verify_state,
> +                                info.digest,
> +                                &format!(
> +                                    "can't verify missing chunk with digest {}",
> +                                    hex::encode(info.digest)
> +                                ),
> +                            );
> +                            break;
> +                        }
> +                        Err(err) => {
> +                            verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> +                            error!(
> +                                "can't verify chunk with digest {}, load failed - {err}",
> +                                hex::encode(info.digest),
> +                            );
> +                            break;
> +                        }
>                      }
>                  }
>              }
> -- 
> 2.47.3
> 
> 
> 
> 
> 
> 




  reply	other threads:[~2026-11-12  9:49 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11  0:29   ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12  9:39   ` Fabian Grünbichler
2026-11-12 10:06     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12  9:48   ` Fabian Grünbichler [this message]
2026-11-12 10:09     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12  9:41   ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1794476595.f1ixh0zxb1.astroid@yuna.none \
    --to=f.gruenbichler@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal