From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors
Date: Thu, 12 Nov 2026 10:48:54 +0100 [thread overview]
Message-ID: <1794476595.f1ixh0zxb1.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-5-c.ebner@proxmox.com>
On November 10, 2026 4:37 pm, Christian Ebner wrote:
> If collecting the response body for the S3 object fetching requests
> fail, retry with exponential backoff time and only consider this
> as transient fetch error on retry exhaustion. Sending the requests
> is already retried by the s3-client, so not to be repeated.
>
> This avoids that an otherwise long running verification which already
> downloaded a lot of chunks needs to be redone due to transient
> connection issues.
not super happy about backoff handling in two places now.. could we move
this into proxmox_s3 by adding a `get_object_content` helper there?
except for the "does S3 work" checks, we always immediately collect the
object response anyway on the PBS side, and I guess if we want to retry
collecting when verifying, we also want when restoring/.. ?
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> src/backup/verify.rs | 102 ++++++++++++++++++++++++++-----------------
> 1 file changed, 63 insertions(+), 39 deletions(-)
>
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index 670a899ca..edfb52a5d 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -2,7 +2,7 @@ use pbs_config::BackupLockGuard;
> use std::collections::HashSet;
> use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
> use std::sync::{Arc, Mutex};
> -use std::time::Instant;
> +use std::time::{Duration, Instant};
>
> use anyhow::{Error, format_err};
> use http_body_util::BodyExt;
> @@ -309,48 +309,72 @@ impl VerifyWorker {
> },
> DatastoreBackend::S3(s3_client) => {
> let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
> - match proxmox_async::runtime::block_on(s3_client.get_object(object_key)) {
> - Ok(Some(response)) => {
> - match proxmox_async::runtime::block_on(response.content.collect()) {
> - Ok(raw_chunk) => {
> - match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
> - Ok(chunk) => {
> - let size = info.size();
> - verify_state
> - .read_bytes
> - .fetch_add(chunk.raw_size(), Ordering::SeqCst);
> - decoder_pool.send((chunk, info.digest, size))?;
> - verify_state
> - .decoded_bytes
> - .fetch_add(size, Ordering::SeqCst);
> - }
> - Err(err) => Self::add_corrupt_chunk(
> - verify_state,
> - info.digest,
> - &format!(
> - "can't verify chunk with digest {} - {err}",
> - hex::encode(info.digest)
> +
> + for retry in 0..3 {
> + match proxmox_async::runtime::block_on(s3_client.get_object(object_key.clone()))
> + {
> + Ok(Some(response)) => {
> + match proxmox_async::runtime::block_on(response.content.collect()) {
> + Ok(raw_chunk) => {
> + match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
> + Ok(chunk) => {
> + let size = info.size();
> + verify_state
> + .read_bytes
> + .fetch_add(chunk.raw_size(), Ordering::SeqCst);
> + decoder_pool.send((chunk, info.digest, size))?;
> + verify_state
> + .decoded_bytes
> + .fetch_add(size, Ordering::SeqCst);
> + }
> + Err(err) => Self::add_corrupt_chunk(
> + verify_state,
> + info.digest,
> + &format!(
> + "can't verify chunk with digest {} - {err}",
> + hex::encode(info.digest)
> + ),
> ),
> - ),
> + }
> + break;
> + }
> + Err(err) => {
> + if retry < 2 {
> + warn!(
> + "retry reading chunk with digest {}, load failed - {err}",
> + hex::encode(info.digest),
> + );
> + let backoff = Duration::from_secs(3_u64.pow(retry));
> + std::thread::sleep(backoff);
> + } else {
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> + error!(
> + "can't verify chunk with digest {}, load failed - {err}",
> + hex::encode(info.digest),
> + );
> + }
> }
> - }
> - Err(err) => {
> - verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> - error!("can't verify chunk, load failed - {err}");
> }
> }
> - }
> - Ok(None) => Self::add_corrupt_chunk(
> - verify_state,
> - info.digest,
> - &format!(
> - "can't verify missing chunk with digest {}",
> - hex::encode(info.digest)
> - ),
> - ),
> - Err(err) => {
> - verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> - error!("can't verify chunk, load failed - {err}");
> + Ok(None) => {
> + Self::add_corrupt_chunk(
> + verify_state,
> + info.digest,
> + &format!(
> + "can't verify missing chunk with digest {}",
> + hex::encode(info.digest)
> + ),
> + );
> + break;
> + }
> + Err(err) => {
> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
> + error!(
> + "can't verify chunk with digest {}, load failed - {err}",
> + hex::encode(info.digest),
> + );
> + break;
> + }
> }
> }
> }
> --
> 2.47.3
>
>
>
>
>
>
next prev parent reply other threads:[~2026-11-12 9:49 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler [this message]
2026-11-12 10:09 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1794476595.f1ixh0zxb1.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox