From: Christian Ebner <c.ebner@proxmox.com>
To: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>,
pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors
Date: Thu, 12 Nov 2026 11:09:10 +0100 [thread overview]
Message-ID: <c4886f17-e1b0-4dde-aadf-d5557ded508f@proxmox.com> (raw)
In-Reply-To: <1794476595.f1ixh0zxb1.astroid@yuna.none>
On 11/12/26 10:48 AM, Fabian Grünbichler wrote:
> On November 10, 2026 4:37 pm, Christian Ebner wrote:
>> If collecting the response body for the S3 object fetching requests
>> fail, retry with exponential backoff time and only consider this
>> as transient fetch error on retry exhaustion. Sending the requests
>> is already retried by the s3-client, so not to be repeated.
>>
>> This avoids that an otherwise long running verification which already
>> downloaded a lot of chunks needs to be redone due to transient
>> connection issues.
>
> not super happy about backoff handling in two places now.. could we move
> this into proxmox_s3 by adding a `get_object_content` helper there?
>
> except for the "does S3 work" checks, we always immediately collect the
> object response anyway on the PBS side, and I guess if we want to retry
> collecting when verifying, we also want when restoring/.. ?
Yeah, agreed: best to move and integrate into proxmox-s3-client crate
and adapt all object fetching related call sites to use that as well.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>> src/backup/verify.rs | 102 ++++++++++++++++++++++++++-----------------
>> 1 file changed, 63 insertions(+), 39 deletions(-)
>>
>> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
>> index 670a899ca..edfb52a5d 100644
>> --- a/src/backup/verify.rs
>> +++ b/src/backup/verify.rs
>> @@ -2,7 +2,7 @@ use pbs_config::BackupLockGuard;
>> use std::collections::HashSet;
>> use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
>> use std::sync::{Arc, Mutex};
>> -use std::time::Instant;
>> +use std::time::{Duration, Instant};
>>
>> use anyhow::{Error, format_err};
>> use http_body_util::BodyExt;
>> @@ -309,48 +309,72 @@ impl VerifyWorker {
>> },
>> DatastoreBackend::S3(s3_client) => {
>> let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
>> - match proxmox_async::runtime::block_on(s3_client.get_object(object_key)) {
>> - Ok(Some(response)) => {
>> - match proxmox_async::runtime::block_on(response.content.collect()) {
>> - Ok(raw_chunk) => {
>> - match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
>> - Ok(chunk) => {
>> - let size = info.size();
>> - verify_state
>> - .read_bytes
>> - .fetch_add(chunk.raw_size(), Ordering::SeqCst);
>> - decoder_pool.send((chunk, info.digest, size))?;
>> - verify_state
>> - .decoded_bytes
>> - .fetch_add(size, Ordering::SeqCst);
>> - }
>> - Err(err) => Self::add_corrupt_chunk(
>> - verify_state,
>> - info.digest,
>> - &format!(
>> - "can't verify chunk with digest {} - {err}",
>> - hex::encode(info.digest)
>> +
>> + for retry in 0..3 {
>> + match proxmox_async::runtime::block_on(s3_client.get_object(object_key.clone()))
>> + {
>> + Ok(Some(response)) => {
>> + match proxmox_async::runtime::block_on(response.content.collect()) {
>> + Ok(raw_chunk) => {
>> + match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
>> + Ok(chunk) => {
>> + let size = info.size();
>> + verify_state
>> + .read_bytes
>> + .fetch_add(chunk.raw_size(), Ordering::SeqCst);
>> + decoder_pool.send((chunk, info.digest, size))?;
>> + verify_state
>> + .decoded_bytes
>> + .fetch_add(size, Ordering::SeqCst);
>> + }
>> + Err(err) => Self::add_corrupt_chunk(
>> + verify_state,
>> + info.digest,
>> + &format!(
>> + "can't verify chunk with digest {} - {err}",
>> + hex::encode(info.digest)
>> + ),
>> ),
>> - ),
>> + }
>> + break;
>> + }
>> + Err(err) => {
>> + if retry < 2 {
>> + warn!(
>> + "retry reading chunk with digest {}, load failed - {err}",
>> + hex::encode(info.digest),
>> + );
>> + let backoff = Duration::from_secs(3_u64.pow(retry));
>> + std::thread::sleep(backoff);
>> + } else {
>> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> + error!(
>> + "can't verify chunk with digest {}, load failed - {err}",
>> + hex::encode(info.digest),
>> + );
>> + }
>> }
>> - }
>> - Err(err) => {
>> - verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> - error!("can't verify chunk, load failed - {err}");
>> }
>> }
>> - }
>> - Ok(None) => Self::add_corrupt_chunk(
>> - verify_state,
>> - info.digest,
>> - &format!(
>> - "can't verify missing chunk with digest {}",
>> - hex::encode(info.digest)
>> - ),
>> - ),
>> - Err(err) => {
>> - verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> - error!("can't verify chunk, load failed - {err}");
>> + Ok(None) => {
>> + Self::add_corrupt_chunk(
>> + verify_state,
>> + info.digest,
>> + &format!(
>> + "can't verify missing chunk with digest {}",
>> + hex::encode(info.digest)
>> + ),
>> + );
>> + break;
>> + }
>> + Err(err) => {
>> + verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
>> + error!(
>> + "can't verify chunk with digest {}, load failed - {err}",
>> + hex::encode(info.digest),
>> + );
>> + break;
>> + }
>> }
>> }
>> }
>> --
>> 2.47.3
>>
>>
>>
>>
>>
>>
next prev parent reply other threads:[~2026-11-12 10:09 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler
2026-11-12 10:09 ` Christian Ebner [this message]
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c4886f17-e1b0-4dde-aadf-d5557ded508f@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=f.gruenbichler@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox