all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry
@ 2026-11-16  9:54 Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
                   ` (11 more replies)
  0 siblings, 12 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Transient errors during fetching of objects from S3 backend currently
lead to snapshots being marked as verify failed. Subsequent runs of
verification on the same snapshot might succeed without issues, since
chunks are only flagged as bad if the chunk blob object could not be
decoded or content verification failed.

To fix this, discriminate between transient, permanent and task
runtime errors and account for these separately. For snapshots
encountering only transient errors, but no permanent errors during
verification, do not set or strip the verify okay state, but always
keep a pre-existing verification failed state. This is to never allow
reuse of a snapshot as reference for backup/sync jobs, when the state
was bad before and could not be verified again. For the other case
there is not much to lose, if it was okay or not verified before
falling back to not verified reflects the current snapshot state.

Avoid manifest updates which are not needed, since this causes
additional put requests to S3 backends and might fail as well.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7606

Changes since v4:
- Add retry quota implementation and apply potential slowdowns also to new
  requests

Changes since v3:
- Try falling back to Retry-After header if x-amz-retry-after is not present
- Double the base delay, in order to keep them closer to previous retry
  delays used by the s3-client
- fix stray newline

Changes since v2 (thanks @Fabian for review):
- move object content fetching retry logic into proxmox-s3-client
  helper and use it across all callsites in proxmox-backup
- extend proxmox-s3-client retry handling with some additional
  improvements:
  - also retry on `502 Bad Gateway` responses
  - follow AWS SDK standard retry backoff delays, distinguishing
    transient errors and throttled requests
  - detect throttling error codes from S3 XML error response bodies
  - add uniform jitter to retry backoff delays and honor
    `x-amz-retry-after` response headers
  - add backoff delay between object upload retries
- distinguish task abort/shutdown from transient and permanent
  verification errors, leaving the snapshot verify state untouched
  for these errors
- do not swallow any previous permanent errors when chunk listing,
  reading or decoding has errors as well
- instead of stopping after the first transient fetch error, only stop
  verifying the current snapshot once 3 consecutive S3 chunk fetches
  fail, resetting the counter on successful fetches and per snapshot

Changes since v1:
- fix off-by-one error in retry count check, only up to 2 retries are
  being performed.
- refactor manifest verify state parsing error handling, it cannot
  propagate an error as that is interpreted as task abort. try to
  update the manifest's verify state instead and log the parsing
  issue.
- early stop verification on transient errors for datastores backed
  by S3, as this causes additional requests, costs and runtime at no
  benefit.
- improve on logging and code style in general


proxmox:

Christian Ebner (8):
  s3-client: also retry requests on bad gateway responses
  s3-client: distinguish transient errors and throttled requests
  s3-client: detect throttling request from the response body on error
  s3-client: avoid retry bursts by adding jitter to retry backoff delay
  s3-client: honor response header directed retry delays
  s3-client: add additional backoff delay between upload retries
  s3-client: add helper to fetch object contents with retry
  s3-client: add shared rate limiter based retry quota

 proxmox-s3-client/src/client.rs          | 384 ++++++++++++++++++++++-
 proxmox-s3-client/src/response_reader.rs |  36 +++
 proxmox-s3-client/src/timestamps.rs      |   7 +
 3 files changed, 413 insertions(+), 14 deletions(-)


proxmox-backup:

Christian Ebner (4):
  s3: retry fetching object contents on body collection errors
  datastore: snapshot: propagate errors in manifest update callback
  fix #7606: verify: s3: handle transient chunk load errors
  verify: s3: stop early if the endpoint appears unreachable

 pbs-datastore/Cargo.toml                      |   1 -
 pbs-datastore/src/backup_info.rs              |   4 +-
 pbs-datastore/src/datastore.rs                |  13 +-
 pbs-datastore/src/local_chunk_reader.rs       |   4 +-
 .../src/local_datastore_lru_cache.rs          |   6 +-
 src/api2/config/datastore.rs                  |  13 +-
 src/api2/reader/mod.rs                        |   4 +-
 src/backup/verify.rs                          | 364 ++++++++++++++----
 8 files changed, 297 insertions(+), 112 deletions(-)


Summary over all repositories:
  11 files changed, 710 insertions(+), 126 deletions(-)

-- 
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
                   ` (10 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

A bad gateway response is typically returned by proxies or load
balancers sitting in front of the S3 compatible object store if the
latter is temporarily not reachable. This is not as frequent as
already retried status codes, but should be retried as well.

Since such a status code signals a transient failure, add it to the
list of retriable responses.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 6a87bd0c..3f2a7107 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -510,6 +510,7 @@ impl S3Client {
                 Ok(response) => matches!(
                     response.status(),
                     StatusCode::INTERNAL_SERVER_ERROR
+                        | StatusCode::BAD_GATEWAY
                         | StatusCode::SERVICE_UNAVAILABLE
                         | StatusCode::GATEWAY_TIMEOUT
                         | StatusCode::TOO_MANY_REQUESTS
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
                   ` (9 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Currently failed requests are retried after a fixed 1 and 3 seconds,
regardless of the cause. This is longer than needed for connection or
server errors, only intended as graceful recovery period for heavily
loaded S3 APIs in the previous implementation.

Therefore, distinguish these 2 modes by following the backoff logic
use by the AWS SDK's standard retry mode instead: wait base * 2^retry,
capped at 20 seconds, with a base delay of 100 ms for transient errors
and 2 second for throttling. These values are chosen to be double the
ones provided by the guide, in order to give on average the same
throttling backoff delay as previously used by the client, once
uniformly distributed jitter is introduced as well in subsequent
changes.

Cover only the too many requests http status code for now, additional
status codes will use this based on the error code in the response
body parsing as extended in subsequent changes.

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 106 +++++++++++++++++++++++++++-----
 1 file changed, 92 insertions(+), 14 deletions(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 3f2a7107..28646aee 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -47,7 +47,11 @@ const MAX_S3_UPLOAD_RETRY: usize = 3;
 // Assumed minimum upload rate of 1 KiB/s for dynamic put object request timeout calculation.
 const S3_MIN_ASSUMED_UPLOAD_RATE: u64 = 1024;
 const MAX_S3_HTTP_REQUEST_RETRY: usize = 3;
-const S3_HTTP_REQUEST_RETRY_BACKOFF_DEFAULT: Duration = Duration::from_secs(1);
+// Retry backoff as used by the AWS SDKs standard retry mode, see
+// https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
+const S3_RETRY_BASE_DELAY_TRANSIENT: Duration = Duration::from_millis(100);
+const S3_RETRY_BASE_DELAY_THROTTLING: Duration = Duration::from_secs(2);
+const S3_RETRY_MAX_BACKOFF: Duration = Duration::from_secs(20);
 
 static SHARED_COUNTER_FLUSHER: LazyLock<RwLock<MmapFlusher>> =
     LazyLock::new(|| RwLock::new(MmapFlusher::new()));
@@ -506,19 +510,20 @@ impl S3Client {
                 self.client.request(request).await
             };
 
-            let do_retry = match &response {
-                Ok(response) => matches!(
-                    response.status(),
+            let retry_reason = match &response {
+                Ok(response) => match response.status() {
                     StatusCode::INTERNAL_SERVER_ERROR
-                        | StatusCode::BAD_GATEWAY
-                        | StatusCode::SERVICE_UNAVAILABLE
-                        | StatusCode::GATEWAY_TIMEOUT
-                        | StatusCode::TOO_MANY_REQUESTS
-                ),
-                Err(_) => true,
+                    | StatusCode::BAD_GATEWAY
+                    | StatusCode::SERVICE_UNAVAILABLE
+                    | StatusCode::GATEWAY_TIMEOUT => Some(RetryReason::Transient),
+                    StatusCode::TOO_MANY_REQUESTS => Some(RetryReason::Throttling),
+                    _ => None,
+                },
+                Err(_) => Some(RetryReason::Transient),
             };
 
-            if !do_retry || retry >= MAX_S3_HTTP_REQUEST_RETRY - 1 {
+            let Some(retry_reason) = retry_reason.filter(|_| retry < MAX_S3_HTTP_REQUEST_RETRY - 1)
+            else {
                 if let Some(counters) = self.request_counters.as_ref()
                     && response.is_ok()
                 {
@@ -533,10 +538,9 @@ impl S3Client {
                     let _ = SHARED_COUNTER_FLUSHER.read().unwrap().request_flush();
                 }
                 return Ok(response?);
-            }
+            };
 
-            let backoff_secs = S3_HTTP_REQUEST_RETRY_BACKOFF_DEFAULT * 3_u32.pow(retry as u32);
-            tokio::time::sleep(backoff_secs).await;
+            tokio::time::sleep(retry_backoff(retry_reason, retry)).await;
         }
 
         bail!("failed to send request exceeding retries");
@@ -1026,10 +1030,84 @@ impl S3Client {
     }
 }
 
+#[derive(Debug, PartialEq)]
+enum RetryReason {
+    Transient,
+    Throttling,
+}
+
+impl RetryReason {
+    fn base_delay(self) -> Duration {
+        match self {
+            Self::Transient => S3_RETRY_BASE_DELAY_TRANSIENT,
+            Self::Throttling => S3_RETRY_BASE_DELAY_THROTTLING,
+        }
+    }
+}
+
+/// Helper to get exponential backoff duration based on retry reason and count.
+///
+/// Calculates how long to wait before the next retry attempt for a request.
+/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`].
+fn retry_backoff(reason: RetryReason, retry: usize) -> Duration {
+    u32::try_from(retry)
+        .ok()
+        .and_then(|retry| 2_u32.checked_pow(retry))
+        .and_then(|factor| reason.base_delay().checked_mul(factor))
+        .map_or(S3_RETRY_MAX_BACKOFF, |backoff| {
+            backoff.min(S3_RETRY_MAX_BACKOFF)
+        })
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    #[test]
+    fn retry_backoff_by_reason() {
+        assert_eq!(
+            retry_backoff(RetryReason::Transient, 0),
+            Duration::from_millis(100)
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Transient, 1),
+            Duration::from_millis(200)
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Throttling, 0),
+            Duration::from_secs(2)
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Throttling, 1),
+            Duration::from_secs(4)
+        );
+    }
+
+    #[test]
+    fn retry_backoff_is_capped() {
+        assert_eq!(
+            retry_backoff(RetryReason::Throttling, 2),
+            Duration::from_secs(8)
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Transient, 4),
+            Duration::from_millis(1_600)
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Transient, 8),
+            S3_RETRY_MAX_BACKOFF
+        );
+        // must not overflow
+        assert_eq!(
+            retry_backoff(RetryReason::Throttling, 64),
+            S3_RETRY_MAX_BACKOFF
+        );
+        assert_eq!(
+            retry_backoff(RetryReason::Transient, usize::MAX),
+            S3_RETRY_MAX_BACKOFF
+        );
+    }
+
     struct TestDirectory(PathBuf);
 
     impl Drop for TestDirectory {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
                   ` (8 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

AWS S3 signals throttling with a service unavailable response and the
SlowDown error code, which is now classified as transient errors
since the previous change introducing this distinction and therefore
retried after the short backoff of a transient error.

Follow the AWS SDKs implementation guide on this and try to identify
retryable errors by the error code of the response body if present
instead, the body will be discarded on retry anyways.
Using the longer throttling backoff delay for retries introduced by
the previous change for the known throttling codes.

Other S3 compatible providers do use similar codes to signal rate
limiting [0].

[0] https://forum.proxmox.com/threads/184371/post-858173
[1] https://forum.proxmox.com/threads/176950/

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs          | 71 ++++++++++++++++++++++++
 proxmox-s3-client/src/response_reader.rs | 36 ++++++++++++
 2 files changed, 107 insertions(+)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 28646aee..2d04ccb7 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -5,6 +5,7 @@ use std::sync::{Arc, LazyLock, Mutex, RwLock};
 use std::time::{Duration, Instant};
 
 use anyhow::{Context, Error, bail, format_err};
+use http_body_util::BodyExt;
 use hyper::body::{Bytes, Incoming};
 use hyper::http::method::Method;
 use hyper::http::uri::{Authority, Parts, PathAndQuery, Scheme};
@@ -52,6 +53,23 @@ const MAX_S3_HTTP_REQUEST_RETRY: usize = 3;
 const S3_RETRY_BASE_DELAY_TRANSIENT: Duration = Duration::from_millis(100);
 const S3_RETRY_BASE_DELAY_THROTTLING: Duration = Duration::from_secs(2);
 const S3_RETRY_MAX_BACKOFF: Duration = Duration::from_secs(20);
+// Response body error codes interpreted as retry with throttled delay instead of transient delay.
+const S3_THROTTLING_ERROR_CODES: &[&str] = &[
+    "Throttling",
+    "ThrottlingException",
+    "ThrottledException",
+    "RequestThrottledException",
+    "TooManyRequestsException",
+    "ProvisionedThroughputExceededException",
+    "LimitExceededException",
+    "PriorRequestNotComplete",
+    "RequestThrottled",
+    "RequestLimitExceeded",
+    "SlowDown",
+    "BandwidthLimitExceeded",
+];
+// Maximum time to wait when reading an error response body to check for throttling codes.
+const S3_ERROR_BODY_READ_TIMEOUT: Duration = Duration::from_secs(5);
 
 static SHARED_COUNTER_FLUSHER: LazyLock<RwLock<MmapFlusher>> =
     LazyLock::new(|| RwLock::new(MmapFlusher::new()));
@@ -540,6 +558,28 @@ impl S3Client {
                 return Ok(response?);
             };
 
+            // The response is discarded on retry, so use it to check the error code to detect
+            // throttling signaled with a server error status.
+            let retry_reason = match response {
+                Ok(response) if retry_reason == RetryReason::Transient => {
+                    let response_body = response.into_body().collect();
+                    let read_deadline = tokio::time::Instant::now() + S3_ERROR_BODY_READ_TIMEOUT;
+                    let body_deadline = match deadline {
+                        Some(deadline) => read_deadline.min(deadline),
+                        None => read_deadline,
+                    };
+
+                    if let Ok(Ok(body)) =
+                        tokio::time::timeout_at(body_deadline, response_body).await
+                    {
+                        RetryReason::from_error_response_body(&body.to_bytes())
+                    } else {
+                        RetryReason::Transient
+                    }
+                }
+                _ => retry_reason,
+            };
+
             tokio::time::sleep(retry_backoff(retry_reason, retry)).await;
         }
 
@@ -1037,6 +1077,16 @@ enum RetryReason {
 }
 
 impl RetryReason {
+    fn from_error_response_body(body: &[u8]) -> Self {
+        if let Some(error_code) = crate::response_reader::parse_error_code(body)
+            && S3_THROTTLING_ERROR_CODES.contains(&error_code.as_str())
+        {
+            return Self::Throttling;
+        }
+
+        Self::Transient
+    }
+
     fn base_delay(self) -> Duration {
         match self {
             Self::Transient => S3_RETRY_BASE_DELAY_TRANSIENT,
@@ -1083,6 +1133,27 @@ mod tests {
         );
     }
 
+    #[test]
+    fn retry_reason_from_error_code() {
+        let error_body = |code: &str| format!("<Error><Code>{code}</Code></Error>");
+        assert_eq!(
+            RetryReason::from_error_response_body(error_body("SlowDown").as_bytes()),
+            RetryReason::Throttling
+        );
+        assert_eq!(
+            RetryReason::from_error_response_body(error_body("InternalError").as_bytes()),
+            RetryReason::Transient
+        );
+        assert_eq!(
+            RetryReason::from_error_response_body(b""),
+            RetryReason::Transient
+        );
+        assert_eq!(
+            RetryReason::from_error_response_body(b"<html>Service Unavailable</html>"),
+            RetryReason::Transient
+        );
+    }
+
     #[test]
     fn retry_backoff_is_capped() {
         assert_eq!(
diff --git a/proxmox-s3-client/src/response_reader.rs b/proxmox-s3-client/src/response_reader.rs
index 1a3a9add..f941aaa2 100644
--- a/proxmox-s3-client/src/response_reader.rs
+++ b/proxmox-s3-client/src/response_reader.rs
@@ -605,6 +605,42 @@ impl ResponseReader {
     }
 }
 
+#[derive(Deserialize)]
+#[serde(rename_all = "PascalCase")]
+/// Common error response body of S3 api calls, only the error code is of interest.
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#RESTErrorResponses
+struct ErrorResponseBody {
+    code: Option<String>,
+}
+
+/// Parse the error code from an S3 api error response body, if any.
+pub(crate) fn parse_error_code(body: &[u8]) -> Option<String> {
+    let body = std::str::from_utf8(body).ok()?;
+    let error: ErrorResponseBody = serde_xml_rs::from_str(body).ok()?;
+    error.code
+}
+
+#[test]
+fn parse_error_code_test() {
+    let response_body = r#"<?xml version="1.0" encoding="UTF-8"?>
+        <Error>
+            <Code>SlowDown</Code>
+            <Message>Please reduce your request rate.</Message>
+            <RequestId>4442587FB7D0A2F9</RequestId>
+        </Error>"#;
+    assert_eq!(
+        parse_error_code(response_body.as_bytes()),
+        Some("SlowDown".to_string())
+    );
+    assert_eq!(
+        parse_error_code(b"<Error><Message>no code</Message></Error>"),
+        None
+    );
+    assert_eq!(parse_error_code(b""), None);
+    assert_eq!(parse_error_code(b"<html>Bad Gateway"), None);
+    assert_eq!(parse_error_code(&[0xff, 0xfe]), None);
+}
+
 #[test]
 fn parse_list_objects_v2_response_test() {
     let response_body = r#"<?xml version="1.0" encoding="UTF-8"?>
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (2 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
                   ` (7 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Requests failing at the same time, are all retried after exactly the
same delay due to the deterministic backoff calculation. This causes
bursts of retries hitting the server at once, which makes it more
likely for them to fail again.

Therefore, randomize the delay uniformly between zero and the
computed backoff, which spreads the retries over the whole backoff
window. If no random value can be generated, wait for the full
backoff rather than not at all.

This follows the AWS SDKs implementation guide.

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 2d04ccb7..149b421c 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -580,7 +580,7 @@ impl S3Client {
                 _ => retry_reason,
             };
 
-            tokio::time::sleep(retry_backoff(retry_reason, retry)).await;
+            tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await;
         }
 
         bail!("failed to send request exceeding retries");
@@ -1095,6 +1095,16 @@ impl RetryReason {
     }
 }
 
+/// Helper to get exponential backoff duration based on retry reason and count with
+/// uniform jitter.
+///
+/// Calculates how long to wait before the next retry attempt for a request.
+/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`], jitter applied
+/// afterwards.
+fn retry_backoff_with_jitter(reason: RetryReason, retry: usize) -> Duration {
+    with_jitter(retry_backoff(reason, retry))
+}
+
 /// Helper to get exponential backoff duration based on retry reason and count.
 ///
 /// Calculates how long to wait before the next retry attempt for a request.
@@ -1109,6 +1119,17 @@ fn retry_backoff(reason: RetryReason, retry: usize) -> Duration {
         })
 }
 
+/// Randomize the given backoff uniformly between zero and its full duration, so clients which
+/// failed at the same time do not retry in lockstep.
+fn with_jitter(backoff: Duration) -> Duration {
+    let mut random = [0u8; 8];
+    match openssl::rand::rand_bytes(&mut random) {
+        Ok(()) => backoff.mul_f64(u64::from_ne_bytes(random) as f64 / u64::MAX as f64),
+        // rather wait too long than retry without any delay
+        Err(_) => backoff,
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1154,6 +1175,15 @@ mod tests {
         );
     }
 
+    #[test]
+    fn with_jitter_within_backoff() {
+        let backoff = Duration::from_secs(1);
+        let delays: Vec<Duration> = (0..1000).map(|_| with_jitter(backoff)).collect();
+        assert!(delays.iter().all(|delay| *delay <= backoff));
+        assert!(delays.iter().any(|delay| *delay != delays[0]));
+        assert_eq!(with_jitter(Duration::ZERO), Duration::ZERO);
+    }
+
     #[test]
     fn retry_backoff_is_capped() {
         assert_eq!(
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (3 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
                   ` (6 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

S3 API servers can tell clients when they expect to have capacity
available again by setting the x-amz-retry-after or Retry-After
header. The former is provided in milliseconds, the latter either as
integer seconds or http date. Since retrying earlier is likely to
fail again and only adds load, try to honor these server provided
delays before the next retry.

Use the server's requested delay instead of the computed backoff, but
never wait less than the backoff and at most 5 seconds longer, so a
misbehaving server cannot stall requests for a long time. No jitter
is applied, the server is expected to randomize the delay itself.

This follows the AWS SDKs implementation guide with respect to
x-amz-retry-after and general http protocol for Retry-After.

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Link: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs     | 99 ++++++++++++++++++++++++++++-
 proxmox-s3-client/src/timestamps.rs |  7 ++
 2 files changed, 103 insertions(+), 3 deletions(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 149b421c..c4fb2733 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -53,6 +53,8 @@ const MAX_S3_HTTP_REQUEST_RETRY: usize = 3;
 const S3_RETRY_BASE_DELAY_TRANSIENT: Duration = Duration::from_millis(100);
 const S3_RETRY_BASE_DELAY_THROTTLING: Duration = Duration::from_secs(2);
 const S3_RETRY_MAX_BACKOFF: Duration = Duration::from_secs(20);
+// Maximum time a header directed retry delay may exceed the computed backoff.
+const S3_RETRY_AFTER_MAX_EXCESS: Duration = Duration::from_secs(5);
 // Response body error codes interpreted as retry with throttled delay instead of transient delay.
 const S3_THROTTLING_ERROR_CODES: &[&str] = &[
     "Throttling",
@@ -558,8 +560,13 @@ impl S3Client {
                 return Ok(response?);
             };
 
-            // The response is discarded on retry, so use it to check the error code to detect
-            // throttling signaled with a server error status.
+            let retry_after = response
+                .as_ref()
+                .ok()
+                .and_then(|response| parse_retry_after(response.headers()));
+
+            // The response is discarded for the retry anyway, so check its error code to detect
+            // throttling signaled with a server error status, for example S3's 503 SlowDown.
             let retry_reason = match response {
                 Ok(response) if retry_reason == RetryReason::Transient => {
                     let response_body = response.into_body().collect();
@@ -580,7 +587,8 @@ impl S3Client {
                 _ => retry_reason,
             };
 
-            tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await;
+            let backoff = retry_backoff(retry_reason, retry);
+            tokio::time::sleep(server_directed_delay(retry_after, backoff)).await;
         }
 
         bail!("failed to send request exceeding retries");
@@ -1130,6 +1138,42 @@ fn with_jitter(backoff: Duration) -> Duration {
     }
 }
 
+/// Parse the server directed retry delay from the response headers, if any.
+///
+/// Checks `x-amz-retry-after` (given in milliseconds) first, and falls back to the standard HTTP
+/// `Retry-After` header (given in integer seconds or as an HTTP-date) if not set.
+fn parse_retry_after(headers: &header::HeaderMap) -> Option<Duration> {
+    if let Some(millis) = headers
+        .get("x-amz-retry-after")
+        .and_then(|value| value.to_str().ok())
+        .and_then(|value| value.trim().parse().ok())
+    {
+        return Some(Duration::from_millis(millis));
+    }
+
+    let value = headers.get(header::RETRY_AFTER)?.to_str().ok()?.trim();
+    if let Ok(secs) = value.parse::<u64>() {
+        return Some(Duration::from_secs(secs));
+    }
+
+    let date = crate::HttpDate::from_str(value).ok()?;
+    let delay_secs = date
+        .epoch()
+        .saturating_sub(proxmox_time::epoch_i64())
+        .max(0) as u64;
+    Some(Duration::from_secs(delay_secs))
+}
+
+/// Use the response header directed retry delay if given, but never wait less than the computed
+/// backoff and at most a few seconds longer. The server is expected to randomize the delay, so
+/// no further jitter is applied.
+fn server_directed_delay(retry_after: Option<Duration>, backoff: Duration) -> Duration {
+    match retry_after {
+        Some(retry_after) => retry_after.clamp(backoff, backoff + S3_RETRY_AFTER_MAX_EXCESS),
+        None => with_jitter(backoff),
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1175,6 +1219,55 @@ mod tests {
         );
     }
 
+    #[test]
+    fn retry_after_header() {
+        let mut headers = header::HeaderMap::new();
+        assert_eq!(parse_retry_after(&headers), None);
+        headers.insert("x-amz-retry-after", HeaderValue::from_static("1500"));
+        assert_eq!(
+            parse_retry_after(&headers),
+            Some(Duration::from_millis(1500))
+        );
+        headers.insert("x-amz-retry-after", HeaderValue::from_static("soon"));
+        assert_eq!(parse_retry_after(&headers), None);
+        headers.insert("x-amz-retry-after", HeaderValue::from_static("-1"));
+        assert_eq!(parse_retry_after(&headers), None);
+
+        headers.clear();
+        headers.insert(header::RETRY_AFTER, HeaderValue::from_static("3"));
+        assert_eq!(parse_retry_after(&headers), Some(Duration::from_secs(3)));
+        headers.insert(
+            header::RETRY_AFTER,
+            HeaderValue::from_static("Wed, 12 Oct 2009 17:50:00 GMT"),
+        );
+        assert_eq!(parse_retry_after(&headers), Some(Duration::ZERO));
+        headers.insert(
+            header::RETRY_AFTER,
+            HeaderValue::from_static("Thu, 01 Jan 2099 00:00:00 GMT"),
+        );
+        assert!(parse_retry_after(&headers).is_some_and(|delay| delay > Duration::ZERO));
+        headers.insert(header::RETRY_AFTER, HeaderValue::from_static("invalid"));
+        assert_eq!(parse_retry_after(&headers), None);
+    }
+
+    #[test]
+    fn server_directed_delay_is_clamped() {
+        let backoff = Duration::from_secs(1);
+        assert!(server_directed_delay(None, backoff) <= backoff);
+        assert_eq!(
+            server_directed_delay(Some(Duration::from_secs(3)), backoff),
+            Duration::from_secs(3)
+        );
+        assert_eq!(
+            server_directed_delay(Some(Duration::from_millis(10)), backoff),
+            backoff
+        );
+        assert_eq!(
+            server_directed_delay(Some(Duration::from_secs(60)), backoff),
+            Duration::from_secs(6)
+        );
+    }
+
     #[test]
     fn with_jitter_within_backoff() {
         let backoff = Duration::from_secs(1);
diff --git a/proxmox-s3-client/src/timestamps.rs b/proxmox-s3-client/src/timestamps.rs
index af4c7d71..163a116e 100644
--- a/proxmox-s3-client/src/timestamps.rs
+++ b/proxmox-s3-client/src/timestamps.rs
@@ -33,6 +33,13 @@ pub struct HttpDate {
     _epoch: i64,
 }
 
+impl HttpDate {
+    /// Returns the Unix epoch timestamp in seconds.
+    pub(crate) fn epoch(&self) -> i64 {
+        self._epoch
+    }
+}
+
 impl std::str::FromStr for HttpDate {
     type Err = Error;
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (4 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
                   ` (5 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Uploads are repeated immediately if the server reports a conflicting
concurrent operation on the same object. Retrying right away gives a
concurrent operation no time to finish and adds load to a server
which already failed to handle the request.

Wait for the jittered transient backoff before repeating the upload,
as done for the individual requests.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index c4fb2733..12d9a206 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -1027,6 +1027,7 @@ impl S3Client {
                     }
                 }
             }
+            tokio::time::sleep(retry_backoff_with_jitter(RetryReason::Transient, retry)).await;
         }
         Ok(false)
     }
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (5 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
                   ` (4 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Sending requests is retried on connection errors and some error
responses, but the connection might still break while receiving the
response body.

Add a helper which fetches the object and collects its contents,
repeating the request with the jittered backoff for transient errors
if collecting the body fails, as such failures are caused by broken
connections. A new request is required, as a partially received body
cannot be resumed.

This is intended for callers which would collect the objects body
data right away anyways and allows to not have to reimplement custom
retry logic for that.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 36 +++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 12d9a206..aae3961a 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -660,6 +660,42 @@ impl S3Client {
         response_reader.get_object_response().await
     }
 
+    /// Fetch an object from object store and collect its contents.
+    ///
+    /// Returns `Ok(None)` if the object does not exist. Sending the request is retried by the
+    /// client already, but the connection can still break while the response body is being
+    /// received. Since a partially received body cannot be resumed, the whole request is repeated
+    /// with backoff in that case, up to the same number of attempts as for sending requests.
+    pub async fn get_object_content(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<Option<Bytes>, Error> {
+        let mut retry = 0;
+        loop {
+            let Some(response) = self.get_object(object_key.clone()).await? else {
+                return Ok(None);
+            };
+
+            let collect_result =
+                tokio::time::timeout(S3_HTTP_REQUEST_TIMEOUT, response.content.collect())
+                    .await
+                    .context("timeout while collecting object contents")
+                    .and_then(|res| res.context("failed to collect object contents"));
+
+            match collect_result {
+                Ok(content) => return Ok(Some(content.to_bytes())),
+                Err(err) => {
+                    if retry >= MAX_S3_HTTP_REQUEST_RETRY - 1 {
+                        return Err(err);
+                    }
+                    tokio::time::sleep(retry_backoff_with_jitter(RetryReason::Transient, retry))
+                        .await;
+                    retry += 1;
+                }
+            }
+        }
+    }
+
     /// Returns some or all (up to 1,000) of the objects in a bucket with each request.
     /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
     pub async fn list_objects_v2(
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (6 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
                   ` (3 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Delays for requests are currently only applied if either a shared
rate limiter is configured or if the request is being retried. New
requests are never delayed for latter case. If the server requested a
slow down, it makes senses to rate limit also new requests.

Therefore, implement a shared token bucket retry quota and reuse the
SharedRateLimiter to account for all requests. Use the values
indicated by the AWS SDK for token cost and refill behaviour of the
retry quota. In particular:
- 500 bucket size
- 50 tokens/s refill rate
- cost 5 for throttling retries and cost 10 for transient retries

When a request has not yet reached its retry limit and the retry
quota is exhausted, add the rate limiter's delay to the retry backoff
instead of failing hard, avoiding premature failure.

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 58 +++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 6 deletions(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index aae3961a..3c58946a 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -72,6 +72,12 @@ const S3_THROTTLING_ERROR_CODES: &[&str] = &[
 ];
 // Maximum time to wait when reading an error response body to check for throttling codes.
 const S3_ERROR_BODY_READ_TIMEOUT: Duration = Duration::from_secs(5);
+// Token bucket retry quota parameters based on the AWS SDKs standard retry mode, see
+// https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
+const S3_RETRY_QUOTA_BUCKET_SIZE: u64 = 500;
+const S3_RETRY_QUOTA_RATE: u64 = 50;
+const S3_RETRY_QUOTA_COST: u64 = 5;
+const S3_RETRY_QUOTA_TIMEOUT_COST: u64 = 10;
 
 static SHARED_COUNTER_FLUSHER: LazyLock<RwLock<MmapFlusher>> =
     LazyLock::new(|| RwLock::new(MmapFlusher::new()));
@@ -201,6 +207,7 @@ pub struct S3Client {
     authority: Authority,
     active_request_rate_limiter: Option<Arc<SharedRateLimiter>>,
     passive_request_rate_limiter: Option<Arc<SharedRateLimiter>>,
+    retry_rate_limiter: Option<Arc<SharedRateLimiter>>,
     // TODO: Drop for PBS 5.
     put_rate_limiter: Option<Arc<Mutex<RateLimiter>>>,
     request_counters: Option<Arc<SharedRequestCounters>>,
@@ -267,7 +274,11 @@ impl S3Client {
             S3_TCP_KEEPIDLE_TIME,
         );
 
-        let (mut passive_request_rate_limiter, mut active_request_rate_limiter) = (None, None);
+        let (
+            mut passive_request_rate_limiter,
+            mut active_request_rate_limiter,
+            mut retry_rate_limiter,
+        ) = (None, None, None);
         if let Some(limiter_config) = &options.rate_limiter_config {
             if let Some(limit) = limiter_config.rate_in {
                 let limiter = SharedRateLimiter::mmap_shmem(
@@ -312,6 +323,15 @@ impl S3Client {
                 )?;
                 passive_request_rate_limiter = Some(Arc::new(limiter));
             }
+
+            let limiter = SharedRateLimiter::mmap_shmem(
+                &format!("{}.retry-quota", limiter_config.options.id),
+                S3_RETRY_QUOTA_RATE,
+                S3_RETRY_QUOTA_BUCKET_SIZE,
+                limiter_config.options.user.clone(),
+                limiter_config.options.base_path.clone(),
+            )?;
+            retry_rate_limiter = Some(Arc::new(limiter));
         }
 
         if let Some(proxy_config) = &options.proxy_config {
@@ -371,6 +391,7 @@ impl S3Client {
             authority,
             active_request_rate_limiter,
             passive_request_rate_limiter,
+            retry_rate_limiter,
             put_rate_limiter,
             request_counters,
         })
@@ -495,6 +516,11 @@ impl S3Client {
 
         let deadline = timeout.map(|timeout| tokio::time::Instant::now() + timeout);
 
+        if let Some(limiter) = &self.retry_rate_limiter {
+            let sleep = limiter.register_traffic(Instant::now(), 0);
+            tokio::time::sleep(sleep).await;
+        }
+
         for retry in 0..MAX_S3_HTTP_REQUEST_RETRY {
             let mut request = Request::from_parts(parts.clone(), Body::from(body_bytes.clone()));
             if let Some(limiter) = &self.active_request_rate_limiter {
@@ -588,7 +614,11 @@ impl S3Client {
             };
 
             let backoff = retry_backoff(retry_reason, retry);
-            tokio::time::sleep(server_directed_delay(retry_after, backoff)).await;
+            let mut delay = server_directed_delay(retry_after, backoff);
+            if let Some(limiter) = &self.retry_rate_limiter {
+                delay += limiter.register_traffic(Instant::now(), retry_reason.quota_cost());
+            }
+            tokio::time::sleep(delay).await;
         }
 
         bail!("failed to send request exceeding retries");
@@ -688,8 +718,12 @@ impl S3Client {
                     if retry >= MAX_S3_HTTP_REQUEST_RETRY - 1 {
                         return Err(err);
                     }
-                    tokio::time::sleep(retry_backoff_with_jitter(RetryReason::Transient, retry))
-                        .await;
+                    let mut delay = retry_backoff_with_jitter(RetryReason::Transient, retry);
+                    if let Some(limiter) = &self.retry_rate_limiter {
+                        delay += limiter
+                            .register_traffic(Instant::now(), RetryReason::Transient.quota_cost());
+                    }
+                    tokio::time::sleep(delay).await;
                     retry += 1;
                 }
             }
@@ -1063,7 +1097,12 @@ impl S3Client {
                     }
                 }
             }
-            tokio::time::sleep(retry_backoff_with_jitter(RetryReason::Transient, retry)).await;
+            let mut delay = retry_backoff_with_jitter(RetryReason::Transient, retry);
+            if let Some(limiter) = &self.retry_rate_limiter {
+                delay +=
+                    limiter.register_traffic(Instant::now(), RetryReason::Transient.quota_cost());
+            }
+            tokio::time::sleep(delay).await;
         }
         Ok(false)
     }
@@ -1115,7 +1154,7 @@ impl S3Client {
     }
 }
 
-#[derive(Debug, PartialEq)]
+#[derive(Clone, Copy, Debug, PartialEq)]
 enum RetryReason {
     Transient,
     Throttling,
@@ -1138,6 +1177,13 @@ impl RetryReason {
             Self::Throttling => S3_RETRY_BASE_DELAY_THROTTLING,
         }
     }
+
+    fn quota_cost(self) -> u64 {
+        match self {
+            Self::Transient => S3_RETRY_QUOTA_TIMEOUT_COST,
+            Self::Throttling => S3_RETRY_QUOTA_COST,
+        }
+    }
 }
 
 /// Helper to get exponential backoff duration based on retry reason and count with
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (7 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
                   ` (2 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Requests to the S3 API are retried by the client on connection errors
and some error responses, but the connection can still break while
receiving the response body. Since the body is always collected right
after the request, such errors currently fail chunk reads for restore
and verification, cache refills, and the datastore refresh from the
bucket, even though repeating the request would most likely succeed.

Use the s3-client helper which implements the retry logic on fetch
errors.
The in-use marker check now fails if the marker content cannot be
received, instead of falling back to an empty content, but both cases
refuse to reuse the bucket anyway.

As a further positive side-effect the pbs-datastore create now no
longer depends on http-body-util and in particular the error handling
logic for chunk fetching during verification gets simpler.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/Cargo.toml                      |  1 -
 pbs-datastore/src/datastore.rs                | 12 ++---
 pbs-datastore/src/local_chunk_reader.rs       |  4 +-
 .../src/local_datastore_lru_cache.rs          |  6 +--
 src/api2/config/datastore.rs                  | 13 ++---
 src/api2/reader/mod.rs                        |  4 +-
 src/backup/verify.rs                          | 49 +++++++------------
 7 files changed, 30 insertions(+), 59 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index b51438dc6..b60a827f3 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -13,7 +13,6 @@ crc32fast.workspace = true
 endian_trait.workspace = true
 futures.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
-http-body-util.workspace = true
 hyper.workspace = true
 libc.workspace = true
 log.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index e6766a3b9..d62b21f9e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,7 +8,6 @@ use std::sync::{Arc, LazyLock, Mutex};
 use std::time::{Duration, SystemTime};
 
 use anyhow::{Context, Error, bail, format_err};
-use http_body_util::BodyExt;
 use hyper::Method;
 use hyper::body::Bytes;
 use nix::unistd::{UnlinkatFlags, unlinkat};
@@ -3262,18 +3261,13 @@ impl DataStore {
                     .await
                     .with_context(|| format!("failed to create target file {file_path:?}"))?;
 
-                if let Some(response) = s3_client
-                    .get_object(object.key)
+                if let Some(data) = s3_client
+                    .get_object_content(object.key)
                     .await
                     .with_context(|| format!("failed to fetch object {object_path:?}"))?
                 {
-                    let data = response
-                        .content
-                        .collect()
-                        .await
-                        .context("failed to collect object contents")?;
                     target_file
-                        .write_all(&data.to_bytes())
+                        .write_all(&data)
                         .await
                         .context("failed to write to target file")?;
                     file_create_options
diff --git a/pbs-datastore/src/local_chunk_reader.rs b/pbs-datastore/src/local_chunk_reader.rs
index faae43b71..431d6f89b 100644
--- a/pbs-datastore/src/local_chunk_reader.rs
+++ b/pbs-datastore/src/local_chunk_reader.rs
@@ -3,7 +3,6 @@ use std::pin::Pin;
 use std::sync::Arc;
 
 use anyhow::{Error, bail, format_err};
-use http_body_util::BodyExt;
 
 use pbs_api_types::CryptMode;
 use pbs_tools::crypt_config::CryptConfig;
@@ -55,8 +54,7 @@ impl LocalChunkReader {
 
 async fn fetch(s3_client: Arc<S3Client>, digest: &[u8; 32]) -> Result<DataBlob, Error> {
     let object_key = crate::s3::object_key_from_digest(digest)?;
-    if let Some(response) = s3_client.get_object(object_key).await? {
-        let bytes = response.content.collect().await?.to_bytes();
+    if let Some(bytes) = s3_client.get_object_content(object_key).await? {
         DataBlob::from_raw(bytes.to_vec())
     } else {
         bail!("no object with digest {}", hex::encode(digest));
diff --git a/pbs-datastore/src/local_datastore_lru_cache.rs b/pbs-datastore/src/local_datastore_lru_cache.rs
index 1bde8ee5a..12e6e3683 100644
--- a/pbs-datastore/src/local_datastore_lru_cache.rs
+++ b/pbs-datastore/src/local_datastore_lru_cache.rs
@@ -4,7 +4,6 @@
 use std::sync::Arc;
 
 use anyhow::{Error, bail};
-use http_body_util::BodyExt;
 
 use pbs_tools::async_lru_cache::AsyncLruCache;
 use proxmox_s3_client::S3Client;
@@ -124,12 +123,11 @@ impl LocalDatastoreLruCache {
         digest: &[u8; 32],
     ) -> Result<DataBlob, Error> {
         let object_key = crate::s3::object_key_from_digest(digest)?;
-        match client.get_object(object_key).await? {
+        match client.get_object_content(object_key).await? {
             None => {
                 bail!("could not fetch object with key {}", hex::encode(digest))
             }
-            Some(response) => {
-                let bytes = response.content.collect().await?.to_bytes();
+            Some(bytes) => {
                 let chunk = DataBlob::from_raw(bytes.to_vec())?;
                 self.insert(digest, &chunk)?;
                 Ok(chunk)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 350e48acc..19bdd73bc 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -3,7 +3,6 @@ use std::sync::Arc;
 
 use ::serde::{Deserialize, Serialize};
 use anyhow::{Context, Error, bail, format_err};
-use http_body_util::BodyExt;
 use serde_json::Value;
 use tracing::{info, warn};
 
@@ -128,14 +127,12 @@ pub(crate) fn do_create_datastore(
                 if !overwrite_in_use {
                     let object_key = S3ObjectKey::try_from(S3_DATASTORE_IN_USE_MARKER)
                         .context("failed to generate s3 object key")?;
-                    if let Some(response) =
-                        proxmox_async::runtime::block_on(s3_client.get_object(object_key.clone()))
-                            .context("failed to get in-use marker from bucket")?
+                    if let Some(content) = proxmox_async::runtime::block_on(
+                        s3_client.get_object_content(object_key.clone()),
+                    )
+                    .context("failed to get in-use marker from bucket")?
                     {
-                        let content = proxmox_async::runtime::block_on(response.content.collect())
-                            .unwrap_or_default();
-                        let content =
-                            String::from_utf8(content.to_bytes().to_vec()).unwrap_or_default();
+                        let content = String::from_utf8(content.to_vec()).unwrap_or_default();
                         let in_use: InUseContent =
                             serde_json::from_str(&content).unwrap_or_default();
                         if let Some(hostname) = in_use.hostname {
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index 29c00e74c..cbda79579 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -3,7 +3,6 @@
 use anyhow::{Context, Error, bail, format_err};
 use futures::*;
 use hex::FromHex;
-use http_body_util::BodyExt;
 use hyper::body::Incoming;
 use hyper::header::{self, CONNECTION, HeaderValue, UPGRADE};
 use hyper::http::request::Parts;
@@ -351,8 +350,7 @@ fn download_chunk(
 
 async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Result<Body, Error> {
     let object_key = pbs_datastore::s3::object_key_from_digest(digest)?;
-    if let Some(response) = s3_client.get_object(object_key).await? {
-        let data = response.content.collect().await?.to_bytes();
+    if let Some(data) = s3_client.get_object_content(object_key).await? {
         return Ok(Body::from(data));
     }
     bail!("cannot find chunk with digest {}", hex::encode(digest));
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 18fb1ca4c..01aafb4f2 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -5,7 +5,6 @@ use std::sync::{Arc, Mutex};
 use std::time::Instant;
 
 use anyhow::{Error, bail};
-use http_body_util::BodyExt;
 use tracing::{error, info, warn};
 
 use proxmox_parallel_handler::{ParallelHandler, SendHandle};
@@ -271,37 +270,25 @@ impl VerifyWorker {
             },
             DatastoreBackend::S3(s3_client) => {
                 let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
-                match proxmox_async::runtime::block_on(s3_client.get_object(object_key)) {
-                    Ok(Some(response)) => {
-                        match proxmox_async::runtime::block_on(response.content.collect()) {
-                            Ok(raw_chunk) => {
-                                match DataBlob::from_raw(raw_chunk.to_bytes().to_vec()) {
-                                    Ok(chunk) => {
-                                        let size = info.size();
-                                        verify_state
-                                            .read_bytes
-                                            .fetch_add(chunk.raw_size(), Ordering::SeqCst);
-                                        decoder_pool.send((chunk, info.digest, size))?;
-                                        verify_state
-                                            .decoded_bytes
-                                            .fetch_add(size, Ordering::SeqCst);
-                                    }
-                                    Err(err) => Self::add_corrupt_chunk(
-                                        verify_state,
-                                        info.digest,
-                                        &format!(
-                                            "can't verify chunk with digest {} - {err}",
-                                            hex::encode(info.digest)
-                                        ),
-                                    ),
-                                }
-                            }
-                            Err(err) => {
-                                verify_state.errors.fetch_add(1, Ordering::SeqCst);
-                                error!("can't verify chunk, load failed - {err}");
-                            }
+                match proxmox_async::runtime::block_on(s3_client.get_object_content(object_key)) {
+                    Ok(Some(raw_chunk)) => match DataBlob::from_raw(raw_chunk.to_vec()) {
+                        Ok(chunk) => {
+                            let size = info.size();
+                            verify_state
+                                .read_bytes
+                                .fetch_add(chunk.raw_size(), Ordering::SeqCst);
+                            decoder_pool.send((chunk, info.digest, size))?;
+                            verify_state.decoded_bytes.fetch_add(size, Ordering::SeqCst);
                         }
-                    }
+                        Err(err) => Self::add_corrupt_chunk(
+                            verify_state,
+                            info.digest,
+                            &format!(
+                                "can't verify chunk with digest {} - {err}",
+                                hex::encode(info.digest)
+                            ),
+                        ),
+                    },
                     Ok(None) => Self::add_corrupt_chunk(
                         verify_state,
                         info.digest,
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (8 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Allow propagating potential errors from the manifest update callback
to the caller.

In preparation for propagating verify state encoding errors.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 4 ++--
 pbs-datastore/src/datastore.rs   | 1 +
 src/backup/verify.rs             | 1 +
 3 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index cce2b699f..9c136fd3c 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -1001,12 +1001,12 @@ impl BackupDir {
     pub fn update_manifest(
         &self,
         backend: &DatastoreBackend,
-        update_fn: impl FnOnce(&mut BackupManifest),
+        update_fn: impl FnOnce(&mut BackupManifest) -> Result<(), Error>,
     ) -> Result<(), Error> {
         let _guard = self.lock_manifest()?;
         let (mut manifest, _) = self.load_manifest()?;
 
-        update_fn(&mut manifest);
+        update_fn(&mut manifest)?;
 
         // keep existing signature
         let blob = manifest.to_data_blob(None)?;
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index d62b21f9e..d34715b4b 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -3447,6 +3447,7 @@ impl DataStore {
         snapshot
             .update_manifest(&backend, |manifest| {
                 manifest.unprotected["notes"] = notes.into();
+                Ok(())
             })
             .map_err(|err| format_err!("unable to update manifest blob - {err}"))?;
         Ok(())
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 01aafb4f2..d2325ce9a 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -457,6 +457,7 @@ impl VerifyWorker {
             let verify_state = serde_json::to_value(verify_state)?;
             backup_dir.update_manifest(&self.datastore.backend()?, |manifest| {
                 manifest.unprotected["verify_state"] = verify_state;
+                Ok(())
             })
         } {
             info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (9 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

Fetching an object for verification might fail when sending the
request to the S3 API or while receiving the response. Both are
currently handled as hard errors, leading to the manifest state being
set to verify failed. This however does not reflect the actual state,
as a subsequent verification can work just fine.

Therefore, distinguish between permanent, transient and verification
task related errors (abort/shutdown).

If only transient errors, but no permanent errors occur, strip any
verify okay state since then it is not possible to determine if the
snapshot is still fine. That is best represented by not having a
verify state, as consequence sync jobs with verified-only filtering
will then exclude such snapshots as well. Independent of the error
variant, always keep a previously set verify error state in order to
avoid allowing backups to reuse this snapshot as reference. This
covers both the new state as well as the manifest state prior to
verification. On task related errors, the verify state is not
altered at all.

Errors of the chunk listing, parallel chunk readers or decoders also
leave the verification incomplete, but are only considered after all
chunks already sent have been checked, so they never hide permanent
errors.

Avoid manifest updates if not needed, since these cause additional
requests on S3 backed datastores and might fail as well.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7606
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/backup/verify.rs | 252 ++++++++++++++++++++++++++++++++++---------
 1 file changed, 201 insertions(+), 51 deletions(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index d2325ce9a..624c9a9aa 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -1,10 +1,10 @@
 use pbs_config::BackupLockGuard;
 use std::collections::HashSet;
-use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
 use std::sync::{Arc, Mutex};
 use std::time::Instant;
 
-use anyhow::{Error, bail};
+use anyhow::{Error, format_err};
 use tracing::{error, info, warn};
 
 use proxmox_parallel_handler::{ParallelHandler, SendHandle};
@@ -38,6 +38,7 @@ struct IndexVerifyState {
     read_bytes: AtomicU64,
     decoded_bytes: AtomicU64,
     errors: AtomicUsize,
+    fetch_errors: AtomicUsize,
     datastore: Arc<DataStore>,
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
@@ -54,6 +55,7 @@ impl IndexVerifyState {
             read_bytes: AtomicU64::new(0),
             decoded_bytes: AtomicU64::new(0),
             errors: AtomicUsize::new(0),
+            fetch_errors: AtomicUsize::new(0),
             datastore: Arc::clone(datastore),
             corrupt_chunks: Arc::clone(corrupt_chunks),
             verified_chunks: Arc::clone(verified_chunks),
@@ -62,6 +64,17 @@ impl IndexVerifyState {
     }
 }
 
+/// Outcome of a failed archive verification, deciding how the snapshot's verify state is updated.
+enum VerificationError {
+    /// The archive is known to be bad, for example because of corrupt or missing chunks.
+    Permanent(Error),
+    /// The archive could not be fully verified, for example because chunks could not be fetched
+    /// from the S3 backend. Its state is unknown.
+    Transient(Error),
+    /// The task was aborted or the daemon is shutting down, unrelated to the archive itself.
+    Task(Error),
+}
+
 impl VerifyWorker {
     /// Creates a new VerifyWorker for a given task worker and datastore.
     pub fn new(
@@ -94,27 +107,38 @@ impl VerifyWorker {
         })
     }
 
-    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
-        let blob = backup_dir.load_blob(info.filename.as_ref())?;
+    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
+        let blob = backup_dir
+            .load_blob(info.filename.as_ref())
+            .map_err(VerificationError::Permanent)?;
 
         let raw_size = blob.raw_size();
         if raw_size != info.size {
-            bail!("wrong size ({} != {})", info.size, raw_size);
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong size ({} != {})",
+                info.size,
+                raw_size
+            )));
         }
 
         let csum = openssl::sha::sha256(blob.raw_data());
         if csum != info.csum {
-            bail!("wrong index checksum");
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong index checksum"
+            )));
         }
 
-        match blob.crypt_mode()? {
+        match blob.crypt_mode().map_err(VerificationError::Permanent)? {
             CryptMode::Encrypt => Ok(()),
             CryptMode::None => {
                 // digest already verified above
-                blob.decode(None, None)?;
+                blob.decode(None, None)
+                    .map_err(VerificationError::Permanent)?;
                 Ok(())
             }
-            CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
+            CryptMode::SignOnly => Err(VerificationError::Permanent(format_err!(
+                "Invalid CryptMode for blob"
+            ))),
         }
     }
 
@@ -122,7 +146,7 @@ impl VerifyWorker {
         &self,
         index: Box<dyn IndexFile + Send>,
         crypt_mode: CryptMode,
-    ) -> Result<(), Error> {
+    ) -> Result<(), VerificationError> {
         let verify_state = Arc::new(IndexVerifyState::new(
             &self.datastore,
             &self.corrupt_chunks,
@@ -179,17 +203,34 @@ impl VerifyWorker {
             }
         };
 
+        // Track if the task was aborted or shutdown, so only that is treated as task error and any
+        // other error of the chunk ordering leaves the archive unverified.
+        let task_aborted_or_shutdown = AtomicBool::new(false);
         let check_abort = |pos: usize| -> Result<(), Error> {
             if pos & 1023 == 0 {
-                self.worker.check_abort()?;
-                self.worker.fail_on_shutdown()?;
+                let result = self
+                    .worker
+                    .check_abort()
+                    .and_then(|_| self.worker.fail_on_shutdown());
+                task_aborted_or_shutdown.store(result.is_err(), Ordering::Release);
+                result?;
             }
             Ok(())
         };
 
-        let chunk_list = self
-            .datastore
-            .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
+        let (chunk_list, chunk_list_error) =
+            match self
+                .datastore
+                .get_chunks_in_order(&*index, skip_chunk, check_abort)
+            {
+                Ok(chunk_list) => (chunk_list, None),
+                Err(err) => {
+                    if task_aborted_or_shutdown.load(Ordering::Acquire) {
+                        return Err(VerificationError::Task(err));
+                    }
+                    (Vec::new(), Some(err))
+                }
+            };
 
         let reader_pool = ParallelHandler::new("read chunks", self.read_threads, {
             let decoder_pool = decoder_pool.channel();
@@ -205,9 +246,13 @@ impl VerifyWorker {
                 )
             }
         });
+
+        let mut chunk_send_error: Option<Error> = None;
         for (pos, _) in chunk_list {
-            self.worker.check_abort()?;
-            self.worker.fail_on_shutdown()?;
+            self.worker.check_abort().map_err(VerificationError::Task)?;
+            self.worker
+                .fail_on_shutdown()
+                .map_err(VerificationError::Task)?;
 
             let info = index.chunk_info(pos).unwrap();
 
@@ -216,11 +261,15 @@ impl VerifyWorker {
                 continue; // already verified or marked corrupt
             }
 
-            reader_pool.send(info)?;
+            if let Err(err) = reader_pool.send(info) {
+                chunk_send_error = Some(err.into());
+                break;
+            }
         }
 
-        reader_pool.complete()?;
-        decoder_pool.complete()?;
+        // always wait for all workers, so all errors of chunks already sent are accounted for
+        let reader_result = reader_pool.complete();
+        let decoder_result = decoder_pool.complete();
 
         let elapsed = verify_state.start_time.elapsed().as_secs_f64();
 
@@ -234,13 +283,46 @@ impl VerifyWorker {
         let decode_speed = decoded_bytes_mib / elapsed;
 
         let error_count = verify_state.errors.load(Ordering::SeqCst);
+        let fetch_error_count = verify_state.fetch_errors.load(Ordering::SeqCst);
 
         info!(
-            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
+            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors, {fetch_error_count} fetch errors)"
         );
 
-        if verify_state.errors.load(Ordering::SeqCst) > 0 {
-            bail!("chunks could not be verified");
+        if error_count > 0 {
+            return Err(VerificationError::Permanent(format_err!(
+                "chunks could not be verified"
+            )));
+        }
+
+        if let Some(err) = chunk_list_error {
+            return Err(VerificationError::Transient(
+                err.context("incomplete verification because of chunk list error"),
+            ));
+        }
+
+        if let Some(err) = chunk_send_error {
+            return Err(VerificationError::Transient(
+                err.context("incomplete verification because of chunk send error"),
+            ));
+        }
+
+        if let Err(err) = reader_result {
+            return Err(VerificationError::Transient(Error::from(err).context(
+                "incomplete verification because of error in reader pool",
+            )));
+        }
+
+        if let Err(err) = decoder_result {
+            return Err(VerificationError::Transient(Error::from(err).context(
+                "incomplete verification because of error in decoder pool",
+            )));
+        }
+
+        if fetch_error_count > 0 {
+            return Err(VerificationError::Transient(format_err!(
+                "chunks could not be fetched"
+            )));
         }
 
         Ok(())
@@ -298,7 +380,7 @@ impl VerifyWorker {
                         ),
                     ),
                     Err(err) => {
-                        verify_state.errors.fetch_add(1, Ordering::SeqCst);
+                        verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
                         error!("can't verify chunk, load failed - {err}");
                     }
                 }
@@ -320,37 +402,63 @@ impl VerifyWorker {
         }
     }
 
-    fn verify_fixed_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+    fn verify_fixed_index(
+        &self,
+        backup_dir: &BackupDir,
+        info: &FileInfo,
+    ) -> Result<(), VerificationError> {
         let mut path = backup_dir.relative_path();
         path.push(info.filename.as_ref());
 
-        let index = self.datastore.open_fixed_reader(&path)?;
+        let index = self
+            .datastore
+            .open_fixed_reader(&path)
+            .map_err(VerificationError::Permanent)?;
 
         let (csum, size) = index.compute_csum();
         if size != info.size {
-            bail!("wrong size ({} != {})", info.size, size);
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong size ({} != {})",
+                info.size,
+                size
+            )));
         }
 
         if csum != info.csum {
-            bail!("wrong index checksum");
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong index checksum"
+            )));
         }
 
         self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
     }
 
-    fn verify_dynamic_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+    fn verify_dynamic_index(
+        &self,
+        backup_dir: &BackupDir,
+        info: &FileInfo,
+    ) -> Result<(), VerificationError> {
         let mut path = backup_dir.relative_path();
         path.push(info.filename.as_ref());
 
-        let index = self.datastore.open_dynamic_reader(&path)?;
+        let index = self
+            .datastore
+            .open_dynamic_reader(&path)
+            .map_err(VerificationError::Permanent)?;
 
         let (csum, size) = index.compute_csum();
         if size != info.size {
-            bail!("wrong size ({} != {})", info.size, size);
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong size ({} != {})",
+                info.size,
+                size
+            )));
         }
 
         if csum != info.csum {
-            bail!("wrong index checksum");
+            return Err(VerificationError::Permanent(format_err!(
+                "wrong index checksum"
+            )));
         }
 
         self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
@@ -426,7 +534,7 @@ impl VerifyWorker {
 
         let mut error_count = 0;
 
-        let mut verify_result = VerifyState::Ok;
+        let mut verify_result = Some(VerifyState::Ok);
         for info in manifest.files() {
             info!("  check {}", info.filename);
             let result = match info.filename.archive_type() {
@@ -438,30 +546,72 @@ impl VerifyWorker {
             self.worker.check_abort()?;
             self.worker.fail_on_shutdown()?;
 
-            if let Err(err) = result {
-                info!(
-                    "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
-                    file_name = info.filename,
-                );
-                error_count += 1;
-                verify_result = VerifyState::Failed;
+            match result {
+                Ok(_) => (),
+                // leave the verify state untouched, the outcome is not related to the snapshot
+                Err(VerificationError::Task(err)) => return Err(err),
+                Err(VerificationError::Permanent(err)) => {
+                    error_count += 1;
+                    verify_result = Some(VerifyState::Failed);
+
+                    info!(
+                        "verify {datastore_name}:{backup_dir_name}/{file_name} failed with permanent error: {err}",
+                        file_name = info.filename,
+                    );
+                }
+                Err(VerificationError::Transient(err)) => {
+                    error_count += 1;
+                    // strip any VerifyState::Ok when updating the manifest
+                    if let Some(VerifyState::Ok) = verify_result {
+                        verify_result = None;
+                    }
+
+                    info!(
+                        "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
+                        file_name = info.filename,
+                    );
+                }
             }
         }
 
-        let verify_state = SnapshotVerifyState {
-            state: verify_result,
-            upid,
+        let do_update_manifest = match verify_result {
+            Some(_) => true,
+            None => match manifest.verify_state() {
+                Ok(Some(SnapshotVerifyState {
+                    state: VerifyState::Ok,
+                    ..
+                })) => true,
+                // keep failed state (and its original UPID) or unverified state untouched
+                Ok(Some(SnapshotVerifyState {
+                    state: VerifyState::Failed,
+                    ..
+                }))
+                | Ok(None) => false,
+                Err(err) => {
+                    info!(
+                        "failed to parse verify state for {datastore_name}:{backup_dir_name}: {err}"
+                    );
+                    // set to failed since verify state parsing failed
+                    verify_result = Some(VerifyState::Failed);
+                    true
+                }
+            },
         };
 
-        if let Err(err) = {
-            let verify_state = serde_json::to_value(verify_state)?;
-            backup_dir.update_manifest(&self.datastore.backend()?, |manifest| {
-                manifest.unprotected["verify_state"] = verify_state;
+        let verify_state = verify_result.map(|state| SnapshotVerifyState { state, upid });
+        if do_update_manifest {
+            if let Err(err) = backup_dir.update_manifest(&self.backend, |manifest| {
+                if let Some(verify_state) = verify_state {
+                    let verify_state = serde_json::to_value(verify_state)?;
+                    manifest.unprotected["verify_state"] = verify_state;
+                } else if let Some(unprotected) = manifest.unprotected.as_object_mut() {
+                    unprotected.remove("verify_state");
+                }
                 Ok(())
-            })
-        } {
-            info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
-            return Ok(false);
+            }) {
+                info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
+                return Ok(false);
+            }
         }
 
         Ok(error_count == 0)
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable
  2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
                   ` (10 preceding siblings ...)
  2026-11-16  9:54 ` [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
@ 2026-11-16  9:54 ` Christian Ebner
  11 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-11-16  9:54 UTC (permalink / raw)
  To: pbs-devel

If the S3 endpoint cannot be reached, every remaining chunk fetch of
a snapshot fails only after the request retries and their backoff.
This causes useless requests and can block the verification for a
very long time, while the outcome is already known, as the snapshot
cannot be marked as verified anyway.

Stopping on the first transient error is not an option though: a
snapshot without verify state can still be used as reference by the
next backup, so corrupt chunks which would be detected later in the
same run must not be missed because of a single failed request.

Therefore, only stop verifying the chunks and the remaining archives
of a snapshot once a few chunk fetches failed in a row, with any
successful fetch resetting the count. Errors detected up to that
point are still accounted for, so known bad snapshots are marked as
failed. The count is reset for each snapshot, so verification
continues normally once the endpoint becomes reachable again.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/backup/verify.rs | 64 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 624c9a9aa..c29e60678 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -22,6 +22,10 @@ use pbs_datastore::{DataBlob, DataStore, DatastoreBackend, StoreProgress};
 
 use crate::backup::hierarchy::ListAccessibleBackupGroups;
 
+/// Number of chunk fetches from the S3 backend failing in a row, after which the endpoint is
+/// considered unreachable and verification of the current snapshot is stopped.
+const S3_MAX_CONSECUTIVE_FETCH_ERRORS: usize = 3;
+
 /// A VerifyWorker encapsulates a task worker, datastore and information about which chunks have
 /// already been verified or detected as corrupt.
 pub struct VerifyWorker {
@@ -32,6 +36,7 @@ pub struct VerifyWorker {
     backend: DatastoreBackend,
     read_threads: usize,
     verify_threads: usize,
+    consecutive_fetch_errors: Arc<AtomicUsize>,
 }
 
 struct IndexVerifyState {
@@ -43,6 +48,7 @@ struct IndexVerifyState {
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     start_time: Instant,
+    consecutive_fetch_errors: Arc<AtomicUsize>,
 }
 
 impl IndexVerifyState {
@@ -50,6 +56,7 @@ impl IndexVerifyState {
         datastore: &Arc<DataStore>,
         corrupt_chunks: &Arc<Mutex<HashSet<[u8; 32]>>>,
         verified_chunks: &Arc<Mutex<HashSet<[u8; 32]>>>,
+        consecutive_fetch_errors: &Arc<AtomicUsize>,
     ) -> Self {
         Self {
             read_bytes: AtomicU64::new(0),
@@ -60,6 +67,7 @@ impl IndexVerifyState {
             corrupt_chunks: Arc::clone(corrupt_chunks),
             verified_chunks: Arc::clone(verified_chunks),
             start_time: Instant::now(),
+            consecutive_fetch_errors: Arc::clone(consecutive_fetch_errors),
         }
     }
 }
@@ -104,9 +112,17 @@ impl VerifyWorker {
             backend,
             read_threads,
             verify_threads,
+            consecutive_fetch_errors: Arc::new(AtomicUsize::new(0)),
         })
     }
 
+    /// Check if fetching chunks from the S3 backend failed too often in a row. A single transient
+    /// error must not stop the verification, as that could hide corrupt chunks which would then
+    /// be reused by subsequent backups.
+    fn s3_endpoint_unreachable(&self) -> bool {
+        self.consecutive_fetch_errors.load(Ordering::SeqCst) >= S3_MAX_CONSECUTIVE_FETCH_ERRORS
+    }
+
     fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
         let blob = backup_dir
             .load_blob(info.filename.as_ref())
@@ -151,6 +167,7 @@ impl VerifyWorker {
             &self.datastore,
             &self.corrupt_chunks,
             &self.verified_chunks,
+            &self.consecutive_fetch_errors,
         ));
 
         let decoder_pool = ParallelHandler::new("verify chunk decoder", self.verify_threads, {
@@ -248,12 +265,20 @@ impl VerifyWorker {
         });
 
         let mut chunk_send_error: Option<Error> = None;
+        let mut exhausted_error: Option<Error> = None;
         for (pos, _) in chunk_list {
             self.worker.check_abort().map_err(VerificationError::Task)?;
             self.worker
                 .fail_on_shutdown()
                 .map_err(VerificationError::Task)?;
 
+            if self.s3_endpoint_unreachable() {
+                // avoid further requests and long stalls, the archive cannot be verified anyway
+                warn!("stop verifying chunks, too many consecutive fetch errors");
+                exhausted_error = Some(format_err!("S3 endpoint appears to be unreachable"));
+                break;
+            }
+
             let info = index.chunk_info(pos).unwrap();
 
             // we must always recheck this here, the parallel worker below alter it!
@@ -271,6 +296,11 @@ impl VerifyWorker {
         let reader_result = reader_pool.complete();
         let decoder_result = decoder_pool.complete();
 
+        if exhausted_error.is_none() && self.s3_endpoint_unreachable() {
+            warn!("stop verifying chunks, too many consecutive fetch errors");
+            exhausted_error = Some(format_err!("S3 endpoint appears to be unreachable"));
+        }
+
         let elapsed = verify_state.start_time.elapsed().as_secs_f64();
 
         let read_bytes = verify_state.read_bytes.load(Ordering::SeqCst);
@@ -301,6 +331,12 @@ impl VerifyWorker {
             ));
         }
 
+        if let Some(err) = exhausted_error {
+            return Err(VerificationError::Transient(
+                err.context("incomplete verification"),
+            ));
+        }
+
         if let Some(err) = chunk_send_error {
             return Err(VerificationError::Transient(
                 err.context("incomplete verification because of chunk send error"),
@@ -351,8 +387,21 @@ impl VerifyWorker {
                 }
             },
             DatastoreBackend::S3(s3_client) => {
+                if verify_state.consecutive_fetch_errors.load(Ordering::SeqCst)
+                    >= S3_MAX_CONSECUTIVE_FETCH_ERRORS
+                {
+                    verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
+                    return Ok(());
+                }
                 let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
-                match proxmox_async::runtime::block_on(s3_client.get_object_content(object_key)) {
+                let result =
+                    proxmox_async::runtime::block_on(s3_client.get_object_content(object_key));
+                if result.is_ok() {
+                    verify_state
+                        .consecutive_fetch_errors
+                        .store(0, Ordering::SeqCst);
+                }
+                match result {
                     Ok(Some(raw_chunk)) => match DataBlob::from_raw(raw_chunk.to_vec()) {
                         Ok(chunk) => {
                             let size = info.size();
@@ -381,6 +430,9 @@ impl VerifyWorker {
                     ),
                     Err(err) => {
                         verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
+                        verify_state
+                            .consecutive_fetch_errors
+                            .fetch_add(1, Ordering::SeqCst);
                         error!("can't verify chunk, load failed - {err}");
                     }
                 }
@@ -532,6 +584,9 @@ impl VerifyWorker {
 
         info!("verify {datastore_name}:{backup_dir_name}");
 
+        // probe the endpoint again for each snapshot, it might have become reachable again
+        self.consecutive_fetch_errors.store(0, Ordering::SeqCst);
+
         let mut error_count = 0;
 
         let mut verify_result = Some(VerifyState::Ok);
@@ -570,6 +625,13 @@ impl VerifyWorker {
                         "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
                         file_name = info.filename,
                     );
+
+                    if self.s3_endpoint_unreachable() {
+                        warn!(
+                            "skip verifying remaining archives of {datastore_name}:{backup_dir_name}"
+                        );
+                        break;
+                    }
                 }
             }
         }
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2026-11-16  9:55 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16  9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal