all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry
Date: Mon, 16 Nov 2026 10:54:22 +0100	[thread overview]
Message-ID: <20261116095434.110541-1-c.ebner@proxmox.com> (raw)

Transient errors during fetching of objects from S3 backend currently
lead to snapshots being marked as verify failed. Subsequent runs of
verification on the same snapshot might succeed without issues, since
chunks are only flagged as bad if the chunk blob object could not be
decoded or content verification failed.

To fix this, discriminate between transient, permanent and task
runtime errors and account for these separately. For snapshots
encountering only transient errors, but no permanent errors during
verification, do not set or strip the verify okay state, but always
keep a pre-existing verification failed state. This is to never allow
reuse of a snapshot as reference for backup/sync jobs, when the state
was bad before and could not be verified again. For the other case
there is not much to lose, if it was okay or not verified before
falling back to not verified reflects the current snapshot state.

Avoid manifest updates which are not needed, since this causes
additional put requests to S3 backends and might fail as well.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7606

Changes since v4:
- Add retry quota implementation and apply potential slowdowns also to new
  requests

Changes since v3:
- Try falling back to Retry-After header if x-amz-retry-after is not present
- Double the base delay, in order to keep them closer to previous retry
  delays used by the s3-client
- fix stray newline

Changes since v2 (thanks @Fabian for review):
- move object content fetching retry logic into proxmox-s3-client
  helper and use it across all callsites in proxmox-backup
- extend proxmox-s3-client retry handling with some additional
  improvements:
  - also retry on `502 Bad Gateway` responses
  - follow AWS SDK standard retry backoff delays, distinguishing
    transient errors and throttled requests
  - detect throttling error codes from S3 XML error response bodies
  - add uniform jitter to retry backoff delays and honor
    `x-amz-retry-after` response headers
  - add backoff delay between object upload retries
- distinguish task abort/shutdown from transient and permanent
  verification errors, leaving the snapshot verify state untouched
  for these errors
- do not swallow any previous permanent errors when chunk listing,
  reading or decoding has errors as well
- instead of stopping after the first transient fetch error, only stop
  verifying the current snapshot once 3 consecutive S3 chunk fetches
  fail, resetting the counter on successful fetches and per snapshot

Changes since v1:
- fix off-by-one error in retry count check, only up to 2 retries are
  being performed.
- refactor manifest verify state parsing error handling, it cannot
  propagate an error as that is interpreted as task abort. try to
  update the manifest's verify state instead and log the parsing
  issue.
- early stop verification on transient errors for datastores backed
  by S3, as this causes additional requests, costs and runtime at no
  benefit.
- improve on logging and code style in general


proxmox:

Christian Ebner (8):
  s3-client: also retry requests on bad gateway responses
  s3-client: distinguish transient errors and throttled requests
  s3-client: detect throttling request from the response body on error
  s3-client: avoid retry bursts by adding jitter to retry backoff delay
  s3-client: honor response header directed retry delays
  s3-client: add additional backoff delay between upload retries
  s3-client: add helper to fetch object contents with retry
  s3-client: add shared rate limiter based retry quota

 proxmox-s3-client/src/client.rs          | 384 ++++++++++++++++++++++-
 proxmox-s3-client/src/response_reader.rs |  36 +++
 proxmox-s3-client/src/timestamps.rs      |   7 +
 3 files changed, 413 insertions(+), 14 deletions(-)


proxmox-backup:

Christian Ebner (4):
  s3: retry fetching object contents on body collection errors
  datastore: snapshot: propagate errors in manifest update callback
  fix #7606: verify: s3: handle transient chunk load errors
  verify: s3: stop early if the endpoint appears unreachable

 pbs-datastore/Cargo.toml                      |   1 -
 pbs-datastore/src/backup_info.rs              |   4 +-
 pbs-datastore/src/datastore.rs                |  13 +-
 pbs-datastore/src/local_chunk_reader.rs       |   4 +-
 .../src/local_datastore_lru_cache.rs          |   6 +-
 src/api2/config/datastore.rs                  |  13 +-
 src/api2/reader/mod.rs                        |   4 +-
 src/backup/verify.rs                          | 364 ++++++++++++++----
 8 files changed, 297 insertions(+), 112 deletions(-)


Summary over all repositories:
  11 files changed, 710 insertions(+), 126 deletions(-)

-- 
Generated by murpp 0.11.0




             reply	other threads:[~2026-11-16  9:55 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  9:54 Christian Ebner [this message]
2026-11-16  9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-16  9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116095434.110541-1-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal