public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox{,-backup} v3 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry
Date: Thu, 12 Nov 2026 19:01:08 +0100	[thread overview]
Message-ID: <20261112180119.660923-1-c.ebner@proxmox.com> (raw)

Transient errors during fetching of objects from S3 backend currently
lead to snapshots being marked as verify failed. Subsequent runs of
verification on the same snapshot might succeed without issues, since
chunks are only flagged as bad if the chunk blob object could not be
decoded or content verification failed.

To fix this, discriminate between transient, permanent and task
runtime errors and account for these separately. For snapshots
encountering only transient errors, but no permanent errors during
verification, do not set or strip the verify okay state, but always
keep a pre-existing verification failed state. This is to never allow
reuse of a snapshot as reference for backup/sync jobs, when the state
was bad before and could not be verified again. For the other case
there is not much to lose, if it was okay or not verified before
falling back to not verified reflects the current snapshot state.

Avoid manifest updates which are not needed, since this causes
additional put requests to S3 backends and might fail as well.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7606

Changes since v2 (thanks @Fabian for review):
- move object content fetching retry logic into proxmox-s3-client
  helper and use it across all callsites in proxmox-backup
- extend proxmox-s3-client retry handling with some additional
  improvements:
  - also retry on `502 Bad Gateway` responses
  - follow AWS SDK standard retry backoff delays, distinguishing
    transient errors and throttled requests
  - detect throttling error codes from S3 XML error response bodies
  - add uniform jitter to retry backoff delays and honor
    `x-amz-retry-after` response headers
  - add backoff delay between object upload retries
- distinguish task abort/shutdown from transient and permanent
  verification errors, leaving the snapshot verify state untouched
  for these errors
- do not swallow any previous permanent errors when chunk listing,
  reading or decoding has errors as well
- instead of stopping after the first transient fetch error, only stop
  verifying the current snapshot once 3 consecutive S3 chunk fetches
  fail, resetting the counter on successful fetches and per snapshot

Changes since v1:
- fix off-by-one error in retry count check, only up to 2 retries are
  being performed.
- refactor manifest verify state parsing error handling, it cannot
  propagate an error as that is interpreted as task abort. try to
  update the manifest's verify state instead and log the parsing
  issue.
- early stop verification on transient errors for datastores backed
  by S3, as this causes additional requests, costs and runtime at no
  benefit.
- improve on logging and code style in general


proxmox:

Christian Ebner (7):
  s3-client: also retry requests on bad gateway responses
  s3-client: distinguish transient errors and throttled requests
  s3-client: detect throttling request from the response body on error
  s3-client: avoid retry bursts by adding jitter to retry backoff delay
  s3-client: honor response header directed retry delays
  s3-client: add additional backoff delay between upload retries
  s3-client: add helper to fetch object contents with retry

 proxmox-s3-client/src/client.rs          | 305 ++++++++++++++++++++++-
 proxmox-s3-client/src/response_reader.rs |  36 +++
 2 files changed, 328 insertions(+), 13 deletions(-)


proxmox-backup:

Christian Ebner (4):
  s3: retry fetching object contents on body collection errors
  datastore: snapshot: propagate errors in manifest update callback
  fix #7606: verify: s3: handle transient chunk load errors
  verify: s3: stop early if the endpoint appears unreachable

 pbs-datastore/Cargo.toml                      |   1 -
 pbs-datastore/src/backup_info.rs              |   4 +-
 pbs-datastore/src/datastore.rs                |  13 +-
 pbs-datastore/src/local_chunk_reader.rs       |   4 +-
 .../src/local_datastore_lru_cache.rs          |   6 +-
 src/api2/config/datastore.rs                  |  13 +-
 src/api2/reader/mod.rs                        |   4 +-
 src/backup/verify.rs                          | 364 ++++++++++++++----
 8 files changed, 297 insertions(+), 112 deletions(-)


Summary over all repositories:
  10 files changed, 625 insertions(+), 125 deletions(-)

-- 
Generated by murpp 0.11.0




             reply	other threads:[~2026-11-12 18:01 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 18:01 Christian Ebner [this message]
2026-11-12 18:01 ` [PATCH proxmox v3 01/11] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 02/11] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 03/11] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 05/11] s3-client: honor response header directed retry delays Christian Ebner
2026-11-13  8:13   ` Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 06/11] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox v3 07/11] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox-backup v3 08/11] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox-backup v3 09/11] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox-backup v3 10/11] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 18:01 ` [PATCH proxmox-backup v3 11/11] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
2026-11-13 14:23 ` superseded: [PATCH proxmox{,-backup} v3 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261112180119.660923-1-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal