From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry
Date: Fri, 13 Nov 2026 15:21:37 +0100 [thread overview]
Message-ID: <20261113142148.569471-1-c.ebner@proxmox.com> (raw)
Transient errors during fetching of objects from S3 backend currently
lead to snapshots being marked as verify failed. Subsequent runs of
verification on the same snapshot might succeed without issues, since
chunks are only flagged as bad if the chunk blob object could not be
decoded or content verification failed.
To fix this, discriminate between transient, permanent and task
runtime errors and account for these separately. For snapshots
encountering only transient errors, but no permanent errors during
verification, do not set or strip the verify okay state, but always
keep a pre-existing verification failed state. This is to never allow
reuse of a snapshot as reference for backup/sync jobs, when the state
was bad before and could not be verified again. For the other case
there is not much to lose, if it was okay or not verified before
falling back to not verified reflects the current snapshot state.
Avoid manifest updates which are not needed, since this causes
additional put requests to S3 backends and might fail as well.
Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7606
Changes since v3:
- Try falling back to Retry-After header if x-amz-retry-after is not present
- Double the base delay, in order to keep them closer to previous retry
delays used by the s3-client
- fix stray newline
Changes since v2 (thanks @Fabian for review):
- move object content fetching retry logic into proxmox-s3-client
helper and use it across all callsites in proxmox-backup
- extend proxmox-s3-client retry handling with some additional
improvements:
- also retry on `502 Bad Gateway` responses
- follow AWS SDK standard retry backoff delays, distinguishing
transient errors and throttled requests
- detect throttling error codes from S3 XML error response bodies
- add uniform jitter to retry backoff delays and honor
`x-amz-retry-after` response headers
- add backoff delay between object upload retries
- distinguish task abort/shutdown from transient and permanent
verification errors, leaving the snapshot verify state untouched
for these errors
- do not swallow any previous permanent errors when chunk listing,
reading or decoding has errors as well
- instead of stopping after the first transient fetch error, only stop
verifying the current snapshot once 3 consecutive S3 chunk fetches
fail, resetting the counter on successful fetches and per snapshot
Changes since v1:
- fix off-by-one error in retry count check, only up to 2 retries are
being performed.
- refactor manifest verify state parsing error handling, it cannot
propagate an error as that is interpreted as task abort. try to
update the manifest's verify state instead and log the parsing
issue.
- early stop verification on transient errors for datastores backed
by S3, as this causes additional requests, costs and runtime at no
benefit.
- improve on logging and code style in general
proxmox:
Christian Ebner (7):
s3-client: also retry requests on bad gateway responses
s3-client: distinguish transient errors and throttled requests
s3-client: detect throttling request from the response body on error
s3-client: avoid retry bursts by adding jitter to retry backoff delay
s3-client: honor response header directed retry delays
s3-client: add additional backoff delay between upload retries
s3-client: add helper to fetch object contents with retry
proxmox-s3-client/src/client.rs | 336 ++++++++++++++++++++++-
proxmox-s3-client/src/response_reader.rs | 36 +++
proxmox-s3-client/src/timestamps.rs | 7 +
3 files changed, 366 insertions(+), 13 deletions(-)
proxmox-backup:
Christian Ebner (4):
s3: retry fetching object contents on body collection errors
datastore: snapshot: propagate errors in manifest update callback
fix #7606: verify: s3: handle transient chunk load errors
verify: s3: stop early if the endpoint appears unreachable
pbs-datastore/Cargo.toml | 1 -
pbs-datastore/src/backup_info.rs | 4 +-
pbs-datastore/src/datastore.rs | 13 +-
pbs-datastore/src/local_chunk_reader.rs | 4 +-
.../src/local_datastore_lru_cache.rs | 6 +-
src/api2/config/datastore.rs | 13 +-
src/api2/reader/mod.rs | 4 +-
src/backup/verify.rs | 364 ++++++++++++++----
8 files changed, 297 insertions(+), 112 deletions(-)
Summary over all repositories:
11 files changed, 663 insertions(+), 125 deletions(-)
--
Generated by murpp 0.11.0
next reply other threads:[~2026-11-13 14:22 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 14:21 Christian Ebner [this message]
2026-11-13 14:21 ` [PATCH proxmox v4 01/11] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 02/11] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 03/11] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 05/11] s3-client: honor response header directed retry delays Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 06/11] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 07/11] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 08/11] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 09/11] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 10/11] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 11/11] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
2026-11-16 9:55 ` superseded: [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113142148.569471-1-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.