From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 0A0E81FF0BC for ; Mon, 16 Nov 2026 10:55:05 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id F0D8321745; Mon, 16 Nov 2026 10:54:51 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Date: Mon, 16 Nov 2026 10:54:22 +0100 Message-ID: <20261116095434.110541-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794822885031 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.385 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 635NSYVWIQDVRZ6VTLF45YYQD7NAWR6R X-Message-ID-Hash: 635NSYVWIQDVRZ6VTLF45YYQD7NAWR6R X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Transient errors during fetching of objects from S3 backend currently lead to snapshots being marked as verify failed. Subsequent runs of verification on the same snapshot might succeed without issues, since chunks are only flagged as bad if the chunk blob object could not be decoded or content verification failed. To fix this, discriminate between transient, permanent and task runtime errors and account for these separately. For snapshots encountering only transient errors, but no permanent errors during verification, do not set or strip the verify okay state, but always keep a pre-existing verification failed state. This is to never allow reuse of a snapshot as reference for backup/sync jobs, when the state was bad before and could not be verified again. For the other case there is not much to lose, if it was okay or not verified before falling back to not verified reflects the current snapshot state. Avoid manifest updates which are not needed, since this causes additional put requests to S3 backends and might fail as well. Link to the bugtracker issue: https://bugzilla.proxmox.com/show_bug.cgi?id=7606 Changes since v4: - Add retry quota implementation and apply potential slowdowns also to new requests Changes since v3: - Try falling back to Retry-After header if x-amz-retry-after is not present - Double the base delay, in order to keep them closer to previous retry delays used by the s3-client - fix stray newline Changes since v2 (thanks @Fabian for review): - move object content fetching retry logic into proxmox-s3-client helper and use it across all callsites in proxmox-backup - extend proxmox-s3-client retry handling with some additional improvements: - also retry on `502 Bad Gateway` responses - follow AWS SDK standard retry backoff delays, distinguishing transient errors and throttled requests - detect throttling error codes from S3 XML error response bodies - add uniform jitter to retry backoff delays and honor `x-amz-retry-after` response headers - add backoff delay between object upload retries - distinguish task abort/shutdown from transient and permanent verification errors, leaving the snapshot verify state untouched for these errors - do not swallow any previous permanent errors when chunk listing, reading or decoding has errors as well - instead of stopping after the first transient fetch error, only stop verifying the current snapshot once 3 consecutive S3 chunk fetches fail, resetting the counter on successful fetches and per snapshot Changes since v1: - fix off-by-one error in retry count check, only up to 2 retries are being performed. - refactor manifest verify state parsing error handling, it cannot propagate an error as that is interpreted as task abort. try to update the manifest's verify state instead and log the parsing issue. - early stop verification on transient errors for datastores backed by S3, as this causes additional requests, costs and runtime at no benefit. - improve on logging and code style in general proxmox: Christian Ebner (8): s3-client: also retry requests on bad gateway responses s3-client: distinguish transient errors and throttled requests s3-client: detect throttling request from the response body on error s3-client: avoid retry bursts by adding jitter to retry backoff delay s3-client: honor response header directed retry delays s3-client: add additional backoff delay between upload retries s3-client: add helper to fetch object contents with retry s3-client: add shared rate limiter based retry quota proxmox-s3-client/src/client.rs | 384 ++++++++++++++++++++++- proxmox-s3-client/src/response_reader.rs | 36 +++ proxmox-s3-client/src/timestamps.rs | 7 + 3 files changed, 413 insertions(+), 14 deletions(-) proxmox-backup: Christian Ebner (4): s3: retry fetching object contents on body collection errors datastore: snapshot: propagate errors in manifest update callback fix #7606: verify: s3: handle transient chunk load errors verify: s3: stop early if the endpoint appears unreachable pbs-datastore/Cargo.toml | 1 - pbs-datastore/src/backup_info.rs | 4 +- pbs-datastore/src/datastore.rs | 13 +- pbs-datastore/src/local_chunk_reader.rs | 4 +- .../src/local_datastore_lru_cache.rs | 6 +- src/api2/config/datastore.rs | 13 +- src/api2/reader/mod.rs | 4 +- src/backup/verify.rs | 364 ++++++++++++++---- 8 files changed, 297 insertions(+), 112 deletions(-) Summary over all repositories: 11 files changed, 710 insertions(+), 126 deletions(-) -- Generated by murpp 0.11.0