From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 6DB271FF0B4 for ; Tue, 10 Nov 2026 16:38:03 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 38CAD2167B; Tue, 10 Nov 2026 16:37:59 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Date: Tue, 10 Nov 2026 16:37:26 +0100 Message-ID: <20261110153731.449671-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794325073988 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.425 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ZGL7USFQ3BNBJGO7AT5YINTMHG54FDLY X-Message-ID-Hash: ZGL7USFQ3BNBJGO7AT5YINTMHG54FDLY X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Transient errors during fetching of object from S3 backend currently lead to snapshots being marked as verify failed. Subsequent runs of verification on the same snapshot might succeed without issues, since chunks are only flagged as bad if the chunk blob object could not be decoded or content verification failed. To fix this, discriminate between transient errors and permanent errors and account for these separately. For snapshots encountering only transient errors, but no permanent errors during verification, do not set or strip the verify okay state, but always keep a pre-existing verification failed state. This is to never allow reuse of a snapshot as reference for backup/sync jobs, when the state was bad before and could not be verified again. For the other case there is not much to lose, if it was okay or not verified before falling back to not verified reflects the current snapshot state. Link to the bugtracker issue: https://bugzilla.proxmox.com/show_bug.cgi?id=7606 Changes since v1: - fix off-by-one error in retry count check, only up to 2 retries are being performed. - refactor manifest verify state parsing error handling, it cannot propagate an error as that is interpreted as task abort. try to update the manifest's verify state instead and log the parsing issue. - early stop verification on transient errors for datastores backed by S3, as this causes additional requests, costs and runtime at no benefit. - improve on logging and code style in general proxmox-backup: Christian Ebner (5): verify: drop useless try_block! in snapshot verification datastore: snapshot: propagate errors in manifest update callback fix #7606: verify: s3: handle transient chunk load errors verify: s3: retry on transient response body collection errors verify: s3: early exit on chunk fetch or verification errors pbs-datastore/src/backup_info.rs | 4 +- pbs-datastore/src/datastore.rs | 1 + src/backup/verify.rs | 321 ++++++++++++++++++++++--------- 3 files changed, 230 insertions(+), 96 deletions(-) Summary over all repositories: 3 files changed, 230 insertions(+), 96 deletions(-) -- Generated by murpp 0.11.0