From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 50EA81FF09F for ; Thu, 03 Sep 2026 14:16:16 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 22935215AC; Thu, 03 Sep 2026 14:16:15 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 0/2] fix #7606: verify: s3: handle transient fetch errors and add retry Date: Thu, 3 Sep 2026 14:15:55 +0200 Message-ID: <20260903121557.420018-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1788437768706 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.680 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: VUNLSGSRL7C3JR5GUXSCTDNSMCNW6LWP X-Message-ID-Hash: VUNLSGSRL7C3JR5GUXSCTDNSMCNW6LWP X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Transient errors during fetching of object from s3 backend currently lead to snapshots being marked as verify failed. Subsequent runs of verification on the same snapshot might succeed without issues, since chunks are only flagged as bad if the chunk blob object could not be decoded or content verification failed. To fix this, discriminate between transient errors and permanent errors and account for these separately. For snapshots encountering only transient errors, but no permanent errors during verification, do not set or strip the verify okay state, but always keep a pre-existing verification failed state. This is to never allow reuse of a snapshot as reference for backup/sync jobs, when the state was bad before and could not be verified again. For the other case there is not much to loose, if it was okay or not verified before falling back to not verified reflects the current snapshot state. Link to the bugtracker issue: https://bugzilla.proxmox.com/show_bug.cgi?id=7606 proxmox-backup: Christian Ebner (2): fix #7606: verify: s3: handle transient chunk load errors verify: s3: retry on transient response body collection errors src/backup/verify.rs | 235 ++++++++++++++++++++++++++++++------------- 1 file changed, 165 insertions(+), 70 deletions(-) Summary over all repositories: 1 files changed, 165 insertions(+), 70 deletions(-) -- Generated by murpp 0.11.0