From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 061B21FF0B4 for ; Tue, 10 Nov 2026 16:37:59 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 48C3421649; Tue, 10 Nov 2026 16:37:58 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Date: Tue, 10 Nov 2026 16:37:31 +0100 Message-ID: <20261110153731.449671-6-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261110153731.449671-1-c.ebner@proxmox.com> References: <20261110153731.449671-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794325075126 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.427 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FTU45YNW32IZVLJ533MBVZZIBMGSSXVC X-Message-ID-Hash: FTU45YNW32IZVLJ533MBVZZIBMGSSXVC X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: If verification fails due to a transient error on S3 backed datastores, it is better to skip any further verification on the same snapshot to save useless API requests and runtime, the final verification state is already known anyway. This further avoids blocking for a very long time if the S3 API endpoint is not reachable. For permanent error continue, since any bad chunk is best marked as such and must then be uploaded again, so it is best to check all of them. Signed-off-by: Christian Ebner --- src/backup/verify.rs | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/src/backup/verify.rs b/src/backup/verify.rs index edfb52a5d..15293403b 100644 --- a/src/backup/verify.rs +++ b/src/backup/verify.rs @@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHandle}; use proxmox_worker_task::WorkerTaskContext; use pbs_api_types::{ - ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DATASTORE_BACKUP, - PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print_ns_and_snapshot, - print_store_and_ns, + ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, DatastoreBackendType, + PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, + print_ns_and_snapshot, print_store_and_ns, }; use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo}; use pbs_datastore::index::{ChunkReadInfo, IndexFile}; @@ -233,6 +233,15 @@ impl VerifyWorker { .fail_on_shutdown() .map_err(VerificationError::Transient)?; + if self.datastore.backend_type() == DatastoreBackendType::S3 + && verify_state.fetch_errors.load(Ordering::SeqCst) > 0 + { + // early abort on transient errors for S3, avoiding useless requests and + // protect against stalling if S3 endpoint cannot be reached. + warn!("verification failed because of transient errors"); + break; + } + let info = index.chunk_info(pos).unwrap(); // we must always recheck this here, the parallel worker below alter it! @@ -579,6 +588,11 @@ impl VerifyWorker { "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}", file_name = info.filename, ); + + if self.datastore.backend_type() == DatastoreBackendType::S3 { + warn!("early aborting verification for {datastore_name}:{backup_dir_name}"); + break; + } } } } -- 2.47.3