From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A378F1FF0B6 for ; Thu, 12 Nov 2026 10:41:51 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 75A5A21649; Thu, 12 Nov 2026 10:41:51 +0100 (CET) Date: Thu, 12 Nov 2026 10:41:44 +0100 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors To: Christian Ebner , pbs-devel@lists.proxmox.com References: <20261110153731.449671-1-c.ebner@proxmox.com> <20261110153731.449671-6-c.ebner@proxmox.com> In-Reply-To: <20261110153731.449671-6-c.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1794476408.5yedmoj0k4.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794476507744 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.532 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5AQYMIPRG3IEA2GRSRJ3Q4FPRRCKSLUE X-Message-ID-Hash: 5AQYMIPRG3IEA2GRSRJ3Q4FPRRCKSLUE X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On November 10, 2026 4:37 pm, Christian Ebner wrote: > If verification fails due to a transient error on S3 backed > datastores, it is better to skip any further verification on the > same snapshot to save useless API requests and runtime, the final > verification state is already known anyway. This further avoids > blocking for a very long time if the S3 API endpoint is not > reachable. >=20 > For permanent error continue, since any bad chunk is best marked > as such and must then be uploaded again, so it is best to check all > of them. should we guard the early abort with a check for the absence of permanent errors? or maybe detect X fetch attempts failing in a row? without such logic, a single transient error early aborting might mask a lot of corrupt chunks.. >=20 > Signed-off-by: Christian Ebner > --- > src/backup/verify.rs | 20 +++++++++++++++++--- > 1 file changed, 17 insertions(+), 3 deletions(-) >=20 > diff --git a/src/backup/verify.rs b/src/backup/verify.rs > index edfb52a5d..15293403b 100644 > --- a/src/backup/verify.rs > +++ b/src/backup/verify.rs > @@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHan= dle}; > use proxmox_worker_task::WorkerTaskContext; > =20 > use pbs_api_types::{ > - ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DA= TASTORE_BACKUP, > - PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print= _ns_and_snapshot, > - print_store_and_ns, > + ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, Datasto= reBackendType, > + PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, U= PID, VerifyState, > + print_ns_and_snapshot, print_store_and_ns, > }; > use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo}; > use pbs_datastore::index::{ChunkReadInfo, IndexFile}; > @@ -233,6 +233,15 @@ impl VerifyWorker { > .fail_on_shutdown() > .map_err(VerificationError::Transient)?; > =20 > + if self.datastore.backend_type() =3D=3D DatastoreBackendType= ::S3 > + && verify_state.fetch_errors.load(Ordering::SeqCst) > 0 > + { > + // early abort on transient errors for S3, avoiding usel= ess requests and > + // protect against stalling if S3 endpoint cannot be rea= ched. > + warn!("verification failed because of transient errors")= ; > + break; > + } > + > let info =3D index.chunk_info(pos).unwrap(); > =20 > // we must always recheck this here, the parallel worker bel= ow alter it! > @@ -579,6 +588,11 @@ impl VerifyWorker { > "verify {datastore_name}:{backup_dir_name}/{file= _name} failed with transient error: {err}", > file_name =3D info.filename, > ); > + > + if self.datastore.backend_type() =3D=3D DatastoreBac= kendType::S3 { > + warn!("early aborting verification for {datastor= e_name}:{backup_dir_name}"); > + break; > + } > } > } > } > --=20 > 2.47.3 >=20 >=20 >=20 >=20 >=20 >=20