From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors
Date: Thu, 12 Nov 2026 10:41:44 +0100 [thread overview]
Message-ID: <1794476408.5yedmoj0k4.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-6-c.ebner@proxmox.com>
On November 10, 2026 4:37 pm, Christian Ebner wrote:
> If verification fails due to a transient error on S3 backed
> datastores, it is better to skip any further verification on the
> same snapshot to save useless API requests and runtime, the final
> verification state is already known anyway. This further avoids
> blocking for a very long time if the S3 API endpoint is not
> reachable.
>
> For permanent error continue, since any bad chunk is best marked
> as such and must then be uploaded again, so it is best to check all
> of them.
should we guard the early abort with a check for the absence of
permanent errors? or maybe detect X fetch attempts failing in a row?
without such logic, a single transient error early aborting might mask a
lot of corrupt chunks..
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> src/backup/verify.rs | 20 +++++++++++++++++---
> 1 file changed, 17 insertions(+), 3 deletions(-)
>
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index edfb52a5d..15293403b 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHandle};
> use proxmox_worker_task::WorkerTaskContext;
>
> use pbs_api_types::{
> - ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DATASTORE_BACKUP,
> - PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print_ns_and_snapshot,
> - print_store_and_ns,
> + ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, DatastoreBackendType,
> + PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState,
> + print_ns_and_snapshot, print_store_and_ns,
> };
> use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
> use pbs_datastore::index::{ChunkReadInfo, IndexFile};
> @@ -233,6 +233,15 @@ impl VerifyWorker {
> .fail_on_shutdown()
> .map_err(VerificationError::Transient)?;
>
> + if self.datastore.backend_type() == DatastoreBackendType::S3
> + && verify_state.fetch_errors.load(Ordering::SeqCst) > 0
> + {
> + // early abort on transient errors for S3, avoiding useless requests and
> + // protect against stalling if S3 endpoint cannot be reached.
> + warn!("verification failed because of transient errors");
> + break;
> + }
> +
> let info = index.chunk_info(pos).unwrap();
>
> // we must always recheck this here, the parallel worker below alter it!
> @@ -579,6 +588,11 @@ impl VerifyWorker {
> "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
> file_name = info.filename,
> );
> +
> + if self.datastore.backend_type() == DatastoreBackendType::S3 {
> + warn!("early aborting verification for {datastore_name}:{backup_dir_name}");
> + break;
> + }
> }
> }
> }
> --
> 2.47.3
>
>
>
>
>
>
next prev parent reply other threads:[~2026-11-12 9:41 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler
2026-11-12 10:09 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler [this message]
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1794476408.5yedmoj0k4.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox