From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors
Date: Thu, 12 Nov 2026 10:41:44 +0100 [thread overview]
Message-ID: <1794476408.5yedmoj0k4.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-6-c.ebner@proxmox.com>
On November 10, 2026 4:37 pm, Christian Ebner wrote:
> If verification fails due to a transient error on S3 backed
> datastores, it is better to skip any further verification on the
> same snapshot to save useless API requests and runtime, the final
> verification state is already known anyway. This further avoids
> blocking for a very long time if the S3 API endpoint is not
> reachable.
>
> For permanent error continue, since any bad chunk is best marked
> as such and must then be uploaded again, so it is best to check all
> of them.
should we guard the early abort with a check for the absence of
permanent errors? or maybe detect X fetch attempts failing in a row?
without such logic, a single transient error early aborting might mask a
lot of corrupt chunks..
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> src/backup/verify.rs | 20 +++++++++++++++++---
> 1 file changed, 17 insertions(+), 3 deletions(-)
>
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index edfb52a5d..15293403b 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHandle};
> use proxmox_worker_task::WorkerTaskContext;
>
> use pbs_api_types::{
> - ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DATASTORE_BACKUP,
> - PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print_ns_and_snapshot,
> - print_store_and_ns,
> + ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, DatastoreBackendType,
> + PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState,
> + print_ns_and_snapshot, print_store_and_ns,
> };
> use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
> use pbs_datastore::index::{ChunkReadInfo, IndexFile};
> @@ -233,6 +233,15 @@ impl VerifyWorker {
> .fail_on_shutdown()
> .map_err(VerificationError::Transient)?;
>
> + if self.datastore.backend_type() == DatastoreBackendType::S3
> + && verify_state.fetch_errors.load(Ordering::SeqCst) > 0
> + {
> + // early abort on transient errors for S3, avoiding useless requests and
> + // protect against stalling if S3 endpoint cannot be reached.
> + warn!("verification failed because of transient errors");
> + break;
> + }
> +
> let info = index.chunk_info(pos).unwrap();
>
> // we must always recheck this here, the parallel worker below alter it!
> @@ -579,6 +588,11 @@ impl VerifyWorker {
> "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
> file_name = info.filename,
> );
> +
> + if self.datastore.backend_type() == DatastoreBackendType::S3 {
> + warn!("early aborting verification for {datastore_name}:{backup_dir_name}");
> + break;
> + }
> }
> }
> }
> --
> 2.47.3
>
>
>
>
>
>
next prev parent reply other threads:[~2026-11-12 9:41 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11 0:29 ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12 9:39 ` Fabian Grünbichler
2026-11-12 10:06 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12 9:48 ` Fabian Grünbichler
2026-11-12 10:09 ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12 9:41 ` Fabian Grünbichler [this message]
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1794476408.5yedmoj0k4.astroid@yuna.none \
--to=f.gruenbichler@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.