public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors
Date: Thu, 12 Nov 2026 10:41:44 +0100	[thread overview]
Message-ID: <1794476408.5yedmoj0k4.astroid@yuna.none> (raw)
In-Reply-To: <20261110153731.449671-6-c.ebner@proxmox.com>

On November 10, 2026 4:37 pm, Christian Ebner wrote:
> If verification fails due to a transient error on S3 backed
> datastores, it is better to skip any further verification on the
> same snapshot to save useless API requests and runtime, the final
> verification state is already known anyway. This further avoids
> blocking for a very long time if the S3 API endpoint is not
> reachable.
> 
> For permanent error continue, since any bad chunk is best marked
> as such and must then be uploaded again, so it is best to check all
> of them.

should we guard the early abort with a check for the absence of
permanent errors? or maybe detect X fetch attempts failing in a row?

without such logic, a single transient error early aborting might mask a
lot of corrupt chunks..

> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  src/backup/verify.rs | 20 +++++++++++++++++---
>  1 file changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index edfb52a5d..15293403b 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHandle};
>  use proxmox_worker_task::WorkerTaskContext;
>  
>  use pbs_api_types::{
> -    ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DATASTORE_BACKUP,
> -    PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print_ns_and_snapshot,
> -    print_store_and_ns,
> +    ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, DatastoreBackendType,
> +    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState,
> +    print_ns_and_snapshot, print_store_and_ns,
>  };
>  use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
>  use pbs_datastore::index::{ChunkReadInfo, IndexFile};
> @@ -233,6 +233,15 @@ impl VerifyWorker {
>                  .fail_on_shutdown()
>                  .map_err(VerificationError::Transient)?;
>  
> +            if self.datastore.backend_type() == DatastoreBackendType::S3
> +                && verify_state.fetch_errors.load(Ordering::SeqCst) > 0
> +            {
> +                // early abort on transient errors for S3, avoiding useless requests and
> +                // protect against stalling if S3 endpoint cannot be reached.
> +                warn!("verification failed because of transient errors");
> +                break;
> +            }
> +
>              let info = index.chunk_info(pos).unwrap();
>  
>              // we must always recheck this here, the parallel worker below alter it!
> @@ -579,6 +588,11 @@ impl VerifyWorker {
>                          "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
>                          file_name = info.filename,
>                      );
> +
> +                    if self.datastore.backend_type() == DatastoreBackendType::S3 {
> +                        warn!("early aborting verification for {datastore_name}:{backup_dir_name}");
> +                        break;
> +                    }
>                  }
>              }
>          }
> -- 
> 2.47.3
> 
> 
> 
> 
> 
> 




  reply	other threads:[~2026-11-12  9:41 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-10 15:37 [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 1/5] verify: drop useless try_block! in snapshot verification Christian Ebner
2026-11-11  0:29   ` applied: " Thomas Lamprecht
2026-11-10 15:37 ` [PATCH proxmox-backup v2 2/5] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 3/5] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-12  9:39   ` Fabian Grünbichler
2026-11-12 10:06     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 4/5] verify: s3: retry on transient response body collection errors Christian Ebner
2026-11-12  9:48   ` Fabian Grünbichler
2026-11-12 10:09     ` Christian Ebner
2026-11-10 15:37 ` [PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors Christian Ebner
2026-11-12  9:41   ` Fabian Grünbichler [this message]
2026-11-12 18:03 ` superseded: [PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1794476408.5yedmoj0k4.astroid@yuna.none \
    --to=f.gruenbichler@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal