all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox-backup v4 11/11] verify: s3: stop early if the endpoint appears unreachable
Date: Fri, 13 Nov 2026 15:21:48 +0100	[thread overview]
Message-ID: <20261113142148.569471-12-c.ebner@proxmox.com> (raw)
In-Reply-To: <20261113142148.569471-1-c.ebner@proxmox.com>

If the S3 endpoint cannot be reached, every remaining chunk fetch of
a snapshot fails only after the request retries and their backoff.
This causes useless requests and can block the verification for a
very long time, while the outcome is already known, as the snapshot
cannot be marked as verified anyway.

Stopping on the first transient error is not an option though: a
snapshot without verify state can still be used as reference by the
next backup, so corrupt chunks which would be detected later in the
same run must not be missed because of a single failed request.

Therefore, only stop verifying the chunks and the remaining archives
of a snapshot once a few chunk fetches failed in a row, with any
successful fetch resetting the count. Errors detected up to that
point are still accounted for, so known bad snapshots are marked as
failed. The count is reset for each snapshot, so verification
continues normally once the endpoint becomes reachable again.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/backup/verify.rs | 64 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 624c9a9aa..c29e60678 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -22,6 +22,10 @@ use pbs_datastore::{DataBlob, DataStore, DatastoreBackend, StoreProgress};
 
 use crate::backup::hierarchy::ListAccessibleBackupGroups;
 
+/// Number of chunk fetches from the S3 backend failing in a row, after which the endpoint is
+/// considered unreachable and verification of the current snapshot is stopped.
+const S3_MAX_CONSECUTIVE_FETCH_ERRORS: usize = 3;
+
 /// A VerifyWorker encapsulates a task worker, datastore and information about which chunks have
 /// already been verified or detected as corrupt.
 pub struct VerifyWorker {
@@ -32,6 +36,7 @@ pub struct VerifyWorker {
     backend: DatastoreBackend,
     read_threads: usize,
     verify_threads: usize,
+    consecutive_fetch_errors: Arc<AtomicUsize>,
 }
 
 struct IndexVerifyState {
@@ -43,6 +48,7 @@ struct IndexVerifyState {
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     start_time: Instant,
+    consecutive_fetch_errors: Arc<AtomicUsize>,
 }
 
 impl IndexVerifyState {
@@ -50,6 +56,7 @@ impl IndexVerifyState {
         datastore: &Arc<DataStore>,
         corrupt_chunks: &Arc<Mutex<HashSet<[u8; 32]>>>,
         verified_chunks: &Arc<Mutex<HashSet<[u8; 32]>>>,
+        consecutive_fetch_errors: &Arc<AtomicUsize>,
     ) -> Self {
         Self {
             read_bytes: AtomicU64::new(0),
@@ -60,6 +67,7 @@ impl IndexVerifyState {
             corrupt_chunks: Arc::clone(corrupt_chunks),
             verified_chunks: Arc::clone(verified_chunks),
             start_time: Instant::now(),
+            consecutive_fetch_errors: Arc::clone(consecutive_fetch_errors),
         }
     }
 }
@@ -104,9 +112,17 @@ impl VerifyWorker {
             backend,
             read_threads,
             verify_threads,
+            consecutive_fetch_errors: Arc::new(AtomicUsize::new(0)),
         })
     }
 
+    /// Check if fetching chunks from the S3 backend failed too often in a row. A single transient
+    /// error must not stop the verification, as that could hide corrupt chunks which would then
+    /// be reused by subsequent backups.
+    fn s3_endpoint_unreachable(&self) -> bool {
+        self.consecutive_fetch_errors.load(Ordering::SeqCst) >= S3_MAX_CONSECUTIVE_FETCH_ERRORS
+    }
+
     fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), VerificationError> {
         let blob = backup_dir
             .load_blob(info.filename.as_ref())
@@ -151,6 +167,7 @@ impl VerifyWorker {
             &self.datastore,
             &self.corrupt_chunks,
             &self.verified_chunks,
+            &self.consecutive_fetch_errors,
         ));
 
         let decoder_pool = ParallelHandler::new("verify chunk decoder", self.verify_threads, {
@@ -248,12 +265,20 @@ impl VerifyWorker {
         });
 
         let mut chunk_send_error: Option<Error> = None;
+        let mut exhausted_error: Option<Error> = None;
         for (pos, _) in chunk_list {
             self.worker.check_abort().map_err(VerificationError::Task)?;
             self.worker
                 .fail_on_shutdown()
                 .map_err(VerificationError::Task)?;
 
+            if self.s3_endpoint_unreachable() {
+                // avoid further requests and long stalls, the archive cannot be verified anyway
+                warn!("stop verifying chunks, too many consecutive fetch errors");
+                exhausted_error = Some(format_err!("S3 endpoint appears to be unreachable"));
+                break;
+            }
+
             let info = index.chunk_info(pos).unwrap();
 
             // we must always recheck this here, the parallel worker below alter it!
@@ -271,6 +296,11 @@ impl VerifyWorker {
         let reader_result = reader_pool.complete();
         let decoder_result = decoder_pool.complete();
 
+        if exhausted_error.is_none() && self.s3_endpoint_unreachable() {
+            warn!("stop verifying chunks, too many consecutive fetch errors");
+            exhausted_error = Some(format_err!("S3 endpoint appears to be unreachable"));
+        }
+
         let elapsed = verify_state.start_time.elapsed().as_secs_f64();
 
         let read_bytes = verify_state.read_bytes.load(Ordering::SeqCst);
@@ -301,6 +331,12 @@ impl VerifyWorker {
             ));
         }
 
+        if let Some(err) = exhausted_error {
+            return Err(VerificationError::Transient(
+                err.context("incomplete verification"),
+            ));
+        }
+
         if let Some(err) = chunk_send_error {
             return Err(VerificationError::Transient(
                 err.context("incomplete verification because of chunk send error"),
@@ -351,8 +387,21 @@ impl VerifyWorker {
                 }
             },
             DatastoreBackend::S3(s3_client) => {
+                if verify_state.consecutive_fetch_errors.load(Ordering::SeqCst)
+                    >= S3_MAX_CONSECUTIVE_FETCH_ERRORS
+                {
+                    verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
+                    return Ok(());
+                }
                 let object_key = pbs_datastore::s3::object_key_from_digest(&info.digest)?;
-                match proxmox_async::runtime::block_on(s3_client.get_object_content(object_key)) {
+                let result =
+                    proxmox_async::runtime::block_on(s3_client.get_object_content(object_key));
+                if result.is_ok() {
+                    verify_state
+                        .consecutive_fetch_errors
+                        .store(0, Ordering::SeqCst);
+                }
+                match result {
                     Ok(Some(raw_chunk)) => match DataBlob::from_raw(raw_chunk.to_vec()) {
                         Ok(chunk) => {
                             let size = info.size();
@@ -381,6 +430,9 @@ impl VerifyWorker {
                     ),
                     Err(err) => {
                         verify_state.fetch_errors.fetch_add(1, Ordering::SeqCst);
+                        verify_state
+                            .consecutive_fetch_errors
+                            .fetch_add(1, Ordering::SeqCst);
                         error!("can't verify chunk, load failed - {err}");
                     }
                 }
@@ -532,6 +584,9 @@ impl VerifyWorker {
 
         info!("verify {datastore_name}:{backup_dir_name}");
 
+        // probe the endpoint again for each snapshot, it might have become reachable again
+        self.consecutive_fetch_errors.store(0, Ordering::SeqCst);
+
         let mut error_count = 0;
 
         let mut verify_result = Some(VerifyState::Ok);
@@ -570,6 +625,13 @@ impl VerifyWorker {
                         "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
                         file_name = info.filename,
                     );
+
+                    if self.s3_endpoint_unreachable() {
+                        warn!(
+                            "skip verifying remaining archives of {datastore_name}:{backup_dir_name}"
+                        );
+                        break;
+                    }
                 }
             }
         }
-- 
2.47.3





  parent reply	other threads:[~2026-11-13 14:22 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13 14:21 [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 01/11] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 02/11] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 03/11] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 05/11] s3-client: honor response header directed retry delays Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 06/11] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 07/11] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 08/11] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 09/11] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 10/11] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-13 14:21 ` Christian Ebner [this message]
2026-11-16  9:55 ` superseded: [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113142148.569471-12-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal