[PATCH proxmox-backup v2 5/5] verify: s3: early exit on chunk fetch or verification errors

[Christian Ebner <c.ebner@proxmox.com>]

If verification fails due to a transient error on S3 backed
datastores, it is better to skip any further verification on the
same snapshot to save useless API requests and runtime, the final
verification state is already known anyway. This further avoids
blocking for a very long time if the S3 API endpoint is not
reachable.

For permanent error continue, since any bad chunk is best marked
as such and must then be uploaded again, so it is best to check all
of them.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/backup/verify.rs | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index edfb52a5d..15293403b 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -12,9 +12,9 @@ use proxmox_parallel_handler::{ParallelHandler, SendHandle};
 use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
-    ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, PRIV_DATASTORE_BACKUP,
-    PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState, print_ns_and_snapshot,
-    print_store_and_ns,
+    ArchiveType, Authid, BackupNamespace, BackupType, CryptMode, DatastoreBackendType,
+    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY, SnapshotVerifyState, UPID, VerifyState,
+    print_ns_and_snapshot, print_store_and_ns,
 };
 use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
 use pbs_datastore::index::{ChunkReadInfo, IndexFile};
@@ -233,6 +233,15 @@ impl VerifyWorker {
                 .fail_on_shutdown()
                 .map_err(VerificationError::Transient)?;
 
+            if self.datastore.backend_type() == DatastoreBackendType::S3
+                && verify_state.fetch_errors.load(Ordering::SeqCst) > 0
+            {
+                // early abort on transient errors for S3, avoiding useless requests and
+                // protect against stalling if S3 endpoint cannot be reached.
+                warn!("verification failed because of transient errors");
+                break;
+            }
+
             let info = index.chunk_info(pos).unwrap();
 
             // we must always recheck this here, the parallel worker below alter it!
@@ -579,6 +588,11 @@ impl VerifyWorker {
                         "verify {datastore_name}:{backup_dir_name}/{file_name} failed with transient error: {err}",
                         file_name = info.filename,
                     );
+
+                    if self.datastore.backend_type() == DatastoreBackendType::S3 {
+                        warn!("early aborting verification for {datastore_name}:{backup_dir_name}");
+                        break;
+                    }
                 }
             }
         }
-- 
2.47.3