[PATCH proxmox-backup v2 0/5] fix #7606: verify: s3: handle transient fetch errors and add retry

[Christian Ebner <c.ebner@proxmox.com>]

Transient errors during fetching of object from S3 backend currently
lead to snapshots being marked as verify failed. Subsequent runs of
verification on the same snapshot might succeed without issues, since
chunks are only flagged as bad if the chunk blob object could not be
decoded or content verification failed.

To fix this, discriminate between transient errors and permanent
errors and account for these separately. For snapshots encountering
only transient errors, but no permanent errors during verification,
do not set or strip the verify okay state, but always keep a
pre-existing verification failed state. This is to never allow reuse
of a snapshot as reference for backup/sync jobs, when the state
was bad before and could not be verified again. For the other case
there is not much to lose, if it was okay or not verified before
falling back to not verified reflects the current snapshot state.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7606

Changes since v1:
- fix off-by-one error in retry count check, only up to 2 retries are
  being performed.
- refactor manifest verify state parsing error handling, it cannot
  propagate an error as that is interpreted as task abort. try to
  update the manifest's verify state instead and log the parsing
  issue.
- early stop verification on transient errors for datastores backed
  by S3, as this causes additional requests, costs and runtime at no
  benefit.
- improve on logging and code style in general


proxmox-backup:

Christian Ebner (5):
  verify: drop useless try_block! in snapshot verification
  datastore: snapshot: propagate errors in manifest update callback
  fix #7606: verify: s3: handle transient chunk load errors
  verify: s3: retry on transient response body collection errors
  verify: s3: early exit on chunk fetch or verification errors

 pbs-datastore/src/backup_info.rs |   4 +-
 pbs-datastore/src/datastore.rs   |   1 +
 src/backup/verify.rs             | 321 ++++++++++++++++++++++---------
 3 files changed, 230 insertions(+), 96 deletions(-)


Summary over all repositories:
  3 files changed, 230 insertions(+), 96 deletions(-)

-- 
Generated by murpp 0.11.0