[RFC PATCH proxmox-backup 0/4] fix #7903: rebuild local S3 chunk markers

[Shan Shaji <s.shaji@proxmox.com>]

Sending this as an RFC to get an initial feedback. I'd really appreciate
any suggestions or pointers to anything I may have missed.

Reusing an S3-backed datastore currently restores the backup metadata to
the local cache without creating local chunk markers. A garbage collection
run is then needed to recreate markers for chunks referenced by the backup
indexes.

This series addresses #7908 by listing the chunk objects in S3 and creating
zero-byte local markers when a datastore is recreated with
--reuse-datastore. It also adds an optional rebuild-chunk-markers parameter
to the s3-refresh API and exposes it through proxmox-backup-manager:

    proxmox-backup-manager datastore s3-refresh <store> --rebuild-chunk-markers true

The option is disabled by default for explicit refreshes. When enabled,
it creates markers for all chunk objects with valid chunk keys, including
chunks not referenced by the restored backup indexes. Existing cached
chunk files are replaced with empty markers.

Shan Shaji (4):
  datastore: factor out s3 chunk objects pagination into a helper
  fix #7908: datastore: create zero-byte markers when re-using datastore
  datastore: api: add option to rebuild S3 chunk markers
  datastore: backup-manager: expose option to rebuild s3 chunk markers

 pbs-datastore/src/datastore.rs              | 120 +++++++++++++++-----
 src/api2/admin/datastore.rs                 |  23 +++-
 src/api2/config/datastore.rs                |   6 +-
 src/bin/proxmox_backup_manager/datastore.rs |   7 ++
 4 files changed, 121 insertions(+), 35 deletions(-)

--
2.47.3