[PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers

[Shan Shaji <s.shaji@proxmox.com>]

This series depends on the following series: [1]

Reusing an S3-backed datastore currently restores the backup metadata to
the local cache without creating local chunk markers. A garbage collection
run is then required to recreate markers for chunks referenced by the backup
indexes.

To fix this, add an optional 'rebuild-chunk-markers' option during datastore
creation to recreate the markers. If the number of chunks is large,
datastore creation together with marker file creation could take a significant
amount of time. Therefore, the option is disabled by default. When I
tested the changes on cloudflare R2 storage, it took 3m 27.1s to create
352670 markers.

Changes since RFC [2]: Thanks @chris for the feedback and offlist
                       clarifications.

- Remove rebuild-chunk-markers option from the s3-refresh command.
- The rebuild-chunk-markers option is now exclusive to datastore creation. Exposed
  the option in the UI and backup-manager.
- Use 'impl Trait' instead of generics.
- Made the chunkstore dependency on the pagination helper explicit.
- Add a comment explaining why holding the chunk store is acceptable.
- Add progress logs with throttling.
- Move the LogThrottle implementation to the proxmox-log crate.

- [1] https://lore.proxmox.com/pbs-devel/20261014071715.88590-1-c.ebner@proxmox.com/T/
- [2] https://lore.proxmox.com/pbs-devel/2d665615-ed2e-4b82-b166-85d0f5eb41d5@proxmox.com/T/#t

proxmox-backup:

Shan Shaji (8):
  refactor: remove local `LogThrottle` in favor of proxmox-log
  datastore: factor out s3 chunk objects pagination into a helper
  pbs-datastore: inherit proxmox-log crate from workspace
  datastore: add method to create zero-byte chunk markers
  fix #7908: datastore: api: add option to create zero-byte chunk
    markers
  fix #7908: datastore: ui: add option to create zero-byte chunk markers
  fix #7908: backup-manager: add option to create zero-byte chunk
    markers
  docs: add explanation regarding ``rebuild-chunk-markers`` option

 docs/storage.rst                            |   7 +-
 pbs-client/src/lib.rs                       |   3 -
 pbs-client/src/log_throttle.rs              |  36 -----
 pbs-datastore/Cargo.toml                    |   1 +
 pbs-datastore/src/datastore.rs              | 143 +++++++++++++++-----
 proxmox-backup-client/src/main.rs           |   3 +-
 src/api2/config/datastore.rs                |  15 +-
 src/api2/node/disks/directory.rs            |   1 +
 src/api2/node/disks/zfs.rs                  |   1 +
 src/bin/proxmox_backup_manager/datastore.rs |   6 +
 www/window/DataStoreEdit.js                 |  20 +++
 11 files changed, 158 insertions(+), 78 deletions(-)
 delete mode 100644 pbs-client/src/log_throttle.rs


proxmox:

Shan Shaji (1):
  proxmox-log: add LogThrottle implementation

 proxmox-log/src/lib.rs          |  1 +
 proxmox-log/src/log_throttle.rs | 39 +++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 proxmox-log/src/log_throttle.rs


Summary over all repositories:
  13 files changed, 198 insertions(+), 78 deletions(-)

--
Generated by murpp 0.10.0