[PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers

[PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers

[Shan Shaji]

This series depends on the following series: [1]

Reusing an S3-backed datastore currently restores the backup metadata to
the local cache without creating local chunk markers. A garbage collection
run is then required to recreate markers for chunks referenced by the backup
indexes.

To fix this, add an optional 'rebuild-chunk-markers' option during datastore
creation to recreate the markers. If the number of chunks is large,
datastore creation together with marker file creation could take a significant
amount of time. Therefore, the option is disabled by default. When I
tested the changes on cloudflare R2 storage, it took 3m 27.1s to create
352670 markers.

Changes since RFC [2]: Thanks @chris for the feedback and offlist
                       clarifications.

- Remove rebuild-chunk-markers option from the s3-refresh command.
- The rebuild-chunk-markers option is now exclusive to datastore creation. Exposed
  the option in the UI and backup-manager.
- Use 'impl Trait' instead of generics.
- Made the chunkstore dependency on the pagination helper explicit.
- Add a comment explaining why holding the chunk store is acceptable.
- Add progress logs with throttling.
- Move the LogThrottle implementation to the proxmox-log crate.

- [1] https://lore.proxmox.com/pbs-devel/20261014071715.88590-1-c.ebner@proxmox.com/T/
- [2] https://lore.proxmox.com/pbs-devel/2d665615-ed2e-4b82-b166-85d0f5eb41d5@proxmox.com/T/#t

proxmox-backup:

Shan Shaji (8):
  refactor: remove local `LogThrottle` in favor of proxmox-log
  datastore: factor out s3 chunk objects pagination into a helper
  pbs-datastore: inherit proxmox-log crate from workspace
  datastore: add method to create zero-byte chunk markers
  fix #7908: datastore: api: add option to create zero-byte chunk
    markers
  fix #7908: datastore: ui: add option to create zero-byte chunk markers
  fix #7908: backup-manager: add option to create zero-byte chunk
    markers
  docs: add explanation regarding ``rebuild-chunk-markers`` option

 docs/storage.rst                            |   7 +-
 pbs-client/src/lib.rs                       |   3 -
 pbs-client/src/log_throttle.rs              |  36 -----
 pbs-datastore/Cargo.toml                    |   1 +
 pbs-datastore/src/datastore.rs              | 143 +++++++++++++++-----
 proxmox-backup-client/src/main.rs           |   3 +-
 src/api2/config/datastore.rs                |  15 +-
 src/api2/node/disks/directory.rs            |   1 +
 src/api2/node/disks/zfs.rs                  |   1 +
 src/bin/proxmox_backup_manager/datastore.rs |   6 +
 www/window/DataStoreEdit.js                 |  20 +++
 11 files changed, 158 insertions(+), 78 deletions(-)
 delete mode 100644 pbs-client/src/log_throttle.rs


proxmox:

Shan Shaji (1):
  proxmox-log: add LogThrottle implementation

 proxmox-log/src/lib.rs          |  1 +
 proxmox-log/src/log_throttle.rs | 39 +++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 proxmox-log/src/log_throttle.rs


Summary over all repositories:
  13 files changed, 198 insertions(+), 78 deletions(-)

--
Generated by murpp 0.10.0

[PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log

[Shan Shaji]

Remove the local implementation of `LogThrottle` and use the version
provided by the proxmox-log crate instead. This prepares the utility
for re-use in pbs-datastore and updates all existing references
accordingly. No functional changes.

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 pbs-client/src/lib.rs             |  3 ---
 pbs-client/src/log_throttle.rs    | 36 -------------------------------
 proxmox-backup-client/src/main.rs |  3 ++-
 3 files changed, 2 insertions(+), 40 deletions(-)
 delete mode 100644 pbs-client/src/log_throttle.rs

diff --git a/pbs-client/src/lib.rs b/pbs-client/src/lib.rs
index 139f4f4b5..4b8e4e4f4 100644
--- a/pbs-client/src/lib.rs
+++ b/pbs-client/src/lib.rs
@@ -45,7 +45,4 @@ pub use chunk_stream::{ChunkStream, FixedChunkStream, InjectionData};
 mod backup_stats;
 pub use backup_stats::BackupStats;
 
-mod log_throttle;
-pub use log_throttle::LogThrottle;
-
 pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
diff --git a/pbs-client/src/log_throttle.rs b/pbs-client/src/log_throttle.rs
deleted file mode 100644
index cfaa6c494..000000000
--- a/pbs-client/src/log_throttle.rs
+++ /dev/null
@@ -1,36 +0,0 @@
-use std::time::{Duration, Instant};
-
-/// Rate-limits recurring log output, such as progress lines, with an exponential backoff.
-///
-/// [`should_log`](Self::should_log) returns `true` at most once per the current interval. Starting
-/// from an initial interval, the delay doubles after each hit up to a configured maximum, giving
-/// frequent feedback early on while avoiding log spam during long-running operations.
-pub struct LogThrottle {
-    last: Instant,
-    interval: Duration,
-    max_interval: Duration,
-}
-
-impl LogThrottle {
-    /// Creates a throttle with the given initial interval, doubling up to `max_interval` after each
-    /// hit. The initial interval is capped at `max_interval`.
-    pub fn new(initial_interval: Duration, max_interval: Duration) -> Self {
-        Self {
-            last: Instant::now(),
-            interval: initial_interval.min(max_interval),
-            max_interval,
-        }
-    }
-
-    /// Returns `true` if the current interval has elapsed since the last hit, advancing the
-    /// throttle and doubling the interval (capped at the maximum); returns `false` otherwise.
-    pub fn should_log(&mut self) -> bool {
-        let now = Instant::now();
-        if now.duration_since(self.last) < self.interval {
-            return false;
-        }
-        self.last = now;
-        self.interval = (self.interval * 2).min(self.max_interval);
-        true
-    }
-}
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 3e7aa7cfd..93f5a1b7a 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -18,6 +18,7 @@ use pathpatterns::{MatchEntry, MatchType, PatternFlag};
 use proxmox_async::blocking::TokioWriterAdapter;
 use proxmox_human_byte::HumanByte;
 use proxmox_io::StdChannelWriter;
+use proxmox_log::log_throttle::LogThrottle;
 use proxmox_router::{ApiMethod, RpcEnvironment, cli::*};
 use proxmox_schema::api;
 use proxmox_sys::fs::{CreateOptions, file_get_json, image_size, replace_file};
@@ -51,7 +52,7 @@ use pbs_client::{
     BACKUP_SOURCE_SCHEMA, BackupDetectionMode, BackupReader, BackupRepository,
     BackupRepositoryArgs, BackupSpecificationType, BackupStats, BackupTargetArgs, BackupWriter,
     BackupWriterOptions, ChunkStream, FixedChunkStream, HttpClient, IndexType, InjectionData,
-    LogThrottle, PxarBackupStream, RemoteChunkReader, UploadOptions, delete_ticket_info,
+    PxarBackupStream, RemoteChunkReader, UploadOptions, delete_ticket_info,
     parse_backup_specification, view_task_result,
 };
 use pbs_datastore::catalog::{BackupCatalogWriter, CatalogReader, CatalogWriter};
-- 
2.47.3

[PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper

[Shan Shaji]

This is in preparation to re-use the chunk iteration logic for creating
empty marker files during s3 refresh.

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 pbs-datastore/src/datastore.rs | 93 +++++++++++++++++++++-------------
 1 file changed, 58 insertions(+), 35 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index a74a6a824..7bb0c187c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -2505,6 +2505,44 @@ impl DataStore {
         }
     }
 
+    // Fetch chunk objects in batches of max 1000 per request.
+    // For each listed object, invoke `callback` with a reference to the
+    // datastore's local `ChunkStore`, followed by the object's S3 key and size.
+    fn paginate_s3_chunk_objects(
+        &self,
+        worker: &dyn WorkerTaskContext,
+        s3_client: &S3Client,
+        mut callback: impl FnMut(&ChunkStore, S3ObjectKey, u64) -> Result<(), Error>,
+    ) -> Result<(), Error> {
+        let prefix = S3PathPrefix::Some(".chunks/".to_string());
+        let mut list_bucket_result =
+            proxmox_async::runtime::block_on(s3_client.list_objects_v2(&prefix, None))
+                .context("failed to list chunk in s3 object store")?;
+
+        loop {
+            for content in list_bucket_result.contents {
+                worker.check_abort()?;
+                worker.fail_on_shutdown()?;
+
+                callback(&self.inner.chunk_store, content.key, content.size)?;
+            }
+
+            // Process next batch of chunks if there are more.
+            if list_bucket_result.is_truncated {
+                list_bucket_result = proxmox_async::runtime::block_on(s3_client.list_objects_v2(
+                    &prefix,
+                    list_bucket_result.next_continuation_token.as_deref(),
+                ))?;
+
+                continue;
+            }
+
+            break;
+        }
+
+        Ok(())
+    }
+
     fn garbage_collection_impl(
         &self,
         worker: &dyn WorkerTaskContext,
@@ -2596,41 +2634,35 @@ impl DataStore {
 
         if let Some(ref s3_client) = s3_client {
             let mut chunk_count = 0;
-            let prefix = S3PathPrefix::Some(".chunks/".to_string());
-            // Operates in batches of 1000 objects max per request
-            let mut list_bucket_result =
-                proxmox_async::runtime::block_on(s3_client.list_objects_v2(&prefix, None))
-                    .context("failed to list chunk in s3 object store")?;
-
             let mut delete_list =
                 S3DeleteList::with_thresholds(S3_DELETE_BATCH_LIMIT, S3_DELETE_DEFER_LIMIT_SECONDS);
-            loop {
-                for content in list_bucket_result.contents {
-                    worker.check_abort()?;
-                    worker.fail_on_shutdown()?;
+
+            self.paginate_s3_chunk_objects(
+                worker,
+                s3_client,
+                |chunk_store, object_key, size| -> Result<(), Error> {
                     let (chunk_path, digest, bad) =
-                        match self.chunk_path_from_object_key(&content.key) {
+                        match self.chunk_path_from_object_key(&object_key) {
                             Some(path) => path,
-                            None => continue,
+                            None => return Ok(()),
                         };
 
                     let timeout = std::time::Duration::from_secs(0);
-                    let _chunk_guard = match self.inner.chunk_store.lock_chunk(&digest, timeout) {
+                    let _chunk_guard = match chunk_store.lock_chunk(&digest, timeout) {
                         Ok(guard) => guard,
-                        Err(_) => continue,
+                        Err(_) => return Ok(()),
                     };
-                    let _guard = self.inner.chunk_store.mutex().lock().unwrap();
+                    let _guard = chunk_store.mutex().lock().unwrap();
 
                     // Check local markers (created or atime updated during phase1) and
                     // keep or delete chunk based on that.
                     let atime = match std::fs::metadata(&chunk_path) {
                         Ok(stat) => Some(stat.accessed()?),
                         Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
-                            if self.inner.chunk_store.clear_chunk_expected_mark(&digest)? {
+                            if chunk_store.clear_chunk_expected_mark(&digest)? {
                                 unsafe {
                                     // chunk store lock held
-                                    self.inner
-                                        .chunk_store
+                                    chunk_store
                                         .replace_chunk_with_marker_or_create_marker(&digest)?;
                                 }
                                 Some(SystemTime::now())
@@ -2645,11 +2677,11 @@ impl DataStore {
                         let atime = atime.duration_since(SystemTime::UNIX_EPOCH)?.as_secs() as i64;
 
                         unsafe {
-                            self.inner.chunk_store.cond_sweep_chunk(
+                            chunk_store.cond_sweep_chunk(
                                 atime,
                                 min_atime,
                                 oldest_writer,
-                                content.size,
+                                size,
                                 bad,
                                 &mut gc_status,
                                 || {
@@ -2660,15 +2692,15 @@ impl DataStore {
                                             std::fs::remove_file(chunk_path)?;
                                         }
                                     }
-                                    delete_list.push(content.key, _chunk_guard);
+                                    delete_list.push(object_key, _chunk_guard);
                                     Ok(())
                                 },
                             )?;
                         }
                     } else {
                         gc_status.removed_chunks += 1;
-                        gc_status.removed_bytes += content.size;
-                        delete_list.push(content.key, _chunk_guard);
+                        gc_status.removed_bytes += size;
+                        delete_list.push(object_key, _chunk_guard);
                     }
 
                     chunk_count += 1;
@@ -2678,19 +2710,10 @@ impl DataStore {
 
                     // limit pending deletes to avoid holding too many chunk flocks
                     delete_list.conditional_delete_and_drop_locks(s3_client)?;
-                }
-                // Process next batch of chunks if there is more
-                if list_bucket_result.is_truncated {
-                    list_bucket_result =
-                        proxmox_async::runtime::block_on(s3_client.list_objects_v2(
-                            &prefix,
-                            list_bucket_result.next_continuation_token.as_deref(),
-                        ))?;
-                    continue;
-                }
 
-                break;
-            }
+                    Ok(())
+                },
+            )?;
 
             // delete the last batch of objects, if there are any remaining
             delete_list.delete_and_drop_locks(s3_client)?;
-- 
2.47.3

[PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace

[Shan Shaji]

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 pbs-datastore/Cargo.toml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index b51438dc6..312cbc1cb 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -48,6 +48,7 @@ proxmox-systemd.workspace = true
 proxmox-time.workspace = true
 proxmox-uuid.workspace = true
 proxmox-worker-task.workspace = true
+proxmox-log.workspace = true
 
 pbs-api-types.workspace = true
 pbs-buildcfg.workspace = true
-- 
2.47.3

[PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers

[Shan Shaji]

previously, recreating an S3-backed datastore with the --reuse-datastore
flag fetched only the metadata files into the local cache, without
creating chunk markers. Garbage collection (GC) had to be run to
recreate markers for chunks referenced by the backup indexes.

In order to fix this, add method to create zero-byte chunk markers by
listing the chunk objects in S3 when reusing a datastore.

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7bb0c187c..17fa47cc3 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -13,6 +13,7 @@ use hyper::Method;
 use hyper::body::Bytes;
 use nix::unistd::{UnlinkatFlags, unlinkat};
 use pbs_tools::lru_cache::LruCache;
+use proxmox_log::log_throttle::{self, LogThrottle};
 use tokio::io::AsyncWriteExt;
 use tracing::{info, warn};
 
@@ -3292,6 +3293,55 @@ impl DataStore {
         *OLD_LOCKING
     }
 
+    /// The caller must hold the maintenance-mode lock with the datastore in
+    /// `S3Refresh` mode.
+    pub fn create_empty_s3_chunk_markers(
+        &self,
+        worker: &dyn WorkerTaskContext,
+    ) -> Result<(), Error> {
+        let DatastoreBackend::S3(s3_client) = self.backend()? else {
+            bail!("store '{}' not backed by S3", self.name());
+        };
+
+        let start = std::time::Instant::now();
+
+        // Holding the chunk store lock for the whole time here  is
+        // only acceptable since the maintenance mode lock is held
+        // during the entire datastore creation.
+        let _guard = self.inner.chunk_store.mutex().lock().unwrap();
+
+        let mut chunk_marker_count = 0;
+        let mut log_throttle =
+            LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL);
+
+        self.paginate_s3_chunk_objects(
+            worker,
+            s3_client.as_ref(),
+            |chunk_store, object_key, _| -> Result<(), Error> {
+                let (_, digest, _) = match digest_from_object_key(&object_key) {
+                    Some(result) => result,
+                    None => return Ok(()),
+                };
+
+                unsafe {
+                    chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?;
+                }
+
+                chunk_marker_count += 1;
+                if log_throttle.should_log() {
+                    info!("created {chunk_marker_count} chunk markers");
+                }
+
+                Ok(())
+            },
+        )?;
+
+        let elapsed = TimeSpan::from(start.elapsed());
+        info!("created {chunk_marker_count} chunk markers in {elapsed}");
+
+        Ok(())
+    }
+
     /// Fetch contents from S3 object store, clear and replace the local cache store contents.
     /// Returns with error for non-S3 datastore backends.
     pub async fn s3_refresh(self: &Arc<Self>) -> Result<(), Error> {
-- 
2.47.3

[PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option to create zero-byte chunk markers

[Shan Shaji]

Expose the `rebuild-chunk-markers` option inside the `/api2/config/
datastore` endpoint. This allows users to optionally create empty,
zero-byte chunk markers during reconstruction.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 src/api2/config/datastore.rs     | 15 ++++++++++++++-
 src/api2/node/disks/directory.rs |  1 +
 src/api2/node/disks/zfs.rs       |  1 +
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index a390bbd6b..b3832bae5 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -95,12 +95,13 @@ impl Drop for UnmountGuard {
 
 pub(crate) fn do_create_datastore(
     lock: BackupLockGuard,
-    _worker: &dyn WorkerTaskContext,
+    worker: &dyn WorkerTaskContext,
     backend_type: DatastoreBackendType,
     mut datastore: DataStoreConfig,
     prune_job_config: Option<PruneJobConfig>,
     reuse_datastore: bool,
     overwrite_in_use: bool,
+    rebuild_chunk_markers: bool,
 ) -> Result<(), Error> {
     // always start out in maintenance-mode create, so access can be blocked during creation
     // without needing to hold the config lock, only the maintenance-mode lock.
@@ -134,6 +135,10 @@ pub(crate) fn do_create_datastore(
         let datastore_instance =
             DataStore::lookup_datastore(lookup_with(&datastore.name, Operation::Lookup))?;
         proxmox_async::runtime::block_on(datastore_instance.s3_refresh())?;
+
+        if rebuild_chunk_markers {
+            datastore_instance.create_empty_s3_chunk_markers(worker)?;
+        }
     }
 
     unset_maintenance_mode(Some(maintenance_mode_lock), &datastore.name, current_type)?;
@@ -161,6 +166,12 @@ pub(crate) fn do_create_datastore(
                 default: false,
                 description: "Overwrite in use marker (S3 backed datastores only)."
             },
+            "rebuild-chunk-markers": {
+                type: Boolean,
+                optional: true,
+                default: false,
+                description: "Create zero-byte chunk markers (S3 backed datastores only)"
+            }
         },
     },
     access: {
@@ -173,6 +184,7 @@ pub fn create_datastore(
     config: DataStoreConfig,
     reuse_datastore: bool,
     overwrite_in_use: bool,
+    rebuild_chunk_markers: bool,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<String, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
@@ -239,6 +251,7 @@ pub fn create_datastore(
                 prune_job_config,
                 reuse_datastore,
                 overwrite_in_use,
+                rebuild_chunk_markers,
             )
         },
     )
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 61e4f9df6..81b6fe36e 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -258,6 +258,7 @@ pub fn create_datastore_disk(
                     None,
                     false,
                     false,
+                    false,
                 )?;
             }
 
diff --git a/src/api2/node/disks/zfs.rs b/src/api2/node/disks/zfs.rs
index b5a0dbfe6..f8c905ccf 100644
--- a/src/api2/node/disks/zfs.rs
+++ b/src/api2/node/disks/zfs.rs
@@ -318,6 +318,7 @@ pub fn create_zpool(
                     None,
                     false,
                     false,
+                    false,
                 )?;
             }
 
-- 
2.47.3

[PATCH proxmox-backup 6/9] fix #7908: datastore: ui: add option to create zero-byte chunk markers

[Shan Shaji]

Expose the `rebuild-chunk-markers` option inside the datastore creation
window to allow creating empty, zero-byte chunk markers.

This option is disabled and can only be enabled when configuring an
S3-backed datastore.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908
---
 www/window/DataStoreEdit.js | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index 911bf47d6..f536ec659 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -81,6 +81,9 @@ Ext.define('PBS.DataStoreEdit', {
                                 let reuseDatastore = inputPanel
                                     .down('[name=reuse-datastore]')
                                     .getValue();
+                                let rebuildChunkMarkersField = inputPanel.down(
+                                    '[name=rebuild-chunk-markers]',
+                                );
 
                                 uuidEditField.setDisabled(!isRemovable);
                                 uuidEditField.allowBlank = !isRemovable;
@@ -100,6 +103,10 @@ Ext.define('PBS.DataStoreEdit', {
                                 overwriteInUseField.setDisabled(!reuseDatastore);
                                 overwriteInUseField.setValue(false);
 
+                                rebuildChunkMarkersField.setHidden(!isS3);
+                                rebuildChunkMarkersField.setDisabled(!reuseDatastore);
+                                rebuildChunkMarkersField.setValue(false);
+
                                 if (isRemovable) {
                                     pathField.setFieldLabel(gettext('Path on Device'));
                                     pathField.setEmptyText(gettext('A relative path'));
@@ -204,6 +211,12 @@ Ext.define('PBS.DataStoreEdit', {
                                     inputPanel.down('[name=overwrite-in-use]');
                                 overwriteInUseField.setDisabled(!selected);
                                 overwriteInUseField.setValue(false);
+
+                                let rebuildChunkMarkersField = inputPanel.down(
+                                    '[name=rebuild-chunk-markers]',
+                                );
+                                rebuildChunkMarkersField.setDisabled(!selected);
+                                rebuildChunkMarkersField.setValue(false);
                             },
                         },
                     },
@@ -216,6 +229,13 @@ Ext.define('PBS.DataStoreEdit', {
                         hidden: true,
                         disabled: true,
                     },
+                    {
+                        xtype: 'checkbox',
+                        name: 'rebuild-chunk-markers',
+                        fieldLabel: gettext('Rebuild chunk markers'),
+                        hidden: true,
+                        disabled: true,
+                    },
                 ],
 
                 onGetValues: function (values) {
-- 
2.47.3

[PATCH proxmox-backup 7/9] fix #7908: backup-manager: add option to create zero-byte chunk markers

[Shan Shaji]

Add optional --rebuild-chunk-markers flag to datastore create command.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 src/bin/proxmox_backup_manager/datastore.rs | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 4c1d9daf5..588cf05c0 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -130,6 +130,12 @@ fn show_datastore(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value
                 default: false,
                 description: "Overwrite in use marker (S3 backed datastores only)."
             },
+            "rebuild-chunk-markers": {
+                type: Boolean,
+                optional: true,
+                default: false,
+                description: "Create zero-byte chunk markers (S3 backed datastores only)"
+            },
             "output-format": {
                 schema: OUTPUT_FORMAT,
                 optional: true,
-- 
2.47.3

[PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option

[Shan Shaji]

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 docs/storage.rst | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/storage.rst b/docs/storage.rst
index 2c6ba0a85..811b5ff44 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -312,8 +312,11 @@ To setup a new datastore called ``my-s3-store`` placed in a bucket called ``pbs-
 A datastore cannot be shared between multiple Proxmox Backup Server instances, only one instance can
 operate on the datastore at a time. However, datastore contents used on an instance which is no
 longer available can be reused on a fresh installation. To recreate the datastore, you must pass the
-``reuse-datastore`` and ``overwrite-in-use`` flags. Since the datastore name is used as prefix, the
-same datastore name must be used.
+``reuse-datastore`` and ``overwrite-in-use`` flags. The optional ``rebuild-chunk-markers`` flag,
+disabled by default, creates local zero-byte chunk markers when reusing an S3 datastore. These markers
+allow Proxmox Backup Server to identify chunks already stored in S3 when they are not present in
+the in-memory cache, avoiding unnecessary uploads. The time required to create these markers depends
+on the number of chunks.
 
 .. code-block:: console
 
-- 
2.47.3

[PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation

[Shan Shaji]

factor out LogThrottle implementation from pbs-client to proxmox-log
crate.

Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
 proxmox-log/src/lib.rs          |  1 +
 proxmox-log/src/log_throttle.rs | 39 +++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 proxmox-log/src/log_throttle.rs

diff --git a/proxmox-log/src/lib.rs b/proxmox-log/src/lib.rs
index 2d321f20..e82ab258 100644
--- a/proxmox-log/src/lib.rs
+++ b/proxmox-log/src/lib.rs
@@ -14,6 +14,7 @@ mod pve_task_formatter;
 mod tasklog_layer;
 
 pub mod builder;
+pub mod log_throttle;
 pub use builder::Logger;
 pub use file_logger::{FileLogOptions, FileLogger};
 
diff --git a/proxmox-log/src/log_throttle.rs b/proxmox-log/src/log_throttle.rs
new file mode 100644
index 00000000..e88e116f
--- /dev/null
+++ b/proxmox-log/src/log_throttle.rs
@@ -0,0 +1,39 @@
+use std::time::{Duration, Instant};
+
+pub const INITIAL_INTERVAL: Duration = Duration::from_secs(5);
+pub const MAX_INTERVAL: Duration = Duration::from_secs(60);
+
+/// Rate-limits recurring log output, such as progress lines, with an exponential backoff.
+///
+/// [`should_log`](Self::should_log) returns `true` at most once per the current interval. Starting
+/// from an initial interval, the delay doubles after each hit up to a configured maximum, giving
+/// frequent feedback early on while avoiding log spam during long-running operations.
+pub struct LogThrottle {
+    last: Instant,
+    interval: Duration,
+    max_interval: Duration,
+}
+
+impl LogThrottle {
+    /// Creates a throttle with the given initial interval, doubling up to `max_interval` after each
+    /// hit. The initial interval is capped at `max_interval`.
+    pub fn new(initial_interval: Duration, max_interval: Duration) -> Self {
+        Self {
+            last: Instant::now(),
+            interval: initial_interval.min(max_interval),
+            max_interval,
+        }
+    }
+
+    /// Returns `true` if the current interval has elapsed since the last hit, advancing the
+    /// throttle and doubling the interval (capped at the maximum); returns `false` otherwise.
+    pub fn should_log(&mut self) -> bool {
+        let now = Instant::now();
+        if now.duration_since(self.last) < self.interval {
+            return false;
+        }
+        self.last = now;
+        self.interval = (self.interval * 2).min(self.max_interval);
+        true
+    }
+}
-- 
2.47.3