* [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Remove the local implementation of `LogThrottle` and use the version
provided by the proxmox-log crate instead. This prepares the utility
for re-use in pbs-datastore and updates all existing references
accordingly. No functional changes.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/lib.rs | 3 ---
pbs-client/src/log_throttle.rs | 36 -------------------------------
proxmox-backup-client/src/main.rs | 3 ++-
3 files changed, 2 insertions(+), 40 deletions(-)
delete mode 100644 pbs-client/src/log_throttle.rs
diff --git a/pbs-client/src/lib.rs b/pbs-client/src/lib.rs
index 139f4f4b5..4b8e4e4f4 100644
--- a/pbs-client/src/lib.rs
+++ b/pbs-client/src/lib.rs
@@ -45,7 +45,4 @@ pub use chunk_stream::{ChunkStream, FixedChunkStream, InjectionData};
mod backup_stats;
pub use backup_stats::BackupStats;
-mod log_throttle;
-pub use log_throttle::LogThrottle;
-
pub const PROXMOX_BACKUP_TCP_KEEPALIVE_TIME: u32 = 120;
diff --git a/pbs-client/src/log_throttle.rs b/pbs-client/src/log_throttle.rs
deleted file mode 100644
index cfaa6c494..000000000
--- a/pbs-client/src/log_throttle.rs
+++ /dev/null
@@ -1,36 +0,0 @@
-use std::time::{Duration, Instant};
-
-/// Rate-limits recurring log output, such as progress lines, with an exponential backoff.
-///
-/// [`should_log`](Self::should_log) returns `true` at most once per the current interval. Starting
-/// from an initial interval, the delay doubles after each hit up to a configured maximum, giving
-/// frequent feedback early on while avoiding log spam during long-running operations.
-pub struct LogThrottle {
- last: Instant,
- interval: Duration,
- max_interval: Duration,
-}
-
-impl LogThrottle {
- /// Creates a throttle with the given initial interval, doubling up to `max_interval` after each
- /// hit. The initial interval is capped at `max_interval`.
- pub fn new(initial_interval: Duration, max_interval: Duration) -> Self {
- Self {
- last: Instant::now(),
- interval: initial_interval.min(max_interval),
- max_interval,
- }
- }
-
- /// Returns `true` if the current interval has elapsed since the last hit, advancing the
- /// throttle and doubling the interval (capped at the maximum); returns `false` otherwise.
- pub fn should_log(&mut self) -> bool {
- let now = Instant::now();
- if now.duration_since(self.last) < self.interval {
- return false;
- }
- self.last = now;
- self.interval = (self.interval * 2).min(self.max_interval);
- true
- }
-}
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 3e7aa7cfd..93f5a1b7a 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -18,6 +18,7 @@ use pathpatterns::{MatchEntry, MatchType, PatternFlag};
use proxmox_async::blocking::TokioWriterAdapter;
use proxmox_human_byte::HumanByte;
use proxmox_io::StdChannelWriter;
+use proxmox_log::log_throttle::LogThrottle;
use proxmox_router::{ApiMethod, RpcEnvironment, cli::*};
use proxmox_schema::api;
use proxmox_sys::fs::{CreateOptions, file_get_json, image_size, replace_file};
@@ -51,7 +52,7 @@ use pbs_client::{
BACKUP_SOURCE_SCHEMA, BackupDetectionMode, BackupReader, BackupRepository,
BackupRepositoryArgs, BackupSpecificationType, BackupStats, BackupTargetArgs, BackupWriter,
BackupWriterOptions, ChunkStream, FixedChunkStream, HttpClient, IndexType, InjectionData,
- LogThrottle, PxarBackupStream, RemoteChunkReader, UploadOptions, delete_ticket_info,
+ PxarBackupStream, RemoteChunkReader, UploadOptions, delete_ticket_info,
parse_backup_specification, view_task_result,
};
use pbs_datastore::catalog::{BackupCatalogWriter, CatalogReader, CatalogWriter};
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
This is in preparation to re-use the chunk iteration logic for creating
empty marker files during s3 refresh.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-datastore/src/datastore.rs | 93 +++++++++++++++++++++-------------
1 file changed, 58 insertions(+), 35 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index a74a6a824..7bb0c187c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -2505,6 +2505,44 @@ impl DataStore {
}
}
+ // Fetch chunk objects in batches of max 1000 per request.
+ // For each listed object, invoke `callback` with a reference to the
+ // datastore's local `ChunkStore`, followed by the object's S3 key and size.
+ fn paginate_s3_chunk_objects(
+ &self,
+ worker: &dyn WorkerTaskContext,
+ s3_client: &S3Client,
+ mut callback: impl FnMut(&ChunkStore, S3ObjectKey, u64) -> Result<(), Error>,
+ ) -> Result<(), Error> {
+ let prefix = S3PathPrefix::Some(".chunks/".to_string());
+ let mut list_bucket_result =
+ proxmox_async::runtime::block_on(s3_client.list_objects_v2(&prefix, None))
+ .context("failed to list chunk in s3 object store")?;
+
+ loop {
+ for content in list_bucket_result.contents {
+ worker.check_abort()?;
+ worker.fail_on_shutdown()?;
+
+ callback(&self.inner.chunk_store, content.key, content.size)?;
+ }
+
+ // Process next batch of chunks if there are more.
+ if list_bucket_result.is_truncated {
+ list_bucket_result = proxmox_async::runtime::block_on(s3_client.list_objects_v2(
+ &prefix,
+ list_bucket_result.next_continuation_token.as_deref(),
+ ))?;
+
+ continue;
+ }
+
+ break;
+ }
+
+ Ok(())
+ }
+
fn garbage_collection_impl(
&self,
worker: &dyn WorkerTaskContext,
@@ -2596,41 +2634,35 @@ impl DataStore {
if let Some(ref s3_client) = s3_client {
let mut chunk_count = 0;
- let prefix = S3PathPrefix::Some(".chunks/".to_string());
- // Operates in batches of 1000 objects max per request
- let mut list_bucket_result =
- proxmox_async::runtime::block_on(s3_client.list_objects_v2(&prefix, None))
- .context("failed to list chunk in s3 object store")?;
-
let mut delete_list =
S3DeleteList::with_thresholds(S3_DELETE_BATCH_LIMIT, S3_DELETE_DEFER_LIMIT_SECONDS);
- loop {
- for content in list_bucket_result.contents {
- worker.check_abort()?;
- worker.fail_on_shutdown()?;
+
+ self.paginate_s3_chunk_objects(
+ worker,
+ s3_client,
+ |chunk_store, object_key, size| -> Result<(), Error> {
let (chunk_path, digest, bad) =
- match self.chunk_path_from_object_key(&content.key) {
+ match self.chunk_path_from_object_key(&object_key) {
Some(path) => path,
- None => continue,
+ None => return Ok(()),
};
let timeout = std::time::Duration::from_secs(0);
- let _chunk_guard = match self.inner.chunk_store.lock_chunk(&digest, timeout) {
+ let _chunk_guard = match chunk_store.lock_chunk(&digest, timeout) {
Ok(guard) => guard,
- Err(_) => continue,
+ Err(_) => return Ok(()),
};
- let _guard = self.inner.chunk_store.mutex().lock().unwrap();
+ let _guard = chunk_store.mutex().lock().unwrap();
// Check local markers (created or atime updated during phase1) and
// keep or delete chunk based on that.
let atime = match std::fs::metadata(&chunk_path) {
Ok(stat) => Some(stat.accessed()?),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
- if self.inner.chunk_store.clear_chunk_expected_mark(&digest)? {
+ if chunk_store.clear_chunk_expected_mark(&digest)? {
unsafe {
// chunk store lock held
- self.inner
- .chunk_store
+ chunk_store
.replace_chunk_with_marker_or_create_marker(&digest)?;
}
Some(SystemTime::now())
@@ -2645,11 +2677,11 @@ impl DataStore {
let atime = atime.duration_since(SystemTime::UNIX_EPOCH)?.as_secs() as i64;
unsafe {
- self.inner.chunk_store.cond_sweep_chunk(
+ chunk_store.cond_sweep_chunk(
atime,
min_atime,
oldest_writer,
- content.size,
+ size,
bad,
&mut gc_status,
|| {
@@ -2660,15 +2692,15 @@ impl DataStore {
std::fs::remove_file(chunk_path)?;
}
}
- delete_list.push(content.key, _chunk_guard);
+ delete_list.push(object_key, _chunk_guard);
Ok(())
},
)?;
}
} else {
gc_status.removed_chunks += 1;
- gc_status.removed_bytes += content.size;
- delete_list.push(content.key, _chunk_guard);
+ gc_status.removed_bytes += size;
+ delete_list.push(object_key, _chunk_guard);
}
chunk_count += 1;
@@ -2678,19 +2710,10 @@ impl DataStore {
// limit pending deletes to avoid holding too many chunk flocks
delete_list.conditional_delete_and_drop_locks(s3_client)?;
- }
- // Process next batch of chunks if there is more
- if list_bucket_result.is_truncated {
- list_bucket_result =
- proxmox_async::runtime::block_on(s3_client.list_objects_v2(
- &prefix,
- list_bucket_result.next_continuation_token.as_deref(),
- ))?;
- continue;
- }
- break;
- }
+ Ok(())
+ },
+ )?;
// delete the last batch of objects, if there are any remaining
delete_list.delete_and_drop_locks(s3_client)?;
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-datastore/Cargo.toml | 1 +
1 file changed, 1 insertion(+)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index b51438dc6..312cbc1cb 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -48,6 +48,7 @@ proxmox-systemd.workspace = true
proxmox-time.workspace = true
proxmox-uuid.workspace = true
proxmox-worker-task.workspace = true
+proxmox-log.workspace = true
pbs-api-types.workspace = true
pbs-buildcfg.workspace = true
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (2 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
previously, recreating an S3-backed datastore with the --reuse-datastore
flag fetched only the metadata files into the local cache, without
creating chunk markers. Garbage collection (GC) had to be run to
recreate markers for chunks referenced by the backup indexes.
In order to fix this, add method to create zero-byte chunk markers by
listing the chunk objects in S3 when reusing a datastore.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7bb0c187c..17fa47cc3 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -13,6 +13,7 @@ use hyper::Method;
use hyper::body::Bytes;
use nix::unistd::{UnlinkatFlags, unlinkat};
use pbs_tools::lru_cache::LruCache;
+use proxmox_log::log_throttle::{self, LogThrottle};
use tokio::io::AsyncWriteExt;
use tracing::{info, warn};
@@ -3292,6 +3293,55 @@ impl DataStore {
*OLD_LOCKING
}
+ /// The caller must hold the maintenance-mode lock with the datastore in
+ /// `S3Refresh` mode.
+ pub fn create_empty_s3_chunk_markers(
+ &self,
+ worker: &dyn WorkerTaskContext,
+ ) -> Result<(), Error> {
+ let DatastoreBackend::S3(s3_client) = self.backend()? else {
+ bail!("store '{}' not backed by S3", self.name());
+ };
+
+ let start = std::time::Instant::now();
+
+ // Holding the chunk store lock for the whole time here is
+ // only acceptable since the maintenance mode lock is held
+ // during the entire datastore creation.
+ let _guard = self.inner.chunk_store.mutex().lock().unwrap();
+
+ let mut chunk_marker_count = 0;
+ let mut log_throttle =
+ LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL);
+
+ self.paginate_s3_chunk_objects(
+ worker,
+ s3_client.as_ref(),
+ |chunk_store, object_key, _| -> Result<(), Error> {
+ let (_, digest, _) = match digest_from_object_key(&object_key) {
+ Some(result) => result,
+ None => return Ok(()),
+ };
+
+ unsafe {
+ chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?;
+ }
+
+ chunk_marker_count += 1;
+ if log_throttle.should_log() {
+ info!("created {chunk_marker_count} chunk markers");
+ }
+
+ Ok(())
+ },
+ )?;
+
+ let elapsed = TimeSpan::from(start.elapsed());
+ info!("created {chunk_marker_count} chunk markers in {elapsed}");
+
+ Ok(())
+ }
+
/// Fetch contents from S3 object store, clear and replace the local cache store contents.
/// Returns with error for non-S3 datastore backends.
pub async fn s3_refresh(self: &Arc<Self>) -> Result<(), Error> {
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option to create zero-byte chunk markers
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (3 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Expose the `rebuild-chunk-markers` option inside the `/api2/config/
datastore` endpoint. This allows users to optionally create empty,
zero-byte chunk markers during reconstruction.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
src/api2/config/datastore.rs | 15 ++++++++++++++-
src/api2/node/disks/directory.rs | 1 +
src/api2/node/disks/zfs.rs | 1 +
3 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index a390bbd6b..b3832bae5 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -95,12 +95,13 @@ impl Drop for UnmountGuard {
pub(crate) fn do_create_datastore(
lock: BackupLockGuard,
- _worker: &dyn WorkerTaskContext,
+ worker: &dyn WorkerTaskContext,
backend_type: DatastoreBackendType,
mut datastore: DataStoreConfig,
prune_job_config: Option<PruneJobConfig>,
reuse_datastore: bool,
overwrite_in_use: bool,
+ rebuild_chunk_markers: bool,
) -> Result<(), Error> {
// always start out in maintenance-mode create, so access can be blocked during creation
// without needing to hold the config lock, only the maintenance-mode lock.
@@ -134,6 +135,10 @@ pub(crate) fn do_create_datastore(
let datastore_instance =
DataStore::lookup_datastore(lookup_with(&datastore.name, Operation::Lookup))?;
proxmox_async::runtime::block_on(datastore_instance.s3_refresh())?;
+
+ if rebuild_chunk_markers {
+ datastore_instance.create_empty_s3_chunk_markers(worker)?;
+ }
}
unset_maintenance_mode(Some(maintenance_mode_lock), &datastore.name, current_type)?;
@@ -161,6 +166,12 @@ pub(crate) fn do_create_datastore(
default: false,
description: "Overwrite in use marker (S3 backed datastores only)."
},
+ "rebuild-chunk-markers": {
+ type: Boolean,
+ optional: true,
+ default: false,
+ description: "Create zero-byte chunk markers (S3 backed datastores only)"
+ }
},
},
access: {
@@ -173,6 +184,7 @@ pub fn create_datastore(
config: DataStoreConfig,
reuse_datastore: bool,
overwrite_in_use: bool,
+ rebuild_chunk_markers: bool,
rpcenv: &mut dyn RpcEnvironment,
) -> Result<String, Error> {
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
@@ -239,6 +251,7 @@ pub fn create_datastore(
prune_job_config,
reuse_datastore,
overwrite_in_use,
+ rebuild_chunk_markers,
)
},
)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 61e4f9df6..81b6fe36e 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -258,6 +258,7 @@ pub fn create_datastore_disk(
None,
false,
false,
+ false,
)?;
}
diff --git a/src/api2/node/disks/zfs.rs b/src/api2/node/disks/zfs.rs
index b5a0dbfe6..f8c905ccf 100644
--- a/src/api2/node/disks/zfs.rs
+++ b/src/api2/node/disks/zfs.rs
@@ -318,6 +318,7 @@ pub fn create_zpool(
None,
false,
false,
+ false,
)?;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: add option to create zero-byte chunk markers
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (4 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Expose the `rebuild-chunk-markers` option inside the datastore creation
window to allow creating empty, zero-byte chunk markers.
This option is disabled and can only be enabled when configuring an
S3-backed datastore.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908
---
www/window/DataStoreEdit.js | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index 911bf47d6..f536ec659 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -81,6 +81,9 @@ Ext.define('PBS.DataStoreEdit', {
let reuseDatastore = inputPanel
.down('[name=reuse-datastore]')
.getValue();
+ let rebuildChunkMarkersField = inputPanel.down(
+ '[name=rebuild-chunk-markers]',
+ );
uuidEditField.setDisabled(!isRemovable);
uuidEditField.allowBlank = !isRemovable;
@@ -100,6 +103,10 @@ Ext.define('PBS.DataStoreEdit', {
overwriteInUseField.setDisabled(!reuseDatastore);
overwriteInUseField.setValue(false);
+ rebuildChunkMarkersField.setHidden(!isS3);
+ rebuildChunkMarkersField.setDisabled(!reuseDatastore);
+ rebuildChunkMarkersField.setValue(false);
+
if (isRemovable) {
pathField.setFieldLabel(gettext('Path on Device'));
pathField.setEmptyText(gettext('A relative path'));
@@ -204,6 +211,12 @@ Ext.define('PBS.DataStoreEdit', {
inputPanel.down('[name=overwrite-in-use]');
overwriteInUseField.setDisabled(!selected);
overwriteInUseField.setValue(false);
+
+ let rebuildChunkMarkersField = inputPanel.down(
+ '[name=rebuild-chunk-markers]',
+ );
+ rebuildChunkMarkersField.setDisabled(!selected);
+ rebuildChunkMarkersField.setValue(false);
},
},
},
@@ -216,6 +229,13 @@ Ext.define('PBS.DataStoreEdit', {
hidden: true,
disabled: true,
},
+ {
+ xtype: 'checkbox',
+ name: 'rebuild-chunk-markers',
+ fieldLabel: gettext('Rebuild chunk markers'),
+ hidden: true,
+ disabled: true,
+ },
],
onGetValues: function (values) {
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 7/9] fix #7908: backup-manager: add option to create zero-byte chunk markers
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (5 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Add optional --rebuild-chunk-markers flag to datastore create command.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=7908
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
src/bin/proxmox_backup_manager/datastore.rs | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 4c1d9daf5..588cf05c0 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -130,6 +130,12 @@ fn show_datastore(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value
default: false,
description: "Overwrite in use marker (S3 backed datastores only)."
},
+ "rebuild-chunk-markers": {
+ type: Boolean,
+ optional: true,
+ default: false,
+ description: "Create zero-byte chunk markers (S3 backed datastores only)"
+ },
"output-format": {
schema: OUTPUT_FORMAT,
optional: true,
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (6 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
docs/storage.rst | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/storage.rst b/docs/storage.rst
index 2c6ba0a85..811b5ff44 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -312,8 +312,11 @@ To setup a new datastore called ``my-s3-store`` placed in a bucket called ``pbs-
A datastore cannot be shared between multiple Proxmox Backup Server instances, only one instance can
operate on the datastore at a time. However, datastore contents used on an instance which is no
longer available can be reused on a fresh installation. To recreate the datastore, you must pass the
-``reuse-datastore`` and ``overwrite-in-use`` flags. Since the datastore name is used as prefix, the
-same datastore name must be used.
+``reuse-datastore`` and ``overwrite-in-use`` flags. The optional ``rebuild-chunk-markers`` flag,
+disabled by default, creates local zero-byte chunk markers when reusing an S3 datastore. These markers
+allow Proxmox Backup Server to identify chunks already stored in S3 when they are not present in
+the in-memory cache, avoiding unnecessary uploads. The time required to create these markers depends
+on the number of chunks.
.. code-block:: console
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
` (7 preceding siblings ...)
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
@ 2026-10-19 16:03 ` Shan Shaji
8 siblings, 0 replies; 10+ messages in thread
From: Shan Shaji @ 2026-10-19 16:03 UTC (permalink / raw)
To: pbs-devel
factor out LogThrottle implementation from pbs-client to proxmox-log
crate.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
proxmox-log/src/lib.rs | 1 +
proxmox-log/src/log_throttle.rs | 39 +++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
create mode 100644 proxmox-log/src/log_throttle.rs
diff --git a/proxmox-log/src/lib.rs b/proxmox-log/src/lib.rs
index 2d321f20..e82ab258 100644
--- a/proxmox-log/src/lib.rs
+++ b/proxmox-log/src/lib.rs
@@ -14,6 +14,7 @@ mod pve_task_formatter;
mod tasklog_layer;
pub mod builder;
+pub mod log_throttle;
pub use builder::Logger;
pub use file_logger::{FileLogOptions, FileLogger};
diff --git a/proxmox-log/src/log_throttle.rs b/proxmox-log/src/log_throttle.rs
new file mode 100644
index 00000000..e88e116f
--- /dev/null
+++ b/proxmox-log/src/log_throttle.rs
@@ -0,0 +1,39 @@
+use std::time::{Duration, Instant};
+
+pub const INITIAL_INTERVAL: Duration = Duration::from_secs(5);
+pub const MAX_INTERVAL: Duration = Duration::from_secs(60);
+
+/// Rate-limits recurring log output, such as progress lines, with an exponential backoff.
+///
+/// [`should_log`](Self::should_log) returns `true` at most once per the current interval. Starting
+/// from an initial interval, the delay doubles after each hit up to a configured maximum, giving
+/// frequent feedback early on while avoiding log spam during long-running operations.
+pub struct LogThrottle {
+ last: Instant,
+ interval: Duration,
+ max_interval: Duration,
+}
+
+impl LogThrottle {
+ /// Creates a throttle with the given initial interval, doubling up to `max_interval` after each
+ /// hit. The initial interval is capped at `max_interval`.
+ pub fn new(initial_interval: Duration, max_interval: Duration) -> Self {
+ Self {
+ last: Instant::now(),
+ interval: initial_interval.min(max_interval),
+ max_interval,
+ }
+ }
+
+ /// Returns `true` if the current interval has elapsed since the last hit, advancing the
+ /// throttle and doubling the interval (capped at the maximum); returns `false` otherwise.
+ pub fn should_log(&mut self) -> bool {
+ let now = Instant::now();
+ if now.duration_since(self.last) < self.interval {
+ return false;
+ }
+ self.last = now;
+ self.interval = (self.interval * 2).min(self.max_interval);
+ true
+ }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 10+ messages in thread