From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C45EB1FF09C for ; Mon, 19 Oct 2026 18:04:25 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4FE6221666; Mon, 19 Oct 2026 18:04:18 +0200 (CEST) From: Shan Shaji To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Date: Mon, 19 Oct 2026 18:03:54 +0200 Message-ID: <20261019160359.20851-5-s.shaji@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261019160359.20851-1-s.shaji@proxmox.com> References: <20261019160359.20851-1-s.shaji@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792425854212 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.411 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 3SOOK7BBNLXUBOSVNWOJBTTLE5J5GGYW X-Message-ID-Hash: 3SOOK7BBNLXUBOSVNWOJBTTLE5J5GGYW X-MailFrom: s.shaji@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: previously, recreating an S3-backed datastore with the --reuse-datastore flag fetched only the metadata files into the local cache, without creating chunk markers. Garbage collection (GC) had to be run to recreate markers for chunks referenced by the backup indexes. In order to fix this, add method to create zero-byte chunk markers by listing the chunk objects in S3 when reusing a datastore. Signed-off-by: Shan Shaji --- pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index 7bb0c187c..17fa47cc3 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -13,6 +13,7 @@ use hyper::Method; use hyper::body::Bytes; use nix::unistd::{UnlinkatFlags, unlinkat}; use pbs_tools::lru_cache::LruCache; +use proxmox_log::log_throttle::{self, LogThrottle}; use tokio::io::AsyncWriteExt; use tracing::{info, warn}; @@ -3292,6 +3293,55 @@ impl DataStore { *OLD_LOCKING } + /// The caller must hold the maintenance-mode lock with the datastore in + /// `S3Refresh` mode. + pub fn create_empty_s3_chunk_markers( + &self, + worker: &dyn WorkerTaskContext, + ) -> Result<(), Error> { + let DatastoreBackend::S3(s3_client) = self.backend()? else { + bail!("store '{}' not backed by S3", self.name()); + }; + + let start = std::time::Instant::now(); + + // Holding the chunk store lock for the whole time here is + // only acceptable since the maintenance mode lock is held + // during the entire datastore creation. + let _guard = self.inner.chunk_store.mutex().lock().unwrap(); + + let mut chunk_marker_count = 0; + let mut log_throttle = + LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL); + + self.paginate_s3_chunk_objects( + worker, + s3_client.as_ref(), + |chunk_store, object_key, _| -> Result<(), Error> { + let (_, digest, _) = match digest_from_object_key(&object_key) { + Some(result) => result, + None => return Ok(()), + }; + + unsafe { + chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?; + } + + chunk_marker_count += 1; + if log_throttle.should_log() { + info!("created {chunk_marker_count} chunk markers"); + } + + Ok(()) + }, + )?; + + let elapsed = TimeSpan::from(start.elapsed()); + info!("created {chunk_marker_count} chunk markers in {elapsed}"); + + Ok(()) + } + /// Fetch contents from S3 object store, clear and replace the local cache store contents. /// Returns with error for non-S3 datastore backends. pub async fn s3_refresh(self: &Arc) -> Result<(), Error> { -- 2.47.3