From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 399EC1FF0AA for ; Tue, 17 Nov 2026 14:14:40 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E29C721571; Tue, 17 Nov 2026 14:14:39 +0100 (CET) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Tue, 17 Nov 2026 14:14:32 +0100 Message-Id: From: "Shan Shaji" To: "Christian Ebner" , Subject: Re: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers X-Mailer: aerc 0.20.0 References: <20261019160359.20851-1-s.shaji@proxmox.com> <20261019160359.20851-5-s.shaji@proxmox.com> <3f1efc7a-04ed-4dfa-9c8f-2d2eb9aeaa68@proxmox.com> In-Reply-To: <3f1efc7a-04ed-4dfa-9c8f-2d2eb9aeaa68@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794921272862 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.345 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: Y25PZ2VVOOSSA6X4722UTO66PHPCCTWN X-Message-ID-Hash: Y25PZ2VVOOSSA6X4722UTO66PHPCCTWN X-MailFrom: s.shaji@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed Nov 11, 2026 at 4:04 PM CET, Christian Ebner wrote: > One comment inline. > > On 10/19/26 6:04 PM, Shan Shaji wrote: >> previously, recreating an S3-backed datastore with the --reuse-datastore >> flag fetched only the metadata files into the local cache, without >> creating chunk markers. Garbage collection (GC) had to be run to >> recreate markers for chunks referenced by the backup indexes. >> >> In order to fix this, add method to create zero-byte chunk markers by >> listing the chunk objects in S3 when reusing a datastore. >> >> Signed-off-by: Shan Shaji >> --- >> pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++ >> 1 file changed, 50 insertions(+) >> >> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastor= e.rs >> index 7bb0c187c..17fa47cc3 100644 >> --- a/pbs-datastore/src/datastore.rs >> +++ b/pbs-datastore/src/datastore.rs >> @@ -13,6 +13,7 @@ use hyper::Method; >> use hyper::body::Bytes; >> use nix::unistd::{UnlinkatFlags, unlinkat}; >> use pbs_tools::lru_cache::LruCache; >> +use proxmox_log::log_throttle::{self, LogThrottle}; >> use tokio::io::AsyncWriteExt; >> use tracing::{info, warn}; >> >> @@ -3292,6 +3293,55 @@ impl DataStore { >> *OLD_LOCKING >> } >> >> + /// The caller must hold the maintenance-mode lock with the datasto= re in >> + /// `S3Refresh` mode. >> + pub fn create_empty_s3_chunk_markers( >> + &self, >> + worker: &dyn WorkerTaskContext, >> + ) -> Result<(), Error> { >> + let DatastoreBackend::S3(s3_client) =3D self.backend()? else { >> + bail!("store '{}' not backed by S3", self.name()); >> + }; >> + >> + let start =3D std::time::Instant::now(); >> + >> + // Holding the chunk store lock for the whole time here is >> + // only acceptable since the maintenance mode lock is held >> + // during the entire datastore creation. >> + let _guard =3D self.inner.chunk_store.mutex().lock().unwrap(); >> + >> + let mut chunk_marker_count =3D 0; >> + let mut log_throttle =3D >> + LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_thrott= le::MAX_INTERVAL); >> + >> + self.paginate_s3_chunk_objects( >> + worker, >> + s3_client.as_ref(), >> + |chunk_store, object_key, _| -> Result<(), Error> { >> + let (_, digest, _) =3D match digest_from_object_key(&ob= ject_key) { >> + Some(result) =3D> result, >> + None =3D> return Ok(()), >> + }; > > comment: the object keys here do also potentially include bad chunks > with their respective suffix, so the markers would be created for these > as well, leading to no further good chunks being uploaded and corrupt > backups. Thank you for catching this. I think I should either skip marker creation for bad chunks or create the corresponding .N.bad marker files. I haven't tested this yet. I'll check and get back. > Further I think there still is a pre-existing issue with the marker > files for bad chunks potentially being missing, which would lead to > removing these from the S3 backend even if no new, good chunk exists. > Do you mean the garbage-collection code path in [1]? If I understand correctly, phase 1 updates existing local .N.bad files while iterating over the index files [2], but does not recreate missing ones. During phase 2, if the corresponding local .N.bad file is missing and clear_chunk_expected_mark() returns false, atime becomes None. This causes the bad chunk object to be queued for deletion from S3, even if no good replacement exists. - [1] https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-data= store/src/datastore.rs;h=3De6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=3DHE= AD#l2544 - [2] https://git.proxmox.com/?p=3Dproxmox-backup.git;a=3Dblob;f=3Dpbs-data= store/src/datastore.rs;h=3De6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=3DHE= AD#l2175 > Above might further be written a bit more concise with the chunk path > helper to already check for bad chunk suffix by: > ``` > - let (_, digest, _) =3D match digest_from_object_key(&object_key) { > - Some(result) =3D> result, > - None =3D> return Ok(()), > + let Some((_, digest, bad)) =3D > self.chunk_path_from_object_key(&object_key) else { > + return Ok(()); > ``` > >> + >> + unsafe { >> + chunk_store.replace_chunk_with_marker_or_create_mar= ker(&digest)?; >> + } >> + >> + chunk_marker_count +=3D 1; >> + if log_throttle.should_log() { >> + info!("created {chunk_marker_count} chunk markers")= ; >> + } >> + >> + Ok(()) >> + }, >> + )?; >> + >> + let elapsed =3D TimeSpan::from(start.elapsed()); >> + info!("created {chunk_marker_count} chunk markers in {elapsed}"= ); >> + >> + Ok(()) >> + } >> + >> /// Fetch contents from S3 object store, clear and replace the loc= al cache store contents. >> /// Returns with error for non-S3 datastore backends. >> pub async fn s3_refresh(self: &Arc) -> Result<(), Error> {