From: "Shan Shaji" <s.shaji@proxmox.com>
To: "Christian Ebner" <c.ebner@proxmox.com>, <pbs-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers
Date: Tue, 17 Nov 2026 14:14:32 +0100 [thread overview]
Message-ID: <DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com> (raw)
In-Reply-To: <3f1efc7a-04ed-4dfa-9c8f-2d2eb9aeaa68@proxmox.com>
On Wed Nov 11, 2026 at 4:04 PM CET, Christian Ebner wrote:
> One comment inline.
>
> On 10/19/26 6:04 PM, Shan Shaji wrote:
>> previously, recreating an S3-backed datastore with the --reuse-datastore
>> flag fetched only the metadata files into the local cache, without
>> creating chunk markers. Garbage collection (GC) had to be run to
>> recreate markers for chunks referenced by the backup indexes.
>>
>> In order to fix this, add method to create zero-byte chunk markers by
>> listing the chunk objects in S3 when reusing a datastore.
>>
>> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
>> ---
>> pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++
>> 1 file changed, 50 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 7bb0c187c..17fa47cc3 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -13,6 +13,7 @@ use hyper::Method;
>> use hyper::body::Bytes;
>> use nix::unistd::{UnlinkatFlags, unlinkat};
>> use pbs_tools::lru_cache::LruCache;
>> +use proxmox_log::log_throttle::{self, LogThrottle};
>> use tokio::io::AsyncWriteExt;
>> use tracing::{info, warn};
>>
>> @@ -3292,6 +3293,55 @@ impl DataStore {
>> *OLD_LOCKING
>> }
>>
>> + /// The caller must hold the maintenance-mode lock with the datastore in
>> + /// `S3Refresh` mode.
>> + pub fn create_empty_s3_chunk_markers(
>> + &self,
>> + worker: &dyn WorkerTaskContext,
>> + ) -> Result<(), Error> {
>> + let DatastoreBackend::S3(s3_client) = self.backend()? else {
>> + bail!("store '{}' not backed by S3", self.name());
>> + };
>> +
>> + let start = std::time::Instant::now();
>> +
>> + // Holding the chunk store lock for the whole time here is
>> + // only acceptable since the maintenance mode lock is held
>> + // during the entire datastore creation.
>> + let _guard = self.inner.chunk_store.mutex().lock().unwrap();
>> +
>> + let mut chunk_marker_count = 0;
>> + let mut log_throttle =
>> + LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL);
>> +
>> + self.paginate_s3_chunk_objects(
>> + worker,
>> + s3_client.as_ref(),
>> + |chunk_store, object_key, _| -> Result<(), Error> {
>> + let (_, digest, _) = match digest_from_object_key(&object_key) {
>> + Some(result) => result,
>> + None => return Ok(()),
>> + };
>
> comment: the object keys here do also potentially include bad chunks
> with their respective suffix, so the markers would be created for these
> as well, leading to no further good chunks being uploaded and corrupt
> backups.
Thank you for catching this. I think I should either skip marker
creation for bad chunks or create the corresponding .N.bad marker files.
I haven't tested this yet. I'll check and get back.
> Further I think there still is a pre-existing issue with the marker
> files for bad chunks potentially being missing, which would lead to
> removing these from the S3 backend even if no new, good chunk exists.
>
Do you mean the garbage-collection code path in [1]? If I understand
correctly, phase 1 updates existing local .N.bad files while iterating
over the index files [2], but does not recreate missing ones. During
phase 2, if the corresponding local .N.bad file is missing and
clear_chunk_expected_mark() returns false, atime becomes None. This
causes the bad chunk object to be queued for deletion from S3,
even if no good replacement exists.
- [1] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2544
- [2] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2175
> Above might further be written a bit more concise with the chunk path
> helper to already check for bad chunk suffix by:
> ```
> - let (_, digest, _) = match digest_from_object_key(&object_key) {
> - Some(result) => result,
> - None => return Ok(()),
> + let Some((_, digest, bad)) =
> self.chunk_path_from_object_key(&object_key) else {
> + return Ok(());
> ```
>
>> +
>> + unsafe {
>> + chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?;
>> + }
>> +
>> + chunk_marker_count += 1;
>> + if log_throttle.should_log() {
>> + info!("created {chunk_marker_count} chunk markers");
>> + }
>> +
>> + Ok(())
>> + },
>> + )?;
>> +
>> + let elapsed = TimeSpan::from(start.elapsed());
>> + info!("created {chunk_marker_count} chunk markers in {elapsed}");
>> +
>> + Ok(())
>> + }
>> +
>> /// Fetch contents from S3 object store, clear and replace the local cache store contents.
>> /// Returns with error for non-S3 datastore backends.
>> pub async fn s3_refresh(self: &Arc<Self>) -> Result<(), Error> {
next prev parent reply other threads:[~2026-11-17 13:14 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
2026-11-11 15:05 ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
2026-11-11 15:04 ` Christian Ebner
2026-11-17 13:14 ` Shan Shaji [this message]
2026-11-17 13:26 ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
2026-11-11 15:04 ` Christian Ebner
2026-11-16 14:38 ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
2026-11-11 15:05 ` Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com \
--to=s.shaji@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox