all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "Shan Shaji" <s.shaji@proxmox.com>
To: "Christian Ebner" <c.ebner@proxmox.com>, <pbs-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers
Date: Tue, 17 Nov 2026 14:14:32 +0100	[thread overview]
Message-ID: <DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com> (raw)
In-Reply-To: <3f1efc7a-04ed-4dfa-9c8f-2d2eb9aeaa68@proxmox.com>

On Wed Nov 11, 2026 at 4:04 PM CET, Christian Ebner wrote:
> One comment inline.
>
> On 10/19/26 6:04 PM, Shan Shaji wrote:
>> previously, recreating an S3-backed datastore with the --reuse-datastore
>> flag fetched only the metadata files into the local cache, without
>> creating chunk markers. Garbage collection (GC) had to be run to
>> recreate markers for chunks referenced by the backup indexes.
>>
>> In order to fix this, add method to create zero-byte chunk markers by
>> listing the chunk objects in S3 when reusing a datastore.
>>
>> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
>> ---
>>   pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 50 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 7bb0c187c..17fa47cc3 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -13,6 +13,7 @@ use hyper::Method;
>>   use hyper::body::Bytes;
>>   use nix::unistd::{UnlinkatFlags, unlinkat};
>>   use pbs_tools::lru_cache::LruCache;
>> +use proxmox_log::log_throttle::{self, LogThrottle};
>>   use tokio::io::AsyncWriteExt;
>>   use tracing::{info, warn};
>>
>> @@ -3292,6 +3293,55 @@ impl DataStore {
>>           *OLD_LOCKING
>>       }
>>
>> +    /// The caller must hold the maintenance-mode lock with the datastore in
>> +    /// `S3Refresh` mode.
>> +    pub fn create_empty_s3_chunk_markers(
>> +        &self,
>> +        worker: &dyn WorkerTaskContext,
>> +    ) -> Result<(), Error> {
>> +        let DatastoreBackend::S3(s3_client) = self.backend()? else {
>> +            bail!("store '{}' not backed by S3", self.name());
>> +        };
>> +
>> +        let start = std::time::Instant::now();
>> +
>> +        // Holding the chunk store lock for the whole time here  is
>> +        // only acceptable since the maintenance mode lock is held
>> +        // during the entire datastore creation.
>> +        let _guard = self.inner.chunk_store.mutex().lock().unwrap();
>> +
>> +        let mut chunk_marker_count = 0;
>> +        let mut log_throttle =
>> +            LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL);
>> +
>> +        self.paginate_s3_chunk_objects(
>> +            worker,
>> +            s3_client.as_ref(),
>> +            |chunk_store, object_key, _| -> Result<(), Error> {
>> +                let (_, digest, _) = match digest_from_object_key(&object_key) {
>> +                    Some(result) => result,
>> +                    None => return Ok(()),
>> +                };
>
> comment: the object keys here do also potentially include bad chunks
> with their respective suffix, so the markers would be created for these
> as well, leading to no further good chunks being uploaded and corrupt
> backups.

Thank you for catching this. I think I should either skip marker
creation for bad chunks or create the corresponding .N.bad marker files.
I haven't tested this yet. I'll check and get back.

> Further I think there still is a pre-existing issue with the marker
> files for bad chunks potentially being missing, which would lead to
> removing these from the S3 backend even if no new, good chunk exists.
>

Do you mean the garbage-collection code path in [1]? If I understand
correctly, phase 1 updates existing local .N.bad files while iterating
over the index files [2], but does not recreate missing ones. During
phase 2, if the corresponding local .N.bad file is missing and
clear_chunk_expected_mark() returns false, atime becomes None. This
causes the bad chunk object to be queued for deletion from S3,
even if no good replacement exists.

- [1] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2544
- [2] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2175

> Above might further be written a bit more concise with the chunk path
> helper to already check for bad chunk suffix by:
> ```
> - let (_, digest, _) = match digest_from_object_key(&object_key) {
> -     Some(result) => result,
> -     None => return Ok(()),
> + let Some((_, digest, bad)) =
> self.chunk_path_from_object_key(&object_key) else {
> +     return Ok(());
> ```
>
>> +
>> +                unsafe {
>> +                    chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?;
>> +                }
>> +
>> +                chunk_marker_count += 1;
>> +                if log_throttle.should_log() {
>> +                    info!("created {chunk_marker_count} chunk markers");
>> +                }
>> +
>> +                Ok(())
>> +            },
>> +        )?;
>> +
>> +        let elapsed = TimeSpan::from(start.elapsed());
>> +        info!("created {chunk_marker_count} chunk markers in {elapsed}");
>> +
>> +        Ok(())
>> +    }
>> +
>>       /// Fetch contents from S3 object store, clear and replace the local cache store contents.
>>       /// Returns with error for non-S3 datastore backends.
>>       pub async fn s3_refresh(self: &Arc<Self>) -> Result<(), Error> {





  reply	other threads:[~2026-11-17 13:14 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
2026-11-11 15:05   ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-17 13:14     ` Shan Shaji [this message]
2026-11-17 13:26       ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-16 14:38     ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
2026-11-11 15:05   ` Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com \
    --to=s.shaji@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal