public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Shan Shaji" <s.shaji@proxmox.com>
To: "Christian Ebner" <c.ebner@proxmox.com>, <pbs-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers
Date: Tue, 17 Nov 2026 14:14:32 +0100	[thread overview]
Message-ID: <DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com> (raw)
In-Reply-To: <3f1efc7a-04ed-4dfa-9c8f-2d2eb9aeaa68@proxmox.com>

On Wed Nov 11, 2026 at 4:04 PM CET, Christian Ebner wrote:
> One comment inline.
>
> On 10/19/26 6:04 PM, Shan Shaji wrote:
>> previously, recreating an S3-backed datastore with the --reuse-datastore
>> flag fetched only the metadata files into the local cache, without
>> creating chunk markers. Garbage collection (GC) had to be run to
>> recreate markers for chunks referenced by the backup indexes.
>>
>> In order to fix this, add method to create zero-byte chunk markers by
>> listing the chunk objects in S3 when reusing a datastore.
>>
>> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
>> ---
>>   pbs-datastore/src/datastore.rs | 50 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 50 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 7bb0c187c..17fa47cc3 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -13,6 +13,7 @@ use hyper::Method;
>>   use hyper::body::Bytes;
>>   use nix::unistd::{UnlinkatFlags, unlinkat};
>>   use pbs_tools::lru_cache::LruCache;
>> +use proxmox_log::log_throttle::{self, LogThrottle};
>>   use tokio::io::AsyncWriteExt;
>>   use tracing::{info, warn};
>>
>> @@ -3292,6 +3293,55 @@ impl DataStore {
>>           *OLD_LOCKING
>>       }
>>
>> +    /// The caller must hold the maintenance-mode lock with the datastore in
>> +    /// `S3Refresh` mode.
>> +    pub fn create_empty_s3_chunk_markers(
>> +        &self,
>> +        worker: &dyn WorkerTaskContext,
>> +    ) -> Result<(), Error> {
>> +        let DatastoreBackend::S3(s3_client) = self.backend()? else {
>> +            bail!("store '{}' not backed by S3", self.name());
>> +        };
>> +
>> +        let start = std::time::Instant::now();
>> +
>> +        // Holding the chunk store lock for the whole time here  is
>> +        // only acceptable since the maintenance mode lock is held
>> +        // during the entire datastore creation.
>> +        let _guard = self.inner.chunk_store.mutex().lock().unwrap();
>> +
>> +        let mut chunk_marker_count = 0;
>> +        let mut log_throttle =
>> +            LogThrottle::new(log_throttle::INITIAL_INTERVAL, log_throttle::MAX_INTERVAL);
>> +
>> +        self.paginate_s3_chunk_objects(
>> +            worker,
>> +            s3_client.as_ref(),
>> +            |chunk_store, object_key, _| -> Result<(), Error> {
>> +                let (_, digest, _) = match digest_from_object_key(&object_key) {
>> +                    Some(result) => result,
>> +                    None => return Ok(()),
>> +                };
>
> comment: the object keys here do also potentially include bad chunks
> with their respective suffix, so the markers would be created for these
> as well, leading to no further good chunks being uploaded and corrupt
> backups.

Thank you for catching this. I think I should either skip marker
creation for bad chunks or create the corresponding .N.bad marker files.
I haven't tested this yet. I'll check and get back.

> Further I think there still is a pre-existing issue with the marker
> files for bad chunks potentially being missing, which would lead to
> removing these from the S3 backend even if no new, good chunk exists.
>

Do you mean the garbage-collection code path in [1]? If I understand
correctly, phase 1 updates existing local .N.bad files while iterating
over the index files [2], but does not recreate missing ones. During
phase 2, if the corresponding local .N.bad file is missing and
clear_chunk_expected_mark() returns false, atime becomes None. This
causes the bad chunk object to be queued for deletion from S3,
even if no good replacement exists.

- [1] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2544
- [2] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2175

> Above might further be written a bit more concise with the chunk path
> helper to already check for bad chunk suffix by:
> ```
> - let (_, digest, _) = match digest_from_object_key(&object_key) {
> -     Some(result) => result,
> -     None => return Ok(()),
> + let Some((_, digest, bad)) =
> self.chunk_path_from_object_key(&object_key) else {
> +     return Ok(());
> ```
>
>> +
>> +                unsafe {
>> +                    chunk_store.replace_chunk_with_marker_or_create_marker(&digest)?;
>> +                }
>> +
>> +                chunk_marker_count += 1;
>> +                if log_throttle.should_log() {
>> +                    info!("created {chunk_marker_count} chunk markers");
>> +                }
>> +
>> +                Ok(())
>> +            },
>> +        )?;
>> +
>> +        let elapsed = TimeSpan::from(start.elapsed());
>> +        info!("created {chunk_marker_count} chunk markers in {elapsed}");
>> +
>> +        Ok(())
>> +    }
>> +
>>       /// Fetch contents from S3 object store, clear and replace the local cache store contents.
>>       /// Returns with error for non-S3 datastore backends.
>>       pub async fn s3_refresh(self: &Arc<Self>) -> Result<(), Error> {





  reply	other threads:[~2026-11-17 13:14 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
2026-11-11 15:05   ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-17 13:14     ` Shan Shaji [this message]
2026-11-17 13:26       ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-16 14:38     ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
2026-11-11 15:05   ` Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com \
    --to=s.shaji@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal