all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: Shan Shaji <s.shaji@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers
Date: Tue, 17 Nov 2026 14:26:07 +0100	[thread overview]
Message-ID: <db5e4467-ccdd-4cf3-883a-d2e88afa1db1@proxmox.com> (raw)
In-Reply-To: <DMXIHLAR8L2E.144KCUPJHPZU6@proxmox.com>

On 11/17/26 2:14 PM, Shan Shaji wrote:
> On Wed Nov 11, 2026 at 4:04 PM CET, Christian Ebner wrote:
>> One comment inline.

[..]

>> comment: the object keys here do also potentially include bad chunks
>> with their respective suffix, so the markers would be created for these
>> as well, leading to no further good chunks being uploaded and corrupt
>> backups.
> 
> Thank you for catching this. I think I should either skip marker
> creation for bad chunks or create the corresponding .N.bad marker files.
> I haven't tested this yet. I'll check and get back.
> 
>> Further I think there still is a pre-existing issue with the marker
>> files for bad chunks potentially being missing, which would lead to
>> removing these from the S3 backend even if no new, good chunk exists.
>>
> 
> Do you mean the garbage-collection code path in [1]? If I understand
> correctly, phase 1 updates existing local .N.bad files while iterating
> over the index files [2], but does not recreate missing ones. During
> phase 2, if the corresponding local .N.bad file is missing and
> clear_chunk_expected_mark() returns false, atime becomes None. This
> causes the bad chunk object to be queued for deletion from S3,
> even if no good replacement exists.
> 
> - [1] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2544
> - [2] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/datastore.rs;h=e6766a3b93c6c350d8ac5150a6e23f86a7659af9;hb=HEAD#l2175

Yes, as mentioned off-list, this is incomplete and should be covered as 
well. bad files should not be cleaned up by GC unless there is a good 
chunk present, so if no good chunk marker does (yet) exist, and the bad 
one does neither, it would be best to create it and avoid cleanup. That 
way we could recover any missing bad chunk markers after a datastore 
recreation from a bucket I think.





  reply	other threads:[~2026-11-17 13:26 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-19 16:03 [PATCH proxmox{-backup,} 0/9] fix #7903: rebuild local S3 chunk markers Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 1/9] refactor: remove local `LogThrottle` in favor of proxmox-log Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 2/9] datastore: factor out s3 chunk objects pagination into a helper Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 3/9] pbs-datastore: inherit proxmox-log crate from workspace Shan Shaji
2026-11-11 15:05   ` Christian Ebner
2026-10-19 16:03 ` [PATCH proxmox-backup 4/9] datastore: add method to create zero-byte chunk markers Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-17 13:14     ` Shan Shaji
2026-11-17 13:26       ` Christian Ebner [this message]
2026-10-19 16:03 ` [PATCH proxmox-backup 5/9] fix #7908: datastore: api: add option " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 6/9] fix #7908: datastore: ui: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 7/9] fix #7908: backup-manager: " Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox-backup 8/9] docs: add explanation regarding ``rebuild-chunk-markers`` option Shan Shaji
2026-11-11 15:04   ` Christian Ebner
2026-11-16 14:38     ` Shan Shaji
2026-10-19 16:03 ` [PATCH proxmox 9/9] proxmox-log: add LogThrottle implementation Shan Shaji
2026-11-11 15:05   ` Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=db5e4467-ccdd-4cf3-883a-d2e88afa1db1@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    --cc=s.shaji@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal