From: Christian Ebner <c.ebner@proxmox.com>
To: "Proxmox Backup Server development discussion"
<pbs-devel@lists.proxmox.com>,
"Fabian Grünbichler" <f.gruenbichler@proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox-backup 3/3] GC: S3: phase2: delete last partial batch of objects at the very end
Date: Fri, 21 Nov 2025 10:31:20 +0100 [thread overview]
Message-ID: <e5e652d2-02f2-4a0d-b349-5e4fcd7d76f8@proxmox.com> (raw)
In-Reply-To: <20251121090605.262675-4-f.gruenbichler@proxmox.com>
While going trough the rest of the series in detail now, one idea right
away.
On 11/21/25 10:06 AM, Fabian Grünbichler wrote:
> instead of after every processing every batch of 1000 listed objects. this
> reduces the number of delete calls made to the backend, making regular garbage
> collections that do not delete most objects cheaper, but means holding the
> flocks for garbage chunks/objects longer.
We could avoid holding the flock for to long (e.g. GC over several days
because of super slow local datastore cache, S3 backend, ...) by setting
(or resetting) a timer on each last delete list insert, and not only
using the batch size to decide if to perform the deleteObjects() call,
but rather compare if a timeout has been elapsed.
This would safeguard us from locking some chunks way to long, causing
potential issues with concurrent backups, but not trow out all the
benefits this patch brings.
What do you think? I could send that as followup if you like.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
next prev parent reply other threads:[~2025-11-21 9:31 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-21 9:05 [pbs-devel] [PATCH proxmox-backup 0/3] reduce GC S3 locking Fabian Grünbichler
2025-11-21 9:05 ` [pbs-devel] [PATCH proxmox-backup 1/3] GC: S3: reduce number of open FDs for to-be-deleted objects Fabian Grünbichler
2025-11-21 9:43 ` Christian Ebner
2025-11-21 9:06 ` [pbs-devel] [PATCH proxmox-backup 2/3] GC: S3: factor out batch object deletion Fabian Grünbichler
2025-11-21 9:06 ` [pbs-devel] [PATCH proxmox-backup 3/3] GC: S3: phase2: delete last partial batch of objects at the very end Fabian Grünbichler
2025-11-21 9:31 ` Christian Ebner [this message]
2025-11-21 9:46 ` Fabian Grünbichler
2025-11-21 9:53 ` Christian Ebner
2025-11-21 10:05 ` [pbs-devel] [PATCH proxmox-backup 0/3] reduce GC S3 locking Christian Ebner
2025-11-21 10:19 ` [pbs-devel] superseded: " Fabian Grünbichler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e5e652d2-02f2-4a0d-b349-5e4fcd7d76f8@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=f.gruenbichler@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox