From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 60F621FF172
for <inbox@lore.proxmox.com>; Wed, 16 Apr 2025 16:18:24 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 28FB5D3B;
Wed, 16 Apr 2025 16:18:22 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Wed, 16 Apr 2025 16:17:59 +0200
Message-Id: <20250416141803.479125-1-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.028 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [datastore.rs]
Subject: [pbs-devel] [RFC proxmox-backup 0/4] implement trash can for
snapshots
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
In an effort to simplify the GC phase 1 logic introduced by commit
cb9814e3 ("garbage collection: fix rare race in chunk marking phase")
this patch series implement a trash can functionality for snapshots.
The main intention is to allow snapshot's index files, pruned while
ongoing phase 1 of garbage collection, to be read and their chunks
marked as in use as well. This will allow to get rid of the currently
implemented and rather complex retry looping logic, which could in
theory lead to failing GC or backups when trying to lock the whole
group exclusively following the 10-th retry.
To achieve this, pruning of snapshots does not remove them
immediately, but rather moves them to a `.trash` subfolder in the
datastores base directory. This directory will then be cleared before
starting of GC phase 1, meaning that any index file could be restored
until the next GC run.
This however comes with it's own set of issues, therefore sending
these patches as RFC for now. Open questions and known limitations
are:
- Pruning does not cleanup any space, on the contrary it might
require additional space on COW filesystem. Should there be a flag
to bypass the trash, also given that sometimes users truly want to
remove a snapshot immediately? Although that would re-introduce the
issue with new snapshot ceration and concurrent GC on a last
snapshot.
- Prune + sync + prune might lead to the same snapshot being pruned
multiple times, currently any second prune on a snapshot would
fail. Should this overwrite the trashed snapshot?
- GC might now read the same index twice, once before it was pruned
followed by a prune while phase 1 is still ongoing and the second
time as read from the trash. Not really an issue, but rather a
limitation.
- Further issues I'm currently overlooking
Christian Ebner (4):
datastore: always skip over base directory when listing index files
datastore: allow to specify sub-directory for index file listing
datastore: move snapshots to trash folder on destroy
garbage collection: read pruned snapshot index files from trash
pbs-datastore/src/backup_info.rs | 14 ++-
pbs-datastore/src/datastore.rs | 158 +++++++++++++++----------------
2 files changed, 89 insertions(+), 83 deletions(-)
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel