From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 4CA2A1FF173
for <inbox@lore.proxmox.com>; Mon, 10 Mar 2025 12:17:31 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 41EE1179C1;
Mon, 10 Mar 2025 12:17:22 +0100 (CET)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Mon, 10 Mar 2025 12:16:33 +0100
Message-Id: <20250310111634.162156-4-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250310111634.162156-1-c.ebner@proxmox.com>
References: <20250310111634.162156-1-c.ebner@proxmox.com>
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.031 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] [PATCH v2 proxmox-backup 3/4] garbage collection: allow
to keep track of already touched chunks
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
Implements the `TouchedChunks` struct and methods to keep track of
already touched chunks during garbage collection phase 1, to avoid
multiple computational and I/O intensive atime updates via a syscall.
By inserting a digest, the chunk will be considered as touched and
can be ignored for subsequent encounters. To limit memory usage, the
structure allows to reset the chunk status, flagging them as seen
previous to the reset. A subsequent insert will then flag it as seen
after the reset. Chunks not seen after a reset, will be cleared from
the structure by the next reset call, eliminating them from memory.
This allows to reset the tracking stat after each processes image
index file, to mimic the incremental backup behaviour of known chunks
and limit memory footprint.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- no changes
pbs-datastore/src/datastore.rs | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 72bc9f77f..fdbb33a98 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1585,3 +1585,32 @@ impl DataStore {
Ok(())
}
}
+
+struct TouchedChunks {
+ list: HashMap<[u8; 32], bool>,
+}
+
+impl TouchedChunks {
+ fn new() -> Self {
+ Self {
+ list: HashMap::new(),
+ }
+ }
+
+ // Clear untouched chunks and reset the touched marker for others.
+ fn reset(&mut self) {
+ let mut new_list = HashMap::new();
+ for (digest, touched) in self.list.drain() {
+ if touched {
+ new_list.insert(digest, false);
+ }
+ }
+ self.list = new_list;
+ }
+
+ // Insert the digest in the list of touched chunks.
+ // Returns true if the chunk was already present, false otherwise.
+ fn insert(&mut self, digest: [u8; 32]) -> bool {
+ self.list.insert(digest, true).is_some()
+ }
+}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel