From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 78ADA1FF17F
for <inbox@lore.proxmox.com>; Mon, 19 May 2025 07:55:20 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id C45393FEEC;
Mon, 19 May 2025 07:55:44 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Mon, 19 May 2025 07:55:18 +0200
Message-Id: <20250519055518.3747-5-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250519055518.3747-1-c.ebner@proxmox.com>
References: <20250519055518.3747-1-c.ebner@proxmox.com>
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.029 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] [PATCH v2 proxmox-backup 4/4] garbage collection: track
chunk cache stats and show in task log
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
Count the chunk cache hits and misses and display the resulting
values and the hit ratio in the garbage collection task log summary.
This allows to investigate possible issues and tune cache capacity,
also by being able to compare to other values in the summary such
as the on disk chunk count.
Exemplary output
```
2025-05-16T22:31:53+02:00: Chunk cache: hits 15817, misses 873 (hit ratio 94.77%)
2025-05-16T22:31:53+02:00: Removed garbage: 0 B
2025-05-16T22:31:53+02:00: Removed chunks: 0
2025-05-16T22:31:53+02:00: Original data usage: 64.961 GiB
2025-05-16T22:31:53+02:00: On-Disk usage: 1.037 GiB (1.60%)
2025-05-16T22:31:53+02:00: On-Disk chunks: 874
2025-05-16T22:31:53+02:00: Deduplication factor: 62.66
2025-05-16T22:31:53+02:00: Average chunk size: 1.215 MiB
```
Sidenote: the discrepancy between cache miss counter and on-disk
chunk count in the output shown above can be attributed to the all
zero chunk, inserted during the atime update check at the start of
garbage collection, however not being referenced by any index file in
this examplary case.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 1:
- add cache hit ratio to output
pbs-datastore/src/datastore.rs | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index dbff84bf3..fcfa7e694 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1087,8 +1087,10 @@ impl DataStore {
// Avoid multiple expensive atime updates by utimensat
if let Some(chunk_lru_cache) = chunk_lru_cache {
if chunk_lru_cache.insert(*digest, ()) {
+ status.cache_hits += 1;
continue;
}
+ status.cache_misses += 1;
}
if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
@@ -1355,6 +1357,15 @@ impl DataStore {
worker,
)?;
+ let total_cache_counts = gc_status.cache_hits + gc_status.cache_misses;
+ if total_cache_counts > 0 {
+ let cache_hit_ratio =
+ (gc_status.cache_hits as f64 * 100.) / total_cache_counts as f64;
+ info!(
+ "Chunk cache: hits {}, misses {} (hit ratio {cache_hit_ratio:.2}%)",
+ gc_status.cache_hits, gc_status.cache_misses,
+ );
+ }
info!(
"Removed garbage: {}",
HumanByte::from(gc_status.removed_bytes),
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel