[pbs-devel] [PATCH proxmox-backup v3 1/1] garbage collection: track chunk cache stats and show in task log

[Christian Ebner <c.ebner@proxmox.com>]

Count the chunk cache hits and misses and display the resulting
values and the hit ratio in the garbage collection task log summary.

This allows to investigate possible issues and tune cache capacity,
also by being able to compare to other values in the summary such
as the on disk chunk count.

Exemplary output
```
2025-05-16T22:31:53+02:00: Chunk cache: hits 15817, misses 873 (hit ratio 94.77%)
2025-05-16T22:31:53+02:00: Removed garbage: 0 B
2025-05-16T22:31:53+02:00: Removed chunks: 0
2025-05-16T22:31:53+02:00: Original data usage: 64.961 GiB
2025-05-16T22:31:53+02:00: On-Disk usage: 1.037 GiB (1.60%)
2025-05-16T22:31:53+02:00: On-Disk chunks: 874
2025-05-16T22:31:53+02:00: Deduplication factor: 62.66
2025-05-16T22:31:53+02:00: Average chunk size: 1.215 MiB
```

Sidenote: the discrepancy between cache miss counter and on-disk
chunk count in the output shown above can be attributed to the all
zero chunk, inserted during the atime update check at the start of
garbage collection, however not being referenced by any index file in
this examplary case.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 2:
- Refactor and initialize to use GarbageCollectionCacheStats for cache
  stats

 pbs-datastore/src/datastore.rs | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 924d8cf9c..d663465e2 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -23,8 +23,8 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
-    DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
-    MaintenanceMode, MaintenanceType, Operation, UPID,
+    DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionCacheStats,
+    GarbageCollectionStatus, MaintenanceMode, MaintenanceType, Operation, UPID,
 };
 use pbs_config::BackupLockGuard;
 
@@ -1098,8 +1098,14 @@ impl DataStore {
             // Avoid multiple expensive atime updates by utimensat
             if let Some(chunk_lru_cache) = chunk_lru_cache {
                 if chunk_lru_cache.insert(*digest, ()) {
+                    if let Some(cache_stats) = status.cache_stats.as_mut() {
+                        cache_stats.hits += 1;
+                    }
                     continue;
                 }
+                if let Some(cache_stats) = status.cache_stats.as_mut() {
+                    cache_stats.misses += 1;
+                }
             }
 
             if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
@@ -1304,6 +1310,7 @@ impl DataStore {
 
             let mut gc_status = GarbageCollectionStatus {
                 upid: Some(upid.to_string()),
+                cache_stats: Some(GarbageCollectionCacheStats::default()),
                 ..Default::default()
             };
             let tuning: DatastoreTuning = serde_json::from_value(
@@ -1366,6 +1373,17 @@ impl DataStore {
                 worker,
             )?;
 
+            if let Some(cache_stats) = &gc_status.cache_stats {
+                let total_cache_counts = cache_stats.hits + cache_stats.misses;
+                if total_cache_counts > 0 {
+                    let cache_hit_ratio =
+                        (cache_stats.hits as f64 * 100.) / total_cache_counts as f64;
+                    info!(
+                        "Chunk cache: hits {}, misses {} (hit ratio {cache_hit_ratio:.2}%)",
+                        cache_stats.hits, cache_stats.misses,
+                    );
+                }
+            }
             info!(
                 "Removed garbage: {}",
                 HumanByte::from(gc_status.removed_bytes),
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel