[pbs-devel] [PATCH proxmox proxmox-backup 0/2] add GC chunk cache stats

[pbs-devel] [PATCH proxmox proxmox-backup 0/2] add GC chunk cache stats

[Christian Ebner]

In an effort to give some more insights into phase one of garbage
collection, add counters for hits and misses of the chunk cache to
the garbage collection status and output the obtained values in the
garbage collection task log.

This will allow to easier investigate possible issues and adapt the
tuning parameters based on the output.

proxmox:

Christian Ebner (1):
  pbs api types: extend garbage collection status by cache stats

 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

proxmox-backup:

Christian Ebner (1):
  garbage collection: track chunk cache stats and show in task log

 pbs-datastore/src/datastore.rs | 6 ++++++
 1 file changed, 6 insertions(+)

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox 1/1] pbs api types: extend garbage collection status by cache stats

[Christian Ebner]

Add the number of cache hits and cache misses encountered during
phase 1 of garbage collection in order to display this information
in the garbage collection task log summary.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..4fb1eb80 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1459,6 +1459,10 @@ pub struct GarbageCollectionStatus {
     pub removed_bad: usize,
     /// Number of chunks still marked as .bad after garbage collection.
     pub still_bad: usize,
+    /// Number of atime update cache hits
+    pub cache_hits: usize,
+    /// Number of atime update cache misses
+    pub cache_misses: usize,
 }
 
 #[api(
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 2/2] garbage collection: track chunk cache stats and show in task log

[Christian Ebner]

Count the chunk cache hits and misses and display the resulting
values in the garbage collection task log summary.

This allows to investigate possible issues and tune cache capacity,
also by being able to compare to other values in the summary such
as the on disk chunk count.

Exemplary output
```
2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017
2025-05-16T10:41:29+02:00: Removed garbage: 3.384 GiB
2025-05-16T10:41:29+02:00: Removed chunks: 2835
2025-05-16T10:41:29+02:00: Original data usage: 155.165 GiB
2025-05-16T10:41:29+02:00: On-Disk usage: 7.312 GiB (4.71%)
2025-05-16T10:41:29+02:00: On-Disk chunks: 10017
2025-05-16T10:41:29+02:00: Deduplication factor: 21.22
2025-05-16T10:41:29+02:00: Average chunk size: 765.438 KiB
```

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/datastore.rs | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..479b76cd8 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1086,8 +1086,10 @@ impl DataStore {
 
             // Avoid multiple expensive atime updates by utimensat
             if chunk_lru_cache.insert(*digest, ()) {
+                status.cache_hits += 1;
                 continue;
             }
+            status.cache_misses += 1;
 
             if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
                 let hex = hex::encode(digest);
@@ -1349,6 +1351,10 @@ impl DataStore {
                 worker,
             )?;
 
+            info!(
+                "Chunk cache: hits {}, misses {}",
+                gc_status.cache_hits, gc_status.cache_misses,
+            );
             info!(
                 "Removed garbage: {}",
                 HumanByte::from(gc_status.removed_bytes),
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup 2/2] garbage collection: track chunk cache stats and show in task log

[Lukas Wagner]

On  2025-05-16 10:58, Christian Ebner wrote:
> Count the chunk cache hits and misses and display the resulting
> values in the garbage collection task log summary.
> 
> This allows to investigate possible issues and tune cache capacity,
> also by being able to compare to other values in the summary such
> as the on disk chunk count.
> 
> Exemplary output
> ```
> 2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017

I think it would be nice to also print the hit ratio :) For the
example here it should be 79%, if my math is correct.


-- 
- Lukas



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup 2/2] garbage collection: track chunk cache stats and show in task log

[Christian Ebner]

On 5/16/25 11:54, Lukas Wagner wrote:
> 
> 
> On  2025-05-16 10:58, Christian Ebner wrote:
>> Count the chunk cache hits and misses and display the resulting
>> values in the garbage collection task log summary.
>>
>> This allows to investigate possible issues and tune cache capacity,
>> also by being able to compare to other values in the summary such
>> as the on disk chunk count.
>>
>> Exemplary output
>> ```
>> 2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017
> 
> I think it would be nice to also print the hit ratio :) For the
> example here it should be 79%, if my math is correct.

Sure, can add that as well as it is indeed more straight forward and 
intuitive!


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] superseded: [PATCH proxmox proxmox-backup 0/2] add GC chunk cache stats

[Christian Ebner]

superseded-by version 2:
https://lore.proxmox.com/pbs-devel/20250519055518.3747-1-c.ebner@proxmox.com/T/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel