[pbs-devel] [RFC proxmox-backup 0/4] fix #5799: Gather per-namespace/group/snapshot storage usage stats

[pbs-devel] [RFC proxmox-backup 0/4] fix #5799: Gather per-namespace/group/snapshot storage usage stats

[Christian Ebner]

Disclaimer: These patches are still in a development state and send
as RFC to discuss implementation details especially with respect to
acceptable required memory footprint and performance limitations.
As is, yhey are not intended for production use yet.

Issue #5799 requested to gather and cache information about raw
storage size of uniquely referenced chunks and deduplication factor
for backup groups, with the intent to provide better introspection
for storage optimization by allowing to pruning specific backup
groups/snapshot based on this additional information.

This patches draft an approach to generate such statistics during
garbage collection, by collecting chunk to namespace/group/snapshot
relations an providing an in-memory reverse mapping from chunk
digests to namespaces/groups/snapshots referencing given chunk. This
reverse mapping would further allow to e.g. mark snapshots as invalid
if referenced chunks are missing.

During phase 1, snapshots referencing chunk digests are stored in a
lookup table. The actual namespace, group and snapshot data is stored
in dedicated indexes, only referenced by the respective key in the
lookup table, with the intent to keep the slot size predictable and
small for better allocation.

During phase 2 raw chunk size is collected while iterating over chunk
files.

Finally, the statistics are gathered by accumulating the counts of
each chunk digest for each of the namespace/group/snapshot, taking
advantage of the lookup map.

Currently, the information is being gathered unconditionally and
logged to the garbage collection task log, but it is planned to make
this opt-in and store gathered data on the namespace/group/snapshot
level, to e.g. be shown on the datastore content listings or a
dedicated content listing.

The following differences in RSS max values were observed via
`watch -n 1 "ps -p $(pidof proxmox-backup-proxy) -o rss | tail -n 1 | tee -a ps-rss.out"`
and compared to initial RSS values after service restart (and GC LRU
cache disabled by setting to 0) on 2 datastores:

| Delta RSS (MiB) | index files | chunk count | deduplication factor |
----------------------------------------------------------------------
|         412.355 |        1125 |      982236 |                14.69 |
|         168.414 |         213 |      598312 |                 5.93 |
----------------------------------------------------------------------

Open questions and ideas to discuss:
- Is the observed memory requirement acceptable if provided as opt-in
  feature? Are there other ideas to further reduce the memory
  footprint? I was pondering about a indirection mapping to group
  digests by common prefix and only store individual suffixes, which
  however only scales better when there is no need to store this as
  hashmap, so not really suitable due to diminished lookup performance.
- Conditionally replace the GC LRU cache by the lookup map if this
  feature is enabled. The digests need to be stored anyways, so it
  would make sense to use it to avoid multiple chunk atime updates
  instead.
- Add a dedicated tab to show the contents independent from the
  current datastore contents? This would reduce the risk of
  misinterpretation as this is no real-time data.
- Add this as dedicated task instead of combining it with garbage
  collection? This would allow to perform information gathering on
  specific sub-namespaces, groups or selected snapshots only.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=5799

proxmox-backup:

Christian Ebner (4):
  chunk store: restrict chunk sweep helper method to module parent
  datastore: add namespace/group/snapshot indices for reverse lookups
  datastore: introduce reverse chunk digest lookup table
  fix #5799: GC: track chunk digests and accumulate statistics

 pbs-datastore/src/chunk_store.rs        |  11 +-
 pbs-datastore/src/datastore.rs          |  46 +++-
 pbs-datastore/src/lib.rs                |   1 +
 pbs-datastore/src/reverse_digest_map.rs | 349 ++++++++++++++++++++++++
 4 files changed, 404 insertions(+), 3 deletions(-)
 create mode 100644 pbs-datastore/src/reverse_digest_map.rs


Summary over all repositories:
  4 files changed, 404 insertions(+), 3 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 1/4] chunk store: restrict chunk sweep helper method to module parent

[Christian Ebner]

This method is only used by garbage collection and must not be called
from outside the pbs-datastore crate.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/chunk_store.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index bd1dc353b..e7e94b29f 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -427,7 +427,7 @@ impl ChunkStore {
         &self.mutex
     }
 
-    pub fn sweep_unused_chunks(
+    pub(super) fn sweep_unused_chunks(
         &self,
         oldest_writer: i64,
         min_atime: i64,
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 2/4] datastore: add namespace/group/snapshot indices for reverse lookups

[Christian Ebner]

Adds in-memory indices for namespace, group and snapshots, storing
them in a hashmap for fast lookup based on a generated 64 bit key.

These structs will be used to reference namespaces, groups and
snapshots based on the respective key (NKey, GKey, SKey) in reverse
chunk digest lookup table for better memory allocation.

While namespaces and groups store the plain contents, the snapshot
index in addition to storing the identifying timestamp only, stores
the namespace and group index key, thereby keeping the datastore's
hierarchy context while minimizing memory requirements.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/lib.rs                |   1 +
 pbs-datastore/src/reverse_digest_map.rs | 104 ++++++++++++++++++++++++
 2 files changed, 105 insertions(+)
 create mode 100644 pbs-datastore/src/reverse_digest_map.rs

diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 1f7c54ae8..82c0e33d6 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -225,6 +225,7 @@ pub use hierarchy::{
     ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive, ListSnapshots,
 };
 
+mod reverse_digest_map;
 mod snapshot_reader;
 pub use snapshot_reader::SnapshotReader;
 
diff --git a/pbs-datastore/src/reverse_digest_map.rs b/pbs-datastore/src/reverse_digest_map.rs
new file mode 100644
index 000000000..323e7315d
--- /dev/null
+++ b/pbs-datastore/src/reverse_digest_map.rs
@@ -0,0 +1,104 @@
+use std::collections::{BTreeMap, HashMap};
+use std::hash::BuildHasher;
+
+use pbs_api_types::{BackupDir, BackupGroup, BackupNamespace};
+
+#[derive(Copy, Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
+/// Backup snapshot index key
+struct SKey(u64);
+
+#[derive(Copy, Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
+/// Backup group index key
+struct GKey(u64);
+
+#[derive(Copy, Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
+/// Backup namespace index key
+struct NKey(u64);
+
+#[derive(Copy, Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
+/// Backup snapshot timestamp
+struct BackupTime(i64);
+
+impl From<i64> for BackupTime {
+    fn from(time: i64) -> Self {
+        Self(time)
+    }
+}
+
+impl From<BackupTime> for i64 {
+    fn from(time: BackupTime) -> i64 {
+        time.0
+    }
+}
+
+#[derive(Default)]
+/// Index to keep track and quickly reference snapshots, associated with the respective group and
+/// namespace indexes.
+struct SnapshotIndex {
+    index: HashMap<SKey, (NKey, GKey, BackupTime)>,
+}
+
+impl SnapshotIndex {
+    /// Index given snapshot and associate to group and namespace by their respective index keys,
+    /// if not already present.
+    ///
+    /// Returns the associated index key.
+    fn insert(&mut self, snapshot_time: BackupTime, nkey: NKey, gkey: GKey) -> SKey {
+        let hasher = self.index.hasher();
+        let key = SKey(hasher.hash_one((nkey, gkey, snapshot_time)));
+        self.index.entry(key).or_insert((nkey, gkey, snapshot_time));
+        key
+    }
+
+    /// Lookup backup snapshot timestamp by key in index, returning also associated namespace and
+    /// group index keys.
+    fn lookup(&self, key: &SKey) -> Option<&(NKey, GKey, BackupTime)> {
+        self.index.get(key)
+    }
+}
+
+#[derive(Default)]
+/// Index to keep track and quickly reference backup groups.
+struct GroupIndex {
+    index: HashMap<GKey, BackupGroup>,
+}
+
+impl GroupIndex {
+    /// Index given backup group if not already present.
+    ///
+    /// Returns the associated index key.
+    fn insert(&mut self, group: BackupGroup) -> GKey {
+        let hasher = self.index.hasher();
+        let key = GKey(hasher.hash_one(&group));
+        self.index.entry(key).or_insert(group);
+        key
+    }
+
+    /// Lookup backup group by key in index
+    fn lookup(&self, key: &GKey) -> Option<&BackupGroup> {
+        self.index.get(key)
+    }
+}
+
+#[derive(Default)]
+/// Index to keep track and quickly reference backup groups.
+struct NamespaceIndex {
+    index: HashMap<NKey, BackupNamespace>,
+}
+
+impl NamespaceIndex {
+    /// Index given backup namespace if not already present.
+    ///
+    /// Returns the associated index key.
+    fn insert(&mut self, namespace: BackupNamespace) -> NKey {
+        let hasher = self.index.hasher();
+        let key = NKey(hasher.hash_one(&namespace));
+        self.index.entry(key).or_insert(namespace);
+        key
+    }
+
+    /// Lookup backup namespace by key in index
+    fn lookup(&self, key: &NKey) -> Option<&BackupNamespace> {
+        self.index.get(key)
+    }
+}
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 3/4] datastore: introduce reverse chunk digest lookup table

[Christian Ebner]

Store the [digest]->[namespace]->[group]->[snapshot] relation
by inserting the items in the respective indices, generating their
key and store the referencing keys in a hashmap, allowing for
easy lookup. This will be used to track chunk references of snapshot
index files during phase 1 of garbage collection and accumulate
statistics after filling relevant raw chunk size information during
phase 2 of garbage collection.

The actual hierarchy is only reconstructed for each chunk digest on
accumulation, allowing to identify unique chunks and perform
chunk reference counting.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/reverse_digest_map.rs | 245 ++++++++++++++++++++++++
 1 file changed, 245 insertions(+)

diff --git a/pbs-datastore/src/reverse_digest_map.rs b/pbs-datastore/src/reverse_digest_map.rs
index 323e7315d..1c3efa8a8 100644
--- a/pbs-datastore/src/reverse_digest_map.rs
+++ b/pbs-datastore/src/reverse_digest_map.rs
@@ -1,7 +1,10 @@
 use std::collections::{BTreeMap, HashMap};
 use std::hash::BuildHasher;
 
+use tracing::info;
+
 use pbs_api_types::{BackupDir, BackupGroup, BackupNamespace};
+use proxmox_human_byte::HumanByte;
 
 #[derive(Copy, Clone, Eq, Hash, PartialEq, PartialOrd, Ord)]
 /// Backup snapshot index key
@@ -102,3 +105,245 @@ impl NamespaceIndex {
         self.index.get(key)
     }
 }
+
+#[derive(Default)]
+/// Store metadata associated with given digest such as raw chunk size and
+/// reference counted list of snapshot keys referencing given chunk.
+struct DigestMetadata {
+    raw_size: u64,
+    logical_size: u64,
+    refcount: BTreeMap<SKey, u32>,
+}
+
+impl DigestMetadata {
+    fn from_skey_with_logical_size(skey: SKey, logical_size: u64) -> Self {
+        let mut refcount = BTreeMap::new();
+        refcount.insert(skey, 1);
+        Self {
+            raw_size: 0,
+            logical_size,
+            refcount,
+        }
+    }
+}
+
+#[derive(Default)]
+pub(super) struct ReverseDigestMap {
+    // Map associating digest to snapshot index with reference count (digest metadata)
+    digests: HashMap<[u8; 32], DigestMetadata>,
+    // indexes storing the actual namespace, group and snapshot data
+    namespaces: NamespaceIndex,
+    groups: GroupIndex,
+    snapshots: SnapshotIndex,
+}
+
+impl ReverseDigestMap {
+    /// Associate digest or update reference count for backup snapshot of given namespace.
+    ///
+    /// Returns [`true`] if the digest key was not yet present.
+    pub(super) fn insert(
+        &mut self,
+        digest: &[u8; 32],
+        namespace: &BackupNamespace,
+        snapshot: &BackupDir,
+        logical_size: u64,
+    ) -> bool {
+        let mut is_new = true;
+        let nkey = self.namespaces.insert(namespace.clone());
+        let gkey = self.groups.insert(snapshot.group.clone());
+        let skey = self
+            .snapshots
+            .insert(BackupTime::from(snapshot.time), nkey, gkey);
+
+        self.digests
+            .entry(*digest)
+            .and_modify(|digest_info| {
+                is_new = false;
+                digest_info
+                    .refcount
+                    .entry(skey)
+                    .and_modify(|count| *count += 1)
+                    .or_insert(1);
+                digest_info.logical_size = logical_size;
+            })
+            .or_insert(DigestMetadata::from_skey_with_logical_size(
+                skey,
+                logical_size,
+            ));
+
+        is_new
+    }
+
+    /// Set the raw chunk size metadata information for associated digest.
+    ///
+    /// Inserts the digest without snapshots associated if not present.
+    pub(super) fn set_raw_chunk_size(&mut self, digest: &[u8; 32], raw_size: u64) {
+        self.digests
+            .entry(*digest)
+            .and_modify(|digest_info| digest_info.raw_size = raw_size);
+    }
+}
+
+#[derive(Debug, Default)]
+pub(super) struct AccumulatedStats {
+    raw_size: u64,
+    logical_size: u64,
+    unique_raw_size: u64,
+    refcount: u64,
+    uniquecount: u64,
+    dedupcount: u64,
+}
+
+// Hierarchy for snapshots on individual digest: [NKey] -> [GKey] -> [(SKey, count)]
+type DigestHierarchy = HashMap<NKey, HashMap<GKey, Vec<(SKey, u32)>>>;
+
+fn format_namespace(namespace: &BackupNamespace) -> String {
+    if namespace.is_root() {
+        "[root]".to_string()
+    } else {
+        namespace.display_as_path().to_string()
+    }
+}
+
+#[derive(Default)]
+pub(super) struct DigestStatAccumulator {
+    list: HashMap<String, AccumulatedStats>,
+}
+
+impl DigestStatAccumulator {
+    // Accumulate statistics by consuming the digest map
+    pub(super) fn accumulate_and_list(mut self, digest_map: ReverseDigestMap) {
+        let mut datastore_stats = AccumulatedStats::default();
+
+        for digest_info in digest_map.digests.into_values() {
+            if digest_info.refcount.is_empty() {
+                continue;
+            }
+
+            // Generate [namespaces] -> [groups] -> [snapshots] reference hierarchy for given digest
+            // to account for statistics. The hierarchy is then used to account for chunk uniqueness
+            // and raw size.
+            let mut hierarchy: DigestHierarchy = HashMap::new();
+            for (skey, count) in digest_info.refcount {
+                if let Some((nkey, gkey, _snapshot_time)) = digest_map.snapshots.lookup(&skey) {
+                    hierarchy
+                        .entry(*nkey)
+                        .or_default()
+                        .entry(*gkey)
+                        .or_default()
+                        .push((skey, count));
+                }
+            }
+
+            let mut unique = hierarchy.len() <= 1;
+            let raw_size = digest_info.raw_size;
+            let logical_size = digest_info.logical_size;
+
+            for (nkey, gkey_map) in hierarchy {
+                if gkey_map.len() > 1 {
+                    unique = false;
+                }
+
+                let namespace = digest_map.namespaces.lookup(&nkey).unwrap();
+                let mut logical_namespace_sum = 0;
+
+                for (gkey, skey_list) in gkey_map {
+                    if skey_list.len() > 1 {
+                        unique = false;
+                    }
+
+                    let group = digest_map.groups.lookup(&gkey).unwrap();
+                    let mut logical_group_sum = 0;
+
+                    for (skey, duplicates) in skey_list {
+                        if duplicates > 1 {
+                            unique = false;
+                        }
+
+                        let (_nkey, _gkey, timestamp) = digest_map.snapshots.lookup(&skey).unwrap();
+                        let snapshot =
+                            BackupDir::from((group.clone(), Into::<i64>::into(*timestamp)));
+
+                        let snapshot = format!("{}/{snapshot}", format_namespace(namespace));
+                        let logical_snapshot_sum = duplicates as u64 * logical_size;
+                        self.account(snapshot, raw_size, logical_snapshot_sum, unique);
+                        logical_group_sum += logical_snapshot_sum;
+                    }
+
+                    let group = format!("{}/{group}", format_namespace(namespace));
+                    self.account(group, raw_size, logical_group_sum, unique);
+                    logical_namespace_sum += logical_group_sum;
+                }
+
+                let namespace = format_namespace(namespace);
+                self.account(namespace, raw_size, logical_namespace_sum, unique);
+            }
+
+            datastore_stats.raw_size += raw_size;
+            datastore_stats.refcount += 1;
+            if unique {
+                datastore_stats.uniquecount += 1;
+                datastore_stats.unique_raw_size += raw_size;
+            } else {
+                datastore_stats.dedupcount += 1;
+            }
+        }
+
+        let mut list: Vec<(String, AccumulatedStats)> = self.list.into_iter().collect();
+        list.sort_unstable_by(|a, b| a.0.cmp(&b.0));
+
+        for (name, stats) in list {
+            info!(
+                "{name}: {} ({} referenced / {} unique ({}) / {} deduped chunks), deduplication factor: {:.2}",
+                HumanByte::from(stats.raw_size),
+                stats.refcount,
+                stats.uniquecount,
+                HumanByte::from(stats.unique_raw_size),
+                stats.dedupcount,
+                Self::dedup_factor(stats.raw_size, stats.logical_size),
+            );
+        }
+
+        info!("===");
+        info!(
+            "Datastore: {} ({} referenced / {} unique / {} deduped chunks)",
+            HumanByte::from(datastore_stats.raw_size),
+            datastore_stats.refcount,
+            datastore_stats.uniquecount,
+            datastore_stats.dedupcount,
+        );
+        info!("===");
+    }
+
+    /// Account for statistics on given key, inserted if not yet present
+    fn account(&mut self, key: String, raw_size: u64, logical_size: u64, unique: bool) {
+        let (uniquecount, dedupcount, unique_raw_size) =
+            if unique { (1, 0, raw_size) } else { (0, 1, 0) };
+        self.list
+            .entry(key)
+            .and_modify(|stats| {
+                stats.raw_size += raw_size;
+                stats.unique_raw_size += unique_raw_size;
+                stats.logical_size += logical_size;
+                stats.refcount += 1;
+                stats.uniquecount += uniquecount;
+                stats.dedupcount += dedupcount;
+            })
+            .or_insert(AccumulatedStats {
+                raw_size,
+                unique_raw_size,
+                logical_size,
+                refcount: 1,
+                uniquecount,
+                dedupcount,
+            });
+    }
+
+    fn dedup_factor(raw_size: u64, logical_size: u64) -> f64 {
+        if raw_size > 0 {
+            logical_size as f64 / raw_size as f64
+        } else {
+            1.0
+        }
+    }
+}
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup 4/4] fix #5799: GC: track chunk digests and accumulate statistics

[Christian Ebner]

Keep track of all digests referenced by snapshots index files
encountered during phase 1 of garbage collection in the reverse
lookup table and fill in raw chunk size information during phase 2.

Allows to finally gather the unique count and raw size information
printed at the end of garbage collection.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/src/chunk_store.rs |  9 +++++++
 pbs-datastore/src/datastore.rs   | 46 ++++++++++++++++++++++++++++++--
 2 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index e7e94b29f..3bf21e1eb 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -434,6 +434,7 @@ impl ChunkStore {
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
         cache: Option<&LocalDatastoreLruCache>,
+        mut digest_map: Option<&mut crate::reverse_digest_map::ReverseDigestMap>,
     ) -> Result<(), Error> {
         // unwrap: only `None` in unit tests
         assert!(self.locker.is_some());
@@ -524,6 +525,14 @@ impl ChunkStore {
                         },
                     )?;
                 }
+
+                // Chunk info not inserted if no already present in mapping
+                if chunk_ext == ChunkExt::None {
+                    if let Some(ref mut map) = digest_map {
+                        let digest = <[u8; 32]>::from_hex(filename.to_bytes())?;
+                        map.set_raw_chunk_size(&digest, stat.st_size as u64);
+                    }
+                }
             }
             drop(lock);
         }
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7ad3d917d..7efa15335 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -45,6 +45,7 @@ use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
 use crate::hierarchy::{ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive};
 use crate::index::IndexFile;
+use crate::reverse_digest_map::{DigestStatAccumulator, ReverseDigestMap};
 use crate::s3::S3_CONTENT_PREFIX;
 use crate::task_tracking::{self, update_active_operations};
 use crate::{DataBlob, LocalDatastoreLruCache};
@@ -1433,6 +1434,7 @@ impl DataStore {
         index: Box<dyn IndexFile>,
         file_name: &Path, // only used for error reporting
         chunk_lru_cache: &mut Option<LruCache<[u8; 32], ()>>,
+        mut digest_map: Option<ReverseMap>,
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
         s3_client: Option<Arc<S3Client>>,
@@ -1443,7 +1445,13 @@ impl DataStore {
         for pos in 0..index.index_count() {
             worker.check_abort()?;
             worker.fail_on_shutdown()?;
-            let digest = index.index_digest(pos).unwrap();
+            let chunk_info = index.chunk_info(pos).unwrap();
+            let digest = &chunk_info.digest;
+
+            if let Some(map) = digest_map.as_mut() {
+                map.digests
+                    .insert(digest, map.namespace, map.snapshot, chunk_info.size());
+            }
 
             // Avoid multiple expensive atime updates by utimensat
             if let Some(chunk_lru_cache) = chunk_lru_cache {
@@ -1493,6 +1501,7 @@ impl DataStore {
         worker: &dyn WorkerTaskContext,
         cache_capacity: usize,
         s3_client: Option<Arc<S3Client>>,
+        mut digest_map: Option<&mut ReverseDigestMap>,
     ) -> Result<(), Error> {
         // Iterate twice over the datastore to fetch index files, even if this comes with an
         // additional runtime cost:
@@ -1522,7 +1531,7 @@ impl DataStore {
             .context("creating namespace iterator failed")?
         {
             let namespace = namespace.context("iterating namespaces failed")?;
-            for group in arc_self.iter_backup_groups(namespace)? {
+            for group in arc_self.iter_backup_groups(namespace.clone())? {
                 let group = group.context("iterating backup groups failed")?;
 
                 // Avoid race between listing/marking of snapshots by GC and pruning the last
@@ -1580,10 +1589,21 @@ impl DataStore {
                                 }
                             };
 
+                            let digest_map = if let Some(digests) = digest_map.as_mut() {
+                                Some(ReverseMap {
+                                    digests,
+                                    namespace: &namespace,
+                                    snapshot: snapshot.backup_dir.dir(),
+                                })
+                            } else {
+                                None
+                            };
+
                             self.index_mark_used_chunks(
                                 index,
                                 &path,
                                 &mut chunk_lru_cache,
+                                digest_map,
                                 status,
                                 worker,
                                 s3_client.as_ref().cloned(),
@@ -1625,6 +1645,7 @@ impl DataStore {
                 index,
                 &path,
                 &mut chunk_lru_cache,
+                None,
                 status,
                 worker,
                 s3_client.as_ref().cloned(),
@@ -1766,11 +1787,14 @@ impl DataStore {
 
         info!("Start GC phase1 (mark used chunks)");
 
+        let mut digest_map = Some(ReverseDigestMap::default());
+
         self.mark_used_chunks(
             &mut gc_status,
             worker,
             gc_cache_capacity,
             s3_client.as_ref().cloned(),
+            digest_map.as_mut(),
         )
         .context("marking used chunks failed")?;
 
@@ -1796,6 +1820,10 @@ impl DataStore {
                             None => continue,
                         };
 
+                    if let Some(map) = digest_map.as_mut() {
+                        map.set_raw_chunk_size(&digest, content.size);
+                    }
+
                     let timeout = std::time::Duration::from_secs(0);
                     let _chunk_guard = match self.inner.chunk_store.lock_chunk(&digest, timeout) {
                         Ok(guard) => guard,
@@ -1892,6 +1920,7 @@ impl DataStore {
                 &mut tmp_gc_status,
                 worker,
                 self.cache(),
+                None,
             )?;
         } else {
             self.inner.chunk_store.sweep_unused_chunks(
@@ -1900,6 +1929,7 @@ impl DataStore {
                 &mut gc_status,
                 worker,
                 None,
+                digest_map.as_mut(),
             )?;
         }
 
@@ -1913,6 +1943,12 @@ impl DataStore {
                 );
             }
         }
+
+        if let Some(digest_map) = digest_map.take() {
+            let accumulator = DigestStatAccumulator::default();
+            accumulator.accumulate_and_list(digest_map);
+        }
+
         info!(
             "Removed garbage: {}",
             HumanByte::from(gc_status.removed_bytes),
@@ -2877,3 +2913,9 @@ impl S3DeleteList {
         Ok(())
     }
 }
+
+struct ReverseMap<'a> {
+    digests: &'a mut ReverseDigestMap,
+    namespace: &'a pbs_api_types::BackupNamespace,
+    snapshot: &'a pbs_api_types::BackupDir,
+}
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel