[PATCH proxmox{,-backup} v2 00/14] fix #5799: gather datastore usage statistics on per-entity level

[Christian Ebner <c.ebner@proxmox.com>]

Issue #5799 requested to gather and cache information about raw
storage size of uniquely referenced chunks and deduplication factor
for backup groups, with the intent to provide better introspection
for storage optimization by allowing to pruning specific backup
groups/snapshot based on this additional information.

This patches implement the accumulation of such statistics during
garbage collection, by tracking chunk to namespace/group/snapshot
association in memory efficient lookup structures by storing compact
vectors but providing also fast lookups via a hashtable where
required.

During phase 1, snapshots referencing chunk digests are stored
together with their association to namespace, group and snapshot
hierarchy. During phase 2 raw chunk size is collected while iterating
over chunk files.

Finally, the statistics are gathered by accumulating the counts of
each chunk digest for each of the namespace/group/snapshot, taking
advantage of the fast lookup map.

Since this feature requires additional memory during garbage
collection it is provided as opt-in feature, which can be enabled
per-datastore via the `extended-usage-statistics` in the datastore
config.

Additionally, the gathered statistics are stored on the datastore
and can be retrieved via CLI or inspected in a new `Statistics`
tab.

Memory usage was evaluated using the following procedure:
Differences in RSS for a garbage collection run with and without
the statistics gathering was performed.
Max memory usage during garbage collection was measured by taking the
maximum obtained from
`watch -n 1 "ps -p $(pidof proxmox-backup-proxy) -o rss | tail -n 1 | tee -a ps-rss.out"`
and subtracted by the initial RSS values after service restart.
This was performed on 2 independent datastores with different contents
and GC LRU cache disabled by setting to 0:

| Delta RSS (MiB) | index files | chunk count | deduplication factor |
----------------------------------------------------------------------
|          85.890 |         179 |      778005 |                 3.37 |
|          71.378 |         157 |      555392 |                 4.99 |
----------------------------------------------------------------------

Changes since RFC:
This is basically a complete rewrite in order to optimize memory usage
during chunk gathering while keeping lookups fast. Therefore, IndexMap
and IndexSet have been implemented, storing the chunk info and their
association to the respective hierarchical datastore entity in a
compact vector while keeping fast lookup capability via hashes.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=5799


proxmox:

Christian Ebner (4):
  pbs-api-types: avoid reimplementation of is_false helper
  pbs-api-types: add types for fine-grained datastore usage statistics
  pbs-api-types: add schema for datastore usage statistics listing
  pbs-api-types: add flag to control usage statistics collection

 pbs-api-types/src/datastore.rs | 116 +++++++++++++++++++++++++++++++++
 pbs-api-types/src/jobs.rs      |   6 +-
 pbs-api-types/src/lib.rs       |   5 ++
 pbs-api-types/src/user.rs      |   6 +-
 4 files changed, 123 insertions(+), 10 deletions(-)


proxmox-backup:

Christian Ebner (10):
  chunk store: restrict chunk sweep helper method to module parent
  datastore: implement index map for compact chunk usage tracking
  datastore: implement chunk map for collecting chunk usage statistics
  datastore: cleanup persisted statistics during datastore destruction
  fix #5799: GC: track chunk digests and accumulate statistics
  api: config: allow modification of extended-usage-statistics flag
  ui: expose extended usage statistics flag in datastore options
  api: admin: add endpoint to fetch fine-grained datastore usage stats
  bin: manager: allow listing of extended usage statistics via cli
  ui: implement statistics tab for datastore panel

 Cargo.toml                                  |   1 +
 pbs-datastore/Cargo.toml                    |   1 +
 pbs-datastore/src/chunk_map.rs              | 315 ++++++++++++++
 pbs-datastore/src/chunk_store.rs            |  11 +-
 pbs-datastore/src/datastore.rs              |  84 +++-
 pbs-datastore/src/index_map.rs              | 101 +++++
 pbs-datastore/src/lib.rs                    |   2 +
 src/api2/admin/datastore.rs                 |  35 +-
 src/api2/config/datastore.rs                |   9 +
 src/bin/proxmox_backup_manager/datastore.rs |  47 ++
 www/Makefile                                |   1 +
 www/datastore/OptionView.js                 |  19 +
 www/datastore/Panel.js                      |   8 +
 www/datastore/Statistics.js                 | 452 ++++++++++++++++++++
 14 files changed, 1077 insertions(+), 9 deletions(-)
 create mode 100644 pbs-datastore/src/chunk_map.rs
 create mode 100644 pbs-datastore/src/index_map.rs
 create mode 100644 www/datastore/Statistics.js


Summary over all repositories:
  18 files changed, 1200 insertions(+), 19 deletions(-)

-- 
Generated by murpp 0.11.0