From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 007E91FF0B3 for ; Wed, 14 Oct 2026 16:44:16 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D4D6021570; Wed, 14 Oct 2026 16:44:06 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{,-backup} v2 00/14] fix #5799: gather datastore usage statistics on per-entity level Date: Wed, 14 Oct 2026 16:43:43 +0200 Message-ID: <20261014144357.771051-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1791989040618 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.134 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: CTIHREN7IICHGICZ3W7TPI35NPKW52DC X-Message-ID-Hash: CTIHREN7IICHGICZ3W7TPI35NPKW52DC X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Issue #5799 requested to gather and cache information about raw storage size of uniquely referenced chunks and deduplication factor for backup groups, with the intent to provide better introspection for storage optimization by allowing to pruning specific backup groups/snapshot based on this additional information. This patches implement the accumulation of such statistics during garbage collection, by tracking chunk to namespace/group/snapshot association in memory efficient lookup structures by storing compact vectors but providing also fast lookups via a hashtable where required. During phase 1, snapshots referencing chunk digests are stored together with their association to namespace, group and snapshot hierarchy. During phase 2 raw chunk size is collected while iterating over chunk files. Finally, the statistics are gathered by accumulating the counts of each chunk digest for each of the namespace/group/snapshot, taking advantage of the fast lookup map. Since this feature requires additional memory during garbage collection it is provided as opt-in feature, which can be enabled per-datastore via the `extended-usage-statistics` in the datastore config. Additionally, the gathered statistics are stored on the datastore and can be retrieved via CLI or inspected in a new `Statistics` tab. Memory usage was evaluated using the following procedure: Differences in RSS for a garbage collection run with and without the statistics gathering was performed. Max memory usage during garbage collection was measured by taking the maximum obtained from `watch -n 1 "ps -p $(pidof proxmox-backup-proxy) -o rss | tail -n 1 | tee -a ps-rss.out"` and subtracted by the initial RSS values after service restart. This was performed on 2 independent datastores with different contents and GC LRU cache disabled by setting to 0: | Delta RSS (MiB) | index files | chunk count | deduplication factor | ---------------------------------------------------------------------- | 85.890 | 179 | 778005 | 3.37 | | 71.378 | 157 | 555392 | 4.99 | ---------------------------------------------------------------------- Changes since RFC: This is basically a complete rewrite in order to optimize memory usage during chunk gathering while keeping lookups fast. Therefore, IndexMap and IndexSet have been implemented, storing the chunk info and their association to the respective hierarchical datastore entity in a compact vector while keeping fast lookup capability via hashes. Link to the bugtracker issue: https://bugzilla.proxmox.com/show_bug.cgi?id=5799 proxmox: Christian Ebner (4): pbs-api-types: avoid reimplementation of is_false helper pbs-api-types: add types for fine-grained datastore usage statistics pbs-api-types: add schema for datastore usage statistics listing pbs-api-types: add flag to control usage statistics collection pbs-api-types/src/datastore.rs | 116 +++++++++++++++++++++++++++++++++ pbs-api-types/src/jobs.rs | 6 +- pbs-api-types/src/lib.rs | 5 ++ pbs-api-types/src/user.rs | 6 +- 4 files changed, 123 insertions(+), 10 deletions(-) proxmox-backup: Christian Ebner (10): chunk store: restrict chunk sweep helper method to module parent datastore: implement index map for compact chunk usage tracking datastore: implement chunk map for collecting chunk usage statistics datastore: cleanup persisted statistics during datastore destruction fix #5799: GC: track chunk digests and accumulate statistics api: config: allow modification of extended-usage-statistics flag ui: expose extended usage statistics flag in datastore options api: admin: add endpoint to fetch fine-grained datastore usage stats bin: manager: allow listing of extended usage statistics via cli ui: implement statistics tab for datastore panel Cargo.toml | 1 + pbs-datastore/Cargo.toml | 1 + pbs-datastore/src/chunk_map.rs | 315 ++++++++++++++ pbs-datastore/src/chunk_store.rs | 11 +- pbs-datastore/src/datastore.rs | 84 +++- pbs-datastore/src/index_map.rs | 101 +++++ pbs-datastore/src/lib.rs | 2 + src/api2/admin/datastore.rs | 35 +- src/api2/config/datastore.rs | 9 + src/bin/proxmox_backup_manager/datastore.rs | 47 ++ www/Makefile | 1 + www/datastore/OptionView.js | 19 + www/datastore/Panel.js | 8 + www/datastore/Statistics.js | 452 ++++++++++++++++++++ 14 files changed, 1077 insertions(+), 9 deletions(-) create mode 100644 pbs-datastore/src/chunk_map.rs create mode 100644 pbs-datastore/src/index_map.rs create mode 100644 www/datastore/Statistics.js Summary over all repositories: 18 files changed, 1200 insertions(+), 19 deletions(-) -- Generated by murpp 0.11.0