From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 045691FF183 for <inbox@lore.proxmox.com>; Wed, 4 Jun 2025 17:35:19 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3703885F1; Wed, 4 Jun 2025 17:35:38 +0200 (CEST) From: Christian Ebner <c.ebner@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Wed, 4 Jun 2025 17:34:49 +0200 Message-Id: <20250604153449.482640-3-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250604153449.482640-1-c.ebner@proxmox.com> References: <20250604153449.482640-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.032 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup v3 1/1] garbage collection: track chunk cache stats and show in task log X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> Count the chunk cache hits and misses and display the resulting values and the hit ratio in the garbage collection task log summary. This allows to investigate possible issues and tune cache capacity, also by being able to compare to other values in the summary such as the on disk chunk count. Exemplary output ``` 2025-05-16T22:31:53+02:00: Chunk cache: hits 15817, misses 873 (hit ratio 94.77%) 2025-05-16T22:31:53+02:00: Removed garbage: 0 B 2025-05-16T22:31:53+02:00: Removed chunks: 0 2025-05-16T22:31:53+02:00: Original data usage: 64.961 GiB 2025-05-16T22:31:53+02:00: On-Disk usage: 1.037 GiB (1.60%) 2025-05-16T22:31:53+02:00: On-Disk chunks: 874 2025-05-16T22:31:53+02:00: Deduplication factor: 62.66 2025-05-16T22:31:53+02:00: Average chunk size: 1.215 MiB ``` Sidenote: the discrepancy between cache miss counter and on-disk chunk count in the output shown above can be attributed to the all zero chunk, inserted during the atime update check at the start of garbage collection, however not being referenced by any index file in this examplary case. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 2: - Refactor and initialize to use GarbageCollectionCacheStats for cache stats pbs-datastore/src/datastore.rs | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index 924d8cf9c..d663465e2 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -23,8 +23,8 @@ use proxmox_worker_task::WorkerTaskContext; use pbs_api_types::{ ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder, - DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus, - MaintenanceMode, MaintenanceType, Operation, UPID, + DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionCacheStats, + GarbageCollectionStatus, MaintenanceMode, MaintenanceType, Operation, UPID, }; use pbs_config::BackupLockGuard; @@ -1098,8 +1098,14 @@ impl DataStore { // Avoid multiple expensive atime updates by utimensat if let Some(chunk_lru_cache) = chunk_lru_cache { if chunk_lru_cache.insert(*digest, ()) { + if let Some(cache_stats) = status.cache_stats.as_mut() { + cache_stats.hits += 1; + } continue; } + if let Some(cache_stats) = status.cache_stats.as_mut() { + cache_stats.misses += 1; + } } if !self.inner.chunk_store.cond_touch_chunk(digest, false)? { @@ -1304,6 +1310,7 @@ impl DataStore { let mut gc_status = GarbageCollectionStatus { upid: Some(upid.to_string()), + cache_stats: Some(GarbageCollectionCacheStats::default()), ..Default::default() }; let tuning: DatastoreTuning = serde_json::from_value( @@ -1366,6 +1373,17 @@ impl DataStore { worker, )?; + if let Some(cache_stats) = &gc_status.cache_stats { + let total_cache_counts = cache_stats.hits + cache_stats.misses; + if total_cache_counts > 0 { + let cache_hit_ratio = + (cache_stats.hits as f64 * 100.) / total_cache_counts as f64; + info!( + "Chunk cache: hits {}, misses {} (hit ratio {cache_hit_ratio:.2}%)", + cache_stats.hits, cache_stats.misses, + ); + } + } info!( "Removed garbage: {}", HumanByte::from(gc_status.removed_bytes), -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel