[pbs-devel] [PATCH proxmox-backup] GC: chunk store: fix chunk using markers cleanup

* [pbs-devel] [PATCH proxmox-backup] GC: chunk store: fix chunk using markers cleanup
@ 2025-11-25 12:58 Christian Ebner
  2025-11-25 14:01 ` [pbs-devel] superseded: " Christian Ebner
  0 siblings, 1 reply; 2+ messages in thread
From: Christian Ebner @ 2025-11-25 12:58 UTC (permalink / raw)
  To: pbs-devel

Since commit 9510ef1a ("GC: assure chunk exists on s3 store when
creating missing chunk marker") chunks which are referenced by
an index file but do not have a local marker file are marked by a
file with the `using` extension, so they are not cleaned up during
phase 2 if the chunk is still present on the backend.

If the chunk is however not encountered, phase 3 will see the marker
and tries to clean it up, which currently however fails because
it is first tried to be cleaned up from the LRU cache, the filename
being converted to the chunk digest.

Therefore, as stop gap make sure using markers are only cleaned up
by unlinking the file, not by going over the datastore cache.

Fixes: https://forum.proxmox.com/threads/176567/post-819437
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
Note:
Make sure to set gc-atime-cutoff to be low enough to not have to wait
for the default.

 pbs-datastore/src/chunk_store.rs | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index f53460664..f3fe8ec16 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -25,6 +25,8 @@ use crate::file_formats::{
 };
 use crate::{DataBlob, LocalDatastoreLruCache};
 
+const USING_MARKER_FILENAME_EXT: &str = "using";
+
 /// File system based chunk store
 pub struct ChunkStore {
     name: String, // used for error reporting
@@ -440,7 +442,11 @@ impl ChunkStore {
                         || {
                             // non-bad S3 chunks need to be removed via cache
                             if let Some(cache) = cache {
-                                if !bad {
+                                if !bad
+                                    && !filename
+                                        .to_bytes()
+                                        .ends_with(USING_MARKER_FILENAME_EXT.as_bytes())
+                                {
                                     let digest = <[u8; 32]>::from_hex(filename.to_bytes())?;
 
                                     // unless there is a concurrent upload pending,
@@ -776,7 +782,7 @@ impl ChunkStore {
     /// Helper to generate marker file path for expected chunks
     fn chunk_expected_marker_path(&self, digest: &[u8; 32]) -> PathBuf {
         let (mut path, _digest_str) = self.chunk_path(digest);
-        path.set_extension("using");
+        path.set_extension(USING_MARKER_FILENAME_EXT);
         path
     }
 
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 2+ messages in thread