From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 445B61FF17A for ; Tue, 11 Nov 2025 15:30:11 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id ACCC5E16B; Tue, 11 Nov 2025 15:30:55 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 11 Nov 2025 15:30:01 +0100 Message-ID: <20251111143002.759901-19-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251111143002.759901-1-c.ebner@proxmox.com> References: <20251111143002.759901-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1762871396959 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.048 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup v5 18/19] GC: assure chunk exists on s3 store when creating missing chunk marker X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Currently it is not assured the chunk is still present on the s3 object store before re-creating the chunk marker file. That will however lead to the chunk not being re-inserted if re-uploaded. Since checking the presence right away is expensive as it requires additional api requests, mark the chunk as expected instead and delay the existence check to phase 2 which must fetch the chunks anyways. Rely on the per-chunk file locks for consistency. Signed-off-by: Christian Ebner --- pbs-datastore/src/datastore.rs | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index 549bc3b41..bf06d6fda 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -1347,13 +1347,7 @@ impl DataStore { if !self.inner.chunk_store.cond_touch_chunk(digest, false)? && !is_bad { // Insert empty file as marker to tell GC phase2 that this is // a chunk still in-use, so to keep in the S3 object store. - std::fs::File::options() - .write(true) - .create_new(true) - .open(&chunk_path) - .with_context(|| { - format!("failed to create marker for chunk {}", hex::encode(digest)) - })?; + self.inner.chunk_store.mark_chunk_as_expected(digest)?; } } else { let hex = hex::encode(digest); @@ -1683,8 +1677,16 @@ impl DataStore { let atime = match std::fs::metadata(&chunk_path) { Ok(stat) => stat.accessed()?, Err(err) if err.kind() == std::io::ErrorKind::NotFound => { - // File not found, delete by setting atime to unix epoch - SystemTime::UNIX_EPOCH + if self.inner.chunk_store.clear_chunk_expected_mark(&digest)? { + unsafe { + // chunk store lock held + self.inner.chunk_store.replace_chunk_with_marker(&digest)?; + } + SystemTime::now() + } else { + // File not found, delete by setting atime to unix epoch + SystemTime::UNIX_EPOCH + } } Err(err) => return Err(err.into()), }; @@ -1980,7 +1982,8 @@ impl DataStore { ) .map_err(|err| format_err!("failed to upload chunk to s3 backend - {err:#}"))?; tracing::info!("Caching of chunk {}", hex::encode(digest)); - self.cache_insert(&digest, &chunk)?; + self.cache_insert(digest, chunk)?; + self.inner.chunk_store.clear_chunk_expected_mark(digest)?; Ok((is_duplicate, chunk_size)) } -- 2.47.3 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel