From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 8D9C71FF191 for ; Tue, 4 Nov 2025 14:07:42 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C48C7CF62; Tue, 4 Nov 2025 14:08:21 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 4 Nov 2025 14:06:56 +0100 Message-ID: <20251104130659.435139-17-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251104130659.435139-1-c.ebner@proxmox.com> References: <20251104130659.435139-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1762261622732 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.047 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup v2 16/19] datastore: add locking to protect against races on chunk insert for s3 X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Acquire the per-chunk file lock to get exclusive access to the chunk on insert and make it possible to guarantee that s3 backend operations and the local caches reflect the consistent state without other operations such as verify chunk renaming or garbage collection interfering. Signed-off-by: Christian Ebner --- Changes since version 1: - inline per-chunk file locking pbs-datastore/src/datastore.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs index 698486ff4..f869320d8 100644 --- a/pbs-datastore/src/datastore.rs +++ b/pbs-datastore/src/datastore.rs @@ -1919,6 +1919,10 @@ impl DataStore { match backend { DatastoreBackend::Filesystem => self.inner.chunk_store.insert_chunk(chunk, digest), DatastoreBackend::S3(s3_client) => { + let _chunk_guard = self + .inner + .chunk_store + .lock_chunk(digest, CHUNK_LOCK_TIMEOUT)?; let chunk_data: Bytes = chunk.raw_data().to_vec().into(); let chunk_size = chunk_data.len() as u64; let object_key = crate::s3::object_key_from_digest(digest)?; @@ -1939,6 +1943,10 @@ impl DataStore { ) -> Result<(bool, u64), Error> { let chunk_data = chunk.raw_data(); let chunk_size = chunk_data.len() as u64; + let _chunk_guard = self + .inner + .chunk_store + .lock_chunk(digest, CHUNK_LOCK_TIMEOUT)?; // Avoid re-upload to S3 if the chunk is either present in the in-memory LRU cache // or the chunk marker file exists on filesystem. The latter means the chunk has -- 2.47.3 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel