From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 5721C1FF179 for ; Wed, 15 Oct 2025 18:40:38 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7E07CAC1; Wed, 15 Oct 2025 18:40:55 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Wed, 15 Oct 2025 18:40:08 +0200 Message-ID: <20251015164008.975591-11-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251015164008.975591-1-c.ebner@proxmox.com> References: <20251015164008.975591-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1760546418953 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.042 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup v3 8/8] pull: guard chunk upload and only insert into cache after upload X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Inserting the chunk into the local datastore cache leads to it being considered as present on the backend. The upload might however still fail, leading to missing chunks. Therefore, only insert the chunk after the upload completed with success and guard the upload by the backend upload marker file to avoid races with garbage collection. Signed-off-by: Christian Ebner --- src/server/pull.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/server/pull.rs b/src/server/pull.rs index c0b6fef7c..0bc619c68 100644 --- a/src/server/pull.rs +++ b/src/server/pull.rs @@ -176,14 +176,17 @@ async fn pull_index_chunks( if target2.cache_contains(&digest) { return Ok(()); } - target2.cache_insert(&digest, &chunk)?; let data = chunk.raw_data().to_vec(); let upload_data = hyper::body::Bytes::from(data); let object_key = pbs_datastore::s3::object_key_from_digest(&digest)?; + if !target2.touch_backend_upload_marker(&digest)? { + return Ok(()); + } let _is_duplicate = proxmox_async::runtime::block_on( s3_client.upload_replace_on_final_retry(object_key, upload_data), ) .context("failed to upload chunk to s3 backend")?; + target2.cache_insert(&digest, &chunk)?; } } Ok(()) -- 2.47.3 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel