From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id E57291FF163 for ; Mon, 6 Oct 2025 12:42:05 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3E396764; Mon, 6 Oct 2025 12:42:06 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Mon, 6 Oct 2025 12:41:51 +0200 Message-ID: <20251006104151.487202-8-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251006104151.487202-1-c.ebner@proxmox.com> References: <20251006104151.487202-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1759747294009 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.043 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup 7/7] pull: guard chunk upload and only insert into cache after upload X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Inserting the chunk into the local datastore cache leads to it being considered as present on the backend. The upload might however still fail, leading to missing chunks. Therefore, only insert the chunk after the upload completed with success and guard the upload by the backend upload marker file to avoid races with garbage collection. Signed-off-by: Christian Ebner --- src/server/pull.rs | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/src/server/pull.rs b/src/server/pull.rs index 08e6e6b64..639415107 100644 --- a/src/server/pull.rs +++ b/src/server/pull.rs @@ -176,14 +176,20 @@ async fn pull_index_chunks( if target2.cache_contains(&digest) { return Ok(()); } - target2.cache_insert(&digest, &chunk)?; let data = chunk.raw_data().to_vec(); let upload_data = hyper::body::Bytes::from(data); let object_key = pbs_datastore::s3::object_key_from_digest(&digest)?; - let _is_duplicate = proxmox_async::runtime::block_on( + target2.insert_backend_upload_marker(&digest)?; + match proxmox_async::runtime::block_on( s3_client.upload_no_replace_with_retry(object_key, upload_data), - ) - .context("failed to upload chunk to s3 backend")?; + ) { + Ok(_is_duplicate) => (), + Err(err) => { + target2.cleanup_backend_upload_marker(&digest)?; + return Err(err.context("failed to upload chunk to s3 backend")); + } + } + target2.cache_insert(&digest, &chunk)?; } } Ok(()) -- 2.47.3 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel