From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 6D5DF1FF187 for ; Mon, 6 Oct 2025 15:18:47 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 010DF4D7B; Mon, 6 Oct 2025 15:18:51 +0200 (CEST) Date: Mon, 06 Oct 2025 15:18:14 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox Backup Server development discussion References: <20251006104151.487202-1-c.ebner@proxmox.com> <20251006104151.487202-8-c.ebner@proxmox.com> In-Reply-To: <20251006104151.487202-8-c.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1759754556.zqybqz2req.astroid@yuna.none> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1759756669152 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.049 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox-backup 7/7] pull: guard chunk upload and only insert into cache after upload X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On October 6, 2025 12:41 pm, Christian Ebner wrote: > Inserting the chunk into the local datastore cache leads to it being > considered as present on the backend. The upload might however still > fail, leading to missing chunks. > > Therefore, only insert the chunk after the upload completed with > success and guard the upload by the backend upload marker file to > avoid races with garbage collection. > > Signed-off-by: Christian Ebner > --- > src/server/pull.rs | 14 ++++++++++---- > 1 file changed, 10 insertions(+), 4 deletions(-) > > diff --git a/src/server/pull.rs b/src/server/pull.rs > index 08e6e6b64..639415107 100644 > --- a/src/server/pull.rs > +++ b/src/server/pull.rs > @@ -176,14 +176,20 @@ async fn pull_index_chunks( > if target2.cache_contains(&digest) { > return Ok(()); > } > - target2.cache_insert(&digest, &chunk)?; > let data = chunk.raw_data().to_vec(); > let upload_data = hyper::body::Bytes::from(data); > let object_key = pbs_datastore::s3::object_key_from_digest(&digest)?; > - let _is_duplicate = proxmox_async::runtime::block_on( > + target2.insert_backend_upload_marker(&digest)?; > + match proxmox_async::runtime::block_on( > s3_client.upload_no_replace_with_retry(object_key, upload_data), > - ) > - .context("failed to upload chunk to s3 backend")?; > + ) { > + Ok(_is_duplicate) => (), > + Err(err) => { > + target2.cleanup_backend_upload_marker(&digest)?; > + return Err(err.context("failed to upload chunk to s3 backend")); > + } > + } > + target2.cache_insert(&digest, &chunk)?; same issue as described in patch #5 exists here as well.. > } > } > Ok(()) > -- > 2.47.3 > > > > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel > > > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel