From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id EB6F31FF179 for ; Wed, 26 Nov 2025 19:11:17 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1D151136F8; Wed, 26 Nov 2025 19:11:31 +0100 (CET) Message-ID: <9dc8c7f6-d125-4e47-b21c-39cfc5b02dc1@proxmox.com> Date: Wed, 26 Nov 2025 19:10:57 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Proxmox Backup Server development discussion , Christian Ebner References: <20251126162815.814841-1-c.ebner@proxmox.com> Content-Language: en-US From: Thomas Lamprecht In-Reply-To: <20251126162815.814841-1-c.ebner@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1764180620375 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.023 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox-backup] api: chunk reader: make reading from filesystem fully async X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Am 26.11.25 um 17:28 schrieb Christian Ebner: > Blocking the thread is problematic here and must be avoided, so > read the chunk data via tokio::fs::read() instead of std::fs::read() > and make the full loading from filesystem branch async. Nothing against that, but "async" here comes a bit with a bigger asterisks, as: "This operation is implemented by running the equivalent blocking operation on a separate thread pool using spawn_blocking." -- https://docs.rs/tokio/latest/tokio/fs/fn.read.html So technically async, but not really does any async IO (tokio io uring when? ;)). The important thing is that it cannot block anything, so it _is_ an OK solution here, might be nice to adapt the commit message slightly though, e.g. something like: ...::read() to move the blocking file read in the "full loading from filesystem" branch to it's own thread pool. Can be done on applying though. > > Encountered while investigating a user provided backtrace looking for > possible causes of hanging backups reported in [0]. > > [0] https://forum.proxmox.com/threads/176444/post-819858 > > Signed-off-by: Christian Ebner > --- > src/api2/reader/mod.rs | 7 ++++--- > 1 file changed, 4 insertions(+), 3 deletions(-) > > diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs > index f7adc366f..1e74b0758 100644 > --- a/src/api2/reader/mod.rs > +++ b/src/api2/reader/mod.rs > @@ -321,7 +321,7 @@ fn download_chunk( > } > > let body = match &env.backend { > - DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?, > + DatastoreBackend::Filesystem => load_from_filesystem(env, &digest).await?, > DatastoreBackend::S3(s3_client) => match env.datastore.cache() { > None => fetch_from_object_store(s3_client, &digest).await?, > Some(cache) => { > @@ -357,13 +357,14 @@ async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Res > bail!("cannot find chunk with digest {}", hex::encode(digest)); > } > > -fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result { > +async fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result { > let (path, _) = env.datastore.chunk_path(digest); > let path2 = path.clone(); > > env.debug(format!("download chunk {path:?}")); > > - let data = proxmox_async::runtime::block_in_place(|| std::fs::read(path)) > + let data = tokio::fs::read(path) > + .await > .map_err(move |err| http_err!(BAD_REQUEST, "reading file {path2:?} failed: {err}"))?; > Ok(Body::from(data)) > } _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel