From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 58CD21FF17E for ; Thu, 27 Nov 2025 09:55:21 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B001E1CC5A; Thu, 27 Nov 2025 09:55:34 +0100 (CET) Message-ID: Date: Thu, 27 Nov 2025 09:55:31 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Thomas Lamprecht , Proxmox Backup Server development discussion References: <20251126162815.814841-1-c.ebner@proxmox.com> <9dc8c7f6-d125-4e47-b21c-39cfc5b02dc1@proxmox.com> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <9dc8c7f6-d125-4e47-b21c-39cfc5b02dc1@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1764233693910 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.047 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [docs.rs, mod.rs, proxmox.com] Subject: Re: [pbs-devel] [PATCH proxmox-backup] api: chunk reader: make reading from filesystem fully async X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On 11/26/25 7:10 PM, Thomas Lamprecht wrote: > Am 26.11.25 um 17:28 schrieb Christian Ebner: >> Blocking the thread is problematic here and must be avoided, so >> read the chunk data via tokio::fs::read() instead of std::fs::read() >> and make the full loading from filesystem branch async. > > Nothing against that, but "async" here comes a bit with a bigger asterisks, > as: > > "This operation is implemented by running the equivalent blocking operation > on a separate thread pool using spawn_blocking." > -- https://docs.rs/tokio/latest/tokio/fs/fn.read.html > > So technically async, but not really does any async IO (tokio io uring when? ;)). > > The important thing is that it cannot block anything, so it _is_ an OK solution > here, might be nice to adapt the commit message slightly though, e.g. something > like: > > ...::read() to move the blocking file read in the "full loading from filesystem" > branch to it's own thread pool. Can be done on applying though. True, fully async is indeed overreaching and incorrect. Can send a v2 with the commit message adapted if requested. Just to clarify as this came up in off-list discussion with Fabian. I do not expect this to be the cause of the issues as reported by the users, so finding that has priority. > >> >> Encountered while investigating a user provided backtrace looking for >> possible causes of hanging backups reported in [0]. >> >> [0] https://forum.proxmox.com/threads/176444/post-819858 >> >> Signed-off-by: Christian Ebner >> --- >> src/api2/reader/mod.rs | 7 ++++--- >> 1 file changed, 4 insertions(+), 3 deletions(-) >> >> diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs >> index f7adc366f..1e74b0758 100644 >> --- a/src/api2/reader/mod.rs >> +++ b/src/api2/reader/mod.rs >> @@ -321,7 +321,7 @@ fn download_chunk( >> } >> >> let body = match &env.backend { >> - DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?, >> + DatastoreBackend::Filesystem => load_from_filesystem(env, &digest).await?, >> DatastoreBackend::S3(s3_client) => match env.datastore.cache() { >> None => fetch_from_object_store(s3_client, &digest).await?, >> Some(cache) => { >> @@ -357,13 +357,14 @@ async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Res >> bail!("cannot find chunk with digest {}", hex::encode(digest)); >> } >> >> -fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result { >> +async fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result { >> let (path, _) = env.datastore.chunk_path(digest); >> let path2 = path.clone(); >> >> env.debug(format!("download chunk {path:?}")); >> >> - let data = proxmox_async::runtime::block_in_place(|| std::fs::read(path)) >> + let data = tokio::fs::read(path) >> + .await >> .map_err(move |err| http_err!(BAD_REQUEST, "reading file {path2:?} failed: {err}"))?; >> Ok(Body::from(data)) >> } > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel