From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id AE0C61FF09B for ; Mon, 26 Oct 2026 14:46:27 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4E175213A6; Mon, 26 Oct 2026 14:46:26 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [RFC PATCH proxmox-backup 1/2] api: optimize content streaming reader for file download Date: Mon, 26 Oct 2026 14:45:58 +0100 Message-ID: <20261026134559.352926-2-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261026134559.352926-1-c.ebner@proxmox.com> References: <20261026134559.352926-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793022378421 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.653 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FRQAZI2L3JUCQ6IEQIXP5KPH67AXYNTK X-Message-ID-Hash: FRQAZI2L3JUCQ6IEQIXP5KPH67AXYNTK X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The current implementation using FramedRead and BytesCodec reads the file in 8K chunks only. FramedRead starts out with a default buffer capacity of 8K [0], the reader filling only up to this capacity [1], which BytesCodec::decode() consumes completely [2]. Therefore, the next read will recreate the same 8K buffer again because BytesMut::reserve() [3] will fill to the initial capacity again. This has additional overhead since tokio's async reader uses the blocking thread pool for each read and is especially problematic if the blocking pool is heavily utilized, e.g. by concurrent operations such as chunk inserts. Latter also run on the blocking pool since commit 23400016 ("api: do not block tokio worker threads during chunk inserts"). Use the ReaderStream with tokio's default maximum buffer size for blocking I/O instead [4]. [0] https://docs.rs/tokio-util/0.7.10/src/tokio_util/codec/framed_impl.rs.html#53 [1] https://docs.rs/tokio-util/0.7.10/src/tokio_util/codec/framed_impl.rs.html#221 [2] https://docs.rs/tokio-util/0.7.10/src/tokio_util/codec/bytes_codec.rs.html#58 [3] https://docs.rs/tokio-util/0.7.10/src/tokio_util/codec/framed_impl.rs.html#221 [4] https://docs.rs/tokio/1.54.1/src/tokio/io/blocking.rs.html#29 Signed-off-by: Christian Ebner --- src/api2/helpers.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/api2/helpers.rs b/src/api2/helpers.rs index aa582a53c..7d1fb8e35 100644 --- a/src/api2/helpers.rs +++ b/src/api2/helpers.rs @@ -1,12 +1,14 @@ use std::path::PathBuf; use anyhow::Error; -use futures::stream::TryStreamExt; use hyper::{Response, StatusCode, header}; +use tokio_util::io::ReaderStream; use proxmox_http::Body; use proxmox_router::http_bail; +const DOWNLOAD_BUFFER_SIZE: usize = 2 * 1024 * 1024; + pub async fn create_download_response(path: PathBuf) -> Result, Error> { let file = match tokio::fs::File::open(path.clone()).await { Ok(file) => file, @@ -16,9 +18,7 @@ pub async fn create_download_response(path: PathBuf) -> Result, E Err(err) => http_bail!(BAD_REQUEST, "open file {:?} failed: {}", path, err), }; - let payload = tokio_util::codec::FramedRead::new(file, tokio_util::codec::BytesCodec::new()) - .map_ok(|bytes| bytes.freeze()); - + let payload = ReaderStream::with_capacity(file, DOWNLOAD_BUFFER_SIZE); let body = Body::wrap_stream(payload); // fixme: set other headers ? -- 2.47.3