From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 00A9BF3C7 for ; Thu, 20 Jul 2023 19:15:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D357617AF5 for ; Thu, 20 Jul 2023 19:15:12 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 20 Jul 2023 19:15:11 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id F123B41FB8 for ; Thu, 20 Jul 2023 19:15:10 +0200 (CEST) From: Max Carrara To: pbs-devel@lists.proxmox.com Date: Thu, 20 Jul 2023 19:15:05 +0200 Message-Id: <20230720171505.1053912-2-m.carrara@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230720171505.1053912-1-m.carrara@proxmox.com> References: <20230720171505.1053912-1-m.carrara@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.009 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] [PATCH pxar 2/2] decoder: aio: improve performance of async file reads X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jul 2023 17:15:43 -0000 In order to bring `aio::Decoder` on par with its `sync` counterpart as well as `sync::Accessor` and `aio::Accessor`, its input is now buffered. As the `tokio` docs mention themselves [0], it can be really inefficient to directly work with an (unbuffered) `AsyncRead` instance. The other aforementioned types already buffer their reads in one way or another, so wrapping the input reader in `tokio::io::BufReader` results in a substantial performance gain. [1] [0]: https://docs.rs/tokio/1.29.1/tokio/io/struct.BufReader.html [1]: Tested via a custom CLI utility that opens and traverses a large (13GB) pxar archive with each decoder and accessor Before: > First pass > With aio::Decoder: Ok(()) (elapsed: 25.827150007s) > With sync::Decoder: Ok(()) (elapsed: 3.577611655s) > With aio::Accessor: Ok(()) (elapsed: 3.962754675s) > With sync::Accessor: Ok(()) (elapsed: 3.961245996s) > > Second pass > With aio::Decoder: Ok(()) (elapsed: 21.045064325s) > With sync::Decoder: Ok(()) (elapsed: 3.644003471s) > With aio::Accessor: Ok(()) (elapsed: 4.054085818s) > With sync::Accessor: Ok(()) (elapsed: 4.036097687s) After: > First pass: > With aio::Decoder: Ok(()) (elapsed: 7.07321221s) > With sync::Decoder: Ok(()) (elapsed: 3.431787191s) > With aio::Accessor: Ok(()) (elapsed: 3.930457465s) > With sync::Accessor: Ok(()) (elapsed: 4.007415416s) > > Second pass: > With aio::Decoder: Ok(()) (elapsed: 6.826005792s) > With sync::Decoder: Ok(()) (elapsed: 3.437391887s) > With aio::Accessor: Ok(()) (elapsed: 3.833275725s) > With sync::Accessor: Ok(()) (elapsed: 3.909827322s) Signed-off-by: Max Carrara --- src/decoder/aio.rs | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/decoder/aio.rs b/src/decoder/aio.rs index 200dd3d..174551b 100644 --- a/src/decoder/aio.rs +++ b/src/decoder/aio.rs @@ -79,14 +79,20 @@ mod tok { use std::pin::Pin; use std::task::{Context, Poll}; - /// Read adapter for `futures::io::AsyncRead` + use tokio::io::AsyncRead; + + /// Read adapter for `tokio::io::AsyncRead` pub struct TokioReader { - inner: T, + inner: tokio::io::BufReader, } impl TokioReader { pub fn new(inner: T) -> Self { - Self { inner } + // buffer size "sweet spot" - larger sizes don't seem to provide any benefit + const BUF_SIZE: usize = 1024 * 16; + Self { + inner: tokio::io::BufReader::with_capacity(BUF_SIZE, inner), + } } } -- 2.39.2