From: Dietmar Maurer <dietmar@proxmox.com>
To: "Proxmox Backup Server development discussion"
<pbs-devel@lists.proxmox.com>,
"Fabian Grünbichler" <f.gruenbichler@proxmox.com>
Subject: Re: [pbs-devel] [RFC proxmox-backup 2/2] image backup: use 4M input buffer
Date: Fri, 19 Jul 2024 09:28:09 +0200 (CEST) [thread overview]
Message-ID: <914687146.1663.1721374089431@webmail.proxmox.com> (raw)
In-Reply-To: <20240717130827.1073067-3-f.gruenbichler@proxmox.com>
I get the following from git blame:
db0cb9ce0 src/bin/proxmox-backup-client.rs (Wolfgang Bumiller 2019-12-12 15:27:07 +0100 289) let stream = tokio_util::codec::FramedRead::new(file, tokio_util::codec::BytesCodec::new())
So we comited that code in 2019, while FramedRead::with_capacity was
added later in 2020:
https://github.com/tokio-rs/tokio/pull/2215
So I think that was the reason for using the small buffer.
> On 17.7.2024 15:08 CEST Fabian Grünbichler <f.gruenbichler@proxmox.com> wrote:
>
>
> with the default 8k input buffer size, the client will spend most of the time
> polling instead of reading/chunking/uploading.
>
> tested with 16G random data file from tmpfs to fresh datastore backed by tmpfs,
> without encryption.
>
> stock:
>
> Time (mean ± σ): 36.064 s ± 0.655 s [User: 21.079 s, System: 26.415 s]
> Range (min … max): 35.663 s … 36.819 s 3 runs
>
> patched:
>
> Time (mean ± σ): 23.591 s ± 0.807 s [User: 16.532 s, System: 18.629 s]
> Range (min … max): 22.663 s … 24.125 s 3 runs
>
> Summary
> patched ran
> 1.53 ± 0.06 times faster than stock
>
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> obviously, this slightly increases memory usage..
>
> the effect here is less pronounced than for the example because the actual
> reading and chunking part is not all the client has to do for a backup - digest
> calculation and TLS crypto make up the bulk of the rest..
>
> proxmox-backup-client/src/main.rs | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
> index 6a7d09047..5edb2a824 100644
> --- a/proxmox-backup-client/src/main.rs
> +++ b/proxmox-backup-client/src/main.rs
> @@ -286,8 +286,12 @@ async fn backup_image<P: AsRef<Path>>(
>
> let file = tokio::fs::File::open(path).await?;
>
> - let stream = tokio_util::codec::FramedRead::new(file, tokio_util::codec::BytesCodec::new())
> - .map_err(Error::from);
> + let stream = tokio_util::codec::FramedRead::with_capacity(
> + file,
> + tokio_util::codec::BytesCodec::new(),
> + 4 * 1024 * 1024,
> + )
> + .map_err(Error::from);
>
> let stream = FixedChunkStream::new(stream, chunk_size.unwrap_or(4 * 1024 * 1024));
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
next prev parent reply other threads:[~2024-07-19 7:28 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-17 13:08 [pbs-devel] [PATCH proxmox-backup 0/2] improve fixed size chunker performance Fabian Grünbichler
2024-07-17 13:08 ` [pbs-devel] [PATCH proxmox-backup 1/2] example: improve chunking speed example Fabian Grünbichler
2024-07-17 13:08 ` [pbs-devel] [RFC proxmox-backup 2/2] image backup: use 4M input buffer Fabian Grünbichler
2024-07-19 7:28 ` Dietmar Maurer [this message]
2024-07-22 8:03 ` [pbs-devel] applied: [PATCH proxmox-backup 0/2] improve fixed size chunker performance Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=914687146.1663.1721374089431@webmail.proxmox.com \
--to=dietmar@proxmox.com \
--cc=f.gruenbichler@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox