From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: applied: [PATCH qemu] PVE backup: pbs: process chunks in parallel
Date: Thu, 12 Nov 2026 23:35:16 +0100 [thread overview]
Message-ID: <c5da9021-c131-4bcc-9e11-97cb6a4ffdb8@proxmox.com> (raw)
In-Reply-To: <20261111003918.452105-1-t.lamprecht@proxmox.com>
Am 11.11.26 um 01:39 schrieb Thomas Lamprecht:
> For each chunk of data it reads, the backup job calls a dump callback,
> which writes the chunk either to a VMA archive or, through the
> proxmox-backup-qemu library, to a Proxmox Backup Server. The job can
> have up to max-workers such writes in flight at once, 16 by default.
>
> The VMA writer is not thread safe, so the VMA callback holds the dump
> callback mutex while writing. But the PBS callback took that mutex as
> well and held it while the library hashes, compresses and encrypts a
> chunk and hands it to the HTTP/2 stream. So only one chunk got
> processed at a time, independent of max-workers and of the four
> threads of the library's runtime.
>
> Stop taking the mutex in the PBS callback, as the library supports
> concurrent writes: the index records each chunk at its offset, its
> shared state has its own locks, and pending writes fail once the
> backup gets aborted. The check for a failed or canceled backup stays,
> so that callbacks after that still skip the write.
>
> Backing up 12 GiB of /usr content in stop mode to a PBS on the same
> host, an EPYC 9135 with the datastore on NVMe, two runs each with and
> without encryption (which made no significant difference):
>
> before after
> time 26-27 s 8-9 s
> library CPU usage 1 core 3.5-4 cores
> QEMU peak RSS 140 MiB 190-300 MiB
>
> With max-workers set to 1, both took 30 s, with the library using less
> than one core and QEMU peaking at about 80 MiB. So that setting can
> limit the impact on the host where a faster backup is not worth it.
>
> Against a remote PBS over 10G, the disk got read at 870-930 MiB/s
> instead of 530-560 MiB/s, but the total time stayed the same, as that
> PBS datastore could not take more.
>
> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
> ---
> pve-backup.c | 6 ------
> 1 file changed, 6 deletions(-)
>
>
applied this one now
prev parent reply other threads:[~2026-11-12 22:35 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 0:38 [PATCH qemu] PVE backup: pbs: process chunks in parallel Thomas Lamprecht
2026-11-12 22:35 ` Thomas Lamprecht [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c5da9021-c131-4bcc-9e11-97cb6a4ffdb8@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.