Re: [pve-devel] Discussion of major PBS restore speedup in proxmox-backup-qemu

[Adam Kalisz via pve-devel <pve-devel@lists.proxmox.com>]

[-- Attachment #1: Type: message/rfc822, Size: 9502 bytes --]

From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: Discussion of major PBS restore speedup in proxmox-backup-qemu
Date: Tue, 24 Jun 2025 12:11:28 +0200
Message-ID: <191bc676b6e091844291b2752a7b13fba0f3d4b6.camel@notnullmakers.com>

Hi Alexandre,

yes, having configurable CONCURRENT_REQUESTS and max_blocking_threads
would be great. However we would need to wire it up all the way to
qmrestore or similar or ensure it is read from some env vars. I didn't
feel confident to introduce this kind of infrastructure as a first time
contribution. Btw. in this case the concurrency applies mostly to
fetching requests the writer is single thread and there should still be
reasonable locality.

If you have a spinning rust setup I would be very glad if you could do
a performance test vs the current implementation.

Best regards
Adam Kalisz

On Tue, 2025-06-24 at 09:09 +0000, DERUMIER, Alexandre wrote:
> Hi, 
> 
> nice work !
> 
> Could it be possible to have an option to configure the
> CONCURRENT_REQUESTS  ?
> 
> (to avoid to put too much load on slow spinning storage)
> 
> 
> 
> 
> -------- Message initial --------
> De: Adam Kalisz <adam.kalisz@notnullmakers.com>
> À: pve-devel@lists.proxmox.com
> Objet: Discussion of major PBS restore speedup in proxmox-backup-qemu
> Date: 23/06/2025 18:10:01
> 
> Hi list,
> 
> before I go through all the hoops to submit a patch I wanted to
> discuss
> the current form of the patch that can be found here:
> 
> https://github.com/NOT-NULL-Makers/proxmox-backup-
> qemu/commit/e91f09cfd1654010d6205d8330d9cca71358e030
> 
> The speedup process was discussed here:
> 
> https://forum.proxmox.com/threads/abysmally-slow-restore-from-
> backup.133602/
> 
> The current numbers are:
> 
> With the most current snapshot of a VM with 10 GiB system disk and 2x
> 100 GiB disks with random data:
> 
> Original as of 1.5.1:
> 10 GiB system:    duration=11.78s,  speed=869.34MB/s
> 100 GiB random 1: duration=412.85s, speed=248.03MB/s
> 100 GiB random 2: duration=422.42s, speed=242.41MB/s
> 
> With the 12-way concurrent fetching:
> 
> 10 GiB system:    duration=2.05s,   speed=4991.99MB/s
> 100 GiB random 1: duration=100.54s, speed=1018.48MB/s
> 100 GiB random 2: duration=100.10s, speed=1022.97MB/s
> 
> The hardware is on the PVE side:
> 2x Intel Xeon Gold 6244, 1 TB RAM, 2x 100 Gbps Mellanox, 14x Samsung
> NVMe 3,8 TB drives in RAID10 using mdadm/ LVM-thin.
> 
> On the PBS side:
> 2x Intel Xeon Gold 6334, 1 TB RAM, 2x 100 Gbps Mellanox, 8x Samsung
> NVMe in RAID using 4 ZFS mirrors with recordsize 1M, lz4 compression.
> 
> Similar or slightly better speeds were achieved on Hetzner AX52 with
> AMD Ryzen 7 7700 with 64 GB RAM and 2x 1 TB NVMe in stripe on PVE
> with
> recordsize 16k connected to another Hetzner AX52 using a 10 Gbps
> connection. The PBS has normal NVMe ZFS mirror again with recordsize
> 1M.
> 
> On bigger servers a 16-way concurrency was even better on smaller
> servers with high frequency CPUs 8-way concurrency performed better.
> The 12-way concurrency is a compromise. We seem to hit a bottleneck
> somewhere in the realm of TLS connection and shallow buffers. The
> network on the 100 Gbps servers can support up to about 3 GBps
> (almost
> 20 Gbps) of traffic in a single TCP connection using mbuffer. The
> storage can keep up with such a speed.
> 
> Before I submit the patch, I would also like to do the most up to
> date
> build but I have trouble updating my build environment to reflect the
> latest commits. What do I have to put in my /etc/apt/sources.list to
> be
> able to install e.g. librust-cbindgen-0.27+default-dev librust-http-
> body-util-0.1+default-dev librust-hyper-1+default-dev and all the
> rest?
> 
> This work was sponsored by ČMIS s.r.o. and consulted with the General
> Manager Václav Svátek (ČMIS), Daniel Škarda (NOT NULL Makers s.r.o.)
> and Linux team leader Roman Müller (ČMIS).
> 
> Best regards
> Adam Kalisz


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel