From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	by lore.proxmox.com (Postfix) with ESMTPS id ABE761FF16F
	for <inbox@lore.proxmox.com>; Tue, 24 Jun 2025 12:11:34 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id C5D263338B;
	Tue, 24 Jun 2025 12:12:08 +0200 (CEST)
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Date: Tue, 24 Jun 2025 12:11:28 +0200
In-Reply-To: <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com>
References: <9995c68d9c0d6e699578f5a45edb2731b5724ef1.camel@notnullmakers.com>
 <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com>
MIME-Version: 1.0
Message-ID: <mailman.571.1750759927.395.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Adam Kalisz via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Adam Kalisz <adam.kalisz@notnullmakers.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: Re: [pve-devel] Discussion of major PBS restore speedup in
 proxmox-backup-qemu
Content-Type: multipart/mixed; boundary="===============2911394534801260041=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>

--===============2911394534801260041==
Content-Type: message/rfc822
Content-Disposition: inline

Return-Path: <adam.kalisz@notnullmakers.com>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits))
	(No client certificate requested)
	by lists.proxmox.com (Postfix) with ESMTPS id DC010D09C4
	for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:12:06 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id BE3F633301
	for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:11:36 +0200 (CEST)
Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by firstgate.proxmox.com (Proxmox) with ESMTPS
	for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:11:36 +0200 (CEST)
Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-ad883afdf0cso48719466b.0
        for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 03:11:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=notnullmakers.com; s=google; t=1750759889; x=1751364689; darn=lists.proxmox.com;
        h=mime-version:user-agent:content-transfer-encoding:references
         :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date
         :message-id:reply-to;
        bh=BjbYM84sj6YCVQ8HQzmQXYm/tyXejMThB54gM8ABNNo=;
        b=aWpJlXDClTXi5Wu+WPkq5Ui2wxXZ9IYeOFSdCUvTgUU5ipoBjievymZGsQK9PxQksr
         KvU96h90zlheHiiicPGTgJHRAEgTVf7pOEBHyGnnRsofp674jItg5gEv8LOLjkw2lOff
         gnS+9KL2SR39cxfXBRC/+AwAeBL9PuSLBJkT0bETmQR7P9lGS0LG9WKK5K5uk/Az8J6K
         HvC1zGAwloYNd/1PUZJF2zWFjV1qrKXgtE2wDdTzKpNJPjkd6qys/g/NvWMgFpIIK9l2
         xnmNIjzx0YeHSaK2NGOFTN4ph8j35zpNGFdCIv49Y0zDhx4k1oiNicPEbkNDiki7NR5h
         XSfQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1750759889; x=1751364689;
        h=mime-version:user-agent:content-transfer-encoding:references
         :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from
         :to:cc:subject:date:message-id:reply-to;
        bh=BjbYM84sj6YCVQ8HQzmQXYm/tyXejMThB54gM8ABNNo=;
        b=BhkZKFUNYxbSC50ozu/Yqdho6khUv5/OfetySDAKZBzjtz7jHkwy1vXYgpY3JbijVe
         iED07uV7/Pg4UdU6X2VgCUe/tGIpFMW0XvOLXw6iXDH+JwO9UByyAIOHCnzQ2zga7K+B
         e/b+2+yxz9cLhv2JT9nk7tUyYodBrbc55zuu8TBC942hLvZ/Wx69uGmoAYYXuXDqNCOE
         HrQ2th12owkY3CU1fTPC5tmfyew7ZFEsN/VBRpuBTEB2EU/NsM7xwFIksq+gNHbpDSYS
         6qUGP5LrH67nRZZzx1LvoSNGcDb/m4ZXCii+DZt1/5kCO9+Tz4LFSmnoAX7KNayvtjnL
         WAfQ==
X-Gm-Message-State: AOJu0YzXXI/5DNT/gZLkvZiy4Gt40H5QIPQVpAl5vDmvW2pD4PgY4cQe
	iHB0THGlnYlYHx/NKt6H9guPrYNV9LsdLNVmFfDwFoSE6H7pVr2u1fw7q+5X2vgMIDcBMKVcx2z
	IelAa
X-Gm-Gg: ASbGncsWxLjNSwavDWTqM3HevrE5xJdioF0arn4/sfBZw1SBcuqQVgc+9zacdmqeJL/
	BWUZYNXStX4XixSOXOBn7bIwdNM9qN/g1+3B0mbaIOvz4l/QNecKv6ZcthqCSRh0sm13ZLncXbL
	vpqqAuzgSXNoIB7opsWvr91WJTRtoIMt5CFByO5k9I56jW6AG8tKs6FpzCp5BG39Dewaav5LRWP
	hoLwCyuccLumkipgG1W3qsQr0TuW/Y6GslTFeY6cw+Eu1cXQjIR53SbYV49AXL7uBtBcWUtUh3D
	VjuI/+X7oWTw9XNk6dsCoCnmAwwXFud3jXdUgTjR7xwL6c5eh+g492tbCtavoH2EpfDs4SRiNI5
	U9A4E5ICnUa+NxEOs++iNAgITYEnMQiE=
X-Google-Smtp-Source: AGHT+IHROoYn5SS29qfIFuUrgGxj9NWTUvzo6q01WpHLCM4fX2OMp8vQj1VDEnWGp9vF2HoGJR7zYQ==
X-Received: by 2002:a17:907:72d5:b0:ad8:9e80:6bb8 with SMTP id a640c23a62f3a-ae0578d850bmr1510762366b.6.1750759889058;
        Tue, 24 Jun 2025 03:11:29 -0700 (PDT)
Received: from [192.168.8.143] (78-80-24-90.customers.tmcz.cz. [78.80.24.90])
        by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ae054209b49sm842539566b.164.2025.06.24.03.11.28
        for <pve-devel@lists.proxmox.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 24 Jun 2025 03:11:28 -0700 (PDT)
Message-ID: <191bc676b6e091844291b2752a7b13fba0f3d4b6.camel@notnullmakers.com>
Subject: Re: Discussion of major PBS restore speedup in proxmox-backup-qemu
From: Adam Kalisz <adam.kalisz@notnullmakers.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Date: Tue, 24 Jun 2025 12:11:28 +0200
In-Reply-To: <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com>
References: 	<9995c68d9c0d6e699578f5a45edb2731b5724ef1.camel@notnullmakers.com>
	 <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.56.1-1 
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results:  0
	BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
	DKIM_SIGNED               0.1 Message has a DKIM or DK signature, not necessarily valid
	DKIM_VALID               -0.1 Message has at least one valid DKIM or DK signature
	DKIM_VALID_AU            -0.1 Message has a valid DKIM or DK signature from author's domain
	DKIM_VALID_EF            -0.1 Message has a valid DKIM or DK signature from envelope-from domain
	DMARC_PASS               -0.1 DMARC pass policy
	RCVD_IN_DNSWL_NONE     -0.0001 Sender listed at https://www.dnswl.org/, no trust
	SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
	SPF_PASS               -0.001 SPF: sender matches SPF record
	URIBL_BLOCKED           0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked.  See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com,notnullmakers.com]

Hi Alexandre,

yes, having configurable CONCURRENT_REQUESTS and max_blocking_threads
would be great. However we would need to wire it up all the way to
qmrestore or similar or ensure it is read from some env vars. I didn't
feel confident to introduce this kind of infrastructure as a first time
contribution. Btw. in this case the concurrency applies mostly to
fetching requests the writer is single thread and there should still be
reasonable locality.

If you have a spinning rust setup I would be very glad if you could do
a performance test vs the current implementation.

Best regards
Adam Kalisz

On Tue, 2025-06-24 at 09:09 +0000, DERUMIER, Alexandre wrote:
> Hi,=20
>=20
> nice work !
>=20
> Could it be possible to have an option to configure the
> CONCURRENT_REQUESTS=C2=A0 ?
>=20
> (to avoid to put too much load on slow spinning storage)
>=20
>=20
>=20
>=20
> -------- Message initial --------
> De: Adam Kalisz <adam.kalisz@notnullmakers.com>
> =C3=80: pve-devel@lists.proxmox.com
> Objet: Discussion of major PBS restore speedup in proxmox-backup-qemu
> Date: 23/06/2025 18:10:01
>=20
> Hi list,
>=20
> before I go through all the hoops to submit a patch I wanted to
> discuss
> the current form of the patch that can be found here:
>=20
> https://github.com/NOT-NULL-Makers/proxmox-backup-
> qemu/commit/e91f09cfd1654010d6205d8330d9cca71358e030
>=20
> The speedup process was discussed here:
>=20
> https://forum.proxmox.com/threads/abysmally-slow-restore-from-
> backup.133602/
>=20
> The current numbers are:
>=20
> With the most current snapshot of a VM with 10 GiB system disk and 2x
> 100 GiB disks with random data:
>=20
> Original as of 1.5.1:
> 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D11.78s,=C2=A0 speed=3D869.34M=
B/s
> 100 GiB random 1: duration=3D412.85s, speed=3D248.03MB/s
> 100 GiB random 2: duration=3D422.42s, speed=3D242.41MB/s
>=20
> With the 12-way concurrent fetching:
>=20
> 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D2.05s,=C2=A0=C2=A0 speed=3D49=
91.99MB/s
> 100 GiB random 1: duration=3D100.54s, speed=3D1018.48MB/s
> 100 GiB random 2: duration=3D100.10s, speed=3D1022.97MB/s
>=20
> The hardware is on the PVE side:
> 2x Intel Xeon Gold 6244, 1 TB RAM, 2x 100 Gbps Mellanox, 14x Samsung
> NVMe 3,8 TB drives in RAID10 using mdadm/ LVM-thin.
>=20
> On the PBS side:
> 2x Intel Xeon Gold 6334, 1 TB RAM, 2x 100 Gbps Mellanox, 8x Samsung
> NVMe in RAID using 4 ZFS mirrors with recordsize 1M, lz4 compression.
>=20
> Similar or slightly better speeds were achieved on Hetzner AX52 with
> AMD Ryzen 7 7700 with 64 GB RAM and 2x 1 TB NVMe in stripe on PVE
> with
> recordsize 16k connected to another Hetzner AX52 using a 10 Gbps
> connection. The PBS has normal NVMe ZFS mirror again with recordsize
> 1M.
>=20
> On bigger servers a 16-way concurrency was even better on smaller
> servers with high frequency CPUs 8-way concurrency performed better.
> The 12-way concurrency is a compromise. We seem to hit a bottleneck
> somewhere in the realm of TLS connection and shallow buffers. The
> network on the 100 Gbps servers can support up to about 3 GBps
> (almost
> 20 Gbps) of traffic in a single TCP connection using mbuffer. The
> storage can keep up with such a speed.
>=20
> Before I submit the patch, I would also like to do the most up to
> date
> build but I have trouble updating my build environment to reflect the
> latest commits. What do I have to put in my /etc/apt/sources.list to
> be
> able to install e.g. librust-cbindgen-0.27+default-dev librust-http-
> body-util-0.1+default-dev librust-hyper-1+default-dev and all the
> rest?
>=20
> This work was sponsored by =C4=8CMIS s.r.o. and consulted with the Genera=
l
> Manager V=C3=A1clav Sv=C3=A1tek (=C4=8CMIS), Daniel =C5=A0karda (NOT NULL=
 Makers s.r.o.)
> and Linux team leader Roman M=C3=BCller (=C4=8CMIS).
>=20
> Best regards
> Adam Kalisz


--===============2911394534801260041==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

--===============2911394534801260041==--