From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id ABE761FF16F for <inbox@lore.proxmox.com>; Tue, 24 Jun 2025 12:11:34 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C5D263338B; Tue, 24 Jun 2025 12:12:08 +0200 (CEST) To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com> Date: Tue, 24 Jun 2025 12:11:28 +0200 In-Reply-To: <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com> References: <9995c68d9c0d6e699578f5a45edb2731b5724ef1.camel@notnullmakers.com> <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com> MIME-Version: 1.0 Message-ID: <mailman.571.1750759927.395.pve-devel@lists.proxmox.com> List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Post: <mailto:pve-devel@lists.proxmox.com> From: Adam Kalisz via pve-devel <pve-devel@lists.proxmox.com> Precedence: list Cc: Adam Kalisz <adam.kalisz@notnullmakers.com> X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> Subject: Re: [pve-devel] Discussion of major PBS restore speedup in proxmox-backup-qemu Content-Type: multipart/mixed; boundary="===============2911394534801260041==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> --===============2911394534801260041== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: <adam.kalisz@notnullmakers.com> X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DC010D09C4 for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:12:06 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BE3F633301 for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:11:36 +0200 (CEST) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 12:11:36 +0200 (CEST) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-ad883afdf0cso48719466b.0 for <pve-devel@lists.proxmox.com>; Tue, 24 Jun 2025 03:11:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1750759889; x=1751364689; darn=lists.proxmox.com; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=BjbYM84sj6YCVQ8HQzmQXYm/tyXejMThB54gM8ABNNo=; b=aWpJlXDClTXi5Wu+WPkq5Ui2wxXZ9IYeOFSdCUvTgUU5ipoBjievymZGsQK9PxQksr KvU96h90zlheHiiicPGTgJHRAEgTVf7pOEBHyGnnRsofp674jItg5gEv8LOLjkw2lOff gnS+9KL2SR39cxfXBRC/+AwAeBL9PuSLBJkT0bETmQR7P9lGS0LG9WKK5K5uk/Az8J6K HvC1zGAwloYNd/1PUZJF2zWFjV1qrKXgtE2wDdTzKpNJPjkd6qys/g/NvWMgFpIIK9l2 xnmNIjzx0YeHSaK2NGOFTN4ph8j35zpNGFdCIv49Y0zDhx4k1oiNicPEbkNDiki7NR5h XSfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750759889; x=1751364689; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BjbYM84sj6YCVQ8HQzmQXYm/tyXejMThB54gM8ABNNo=; b=BhkZKFUNYxbSC50ozu/Yqdho6khUv5/OfetySDAKZBzjtz7jHkwy1vXYgpY3JbijVe iED07uV7/Pg4UdU6X2VgCUe/tGIpFMW0XvOLXw6iXDH+JwO9UByyAIOHCnzQ2zga7K+B e/b+2+yxz9cLhv2JT9nk7tUyYodBrbc55zuu8TBC942hLvZ/Wx69uGmoAYYXuXDqNCOE HrQ2th12owkY3CU1fTPC5tmfyew7ZFEsN/VBRpuBTEB2EU/NsM7xwFIksq+gNHbpDSYS 6qUGP5LrH67nRZZzx1LvoSNGcDb/m4ZXCii+DZt1/5kCO9+Tz4LFSmnoAX7KNayvtjnL WAfQ== X-Gm-Message-State: AOJu0YzXXI/5DNT/gZLkvZiy4Gt40H5QIPQVpAl5vDmvW2pD4PgY4cQe iHB0THGlnYlYHx/NKt6H9guPrYNV9LsdLNVmFfDwFoSE6H7pVr2u1fw7q+5X2vgMIDcBMKVcx2z IelAa X-Gm-Gg: ASbGncsWxLjNSwavDWTqM3HevrE5xJdioF0arn4/sfBZw1SBcuqQVgc+9zacdmqeJL/ BWUZYNXStX4XixSOXOBn7bIwdNM9qN/g1+3B0mbaIOvz4l/QNecKv6ZcthqCSRh0sm13ZLncXbL vpqqAuzgSXNoIB7opsWvr91WJTRtoIMt5CFByO5k9I56jW6AG8tKs6FpzCp5BG39Dewaav5LRWP hoLwCyuccLumkipgG1W3qsQr0TuW/Y6GslTFeY6cw+Eu1cXQjIR53SbYV49AXL7uBtBcWUtUh3D VjuI/+X7oWTw9XNk6dsCoCnmAwwXFud3jXdUgTjR7xwL6c5eh+g492tbCtavoH2EpfDs4SRiNI5 U9A4E5ICnUa+NxEOs++iNAgITYEnMQiE= X-Google-Smtp-Source: AGHT+IHROoYn5SS29qfIFuUrgGxj9NWTUvzo6q01WpHLCM4fX2OMp8vQj1VDEnWGp9vF2HoGJR7zYQ== X-Received: by 2002:a17:907:72d5:b0:ad8:9e80:6bb8 with SMTP id a640c23a62f3a-ae0578d850bmr1510762366b.6.1750759889058; Tue, 24 Jun 2025 03:11:29 -0700 (PDT) Received: from [192.168.8.143] (78-80-24-90.customers.tmcz.cz. [78.80.24.90]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ae054209b49sm842539566b.164.2025.06.24.03.11.28 for <pve-devel@lists.proxmox.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 03:11:28 -0700 (PDT) Message-ID: <191bc676b6e091844291b2752a7b13fba0f3d4b6.camel@notnullmakers.com> Subject: Re: Discussion of major PBS restore speedup in proxmox-backup-qemu From: Adam Kalisz <adam.kalisz@notnullmakers.com> To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com> Date: Tue, 24 Jun 2025 12:11:28 +0200 In-Reply-To: <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com> References: <9995c68d9c0d6e699578f5a45edb2731b5724ef1.camel@notnullmakers.com> <32645c96c7a1e247202d9d34e6102f08a7f08c97.camel@groupe-cyllene.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.1-1 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com,notnullmakers.com] Hi Alexandre, yes, having configurable CONCURRENT_REQUESTS and max_blocking_threads would be great. However we would need to wire it up all the way to qmrestore or similar or ensure it is read from some env vars. I didn't feel confident to introduce this kind of infrastructure as a first time contribution. Btw. in this case the concurrency applies mostly to fetching requests the writer is single thread and there should still be reasonable locality. If you have a spinning rust setup I would be very glad if you could do a performance test vs the current implementation. Best regards Adam Kalisz On Tue, 2025-06-24 at 09:09 +0000, DERUMIER, Alexandre wrote: > Hi,=20 >=20 > nice work ! >=20 > Could it be possible to have an option to configure the > CONCURRENT_REQUESTS=C2=A0 ? >=20 > (to avoid to put too much load on slow spinning storage) >=20 >=20 >=20 >=20 > -------- Message initial -------- > De: Adam Kalisz <adam.kalisz@notnullmakers.com> > =C3=80: pve-devel@lists.proxmox.com > Objet: Discussion of major PBS restore speedup in proxmox-backup-qemu > Date: 23/06/2025 18:10:01 >=20 > Hi list, >=20 > before I go through all the hoops to submit a patch I wanted to > discuss > the current form of the patch that can be found here: >=20 > https://github.com/NOT-NULL-Makers/proxmox-backup- > qemu/commit/e91f09cfd1654010d6205d8330d9cca71358e030 >=20 > The speedup process was discussed here: >=20 > https://forum.proxmox.com/threads/abysmally-slow-restore-from- > backup.133602/ >=20 > The current numbers are: >=20 > With the most current snapshot of a VM with 10 GiB system disk and 2x > 100 GiB disks with random data: >=20 > Original as of 1.5.1: > 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D11.78s,=C2=A0 speed=3D869.34M= B/s > 100 GiB random 1: duration=3D412.85s, speed=3D248.03MB/s > 100 GiB random 2: duration=3D422.42s, speed=3D242.41MB/s >=20 > With the 12-way concurrent fetching: >=20 > 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D2.05s,=C2=A0=C2=A0 speed=3D49= 91.99MB/s > 100 GiB random 1: duration=3D100.54s, speed=3D1018.48MB/s > 100 GiB random 2: duration=3D100.10s, speed=3D1022.97MB/s >=20 > The hardware is on the PVE side: > 2x Intel Xeon Gold 6244, 1 TB RAM, 2x 100 Gbps Mellanox, 14x Samsung > NVMe 3,8 TB drives in RAID10 using mdadm/ LVM-thin. >=20 > On the PBS side: > 2x Intel Xeon Gold 6334, 1 TB RAM, 2x 100 Gbps Mellanox, 8x Samsung > NVMe in RAID using 4 ZFS mirrors with recordsize 1M, lz4 compression. >=20 > Similar or slightly better speeds were achieved on Hetzner AX52 with > AMD Ryzen 7 7700 with 64 GB RAM and 2x 1 TB NVMe in stripe on PVE > with > recordsize 16k connected to another Hetzner AX52 using a 10 Gbps > connection. The PBS has normal NVMe ZFS mirror again with recordsize > 1M. >=20 > On bigger servers a 16-way concurrency was even better on smaller > servers with high frequency CPUs 8-way concurrency performed better. > The 12-way concurrency is a compromise. We seem to hit a bottleneck > somewhere in the realm of TLS connection and shallow buffers. The > network on the 100 Gbps servers can support up to about 3 GBps > (almost > 20 Gbps) of traffic in a single TCP connection using mbuffer. The > storage can keep up with such a speed. >=20 > Before I submit the patch, I would also like to do the most up to > date > build but I have trouble updating my build environment to reflect the > latest commits. What do I have to put in my /etc/apt/sources.list to > be > able to install e.g. librust-cbindgen-0.27+default-dev librust-http- > body-util-0.1+default-dev librust-hyper-1+default-dev and all the > rest? >=20 > This work was sponsored by =C4=8CMIS s.r.o. and consulted with the Genera= l > Manager V=C3=A1clav Sv=C3=A1tek (=C4=8CMIS), Daniel =C5=A0karda (NOT NULL= Makers s.r.o.) > and Linux team leader Roman M=C3=BCller (=C4=8CMIS). >=20 > Best regards > Adam Kalisz --===============2911394534801260041== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============2911394534801260041==--