From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id DB0ED1FF165 for ; Thu, 3 Jul 2025 10:29:18 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5AE20E09B; Thu, 3 Jul 2025 10:29:56 +0200 (CEST) To: Proxmox VE development discussion Date: Thu, 03 Jul 2025 10:29:43 +0200 In-Reply-To: <644731967.7191.1750761799812@webmail.proxmox.com> References: <1842547039.6962.1750750098094@webmail.proxmox.com> <644731967.7191.1750761799812@webmail.proxmox.com> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Adam Kalisz via pve-devel Precedence: list Cc: Adam Kalisz X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] Discussion of major PBS restore speedup in proxmox-backup-qemu Content-Type: multipart/mixed; boundary="===============1076329132905790159==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============1076329132905790159== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6783AD4B28 for ; Thu, 3 Jul 2025 10:29:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5179EE063 for ; Thu, 3 Jul 2025 10:29:54 +0200 (CEST) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 3 Jul 2025 10:29:52 +0200 (CEST) Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-60700a745e5so10450504a12.3 for ; Thu, 03 Jul 2025 01:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1751531385; x=1752136185; darn=lists.proxmox.com; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=aD53U+web+rqr8u/qBI73b8ARTjplatHO4aoDqZ9iXo=; b=Sd79+v/N6pui6cQ7JEpeEzxYGoIlcEiU2kvwIzqbyK4FknmBGpaztIkJvtWXpzVBWj 9Yd8xjkA2gesTWOKmrC3XeQYErL7V6fdAnbdVRD1ep/j4cl2TdCR4KVQXIS0hyXhKZLU JOjoRRYlew3Ew1yIt5IxSR45lM7lSQOXqbSLBaucZtwNs0YEu05q3jR+pQbzzcTVSRsV ZLZkgwhExSd8oqp82iyV6QDq61VcrmRDAYznF+6tU/Zp1IN6VmNIdxk6+nLH9CAGkZFa f2OEoF2LXv4v/xQHZYuYTCPwOIzkulBAkYBGXlJtT8GVCys2ziMjfOfDYW6oHkxYle4n 6yMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751531385; x=1752136185; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=aD53U+web+rqr8u/qBI73b8ARTjplatHO4aoDqZ9iXo=; b=GCTVsz1a5y/lBo3eZeB0xkV+vA6ewE5lSrtoVHd5bcDtbDCVMOmxZXciKLgUJ+gxPv cPVuBbjBrlrt5aoHkfwNpvEWTZ92/ozP8JFqrej3sRpEovkdsubFYdir0l9OLMeyjZHx IgDjhJQlhNQlz7N3Wu1fhEEWdoXyH+dSrEjXQKzVF8kqstAkRWrZ2XwhpKO75/VlaKaD wB6986ToPatE2ejXvx68zkWupHqyZPT1zSYadR/E7dC0R7D75vQYPApn34xtNSShJ6hb W3jeIXhlrELLw2UzK/IUS0rkPyt8huiqi2MiaTR2lkxmNNMWiXauvbVEndu/aIbT4W/W hmgA== X-Gm-Message-State: AOJu0YwE0n1DwRpGta+Wzxe7sDRcQZTqBJMkGTSxzJ4GGBkf330F5oC0 md4i4dHHvmCi0MwbusuFHaeW4uxAhOIbA2UyfnLAOSRB+iHGEAgip5RePz6ejR8mGB9G6qD+Kva 4kS7v X-Gm-Gg: ASbGncsrIDm46aaHZA8Xgp23HqBkeDVfHtXmA+Ohgzba3YZlkXbZl/v60pW55tU1sbD 8Bnb1YpPIzxF7dsZ4GKwt3UanKVYp43bEwC/4dCgTbxnn6fSpnxmB5qZkKr/8vxxNP9ZbbrbYYP xZfQI9POWFFUPbC9UT1jPiGKO+h/baJcG5M0b1e7QPtaXJAcRGMcHYBJsyBcptpuQ0d+7Nc44eI t9ZmL5ID4CZ7MAH4Pt4aBM9ukmJss2VHFze5JL89114sjyiqJb82XmLnQjavbFjQ/T4MEb6v1w1 2JK52xGbzprZ8LC7Y96s0T7XWAIBNn3VxOmKsTOskhZF4CBL5Qh5e6+LxFcJHgi5mJ9g1dQPDqN /f/AC4c8nfg== X-Google-Smtp-Source: AGHT+IGVnnRtchHjO4SaEAGv/jrF5/TBC7mfmbK9HiWx75iyex5ftUxrW6vgNZ3zO9iLHE2dU40pcw== X-Received: by 2002:a17:907:3e8b:b0:ade:32fa:739e with SMTP id a640c23a62f3a-ae3c2a904e4mr601223366b.2.1751531385240; Thu, 03 Jul 2025 01:29:45 -0700 (PDT) Received: from ?IPv6:2a02:8308:299:4600::5753? ([2a02:8308:299:4600::5753]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ae35363b244sm1215118066b.30.2025.07.03.01.29.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Jul 2025 01:29:44 -0700 (PDT) Message-ID: <204986487f1d360ba068f5a1301ee809fd2846f0.camel@notnullmakers.com> Subject: Re: [pve-devel] Discussion of major PBS restore speedup in proxmox-backup-qemu From: Adam Kalisz To: Proxmox VE development discussion Date: Thu, 03 Jul 2025 10:29:43 +0200 In-Reply-To: <644731967.7191.1750761799812@webmail.proxmox.com> References: <1842547039.6962.1750750098094@webmail.proxmox.com> <644731967.7191.1750761799812@webmail.proxmox.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.1-1 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com,notnullmakers.com] Hi, On Friday I have submitted the patch with a slight edit to allow setting the number of threads from an environment variable. On Tue, 2025-06-24 at 12:43 +0200, Fabian Gr=C3=BCnbichler wrote: >=20 > > Adam Kalisz hat am 24.06.2025 12:22 > > CEST geschrieben: > > Hi Fabian, >=20 > CCing the list again, assuming it got dropped by accident. >=20 > > the CPU usage is higher, I see about 400% for the restore process. > > I > > didn't investigate the original much because it's unbearably slow. > >=20 > > Yes, having configurable CONCURRENT_REQUESTS and > > max_blocking_threads > > would be great. However we would need to wire it up all the way to > > qmrestore or similar or ensure it is read from some env vars. I > > didn't > > feel confident to introduce this kind of infrastructure as a first > > time > > contribution. >=20 > we can guide you if you want, but it's also possible to follow-up on > our end with that as part of applying the change. That would be great, it shouldn't be too much work for somebody more familiar with the project structure where everything needs to be. > > The writer to disk is single thread still so a CPU that can ramp up > > a > > single core to a high frequency/ IPC will usually do better on the > > benchmarks. >=20 > I think that limitation is no longer there on the QEMU side nowadays, > but it would likely require some more changes to actually make use of > multiple threads submitting IO. The storage writing seemed to be less of a bottleneck than the fetching of chunks. It seems to me there still is a bottleneck in the network part because I haven't seen an instance with substantially higher speed than 1.1 GBps. Perhaps we could have a discussion about the backup, restore and synchronization speeds and strategies for debugging and improving the situation after we have taken the intermediate step of improving the restore speed as proposed to gather more feedback from the field? > > What are the chances of this getting accepted more or less as is? >=20 > proper review and discussion of potential follow-ups (no matter who > ends > up doing them) would require submitting a properly signed-off patch > and a CLA - see > https://pve.proxmox.com/wiki/Developer_Documentation#Software_License_and= _Copyright I have cleared the CLA with the Proxmox office last week. > Fabian Adam > > On Tue, 2025-06-24 at 09:28 +0200, Fabian Gr=C3=BCnbichler wrote: > > >=20 > > > > Adam Kalisz via pve-devel hat am > > > > 23.06.2025 18:10 CEST geschrieben: > > > > Hi list, > > >=20 > > > Hi! > > >=20 > > > > before I go through all the hoops to submit a patch I wanted to > > > > discuss > > > > the current form of the patch that can be found here: > > > >=20 > > > > https://github.com/NOT-NULL-Makers/proxmox-backup-qemu/commit/e91f0= 9cfd1654010d6205d8330d9cca71358e030 > > > >=20 > > > > The speedup process was discussed here: > > > >=20 > > > > https://forum.proxmox.com/threads/abysmally-slow-restore-from-backu= p.133602/ > > > >=20 > > > > The current numbers are: > > > >=20 > > > > With the most current snapshot of a VM with 10 GiB system disk > > > > and > > > > 2x > > > > 100 GiB disks with random data: > > > >=20 > > > > Original as of 1.5.1: > > > > 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D11.78s,=C2=A0 speed=3D8= 69.34MB/s > > > > 100 GiB random 1: duration=3D412.85s, speed=3D248.03MB/s > > > > 100 GiB random 2: duration=3D422.42s, speed=3D242.41MB/s > > > >=20 > > > > With the 12-way concurrent fetching: > > > >=20 > > > > 10 GiB system:=C2=A0=C2=A0=C2=A0 duration=3D2.05s,=C2=A0=C2=A0 spee= d=3D4991.99MB/s > > > > 100 GiB random 1: duration=3D100.54s, speed=3D1018.48MB/s > > > > 100 GiB random 2: duration=3D100.10s, speed=3D1022.97MB/s > > >=20 > > > Those numbers do look good - do you also have CPU usage stats > > > before > > > and after? > > >=20 > > > > The hardware is on the PVE side: > > > > 2x Intel Xeon Gold 6244, 1 TB RAM, 2x 100 Gbps Mellanox, 14x > > > > Samsung > > > > NVMe 3,8 TB drives in RAID10 using mdadm/ LVM-thin. > > > >=20 > > > > On the PBS side: > > > > 2x Intel Xeon Gold 6334, 1 TB RAM, 2x 100 Gbps Mellanox, 8x > > > > Samsung > > > > NVMe in RAID using 4 ZFS mirrors with recordsize 1M, lz4 > > > > compression. > > > >=20 > > > > Similar or slightly better speeds were achieved on Hetzner AX52 > > > > with > > > > AMD Ryzen 7 7700 with 64 GB RAM and 2x 1 TB NVMe in stripe on > > > > PVE > > > > with > > > > recordsize 16k connected to another Hetzner AX52 using a 10 > > > > Gbps > > > > connection. The PBS has normal NVMe ZFS mirror again with > > > > recordsize > > > > 1M. > > > >=20 > > > > On bigger servers a 16-way concurrency was even better on > > > > smaller > > > > servers with high frequency CPUs 8-way concurrency performed > > > > better. > > > > The 12-way concurrency is a compromise. We seem to hit a > > > > bottleneck > > > > somewhere in the realm of TLS connection and shallow buffers. > > > > The > > > > network on the 100 Gbps servers can support up to about 3 GBps > > > > (almost > > > > 20 Gbps) of traffic in a single TCP connection using mbuffer. > > > > The > > > > storage can keep up with such a speed. > > >=20 > > > This sounds like it might make sense to make the number of > > > threads > > > configurable (the second lower count can probably be derived from > > > it?) > > > to allow high-end systems to make the most of it, without > > > overloading > > > smaller setups. Or maybe deriving it from the host CPU count > > > would > > > also work? > > >=20 > > > > Before I submit the patch, I would also like to do the most up > > > > to > > > > date > > > > build but I have trouble updating my build environment to > > > > reflect > > > > the > > > > latest commits. What do I have to put in my > > > > /etc/apt/sources.list > > > > to be > > > > able to install e.g. librust-cbindgen-0.27+default-dev librust- > > > > http- > > > > body-util-0.1+default-dev librust-hyper-1+default-dev and all > > > > the > > > > rest? > > >=20 > > > We are currently in the process of rebasing all our repositories > > > on > > > top > > > of the upcoming Debian Trixie release. The built packages are not > > > yet > > > available for public testing, so you'd either need to wait a bit > > > (in > > > the > > > order of a few weeks at most), or submit the patches for the > > > current > > > stable Bookworm-based version and let us forward port them. > > >=20 > > > > This work was sponsored by =C4=8CMIS s.r.o. and consulted with the > > > > General > > > > Manager V=C3=A1clav Sv=C3=A1tek (=C4=8CMIS), Daniel =C5=A0karda (NO= T NULL Makers > > > > s.r.o.) > > > > and Linux team leader Roman M=C3=BCller (=C4=8CMIS). > > >=20 > > > Nice! Looking forward to the "official" patch submission! > > > Fabian --===============1076329132905790159== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============1076329132905790159==--