From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 0E6D41FF185 for ; Mon, 23 Jun 2025 18:10:10 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2C35617316; Mon, 23 Jun 2025 18:10:42 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Mon, 23 Jun 2025 18:10:01 +0200 MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Adam Kalisz via pve-devel Precedence: list Cc: Adam Kalisz X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] Discussion of major PBS restore speedup in proxmox-backup-qemu Content-Type: multipart/mixed; boundary="===============7606155690521019207==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============7606155690521019207== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 48142D0673 for ; Mon, 23 Jun 2025 18:10:40 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2306317262 for ; Mon, 23 Jun 2025 18:10:10 +0200 (CEST) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 23 Jun 2025 18:10:09 +0200 (CEST) Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-60707b740a6so6632942a12.0 for ; Mon, 23 Jun 2025 09:10:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1750695002; x=1751299802; darn=lists.proxmox.com; h=mime-version:user-agent:content-transfer-encoding:date:to:from :subject:message-id:from:to:cc:subject:date:message-id:reply-to; bh=oan795gTR9aNHKx6ODjb0sf1qLKmURWmjHqzusWeKkA=; b=aPCzxCV5ilMu45s1n6iTh1UxEl7Ae0anKQQEUw929x9VFcDjN7g/BQ3hSOASUbVMTW URtc9/UFVBXypXF31dLOW1lG5/PQ/YtAnOzABGb/R5fOsEle6xgwrmb631jBvOJETRwi HvuCZU+hpPBZEm5pEObtEDuapp2F7jfNf0kS4N3cZzzvy/9nMABTt1iUpLpU7AWHKnGN Tma9l1cVNbphkRO0MBLoO7ZXkS67TBzYcIB1LC314NmHBUBNH8XcPk6PW2eBNHLEyG8w WlphYxizdxKezBi6Nv+9aoximcgfPL26PmuKM4iHY3L17CoMmMCNB9xF6UsI+58YXvPN uVmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750695002; x=1751299802; h=mime-version:user-agent:content-transfer-encoding:date:to:from :subject:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=oan795gTR9aNHKx6ODjb0sf1qLKmURWmjHqzusWeKkA=; b=VnUoPvAdkOYYkCRYdA8QhWFkZW8ibFl43tGSWtfvU8E0HmDgaD2l+48gHxPoPmxdeV qtYQOfQvRLLcDupTGwfcpup731/4cCV39kHuDvvg3SGIyT3YI1N5DWmP4d6DOvpfSY22 xfr9ZhzxYnA7ELkvWcAqw+k7hSIOGCPuzhXfZPWBfWWPDy64m2EwNV98lAgv5X4S6piU 8A/3ji200SJLpqmsS5/XDEl0WWzeD67xALvBdHTeSxM+doZAgN1dsNZkWrf5IHZBPRRY vnlOBih0c4e0HtvtyX3lLJoci9JON2IPHhnsK4aLMFmRpKmPwlVX6SKfzEg2Yqp4ITZa DgYQ== X-Gm-Message-State: AOJu0YzxKhR8GFfywqqsJeAz+pgYX3Ljb5a/SGXIHR64eYGABVUJNSwM ALZhNeq1KZsmJ0tkj9qTFNWvPdg9hjwfM11kEo0AEKvusJZRrpDCTzFdNdU24nG5Nb+DUOYFZFK yUtSa X-Gm-Gg: ASbGncu2bzzRy1n0iuFagWw36AMQ4h8kYPysoTboXrrm6Q1kkuLn69JG2g+tUVIxgUt 4evdu0GrAJG2rRqtf8gR8WXsgmueCZGHvRcuMeU+G0j9oxGvAk8eAnW7/XbDkPd+kN63f8iCxNL mepJi9Pu5A1mXwDv1UiSJkFifG+Pvd0yXC7n8UUQY6fZFuLuW1RxOF7FhXjxG2K56lz8tPCwVK5 E9TCGFGSGuj18HXcPSNsXChN2tUeEJ3OMMHUviqbsh/DEdWFrthJaUNlEIqqpj5v9h9x30mRxwH 7k/HXljgJakWuynTqgXSJd6vOqphHLGvipN6WXtDFt41lFmm1dJk29PvOtb13PjfaDpBz4XckK+ VjcNKPYzsNQ== X-Google-Smtp-Source: AGHT+IHQwwEpGOYUxOsJQO1nBA9kNTeLadfxtCcuT6PZhQB82VIIhsBox5wWJgmIQGYAV1T62+poOA== X-Received: by 2002:aa7:dacd:0:b0:606:f836:c657 with SMTP id 4fb4d7f45d1cf-60a1d19085cmr10828709a12.28.1750695002453; Mon, 23 Jun 2025 09:10:02 -0700 (PDT) Received: from ?IPv6:2a02:8308:299:4600::5753? ([2a02:8308:299:4600::5753]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-60a18cbd61asm6213296a12.66.2025.06.23.09.10.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jun 2025 09:10:02 -0700 (PDT) Message-ID: <9995c68d9c0d6e699578f5a45edb2731b5724ef1.camel@notnullmakers.com> Subject: Discussion of major PBS restore speedup in proxmox-backup-qemu From: Adam Kalisz To: pve-devel@lists.proxmox.com Date: Mon, 23 Jun 2025 18:10:01 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.1-1 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Hi list, before I go through all the hoops to submit a patch I wanted to discuss the current form of the patch that can be found here: https://github.com/NOT-NULL-Makers/proxmox-backup-qemu/commit/e91f09cfd1654= 010d6205d8330d9cca71358e030 The speedup process was discussed here: https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602= / The current numbers are: With the most current snapshot of a VM with 10 GiB system disk and 2x 100 GiB disks with random data: Original as of 1.5.1: 10 GiB system: duration=3D11.78s, speed=3D869.34MB/s 100 GiB random 1: duration=3D412.85s, speed=3D248.03MB/s 100 GiB random 2: duration=3D422.42s, speed=3D242.41MB/s With the 12-way concurrent fetching: 10 GiB system: duration=3D2.05s, speed=3D4991.99MB/s 100 GiB random 1: duration=3D100.54s, speed=3D1018.48MB/s 100 GiB random 2: duration=3D100.10s, speed=3D1022.97MB/s The hardware is on the PVE side: 2x Intel Xeon Gold 6244, 1 TB RAM, 2x 100 Gbps Mellanox, 14x Samsung NVMe 3,8 TB drives in RAID10 using mdadm/ LVM-thin. On the PBS side: 2x Intel Xeon Gold 6334, 1 TB RAM, 2x 100 Gbps Mellanox, 8x Samsung NVMe in RAID using 4 ZFS mirrors with recordsize 1M, lz4 compression. Similar or slightly better speeds were achieved on Hetzner AX52 with AMD Ryzen 7 7700 with 64 GB RAM and 2x 1 TB NVMe in stripe on PVE with recordsize 16k connected to another Hetzner AX52 using a 10 Gbps connection. The PBS has normal NVMe ZFS mirror again with recordsize 1M. On bigger servers a 16-way concurrency was even better on smaller servers with high frequency CPUs 8-way concurrency performed better. The 12-way concurrency is a compromise. We seem to hit a bottleneck somewhere in the realm of TLS connection and shallow buffers. The network on the 100 Gbps servers can support up to about 3 GBps (almost 20 Gbps) of traffic in a single TCP connection using mbuffer. The storage can keep up with such a speed. Before I submit the patch, I would also like to do the most up to date build but I have trouble updating my build environment to reflect the latest commits. What do I have to put in my /etc/apt/sources.list to be able to install e.g. librust-cbindgen-0.27+default-dev librust-http- body-util-0.1+default-dev librust-hyper-1+default-dev and all the rest? This work was sponsored by =C4=8CMIS s.r.o. and consulted with the General Manager V=C3=A1clav Sv=C3=A1tek (=C4=8CMIS), Daniel =C5=A0karda (NOT NULL M= akers s.r.o.) and Linux team leader Roman M=C3=BCller (=C4=8CMIS). Best regards Adam Kalisz --===============7606155690521019207== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============7606155690521019207==--