From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 767081FF15C for ; Fri, 27 Jun 2025 21:24:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 87C321AC4B; Fri, 27 Jun 2025 21:25:18 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Fri, 27 Jun 2025 21:24:25 +0200 MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Adam Kalisz via pve-devel Precedence: list Cc: Adam Kalisz X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] [PATCH PBS restore 0/1] Speedup PBS restore with concurrent Content-Type: multipart/mixed; boundary="===============2563737609333581865==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============2563737609333581865== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id B73F1D2458 for ; Fri, 27 Jun 2025 21:25:17 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 962601A6E8 for ; Fri, 27 Jun 2025 21:24:47 +0200 (CEST) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 27 Jun 2025 21:24:46 +0200 (CEST) Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-60867565fb5so4283168a12.3 for ; Fri, 27 Jun 2025 12:24:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1751052280; x=1751657080; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=BW3fNBZZF68oHzeeJdVyDZHEQIjAsAo5OAT1AmRtklY=; b=EziFhMj+/hTOU3TDpN+a5wQEJVhBkZLx8PmapthGAKMRSjHX+pOfG/aTIgzPtYBXPj jEyfTBHtU23/4Fj7t9zKe+mC5A8y6ro6vBAA2coDf02CXJ08jGWFwQAaLePfmtZg/+cy Qijz/gmrfq12ElFHUMPh01uruLTYi7mznqYelTK+/u0zHenTXriH1yPQg5ftGbYv/AHK 1M9RbI+nLe8ktXjraO++m6PTuSMkZJhqJ3kvqcgjaZ9aNcn0rxD905x8HGztDjeBvynn /JfLKvIu/Wn1aNHLDpfiMVT210J77YIz1+F3eHPbhWpg/hnNee6LuEHuDaPRJHru9NYx zx7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751052280; x=1751657080; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BW3fNBZZF68oHzeeJdVyDZHEQIjAsAo5OAT1AmRtklY=; b=OGTVxcq2NgHz71neNRsQoRjKp/hPpR2qSMSaIM2qP8tH9QNb+z1hyK6d3wRtZabtAS CriMD1M648KUG24Kv2ikDVImhA26E4Ry0xaEtR3ew+LhnjM91NwkXAE0YC7X8CHLrplv JK74n3PxSxKfeX7EESYriXE99oKn48Hps924oAlUWCAqh7SMOm7Jqk8DJu9N68WMTxRQ 7Iq1Ki//GAGwcRfL45rROs1MfjPV/7Vm0E/pLEi+5qBUSDwCunX1H2Fb3b88RIgy8UNX y+Cak4SK8xHAtaZIjAK1Q/aXcCBqyd6wxqyRqGcrAAl9e/BbIZv/Z2WskkCID4uQDInj IFFA== X-Gm-Message-State: AOJu0Yxm35BVA5z5xOtUXFDCpHZNMpOPRYMKkLgkS/HKt/uhVrAxPJpB eD5fRRUFBtOxTs23Q8cYKFq4dt3n1b4Ned04G5PaWd9sBskdNVy5n/5sWTIF7/TMWVvLlJspPAa bTI1I X-Gm-Gg: ASbGnctjbmyeXNrcg+Iv5by30l3IrVY/+VU96XFspkhIIdh+zO9i8t1iOiovIluIWL/ ZcOjuFFPptIBrAAn8C14Slg4Tzpip1/Qr/vLa+F0BTofvInbPpe+O5V7M3hM/IHSz7+jg2mn8Fg EWsgB5RK1deOtSAS6C936WZDggBuwMRJ8lQv6jcwvyxqEvdMbI2ILxAFE+G6Niyy08xpNGCeq58 852LJuedmlmwFFd6slxszvyGdvX2aUrrVJfQWa60QENn+GgYyRx3uafxLuOBdG8EIjLTNcge9ds i95J8dwStRPtleToEm22aY2fy9HS/KeS4o6uXUmvf68gN58F63Qg02QQNPdiRc1Tu2EQhZIh X-Google-Smtp-Source: AGHT+IF6uX5THlmaT2B3bNCUBn0Moxks8/CvUBzyUaO/cSd6bBcwo7MpNliuEqqzQ6g6u23B9jVikg== X-Received: by 2002:a05:6402:455a:b0:607:f513:4800 with SMTP id 4fb4d7f45d1cf-60c88b56830mr3621345a12.10.1751052279298; Fri, 27 Jun 2025 12:24:39 -0700 (PDT) Received: from localhost.localdomain ([2a02:8308:299:4600::5753]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-60c8290e37csm1862340a12.31.2025.06.27.12.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Jun 2025 12:24:38 -0700 (PDT) From: Adam Kalisz To: pve-devel@lists.proxmox.com Subject: [PATCH PBS restore 0/1] Speedup PBS restore with concurrent Date: Fri, 27 Jun 2025 21:24:25 +0200 Message-ID: <20250627192426.582978-1-adam.kalisz@notnullmakers.com> X-Mailer: git-send-email 2.47.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs,notnullmakers.com] Speedup PBS restore by fetching chunks concurrently using a configurable number of threads. 4 threads are used by default, the environment variable PBS_RESTORE_CONCURRENCY should be between 1 and 32 inclusive. The upper limit is max_blocking_threads in BackupClient. Adam Kalisz (1): Make retrieval of chunks concurrent when restoring backup, add more statistics on chunk fetching and writing to storage. Allow configuring the number of threads fetching chunks using PBS_RESTORE_CONCURRENCY environment variable. src/restore.rs | 101 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 84 insertions(+), 17 deletions(-) -- 2.47.2 --===============2563737609333581865== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============2563737609333581865==--