From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id BB4F71FF38E for ; Tue, 7 May 2024 09:30:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id EA3EF815C; Tue, 7 May 2024 09:30:29 +0200 (CEST) From: Dominik Csapak To: pbs-devel@lists.proxmox.com Date: Tue, 7 May 2024 09:29:52 +0200 Message-Id: <20240507072955.364206-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.016 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com, datastore.rs, verify.rs] Subject: [pbs-devel] [PATCH proxmox-backup v2 0/3] tape/verify: improve multithreaded X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" this is a replacement series for my previous patches[0] in this version, i reversed the order of patches, so that the tape improvement comes first. I did this, because for tapes I could measure a very real and beneficial improvement, while for verification the situation is more complicated. There we have currently only the hashing of chunks parallelized, not the reading of data, but this could also improve throughput (as seen in the benchmarks of 3/3). patch 2 and 3 are RFCs because I'm not sure how to proceed with these. One one hand they can improve the situation (when both are applied) but makes it worse for other situations. We could go even further and introduce another 'verification-read-threads' option and have a seperate thread pool for reading the data and another for verifying them, but I'm not sure if it would be worth it to spend even more time on it (incl testing, which takes a lot of time) changes from v1: * reordered code from verify,tape to tape,verify * marked verify patches as rfc * use a struct for saving the values in the datastore (as thomas suggested) * introduces another verify patch to merge the chunk loading into the worker threads 0: https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009162.html Dominik Csapak (3): tape: introduce a tape backup read thread tuning option verify: add tuning option for number of threads to use verify: move chunk loading into the worker threads pbs-api-types/src/datastore.rs | 15 +++++ pbs-datastore/src/datastore.rs | 31 ++++++++++ src/backup/verify.rs | 67 +++++++++------------ src/tape/pool_writer/new_chunks_iterator.rs | 45 +++++++++----- www/Utils.js | 10 +++ www/datastore/OptionView.js | 16 +++++ 6 files changed, 130 insertions(+), 54 deletions(-) -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel