From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
	by lore.proxmox.com (Postfix) with ESMTPS id BB4F71FF38E
	for <inbox@lore.proxmox.com>; Tue,  7 May 2024 09:30:25 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id EA3EF815C;
	Tue,  7 May 2024 09:30:29 +0200 (CEST)
From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Tue,  7 May 2024 09:29:52 +0200
Message-Id: <20240507072955.364206-1-d.csapak@proxmox.com>
X-Mailer: git-send-email 2.39.2
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results:  0
 AWL 0.016 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DMARC_MISSING             0.1 Missing DMARC policy
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
 URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
 information. [proxmox.com, datastore.rs, verify.rs]
Subject: [pbs-devel] [PATCH proxmox-backup v2 0/3] tape/verify: improve
 multithreaded
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>

this is a replacement series for my previous patches[0]

in this version, i reversed the order of patches, so that the tape
improvement comes first. I did this, because for tapes I could measure a
very real and beneficial improvement, while for verification the
situation is more complicated.

There we have currently only the hashing of chunks parallelized, not the
reading of data, but this could also improve throughput (as seen in the
benchmarks of 3/3).

patch 2 and 3 are RFCs because I'm not sure how to proceed with these.
One one hand they can improve the situation (when both are applied) but
makes it worse for other situations.

We could go even further and introduce another
'verification-read-threads' option and have a seperate thread pool for
reading the data and another for verifying them, but I'm not sure if it
would be worth it to spend even more time on it (incl testing, which
takes a lot of time)

changes from v1:
* reordered code from verify,tape to tape,verify
* marked verify patches as rfc
* use a struct for saving the values in the datastore (as thomas
  suggested)
* introduces another verify patch to merge the chunk loading into the
  worker threads

0: https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009162.html

Dominik Csapak (3):
  tape: introduce a tape backup read thread tuning option
  verify: add tuning option for number of threads to use
  verify: move chunk loading into the worker threads

 pbs-api-types/src/datastore.rs              | 15 +++++
 pbs-datastore/src/datastore.rs              | 31 ++++++++++
 src/backup/verify.rs                        | 67 +++++++++------------
 src/tape/pool_writer/new_chunks_iterator.rs | 45 +++++++++-----
 www/Utils.js                                | 10 +++
 www/datastore/OptionView.js                 | 16 +++++
 6 files changed, 130 insertions(+), 54 deletions(-)

-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel