[pbs-devel] [PATCH proxmox{, -backup} v2 0/7] parallelize chunk reads in verification

[Nicolas Frey <n.frey@proxmox.com>]

This patch series aims to expand on Dominik's series [0] written for
pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
thread pool from the verification.

The number of threads was previously hard-coded, but is now
configurable via the API and GUI with new properties called
`{read,verify}-threads`, similarly to tape backups.

In my local tests I measured the following speed difference:
verified a single snapshot with ~32 GiB (4x the RAM size) with 4
cores (just changing the read threads)

1 thread:    ~440MiB/s
2 threads:   ~780MiB/s
4 threads:   ~1140MiB/s

[0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t

Changes since v1, thanks to Chris:
* define dedicated schema for {worker,read,verify}-threads
* rebase proxmox-backup
* introduce new state struct `IndexVerifyState` to reduce the amount
  of Arc clones and overall better bundling
* adjust update endpoint and UI to the new properties

proxmox:

Nicolas Frey (3):
  pbs-api-types: add schema for {worker,read,verify}-threads
  pbs-api-types: jobs: add {read,verify}-threads to
    VerificationJobConfig
  pbs-api-types: use worker-threads schema for TapeBackupJobSetup

 pbs-api-types/src/datastore.rs | 14 ++++++++++++++
 pbs-api-types/src/jobs.rs      | 20 +++++++++++++++-----
 2 files changed, 29 insertions(+), 5 deletions(-)


proxmox-backup:

Nicolas Frey (4):
  api: verify: move chunk loading into parallel handler
  api: verify: determine the number of threads to use with
    {read,verify}-threads
  api: verify: add {read,verify}-threads to update endpoint
  ui: verify: add option to set number of threads for job

 src/api2/admin/datastore.rs    |  18 +++-
 src/api2/backup/environment.rs |   2 +-
 src/api2/config/verify.rs      |  16 ++++
 src/backup/verify.rs           | 149 ++++++++++++++++++++-------------
 src/server/verify_job.rs       |   7 +-
 www/window/VerifyAll.js        |  18 ++++
 www/window/VerifyJobEdit.js    |  24 ++++++
 7 files changed, 172 insertions(+), 62 deletions(-)


Summary over all repositories:
  9 files changed, 201 insertions(+), 67 deletions(-)

-- 
Generated by git-murpp 0.8.1

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel