[pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification

[Nicolas Frey <n.frey@proxmox.com>]

This patch series aims to expand on Dominik's series [0] written for
pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
thread pool from the verification.

The number of threads was previously hard-coded, but is now
configurable via the API and GUI with new properties called
`{read,verify}-threads`, similarly to tape backups.

The number of threads should also be configurable through tuning
options or datastore config (as discussed by Chris & Thomas on list),
which can be added in a follow up patch series.

In my local tests I measured the following speed difference:
verified a single snapshot with ~32 GiB (4x the RAM size) with 4
cores (just changing the read threads)

1 thread:    ~440MiB/s
2 threads:   ~780MiB/s
4 threads:   ~1140MiB/s

[0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t

Changes since v3, thanks Chris!:
* correct the ordering of patches
* move thread schemas into jobs.rs, rename them and add more specific
  descriptions
* add api schema for {read,verify}-threads in backup manager
* add doc for new parameters
* rebase

Changes since v2:
* split move to parallel handler into 2 seperate commits
* add constructor to `IndexVerifyState` and clean up nits

Changes since v1, thanks to Chris:
* define dedicated schema for {worker,read,verify}-threads
* rebase proxmox-backup
* introduce new state struct `IndexVerifyState` to reduce the amount
  of Arc clones and overall better bundling
* adjust update endpoint and UI to the new properties

proxmox:

Nicolas Frey (3):
  pbs-api-types: add schema for {worker,read,verify}-threads
  pbs-api-types: jobs: add {read,verify}-threads to
    VerificationJobConfig
  pbs-api-types: use worker-threads schema for TapeBackupJobSetup

 pbs-api-types/src/jobs.rs | 41 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)


proxmox-backup:

Nicolas Frey (9):
  verify: correct typo in comment
  verify: introduce new state struct
  verify: refactor into associated functions to use new state struct
  verify: move chunk loading into parallel handler
  verify: determine the number of threads to use with
    {read,verify}-threads
  verify: add {read,verify}-threads to update endpoint
  verify: add {read,verify}-threads to api schema in backup manager
  ui: verify: add option to set number of threads for job
  docs: verify: document {read,verify}-threads and update screenshot

 .../pbs-gui-datastore-verifyjob-add.png       | Bin 15455 -> 41433 bytes
 docs/maintenance.rst                          |  11 +-
 src/api2/admin/datastore.rs                   |  16 +-
 src/api2/backup/environment.rs                |   4 +-
 src/api2/config/verify.rs                     |  16 ++
 src/backup/verify.rs                          | 171 +++++++++++-------
 src/bin/proxmox-backup-manager.rs             |  11 +-
 src/server/verify_job.rs                      |   7 +-
 www/window/VerifyAll.js                       |  18 ++
 www/window/VerifyJobEdit.js                   |  24 +++
 10 files changed, 210 insertions(+), 68 deletions(-)


Summary over all repositories:
  11 files changed, 246 insertions(+), 73 deletions(-)

-- 
Generated by git-murpp 0.8.1

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel