all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification
@ 2025-11-13  9:31 Nicolas Frey
  2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox v4 1/3] pbs-api-types: add schema for {worker, read, verify}-threads Nicolas Frey
                   ` (15 more replies)
  0 siblings, 16 replies; 18+ messages in thread
From: Nicolas Frey @ 2025-11-13  9:31 UTC (permalink / raw)
  To: pbs-devel

This patch series aims to expand on Dominik's series [0] written for
pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
thread pool from the verification.

The number of threads was previously hard-coded, but is now
configurable via the API and GUI with new properties called
`{read,verify}-threads`, similarly to tape backups.

The number of threads should also be configurable through tuning
options or datastore config (as discussed by Chris & Thomas on list),
which can be added in a follow up patch series.

In my local tests I measured the following speed difference:
verified a single snapshot with ~32 GiB (4x the RAM size) with 4
cores (just changing the read threads)

1 thread:    ~440MiB/s
2 threads:   ~780MiB/s
4 threads:   ~1140MiB/s

[0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t

Changes since v3, thanks Chris!:
* correct the ordering of patches
* move thread schemas into jobs.rs, rename them and add more specific
  descriptions
* add api schema for {read,verify}-threads in backup manager
* add doc for new parameters
* rebase

Changes since v2:
* split move to parallel handler into 2 seperate commits
* add constructor to `IndexVerifyState` and clean up nits

Changes since v1, thanks to Chris:
* define dedicated schema for {worker,read,verify}-threads
* rebase proxmox-backup
* introduce new state struct `IndexVerifyState` to reduce the amount
  of Arc clones and overall better bundling
* adjust update endpoint and UI to the new properties

proxmox:

Nicolas Frey (3):
  pbs-api-types: add schema for {worker,read,verify}-threads
  pbs-api-types: jobs: add {read,verify}-threads to
    VerificationJobConfig
  pbs-api-types: use worker-threads schema for TapeBackupJobSetup

 pbs-api-types/src/jobs.rs | 41 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)


proxmox-backup:

Nicolas Frey (9):
  verify: correct typo in comment
  verify: introduce new state struct
  verify: refactor into associated functions to use new state struct
  verify: move chunk loading into parallel handler
  verify: determine the number of threads to use with
    {read,verify}-threads
  verify: add {read,verify}-threads to update endpoint
  verify: add {read,verify}-threads to api schema in backup manager
  ui: verify: add option to set number of threads for job
  docs: verify: document {read,verify}-threads and update screenshot

 .../pbs-gui-datastore-verifyjob-add.png       | Bin 15455 -> 41433 bytes
 docs/maintenance.rst                          |  11 +-
 src/api2/admin/datastore.rs                   |  16 +-
 src/api2/backup/environment.rs                |   4 +-
 src/api2/config/verify.rs                     |  16 ++
 src/backup/verify.rs                          | 171 +++++++++++-------
 src/bin/proxmox-backup-manager.rs             |  11 +-
 src/server/verify_job.rs                      |   7 +-
 www/window/VerifyAll.js                       |  18 ++
 www/window/VerifyJobEdit.js                   |  24 +++
 10 files changed, 210 insertions(+), 68 deletions(-)


Summary over all repositories:
  11 files changed, 246 insertions(+), 73 deletions(-)

-- 
Generated by git-murpp 0.8.1

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2025-11-14 22:17 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-11-13  9:31 [pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox v4 1/3] pbs-api-types: add schema for {worker, read, verify}-threads Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox v4 2/3] pbs-api-types: jobs: add {read, verify}-threads to VerificationJobConfig Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox v4 3/3] pbs-api-types: use worker-threads schema for TapeBackupJobSetup Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 1/9] verify: correct typo in comment Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 2/9] verify: introduce new state struct Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 3/9] verify: refactor into associated functions to use " Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 4/9] verify: move chunk loading into parallel handler Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 5/9] verify: determine the number of threads to use with {read, verify}-threads Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 6/9] verify: add {read, verify}-threads to update endpoint Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 7/9] verify: add {read, verify}-threads to api schema in backup manager Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 8/9] ui: verify: add option to set number of threads for job Nicolas Frey
2025-11-13  9:31 ` [pbs-devel] [PATCH proxmox-backup v4 9/9] docs: verify: document {read, verify}-threads and update screenshot Nicolas Frey
2025-11-14 10:41   ` Christian Ebner
2025-11-13 12:53 ` [pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification Fabian Grünbichler
2025-11-14 10:46 ` Christian Ebner
2025-11-14 21:36 ` [pbs-devel] applied: " Thomas Lamprecht
2025-11-14 22:17 ` Thomas Lamprecht

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal