[PATCH proxmox{,-backup} v5 00/12] fix #4182: concurrent group pull/push support for sync jobs

[Christian Ebner <c.ebner@proxmox.com>]

Syncing contents from/to a remote source via a sync job suffers from
low throughput on high latency networks because of limitations by the
HTTP/2 connection, as described in [0]. To improve, syncing multiple
groups in parallel by establishing multiple reader instances has been
suggested.

This patch series implements the functionality by adding the sync job
configuration property `worker-threads`, allowing to define the
number of groups pull/push tokio tasks to be executed in parallel on
the runtime during each job.

Examplary configuration:
```
sync: s-8764c440-3a6c
        ...
	store datastore
	sync-direction push
	worker-threads 4
```

Since log messages are now also written concurrently, prefix logs
related to groups, snapshots and archives with their respective
context prefix and add context to error messages.

Further, improve logging especially for sync jobs in push direction,
which only displayed limited information so far.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182

Change since version 4 (thanks @Max for review):
- Use dedicated tokio tasks to run in parallel on different runtime threads,
  not just multiple concurrent futures on the same thread.
- Rework store progress accounting logic to avoid mutex locks when possible,
  use atomic counters instead.
- Expose setting also in the sync job edit window, not just the config.


proxmox:

Christian Ebner (1):
  pbs api types: add `worker-threads` to sync job config

 pbs-api-types/src/jobs.rs | 11 +++++++++++
 1 file changed, 11 insertions(+)


proxmox-backup:

Christian Ebner (11):
  client: backup writer: fix upload stats size and rate for push sync
  api: config/sync: add optional `worker-threads` property
  sync: pull: revert avoiding reinstantiation for encountered chunks map
  sync: pull: factor out backup group locking and owner check
  sync: pull: prepare pull parameters to be shared across parallel tasks
  fix #4182: server: sync: allow pulling backup groups in parallel
  server: pull: prefix log messages and add error context
  sync: push: prepare push parameters to be shared across parallel tasks
  server: sync: allow pushing groups concurrently
  server: push: prefix log messages and add additional logging
  ui: expose group worker setting in sync job edit window

 pbs-client/src/backup_stats.rs  |  20 +--
 pbs-client/src/backup_writer.rs |   4 +-
 src/api2/config/sync.rs         |  10 ++
 src/api2/pull.rs                |   9 +-
 src/api2/push.rs                |   8 +-
 src/server/pull.rs              | 246 +++++++++++++++++++-------------
 src/server/push.rs              | 178 +++++++++++++++++------
 src/server/sync.rs              |  90 +++++++++++-
 www/window/SyncJobEdit.js       |  11 ++
 9 files changed, 411 insertions(+), 165 deletions(-)


Summary over all repositories:
  10 files changed, 422 insertions(+), 165 deletions(-)

-- 
Generated by murpp 0.9.0