[PATCH proxmox{,-backup} v7 0/9] fix #4182: concurrent group pull/push support for sync jobs

[Christian Ebner <c.ebner@proxmox.com>]

Syncing contents from/to a remote source via a sync job suffers from
low throughput on high latency networks because of limitations by the
HTTP/2 connection, as described in [0]. To improve, syncing multiple
groups in parallel by establishing multiple reader instances has been
suggested.

This patch series implements the functionality by adding the sync job
configuration property `worker-threads`, allowing to define the
number of groups pull/push tokio tasks to be executed in parallel on
the runtime during each job.

Examplary configuration:
```
sync: s-8764c440-3a6c
	ns
	owner root@pam
	remote local
	remote-ns
	remote-store push-target-store
	remove-vanished false
	store datastore
	sync-direction push
	worker-threads 4
```

Since log messages are now also written concurrently, prefix logs
related to groups, snapshots and archives with their respective
context prefix and add context to error messages.

To reduce interwoven log messages from log lines arriving in fast
succession from different group workers, implement a buffer logic to
keep up to 5 lines buffered with a timeout of 1 second. This helps to
follow log lines.

Further, improve logging especially for sync jobs in push direction,
which only displayed limited information so far.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182

Change since version 6 (thanks @Fabian):
- Implement LogSender trait and UnbufferedLogLineSender to completely
  bypass BufferedLogger in non-parallel case.
- Store per-label last logged timestamp and use that to flush in main
  logger loop to assure labels are also flushed when other labels are
  receiving new messages.
- Refactor internal logger methods and fix various docstrings and comments.
- Make sure all available results in bounded join set are returned when
  spawning a new task, not just the first one.
- Adapt accounting logic for store progress with respect to shared group
  progress, to improve logging output for parallel syncs.
- Adapt snapshot-archive prefix separator from {snapshot}: {archive} to
  {snapshot}/{archive}.
- Conditionally show either percentual progress for sequential syncs or
  snapshot/group count based progress for parallel syncs.

Change since version 5 (thanks @Fabian):
- Implement buffered logger for better grouping of fast succession log lines
- Refactor group worker into standalone BoundedJoinSet implementation.
- Improve log output by using better prefixes
- Add missing error contexts

Change since version 4:
- Use dedicated tokio tasks to run in parallel on different runtime threads,
  not just multiple concurrent futures on the same thread.
- Rework store progress accounting logic to avoid mutex locks when possible,
  use atomic counters instead.
- Expose setting also in the sync job edit window, not just the config.


proxmox:

Christian Ebner (1):
  pbs api types: add `worker-threads` to sync job config

 pbs-api-types/src/jobs.rs | 11 +++++++++++
 1 file changed, 11 insertions(+)


proxmox-backup:

Christian Ebner (8):
  tools: implement buffered logger for concurrent log messages
  tools: add bounded join set to run concurrent tasks bound by limit
  api: config/sync: add optional `worker-threads` property
  fix #4182: server: sync: allow pulling backup groups in parallel
  server: pull: prefix log messages and add error context
  server: sync: allow pushing groups concurrently
  server: push: prefix log messages and add additional logging
  ui: expose group worker setting in sync job edit window

 pbs-tools/Cargo.toml              |   3 +
 pbs-tools/src/bounded_join_set.rs |  74 ++++++
 pbs-tools/src/buffered_logger.rs  | 253 ++++++++++++++++++
 pbs-tools/src/lib.rs              |   2 +
 src/api2/config/sync.rs           |  10 +
 src/api2/pull.rs                  |   9 +-
 src/api2/push.rs                  |   8 +-
 src/server/pull.rs                | 415 +++++++++++++++++++++++-------
 src/server/push.rs                | 351 +++++++++++++++++++++----
 src/server/sync.rs                |  40 ++-
 www/window/SyncJobEdit.js         |  11 +
 11 files changed, 1018 insertions(+), 158 deletions(-)
 create mode 100644 pbs-tools/src/bounded_join_set.rs
 create mode 100644 pbs-tools/src/buffered_logger.rs


Summary over all repositories:
  12 files changed, 1029 insertions(+), 158 deletions(-)

-- 
Generated by murpp 0.11.0