[pbs-devel] [PATCH v3 proxmox proxmox-backup 0/7] fix #4182: concurrent group pull/push support for sync jobs

[Christian Ebner <c.ebner@proxmox.com>]

Syncing contents from/to a remote source via a sync job suffers from
low throughput on high latency networks because of limitations by the
HTTP/2 connection, as described in [0]. To improve, syncing multiple
groups in parallel by establishing multiple reader instances has been
suggested.

This patch series implements the functionality by adding the sync job
configuration property `parallel-groups`, allowing to define the
number of concurrent groups pull/push futures to be instantiated and
executed for each job.
The property is currently not exposed on the UI, as intended to be
set in the config directly for now.

Examplary configuration:
```
sync: s-8764c440-3a6c
	ns
	owner root@pam
	remote local
	remote-ns
	remote-store push-target-store
	remove-vanished false
	store datastore
	sync-direction push
	parallel-groups 4
```

Since log messages are now also written concurrently, prefix logs
related to groups, snapshots and archives with their respective
context prefix and add context to error messages.

Further, improve logging especially for sync jobs in push direction,
which only displayed limited information so far.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182

proxmox:

Christian Ebner (1):
  pbs api types: add 'parallel-groups' to sync job config

 pbs-api-types/src/jobs.rs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

proxmox-backup:

Christian Ebner (6):
  client: backup writer: fix upload stats size and rate for push sync
  api: config/sync: add optional `parallel-groups` property
  fix #4182: server: sync: allow pulling groups concurrently
  server: pull: prefix log messages and add error context
  server: sync: allow pushing groups concurrently
  server: push: prefix log messages and add additional logging

 pbs-client/src/backup_stats.rs      |  20 +--
 pbs-client/src/backup_writer.rs     |   4 +-
 pbs-datastore/src/store_progress.rs |   2 +-
 src/api2/config/sync.rs             |  10 ++
 src/api2/pull.rs                    |   9 +-
 src/api2/push.rs                    |   9 +-
 src/server/pull.rs                  | 217 +++++++++++++++++-----------
 src/server/push.rs                  | 147 ++++++++++++++-----
 src/server/sync.rs                  |  10 +-
 9 files changed, 292 insertions(+), 136 deletions(-)

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel