From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 7EB191FF136 for ; Mon, 09 Mar 2026 17:21:45 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 13E853AA2; Mon, 9 Mar 2026 17:21:36 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{,-backup} v5 00/12] fix #4182: concurrent group pull/push support for sync jobs Date: Mon, 9 Mar 2026 17:20:38 +0100 Message-ID: <20260309162050.1047341-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1773073227677 X-SPAM-LEVEL: Spam detection results: 0 AWL -1.011 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.408 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.819 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.903 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: W2VCHZXM5EYGIMSNKYQQHJNAU7FVPIDH X-Message-ID-Hash: W2VCHZXM5EYGIMSNKYQQHJNAU7FVPIDH X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Syncing contents from/to a remote source via a sync job suffers from low throughput on high latency networks because of limitations by the HTTP/2 connection, as described in [0]. To improve, syncing multiple groups in parallel by establishing multiple reader instances has been suggested. This patch series implements the functionality by adding the sync job configuration property `worker-threads`, allowing to define the number of groups pull/push tokio tasks to be executed in parallel on the runtime during each job. Examplary configuration: ``` sync: s-8764c440-3a6c ... store datastore sync-direction push worker-threads 4 ``` Since log messages are now also written concurrently, prefix logs related to groups, snapshots and archives with their respective context prefix and add context to error messages. Further, improve logging especially for sync jobs in push direction, which only displayed limited information so far. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182 Change since version 4 (thanks @Max for review): - Use dedicated tokio tasks to run in parallel on different runtime threads, not just multiple concurrent futures on the same thread. - Rework store progress accounting logic to avoid mutex locks when possible, use atomic counters instead. - Expose setting also in the sync job edit window, not just the config. proxmox: Christian Ebner (1): pbs api types: add `worker-threads` to sync job config pbs-api-types/src/jobs.rs | 11 +++++++++++ 1 file changed, 11 insertions(+) proxmox-backup: Christian Ebner (11): client: backup writer: fix upload stats size and rate for push sync api: config/sync: add optional `worker-threads` property sync: pull: revert avoiding reinstantiation for encountered chunks map sync: pull: factor out backup group locking and owner check sync: pull: prepare pull parameters to be shared across parallel tasks fix #4182: server: sync: allow pulling backup groups in parallel server: pull: prefix log messages and add error context sync: push: prepare push parameters to be shared across parallel tasks server: sync: allow pushing groups concurrently server: push: prefix log messages and add additional logging ui: expose group worker setting in sync job edit window pbs-client/src/backup_stats.rs | 20 +-- pbs-client/src/backup_writer.rs | 4 +- src/api2/config/sync.rs | 10 ++ src/api2/pull.rs | 9 +- src/api2/push.rs | 8 +- src/server/pull.rs | 246 +++++++++++++++++++------------- src/server/push.rs | 178 +++++++++++++++++------ src/server/sync.rs | 90 +++++++++++- www/window/SyncJobEdit.js | 11 ++ 9 files changed, 411 insertions(+), 165 deletions(-) Summary over all repositories: 10 files changed, 422 insertions(+), 165 deletions(-) -- Generated by murpp 0.9.0