From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 819B41FF13B for ; Wed, 22 Apr 2026 15:18:42 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5B33C1F2C7; Wed, 22 Apr 2026 15:18:42 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{,-backup} v8 00/10] fix #4182: concurrent group pull/push support for sync jobs Date: Wed, 22 Apr 2026 15:18:10 +0200 Message-ID: <20260422131820.769620-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776863829361 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.071 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com,jobs.rs,pull.rs,push.rs,lib.rs,sync.rs] Message-ID-Hash: ZGESBSZEXO6KICELM7M5UCUKX5HTH7QQ X-Message-ID-Hash: ZGESBSZEXO6KICELM7M5UCUKX5HTH7QQ X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Syncing contents from/to a remote source via a sync job suffers from low throughput on high latency networks because of limitations by the HTTP/2 connection, as described in [0]. To improve, syncing multiple groups in parallel by establishing multiple reader instances has been suggested. This patch series implements the functionality by adding the sync job configuration property `worker-threads`, allowing to define the number of groups pull/push tokio tasks to be executed in parallel on the runtime during each job. Examplary configuration: ``` sync: s-8764c440-3a6c ns owner root@pam remote local remote-ns remote-store push-target-store remove-vanished false store datastore sync-direction push worker-threads 4 ``` Since log messages are now also written concurrently, prefix logs related to groups, snapshots and archives with their respective context prefix and add context to error messages. To reduce interwoven log messages from log lines arriving in fast succession from different group workers, implement a buffer logic to keep up to 5 lines buffered with a timeout of 1 second. This helps to follow log lines. Further, improve logging especially for sync jobs in push direction, which only displayed limited information so far. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182 Change since version 6 (thanks for off list suggestions @Fabian): - Fold UnbufferedLogLineSender into LogLineSender implementation and drop trait - Add close method to really finish of logging and flush all buffered - Refactor UnboundJoinSet to use join_next() instead of join_spawned_tasks() to get results early. - Fix incorrect logger instance usage due to logic bug in number of worker threads check - also log skipped because in-progress snapshots through logger instance Change since version 6 (thanks @Fabian): - Implement LogSender trait and UnbufferedLogLineSender to completely bypass BufferedLogger in non-parallel case. - Store per-label last logged timestamp and use that to flush in main logger loop to assure labels are also flushed when other labels are receiving new messages. - Refactor internal logger methods and fix various docstrings and comments. - Make sure all available results in bounded join set are returned when spawning a new task, not just the first one. - Adapt accounting logic for store progress with respect to shared group progress, to improve logging output for parallel syncs. - Adapt snapshot-archive prefix separator from {snapshot}: {archive} to {snapshot}/{archive}. - Conditionally show either percentual progress for sequential syncs or snapshot/group count based progress for parallel syncs. Change since version 5 (thanks @Fabian): - Implement buffered logger for better grouping of fast succession log lines - Refactor group worker into standalone BoundedJoinSet implementation. - Improve log output by using better prefixes - Add missing error contexts Change since version 4: - Use dedicated tokio tasks to run in parallel on different runtime threads, not just multiple concurrent futures on the same thread. - Rework store progress accounting logic to avoid mutex locks when possible, use atomic counters instead. - Expose setting also in the sync job edit window, not just the config. proxmox: Christian Ebner (1): pbs api types: add `worker-threads` to sync job config pbs-api-types/src/jobs.rs | 11 +++++++++++ 1 file changed, 11 insertions(+) proxmox-backup: Christian Ebner (9): tools: implement buffered logger for concurrent log messages tools: add bounded join set to run concurrent tasks bound by limit api: config/sync: add optional `worker-threads` property fix #4182: server: sync: allow pulling backup groups in parallel server: pull: prefix log messages and add error context server: sync: allow pushing groups concurrently server: push: prefix log messages and add additional logging sync: move in-progress snapshot filter to helper and use log line sender ui: expose group worker setting in sync job edit window pbs-tools/Cargo.toml | 3 + pbs-tools/src/bounded_join_set.rs | 81 ++++++ pbs-tools/src/buffered_logger.rs | 238 +++++++++++++++++ pbs-tools/src/lib.rs | 2 + src/api2/config/sync.rs | 10 + src/api2/pull.rs | 9 +- src/api2/push.rs | 8 +- src/server/pull.rs | 413 +++++++++++++++++++++++------- src/server/push.rs | 349 +++++++++++++++++++++---- src/server/sync.rs | 79 ++++-- www/window/SyncJobEdit.js | 11 + 11 files changed, 1030 insertions(+), 173 deletions(-) create mode 100644 pbs-tools/src/bounded_join_set.rs create mode 100644 pbs-tools/src/buffered_logger.rs Summary over all repositories: 12 files changed, 1041 insertions(+), 173 deletions(-) -- Generated by murpp 0.11.0