From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id F39341FF16E for ; Mon, 20 Jan 2025 11:51:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id EE8C017CEA; Mon, 20 Jan 2025 11:51:47 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Mon, 20 Jan 2025 11:50:58 +0100 Message-Id: <20250120105104.206718-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.028 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH v2 proxmox-backup 0/6] concurrent group pull/push support for sync jobs X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Syncing contents from/to a remote source via a sync job suffers from low throughput on high latency networks because of limitations by the HTTP/2 connection, as described in [0]. To improve, syncing multiple groups in parallel by establishing multiple reader instances has been suggested. This patch series implements the functionality by adding the sync job configuration property `parallel-groups`, allowing to define the number of concurrent groups pull/push futures to be instantiated and executed for each job. The property is currently not exposed on the UI, as intended to be set in the config directly for now. Examplary configuration: ``` sync: s-8764c440-3a6c ns owner root@pam remote local remote-ns remote-store push-target-store remove-vanished false store datastore sync-direction push parallel-groups 4 ``` Since log messages are now also written concurrently, prefix logs related to groups, snapshots and archives with their respective context prefix and add context to error messages. Further, improve logging especially for sync jobs in push direction, which only displayed limited information so far. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182 The previous version (RFC) of the patches can be found here: https://lore.proxmox.com/pbs-devel/20240725101922.231053-1-c.ebner@proxmox.com/ Christian Ebner (6): client: backup writer: fix upload stats size and rate for push sync api: config/sync: add optional `parallel-groups` property fix #4182: server: sync: allow pulling groups concurrently server: pull: prefix log messages and add error context server: sync: allow pushing groups concurrently server: push: prefix log messages and add additional logging pbs-api-types/src/jobs.rs | 14 ++ pbs-client/src/backup_stats.rs | 20 +-- pbs-client/src/backup_writer.rs | 4 +- pbs-datastore/src/store_progress.rs | 2 +- src/api2/config/sync.rs | 10 ++ src/api2/pull.rs | 9 +- src/api2/push.rs | 9 +- src/server/pull.rs | 217 +++++++++++++++++----------- src/server/push.rs | 147 ++++++++++++++----- src/server/sync.rs | 10 +- 10 files changed, 306 insertions(+), 136 deletions(-) -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel