From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id 612061FF15D
for <inbox@lore.proxmox.com>; Thu, 25 Jul 2024 12:19:52 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id DBA5934AF;
Thu, 25 Jul 2024 12:19:50 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Thu, 25 Jul 2024 12:19:21 +0200
Message-Id: <20240725101922.231053-4-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.2
In-Reply-To: <20240725101922.231053-1-c.ebner@proxmox.com>
References: <20240725101922.231053-1-c.ebner@proxmox.com>
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.021 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] [RFC proxmox-backup 3/4] fix #4182: server: sync: allow
pulling groups concurrently
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
Currently, a sync job sequentially pulls the backup groups and the
snapshots contained within them, therefore being limited in download
speed by the http2 connection of the source reader instance in case
of remote syncs. High latency networks suffer from limited download
speed.
Improve the throughput by allowing to pull up to a configured number
of backup groups concurrently, by creating tasks connecting and
pulling from the remote source in parallel.
Link to issue in bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=4182
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
src/server/pull.rs | 50 ++++++++++++++++++++++++++++++++++++++--------
1 file changed, 42 insertions(+), 8 deletions(-)
diff --git a/src/server/pull.rs b/src/server/pull.rs
index e2d155c78..0a54217d4 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -10,6 +10,8 @@ use std::sync::{Arc, Mutex};
use std::time::{Duration, SystemTime};
use anyhow::{bail, format_err, Error};
+use futures::stream::FuturesUnordered;
+use futures::StreamExt;
use http::StatusCode;
use proxmox_human_byte::HumanByte;
use proxmox_router::HttpError;
@@ -1452,16 +1454,48 @@ pub(crate) async fn pull_ns(
new_groups.insert(group.clone());
}
- let mut progress = StoreProgress::new(list.len() as u64);
- let mut pull_stats = PullStats::default();
+ let mut store_progress = StoreProgress::new(list.len() as u64);
let target_ns = namespace.map_prefix(¶ms.source.get_ns(), ¶ms.target.ns)?;
- for (done, group) in list.into_iter().enumerate() {
- progress.done_groups = done as u64;
- progress.done_snapshots = 0;
- progress.group_snapshots = 0;
- pull_group_task(params, &group, namespace, &target_ns, progress.clone()).await?;
+ let mut pull_group_tasks = FuturesUnordered::new();
+
+ let mut list_iter = list.iter();
+ // queue up to requested number of initial group sync tasks to the task pool
+ for _ in 0..params.group_sync_tasks.unwrap_or(1) {
+ if let Some(group) = list_iter.next() {
+ let task_progress = StoreProgress::new(list.len() as u64);
+ pull_group_tasks.push(pull_group_task(
+ params,
+ group,
+ namespace,
+ &target_ns,
+ task_progress,
+ ));
+ }
+ }
+
+ let mut pull_stats = PullStats::default();
+ // poll to initiate tasks, queue another remaining tasks for each finished one
+ while let Some(result) = pull_group_tasks.next().await {
+ let (progress, stats, has_errors) = result?;
+ errors |= has_errors;
+ pull_stats.add(stats);
+ store_progress.done_groups += progress.done_groups;
+ store_progress.done_snapshots += progress.done_snapshots;
+
+ matches!(params.group_sync_tasks, Some(n) if n > 1);
+ // queue another remaining group sync to the task pool
+ if let Some(group) = list_iter.next() {
+ let task_progress = StoreProgress::new(list.len() as u64);
+ pull_group_tasks.push(pull_group_task(
+ params,
+ group,
+ namespace,
+ &target_ns,
+ task_progress,
+ ));
+ }
}
if params.remove_vanished {
@@ -1516,5 +1550,5 @@ pub(crate) async fn pull_ns(
};
}
- Ok((progress, pull_stats, errors))
+ Ok((store_progress, pull_stats, errors))
}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel