From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 181D71FF09D for ; Mon, 02 Nov 2026 00:00:56 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1C66F21841; Sun, 01 Nov 2026 23:59:18 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Date: Sun, 1 Nov 2026 23:58:27 +0100 Message-ID: <20261101225849.884587-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793573947811 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.389 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RESJ655LKJLZ4VDWICMM6XPUPT4OIQXA X-Message-ID-Hash: RESJ655LKJLZ4VDWICMM6XPUPT4OIQXA X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A node runs one backup job at a time, and that job backs up its guests one after the other. This leads to odd load distributions depending on how the guests of a job are distributed in a cluster. The extreme cases: If all are on one node all get backed up in series, adding potentially a lot of delay but being rather lenient on the backup target (storage/server). If all are spread such that each guest is on a different node all will get backed up concurrently, being potentially much faster but also putting a lot of load towards the target. There is no real control over that so depending on current guest distribution a job might run slow, ok or fast, overload or underuse targets. This series adds: - parallel guest backups within a job, up to a per-node backup concurrency, set as datacenter default or per node; a job can lower it with its new 'concurrency' option, - the 'overlap' job option to run alongside other such jobs, together within the node's backup concurrency, - cluster-wide and per-storage limits that protect shared storages and backup servers from too many guest backups at the same time. Without configuration nothing changes: each node backs up one guest at a time and jobs do not overlap. The node limit and the limits of local storages use file locks on each node. The cluster-wide limit and those of shared storages use a slot table on pmxcfs, changed under a cluster lock. Waiting guest backups get their slots in order, and one held up by a full storage cannot starve. These limits are only enforced once all online nodes run a version that supports them; until then the task log warns. Patches 5 to 10 fix or improve existing behavior and could be applied ahead of the rest, for example stopping all backup jobs of a node, and naming the error and the skipped guests of a job that could not start. Repo (build-)dependencies: pve-manager -> pve-guest-common, pve-cluster, pve-storage A separate series on top adds a start window option for backup jobs. Besides the unit tests, I tested this end to end on nested clusters and on a three node cluster with Ceph and HA, also with PBS and NFS targets, guests that keep writing during their backup, node failures, quorum loss, stopped jobs and a rolling upgrade. pve-cluster: Thomas Lamprecht (2): datacenter config: add backup concurrency setting datacenter config: backup concurrency: add cluster-wide limit src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) pve-storage: Thomas Lamprecht (1): add backup concurrency option to storages src/PVE/Storage/BTRFSPlugin.pm | 1 + src/PVE/Storage/CIFSPlugin.pm | 1 + src/PVE/Storage/CephFSPlugin.pm | 1 + src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/ISCSIDirectPlugin.pm | 1 + src/PVE/Storage/ISCSIPlugin.pm | 1 + src/PVE/Storage/LVMPlugin.pm | 1 + src/PVE/Storage/LvmThinPlugin.pm | 1 + src/PVE/Storage/NFSPlugin.pm | 1 + src/PVE/Storage/PBSPlugin.pm | 1 + src/PVE/Storage/Plugin.pm | 9 +++++++++ src/PVE/Storage/RBDPlugin.pm | 1 + src/PVE/Storage/ZFSPlugin.pm | 1 + src/PVE/Storage/ZFSPoolPlugin.pm | 1 + 14 files changed, 22 insertions(+) pve-guest-common: Thomas Lamprecht (1): vzdump: add options for parallel guest backups and overlapping jobs src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) pve-container: Thomas Lamprecht (1): vzdump: honor bandwidth limit for backups to Proxmox Backup Server src/PVE/VZDump/LXC.pm | 1 + 1 file changed, 1 insertion(+) pve-manager: Thomas Lamprecht (13): vzdump: notification: capture guest logs right after each backup vzdump: stop all backup jobs of the node vzdump: show the error if a job could not get the global lock vzdump: run job-init hook and set up target once the job may start vzdump: list guests as skipped if the job could not start vzdump: back up guests of a job in parallel vzdump: allow jobs to run alongside each other ui: options: add backup concurrency for datacenter and nodes ui: backup job: add concurrency and overlap options vzdump: limit concurrent guest backups cluster-wide vzdump: honor backup concurrency limits of storages api: backup info: add endpoints for cluster-wide backup slots ui: add cluster-wide and storage backup concurrency limits PVE/API2/Cluster/BackupInfo.pm | 106 ++- PVE/API2/VZDump.pm | 10 +- PVE/Makefile | 2 +- PVE/NodeConfig.pm | 7 + PVE/Service/pvescheduler.pm | 4 + PVE/VZDump.pm | 783 ++++++++++++++++++-- PVE/VZDump/ClusterSlots.pm | 299 ++++++++ PVE/VZDump/Makefile | 15 + test/Makefile | 10 +- test/vzdump_cluster_slots_test.pl | 271 +++++++ test/vzdump_concurrency_test.pl | 89 +++ test/vzdump_new_test.pl | 1 + www/manager6/dc/OptionView.js | 22 + www/manager6/node/NodeOptionsView.js | 8 + www/manager6/panel/BackupAdvancedOptions.js | 40 +- www/manager6/storage/Base.js | 17 + 16 files changed, 1623 insertions(+), 61 deletions(-) pve-docs: Thomas Lamprecht (1): vzdump: document concurrent backups and their limits vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) -- 2.47.3