From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 37FA91FF0C0 for ; Wed, 18 Nov 2026 14:31:50 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C3DD82170B; Wed, 18 Nov 2026 14:30:17 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Date: Wed, 18 Nov 2026 14:18:24 +0100 Message-ID: <20261118132947.1093069-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795008592016 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.172 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NA7XLPJHBRI4YOBSVHDT4A3N7Z7RING5 X-Message-ID-Hash: NA7XLPJHBRI4YOBSVHDT4A3N7Z7RING5 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is v2 of the series [0], addressing Fiona's review (thanks!) and some additional findings since v1. Patch 16 now uses node_pvecfg_version_at_least() from pve-cluster [1], which got applied in the meantime. changes v1 -> v2: - drop the container patch, it got applied - new patches 5 and 15 - fix the node version check, which failed jobs with a cluster-wide or shared storage limit on clusters with qemu-server 9.2.15 or newer - after lowering a limit, guest backups start again as soon as fewer than the new limit run, and waiting jobs pick up changed limits - less noise in the task log while waiting, and better handling of quorum loss, offline nodes and an unparsable slot table - rebase onto current master - details are in the per-patch changelogs, those of patches 6 and 11 also explain the review points not followed [0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/ Original cover letter ===================== A node runs one backup job at a time, and that job backs up its guests one after the other. This leads to odd load distributions depending on how the guests of a job are distributed in a cluster. The extreme cases: If all are on one node all get backed up in series, adding potentially a lot of delay but being rather lenient on the backup target (storage/server). If all are spread such that each guest is on a different node all will get backed up concurrently, being potentially much faster but also putting a lot of load towards the target. There is no real control over that so depending on current guest distribution a job might run slow, ok or fast, overload or underuse targets. This series adds: - parallel guest backups within a job, up to a per-node backup concurrency, set as datacenter default or per node; a job can lower it with its new 'concurrency' option, - the 'overlap' job option to run alongside other such jobs, together within the node's backup concurrency, - cluster-wide and per-storage limits that protect shared storages and backup servers from too many guest backups at the same time. The new settings are opt-in: without them, each node still backs up one guest at a time and jobs do not overlap. The node limit and the limits of local storages use file locks on each node. The cluster-wide limit and those of shared storages use a slot table on pmxcfs, changed under a cluster lock. Waiting guest backups get their slots in order, and one held up by a full storage cannot starve. The slot table is only used once all online nodes run a version that supports it; until then, the cluster-wide and shared storage limits are not enforced and the task log warns. Patches 5 to 10 fix or improve existing behavior, for example stopping all backup jobs of a node, and naming the error and the skipped guests of a job that could not start. They could be applied ahead of the rest. Repo (build-)dependencies: pve-manager -> pve-guest-common, pve-cluster, pve-storage A separate series on top adds a start window option for backup jobs. Besides the unit tests, I tested this end to end on nested clusters and on a three node cluster with Ceph and HA, also with PBS and NFS targets, guests that keep writing during their backup, node failures, quorum loss, stopped jobs and a rolling upgrade. pve-cluster: Thomas Lamprecht (2): datacenter config: add backup concurrency setting datacenter config: backup concurrency: add cluster-wide limit src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) pve-storage: Thomas Lamprecht (1): add backup concurrency option to storages src/PVE/Storage/BTRFSPlugin.pm | 1 + src/PVE/Storage/CIFSPlugin.pm | 1 + src/PVE/Storage/CephFSPlugin.pm | 1 + src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/ISCSIDirectPlugin.pm | 1 + src/PVE/Storage/ISCSIPlugin.pm | 1 + src/PVE/Storage/LVMPlugin.pm | 1 + src/PVE/Storage/LvmThinPlugin.pm | 1 + src/PVE/Storage/NFSPlugin.pm | 1 + src/PVE/Storage/PBSPlugin.pm | 1 + src/PVE/Storage/Plugin.pm | 9 +++++++++ src/PVE/Storage/RBDPlugin.pm | 1 + src/PVE/Storage/ZFSPlugin.pm | 1 + src/PVE/Storage/ZFSPoolPlugin.pm | 1 + src/test/plugin_properties_test.pm | 5 +++-- 15 files changed, 25 insertions(+), 2 deletions(-) pve-guest-common: Thomas Lamprecht (1): vzdump: add options for parallel guest backups and overlapping jobs src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) pve-manager: Thomas Lamprecht (15): vzdump: drop never-set fallback for the lock wait time vzdump: notification: capture guest logs right after each backup vzdump: stop all running backup jobs of the node vzdump: show the error if a job could not get the global lock vzdump: run job-init hook and set up target once the job may start vzdump: list guests as skipped if the job could not start vzdump: back up guests of a job in parallel vzdump: allow jobs to run alongside each other ui: options: add backup concurrency for datacenter and nodes ui: backup job: add concurrency and overlap options ui: backup now: add overlap option vzdump: limit concurrent guest backups cluster-wide vzdump: honor backup concurrency limits of storages api: backup info: add endpoints for cluster-wide backup slots ui: add cluster-wide and storage backup concurrency limits PVE/API2/Cluster/BackupInfo.pm | 106 ++- PVE/API2/VZDump.pm | 14 +- PVE/Makefile | 2 +- PVE/NodeConfig.pm | 7 + PVE/Service/pvescheduler.pm | 4 + PVE/VZDump.pm | 893 ++++++++++++++++++-- PVE/VZDump/ClusterSlots.pm | 296 +++++++ PVE/VZDump/Makefile | 15 + debian/tmpfiles | 5 +- test/Makefile | 10 +- test/vzdump_cluster_slots_test.pl | 282 +++++++ test/vzdump_concurrency_test.pl | 89 ++ test/vzdump_new_test.pl | 1 + www/manager6/dc/OptionView.js | 22 + www/manager6/node/NodeOptionsView.js | 8 + www/manager6/panel/BackupAdvancedOptions.js | 40 +- www/manager6/storage/Base.js | 17 + www/manager6/window/Backup.js | 28 +- 18 files changed, 1773 insertions(+), 66 deletions(-) pve-docs: Thomas Lamprecht (1): vzdump: document concurrent backups and their limits vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) -- 2.47.3