From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
Date: Wed, 18 Nov 2026 14:18:24 +0100 [thread overview]
Message-ID: <20261118132947.1093069-1-t.lamprecht@proxmox.com> (raw)
This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.
changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
also explain the review points not followed
[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/
Original cover letter
=====================
A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.
This series adds:
- parallel guest backups within a job, up to a per-node backup
concurrency, set as datacenter default or per node; a job can lower
it with its new 'concurrency' option,
- the 'overlap' job option to run alongside other such jobs, together
within the node's backup concurrency,
- cluster-wide and per-storage limits that protect shared storages and
backup servers from too many guest backups at the same time.
The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.
The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.
Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.
Repo (build-)dependencies:
pve-manager -> pve-guest-common, pve-cluster, pve-storage
A separate series on top adds a start window option for backup jobs.
Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.
pve-cluster:
Thomas Lamprecht (2):
datacenter config: add backup concurrency setting
datacenter config: backup concurrency: add cluster-wide limit
src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
pve-storage:
Thomas Lamprecht (1):
add backup concurrency option to storages
src/PVE/Storage/BTRFSPlugin.pm | 1 +
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
src/PVE/Storage/ISCSIPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 1 +
src/PVE/Storage/LvmThinPlugin.pm | 1 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/PBSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 9 +++++++++
src/PVE/Storage/RBDPlugin.pm | 1 +
src/PVE/Storage/ZFSPlugin.pm | 1 +
src/PVE/Storage/ZFSPoolPlugin.pm | 1 +
src/test/plugin_properties_test.pm | 5 +++--
15 files changed, 25 insertions(+), 2 deletions(-)
pve-guest-common:
Thomas Lamprecht (1):
vzdump: add options for parallel guest backups and overlapping jobs
src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
pve-manager:
Thomas Lamprecht (15):
vzdump: drop never-set fallback for the lock wait time
vzdump: notification: capture guest logs right after each backup
vzdump: stop all running backup jobs of the node
vzdump: show the error if a job could not get the global lock
vzdump: run job-init hook and set up target once the job may start
vzdump: list guests as skipped if the job could not start
vzdump: back up guests of a job in parallel
vzdump: allow jobs to run alongside each other
ui: options: add backup concurrency for datacenter and nodes
ui: backup job: add concurrency and overlap options
ui: backup now: add overlap option
vzdump: limit concurrent guest backups cluster-wide
vzdump: honor backup concurrency limits of storages
api: backup info: add endpoints for cluster-wide backup slots
ui: add cluster-wide and storage backup concurrency limits
PVE/API2/Cluster/BackupInfo.pm | 106 ++-
PVE/API2/VZDump.pm | 14 +-
PVE/Makefile | 2 +-
PVE/NodeConfig.pm | 7 +
PVE/Service/pvescheduler.pm | 4 +
PVE/VZDump.pm | 893 ++++++++++++++++++--
PVE/VZDump/ClusterSlots.pm | 296 +++++++
PVE/VZDump/Makefile | 15 +
debian/tmpfiles | 5 +-
test/Makefile | 10 +-
test/vzdump_cluster_slots_test.pl | 282 +++++++
test/vzdump_concurrency_test.pl | 89 ++
test/vzdump_new_test.pl | 1 +
www/manager6/dc/OptionView.js | 22 +
www/manager6/node/NodeOptionsView.js | 8 +
www/manager6/panel/BackupAdvancedOptions.js | 40 +-
www/manager6/storage/Base.js | 17 +
www/manager6/window/Backup.js | 28 +-
18 files changed, 1773 insertions(+), 66 deletions(-)
pve-docs:
Thomas Lamprecht (1):
vzdump: document concurrent backups and their limits
vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 116 insertions(+)
--
2.47.3
next reply other threads:[~2026-11-18 13:31 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 13:18 Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118132947.1093069-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox