From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
Date: Wed, 18 Nov 2026 14:18:24 +0100 [thread overview]
Message-ID: <20261118132947.1093069-1-t.lamprecht@proxmox.com> (raw)
This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.
changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
also explain the review points not followed
[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/
Original cover letter
=====================
A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.
This series adds:
- parallel guest backups within a job, up to a per-node backup
concurrency, set as datacenter default or per node; a job can lower
it with its new 'concurrency' option,
- the 'overlap' job option to run alongside other such jobs, together
within the node's backup concurrency,
- cluster-wide and per-storage limits that protect shared storages and
backup servers from too many guest backups at the same time.
The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.
The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.
Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.
Repo (build-)dependencies:
pve-manager -> pve-guest-common, pve-cluster, pve-storage
A separate series on top adds a start window option for backup jobs.
Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.
pve-cluster:
Thomas Lamprecht (2):
datacenter config: add backup concurrency setting
datacenter config: backup concurrency: add cluster-wide limit
src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
pve-storage:
Thomas Lamprecht (1):
add backup concurrency option to storages
src/PVE/Storage/BTRFSPlugin.pm | 1 +
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
src/PVE/Storage/ISCSIPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 1 +
src/PVE/Storage/LvmThinPlugin.pm | 1 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/PBSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 9 +++++++++
src/PVE/Storage/RBDPlugin.pm | 1 +
src/PVE/Storage/ZFSPlugin.pm | 1 +
src/PVE/Storage/ZFSPoolPlugin.pm | 1 +
src/test/plugin_properties_test.pm | 5 +++--
15 files changed, 25 insertions(+), 2 deletions(-)
pve-guest-common:
Thomas Lamprecht (1):
vzdump: add options for parallel guest backups and overlapping jobs
src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
pve-manager:
Thomas Lamprecht (15):
vzdump: drop never-set fallback for the lock wait time
vzdump: notification: capture guest logs right after each backup
vzdump: stop all running backup jobs of the node
vzdump: show the error if a job could not get the global lock
vzdump: run job-init hook and set up target once the job may start
vzdump: list guests as skipped if the job could not start
vzdump: back up guests of a job in parallel
vzdump: allow jobs to run alongside each other
ui: options: add backup concurrency for datacenter and nodes
ui: backup job: add concurrency and overlap options
ui: backup now: add overlap option
vzdump: limit concurrent guest backups cluster-wide
vzdump: honor backup concurrency limits of storages
api: backup info: add endpoints for cluster-wide backup slots
ui: add cluster-wide and storage backup concurrency limits
PVE/API2/Cluster/BackupInfo.pm | 106 ++-
PVE/API2/VZDump.pm | 14 +-
PVE/Makefile | 2 +-
PVE/NodeConfig.pm | 7 +
PVE/Service/pvescheduler.pm | 4 +
PVE/VZDump.pm | 893 ++++++++++++++++++--
PVE/VZDump/ClusterSlots.pm | 296 +++++++
PVE/VZDump/Makefile | 15 +
debian/tmpfiles | 5 +-
test/Makefile | 10 +-
test/vzdump_cluster_slots_test.pl | 282 +++++++
test/vzdump_concurrency_test.pl | 89 ++
test/vzdump_new_test.pl | 1 +
www/manager6/dc/OptionView.js | 22 +
www/manager6/node/NodeOptionsView.js | 8 +
www/manager6/panel/BackupAdvancedOptions.js | 40 +-
www/manager6/storage/Base.js | 17 +
www/manager6/window/Backup.js | 28 +-
18 files changed, 1773 insertions(+), 66 deletions(-)
pve-docs:
Thomas Lamprecht (1):
vzdump: document concurrent backups and their limits
vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 116 insertions(+)
--
2.47.3
next reply other threads:[~2026-11-18 13:31 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 13:18 Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118132947.1093069-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.