* [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits
@ 2026-11-01 22:58 Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 01/19] datacenter config: add backup concurrency setting Thomas Lamprecht
` (18 more replies)
0 siblings, 19 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.
This series adds:
- parallel guest backups within a job, up to a per-node backup
concurrency, set as datacenter default or per node; a job can lower
it with its new 'concurrency' option,
- the 'overlap' job option to run alongside other such jobs, together
within the node's backup concurrency,
- cluster-wide and per-storage limits that protect shared storages and
backup servers from too many guest backups at the same time.
Without configuration nothing changes: each node backs up one guest at
a time and jobs do not overlap.
The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. These limits are only enforced once all online nodes run a
version that supports them; until then the task log warns.
Patches 5 to 10 fix or improve existing behavior and could be applied
ahead of the rest, for example stopping all backup jobs of a node, and
naming the error and the skipped guests of a job that could not start.
Repo (build-)dependencies:
pve-manager -> pve-guest-common, pve-cluster, pve-storage
A separate series on top adds a start window option for backup jobs.
Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.
pve-cluster:
Thomas Lamprecht (2):
datacenter config: add backup concurrency setting
datacenter config: backup concurrency: add cluster-wide limit
src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
pve-storage:
Thomas Lamprecht (1):
add backup concurrency option to storages
src/PVE/Storage/BTRFSPlugin.pm | 1 +
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
src/PVE/Storage/ISCSIPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 1 +
src/PVE/Storage/LvmThinPlugin.pm | 1 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/PBSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 9 +++++++++
src/PVE/Storage/RBDPlugin.pm | 1 +
src/PVE/Storage/ZFSPlugin.pm | 1 +
src/PVE/Storage/ZFSPoolPlugin.pm | 1 +
14 files changed, 22 insertions(+)
pve-guest-common:
Thomas Lamprecht (1):
vzdump: add options for parallel guest backups and overlapping jobs
src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
pve-container:
Thomas Lamprecht (1):
vzdump: honor bandwidth limit for backups to Proxmox Backup Server
src/PVE/VZDump/LXC.pm | 1 +
1 file changed, 1 insertion(+)
pve-manager:
Thomas Lamprecht (13):
vzdump: notification: capture guest logs right after each backup
vzdump: stop all backup jobs of the node
vzdump: show the error if a job could not get the global lock
vzdump: run job-init hook and set up target once the job may start
vzdump: list guests as skipped if the job could not start
vzdump: back up guests of a job in parallel
vzdump: allow jobs to run alongside each other
ui: options: add backup concurrency for datacenter and nodes
ui: backup job: add concurrency and overlap options
vzdump: limit concurrent guest backups cluster-wide
vzdump: honor backup concurrency limits of storages
api: backup info: add endpoints for cluster-wide backup slots
ui: add cluster-wide and storage backup concurrency limits
PVE/API2/Cluster/BackupInfo.pm | 106 ++-
PVE/API2/VZDump.pm | 10 +-
PVE/Makefile | 2 +-
PVE/NodeConfig.pm | 7 +
PVE/Service/pvescheduler.pm | 4 +
PVE/VZDump.pm | 783 ++++++++++++++++++--
PVE/VZDump/ClusterSlots.pm | 299 ++++++++
PVE/VZDump/Makefile | 15 +
test/Makefile | 10 +-
test/vzdump_cluster_slots_test.pl | 271 +++++++
test/vzdump_concurrency_test.pl | 89 +++
test/vzdump_new_test.pl | 1 +
www/manager6/dc/OptionView.js | 22 +
www/manager6/node/NodeOptionsView.js | 8 +
www/manager6/panel/BackupAdvancedOptions.js | 40 +-
www/manager6/storage/Base.js | 17 +
16 files changed, 1623 insertions(+), 61 deletions(-)
pve-docs:
Thomas Lamprecht (1):
vzdump: document concurrent backups and their limits
vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 108 insertions(+)
--
2.47.3
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH cluster 01/19] datacenter config: add backup concurrency setting
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 02/19] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
` (17 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Allows configuring how many guests a node backs up at the same time,
as default for all nodes. The node configuration can override it, and
a backup job can only lower it.
The setting is a property string with the per-node limit as default
key, so further limits fit in without changing its type.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 004122e..baff829 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -219,6 +219,18 @@ my $next_id_format = {
},
};
+my $backup_concurrency_format = {
+ node => {
+ type => 'integer',
+ description => "Maximal number of guests backed up at the same time on each node,"
+ . " unless the node configuration overrides it.",
+ minimum => 1,
+ default => 1,
+ default_key => 1,
+ optional => 1,
+ },
+};
+
my $u2f_format = {
appid => {
type => 'string',
@@ -427,6 +439,12 @@ my $datacenter_schema = {
format => $replication_format,
description => "For cluster wide replication settings.",
},
+ 'backup-concurrency' => {
+ optional => 1,
+ type => 'string',
+ format => $backup_concurrency_format,
+ description => "Limits for backing up guests at the same time.",
+ },
console => {
optional => 1,
type => 'string',
@@ -599,6 +617,11 @@ sub parse_datacenter_config {
$res->{'next-id'} = parse_property_string($next_id_format, $next_id);
}
+ if (my $backup_concurrency = $res->{'backup-concurrency'}) {
+ $res->{'backup-concurrency'} =
+ parse_property_string($backup_concurrency_format, $backup_concurrency);
+ }
+
if (my $ha = $res->{ha}) {
$res->{ha} = parse_property_string($ha_format, $ha);
}
@@ -692,6 +715,11 @@ sub write_datacenter_config {
$cfg->{'next-id'} = PVE::JSONSchema::print_property_string($next_id, $next_id_format);
}
+ if (ref(my $backup_concurrency = $cfg->{'backup-concurrency'})) {
+ $cfg->{'backup-concurrency'} =
+ PVE::JSONSchema::print_property_string($backup_concurrency, $backup_concurrency_format);
+ }
+
if (ref(my $ha = $cfg->{ha})) {
$cfg->{ha} = PVE::JSONSchema::print_property_string($ha, $ha_format);
}
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH cluster 02/19] datacenter config: backup concurrency: add cluster-wide limit
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 01/19] datacenter config: add backup concurrency setting Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH storage 03/19] add backup concurrency option to storages Thomas Lamprecht
` (16 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Nodes run their part of a backup job on their own, so how many guests
are backed up at the same time depends on how many nodes the guests of
a job are spread over. In larger clusters this can overload shared
storages and backup servers, while lowering the per-node limit to
protect them slows down jobs whose guests are mostly on one node.
Add an optional limit for the whole cluster, enforced by the backup
code of all nodes together.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/DataCenterConfig.pm | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index baff829..f6ac5b9 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -229,6 +229,13 @@ my $backup_concurrency_format = {
default_key => 1,
optional => 1,
},
+ cluster => {
+ type => 'integer',
+ description =>
+ "Maximal number of guests backed up at the same time in the whole cluster.",
+ minimum => 1,
+ optional => 1,
+ },
};
my $u2f_format = {
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH storage 03/19] add backup concurrency option to storages
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 01/19] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 02/19] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH guest-common 04/19] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
` (15 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Allows limiting how many guest backups use a storage at the same time,
be it as backup target or because guest disks live on it. A backup
server or a Ceph cluster can thus be protected from more backup load
than it can handle, independent of how many nodes and jobs take part.
For shared storages the limit counts for the whole cluster, for local
storages on each node. The ESXi import storage is not a backup source
or target and does not get the option.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/BTRFSPlugin.pm | 1 +
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
src/PVE/Storage/ISCSIPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 1 +
src/PVE/Storage/LvmThinPlugin.pm | 1 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/PBSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 9 +++++++++
src/PVE/Storage/RBDPlugin.pm | 1 +
src/PVE/Storage/ZFSPlugin.pm | 1 +
src/PVE/Storage/ZFSPoolPlugin.pm | 1 +
14 files changed, 22 insertions(+)
diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index fa28592e..30e28402 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -66,6 +66,7 @@ sub options {
return {
path => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CIFSPlugin.pm b/src/PVE/Storage/CIFSPlugin.pm
index 54f0f4ec..2d358809 100644
--- a/src/PVE/Storage/CIFSPlugin.pm
+++ b/src/PVE/Storage/CIFSPlugin.pm
@@ -166,6 +166,7 @@ sub options {
share => { fixed => 1 },
subdir => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index fbc97113..135d0aee 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -142,6 +142,7 @@ sub options {
'content-dirs' => { optional => 1 },
monhost => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
subdir => { optional => 1 },
disable => { optional => 1 },
options => { optional => 1 },
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 80c4a031..4161fa77 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -82,6 +82,7 @@ sub options {
path => { fixed => 1 },
'content-dirs' => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index fb732850..ad13966c 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -75,6 +75,7 @@ sub options {
portal => { fixed => 1 },
target => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm
index 801b5d1b..808081fd 100644
--- a/src/PVE/Storage/ISCSIPlugin.pm
+++ b/src/PVE/Storage/ISCSIPlugin.pm
@@ -357,6 +357,7 @@ sub options {
portal => { fixed => 1 },
target => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 98dd48c6..47ed6d9c 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -795,6 +795,7 @@ sub options {
return {
vgname => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
saferemove => { optional => 1 },
diff --git a/src/PVE/Storage/LvmThinPlugin.pm b/src/PVE/Storage/LvmThinPlugin.pm
index 6b953693..0ace1b0d 100644
--- a/src/PVE/Storage/LvmThinPlugin.pm
+++ b/src/PVE/Storage/LvmThinPlugin.pm
@@ -55,6 +55,7 @@ sub options {
thinpool => { fixed => 1 },
vgname => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/NFSPlugin.pm b/src/PVE/Storage/NFSPlugin.pm
index 4cc02c9e..a5a4d815 100644
--- a/src/PVE/Storage/NFSPlugin.pm
+++ b/src/PVE/Storage/NFSPlugin.pm
@@ -92,6 +92,7 @@ sub options {
server => { fixed => 1 },
export => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
index 6adafa34..c05e0fd7 100644
--- a/src/PVE/Storage/PBSPlugin.pm
+++ b/src/PVE/Storage/PBSPlugin.pm
@@ -67,6 +67,7 @@ sub options {
namespace => { optional => 1 },
port => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
username => { optional => 1 },
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3d07f749..67ee4e9c 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -195,6 +195,15 @@ my $defaultData = {
default =>
"Unlimited for users with Datastore.Allocate privilege, 5 for other users",
},
+ 'backup-concurrency' => {
+ description =>
+ "Maximal number of guest backups using this storage at the same time,"
+ . " either as backup target or for guest disks. Counted for the whole cluster if"
+ . " the storage is shared, for each node otherwise.",
+ type => 'integer',
+ minimum => 1,
+ optional => 1,
+ },
shared => {
description =>
"Indicate that this is a single storage with the same contents on all "
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index ea875463..fa7ed375 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -462,6 +462,7 @@ sub properties {
sub options {
return {
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
monhost => { optional => 1 },
pool => { optional => 1 },
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index eba85bfa..70690372 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -222,6 +222,7 @@ sub properties {
sub options {
return {
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
portal => { fixed => 1 },
target => { fixed => 1 },
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index 1a4ebc65..68de1de8 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -92,6 +92,7 @@ sub options {
blocksize => { optional => 1 },
sparse => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH guest-common 04/19] vzdump: add options for parallel guest backups and overlapping jobs
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (2 preceding siblings ...)
2026-11-01 22:58 ` [PATCH storage 03/19] add backup concurrency option to storages Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH container 05/19] vzdump: honor bandwidth limit for backups to Proxmox Backup Server Thomas Lamprecht
` (14 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
A node only ever runs one backup job, and that job backs up its guests
one after the other. How long a job takes thus depends on how its
guests happen to be spread over the nodes, and a short job has to wait
until a long one finished, failing once the lock wait time is
exceeded.
The new 'concurrency' option allows backing up several guests of a job
at the same time, up to the backup concurrency of the node. The
bandwidth limit then applies to each of them, as its description now
says. The 'overlap' option lets a job run alongside other jobs that
allow it as well. It is off by default, as existing hook scripts may
rely on job phases never overlapping.
With overlapping jobs the same guest can be part of two running
backups, and with cluster-wide limits a guest backup can wait for a
slot, so the lock wait now also bounds these waits.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 67a4cf4..f98c9ae 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -297,7 +297,7 @@ my $confdesc = {
},
bwlimit => {
type => 'integer',
- description => "Limit I/O bandwidth (in KiB/s).",
+ description => "Limit I/O bandwidth of each guest backup (in KiB/s).",
optional => 1,
minimum => 0,
default => 0,
@@ -327,11 +327,27 @@ my $confdesc = {
},
lockwait => {
type => 'integer',
- description => "Maximal time to wait for the global lock (minutes).",
+ description =>
+ "Maximal time to wait for the global lock, for guests that another backup"
+ . " is currently processing, and for cluster-wide backup slots (minutes).",
optional => 1,
minimum => 0,
default => 3 * 60, # 3 hours
},
+ concurrency => {
+ type => 'integer',
+ description => "Maximal number of guests to back up at the same time on each node."
+ . " Defaults to, and is limited by, the backup concurrency configured for the node.",
+ optional => 1,
+ minimum => 1,
+ },
+ overlap => {
+ type => 'boolean',
+ description => "Allow running at the same time as other backups on the same node that"
+ . " also allow it. The job phases of their hook scripts can then overlap.",
+ optional => 1,
+ default => 0,
+ },
stopwait => {
type => 'integer',
description => "Maximal time to wait until a guest system is stopped (minutes).",
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH container 05/19] vzdump: honor bandwidth limit for backups to Proxmox Backup Server
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (3 preceding siblings ...)
2026-11-01 22:58 ` [PATCH guest-common 04/19] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 06/19] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
` (13 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
The bandwidth limit applies to container backups to file-based
storages and to the rsync passes of suspend mode, but it had no effect
on the upload to a Proxmox Backup Server. Pass it on as the client's
rate limit.
VM backups are not affected, as QEMU applies their limit.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/VZDump/LXC.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/PVE/VZDump/LXC.pm b/src/PVE/VZDump/LXC.pm
index 9bf95af..586857b 100644
--- a/src/PVE/VZDump/LXC.pm
+++ b/src/PVE/VZDump/LXC.pm
@@ -467,6 +467,7 @@ sub archive {
push @$param, '--backup-time', $task->{backup_time};
push @$param, '--change-detection-mode', $opts->{"pbs-change-detection-mode"}
if $opts->{"pbs-change-detection-mode"};
+ push @$param, '--rate', "$opts->{bwlimit}KiB" if $opts->{bwlimit};
if (my $entries_max = $opts->{"performance"}->{"pbs-entries-max"}) {
push $param->@*, '--entries-max', $entries_max;
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 06/19] vzdump: notification: capture guest logs right after each backup
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (4 preceding siblings ...)
2026-11-01 22:58 ` [PATCH container 05/19] vzdump: honor bandwidth limit for backups to Proxmox Backup Server Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 07/19] vzdump: stop all running backup jobs of the node Thomas Lamprecht
` (12 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
The log of a guest backup is written to a path derived from the guest
alone and was only read back when sending the notification at the end
of the job. This relies on no other backup of the guest starting in the
meantime, which no longer holds once backup jobs can run alongside each
other.
Read the log as the last step of each guest backup instead. Logs
longer than the notification size limit are cut, as they would not be
included anyway.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/VZDump.pm | 27 +++++++++++++++++++--------
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index defe9e0a5..d07ced61b 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -368,24 +368,35 @@ sub read_vzdump_defaults {
return $res;
}
-my sub read_backup_task_logs {
+use constant MAX_LOG_SIZE => 1024 * 1024;
+
+my sub read_guest_log {
+ my ($path) = @_;
+
+ open(my $fh, '<', $path) or return undef;
+ my $log = '';
+ read($fh, $log, MAX_LOG_SIZE + 1); # anything longer is not sent anyway
+ close($fh);
+
+ return $log;
+}
+
+my sub format_backup_task_logs {
my ($task_list) = @_;
my $task_logs = "";
for my $task (@$task_list) {
my $vmid = $task->{vmid};
- my $log_file = $task->{tmplog};
if (!$task->{tmplog}) {
$task_logs .= "$vmid: no log available\n\n";
next;
}
- if (open(my $TMP, '<', "$log_file")) {
- while (my $line = <$TMP>) {
+ if (defined(my $log = $task->{log})) {
+ for my $line (split(/^/m, $log)) {
next if $line =~ /^status: \d+/; # not useful in mails
$task_logs .= encode8bit("$vmid: $line");
}
- close($TMP);
} else {
$task_logs .= "$vmid: Could not open log file\n\n";
}
@@ -484,8 +495,6 @@ my sub get_hostname {
return $hostname;
}
-use constant MAX_LOG_SIZE => 1024 * 1024;
-
sub send_notification {
my ($self, $tasklist, $total_time, $err, $detail_pre, $detail_post) = @_;
@@ -514,7 +523,7 @@ sub send_notification {
my $text_log_part = "$cmdline\n\n";
$text_log_part .= $detail_pre . "\n" if defined($detail_pre);
- $text_log_part .= read_backup_task_logs($tasklist);
+ $text_log_part .= format_backup_task_logs($tasklist);
$text_log_part .= $detail_post if defined($detail_post);
if (length($text_log_part) > MAX_LOG_SIZE) {
@@ -1385,6 +1394,8 @@ sub exec_backup_task {
eval { $self->run_hook_script('log-end', $task); };
debugmsg('warn', $@) if $@; # message already contains command with phase name
+ $task->{log} = read_guest_log($task->{tmplog}) if $task->{tmplog};
+
die $err if $err && $err =~ m/^interrupted by signal$/;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 07/19] vzdump: stop all running backup jobs of the node
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (5 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 06/19] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 08/19] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
` (11 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Stopping running backups only signalled the job recorded in the pid
file, which is the one holding the global lock. Jobs waiting for the
lock went on afterwards, and once jobs can run alongside each other,
the pid file cannot represent all of them anyway.
Signal every backup task of the node instead, including those still
waiting to start, and drop the pid file. As when stopping a task, the
signal goes to the whole process group.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/VZDump.pm | 37 ++++++++++++++++---------------------
1 file changed, 16 insertions(+), 21 deletions(-)
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index d07ced61b..a9a2256a8 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -44,7 +44,6 @@ PVE::JSONSchema::register_standard_option(
my @posix_filesystems = qw(ext3 ext4 nfs nfs4 reiserfs xfs);
my $lockfile = '/var/run/vzdump.lock';
-my $pidfile = '/var/run/vzdump.pid';
my $logdir = '/var/log/vzdump';
my @plugins = qw();
@@ -811,8 +810,6 @@ sub getlock {
debugmsg('info', "got global lock", undef, 1);
}
- PVE::Tools::file_set_contents($pidfile, $upid);
-
return $SERVER_FLCK;
}
@@ -1500,8 +1497,6 @@ sub exec_backup {
die $err if $err;
die "job errors\n" if $errcount;
-
- unlink $pidfile;
}
sub option_exists {
@@ -1566,26 +1561,26 @@ sub verify_vzdump_parameters {
}
sub stop_running_backups {
- my ($self) = @_;
+ my $nodename = PVE::INotify::nodename();
- my $upid = PVE::Tools::file_read_firstline($pidfile);
- return if !$upid;
+ my $running = [];
+ for my $worker (PVE::RPCEnvironment::get()->active_workers()->@*) {
+ my $task = PVE::Tools::upid_decode($worker->{upid}, 1) or next;
+ next if $task->{type} ne 'vzdump' || $task->{node} ne $nodename;
+ next if !PVE::ProcFSTools::check_process_running($task->{pid}, $task->{pstart});
- my $task = PVE::Tools::upid_decode($upid);
+ # the whole process group, like stopping the task does
+ kill(15, -$task->{pid});
+ push $running->@*, $task;
+ }
- if (
- PVE::ProcFSTools::check_process_running($task->{pid}, $task->{pstart})
- && PVE::ProcFSTools::read_proc_starttime($task->{pid}) == $task->{pstart}
- ) {
- kill(15, $task->{pid});
- # wait max 15 seconds to shut down (else, do nothing for now)
- my $i;
- for ($i = 15; $i > 0; $i--) {
- last if !PVE::ProcFSTools::check_process_running(($task->{pid}, $task->{pstart}));
- sleep(1);
- }
- die "stopping backup process $task->{pid} failed\n" if $i == 0;
+ for (1 .. 15) {
+ $running = [grep { PVE::ProcFSTools::check_process_running($_->{pid}, $_->{pstart}) }
+ $running->@*];
+ return if !scalar($running->@*);
+ sleep(1);
}
+ die "stopping backup process " . join(', ', map { $_->{pid} } $running->@*) . " failed\n";
}
sub get_included_guests {
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 08/19] vzdump: show the error if a job could not get the global lock
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (6 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 07/19] vzdump: stop all running backup jobs of the node Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 09/19] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
` (10 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
When a job could not get the global lock, the notification subject only
said "multiple problems", as the error message ended with a newline.
The worker then exited directly, so the task showed "unexpected
status" instead of the reason.
Remove the newline and die with the error, so the notification subject
and the task status both name the actual problem.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/VZDump.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 3856ec6bf..c8b66169e 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -140,8 +140,9 @@ __PACKAGE__->register_method({
$vzdump->getlock($upid); # only one process allowed
};
if (my $err = $@) {
+ chomp($err); # otherwise reported as multiple problems
$vzdump->send_notification([], 0, $err);
- exit(-1);
+ die "$err\n";
}
if (defined($param->{ionice})) {
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 09/19] vzdump: run job-init hook and set up target once the job may start
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (7 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 08/19] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 10/19] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
` (9 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
The job-init hook and the activation of the backup storage ran before
taking the global lock. A job waiting for another one thus already
called its hook script, for example to mount or wake up its target,
while the other job was still running, and then possibly waited for
hours.
Take the lock first and do the job setup afterwards, handling setup
errors like a failure to get the lock. Hook scripts now see the job
phases of different jobs strictly in sequence.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/VZDump.pm | 4 +++-
PVE/VZDump.pm | 20 ++++++++++++--------
test/vzdump_new_test.pl | 1 +
3 files changed, 16 insertions(+), 9 deletions(-)
diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index c8b66169e..083b9ea35 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -137,7 +137,9 @@ __PACKAGE__->register_method({
my $vzdump = PVE::VZDump->new($cmdline, $param, $skiplist);
my $LOCK_FH = eval {
- $vzdump->getlock($upid); # only one process allowed
+ my $lock_fh = $vzdump->getlock($upid); # only one process allowed
+ $vzdump->init_job();
+ $lock_fh;
};
if (my $err = $@) {
chomp($err); # otherwise reported as multiple problems
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index a9a2256a8..a86be4f09 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -674,6 +674,16 @@ sub new {
delete $opts->{$_} for qw(notes-template protected);
}
+ $self->{'default-prune-backups'} = $defaults->{'prune-backups'};
+
+ return $self;
+}
+
+sub init_job {
+ my ($self) = @_;
+
+ my $opts = $self->{opts};
+
my $errors = '';
my $add_error = sub {
my ($error) = @_;
@@ -714,7 +724,7 @@ sub new {
die "internal error";
}
- $opts->{'prune-backups'} //= $defaults->{'prune-backups'};
+ $opts->{'prune-backups'} //= $self->{'default-prune-backups'};
# avoid triggering any remove code path if keep-all is set
$opts->{remove} = 0 if $opts->{'prune-backups'}->{'keep-all'};
@@ -723,13 +733,7 @@ sub new {
$add_error->("tmpdir '$opts->{tmpdir}' does not exist");
}
- if ($errors) {
- eval { $self->send_notification([], 0, $errors); };
- debugmsg('err', $@) if $@;
- die "$errors\n";
- }
-
- return $self;
+ die "$errors\n" if $errors;
}
sub get_mount_info {
diff --git a/test/vzdump_new_test.pl b/test/vzdump_new_test.pl
index db829e70f..490ae4482 100755
--- a/test/vzdump_new_test.pl
+++ b/test/vzdump_new_test.pl
@@ -539,6 +539,7 @@ foreach my $test (@tests) {
PVE::VZDump::parse_mailto_exclude_path($test->{cli_param});
my $vzdump = PVE::VZDump->new('fake cmdline', $test->{cli_param}, undef);
+ $vzdump->init_job();
my $opts = $vzdump->{opts} or die "did not get options\n";
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 10/19] vzdump: list guests as skipped if the job could not start
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (8 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 09/19] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 11/19] vzdump: back up guests of a job in parallel Thomas Lamprecht
` (8 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
If a job could not get the global lock, or its setup failed, for
example because the job-init hook failed or the backup server was not
reachable, none of its guests got backed up, but the notification did
not list any of them.
List them as skipped with the reason, so notifications, including those
sent only on failure, show what is missing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/VZDump.pm | 3 ++-
PVE/VZDump.pm | 48 +++++++++++++++++++++++++++++++++++++---------
2 files changed, 41 insertions(+), 10 deletions(-)
diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 083b9ea35..73291189f 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -143,7 +143,8 @@ __PACKAGE__->register_method({
};
if (my $err = $@) {
chomp($err); # otherwise reported as multiple problems
- $vzdump->send_notification([], 0, $err);
+ my $tasklist = $vzdump->skip_all_guests($rpcenv, $user, 'job did not start');
+ $vzdump->send_notification($tasklist, 0, $err);
die "$err\n";
}
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index a86be4f09..b12281790 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -387,7 +387,10 @@ my sub format_backup_task_logs {
for my $task (@$task_list) {
my $vmid = $task->{vmid};
- if (!$task->{tmplog}) {
+ if ($task->{state} eq 'skipped') {
+ $task_logs .= "$vmid: skipped - $task->{msg}\n\n";
+ next;
+ } elsif (!$task->{tmplog}) {
$task_logs .= "$vmid: no log available\n\n";
next;
}
@@ -1400,18 +1403,11 @@ sub exec_backup_task {
die $err if $err && $err =~ m/^interrupted by signal$/;
}
-sub exec_backup {
+sub get_tasklist {
my ($self, $rpcenv, $authuser) = @_;
my $opts = $self->{opts};
- debugmsg('info', "starting new backup job: $self->{cmdline}", undef, 1);
-
- if (scalar(@{ $self->{skiplist} })) {
- my $skip_string = join(', ', sort { $a <=> $b } @{ $self->{skiplist} });
- debugmsg('info', "skip external VMs: $skip_string");
- }
-
my $tasklist = [];
my $vzdump_plugins = {};
foreach my $plugin (@{ $self->{plugins} }) {
@@ -1438,6 +1434,40 @@ sub exec_backup {
};
}
+ return $tasklist;
+}
+
+my sub skip_task {
+ my ($task, $reason) = @_;
+
+ $task->{state} = 'skipped';
+ $task->{msg} = $reason;
+ debugmsg('err', "Backup of VM $task->{vmid} skipped - $reason");
+}
+
+sub skip_all_guests {
+ my ($self, $rpcenv, $authuser, $reason) = @_;
+
+ my $tasklist = $self->get_tasklist($rpcenv, $authuser);
+ skip_task($_, $reason) for $tasklist->@*;
+
+ return $tasklist;
+}
+
+sub exec_backup {
+ my ($self, $rpcenv, $authuser) = @_;
+
+ my $opts = $self->{opts};
+
+ debugmsg('info', "starting new backup job: $self->{cmdline}", undef, 1);
+
+ if (scalar(@{ $self->{skiplist} })) {
+ my $skip_string = join(', ', sort { $a <=> $b } @{ $self->{skiplist} });
+ debugmsg('info', "skip external VMs: $skip_string");
+ }
+
+ my $tasklist = $self->get_tasklist($rpcenv, $authuser);
+
# Use in-memory files for the outer hook logs to pass them to sendmail.
my $job_start_log = '';
my $job_end_log = '';
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 11/19] vzdump: back up guests of a job in parallel
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (9 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 10/19] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 12/19] vzdump: allow jobs to run alongside each other Thomas Lamprecht
` (7 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Each node backed up the guests of a job strictly one after the other.
How long a job took thus depended on how its guests were spread over
the nodes: guests all on one node were backed up in series, even if
that node and the backup target could handle more.
Allow backing up several guests at the same time, up to the backup
concurrency of the node. It comes from the new node configuration
option or the datacenter-wide default, and stays at one guest unless
configured. A job can lower it with its 'concurrency' option.
Each guest backup then runs in a forked process. The job process
collects the results, prefixes the guest output in the task log with
the guest ID, keeps the warning count, and sends a single notification
as before. A guest backup process that ends without a result is
reported as failed.
Stopping the task signals all its processes, so the guest backups stop
on their own while the job process waits for them. It does not signal
them again, which could interrupt their cleanup. A stop request that
arrives while a guest backup process is forked does not reach that
process, so it only starts its backup once the job process confirmed
that there was none; otherwise the guest counts as not started. A
signal that only reached the job process keeps it from starting
further guests.
A guest listed more than once is backed up once, as its backups would
otherwise run at the same time.
Backups to external providers stay sequential, as providers are not
known to cope with backups from multiple processes. With a concurrency
of one, guests are backed up in the job process as before.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/NodeConfig.pm | 7 +
PVE/VZDump.pm | 244 +++++++++++++++++++++++++++++++-
test/Makefile | 6 +-
test/vzdump_concurrency_test.pl | 89 ++++++++++++
4 files changed, 340 insertions(+), 6 deletions(-)
create mode 100755 test/vzdump_concurrency_test.pl
diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm
index cc1a2a296..fbd7ecf87 100644
--- a/PVE/NodeConfig.pm
+++ b/PVE/NodeConfig.pm
@@ -108,6 +108,13 @@ my $confdesc = {
default => 80,
optional => 1,
},
+ 'backup-concurrency' => {
+ description => "Maximal number of guests backed up at the same time on this node."
+ . " Overrides the datacenter-wide default.",
+ type => 'integer',
+ minimum => 1,
+ optional => 1,
+ },
};
my $wakeonlan_desc = {
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index b12281790..33fe8793e 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -11,6 +11,7 @@ use IO::File;
use IO::Select;
use IPC::Open3;
use POSIX qw(strftime);
+use Storable;
use Time::Local;
use PVE::Cluster qw(cfs_read_file);
@@ -19,6 +20,7 @@ use PVE::Exception qw(raise_param_exc);
use PVE::HA::Config;
use PVE::HA::Env::PVE2;
use PVE::JSONSchema qw(get_standard_option);
+use PVE::NodeConfig;
use PVE::Notify;
use PVE::RPCEnvironment;
use PVE::Storage;
@@ -288,10 +290,11 @@ sub check_vmids {
my (@vmids) = @_;
my $res = [];
+ my $seen = {};
for my $vmid (sort { $a <=> $b } @vmids) {
die "ERROR: strange VM ID '${vmid}'\n" if $vmid !~ m/^\d+$/;
$vmid = int($vmid); # remove leading zeros
- next if !$vmid;
+ next if !$vmid || $seen->{$vmid}++;
push @$res, $vmid;
}
@@ -1454,6 +1457,232 @@ sub skip_all_guests {
return $tasklist;
}
+my sub fail_task {
+ my ($task, $err) = @_;
+
+ $task->{state} = 'err';
+ $task->{msg} = $err;
+ debugmsg('err', "Backup of VM $task->{vmid} failed - $err");
+}
+
+sub node_backup_concurrency {
+ my $node_conf = PVE::NodeConfig::load_config(PVE::INotify::nodename());
+
+ return $node_conf->{'backup-concurrency'}
+ // cfs_read_file('datacenter.cfg')->{'backup-concurrency'}->{node} // 1;
+}
+
+sub guest_backup_concurrency {
+ my ($self) = @_;
+
+ # external providers are not known to cope with backups from multiple processes
+ return 1 if $self->{'backup-provider'};
+
+ my $node_limit = node_backup_concurrency();
+ my $limit = $self->{opts}->{concurrency} // $node_limit;
+
+ return $limit < $node_limit ? $limit : $node_limit;
+}
+
+my $guest_result_keys = [qw(backuptime hostname log msg size state target tmplog vmtype)];
+
+my sub fork_guest_backup {
+ my ($self, $task, $running, $interrupted) = @_;
+
+ pipe(my $output_reader, my $output_writer) or die "unable to create pipe - $!\n";
+ pipe(my $result_reader, my $result_writer) or die "unable to create pipe - $!\n";
+ pipe(my $start_reader, my $start_writer) or die "unable to create pipe - $!\n";
+
+ # Until the child set up its own handlers, a signal would only reach the inherited one of the
+ # job process and get lost.
+ my $signals = POSIX::SigSet->new(
+ POSIX::SIGINT(),
+ POSIX::SIGTERM(),
+ POSIX::SIGQUIT(),
+ POSIX::SIGHUP(),
+ POSIX::SIGPIPE(),
+ );
+ my $old_mask = POSIX::SigSet->new();
+ POSIX::sigprocmask(POSIX::SIG_BLOCK(), $signals, $old_mask)
+ or die "unable to block signals - $!\n";
+
+ my $pid = fork();
+ if (!defined($pid)) {
+ my $err = $!;
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+ die "unable to fork guest backup - $err\n";
+ }
+
+ if (!$pid) {
+ my $interrupted = 0;
+ $SIG{INT} = $SIG{TERM} = $SIG{QUIT} = $SIG{HUP} = $SIG{PIPE} = sub {
+ # a second signal must not abort the cleanup the first one started
+ die "interrupted by signal\n" if !$interrupted++;
+ };
+
+ my $exitcode = 1;
+ eval {
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+ close($output_reader);
+ close($result_reader);
+ close($start_writer);
+ # only the job process reads the output of the other guest backups
+ for my $sibling (values $running->%*) {
+ close($_) for grep { defined } $sibling->@{qw(output result)};
+ }
+ open(STDOUT, '>&', $output_writer) or die "unable to redirect STDOUT - $!\n";
+ open(STDERR, '>&', $output_writer) or die "unable to redirect STDERR - $!\n";
+ close($output_writer);
+
+ sysread($start_reader, my $start, 1) or die "interrupted by signal\n";
+ close($start_reader);
+
+ PVE::INotify::inotify_close();
+
+ eval { $self->exec_backup_task($task) };
+ my $err = $@;
+
+ my $result = { map { $_ => $task->{$_} } $guest_result_keys->@* };
+ print $result_writer Storable::nfreeze($result);
+ close($result_writer);
+
+ die $err if $err;
+ $exitcode = 0;
+ };
+ warn $@ if $@ && $@ !~ m/^interrupted by signal$/;
+ STDOUT->flush();
+ POSIX::_exit($exitcode);
+ }
+
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+
+ close($output_writer);
+ close($result_writer);
+ close($start_reader);
+
+ # A stop request that came in while forking reached only the job process, so the backup
+ # process must not start then.
+ my $cancelled = $$interrupted;
+ if (!$cancelled) {
+ local $SIG{PIPE} = 'IGNORE';
+ syswrite($start_writer, '1');
+ }
+ close($start_writer);
+
+ return {
+ pid => $pid,
+ cancelled => $cancelled,
+ task => $task,
+ output => $output_reader,
+ result => $result_reader,
+ 'output-buffer' => '',
+ 'result-data' => '',
+ };
+}
+
+my sub forward_guest_output {
+ my ($child, $data, $final) = @_;
+
+ my $vmid = $child->{task}->{vmid};
+
+ $child->{'output-buffer'} .= $data;
+ $child->{'output-buffer'} .= "\n" if $final && length($child->{'output-buffer'});
+
+ while ($child->{'output-buffer'} =~ s/^(.*)\n//) {
+ my $line = $1;
+ # warnings of a guest backup process count for the task, but its counter ends with it
+ PVE::RPCEnvironment::get()->{warning_count}++ if $line =~ m/^WARN: /;
+ print STDERR "$vmid: $line\n";
+ }
+}
+
+my sub read_guest_backup_pipe {
+ my ($child, $kind) = @_;
+
+ my $count = sysread($child->{$kind}, my $data, 64 * 1024);
+ return if !defined($count) && ($!{EINTR} || $!{EAGAIN});
+
+ if (!$count) {
+ close(delete $child->{$kind});
+ forward_guest_output($child, '', 1) if $kind eq 'output';
+ } elsif ($kind eq 'output') {
+ forward_guest_output($child, $data);
+ } else {
+ $child->{'result-data'} .= $data;
+ }
+}
+
+my sub finish_guest_backup {
+ my ($child, $status) = @_;
+
+ # the backup process is gone, but helpers it started might still hold the output pipe
+ for my $kind (qw(result output)) {
+ while ($child->{$kind} && IO::Select->new($child->{$kind})->can_read(0)) {
+ read_guest_backup_pipe($child, $kind);
+ }
+ close(delete $child->{$kind}) if $child->{$kind};
+ }
+ forward_guest_output($child, '', 1);
+ return if $child->{cancelled};
+
+ my $task = $child->{task};
+ my $result = eval { Storable::thaw($child->{'result-data'}) };
+ if (ref($result) eq 'HASH') {
+ $task->{$_} = $result->{$_} for keys $result->%*;
+ } else {
+ my $reason = $status & 127 ? "signal " . ($status & 127) : "exit code " . ($status >> 8);
+ fail_task($task, "backup process ended unexpectedly ($reason)");
+ }
+}
+
+my sub exec_parallel_backup_tasks {
+ my ($self, $tasklist, $concurrency) = @_;
+
+ my @queue = $tasklist->@*;
+ my $running = {};
+ my $interrupted = 0;
+ my $job_pid = $$;
+
+ # Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
+ # them again could interrupt their cleanup.
+ local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
+ sub { $interrupted = 1; };
+
+ while (scalar(@queue) || scalar(keys $running->%*)) {
+ while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
+ my $task = shift @queue;
+ my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
+ # a guest backup process signalled while it exits dies into this eval
+ POSIX::_exit(1) if $$ != $job_pid;
+ if (my $err = $@) {
+ fail_task($task, $err);
+ next;
+ }
+ $running->{ $child->{pid} } = $child;
+ unshift @queue, $task if $child->{cancelled};
+ }
+
+ last if $interrupted && !scalar(keys $running->%*);
+
+ my $handles = {};
+ for my $child (values $running->%*) {
+ $handles->{ fileno($child->{$_}) } = [$child, $_]
+ for grep { $child->{$_} } qw(output result);
+ }
+ my $select = IO::Select->new(map { $_->[0]->{ $_->[1] } } values $handles->%*);
+ for my $fh ($select->can_read(1)) {
+ read_guest_backup_pipe($handles->{ fileno($fh) }->@*);
+ }
+
+ for my $pid (keys $running->%*) {
+ next if waitpid($pid, POSIX::WNOHANG()) != $pid;
+ finish_guest_backup(delete $running->{$pid}, $?);
+ }
+ }
+
+ die "interrupted by signal\n" if $interrupted;
+}
+
sub exec_backup {
my ($self, $rpcenv, $authuser) = @_;
@@ -1474,22 +1703,27 @@ sub exec_backup {
open my $job_start_fd, '>', \$job_start_log;
open my $job_end_fd, '>', \$job_end_log;
+ my $concurrency = $self->guest_backup_concurrency();
+ $concurrency = scalar($tasklist->@*) if scalar($tasklist->@*) < $concurrency;
+
my $starttime = time();
- my $errcount = 0;
eval {
$self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
$self->run_hook_script('job-start', undef, $job_start_fd);
- foreach my $task (@$tasklist) {
- $self->exec_backup_task($task);
- $errcount += 1 if $task->{state} ne 'ok';
+ if ($concurrency > 1) {
+ debugmsg('info', "running up to $concurrency guest backups at the same time");
+ exec_parallel_backup_tasks($self, $tasklist, $concurrency);
+ } else {
+ $self->exec_backup_task($_) for $tasklist->@*;
}
$self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
$self->run_hook_script('job-end', undef, $job_end_fd);
};
my $err = $@;
+ my $errcount = grep { $_->{state} ne 'ok' } $tasklist->@*;
if ($err) {
if ($self->{'backup-provider'}) {
diff --git a/test/Makefile b/test/Makefile
index 2dfd20a41..1eb94f4e0 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
./$<
.PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
.PHONY: test-vzdump-guest-included
test-vzdump-guest-included:
@@ -28,6 +28,10 @@ test-vzdump-guest-included:
test-vzdump-new:
./vzdump_new_test.pl
+.PHONY: test-vzdump-concurrency
+test-vzdump-concurrency:
+ ./vzdump_concurrency_test.pl
+
.PHONY: test-osd
test-osd:
./OSD_test.pl
diff --git a/test/vzdump_concurrency_test.pl b/test/vzdump_concurrency_test.pl
new file mode 100755
index 000000000..46d679490
--- /dev/null
+++ b/test/vzdump_concurrency_test.pl
@@ -0,0 +1,89 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib '..';
+
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump;
+
+my $node_config = {};
+my $datacenter_config = {};
+
+my $node_config_module = Test::MockModule->new('PVE::NodeConfig');
+$node_config_module->mock(load_config => sub { return $node_config; });
+
+my $vzdump_module = Test::MockModule->new('PVE::VZDump');
+$vzdump_module->mock(
+ cfs_read_file => sub {
+ my ($filename) = @_;
+ die "unexpected file '$filename'\n" if $filename ne 'datacenter.cfg';
+ return $datacenter_config;
+ },
+);
+
+my $tests = [
+ {
+ description => 'defaults keep backing up one guest at a time',
+ expected => 1,
+ },
+ {
+ description => 'datacenter default applies to the node',
+ datacenter => 4,
+ expected => 4,
+ },
+ {
+ description => 'node configuration overrides datacenter default',
+ datacenter => 4,
+ node => 2,
+ expected => 2,
+ },
+ {
+ description => 'node configuration applies without datacenter default',
+ node => 3,
+ expected => 3,
+ },
+ {
+ description => 'job limit lowers node limit',
+ node => 4,
+ job => 2,
+ expected => 2,
+ },
+ {
+ description => 'job limit cannot raise node limit',
+ node => 2,
+ job => 8,
+ expected => 2,
+ },
+ {
+ description => 'job limit does not raise the default',
+ job => 8,
+ expected => 1,
+ },
+ {
+ description => 'external backup providers back up one guest at a time',
+ node => 4,
+ job => 4,
+ provider => 1,
+ expected => 1,
+ },
+];
+
+for my $test ($tests->@*) {
+ $datacenter_config =
+ defined($test->{datacenter})
+ ? { 'backup-concurrency' => { node => $test->{datacenter} } }
+ : {};
+ $node_config = defined($test->{node}) ? { 'backup-concurrency' => $test->{node} } : {};
+
+ my $self = bless { opts => {} }, 'PVE::VZDump';
+ $self->{opts}->{concurrency} = $test->{job} if defined($test->{job});
+ $self->{'backup-provider'} = {} if $test->{provider};
+
+ is($self->guest_backup_concurrency(), $test->{expected}, $test->{description});
+}
+
+done_testing();
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 12/19] vzdump: allow jobs to run alongside each other
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (10 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 11/19] vzdump: back up guests of a job in parallel Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 13/19] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
` (6 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
A node ran one backup job at a time. A short job, for example an
hourly one to a nearby target, had to wait until a long nightly job
finished, possibly for hours, and failed once the lock wait time ran
out.
Jobs with the new 'overlap' option share the global lock and run at
the same time as other such jobs. Every guest backup on the node, no
matter which job it belongs to, holds one of the node's backup slots
while running, so overlapping jobs together stay within the node's
backup concurrency. When the concurrency gets lowered, slots above
the new limit keep counting until their guest backups end. Jobs
without the option keep running alone, and a job waiting for that
takes precedence over overlapping jobs starting later, so these cannot
starve it. Jobs to external backup providers always run alone.
A guest that another job is backing up is deferred until that backup
finished, waiting at most the lock wait time. Before starting a guest
backup, the job also checks whether the guest moved to another node in
the meantime.
All of these are file locks held by the processes doing the work, so
they get released whenever such a process ends. Jobs that do not
overlap still take the global lock exclusively, which keeps it working
for the boot-time cleanup of stale backup locks and against older
versions during an upgrade.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/VZDump.pm | 2 +-
PVE/VZDump.pm | 193 +++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 188 insertions(+), 7 deletions(-)
diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 73291189f..3e31505f3 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -137,7 +137,7 @@ __PACKAGE__->register_method({
my $vzdump = PVE::VZDump->new($cmdline, $param, $skiplist);
my $LOCK_FH = eval {
- my $lock_fh = $vzdump->getlock($upid); # only one process allowed
+ my $lock_fh = $vzdump->getlock($upid);
$vzdump->init_job();
$lock_fh;
};
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 33fe8793e..f597f7ed7 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -46,6 +46,7 @@ PVE::JSONSchema::register_standard_option(
my @posix_filesystems = qw(ext3 ext4 nfs nfs4 reiserfs xfs);
my $lockfile = '/var/run/vzdump.lock';
+my $lockdir = '/run/vzdump';
my $logdir = '/var/log/vzdump';
my @plugins = qw();
@@ -682,6 +683,14 @@ sub new {
$self->{'default-prune-backups'} = $defaults->{'prune-backups'};
+ # external providers are not known to cope with other backups running at the same time
+ my $uses_provider = $opts->{storage} && eval {
+ PVE::Storage::storage_has_feature(
+ PVE::Storage::config(), $opts->{storage}, 'backup-provider',
+ );
+ };
+ $self->{exclusive} = !$opts->{overlap} || $uses_provider;
+
return $self;
}
@@ -781,14 +790,32 @@ sub getlock {
my $maxwait = $self->{opts}->{lockwait} || $self->{lockwait};
die "missing UPID" if !$upid; # should not happen
+ $self->{upid} = $upid;
+
+ mkdir $lockdir;
my $SERVER_FLCK;
if (!open($SERVER_FLCK, '>>', "$lockfile")) {
debugmsg('err', "can't open lock on file '$lockfile' - $!", undef, 1);
die "can't open lock on file '$lockfile' - $!";
}
+ open(my $gate, '>>', "$lockdir/gate.lock")
+ or die "can't open lock on file '$lockdir/gate.lock' - $!\n";
- if (!flock($SERVER_FLCK, LOCK_EX | LOCK_NB)) {
+ # Jobs that may overlap share the global lock. A job waiting to get it exclusively holds the
+ # gate meanwhile, so overlapping jobs starting later cannot keep it waiting forever.
+ my $mode = $self->{exclusive} ? LOCK_EX : LOCK_SH;
+ my $acquire = sub {
+ my ($flags) = @_;
+
+ flock($gate, $mode | $flags) or return 0;
+ my $locked = flock($SERVER_FLCK, $mode | $flags);
+ flock($gate, LOCK_UN);
+
+ return $locked;
+ };
+
+ if (!$acquire->(LOCK_NB)) {
if (!$maxwait) {
debugmsg('err', "can't acquire lock '$lockfile' (wait = 0)", undef, 1);
die "can't acquire lock '$lockfile' (wait = 0)";
@@ -800,7 +827,7 @@ sub getlock {
local $SIG{ALRM} = sub { alarm(0); die "got timeout\n"; };
- if (!flock($SERVER_FLCK, LOCK_EX)) {
+ if (!$acquire->(0)) {
my $err = $!;
close($SERVER_FLCK);
alarm(0);
@@ -1635,12 +1662,145 @@ my sub finish_guest_backup {
}
}
+my sub try_lock_file {
+ my ($path) = @_;
+
+ open(my $fh, '>>', $path) or die "unable to open lock file '$path' - $!\n";
+ return $fh if flock($fh, LOCK_EX | LOCK_NB);
+
+ close($fh);
+ return undef;
+}
+
+my sub try_guest_lock {
+ my ($self, $vmid) = @_;
+
+ my $fh = try_lock_file("$lockdir/guest-$vmid.lock") or return undef;
+ truncate($fh, 0);
+ print $fh "$self->{upid}\n";
+ $fh->flush();
+
+ return $fh;
+}
+
+# Slots above a lowered limit stay in use until their backups end, no new one starts before.
+my sub try_slot {
+ my ($prefix, $limit) = @_;
+
+ for my $path (glob("$prefix-*.lock")) {
+ my ($slot) = $path =~ m/^\Q$prefix\E-(\d+)\.lock$/ or next;
+ next if $slot <= $limit;
+ my $fh = try_lock_file("$prefix-$slot.lock") or return undef;
+ close($fh);
+ }
+ for my $slot (1 .. $limit) {
+ my $fh = try_lock_file("$prefix-$slot.lock");
+ return $fh if $fh;
+ }
+ return undef;
+}
+
+my sub try_node_slot {
+ return try_slot("$lockdir/slot", node_backup_concurrency());
+}
+
+my sub log_wait {
+ my ($self, $reason) = @_;
+
+ return if ($self->{'waiting-for'} // '') eq $reason;
+ $self->{'waiting-for'} = $reason;
+ debugmsg('info', "waiting for $reason");
+}
+
+my sub guest_moved_to {
+ my ($vmid) = @_;
+
+ PVE::Cluster::cfs_update();
+ my $node = PVE::Cluster::get_vmlist()->{ids}->{$vmid}->{node};
+
+ return defined($node) && $node ne PVE::INotify::nodename() ? $node : undef;
+}
+
+my sub claim_next_guest {
+ my ($self, $queue, $deferred) = @_;
+
+ my $maxwait = ($self->{opts}->{lockwait} || 0) * 60;
+
+ for (my $i = 0; $i < scalar($queue->@*); $i++) {
+ my $task = $queue->[$i];
+ my $vmid = $task->{vmid};
+
+ my $guest_lock = try_guest_lock($self, $vmid);
+ if (!$guest_lock) {
+ my $holder = PVE::Tools::file_read_firstline("$lockdir/guest-$vmid.lock") // '';
+ if (!defined($deferred->{$vmid})) {
+ $deferred->{$vmid} = time();
+ debugmsg('info', "VM $vmid is being backed up by '$holder', deferring");
+ } elsif (time() - $deferred->{$vmid} >= $maxwait) {
+ splice($queue->@*, $i--, 1);
+ skip_task($task, "still being backed up by '$holder'");
+ }
+ next;
+ }
+
+ my $slot = try_node_slot();
+ if (!$slot) {
+ log_wait($self, "a free backup slot on the node");
+ return;
+ }
+ if (my $node = guest_moved_to($vmid)) {
+ splice($queue->@*, $i--, 1);
+ skip_task($task, "guest moved to node '$node'");
+ next;
+ }
+ delete $self->{'waiting-for'};
+
+ splice($queue->@*, $i--, 1);
+ return ($task, [$guest_lock, $slot]);
+ }
+
+ return;
+}
+
+my sub skip_after_error {
+ my ($queue, $err) = @_;
+
+ return if $err =~ m/interrupted by signal$/;
+
+ chomp(my $reason = $err);
+ skip_task($_, "not started due to error - $reason") for splice($queue->@*);
+}
+
+my sub exec_serial_backup_tasks {
+ my ($self, $tasklist) = @_;
+
+ my @queue = $tasklist->@*;
+ my $deferred = {};
+
+ eval {
+ while (scalar(@queue)) {
+ my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+ if ($task) {
+ $self->exec_backup_task($task);
+ } elsif (scalar(@queue)) {
+ sleep(1);
+ }
+ }
+ };
+ if (my $err = $@) {
+ skip_after_error(\@queue, $err);
+ die $err;
+ }
+}
+
my sub exec_parallel_backup_tasks {
my ($self, $tasklist, $concurrency) = @_;
my @queue = $tasklist->@*;
+ my $deferred = {};
my $running = {};
my $interrupted = 0;
+ my $error;
my $job_pid = $$;
# Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
@@ -1649,8 +1809,17 @@ my sub exec_parallel_backup_tasks {
sub { $interrupted = 1; };
while (scalar(@queue) || scalar(keys $running->%*)) {
- while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
- my $task = shift @queue;
+ while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
+ my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+ if (my $err = $@) {
+ # let running guest backups finish, they would get aborted otherwise
+ debugmsg('err', "unable to start further guest backups - $err");
+ $error = $err;
+ last;
+ }
+ last if !$task;
+
+ # the backup process keeps holding the locks through its inherited copies of $locks
my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
# a guest backup process signalled while it exits dies into this eval
POSIX::_exit(1) if $$ != $job_pid;
@@ -1662,7 +1831,14 @@ my sub exec_parallel_backup_tasks {
unshift @queue, $task if $child->{cancelled};
}
- last if $interrupted && !scalar(keys $running->%*);
+ if ($interrupted || defined($error)) {
+ last if !scalar(keys $running->%*);
+ }
+
+ if (!scalar(keys $running->%*)) {
+ sleep(1);
+ next;
+ }
my $handles = {};
for my $child (values $running->%*) {
@@ -1681,6 +1857,11 @@ my sub exec_parallel_backup_tasks {
}
die "interrupted by signal\n" if $interrupted;
+
+ if (defined($error)) {
+ skip_after_error(\@queue, $error);
+ die $error;
+ }
}
sub exec_backup {
@@ -1716,7 +1897,7 @@ sub exec_backup {
debugmsg('info', "running up to $concurrency guest backups at the same time");
exec_parallel_backup_tasks($self, $tasklist, $concurrency);
} else {
- $self->exec_backup_task($_) for $tasklist->@*;
+ exec_serial_backup_tasks($self, $tasklist);
}
$self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 13/19] ui: options: add backup concurrency for datacenter and nodes
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (11 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 12/19] vzdump: allow jobs to run alongside each other Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 14/19] ui: backup job: add concurrency and overlap options Thomas Lamprecht
` (5 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Expose the per-node backup concurrency in the datacenter options and
its override in the node options, so admins can let nodes back up
several guests at the same time without editing configuration files.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/OptionView.js | 14 ++++++++++++++
www/manager6/node/NodeOptionsView.js | 8 ++++++++
2 files changed, 22 insertions(+)
diff --git a/www/manager6/dc/OptionView.js b/www/manager6/dc/OptionView.js
index dc12aa7e1..ac0f0bdd1 100644
--- a/www/manager6/dc/OptionView.js
+++ b/www/manager6/dc/OptionView.js
@@ -360,6 +360,20 @@ Ext.define('PVE.dc.OptionView', {
minValue: 1,
maxValue: 64, // arbitrary but generous limit as limits are good
});
+ me.add_inputpanel_row('backup-concurrency', gettext('Backup Concurrency'), {
+ renderer: PVE.Utils.render_as_property_string,
+ url: '/api2/extjs/cluster/options',
+ items: [
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 'node',
+ fieldLabel: gettext('Per node'),
+ emptyText: '1',
+ minValue: 1,
+ submitValue: true,
+ },
+ ],
+ });
me.add_inputpanel_row('next-id', gettext('Next Free VMID Range'), {
renderer: PVE.Utils.render_as_property_string,
url: '/api2/extjs/cluster/options',
diff --git a/www/manager6/node/NodeOptionsView.js b/www/manager6/node/NodeOptionsView.js
index 5e9ce7915..2dcda66bc 100644
--- a/www/manager6/node/NodeOptionsView.js
+++ b/www/manager6/node/NodeOptionsView.js
@@ -95,5 +95,13 @@ Ext.define('Proxmox.node.NodeOptionsView', {
onlineHelp: 'qm_memory',
renderer: (value) => (value !== undefined ? `${value}%` : gettext('Default (80%)')),
},
+ {
+ xtype: 'integer',
+ name: 'backup-concurrency',
+ text: gettext('Backup concurrency'),
+ minValue: 1,
+ deleteEmpty: true,
+ renderer: (value) => value ?? gettext('Datacenter default'),
+ },
],
});
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 14/19] ui: backup job: add concurrency and overlap options
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (12 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 13/19] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 15/19] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
` (4 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Add the new job options to the advanced tab. An empty concurrency
field uses the fallback from the node's vzdump.conf like the other
options there. An unchecked overlap box is stored as off, like the
fleecing one, as deleting it would let an overlap setting in
vzdump.conf apply while the box shows otherwise.
The bandwidth limit applies to each guest backup on its own, so with
several guests backed up at the same time their total can be higher.
Say so in its description.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/panel/BackupAdvancedOptions.js | 40 ++++++++++++++++++++-
1 file changed, 39 insertions(+), 1 deletion(-)
diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf7..8af0fa77e 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -154,7 +154,45 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
endFlex: 2,
endColumn: {
xtype: 'displayfield',
- value: `${gettext('Limit I/O bandwidth.')} ${Ext.String.format(gettext('Schema default: {0}'), 0)}`,
+ value: `${gettext('Limit I/O bandwidth of each guest backup.')} ${Ext.String.format(gettext('Schema default: {0}'), 0)}`,
+ },
+ },
+ {
+ xtype: 'pveTwoColumnContainer',
+ startColumn: {
+ xtype: 'proxmoxintegerfield',
+ name: 'concurrency',
+ minValue: 1,
+ fieldLabel: gettext('Concurrency'),
+ fieldStyle: 'text-align: right',
+ emptyText: gettext('Fallback'),
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ },
+ endFlex: 2,
+ endColumn: {
+ xtype: 'displayfield',
+ value: gettext(
+ 'Guests backed up at the same time on each node, at most the backup concurrency of the node.',
+ ),
+ },
+ },
+ {
+ xtype: 'pveTwoColumnContainer',
+ startColumn: {
+ xtype: 'proxmoxcheckbox',
+ fieldLabel: gettext('Overlap'),
+ name: 'overlap',
+ uncheckedValue: 0,
+ value: 0,
+ },
+ endFlex: 2,
+ endColumn: {
+ xtype: 'displayfield',
+ value: gettext(
+ 'Run at the same time as other jobs that allow it. Job phases of hook scripts can then overlap.',
+ ),
},
},
{
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 15/19] vzdump: limit concurrent guest backups cluster-wide
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (13 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 14/19] ui: backup job: add concurrency and overlap options Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 16/19] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
` (3 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Each node runs its part of a backup job on its own, so shared storages
and backup servers get as many guest backups at the same time as all
nodes together allow.
With the new cluster-wide limit in the datacenter configuration, every
guest backup also needs a slot in a table on the cluster file system
before it starts. The table is only changed under a cluster lock, and
waiting guest backups get their slots in the order they queued up. A
job has at most one guest waiting for a slot, so waiting does not tie
up the whole node, and waits at most the lock wait time in total.
A guest backup gives back its slot when it is done. If that fails, for
example because the backup process got killed, the scheduler of the
node removes the entry once the guest is no longer locked for backup.
Slots of offline nodes stay taken, as their guest backups might still
run and keep the shared systems busy. Entries of nodes removed from the
cluster are dropped. Granting slots needs quorum; guest backups that
already run continue without it. A table that cannot be read stops
granting instead of getting replaced, which would forget guest backups
that still run.
The limit is only enforced once all other online nodes run a version
that takes part, as told by the version info each node broadcasts.
Older nodes would not count their guest backups, and a node that gets
downgraded broadcasts its older version again.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
Notes:
NOTE: $min_version assumes that this lands with pve-manager 9.2.26; adapt it to the
first release that contains it.
PVE/Makefile | 2 +-
PVE/Service/pvescheduler.pm | 4 +
PVE/VZDump.pm | 157 +++++++++++++++++--
PVE/VZDump/ClusterSlots.pm | 245 ++++++++++++++++++++++++++++++
PVE/VZDump/Makefile | 15 ++
test/Makefile | 6 +-
test/vzdump_cluster_slots_test.pl | 194 +++++++++++++++++++++++
7 files changed, 612 insertions(+), 11 deletions(-)
create mode 100644 PVE/VZDump/ClusterSlots.pm
create mode 100644 PVE/VZDump/Makefile
create mode 100755 test/vzdump_cluster_slots_test.pl
diff --git a/PVE/Makefile b/PVE/Makefile
index efcb250d0..ffdd3a8fd 100644
--- a/PVE/Makefile
+++ b/PVE/Makefile
@@ -1,6 +1,6 @@
include ../defines.mk
-SUBDIRS=API2 Status CLI Service Ceph Jobs
+SUBDIRS=API2 Status CLI Service Ceph Jobs VZDump
PERLSOURCE = \
API2.pm \
diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm
index f7271fc00..e54f94183 100755
--- a/PVE/Service/pvescheduler.pm
+++ b/PVE/Service/pvescheduler.pm
@@ -7,6 +7,7 @@ use POSIX qw(WNOHANG);
use PVE::Jobs;
use PVE::SafeSyslog;
+use PVE::VZDump;
use PVE::API2::Replication;
@@ -114,6 +115,9 @@ sub run {
$fork->(
'jobs',
sub {
+ eval { PVE::VZDump::cleanup_cluster_slots() };
+ syslog('err', "backup slots: $@") if $@;
+
PVE::Jobs::run_jobs($first_run);
},
);
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index f597f7ed7..525a593c6 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -25,6 +25,7 @@ use PVE::Notify;
use PVE::RPCEnvironment;
use PVE::Storage;
use PVE::VZDump::Common;
+use PVE::VZDump::ClusterSlots;
use PVE::VZDump::Plugin;
use PVE::Tools qw(extract_param split_list);
use PVE::API2Tools;
@@ -1683,6 +1684,18 @@ my sub try_guest_lock {
return $fh;
}
+my sub guest_backup_running {
+ my ($vmid, $upid) = @_;
+
+ open(my $fh, '<', "$lockdir/guest-$vmid.lock") or return 0;
+ return 0 if flock($fh, LOCK_SH | LOCK_NB);
+
+ my $holder = <$fh> // '';
+ chomp($holder);
+
+ return $holder eq $upid;
+}
+
# Slots above a lowered limit stay in use until their backups end, no new one starts before.
my sub try_slot {
my ($prefix, $limit) = @_;
@@ -1771,23 +1784,138 @@ my sub skip_after_error {
skip_task($_, "not started due to error - $reason") for splice($queue->@*);
}
+my sub cluster_slot_resources {
+ my ($self) = @_;
+
+ my $limits = PVE::VZDump::ClusterSlots::limits();
+ return undef if !scalar(keys $limits->%*);
+
+ if (!PVE::VZDump::ClusterSlots::supported_by_all_nodes()) {
+ debugmsg(
+ 'warn',
+ "cluster-wide backup limits are not enforced - not all online nodes support them",
+ ) if !$self->{'cluster-slots-unsupported'}++;
+ return undef;
+ }
+
+ return ['cluster'];
+}
+
+my sub release_cluster_slots {
+ my ($key) = @_;
+
+ return if !defined($key);
+
+ eval { PVE::VZDump::ClusterSlots::release($key) };
+ if (my $err = $@) {
+ die $err if $err =~ m/interrupted by signal$/; # also if wrapped by the cluster lock
+ debugmsg('warn', "unable to release cluster-wide backup slot - $err");
+ }
+}
+
+sub cleanup_cluster_slots {
+ PVE::VZDump::ClusterSlots::cleanup_local_entries(\&guest_backup_running);
+}
+
+# A claimed guest that needs cluster-wide slots waits for them as pending guest, only one at a time.
+my sub next_guest_to_start {
+ my ($self, $queue, $schedule) = @_;
+
+ my $pending = $schedule->{pending};
+ if (!$pending) {
+ my ($task, $locks) = claim_next_guest($self, $queue, $schedule->{deferred}) or return;
+ my $resources = eval { cluster_slot_resources($self) };
+ if (my $err = $@) {
+ unshift $queue->@*, $task;
+ die $err;
+ }
+ return ($task, $locks) if !$resources;
+
+ $pending = $schedule->{pending} = {
+ task => $task,
+ locks => $locks,
+ resources => $resources,
+ since => time(),
+ refreshed => time(),
+ };
+ }
+
+ my $task = $pending->{task};
+ my $granted = eval {
+ $pending->{key} //=
+ PVE::VZDump::ClusterSlots::request($self->{upid}, $task->{vmid}, $pending->{resources});
+ if (time() - $pending->{refreshed} >= 30) {
+ PVE::VZDump::ClusterSlots::refresh();
+ $pending->{refreshed} = time();
+ }
+ PVE::VZDump::ClusterSlots::granted($pending->{key});
+ };
+ if (my $err = $@) {
+ die $err if $err =~ m/interrupted by signal$/;
+ chomp($err);
+ debugmsg('warn', "unable to request cluster-wide backup slot - $err")
+ if !defined($pending->{'request-error'});
+ $pending->{'request-error'} = $err;
+ } elsif (!defined($granted)) {
+ $pending->{key} = undef; # vanished, for example if released by an admin
+ } elsif ($granted) {
+ delete $schedule->{pending};
+ $schedule->{'cluster-wait'} += time() - $pending->{since};
+ if (my $node = guest_moved_to($task->{vmid})) {
+ release_cluster_slots($pending->{key});
+ skip_task($task, "guest moved to node '$node'");
+ return;
+ }
+ return ($task, $pending->{locks}, $pending->{key});
+ } else {
+ delete $pending->{'request-error'};
+ }
+
+ my $waited = $schedule->{'cluster-wait'} + time() - $pending->{since};
+ my $maxwait = ($self->{opts}->{lockwait} || 0) * 60;
+ if ($waited >= $maxwait) {
+ release_cluster_slots($pending->{key});
+ delete $schedule->{pending};
+ $schedule->{'cluster-wait'} = $waited;
+ my $reason = 'timeout waiting for cluster-wide backup slot';
+ $reason .= " - $pending->{'request-error'}" if defined($pending->{'request-error'});
+ skip_task($task, $reason);
+ } elsif (!$pending->{logged}++) {
+ debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot");
+ }
+
+ return;
+}
+
+my sub requeue_pending_guest {
+ my ($queue, $schedule) = @_;
+
+ my $pending = delete $schedule->{pending} or return;
+ release_cluster_slots($pending->{key});
+ unshift $queue->@*, $pending->{task};
+}
+
my sub exec_serial_backup_tasks {
my ($self, $tasklist) = @_;
my @queue = $tasklist->@*;
- my $deferred = {};
+ my $schedule = { deferred => {}, 'cluster-wait' => 0 };
eval {
- while (scalar(@queue)) {
- my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+ while (scalar(@queue) || $schedule->{pending}) {
+ my ($task, $locks, $slot_key) = next_guest_to_start($self, \@queue, $schedule);
if ($task) {
- $self->exec_backup_task($task);
- } elsif (scalar(@queue)) {
+ eval { $self->exec_backup_task($task) };
+ my $err = $@;
+ release_cluster_slots($slot_key);
+ die $err if $err;
+ } else {
sleep(1);
}
}
};
if (my $err = $@) {
+ requeue_pending_guest(\@queue, $schedule);
skip_after_error(\@queue, $err);
die $err;
}
@@ -1797,7 +1925,7 @@ my sub exec_parallel_backup_tasks {
my ($self, $tasklist, $concurrency) = @_;
my @queue = $tasklist->@*;
- my $deferred = {};
+ my $schedule = { deferred => {}, 'cluster-wait' => 0 };
my $running = {};
my $interrupted = 0;
my $error;
@@ -1808,9 +1936,9 @@ my sub exec_parallel_backup_tasks {
local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
sub { $interrupted = 1; };
- while (scalar(@queue) || scalar(keys $running->%*)) {
+ while (scalar(@queue) || $schedule->{pending} || scalar(keys $running->%*)) {
while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
- my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+ my ($task, $locks, $slot_key) = eval { next_guest_to_start($self, \@queue, $schedule) };
if (my $err = $@) {
# let running guest backups finish, they would get aborted otherwise
debugmsg('err', "unable to start further guest backups - $err");
@@ -1819,19 +1947,28 @@ my sub exec_parallel_backup_tasks {
}
last if !$task;
+ if ($interrupted) { # getting the slots can take a while
+ release_cluster_slots($slot_key);
+ unshift @queue, $task;
+ last;
+ }
+
# the backup process keeps holding the locks through its inherited copies of $locks
my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
# a guest backup process signalled while it exits dies into this eval
POSIX::_exit(1) if $$ != $job_pid;
if (my $err = $@) {
fail_task($task, $err);
+ release_cluster_slots($slot_key);
next;
}
+ $child->{'slot-key'} = $slot_key;
$running->{ $child->{pid} } = $child;
unshift @queue, $task if $child->{cancelled};
}
if ($interrupted || defined($error)) {
+ requeue_pending_guest(\@queue, $schedule);
last if !scalar(keys $running->%*);
}
@@ -1852,7 +1989,9 @@ my sub exec_parallel_backup_tasks {
for my $pid (keys $running->%*) {
next if waitpid($pid, POSIX::WNOHANG()) != $pid;
- finish_guest_backup(delete $running->{$pid}, $?);
+ my $child = delete $running->{$pid};
+ finish_guest_backup($child, $?);
+ release_cluster_slots($child->{'slot-key'});
}
}
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
new file mode 100644
index 000000000..6537bdce5
--- /dev/null
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -0,0 +1,245 @@
+package PVE::VZDump::ClusterSlots;
+
+# Cluster-wide limits for guest backups running at the same time.
+#
+# Each guest backup that counts against such a limit has an entry in a shared table on pmxcfs. It
+# starts out as waiter and becomes holder once all limits it counts against have room. Entries of
+# offline nodes stay, as their guest backups might still be running.
+
+use strict;
+use warnings;
+
+use JSON;
+
+use PVE::Cluster qw(cfs_read_file);
+use PVE::INotify;
+use PVE::QemuServer::Helpers;
+use PVE::Tools;
+
+my $state_file = '/etc/pve/priv/backup-slots.json';
+
+# first release of pve-manager that takes part in cluster-wide limits
+my $min_version = [9, 2, 26];
+
+sub limits {
+ my $limits = {};
+
+ my $dc_conf = cfs_read_file('datacenter.cfg');
+ if (defined(my $cluster = $dc_conf->{'backup-concurrency'}->{cluster})) {
+ $limits->{cluster} = $cluster;
+ }
+
+ return $limits;
+}
+
+my sub get_members {
+ my $members = PVE::Cluster::get_members();
+ return $members if $members && scalar(keys $members->%*);
+
+ # entries of other nodes would get dropped as if these left the cluster
+ die "unable to get cluster members\n" if -e '/etc/pve/corosync.conf';
+
+ return { PVE::INotify::nodename() => { online => 1 } };
+}
+
+# Offline nodes cannot start guest backups, and their entries stay anyway.
+sub supported_by_all_nodes {
+ my $members = get_members();
+ my $nodename = PVE::INotify::nodename();
+
+ for my $node (keys $members->%*) {
+ next if $node eq $nodename || !$members->{$node}->{online};
+ my $version = PVE::QemuServer::Helpers::get_node_pvecfg_version($node);
+ return 0 if !PVE::QemuServer::Helpers::pvecfg_min_version($version, $min_version->@*);
+ }
+ return 1;
+}
+
+# Checks a copy, so that numbers do not get written back as strings.
+my sub is_count {
+ my ($value) = @_;
+ return defined($value) && !ref($value) && $value =~ m/^\d+$/;
+}
+
+my sub valid_entry {
+ my ($entry) = @_;
+
+ return
+ ref($entry) eq 'HASH'
+ && (grep { !ref($entry->{$_}) && length($entry->{$_} // '') } qw(node upid)) == 2
+ && is_count($entry->{vmid})
+ && is_count($entry->{seq})
+ && ref($entry->{resources}) eq 'ARRAY'
+ && !grep { ref($_) || !length($_ // '') } $entry->{resources}->@*;
+}
+
+sub parse_state {
+ my ($raw) = @_;
+
+ my $state = eval { decode_json($raw) };
+ die "invalid JSON - $@" if $@;
+ die "not a backup slot table\n"
+ if ref($state) ne 'HASH'
+ || (grep { ref($state->{$_}) ne 'HASH' } qw(holders waiters))
+ || !is_count($state->{seq})
+ || grep { !valid_entry($_) } map { values $state->{$_}->%* } qw(holders waiters);
+
+ return $state;
+}
+
+# Fails on anything but a missing file, as writing back a table that lacks holders would let more
+# guest backups start than the limits allow.
+sub read_state {
+ open(my $fh, '<', $state_file) or do {
+ die "unable to open '$state_file' - $!\n" if !$!{ENOENT};
+ return { holders => {}, waiters => {}, seq => 0 };
+ };
+ my $raw = PVE::Tools::safe_read_from($fh, 0, 0, $state_file);
+ my $state = eval { parse_state($raw) };
+ die "unable to parse '$state_file' - $@" if $@;
+
+ return $state;
+}
+
+my sub used_slots {
+ my ($state) = @_;
+
+ my $used = {};
+ for my $holder (values $state->{holders}->%*) {
+ $used->{$_}++ for $holder->{resources}->@*;
+ }
+ return $used;
+}
+
+my sub remove_entries {
+ my ($state, $match) = @_;
+
+ my $count = 0;
+ for my $entries ($state->@{qw(holders waiters)}) {
+ for my $key (keys $entries->%*) {
+ next if !$match->($entries->{$key});
+ delete $entries->{$key};
+ $count++;
+ }
+ }
+ return $count;
+}
+
+# Moves waiters that fit into all limits they count against to the holders, oldest first. Only
+# resources with a limit matter, the others are tracked for information. Waiters of offline nodes
+# neither get granted nor hold up others.
+sub grant_waiters {
+ my ($state, $limits, $members) = @_;
+
+ remove_entries($state, sub { !$members->{ $_[0]->{node} } });
+
+ my $used = used_slots($state);
+
+ my $waiters = $state->{waiters};
+ for my $key (sort { $waiters->{$a}->{seq} <=> $waiters->{$b}->{seq} } keys $waiters->%*) {
+ my $waiter = $waiters->{$key};
+ next if !$members->{ $waiter->{node} }->{online};
+
+ my $limited = [grep { defined($limits->{$_}) } $waiter->{resources}->@*];
+ next if grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+
+ $used->{$_}++ for $waiter->{resources}->@*;
+ $state->{holders}->{$key} = delete $waiters->{$key};
+ $state->{holders}->{$key}->{granted} = time();
+ }
+}
+
+my sub update_state {
+ my ($code) = @_;
+
+ my $res = PVE::Cluster::cfs_lock_domain(
+ 'backup-slots',
+ 10,
+ sub {
+ my $json = JSON->new->canonical();
+
+ my $state = read_state();
+ my $previous = $json->encode($state);
+
+ my $res = $code->($state);
+ grant_waiters($state, limits(), get_members());
+
+ my $raw = $json->encode($state);
+ PVE::Tools::file_set_contents($state_file, $raw) if $raw ne $previous;
+
+ return $res;
+ },
+ );
+ die $@ if $@;
+
+ return $res;
+}
+
+sub request {
+ my ($upid, $vmid, $resources) = @_;
+
+ my $node = PVE::INotify::nodename();
+ my $key = "$node/$vmid/$upid";
+
+ update_state(sub {
+ my ($state) = @_;
+ return if $state->{holders}->{$key} || $state->{waiters}->{$key};
+
+ $state->{waiters}->{$key} = {
+ node => $node,
+ upid => $upid,
+ vmid => $vmid,
+ resources => $resources,
+ seq => ++$state->{seq},
+ queued => time(),
+ };
+ });
+
+ return $key;
+}
+
+# Returns 1 if granted, 0 if still waiting and undef if the request vanished.
+sub granted {
+ my ($key) = @_;
+
+ my $state = read_state();
+ return 1 if $state->{holders}->{$key};
+ return 0 if $state->{waiters}->{$key};
+ return undef;
+}
+
+# Grants waiters that fit by now, for example after limits got raised.
+sub refresh {
+ update_state(sub { });
+}
+
+sub release {
+ my ($key) = @_;
+
+ update_state(sub {
+ my ($state) = @_;
+ delete $state->{$_}->{$key} for qw(holders waiters);
+ });
+}
+
+# Removes the entries of this node whose guest backup is gone, as told by $running->($vmid, $upid).
+sub cleanup_local_entries {
+ my ($running) = @_;
+
+ my $node = PVE::INotify::nodename();
+
+ my $stale = sub {
+ my ($entry) = @_;
+ return $entry->{node} eq $node && !$running->($entry->{vmid}, $entry->{upid});
+ };
+
+ # avoid the cluster lock if there is nothing to remove
+ return if !remove_entries(read_state(), $stale);
+
+ update_state(sub {
+ my ($state) = @_;
+ remove_entries($state, $stale);
+ });
+}
+
+1;
diff --git a/PVE/VZDump/Makefile b/PVE/VZDump/Makefile
new file mode 100644
index 000000000..6500eb4d4
--- /dev/null
+++ b/PVE/VZDump/Makefile
@@ -0,0 +1,15 @@
+include ../../defines.mk
+
+PERLSOURCE = \
+ ClusterSlots.pm \
+
+all:
+
+.PHONY: clean
+clean:
+ rm -rf *~
+
+.PHONY: install
+install: $(PERLSOURCE)
+ install -d $(PERLLIBDIR)/PVE/VZDump
+ install -m 0644 $(PERLSOURCE) $(PERLLIBDIR)/PVE/VZDump
diff --git a/test/Makefile b/test/Makefile
index 1eb94f4e0..d51095978 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
./$<
.PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency test-vzdump-cluster-slots
.PHONY: test-vzdump-guest-included
test-vzdump-guest-included:
@@ -32,6 +32,10 @@ test-vzdump-new:
test-vzdump-concurrency:
./vzdump_concurrency_test.pl
+.PHONY: test-vzdump-cluster-slots
+test-vzdump-cluster-slots:
+ ./vzdump_cluster_slots_test.pl
+
.PHONY: test-osd
test-osd:
./OSD_test.pl
diff --git a/test/vzdump_cluster_slots_test.pl b/test/vzdump_cluster_slots_test.pl
new file mode 100755
index 000000000..85608b5be
--- /dev/null
+++ b/test/vzdump_cluster_slots_test.pl
@@ -0,0 +1,194 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib '..';
+
+use JSON;
+use Storable qw(dclone);
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump::ClusterSlots;
+
+my $members = {
+ node1 => { online => 1 },
+ node2 => { online => 1 },
+ node3 => { online => 0 },
+};
+
+my sub make_state {
+ my ($holders, $waiters) = @_;
+
+ my $state = { holders => {}, waiters => {}, seq => 0 };
+ for my $holder ($holders->@*) {
+ my ($key, $node, $resources) = $holder->@*;
+ $state->{holders}->{$key} = { node => $node, resources => $resources };
+ }
+ for my $waiter ($waiters->@*) {
+ my ($key, $node, $resources) = $waiter->@*;
+ $state->{waiters}->{$key} = {
+ node => $node,
+ resources => $resources,
+ seq => ++$state->{seq},
+ };
+ }
+ return $state;
+}
+
+my sub granted {
+ my ($state) = @_;
+ return [sort keys $state->{holders}->%*];
+}
+
+my $tests = [
+ {
+ description => 'waiters are granted in order up to the limit',
+ limits => { cluster => 2 },
+ waiters =>
+ [['a', 'node1', ['cluster']], ['b', 'node2', ['cluster']], ['c', 'node1', ['cluster']]],
+ expected => ['a', 'b'],
+ },
+ {
+ description => 'holders count against the limit',
+ limits => { cluster => 2 },
+ holders => [['h', 'node1', ['cluster']]],
+ waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+ expected => ['a', 'h'],
+ },
+ {
+ description => 'resources without limit do not hold up',
+ limits => {},
+ holders => [['h', 'node1', ['cluster']]],
+ waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+ expected => ['a', 'b', 'h'],
+ },
+ {
+ description => 'waiters of offline nodes neither get granted nor hold up others',
+ limits => { cluster => 1 },
+ waiters => [['a', 'node3', ['cluster']], ['b', 'node1', ['cluster']]],
+ expected => ['b'],
+ },
+ {
+ description => 'holders of offline nodes keep counting',
+ limits => { cluster => 1 },
+ holders => [['h', 'node3', ['cluster']]],
+ waiters => [['a', 'node1', ['cluster']]],
+ expected => ['h'],
+ },
+ {
+ description => 'entries of nodes no longer in the cluster are dropped',
+ limits => { cluster => 1 },
+ holders => [['h', 'gone', ['cluster']]],
+ waiters => [['a', 'gone', ['cluster']], ['b', 'node1', ['cluster']]],
+ expected => ['b'],
+ 'expected-waiters' => [],
+ },
+];
+
+for my $test ($tests->@*) {
+ my $state = make_state($test->{holders} // [], $test->{waiters} // []);
+
+ PVE::VZDump::ClusterSlots::grant_waiters($state, $test->{limits}, $members);
+
+ is_deeply(granted($state), $test->{expected}, $test->{description});
+
+ if (my $expected_waiters = $test->{'expected-waiters'}) {
+ is_deeply(
+ [sort keys $state->{waiters}->%*],
+ $expected_waiters,
+ "$test->{description} (waiters)",
+ );
+ }
+}
+
+# Request, grant, release and cleanup against an in-memory table.
+my $table = { holders => {}, waiters => {}, seq => 0 };
+my $writes = 0;
+
+my $slots_module = Test::MockModule->new('PVE::VZDump::ClusterSlots');
+$slots_module->mock(
+ read_state => sub { return dclone($table); },
+ limits => sub { return { cluster => 1 }; },
+);
+my $cluster_module = Test::MockModule->new('PVE::Cluster');
+$cluster_module->mock(
+ cfs_lock_domain => sub {
+ my ($domain, $timeout, $code) = @_;
+ return eval { $code->() };
+ },
+ get_members => sub { return $members; },
+);
+my $tools_module = Test::MockModule->new('PVE::Tools');
+$tools_module->mock(
+ file_set_contents => sub {
+ my ($filename, $raw) = @_;
+ $table = decode_json($raw);
+ $writes++;
+ },
+);
+my $inotify_module = Test::MockModule->new('PVE::INotify');
+$inotify_module->mock(nodename => sub { return 'node1'; });
+
+$table->{holders}->{other} = { node => 'node2', vmid => 200, upid => 'u0', resources => [] };
+
+my $first = PVE::VZDump::ClusterSlots::request('u1', 100, ['cluster']);
+my $second = PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(PVE::VZDump::ClusterSlots::granted($first), 1, 'first request gets the slot');
+is(PVE::VZDump::ClusterSlots::granted($second), 0, 'second request waits');
+
+PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(scalar(keys $table->{waiters}->%*), 1, 'repeated request does not queue again');
+
+my $count = $writes;
+PVE::VZDump::ClusterSlots::refresh();
+is($writes, $count, 'unchanged table is not written');
+
+PVE::VZDump::ClusterSlots::release($first);
+is(PVE::VZDump::ClusterSlots::granted($second), 1, 'released slot goes to the waiting request');
+is(PVE::VZDump::ClusterSlots::granted($first), undef, 'released request is gone');
+
+PVE::VZDump::ClusterSlots::cleanup_local_entries(sub { return 0; });
+is_deeply([keys $table->{holders}->%*], ['other'], 'cleanup only drops stale entries of the node');
+
+my $entry = '{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":["cluster"]}';
+my $parsed = PVE::VZDump::ClusterSlots::parse_state('{"holders":{},"waiters":{},"seq":3}');
+is(JSON->new->canonical->encode($parsed), '{"holders":{},"seq":3,"waiters":{}}', 'valid table');
+ok(
+ PVE::VZDump::ClusterSlots::parse_state(qq({"holders":{"a":$entry},"waiters":{},"seq":3})),
+ 'valid table with an entry',
+);
+for my $raw (
+ 'garbage',
+ '[]',
+ '{"holders":null,"waiters":{},"seq":3}',
+ '{"waiters":{},"seq":3}',
+ '{"holders":{},"waiters":[],"seq":3}',
+ '{"holders":{},"waiters":{}}',
+ '{"holders":{},"waiters":{},"seq":"x"}',
+ '{"holders":{"a":{}},"waiters":{},"seq":3}',
+ '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":"cluster"}},"waiters":{},"seq":3}',
+ '{"holders":{},"waiters":{"a":{"upid":"UPID:x","vmid":100,"seq":1,"resources":[]}},"seq":3}',
+ '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[null]}},"waiters":{},"seq":3}',
+ '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[{}]}},"waiters":{},"seq":3}',
+) {
+ eval { PVE::VZDump::ClusterSlots::parse_state($raw) };
+ ok($@, "rejects $raw");
+}
+
+my $versions = { node2 => '9.2.26', node3 => '9.2.25' };
+$cluster_module->mock(
+ get_node_kv => sub {
+ my ($key, $node) = @_;
+ return {} if $key ne 'version-info' || !defined($versions->{$node});
+ return { $node => encode_json({ version => $versions->{$node} }) };
+ },
+);
+ok(PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older offline node is ignored');
+$versions->{node2} = '9.2.25';
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older online node blocks');
+delete $versions->{node2};
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'online node without version blocks');
+
+done_testing();
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 16/19] vzdump: honor backup concurrency limits of storages
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (14 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 15/19] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 17/19] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
` (2 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Guest backups now also count against the backup concurrency limit of
the storages they use, both the backup target and the storages of the
included guest disks. This protects a backup server or a Ceph cluster
from more parallel backups than it can handle, which a limit for the
whole cluster cannot express once several such systems are in use.
Limits of shared storages count for the whole cluster and use the
cluster-wide slots. Limits of local storages count on each node and
use slot locks like the node limit. A guest whose local storage has no
free slot lets other guests of the job go first.
With several limits, a waiting guest backup can be blocked by a full
storage while younger ones with room in their limits get slots. This
may happen a few times, then a slot of each limit it needs is kept
free for it, so it cannot starve.
The storages are taken from the guest configuration before the guest
gets locked for backup, so waiting for slots does not block changes to
the guest. If its disks moved to another storage with a limit by the
time it got locked, the backup fails instead of running without a slot
for that storage. If the storages cannot be determined, the guest is
skipped for the same reason. Nothing of this happens as long as no
storage has a limit. The task log tells which limit a guest waits for.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/VZDump.pm | 95 +++++++++++++++++++++++++++++--
PVE/VZDump/ClusterSlots.pm | 44 +++++++++++++-
test/vzdump_cluster_slots_test.pl | 79 ++++++++++++++++++++++++-
3 files changed, 209 insertions(+), 9 deletions(-)
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 525a593c6..59809d85c 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -933,6 +933,34 @@ sub get_backup_file_list {
return $bklist;
}
+my sub limited_backup_storages {
+ my ($self, $task) = @_;
+
+ my $storage_cfg = PVE::Storage::config();
+ my $limited = {
+ map { $_ => 1 }
+ grep { defined($storage_cfg->{ids}->{$_}->{'backup-concurrency'}) }
+ keys $storage_cfg->{ids}->%*
+ };
+ return [] if !scalar(keys $limited->%*);
+
+ my $storeids = {};
+ $storeids->{ $self->{opts}->{storage} } = 1 if $self->{opts}->{storage};
+
+ if (my $plugin = $task->{plugin}) {
+ my $config_class = $plugin->type() eq 'qemu' ? 'PVE::QemuConfig' : 'PVE::LXC::Config';
+ my $conf = $config_class->load_config($task->{vmid});
+ for my $volume ($config_class->get_backup_volumes($conf)->@*) {
+ next if !$volume->{included};
+ my $volid = $volume->{volume_config}->{file} // $volume->{volume_config}->{volume};
+ my ($storeid) = PVE::Storage::parse_volume_id($volid, 1);
+ $storeids->{$storeid} = 1 if defined($storeid);
+ }
+ }
+
+ return [sort grep { $limited->{$_} } keys $storeids->%*];
+}
+
sub exec_backup_task {
my ($self, $task) = @_;
@@ -1116,6 +1144,13 @@ sub exec_backup_task {
$cleanup->{unlock} = 1;
+ my $taken = { map { $_ => 1 } $task->{storages}->@* };
+ if (my @missing = grep { !$taken->{$_} } limited_backup_storages($self, $task)->@*) {
+ my $list = join(', ', map { "'$_'" } @missing);
+ die "no backup slot for storage $list - guest disks or storage limits changed while"
+ . " waiting for backup slots\n";
+ }
+
# prepare
my $mode = $running ? $task->{mode} : 'stop';
@@ -1717,6 +1752,25 @@ my sub try_node_slot {
return try_slot("$lockdir/slot", node_backup_concurrency());
}
+# Limits of local storages count on each node, shared ones are handled cluster-wide.
+my sub try_local_storage_slots {
+ my ($storeids) = @_;
+
+ my $storage_cfg = PVE::Storage::config();
+
+ my $locks = [];
+ for my $storeid ($storeids->@*) {
+ my $scfg = $storage_cfg->{ids}->{$storeid} or next;
+ my $limit = $scfg->{'backup-concurrency'};
+ next if $scfg->{shared} || !defined($limit);
+
+ my $lock = try_slot("$lockdir/storage-$storeid", $limit) or return (undef, $storeid);
+ push $locks->@*, $lock;
+ }
+
+ return ($locks);
+}
+
my sub log_wait {
my ($self, $reason) = @_;
@@ -1766,10 +1820,25 @@ my sub claim_next_guest {
skip_task($task, "guest moved to node '$node'");
next;
}
+
+ $task->{storages} = eval { limited_backup_storages($self, $task) };
+ if (my $err = $@) {
+ die $err if $err =~ m/interrupted by signal$/;
+ chomp($err);
+ splice($queue->@*, $i--, 1);
+ skip_task($task, "unable to check storage backup limits - $err");
+ next;
+ }
+
+ my ($storage_locks, $busy_storage) = try_local_storage_slots($task->{storages});
+ if (!$storage_locks) {
+ log_wait($self, "a free backup slot of storage '$busy_storage'");
+ next;
+ }
delete $self->{'waiting-for'};
splice($queue->@*, $i--, 1);
- return ($task, [$guest_lock, $slot]);
+ return ($task, [$guest_lock, $slot, $storage_locks->@*]);
}
return;
@@ -1785,7 +1854,7 @@ my sub skip_after_error {
}
my sub cluster_slot_resources {
- my ($self) = @_;
+ my ($self, $task) = @_;
my $limits = PVE::VZDump::ClusterSlots::limits();
return undef if !scalar(keys $limits->%*);
@@ -1798,7 +1867,15 @@ my sub cluster_slot_resources {
return undef;
}
- return ['cluster'];
+ my $storage_cfg = PVE::Storage::config();
+ my $shared = [grep { $storage_cfg->{ids}->{$_}->{shared} } $task->{storages}->@*];
+
+ return ['cluster', map { "storage/$_" } $shared->@*];
+}
+
+my sub render_cluster_limits {
+ my ($resources) = @_;
+ return join(', ', map { m!^storage/(.*)$! ? "storage '$1'" : "the $_" } $resources->@*);
}
my sub release_cluster_slots {
@@ -1824,7 +1901,7 @@ my sub next_guest_to_start {
my $pending = $schedule->{pending};
if (!$pending) {
my ($task, $locks) = claim_next_guest($self, $queue, $schedule->{deferred}) or return;
- my $resources = eval { cluster_slot_resources($self) };
+ my $resources = eval { cluster_slot_resources($self, $task) };
if (my $err = $@) {
unshift $queue->@*, $task;
die $err;
@@ -1880,8 +1957,14 @@ my sub next_guest_to_start {
my $reason = 'timeout waiting for cluster-wide backup slot';
$reason .= " - $pending->{'request-error'}" if defined($pending->{'request-error'});
skip_task($task, $reason);
- } elsif (!$pending->{logged}++) {
- debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot");
+ } elsif (defined($pending->{key}) && !$pending->{logged}++) {
+ my $reached = eval { PVE::VZDump::ClusterSlots::reached_limits($pending->{key}) };
+ die $@ if $@ =~ m/interrupted by signal$/;
+ my $reason =
+ scalar(($reached // [])->@*)
+ ? "limit reached for " . render_cluster_limits($reached)
+ : "queued behind other guest backups";
+ debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot, $reason");
}
return;
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
index 6537bdce5..f4cac71c7 100644
--- a/PVE/VZDump/ClusterSlots.pm
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -14,6 +14,7 @@ use JSON;
use PVE::Cluster qw(cfs_read_file);
use PVE::INotify;
use PVE::QemuServer::Helpers;
+use PVE::Storage;
use PVE::Tools;
my $state_file = '/etc/pve/priv/backup-slots.json';
@@ -21,6 +22,9 @@ my $state_file = '/etc/pve/priv/backup-slots.json';
# first release of pve-manager that takes part in cluster-wide limits
my $min_version = [9, 2, 26];
+# How often younger requests may get ahead of a waiting one before its limits are reserved for it.
+use constant MAX_BYPASS => 3;
+
sub limits {
my $limits = {};
@@ -29,6 +33,13 @@ sub limits {
$limits->{cluster} = $cluster;
}
+ my $storage_cfg = PVE::Storage::config();
+ for my $storeid (keys $storage_cfg->{ids}->%*) {
+ my $scfg = $storage_cfg->{ids}->{$storeid};
+ next if !$scfg->{shared} || !defined($scfg->{'backup-concurrency'});
+ $limits->{"storage/$storeid"} = $scfg->{'backup-concurrency'};
+ }
+
return $limits;
}
@@ -127,7 +138,8 @@ my sub remove_entries {
# Moves waiters that fit into all limits they count against to the holders, oldest first. Only
# resources with a limit matter, the others are tracked for information. Waiters of offline nodes
-# neither get granted nor hold up others.
+# neither get granted nor hold up others. A waiter that younger ones got ahead of too often
+# reserves a slot of each limit it counts against, so it cannot starve.
sub grant_waiters {
my ($state, $limits, $members) = @_;
@@ -135,13 +147,29 @@ sub grant_waiters {
my $used = used_slots($state);
+ my $passed_over = [];
+
my $waiters = $state->{waiters};
for my $key (sort { $waiters->{$a}->{seq} <=> $waiters->{$b}->{seq} } keys $waiters->%*) {
my $waiter = $waiters->{$key};
next if !$members->{ $waiter->{node} }->{online};
my $limited = [grep { defined($limits->{$_}) } $waiter->{resources}->@*];
- next if grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+ my $fits = !grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+
+ if (!$fits) {
+ if (($waiter->{bypassed} // 0) >= MAX_BYPASS) {
+ $used->{$_}++ for $limited->@*;
+ }
+ push $passed_over->@*, [$waiter, $limited];
+ next;
+ }
+
+ my %mine = map { $_ => 1 } $limited->@*;
+ for my $older ($passed_over->@*) {
+ my ($older_waiter, $older_limited) = $older->@*;
+ $older_waiter->{bypassed}++ if grep { $mine{$_} } $older_limited->@*;
+ }
$used->{$_}++ for $waiter->{resources}->@*;
$state->{holders}->{$key} = delete $waiters->{$key};
@@ -208,6 +236,18 @@ sub granted {
return undef;
}
+sub reached_limits {
+ my ($key) = @_;
+
+ my $state = read_state();
+ my $waiter = $state->{waiters}->{$key} or return [];
+
+ my $used = used_slots($state);
+ my $limits = limits();
+ return [grep { defined($limits->{$_}) && ($used->{$_} // 0) >= $limits->{$_} }
+ $waiter->{resources}->@*];
+}
+
# Grants waiters that fit by now, for example after limits got raised.
sub refresh {
update_state(sub { });
diff --git a/test/vzdump_cluster_slots_test.pl b/test/vzdump_cluster_slots_test.pl
index 85608b5be..5e13512b2 100755
--- a/test/vzdump_cluster_slots_test.pl
+++ b/test/vzdump_cluster_slots_test.pl
@@ -27,11 +27,12 @@ my sub make_state {
$state->{holders}->{$key} = { node => $node, resources => $resources };
}
for my $waiter ($waiters->@*) {
- my ($key, $node, $resources) = $waiter->@*;
+ my ($key, $node, $resources, $bypassed) = $waiter->@*;
$state->{waiters}->{$key} = {
node => $node,
resources => $resources,
seq => ++$state->{seq},
+ bypassed => $bypassed // 0,
};
}
return $state;
@@ -85,6 +86,54 @@ my $tests = [
expected => ['b'],
'expected-waiters' => [],
},
+ {
+ description => 'younger waiter with free resources gets ahead of a blocked one',
+ limits => { x => 1, y => 1 },
+ holders => [['h', 'node1', ['x']]],
+ waiters => [['a', 'node1', ['x', 'y']], ['b', 'node2', ['y']]],
+ expected => ['b', 'h'],
+ bypassed => { a => 1 },
+ },
+ {
+ description => 'waiter passed over too often reserves its resources',
+ limits => { x => 1, y => 1 },
+ holders => [['h', 'node1', ['x']]],
+ waiters => [['a', 'node1', ['x', 'y'], 3], ['b', 'node2', ['y']]],
+ expected => ['h'],
+ },
+ {
+ description => 'waiter passed over too often reserves only one slot of each resource',
+ limits => { cluster => 3, x => 1 },
+ holders => [['h', 'node1', ['cluster', 'x']]],
+ waiters => [
+ ['a', 'node1', ['cluster', 'x'], 3],
+ ['b', 'node2', ['cluster']],
+ ['c', 'node2', ['cluster']],
+ ],
+ expected => ['b', 'h'],
+ },
+ {
+ description => 'reserved resources do not hold up unrelated waiters',
+ limits => { x => 1, y => 1, z => 1 },
+ holders => [['h', 'node1', ['x']]],
+ waiters => [['a', 'node1', ['x', 'y'], 3], ['b', 'node2', ['z']]],
+ expected => ['b', 'h'],
+ },
+ {
+ description => 'resources without limit do not count as passed over',
+ limits => { x => 1 },
+ holders => [['h', 'node1', ['cluster', 'x']]],
+ waiters => [['a', 'node1', ['cluster', 'x']], ['b', 'node2', ['cluster']]],
+ expected => ['b', 'h'],
+ bypassed => { a => 0 },
+ },
+ {
+ description => 'resources without limit do not get reserved',
+ limits => { x => 1 },
+ holders => [['h', 'node1', ['cluster', 'x']]],
+ waiters => [['a', 'node1', ['cluster', 'x'], 3], ['b', 'node2', ['cluster', 'y']]],
+ expected => ['b', 'h'],
+ },
];
for my $test ($tests->@*) {
@@ -101,6 +150,34 @@ for my $test ($tests->@*) {
"$test->{description} (waiters)",
);
}
+ for my $key (sort keys(($test->{bypassed} // {})->%*)) {
+ is(
+ $state->{waiters}->{$key}->{bypassed},
+ $test->{bypassed}->{$key},
+ "$test->{description} ($key bypassed)",
+ );
+ }
+}
+
+{
+ my $limits = { cluster => 2, x => 1 };
+ my $state = make_state([['h', 'node1', ['cluster', 'x']]], [['a', 'node1', ['cluster', 'x']]]);
+ my $add_younger = sub {
+ my ($key) = @_;
+ $state->{waiters}->{$key} =
+ { node => 'node2', resources => ['cluster'], seq => ++$state->{seq} };
+ PVE::VZDump::ClusterSlots::grant_waiters($state, $limits, $members);
+ };
+ for my $younger (qw(b c d)) {
+ $add_younger->($younger);
+ delete $state->{holders}->{$younger};
+ }
+ $add_younger->('e');
+ is_deeply(granted($state), ['h'], 'younger waiters stop getting ahead after a few times');
+
+ delete $state->{holders}->{h};
+ PVE::VZDump::ClusterSlots::grant_waiters($state, $limits, $members);
+ is_deeply(granted($state), ['a', 'e'], 'passed over waiter gets its slots once they are free');
}
# Request, grant, release and cleanup against an in-memory table.
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 17/19] api: backup info: add endpoints for cluster-wide backup slots
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (15 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 16/19] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 18/19] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH docs 19/19] vzdump: document concurrent backups and their limits Thomas Lamprecht
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
List the guest backups that hold or wait for cluster-wide backup
slots, to see where the slots went when guest backups wait.
Slots of an offline node stay taken, as its guest backups might still
be running. Allow releasing them once it is clear that these backups
stopped, for example because the node is powered off. Otherwise the
node releases them itself when it is back online.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/Cluster/BackupInfo.pm | 106 ++++++++++++++++++++++++++++++++-
PVE/VZDump/ClusterSlots.pm | 14 +++++
2 files changed, 118 insertions(+), 2 deletions(-)
diff --git a/PVE/API2/Cluster/BackupInfo.pm b/PVE/API2/Cluster/BackupInfo.pm
index 4ef59ac3d..d5e14a3c9 100644
--- a/PVE/API2/Cluster/BackupInfo.pm
+++ b/PVE/API2/Cluster/BackupInfo.pm
@@ -9,10 +9,11 @@ use PVE::Tools qw(extract_param);
use PVE::Cluster qw(cfs_lock_file cfs_read_file cfs_write_file);
use PVE::RESTHandler;
use PVE::RPCEnvironment;
-use PVE::JSONSchema;
+use PVE::JSONSchema qw(get_standard_option);
use PVE::Storage;
use PVE::Exception qw(raise_param_exc);
use PVE::VZDump;
+use PVE::VZDump::ClusterSlots;
use PVE::VZDump::Common;
use base qw(PVE::RESTHandler);
@@ -57,11 +58,112 @@ __PACKAGE__->register_method({
},
code => sub {
return [
- { subdir => 'not-backed-up' },
+ { subdir => 'not-backed-up' }, { subdir => 'slots' },
];
},
});
+__PACKAGE__->register_method({
+ name => 'get_backup_slots',
+ path => 'slots',
+ method => 'GET',
+ protected => 1,
+ description => "Shows guest backups that hold or wait for cluster-wide backup slots.",
+ permissions => {
+ description => "Only guests with 'VM.Audit' permissions are listed.",
+ check => ['perm', '/', ['Sys.Audit']],
+ },
+ parameters => {
+ additionalProperties => 0,
+ properties => {},
+ },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ properties => {
+ node => get_standard_option('pve-node'),
+ vmid => get_standard_option('pve-vmid'),
+ upid => {
+ type => 'string',
+ description => 'Backup task the guest backup belongs to.',
+ },
+ state => {
+ type => 'string',
+ enum => ['running', 'waiting'],
+ description =>
+ 'Whether the guest backup holds its slots or waits for them.',
+ },
+ since => {
+ type => 'integer',
+ description =>
+ 'When the guest backup got its slots or started waiting for them.',
+ },
+ storages => {
+ type => 'array',
+ items => get_standard_option('pve-storage-id'),
+ description =>
+ 'Shared storages with a backup concurrency limit that the guest'
+ . ' backup uses.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $rpcenv = PVE::RPCEnvironment::get();
+ my $user = $rpcenv->get_user();
+
+ my $state = PVE::VZDump::ClusterSlots::read_state();
+
+ my $result = [];
+ for my $kind (qw(holders waiters)) {
+ for my $entry (values $state->{$kind}->%*) {
+ next if !$rpcenv->check($user, "/vms/$entry->{vmid}", ['VM.Audit'], 1);
+
+ push $result->@*,
+ {
+ node => $entry->{node},
+ vmid => int($entry->{vmid}),
+ upid => $entry->{upid},
+ state => $kind eq 'holders' ? 'running' : 'waiting',
+ since => int($kind eq 'holders' ? $entry->{granted} : $entry->{queued}),
+ storages => [map { m!^storage/(.+)$! ? $1 : () } $entry->{resources}->@*],
+ };
+ }
+ }
+
+ return $result;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'release_backup_slots',
+ path => 'slots/{node}',
+ method => 'DELETE',
+ protected => 1,
+ description =>
+ "Release the cluster-wide backup slots of an offline node. They stay taken because its"
+ . " guest backups might still be running, so only release them once you are sure that"
+ . " these stopped, for example because the node is powered off.",
+ permissions => {
+ check => ['perm', '/', ['Sys.Modify']],
+ },
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ },
+ },
+ returns => { type => 'null' },
+ code => sub {
+ my ($param) = @_;
+
+ PVE::VZDump::ClusterSlots::release_node($param->{node});
+
+ return;
+ },
+});
+
__PACKAGE__->register_method({
name => 'get_guests_not_in_backup',
path => 'not-backed-up',
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
index f4cac71c7..962b270a0 100644
--- a/PVE/VZDump/ClusterSlots.pm
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -282,4 +282,18 @@ sub cleanup_local_entries {
});
}
+sub release_node {
+ my ($node) = @_;
+
+ update_state(sub {
+ my ($state) = @_;
+
+ # checked under the lock, so the node cannot rejoin and take new slots meanwhile
+ die "cannot release backup slots of node '$node' - it is online\n"
+ if get_members()->{$node}->{online};
+
+ remove_entries($state, sub { $_[0]->{node} eq $node });
+ });
+}
+
1;
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH manager 18/19] ui: add cluster-wide and storage backup concurrency limits
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (16 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 17/19] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
2026-11-01 22:58 ` [PATCH docs 19/19] vzdump: document concurrent backups and their limits Thomas Lamprecht
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Expose the cluster-wide limit next to the per-node one in the
datacenter options, and the limit of a storage in the advanced section
of the storage edit window, so admins can protect shared systems from
too many parallel backups without editing configuration files.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/OptionView.js | 8 ++++++++
www/manager6/storage/Base.js | 17 +++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/www/manager6/dc/OptionView.js b/www/manager6/dc/OptionView.js
index ac0f0bdd1..851288a96 100644
--- a/www/manager6/dc/OptionView.js
+++ b/www/manager6/dc/OptionView.js
@@ -372,6 +372,14 @@ Ext.define('PVE.dc.OptionView', {
minValue: 1,
submitValue: true,
},
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 'cluster',
+ fieldLabel: gettext('Cluster-wide'),
+ emptyText: gettext('Unlimited'),
+ minValue: 1,
+ submitValue: true,
+ },
],
});
me.add_inputpanel_row('next-id', gettext('Next Free VMID Range'), {
diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index 8961460b0..c66e0b20f 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -62,6 +62,23 @@ Ext.define('PVE.panel.StorageBase', {
}
};
+ if (me.type !== 'esxi') {
+ addAdvancedWidget({
+ xtype: 'proxmoxintegerfield',
+ name: 'backup-concurrency',
+ fieldLabel: gettext('Backup Concurrency'),
+ emptyText: gettext('Unlimited'),
+ minValue: 1,
+ deleteEmpty: !me.isCreate,
+ autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext(
+ 'Guest backups using this storage at the same time, as target or for guest disks. Counted for the whole cluster if the storage is shared, otherwise on each node.',
+ ),
+ },
+ });
+ }
+
const qemuImgStorageTypes = ['dir', 'btrfs', 'nfs', 'cifs'];
if (qemuImgStorageTypes.includes(me.type)) {
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH docs 19/19] vzdump: document concurrent backups and their limits
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
` (17 preceding siblings ...)
2026-11-01 22:58 ` [PATCH manager 18/19] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
@ 2026-11-01 22:58 ` Thomas Lamprecht
18 siblings, 0 replies; 20+ messages in thread
From: Thomas Lamprecht @ 2026-11-01 22:58 UTC (permalink / raw)
To: pve-devel
Describe how many guests a node backs up at the same time and that the
bandwidth limit applies to each of them, how jobs can run alongside
each other, and the cluster-wide and storage limits, including how to
size them and what happens with slots of an offline node.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 108 insertions(+)
diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..6e65c29 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,114 @@ the `performance` setting, `max-workers` (affects VM backups only) and
`pbs-entries-max` (affects container backups only). See the
xref:vzdump_configuration[configuration options] for details.
+[[vzdump_concurrency]]
+Concurrent Backups
+------------------
+
+By default, a node runs one backup job at a time and backs up its guests one
+after the other. A job that is due while another one runs waits for it, up to
+the `lockwait` time. As each node backs up its own guests, how long a job takes
+and how much load it puts on the backup target depend on how its guests are
+spread over the nodes. The settings described here allow backing up several
+guests of a node at the same time, running jobs alongside each other, and
+limiting the load on shared storages and backup servers.
+
+Node Limit and Job Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backup concurrency of a node limits how many guests it backs up at the same
+time, across all jobs. Set the default for all nodes in the datacenter options
+(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in
+their options (`backup-concurrency` in the node configuration):
+
+----
+# pvesh set /cluster/options --backup-concurrency 4
+# pvenode config set --backup-concurrency 8
+----
+
+A job backs up as many guests at the same time as its node allows. With the
+`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use
+fewer, but not more.
+
+Size the node limit by what the node itself can handle, that is, its CPU,
+memory, network connection, and local disks. Each parallel guest backup runs its
+own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to
+each guest backup on its own, so the bandwidth of a node can add up to the
+configured limit times the number of guests backed up at the same time.
+
+Jobs Running Alongside Each Other
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same
+time as other jobs that have it enabled, for example, a short hourly job
+alongside a long nightly one. Together, they stay within the backup concurrency
+of the node. A job without the option still runs alone, and while it waits for
+running jobs to finish, overlapping jobs that are due later do not start before
+it. Waiting for a free backup slot of the node is not limited by `lockwait`.
+
+If two running jobs include the same guest, the second one backs it up once the
+first one is done with it, waiting at most `lockwait` minutes.
+
+NOTE: The job phases of hook scripts can overlap for overlapping jobs. For
+example, a hook script that mounts a disk in the `job-start` phase and unmounts
+it in the `job-end` phase must not be used for overlapping jobs. With a job
+concurrency above one, the phases of different guests of a job overlap as well.
+
+Cluster-wide and Storage Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each node runs its part of a job on its own, so in a cluster, shared storages
+and backup servers get the backups of all nodes at the same time. To protect
+them, you can limit how many guest backups run at the same time:
+
+* for the whole cluster, with the `cluster` key of `backup-concurrency` in the
+ datacenter options ('Cluster-wide' in the UI),
+* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in
+ the advanced storage settings). It counts guest backups that use the storage
+ as backup target or for included guest disks, but not for fleecing or
+ temporary files. For a shared storage, the limit applies to the whole cluster,
+ for a local storage to each node.
+
+----
+# pvesh set /cluster/options --backup-concurrency node=8,cluster=12
+# pvesm set pbs-main --backup-concurrency 6
+----
+
+A guest backup only starts once all limits it counts against have room. Size
+the cluster-wide and storage limits by what the shared systems can handle,
+independent of the node limits. With such limits in place, raising the node
+limits cannot overload the shared systems, so the node limits only need to fit
+what each node can handle.
+
+TIP: Several storage entries can refer to the same system, for example,
+namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits
+count separately, so use the cluster-wide limit to bound their sum.
+
+Guest backups get cluster-wide slots in the order they started waiting. One that
+waits for a storage at its limit lets others with room go first, but only a few
+times. A job has at most one guest waiting for cluster-wide slots at a time, and
+waits for them at most `lockwait` minutes in total. Like for node slots, waiting
+for free slots of a local storage is not limited by `lockwait`. The task log
+tells which limit a guest waits for.
+To see which guest backups hold or wait for cluster-wide slots, use:
+
+----
+# pvesh get /cluster/backup-info/slots
+----
+
+Granting cluster-wide slots requires quorum, and the limits are only enforced
+once all online nodes of the cluster run a version that supports them. Guest
+backups that started before that, or while neither a cluster-wide limit nor a
+limit on a shared storage was set, do not count against them. Slots of a node
+that goes offline stay taken, as its guest backups might still be running, and
+the node releases them once it is back. Once you made sure that these backups
+stopped, for example because the node is powered off, you can release its slots
+manually:
+
+----
+# pvesh delete /cluster/backup-info/slots/NODE
+----
+
[[vzdump_retention]]
Backup Retention
----------------
--
2.47.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
end of thread, other threads:[~2026-11-01 23:02 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 01/19] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 02/19] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-01 22:58 ` [PATCH storage 03/19] add backup concurrency option to storages Thomas Lamprecht
2026-11-01 22:58 ` [PATCH guest-common 04/19] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-01 22:58 ` [PATCH container 05/19] vzdump: honor bandwidth limit for backups to Proxmox Backup Server Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 06/19] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 07/19] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 08/19] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 09/19] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 10/19] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 11/19] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 12/19] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 13/19] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 14/19] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 15/19] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 16/19] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 17/19] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 18/19] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH docs 19/19] vzdump: document concurrent backups and their limits Thomas Lamprecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.