all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
@ 2026-11-18 13:18 Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
                   ` (19 more replies)
  0 siblings, 20 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.

changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
  shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
  than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
  quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
  also explain the review points not followed

[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/


Original cover letter
=====================

A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.

This series adds:

 - parallel guest backups within a job, up to a per-node backup
   concurrency, set as datacenter default or per node; a job can lower
   it with its new 'concurrency' option,
 - the 'overlap' job option to run alongside other such jobs, together
   within the node's backup concurrency,
 - cluster-wide and per-storage limits that protect shared storages and
   backup servers from too many guest backups at the same time.

The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.

The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.

Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.

Repo (build-)dependencies:
 pve-manager -> pve-guest-common, pve-cluster, pve-storage

A separate series on top adds a start window option for backup jobs.

Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.

pve-cluster:

Thomas Lamprecht (2):
      datacenter config: add backup concurrency setting
      datacenter config: backup concurrency: add cluster-wide limit

 src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

pve-storage:

Thomas Lamprecht (1):
      add backup concurrency option to storages

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 src/test/plugin_properties_test.pm   | 5 +++--
 15 files changed, 25 insertions(+), 2 deletions(-)

pve-guest-common:

Thomas Lamprecht (1):
      vzdump: add options for parallel guest backups and overlapping jobs

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

pve-manager:

Thomas Lamprecht (15):
      vzdump: drop never-set fallback for the lock wait time
      vzdump: notification: capture guest logs right after each backup
      vzdump: stop all running backup jobs of the node
      vzdump: show the error if a job could not get the global lock
      vzdump: run job-init hook and set up target once the job may start
      vzdump: list guests as skipped if the job could not start
      vzdump: back up guests of a job in parallel
      vzdump: allow jobs to run alongside each other
      ui: options: add backup concurrency for datacenter and nodes
      ui: backup job: add concurrency and overlap options
      ui: backup now: add overlap option
      vzdump: limit concurrent guest backups cluster-wide
      vzdump: honor backup concurrency limits of storages
      api: backup info: add endpoints for cluster-wide backup slots
      ui: add cluster-wide and storage backup concurrency limits

 PVE/API2/Cluster/BackupInfo.pm              | 106 ++-
 PVE/API2/VZDump.pm                          |  14 +-
 PVE/Makefile                                |   2 +-
 PVE/NodeConfig.pm                           |   7 +
 PVE/Service/pvescheduler.pm                 |   4 +
 PVE/VZDump.pm                               | 893 ++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm                  | 296 +++++++
 PVE/VZDump/Makefile                         |  15 +
 debian/tmpfiles                             |   5 +-
 test/Makefile                               |  10 +-
 test/vzdump_cluster_slots_test.pl           | 282 +++++++
 test/vzdump_concurrency_test.pl             |  89 ++
 test/vzdump_new_test.pl                     |   1 +
 www/manager6/dc/OptionView.js               |  22 +
 www/manager6/node/NodeOptionsView.js        |   8 +
 www/manager6/panel/BackupAdvancedOptions.js |  40 +-
 www/manager6/storage/Base.js                |  17 +
 www/manager6/window/Backup.js               |  28 +-
 18 files changed, 1773 insertions(+), 66 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      vzdump: document concurrent backups and their limits

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

-- 
2.47.3





^ permalink raw reply	[flat|nested] 21+ messages in thread

* [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
                   ` (18 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Allows configuring how many guests a node backs up at the same time,
as default for all nodes. The node configuration can override it, and
a backup job can only lower it.

The setting is a property string with the per-node limit as default
key, so further limits fit in without changing its type.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

no changes since v1

 src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 56b19bd..1bab9f2 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -234,6 +234,18 @@ my $next_id_format = {
     },
 };
 
+my $backup_concurrency_format = {
+    node => {
+        type => 'integer',
+        description => "Maximal number of guests backed up at the same time on each node,"
+            . " unless the node configuration overrides it.",
+        minimum => 1,
+        default => 1,
+        default_key => 1,
+        optional => 1,
+    },
+};
+
 my $u2f_format = {
     appid => {
         type => 'string',
@@ -494,6 +506,12 @@ my $datacenter_schema = {
             format => $replication_format,
             description => "For cluster wide replication settings.",
         },
+        'backup-concurrency' => {
+            optional => 1,
+            type => 'string',
+            format => $backup_concurrency_format,
+            description => "Limits for backing up guests at the same time.",
+        },
         console => {
             optional => 1,
             type => 'string',
@@ -673,6 +691,11 @@ sub parse_datacenter_config {
         $res->{'next-id'} = parse_property_string($next_id_format, $next_id);
     }
 
+    if (my $backup_concurrency = $res->{'backup-concurrency'}) {
+        $res->{'backup-concurrency'} =
+            parse_property_string($backup_concurrency_format, $backup_concurrency);
+    }
+
     if (my $ha = $res->{ha}) {
         $res->{ha} = parse_property_string($ha_format, $ha);
     }
@@ -770,6 +793,11 @@ sub write_datacenter_config {
         $cfg->{'next-id'} = PVE::JSONSchema::print_property_string($next_id, $next_id_format);
     }
 
+    if (ref(my $backup_concurrency = $cfg->{'backup-concurrency'})) {
+        $cfg->{'backup-concurrency'} =
+            PVE::JSONSchema::print_property_string($backup_concurrency, $backup_concurrency_format);
+    }
+
     if (ref(my $ha = $cfg->{ha})) {
         $cfg->{ha} = PVE::JSONSchema::print_property_string($ha, $ha_format);
     }
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
                   ` (17 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Nodes run their part of a backup job on their own, so how many guests
are backed up at the same time depends on how many nodes the guests of
a job are spread over. In larger clusters this can overload shared
storages and backup servers, while lowering the per-node limit to
protect them slows down jobs whose guests are mostly on one node.

Add an optional limit for the whole cluster, enforced by the backup
code of all nodes together.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

no changes since v1

 src/PVE/DataCenterConfig.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 1bab9f2..ec62dbd 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -244,6 +244,13 @@ my $backup_concurrency_format = {
         default_key => 1,
         optional => 1,
     },
+    cluster => {
+        type => 'integer',
+        description =>
+            "Maximal number of guests backed up at the same time in the whole cluster.",
+        minimum => 1,
+        optional => 1,
+    },
 };
 
 my $u2f_format = {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 storage 03/20] add backup concurrency option to storages
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
                   ` (16 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Allows limiting how many guest backups use a storage at the same time,
be it as backup target or because guest disks live on it. A backup
server or a Ceph cluster can thus be protected from more backup load
than it can handle, independent of how many nodes and jobs take part.

For shared storages the limit counts for the whole cluster, for local
storages on each node. The ESXi import storage is not a backup source
or target and does not get the option. The option is declared in the
base plugin, as the backup code reads it from storages of any type, so
plugins that isolate their properties can offer it as well. Third-party
plugins need to list it in their options() to support it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- rebase onto the property isolation changes; the option is now in the
  pinned lists of the plugin property test

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 src/test/plugin_properties_test.pm   | 5 +++--
 15 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index 72cb39b7..3f5181fd 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -83,6 +83,7 @@ sub options {
     return {
         path => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
         'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CIFSPlugin.pm b/src/PVE/Storage/CIFSPlugin.pm
index 15d5a22d..594ec1de 100644
--- a/src/PVE/Storage/CIFSPlugin.pm
+++ b/src/PVE/Storage/CIFSPlugin.pm
@@ -189,6 +189,7 @@ sub options {
         share => { fixed => 1 },
         subdir => { optional => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         'prune-backups' => { optional => 1 },
         'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index 39b9686b..19a1f7ab 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -164,6 +164,7 @@ sub options {
         'content-dirs' => { optional => 1 },
         monhost => { optional => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         subdir => { optional => 1 },
         disable => { optional => 1 },
         options => { optional => 1 },
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 397d4fe7..56c5d785 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -76,6 +76,7 @@ sub options {
         path => { fixed => 1 },
         'content-dirs' => { optional => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
         'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index d57bc704..f1d133ba 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -77,6 +77,7 @@ sub options {
         portal => { fixed => 1 },
         target => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         content => { optional => 1 },
         bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm
index 1e349cde..9c7b20a4 100644
--- a/src/PVE/Storage/ISCSIPlugin.pm
+++ b/src/PVE/Storage/ISCSIPlugin.pm
@@ -416,6 +416,7 @@ sub options {
         portal => { fixed => 1 },
         target => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         content => { optional => 1 },
         bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 3cdab9e4..60bc8743 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -949,6 +949,7 @@ sub options {
     return {
         vgname => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         shared => { optional => 1 },
         disable => { optional => 1 },
         saferemove => { optional => 1 },
diff --git a/src/PVE/Storage/LvmThinPlugin.pm b/src/PVE/Storage/LvmThinPlugin.pm
index 67f7bb2f..74817e77 100644
--- a/src/PVE/Storage/LvmThinPlugin.pm
+++ b/src/PVE/Storage/LvmThinPlugin.pm
@@ -63,6 +63,7 @@ sub options {
         thinpool => { fixed => 1 },
         vgname => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         content => { optional => 1 },
         bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/NFSPlugin.pm b/src/PVE/Storage/NFSPlugin.pm
index fdc32c01..cf42b619 100644
--- a/src/PVE/Storage/NFSPlugin.pm
+++ b/src/PVE/Storage/NFSPlugin.pm
@@ -119,6 +119,7 @@ sub options {
         server => { fixed => 1 },
         export => { fixed => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         'prune-backups' => { optional => 1 },
         'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
index cf826237..01f8c936 100644
--- a/src/PVE/Storage/PBSPlugin.pm
+++ b/src/PVE/Storage/PBSPlugin.pm
@@ -85,6 +85,7 @@ sub options {
         namespace => { optional => 1 },
         port => { optional => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         content => { optional => 1 },
         username => { optional => 1 },
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 724c2d35..2095e7eb 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -217,6 +217,15 @@ my $defaultData = {
             default =>
                 "Unlimited for users with Datastore.Allocate privilege, 5 for other users",
         },
+        'backup-concurrency' => {
+            description =>
+                "Maximal number of guest backups using this storage at the same time,"
+                . " either as backup target or for guest disks. Counted for the whole cluster if"
+                . " the storage is shared, for each node otherwise.",
+            type => 'integer',
+            minimum => 1,
+            optional => 1,
+        },
         shared => {
             title => "Shared",
             description =>
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 26a1cf81..f4bc702f 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -482,6 +482,7 @@ sub properties {
 sub options {
     return {
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         monhost => { optional => 1 },
         pool => { optional => 1 },
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 732069e0..69316e7d 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -247,6 +247,7 @@ sub properties {
 sub options {
     return {
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         portal => { fixed => 1 },
         target => { fixed => 1 },
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index 1085db94..6c47dc48 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -106,6 +106,7 @@ sub options {
         blocksize => { optional => 1 },
         sparse => { optional => 1 },
         nodes => { optional => 1 },
+        'backup-concurrency' => { optional => 1 },
         disable => { optional => 1 },
         content => { optional => 1 },
         bwlimit => { optional => 1 },
diff --git a/src/test/plugin_properties_test.pm b/src/test/plugin_properties_test.pm
index d8e336d1..25c1a023 100644
--- a/src/test/plugin_properties_test.pm
+++ b/src/test/plugin_properties_test.pm
@@ -12,6 +12,7 @@ use PVE::Storage::Plugin;
 # none can be removed, renamed, narrowed or left unexposed by an in-tree plugin that isolates.
 my $available = {
     authsupported => 'string',
+    'backup-concurrency' => 'integer minimum=1',
     base => 'string:pve-volume-id',
     blocksize => 'string:pve-storage-zfs-blocksize',
     bwlimit => 'string:{clone,default,migration,move,restore}',
@@ -137,8 +138,8 @@ for my $name (sort keys $available->%*) {
 # adding one can break third-party plugins. A property belongs here only if methods of the base
 # plugin or code outside of the plugins read it from storages of any type.
 my $base = [qw(
-    base bwlimit content content-dirs create-base-path create-subdirs disable format
-    max-protected-backups mkdir nodes path preallocation prune-backups shared
+    backup-concurrency base bwlimit content content-dirs create-base-path create-subdirs disable
+    format max-protected-backups mkdir nodes path preallocation prune-backups shared
     snapshot-as-volume-chain storage type
 )];
 my $declared = PVE::Storage::Plugin->base_properties();
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (2 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
                   ` (15 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

A node only ever runs one backup job, and that job backs up its guests
one after the other. How long a job takes thus depends on how its
guests happen to be spread over the nodes, and a short job has to wait
until a long one finishes, failing once the lock wait time is
exceeded.

The new 'concurrency' option allows backing up several guests of a job
at the same time, up to the backup concurrency of the node. The
bandwidth limit then applies to each of them, as its description now
says. The 'overlap' option lets a job run alongside other jobs that
allow it as well. It is off by default, as existing hook scripts may
rely on job phases never overlapping.

With overlapping jobs the same guest can be part of two running
backups, and with cluster-wide limits a guest backup can wait for a
slot, so the lock wait now also bounds these waits.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- reword the commit message

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 2cb48a3..26d7799 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -298,7 +298,7 @@ my $confdesc = {
     },
     bwlimit => {
         type => 'integer',
-        description => "Limit I/O bandwidth (in KiB/s).",
+        description => "Limit I/O bandwidth of each guest backup (in KiB/s).",
         optional => 1,
         minimum => 0,
         default => 0,
@@ -328,11 +328,27 @@ my $confdesc = {
     },
     lockwait => {
         type => 'integer',
-        description => "Maximal time to wait for the global lock (minutes).",
+        description =>
+            "Maximal time to wait for the global lock, for guests that another backup"
+            . " is currently processing, and for cluster-wide backup slots (minutes).",
         optional => 1,
         minimum => 0,
         default => 3 * 60, # 3 hours
     },
+    concurrency => {
+        type => 'integer',
+        description => "Maximal number of guests to back up at the same time on each node."
+            . " Defaults to, and is limited by, the backup concurrency configured for the node.",
+        optional => 1,
+        minimum => 1,
+    },
+    overlap => {
+        type => 'boolean',
+        description => "Allow running at the same time as other backups on the same node that"
+            . " also allow it. The job phases of their hook scripts can then overlap.",
+        optional => 1,
+        default => 0,
+    },
     stopwait => {
         type => 'integer',
         description => "Maximal time to wait until a guest system is stopped (minutes).",
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (3 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
                   ` (14 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

The lock wait time falls back to a value stored on the object, but
nothing has ever set one there since vzdump moved into this package.
The option always has a value anyway, as the constructor fills in the
configured or default one.

No semantic change intended.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

new in v2 [Fiona]

 PVE/VZDump.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index d2861f9e7..48d2541c9 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -761,7 +761,7 @@ sub getlock {
 
     my $fh;
 
-    my $maxwait = $self->{opts}->{lockwait} || $self->{lockwait};
+    my $maxwait = $self->{opts}->{lockwait};
 
     die "missing UPID" if !$upid; # should not happen
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (4 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
                   ` (13 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

The log of a guest backup is written to a path derived from the guest
alone and was only read back when sending the notification at the end
of the job. This relies on no other backup of the guest starting in the
meantime, which no longer holds once backup jobs can run alongside each
other.

Read the log as the last step of each guest backup instead. Logs
longer than the notification size limit are cut, as they would not be
included anyway.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
---

changes v1 -> v2:
- warn if the guest log cannot be opened [Fiona]

Not changed: the guest log under /var/log/vzdump is kept after reading
instead of unlinked [Fiona], as it is the last backup log of the guest,
like before this series.

 PVE/VZDump.pm | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 48d2541c9..92a48babf 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -368,24 +368,39 @@ sub read_vzdump_defaults {
     return $res;
 }
 
-my sub read_backup_task_logs {
+use constant MAX_LOG_SIZE => 1024 * 1024;
+
+my sub read_guest_log {
+    my ($path) = @_;
+
+    my $fh;
+    if (!open($fh, '<', $path)) {
+        debugmsg('warn', "unable to open log file '$path' - $!");
+        return undef;
+    }
+    my $log = '';
+    read($fh, $log, MAX_LOG_SIZE + 1); # anything longer is not sent anyway
+    close($fh);
+
+    return $log;
+}
+
+my sub format_backup_task_logs {
     my ($task_list) = @_;
 
     my $task_logs = "";
 
     for my $task (@$task_list) {
         my $vmid = $task->{vmid};
-        my $log_file = $task->{tmplog};
         if (!$task->{tmplog}) {
             $task_logs .= "$vmid: no log available\n\n";
             next;
         }
-        if (open(my $TMP, '<', "$log_file")) {
-            while (my $line = <$TMP>) {
+        if (defined(my $log = $task->{log})) {
+            for my $line (split(/^/m, $log)) {
                 next if $line =~ /^status: \d+/; # not useful in mails
                 $task_logs .= encode8bit("$vmid: $line");
             }
-            close($TMP);
         } else {
             $task_logs .= "$vmid: Could not open log file\n\n";
         }
@@ -484,8 +499,6 @@ my sub get_hostname {
     return $hostname;
 }
 
-use constant MAX_LOG_SIZE => 1024 * 1024;
-
 sub send_notification {
     my ($self, $tasklist, $total_time, $err, $detail_pre, $detail_post) = @_;
 
@@ -516,7 +529,7 @@ sub send_notification {
 
     my $text_log_part = "$cmdline\n\n";
     $text_log_part .= $detail_pre . "\n" if defined($detail_pre);
-    $text_log_part .= read_backup_task_logs($tasklist);
+    $text_log_part .= format_backup_task_logs($tasklist);
     $text_log_part .= $detail_post if defined($detail_post);
 
     if (length($text_log_part) > MAX_LOG_SIZE) {
@@ -1396,6 +1409,8 @@ sub exec_backup_task {
     eval { $self->run_hook_script('log-end', $task); };
     debugmsg('warn', $@) if $@; # message already contains command with phase name
 
+    $task->{log} = read_guest_log($task->{tmplog}) if $task->{tmplog};
+
     die $err if $err && $err =~ m/^interrupted by signal$/;
 }
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (5 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
                   ` (12 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Stopping running backups only signalled the job recorded in the pid
file, which is the one holding the global lock. Jobs waiting for the
lock went on afterwards, and once jobs can run alongside each other,
the pid file cannot represent all of them anyway.

Signal every backup task of the node instead, including those still
waiting to start, and drop the pid file. As when stopping a task, the
signal goes to the whole process group.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
---

changes v1 -> v2:
- use PVE::UPID, say "processes" if several are left [Fiona]

 PVE/VZDump.pm | 39 ++++++++++++++++++---------------------
 1 file changed, 18 insertions(+), 21 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 92a48babf..9842069a7 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -22,6 +22,7 @@ use PVE::JSONSchema qw(get_standard_option);
 use PVE::Notify;
 use PVE::RPCEnvironment;
 use PVE::Storage;
+use PVE::UPID;
 use PVE::VZDump::Common;
 use PVE::VZDump::Plugin;
 use PVE::Tools qw(extract_param split_list);
@@ -44,7 +45,6 @@ PVE::JSONSchema::register_standard_option(
 my @posix_filesystems = qw(ext3 ext4 nfs nfs4 reiserfs xfs);
 
 my $lockfile = '/var/run/vzdump.lock';
-my $pidfile = '/var/run/vzdump.pid';
 my $logdir = '/var/log/vzdump';
 
 my @plugins = qw();
@@ -816,8 +816,6 @@ sub getlock {
         debugmsg('info', "got global lock", undef, 1);
     }
 
-    PVE::Tools::file_set_contents($pidfile, $upid);
-
     return $SERVER_FLCK;
 }
 
@@ -1515,8 +1513,6 @@ sub exec_backup {
     die $err if $err;
 
     die "job errors\n" if $errcount;
-
-    unlink $pidfile;
 }
 
 sub option_exists {
@@ -1581,26 +1577,27 @@ sub verify_vzdump_parameters {
 }
 
 sub stop_running_backups {
-    my ($self) = @_;
+    my $nodename = PVE::INotify::nodename();
 
-    my $upid = PVE::Tools::file_read_firstline($pidfile);
-    return if !$upid;
+    my $running = [];
+    for my $worker (PVE::RPCEnvironment::get()->active_workers()->@*) {
+        my $task = PVE::UPID::decode($worker->{upid}, 1) or next;
+        next if $task->{type} ne 'vzdump' || $task->{node} ne $nodename;
+        next if !PVE::ProcFSTools::check_process_running($task->{pid}, $task->{pstart});
 
-    my $task = PVE::Tools::upid_decode($upid);
+        # the whole process group, like stopping the task does
+        kill(15, -$task->{pid});
+        push $running->@*, $task;
+    }
 
-    if (
-        PVE::ProcFSTools::check_process_running($task->{pid}, $task->{pstart})
-        && PVE::ProcFSTools::read_proc_starttime($task->{pid}) == $task->{pstart}
-    ) {
-        kill(15, $task->{pid});
-        # wait max 15 seconds to shut down (else, do nothing for now)
-        my $i;
-        for ($i = 15; $i > 0; $i--) {
-            last if !PVE::ProcFSTools::check_process_running(($task->{pid}, $task->{pstart}));
-            sleep(1);
-        }
-        die "stopping backup process $task->{pid} failed\n" if $i == 0;
+    for (1 .. 15) {
+        $running = [grep { PVE::ProcFSTools::check_process_running($_->{pid}, $_->{pstart}) }
+            $running->@*];
+        return if !scalar($running->@*);
+        sleep(1);
     }
+    my $what = scalar($running->@*) > 1 ? 'processes' : 'process';
+    die "stopping backup $what " . join(', ', map { $_->{pid} } $running->@*) . " failed\n";
 }
 
 sub get_included_guests {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (6 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
                   ` (11 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

When a job could not get the global lock, the notification subject only
said "multiple problems", as the error message ended with a newline.
The worker then exited directly, so the task showed "unexpected
status" instead of the reason.

Remove the newline and die with the error, so the notification subject
and the task status both name the actual problem. If sending the
notification fails, log that and still end with the actual problem.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- log a failure to send the notification instead of failing with it,
  as job setup errors got handled before patch 9 [Fiona]

 PVE/API2/VZDump.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 3856ec6bf..ef3b9d169 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -140,8 +140,10 @@ __PACKAGE__->register_method({
                 $vzdump->getlock($upid); # only one process allowed
             };
             if (my $err = $@) {
-                $vzdump->send_notification([], 0, $err);
-                exit(-1);
+                chomp($err); # otherwise reported as multiple problems
+                eval { $vzdump->send_notification([], 0, $err) };
+                PVE::VZDump::debugmsg('err', $@) if $@;
+                die "$err\n";
             }
 
             if (defined($param->{ionice})) {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (7 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
                   ` (10 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

The job-init hook and the activation of the backup storage ran before
taking the global lock. A job waiting for another one thus already
called its hook script, for example to mount or wake up its target,
while the other job was still running, and then possibly waited for
hours.

Take the lock first and do the job setup afterwards, handling setup
errors like a failure to get the lock. Hook scripts now see the job
phases of different jobs strictly in sequence.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

no changes since v1

 PVE/API2/VZDump.pm      |  4 +++-
 PVE/VZDump.pm           | 20 ++++++++++++--------
 test/vzdump_new_test.pl |  1 +
 3 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index ef3b9d169..4e59432f2 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -137,7 +137,9 @@ __PACKAGE__->register_method({
             my $vzdump = PVE::VZDump->new($cmdline, $param, $skiplist);
 
             my $LOCK_FH = eval {
-                $vzdump->getlock($upid); # only one process allowed
+                my $lock_fh = $vzdump->getlock($upid); # only one process allowed
+                $vzdump->init_job();
+                $lock_fh;
             };
             if (my $err = $@) {
                 chomp($err); # otherwise reported as multiple problems
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 9842069a7..352c8f0ba 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -680,6 +680,16 @@ sub new {
         delete $opts->{$_} for qw(notes-template protected);
     }
 
+    $self->{'default-prune-backups'} = $defaults->{'prune-backups'};
+
+    return $self;
+}
+
+sub init_job {
+    my ($self) = @_;
+
+    my $opts = $self->{opts};
+
     my $errors = '';
     my $add_error = sub {
         my ($error) = @_;
@@ -720,7 +730,7 @@ sub new {
         die "internal error";
     }
 
-    $opts->{'prune-backups'} //= $defaults->{'prune-backups'};
+    $opts->{'prune-backups'} //= $self->{'default-prune-backups'};
 
     # avoid triggering any remove code path if keep-all is set
     $opts->{remove} = 0 if $opts->{'prune-backups'}->{'keep-all'};
@@ -729,13 +739,7 @@ sub new {
         $add_error->("tmpdir '$opts->{tmpdir}' does not exist");
     }
 
-    if ($errors) {
-        eval { $self->send_notification([], 0, $errors); };
-        debugmsg('err', $@) if $@;
-        die "$errors\n";
-    }
-
-    return $self;
+    die "$errors\n" if $errors;
 }
 
 sub get_mount_info {
diff --git a/test/vzdump_new_test.pl b/test/vzdump_new_test.pl
index db829e70f..490ae4482 100755
--- a/test/vzdump_new_test.pl
+++ b/test/vzdump_new_test.pl
@@ -539,6 +539,7 @@ foreach my $test (@tests) {
         PVE::VZDump::parse_mailto_exclude_path($test->{cli_param});
 
         my $vzdump = PVE::VZDump->new('fake cmdline', $test->{cli_param}, undef);
+        $vzdump->init_job();
 
         my $opts = $vzdump->{opts} or die "did not get options\n";
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (8 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
                   ` (9 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

If a job could not get the global lock, or its setup failed, for
example because the job-init hook failed or the backup server was not
reachable, none of its guests got backed up, but the notification did
not list any of them.

List them as skipped with the reason, so notifications, including those
sent only on failure, show what is missing.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- list the skipped guests inside the eval that patch 8 added

 PVE/API2/VZDump.pm |  6 +++++-
 PVE/VZDump.pm      | 48 +++++++++++++++++++++++++++++++++++++---------
 2 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 4e59432f2..97b1797f8 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -143,7 +143,11 @@ __PACKAGE__->register_method({
             };
             if (my $err = $@) {
                 chomp($err); # otherwise reported as multiple problems
-                eval { $vzdump->send_notification([], 0, $err) };
+                eval {
+                    my $tasklist =
+                        $vzdump->skip_all_guests($rpcenv, $user, 'job did not start');
+                    $vzdump->send_notification($tasklist, 0, $err);
+                };
                 PVE::VZDump::debugmsg('err', $@) if $@;
                 die "$err\n";
             }
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 352c8f0ba..fd2a74ab9 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -392,7 +392,10 @@ my sub format_backup_task_logs {
 
     for my $task (@$task_list) {
         my $vmid = $task->{vmid};
-        if (!$task->{tmplog}) {
+        if ($task->{state} eq 'skipped') {
+            $task_logs .= "$vmid: skipped - $task->{msg}\n\n";
+            next;
+        } elsif (!$task->{tmplog}) {
             $task_logs .= "$vmid: no log available\n\n";
             next;
         }
@@ -1416,18 +1419,11 @@ sub exec_backup_task {
     die $err if $err && $err =~ m/^interrupted by signal$/;
 }
 
-sub exec_backup {
+sub get_tasklist {
     my ($self, $rpcenv, $authuser) = @_;
 
     my $opts = $self->{opts};
 
-    debugmsg('info', "starting new backup job: $self->{cmdline}", undef, 1);
-
-    if (scalar(@{ $self->{skiplist} })) {
-        my $skip_string = join(', ', sort { $a <=> $b } @{ $self->{skiplist} });
-        debugmsg('info', "skip external VMs: $skip_string");
-    }
-
     my $tasklist = [];
     my $vzdump_plugins = {};
     foreach my $plugin (@{ $self->{plugins} }) {
@@ -1454,6 +1450,40 @@ sub exec_backup {
             };
     }
 
+    return $tasklist;
+}
+
+my sub skip_task {
+    my ($task, $reason) = @_;
+
+    $task->{state} = 'skipped';
+    $task->{msg} = $reason;
+    debugmsg('err', "Backup of VM $task->{vmid} skipped - $reason");
+}
+
+sub skip_all_guests {
+    my ($self, $rpcenv, $authuser, $reason) = @_;
+
+    my $tasklist = $self->get_tasklist($rpcenv, $authuser);
+    skip_task($_, $reason) for $tasklist->@*;
+
+    return $tasklist;
+}
+
+sub exec_backup {
+    my ($self, $rpcenv, $authuser) = @_;
+
+    my $opts = $self->{opts};
+
+    debugmsg('info', "starting new backup job: $self->{cmdline}", undef, 1);
+
+    if (scalar(@{ $self->{skiplist} })) {
+        my $skip_string = join(', ', sort { $a <=> $b } @{ $self->{skiplist} });
+        debugmsg('info', "skip external VMs: $skip_string");
+    }
+
+    my $tasklist = $self->get_tasklist($rpcenv, $authuser);
+
     # Use in-memory files for the outer hook logs to pass them to sendmail.
     my $job_start_log = '';
     my $job_end_log = '';
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (9 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
                   ` (8 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Each node backed up the guests of a job strictly one after the other.
How long a job took thus depended on how its guests were spread over
the nodes: guests all on one node were backed up in series, even if
that node and the backup target could handle more.

Allow backing up several guests at the same time, up to the backup
concurrency of the node. It comes from the new node configuration
option or the datacenter-wide default, and stays at one guest unless
configured. A job can lower it with its 'concurrency' option.

Each guest backup then runs in a forked process. The job process
collects the results, prefixes the guest output in the task log with
the guest ID, keeps the warning count, and sends a single notification
as before. A guest backup process that ends without a result is
reported as failed.

Stopping the task signals all its processes, so the guest backups stop
on their own while the job process waits for them. It does not signal
them again, which could interrupt their cleanup. A stop request that
arrives while a guest backup process is being forked does not reach
that process. The new process therefore starts its backup only after
the job process confirmed that no stop request came in; otherwise the
guest counts as not started. A signal that only reached the job
process keeps it from starting further guests.

A guest listed more than once is backed up once, as its backups would
otherwise run at the same time.

Backups to external providers stay sequential, as providers are not
known to cope with backups from multiple processes. With a concurrency
of one, guests are backed up in the job process as before.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- comment why a backup process without start byte ends as interrupted
  [Fiona]
- flush the output of a guest backup only once [Fiona]
- report a guest backup process that fails before its backup begins as
  failed with the error, it was left as not started

Not changed [Fiona]:
- no separate message for a canceled start: end-of-file means do not
  start, both after a stop request while forking and if the job process
  is gone, and in the latter case nothing reads the output anymore
- no separate exit status check. The result is written once the backup
  and its cleanup finished, so a non-zero exit with a result is either
  an interrupted backup that the result records as failed, or a process
  killed while exiting after a finished backup

 PVE/NodeConfig.pm               |   7 +
 PVE/VZDump.pm                   | 257 +++++++++++++++++++++++++++++++-
 test/Makefile                   |   6 +-
 test/vzdump_concurrency_test.pl |  89 +++++++++++
 4 files changed, 353 insertions(+), 6 deletions(-)
 create mode 100755 test/vzdump_concurrency_test.pl

diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm
index cc1a2a296..fbd7ecf87 100644
--- a/PVE/NodeConfig.pm
+++ b/PVE/NodeConfig.pm
@@ -108,6 +108,13 @@ my $confdesc = {
         default => 80,
         optional => 1,
     },
+    'backup-concurrency' => {
+        description => "Maximal number of guests backed up at the same time on this node."
+            . " Overrides the datacenter-wide default.",
+        type => 'integer',
+        minimum => 1,
+        optional => 1,
+    },
 };
 
 my $wakeonlan_desc = {
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index fd2a74ab9..054665430 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -11,6 +11,7 @@ use IO::File;
 use IO::Select;
 use IPC::Open3;
 use POSIX qw(strftime);
+use Storable;
 use Time::Local;
 
 use PVE::Cluster qw(cfs_read_file);
@@ -19,6 +20,7 @@ use PVE::Exception qw(raise_param_exc);
 use PVE::HA::Config;
 use PVE::HA::Env::PVE2;
 use PVE::JSONSchema qw(get_standard_option);
+use PVE::NodeConfig;
 use PVE::Notify;
 use PVE::RPCEnvironment;
 use PVE::Storage;
@@ -289,10 +291,11 @@ sub check_vmids {
     my (@vmids) = @_;
 
     my $res = [];
+    my $seen = {};
     for my $vmid (sort { $a <=> $b } @vmids) {
         die "ERROR: strange VM ID '${vmid}'\n" if $vmid !~ m/^\d+$/;
         $vmid = int($vmid); # remove leading zeros
-        next if !$vmid;
+        next if !$vmid || $seen->{$vmid}++;
         push @$res, $vmid;
     }
 
@@ -1470,6 +1473,245 @@ sub skip_all_guests {
     return $tasklist;
 }
 
+my sub fail_task {
+    my ($task, $err) = @_;
+
+    $task->{state} = 'err';
+    $task->{msg} = $err;
+    debugmsg('err', "Backup of VM $task->{vmid} failed - $err");
+}
+
+sub node_backup_concurrency {
+    my $node_conf = PVE::NodeConfig::load_config(PVE::INotify::nodename());
+
+    return $node_conf->{'backup-concurrency'}
+        // cfs_read_file('datacenter.cfg')->{'backup-concurrency'}->{node} // 1;
+}
+
+sub guest_backup_concurrency {
+    my ($self) = @_;
+
+    # external providers are not known to cope with backups from multiple processes
+    return 1 if $self->{'backup-provider'};
+
+    my $node_limit = node_backup_concurrency();
+    my $limit = $self->{opts}->{concurrency} // $node_limit;
+
+    return $limit < $node_limit ? $limit : $node_limit;
+}
+
+my $guest_result_keys = [qw(backuptime hostname log msg size state target tmplog vmtype)];
+
+my sub fork_guest_backup {
+    my ($self, $task, $running, $interrupted) = @_;
+
+    pipe(my $output_reader, my $output_writer) or die "unable to create pipe - $!\n";
+    pipe(my $result_reader, my $result_writer) or die "unable to create pipe - $!\n";
+    pipe(my $start_reader, my $start_writer) or die "unable to create pipe - $!\n";
+
+    # Until the child set up its own handlers, a signal would only reach the inherited one of the
+    # job process and get lost.
+    my $signals = POSIX::SigSet->new(
+        POSIX::SIGINT(),
+        POSIX::SIGTERM(),
+        POSIX::SIGQUIT(),
+        POSIX::SIGHUP(),
+        POSIX::SIGPIPE(),
+    );
+    my $old_mask = POSIX::SigSet->new();
+    POSIX::sigprocmask(POSIX::SIG_BLOCK(), $signals, $old_mask)
+        or die "unable to block signals - $!\n";
+
+    my $pid = fork();
+    if (!defined($pid)) {
+        my $err = $!;
+        POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+        die "unable to fork guest backup - $err\n";
+    }
+
+    if (!$pid) {
+        my $interrupted = 0;
+        $SIG{INT} = $SIG{TERM} = $SIG{QUIT} = $SIG{HUP} = $SIG{PIPE} = sub {
+            # a second signal must not abort the cleanup the first one started
+            die "interrupted by signal\n" if !$interrupted++;
+        };
+
+        my $exitcode = 1;
+        eval {
+            POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+            close($output_reader);
+            close($result_reader);
+            close($start_writer);
+            # only the job process reads the output of the other guest backups
+            for my $sibling (values $running->%*) {
+                close($_) for grep { defined } $sibling->@{qw(output result)};
+            }
+            open(STDOUT, '>&', $output_writer) or die "unable to redirect STDOUT - $!\n";
+            open(STDERR, '>&', $output_writer) or die "unable to redirect STDERR - $!\n";
+            close($output_writer);
+
+            # The job process closes the pipe without writing if it got a stop request while
+            # forking, which this process missed, so end as if it got that signal.
+            sysread($start_reader, my $start, 1) or die "interrupted by signal\n";
+            close($start_reader);
+
+            PVE::INotify::inotify_close();
+
+            eval { $self->exec_backup_task($task) };
+            my $err = $@;
+            # errors of the backup itself are handled there, so this one came up before it began
+            if ($err && $err !~ m/^interrupted by signal$/ && $task->{state} eq 'todo') {
+                chomp($err);
+                fail_task($task, $err);
+                undef $err;
+            }
+
+            my $result = { map { $_ => $task->{$_} } $guest_result_keys->@* };
+            print $result_writer Storable::nfreeze($result);
+            close($result_writer);
+
+            die $err if $err;
+            $exitcode = 0;
+        };
+        warn $@ if $@ && $@ !~ m/^interrupted by signal$/;
+        STDOUT->flush();
+        POSIX::_exit($exitcode);
+    }
+
+    POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+
+    close($output_writer);
+    close($result_writer);
+    close($start_reader);
+
+    # A stop request that came in while forking reached only the job process, so the backup
+    # process must not start then.
+    my $cancelled = $$interrupted;
+    if (!$cancelled) {
+        local $SIG{PIPE} = 'IGNORE';
+        syswrite($start_writer, '1');
+    }
+    close($start_writer);
+
+    return {
+        pid => $pid,
+        cancelled => $cancelled,
+        task => $task,
+        output => $output_reader,
+        result => $result_reader,
+        'output-buffer' => '',
+        'result-data' => '',
+    };
+}
+
+my sub forward_guest_output {
+    my ($child, $data, $final) = @_;
+
+    my $vmid = $child->{task}->{vmid};
+
+    $child->{'output-buffer'} .= $data;
+    $child->{'output-buffer'} .= "\n" if $final && length($child->{'output-buffer'});
+
+    while ($child->{'output-buffer'} =~ s/^(.*)\n//) {
+        my $line = $1;
+        # a guest backup process counts its own warnings, and that count is lost when it exits
+        PVE::RPCEnvironment::get()->{warning_count}++ if $line =~ m/^WARN: /;
+        print STDERR "$vmid: $line\n";
+    }
+}
+
+my sub close_guest_backup_pipe {
+    my ($child, $kind) = @_;
+
+    close(delete $child->{$kind});
+    forward_guest_output($child, '', 1) if $kind eq 'output';
+}
+
+my sub read_guest_backup_pipe {
+    my ($child, $kind) = @_;
+
+    my $count = sysread($child->{$kind}, my $data, 64 * 1024);
+    return if !defined($count) && ($!{EINTR} || $!{EAGAIN});
+
+    if (!$count) {
+        close_guest_backup_pipe($child, $kind);
+    } elsif ($kind eq 'output') {
+        forward_guest_output($child, $data);
+    } else {
+        $child->{'result-data'} .= $data;
+    }
+}
+
+my sub finish_guest_backup {
+    my ($child, $status) = @_;
+
+    # the backup process is gone, but helpers it started might still hold the output pipe
+    for my $kind (qw(result output)) {
+        while ($child->{$kind} && IO::Select->new($child->{$kind})->can_read(0)) {
+            read_guest_backup_pipe($child, $kind);
+        }
+        close_guest_backup_pipe($child, $kind) if $child->{$kind};
+    }
+    return if $child->{cancelled};
+
+    my $task = $child->{task};
+    my $result = eval { Storable::thaw($child->{'result-data'}) };
+    if (ref($result) eq 'HASH') {
+        $task->{$_} = $result->{$_} for keys $result->%*;
+    } else {
+        my $reason = $status & 127 ? "signal " . ($status & 127) : "exit code " . ($status >> 8);
+        fail_task($task, "backup process ended unexpectedly ($reason)");
+    }
+}
+
+my sub exec_parallel_backup_tasks {
+    my ($self, $tasklist, $concurrency) = @_;
+
+    my @queue = $tasklist->@*;
+    my $running = {};
+    my $interrupted = 0;
+    my $job_pid = $$;
+
+    # Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
+    # them again could interrupt their cleanup.
+    local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
+        sub { $interrupted = 1; };
+
+    while (scalar(@queue) || scalar(keys $running->%*)) {
+        while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
+            my $task = shift @queue;
+            my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
+            # a guest backup process signalled while it exits dies into this eval
+            POSIX::_exit(1) if $$ != $job_pid;
+            if (my $err = $@) {
+                fail_task($task, $err);
+                next;
+            }
+            $running->{ $child->{pid} } = $child;
+            unshift @queue, $task if $child->{cancelled};
+        }
+
+        last if $interrupted && !scalar(keys $running->%*);
+
+        my $handles = {};
+        for my $child (values $running->%*) {
+            $handles->{ fileno($child->{$_}) } = [$child, $_]
+                for grep { $child->{$_} } qw(output result);
+        }
+        my $select = IO::Select->new(map { $_->[0]->{ $_->[1] } } values $handles->%*);
+        for my $fh ($select->can_read(1)) {
+            read_guest_backup_pipe($handles->{ fileno($fh) }->@*);
+        }
+
+        for my $pid (keys $running->%*) {
+            next if waitpid($pid, POSIX::WNOHANG()) != $pid;
+            finish_guest_backup(delete $running->{$pid}, $?);
+        }
+    }
+
+    die "interrupted by signal\n" if $interrupted;
+}
+
 sub exec_backup {
     my ($self, $rpcenv, $authuser) = @_;
 
@@ -1490,22 +1732,27 @@ sub exec_backup {
     open my $job_start_fd, '>', \$job_start_log;
     open my $job_end_fd, '>', \$job_end_log;
 
+    my $concurrency = $self->guest_backup_concurrency();
+    $concurrency = scalar($tasklist->@*) if scalar($tasklist->@*) < $concurrency;
+
     my $starttime = time();
-    my $errcount = 0;
     eval {
 
         $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
         $self->run_hook_script('job-start', undef, $job_start_fd);
 
-        foreach my $task (@$tasklist) {
-            $self->exec_backup_task($task);
-            $errcount += 1 if $task->{state} ne 'ok';
+        if ($concurrency > 1) {
+            debugmsg('info', "running up to $concurrency guest backups at the same time");
+            exec_parallel_backup_tasks($self, $tasklist, $concurrency);
+        } else {
+            $self->exec_backup_task($_) for $tasklist->@*;
         }
 
         $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
         $self->run_hook_script('job-end', undef, $job_end_fd);
     };
     my $err = $@;
+    my $errcount = grep { $_->{state} ne 'ok' } $tasklist->@*;
 
     if ($err) {
         if ($self->{'backup-provider'}) {
diff --git a/test/Makefile b/test/Makefile
index 91761579d..bbe87a7c3 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
 	./$<
 
 .PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
 
 .PHONY: test-vzdump-guest-included
 test-vzdump-guest-included:
@@ -28,6 +28,10 @@ test-vzdump-guest-included:
 test-vzdump-new:
 	./vzdump_new_test.pl
 
+.PHONY: test-vzdump-concurrency
+test-vzdump-concurrency:
+	./vzdump_concurrency_test.pl
+
 .PHONY: test-osd
 test-osd:
 	./OSD_test.pl
diff --git a/test/vzdump_concurrency_test.pl b/test/vzdump_concurrency_test.pl
new file mode 100755
index 000000000..46d679490
--- /dev/null
+++ b/test/vzdump_concurrency_test.pl
@@ -0,0 +1,89 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib '..';
+
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump;
+
+my $node_config = {};
+my $datacenter_config = {};
+
+my $node_config_module = Test::MockModule->new('PVE::NodeConfig');
+$node_config_module->mock(load_config => sub { return $node_config; });
+
+my $vzdump_module = Test::MockModule->new('PVE::VZDump');
+$vzdump_module->mock(
+    cfs_read_file => sub {
+        my ($filename) = @_;
+        die "unexpected file '$filename'\n" if $filename ne 'datacenter.cfg';
+        return $datacenter_config;
+    },
+);
+
+my $tests = [
+    {
+        description => 'defaults keep backing up one guest at a time',
+        expected => 1,
+    },
+    {
+        description => 'datacenter default applies to the node',
+        datacenter => 4,
+        expected => 4,
+    },
+    {
+        description => 'node configuration overrides datacenter default',
+        datacenter => 4,
+        node => 2,
+        expected => 2,
+    },
+    {
+        description => 'node configuration applies without datacenter default',
+        node => 3,
+        expected => 3,
+    },
+    {
+        description => 'job limit lowers node limit',
+        node => 4,
+        job => 2,
+        expected => 2,
+    },
+    {
+        description => 'job limit cannot raise node limit',
+        node => 2,
+        job => 8,
+        expected => 2,
+    },
+    {
+        description => 'job limit does not raise the default',
+        job => 8,
+        expected => 1,
+    },
+    {
+        description => 'external backup providers back up one guest at a time',
+        node => 4,
+        job => 4,
+        provider => 1,
+        expected => 1,
+    },
+];
+
+for my $test ($tests->@*) {
+    $datacenter_config =
+        defined($test->{datacenter})
+        ? { 'backup-concurrency' => { node => $test->{datacenter} } }
+        : {};
+    $node_config = defined($test->{node}) ? { 'backup-concurrency' => $test->{node} } : {};
+
+    my $self = bless { opts => {} }, 'PVE::VZDump';
+    $self->{opts}->{concurrency} = $test->{job} if defined($test->{job});
+    $self->{'backup-provider'} = {} if $test->{provider};
+
+    is($self->guest_backup_concurrency(), $test->{expected}, $test->{description});
+}
+
+done_testing();
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (10 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
                   ` (7 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

A node ran one backup job at a time. A short job, for example an
hourly one to a nearby target, had to wait until a long nightly job
finished, possibly for hours, and failed once the lock wait time ran
out.

Jobs with the new 'overlap' option share the global lock and run at
the same time as other such jobs. Every guest backup on the node, no
matter which job it belongs to, holds one of the node's backup slots
while running, so overlapping jobs together stay within the node's
backup concurrency. When the concurrency gets lowered, slots above
the new limit keep counting until their guest backups end. Jobs
without the option keep running alone, and a job waiting for that
takes precedence over overlapping jobs starting later, so these cannot
starve it. Jobs to external backup providers always run alone.

A guest that another job is backing up is deferred until that backup
finishes, waiting at most the lock wait time. Before starting a guest
backup, the job also checks whether the guest moved to another node in
the meantime.

The global lock, the node slots and the guest locks are file locks
held by the processes doing the work, so they get released whenever
such a process ends. Jobs that do not overlap still take the global
lock exclusively, which keeps it working for the boot-time cleanup of
stale backup locks and against older versions during an upgrade.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- after lowering a limit, reuse free slots once fewer than the limit are
  occupied, even if slots above it are still in use [Fiona]; only slots
  that were used before get probed for that
- take the node slot before the guest lock, so a job waiting for a slot
  does not lock guests of other jobs for a moment
- pick up a changed datacenter default of the node limit while waiting
- warn and run alone if checking for an external backup provider fails,
  but still end on a stop request [Fiona]
- preserve the global lock error across unlocking the gate [Fiona]
- create /run/vzdump through tmpfiles.d [Fiona]
- rely on the lock wait time that the constructor sets [Fiona]
- use PVE::File [Fiona], match numbers with [0-9]

 PVE/API2/VZDump.pm |   2 +-
 PVE/VZDump.pm      | 244 +++++++++++++++++++++++++++++++++++++++++++--
 debian/tmpfiles    |   5 +-
 3 files changed, 241 insertions(+), 10 deletions(-)

diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 97b1797f8..42017cf9e 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -137,7 +137,7 @@ __PACKAGE__->register_method({
             my $vzdump = PVE::VZDump->new($cmdline, $param, $skiplist);
 
             my $LOCK_FH = eval {
-                my $lock_fh = $vzdump->getlock($upid); # only one process allowed
+                my $lock_fh = $vzdump->getlock($upid);
                 $vzdump->init_job();
                 $lock_fh;
             };
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 054665430..eef338435 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -17,6 +17,7 @@ use Time::Local;
 use PVE::Cluster qw(cfs_read_file);
 use PVE::DataCenterConfig;
 use PVE::Exception qw(raise_param_exc);
+use PVE::File;
 use PVE::HA::Config;
 use PVE::HA::Env::PVE2;
 use PVE::JSONSchema qw(get_standard_option);
@@ -47,6 +48,7 @@ PVE::JSONSchema::register_standard_option(
 my @posix_filesystems = qw(ext3 ext4 nfs nfs4 reiserfs xfs);
 
 my $lockfile = '/var/run/vzdump.lock';
+my $lockdir = '/run/vzdump';
 my $logdir = '/var/log/vzdump';
 
 my @plugins = qw();
@@ -688,6 +690,25 @@ sub new {
 
     $self->{'default-prune-backups'} = $defaults->{'prune-backups'};
 
+    $self->{exclusive} = !$opts->{overlap};
+    if (!$self->{exclusive} && $opts->{storage}) {
+        # external providers are not known to cope with other backups running at the same time
+        $self->{exclusive} = eval {
+            PVE::Storage::storage_has_feature(
+                PVE::Storage::config(), $opts->{storage}, 'backup-provider',
+            );
+        };
+        if (my $err = $@) {
+            die $err if $err =~ m/^interrupted by signal$/;
+            debugmsg(
+                'warn',
+                "unable to check if storage '$opts->{storage}' uses a backup provider, so not"
+                    . " running at the same time as other backups - $err",
+            );
+            $self->{exclusive} = 1;
+        }
+    }
+
     return $self;
 }
 
@@ -787,14 +808,32 @@ sub getlock {
     my $maxwait = $self->{opts}->{lockwait};
 
     die "missing UPID" if !$upid; # should not happen
+    $self->{upid} = $upid;
 
     my $SERVER_FLCK;
     if (!open($SERVER_FLCK, '>>', "$lockfile")) {
         debugmsg('err', "can't open lock on file '$lockfile' - $!", undef, 1);
         die "can't open lock on file '$lockfile' - $!";
     }
+    open(my $gate, '>>', "$lockdir/gate.lock")
+        or die "can't open lock on file '$lockdir/gate.lock' - $!\n";
 
-    if (!flock($SERVER_FLCK, LOCK_EX | LOCK_NB)) {
+    # Jobs that may overlap share the global lock. A job waiting to get it exclusively holds the
+    # gate meanwhile, so overlapping jobs starting later cannot keep it waiting forever.
+    my $mode = $self->{exclusive} ? LOCK_EX : LOCK_SH;
+    my $acquire = sub {
+        my ($flags) = @_;
+
+        flock($gate, $mode | $flags) or return (0, "$!");
+        my $locked = flock($SERVER_FLCK, $mode | $flags);
+        my $err = "$!";
+        flock($gate, LOCK_UN);
+
+        return ($locked, $err);
+    };
+
+    my ($locked) = $acquire->(LOCK_NB);
+    if (!$locked) {
         if (!$maxwait) {
             debugmsg('err', "can't acquire lock '$lockfile' (wait = 0)", undef, 1);
             die "can't acquire lock '$lockfile' (wait = 0)";
@@ -806,8 +845,8 @@ sub getlock {
 
             local $SIG{ALRM} = sub { alarm(0); die "got timeout\n"; };
 
-            if (!flock($SERVER_FLCK, LOCK_EX)) {
-                my $err = $!;
+            ($locked, my $err) = $acquire->(0);
+            if (!$locked) {
                 close($SERVER_FLCK);
                 alarm(0);
                 die "$err\n";
@@ -1664,12 +1703,182 @@ my sub finish_guest_backup {
     }
 }
 
+my sub try_lock_file {
+    my ($path) = @_;
+
+    open(my $fh, '>>', $path) or die "unable to open lock file '$path' - $!\n";
+    return $fh if flock($fh, LOCK_EX | LOCK_NB);
+
+    close($fh);
+    return undef;
+}
+
+my sub try_guest_lock {
+    my ($self, $vmid) = @_;
+
+    my $fh = try_lock_file("$lockdir/guest-$vmid.lock") or return undef;
+    truncate($fh, 0);
+    print $fh "$self->{upid}\n";
+    $fh->flush();
+
+    return $fh;
+}
+
+# Guest backups in slots above a lowered limit keep counting against it until they end.
+my sub try_slot {
+    my ($prefix, $limit) = @_;
+
+    # Otherwise, processes counting the slots in use at the same time could all take the last one.
+    my $select_lock = "$lockdir/select-slot.lock";
+    open(my $selecting, '>>', $select_lock)
+        or die "unable to open lock file '$select_lock' - $!\n";
+    flock($selecting, LOCK_EX) or die "unable to lock '$select_lock' - $!\n";
+
+    # only slots that were used before have a lock file
+    my ($busy, $free, $exists) = (0, undef, {});
+    for my $path (glob("$prefix-*.lock")) {
+        my ($slot) = $path =~ m/^\Q$prefix\E-([0-9]+)\.lock$/ or next;
+        $exists->{$slot} = 1;
+        # rebuilt from the number, as names from glob() are tainted and vzdump runs with -T
+        my $fh = try_lock_file("$prefix-$slot.lock");
+        if (!$fh) {
+            $busy++;
+        } elsif ($slot <= $limit && !$free) {
+            $free = $fh;
+        } else {
+            close($fh);
+        }
+    }
+
+    if ($busy >= $limit) {
+        close($free) if $free;
+        return undef;
+    }
+    return $free if $free;
+
+    # fewer slots are busy than the limit allows, so the first one without lock file is within it
+    my $slot = 1;
+    $slot++ while $exists->{$slot};
+    return try_lock_file("$prefix-$slot.lock");
+}
+
+my sub try_node_slot {
+    return try_slot("$lockdir/slot", node_backup_concurrency());
+}
+
+my sub log_wait {
+    my ($self, $reason) = @_;
+
+    return if $self->{'waiting-for'}->{$reason}++;
+    debugmsg('info', "waiting for $reason");
+}
+
+# Config files from pmxcfs are cached until cfs_update(), refresh them to pick up changed limits.
+my sub refresh_cluster_config {
+    my ($self) = @_;
+
+    return if time() - ($self->{'cluster-config-refreshed'} // 0) < 5;
+
+    PVE::Cluster::cfs_update();
+    $self->{'cluster-config-refreshed'} = time();
+}
+
+my sub guest_moved_to {
+    my ($vmid) = @_;
+
+    PVE::Cluster::cfs_update();
+    my $node = PVE::Cluster::get_vmlist()->{ids}->{$vmid}->{node};
+
+    return defined($node) && $node ne PVE::INotify::nodename() ? $node : undef;
+}
+
+my sub claim_next_guest {
+    my ($self, $queue, $deferred) = @_;
+
+    refresh_cluster_config($self);
+
+    my $maxwait = $self->{opts}->{lockwait} * 60;
+
+    my $slot;
+    for (my $i = 0; $i < scalar($queue->@*); $i++) {
+        my $task = $queue->[$i];
+        my $vmid = $task->{vmid};
+
+        # first, so that a job waiting for a slot does not lock guests of other jobs for a moment
+        $slot //= try_node_slot();
+        if (!$slot) {
+            log_wait($self, "a free backup slot on the node");
+            return;
+        }
+
+        my $guest_lock = try_guest_lock($self, $vmid);
+        if (!$guest_lock) {
+            my $holder = PVE::File::file_read_first_line("$lockdir/guest-$vmid.lock") // '';
+            if (!defined($deferred->{$vmid})) {
+                $deferred->{$vmid} = time();
+                debugmsg('info', "VM $vmid is being backed up by '$holder', deferring");
+            } elsif (time() - $deferred->{$vmid} >= $maxwait) {
+                splice($queue->@*, $i--, 1);
+                skip_task($task, "still being backed up by '$holder'");
+            }
+            next;
+        }
+        delete $deferred->{$vmid};
+
+        if (my $node = guest_moved_to($vmid)) {
+            splice($queue->@*, $i--, 1);
+            skip_task($task, "guest moved to node '$node'");
+            next;
+        }
+        delete $self->{'waiting-for'};
+
+        splice($queue->@*, $i--, 1);
+        return ($task, [$guest_lock, $slot]);
+    }
+
+    return;
+}
+
+my sub skip_after_error {
+    my ($queue, $err) = @_;
+
+    return if $err =~ m/interrupted by signal$/;
+
+    chomp(my $reason = $err);
+    skip_task($_, "not started due to error - $reason") for splice($queue->@*);
+}
+
+my sub exec_serial_backup_tasks {
+    my ($self, $tasklist) = @_;
+
+    my @queue = $tasklist->@*;
+    my $deferred = {};
+
+    eval {
+        while (scalar(@queue)) {
+            # $locks keeps the guest lock and the slots taken until the guest backup is done
+            my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+            if ($task) {
+                $self->exec_backup_task($task);
+            } elsif (scalar(@queue)) {
+                sleep(1);
+            }
+        }
+    };
+    if (my $err = $@) {
+        skip_after_error(\@queue, $err);
+        die $err;
+    }
+}
+
 my sub exec_parallel_backup_tasks {
     my ($self, $tasklist, $concurrency) = @_;
 
     my @queue = $tasklist->@*;
+    my $deferred = {};
     my $running = {};
     my $interrupted = 0;
+    my $error;
     my $job_pid = $$;
 
     # Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
@@ -1678,8 +1887,17 @@ my sub exec_parallel_backup_tasks {
         sub { $interrupted = 1; };
 
     while (scalar(@queue) || scalar(keys $running->%*)) {
-        while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
-            my $task = shift @queue;
+        while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
+            my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+            if (my $err = $@) {
+                # let running guest backups finish, they would get aborted otherwise
+                debugmsg('err', "unable to start further guest backups - $err");
+                $error = $err;
+                last;
+            }
+            last if !$task;
+
+            # the backup process keeps holding the locks through its inherited copies of $locks
             my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
             # a guest backup process signalled while it exits dies into this eval
             POSIX::_exit(1) if $$ != $job_pid;
@@ -1691,7 +1909,14 @@ my sub exec_parallel_backup_tasks {
             unshift @queue, $task if $child->{cancelled};
         }
 
-        last if $interrupted && !scalar(keys $running->%*);
+        if ($interrupted || defined($error)) {
+            last if !scalar(keys $running->%*);
+        }
+
+        if (!scalar(keys $running->%*)) {
+            sleep(1);
+            next;
+        }
 
         my $handles = {};
         for my $child (values $running->%*) {
@@ -1710,6 +1935,11 @@ my sub exec_parallel_backup_tasks {
     }
 
     die "interrupted by signal\n" if $interrupted;
+
+    if (defined($error)) {
+        skip_after_error(\@queue, $error);
+        die $error;
+    }
 }
 
 sub exec_backup {
@@ -1745,7 +1975,7 @@ sub exec_backup {
             debugmsg('info', "running up to $concurrency guest backups at the same time");
             exec_parallel_backup_tasks($self, $tasklist, $concurrency);
         } else {
-            $self->exec_backup_task($_) for $tasklist->@*;
+            exec_serial_backup_tasks($self, $tasklist);
         }
 
         $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
diff --git a/debian/tmpfiles b/debian/tmpfiles
index 1263300fb..1703c789d 100644
--- a/debian/tmpfiles
+++ b/debian/tmpfiles
@@ -1,2 +1,3 @@
-#Type Path     Mode User Group     Age Argument
-d     /run/pve 0750 root www-data  -   -
+#Type Path        Mode User Group     Age Argument
+d     /run/pve    0750 root www-data  -   -
+d     /run/vzdump 0700 root root      -   -
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (11 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
                   ` (6 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Expose the per-node backup concurrency in the datacenter options and
its override in the node options, so admins can let nodes back up
several guests at the same time without editing configuration files.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- use title case for the 'Per Node' label

 www/manager6/dc/OptionView.js        | 14 ++++++++++++++
 www/manager6/node/NodeOptionsView.js |  8 ++++++++
 2 files changed, 22 insertions(+)

diff --git a/www/manager6/dc/OptionView.js b/www/manager6/dc/OptionView.js
index 19e6d54ea..4a7e9692a 100644
--- a/www/manager6/dc/OptionView.js
+++ b/www/manager6/dc/OptionView.js
@@ -360,6 +360,20 @@ Ext.define('PVE.dc.OptionView', {
             minValue: 1,
             maxValue: 64, // arbitrary but generous limit as limits are good
         });
+        me.add_inputpanel_row('backup-concurrency', gettext('Backup Concurrency'), {
+            renderer: PVE.Utils.render_as_property_string,
+            url: '/api2/extjs/cluster/options',
+            items: [
+                {
+                    xtype: 'proxmoxintegerfield',
+                    name: 'node',
+                    fieldLabel: gettext('Per Node'),
+                    emptyText: '1',
+                    minValue: 1,
+                    submitValue: true,
+                },
+            ],
+        });
         me.add_inputpanel_row('next-id', gettext('Next Free Guest ID'), {
             renderer: PVE.Utils.render_as_property_string,
             url: '/api2/extjs/cluster/options',
diff --git a/www/manager6/node/NodeOptionsView.js b/www/manager6/node/NodeOptionsView.js
index 5e9ce7915..2dcda66bc 100644
--- a/www/manager6/node/NodeOptionsView.js
+++ b/www/manager6/node/NodeOptionsView.js
@@ -95,5 +95,13 @@ Ext.define('Proxmox.node.NodeOptionsView', {
             onlineHelp: 'qm_memory',
             renderer: (value) => (value !== undefined ? `${value}%` : gettext('Default (80%)')),
         },
+        {
+            xtype: 'integer',
+            name: 'backup-concurrency',
+            text: gettext('Backup concurrency'),
+            minValue: 1,
+            deleteEmpty: true,
+            renderer: (value) => value ?? gettext('Datacenter default'),
+        },
     ],
 });
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (12 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
                   ` (5 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Add the new job options to the advanced tab. An empty concurrency
field uses the fallback from the node's vzdump.conf like the other
options there. An unchecked overlap box is stored as off, like the
fleecing one, as deleting it would let an overlap setting in
vzdump.conf apply while the box shows otherwise.

The bandwidth limit applies to each guest backup on its own, so with
several guests backed up at the same time their total can be higher.
Say so in its description.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- the description of 'Concurrency' says what an empty field means

 www/manager6/panel/BackupAdvancedOptions.js | 40 ++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf7..49fefb744 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -154,7 +154,45 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
             endFlex: 2,
             endColumn: {
                 xtype: 'displayfield',
-                value: `${gettext('Limit I/O bandwidth.')} ${Ext.String.format(gettext('Schema default: {0}'), 0)}`,
+                value: `${gettext('Limit I/O bandwidth of each guest backup.')} ${Ext.String.format(gettext('Schema default: {0}'), 0)}`,
+            },
+        },
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxintegerfield',
+                name: 'concurrency',
+                minValue: 1,
+                fieldLabel: gettext('Concurrency'),
+                fieldStyle: 'text-align: right',
+                emptyText: gettext('Fallback'),
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: gettext(
+                    'Guests backed up at the same time on each node. Defaults to, and is limited by, the backup concurrency of the node.',
+                ),
+            },
+        },
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxcheckbox',
+                fieldLabel: gettext('Overlap'),
+                name: 'overlap',
+                uncheckedValue: 0,
+                value: 0,
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: gettext(
+                    'Run at the same time as other jobs that allow it. Job phases of hook scripts can then overlap.',
+                ),
             },
         },
         {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 15/20] ui: backup now: add overlap option
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (13 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
                   ` (4 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

An ad-hoc backup runs alone unless the node's vzdump.conf enables
'overlap'. It thus waits for a running job, even if that job allows
overlapping and the node has free backup slots.

Add the option to the dialog, preset from the node's default, so a
single guest can be backed up while a long overlapping job runs. For a
node that does not know the option yet, as told by its defaults, the
box is disabled and the option left out, as the node would reject the
request otherwise.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

new in v2

 www/manager6/window/Backup.js | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
index 0caada651..8adbf938c 100644
--- a/www/manager6/window/Backup.js
+++ b/www/manager6/window/Backup.js
@@ -123,6 +123,17 @@ Ext.define('PVE.window.Backup', {
 
         let initialDefaults = false;
 
+        let overlapCheckbox = Ext.create('Proxmox.form.Checkbox', {
+            name: 'overlap',
+            checked: false,
+            uncheckedValue: 0,
+            fieldLabel: gettext('Overlap'),
+            autoEl: {
+                tag: 'div',
+                'data-qtip': gettext('Run at the same time as other backups that allow it'),
+            },
+        });
+
         var storagesel = Ext.create('PVE.form.StorageSelector', {
             nodename: me.nodename,
             name: 'storage',
@@ -187,6 +198,11 @@ Ext.define('PVE.window.Backup', {
                                     mailtoField.setValue(mailto);
                                 }
                             }
+                            // nodes that do not know the option yet reject it
+                            overlapCheckbox.setDisabled(data.overlap === undefined);
+                            if (!initialDefaults) {
+                                overlapCheckbox.setValue(!!data.overlap);
+                            }
                             if (!initialDefaults && data.mode !== undefined) {
                                 modeSelector.setValue(data.mode);
                             }
@@ -249,7 +265,13 @@ Ext.define('PVE.window.Backup', {
         me.formPanel = Ext.create('Proxmox.panel.InputPanel', {
             bodyPadding: 10,
             border: false,
-            column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
+            column1: [
+                storagesel,
+                modeSelector,
+                protectedCheckbox,
+                overlapCheckbox,
+                pbsChangeDetection,
+            ],
             column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
             columnB: [
                 {
@@ -316,6 +338,10 @@ Ext.define('PVE.window.Backup', {
                     remove: values.remove,
                 };
 
+                if (values.overlap !== undefined) {
+                    params.overlap = values.overlap;
+                }
+
                 if (values.mailto) {
                     params.mailto = values.mailto;
                 }
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (14 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
                   ` (3 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Each node runs its part of a backup job on its own, so shared storages
and backup servers get as many guest backups at the same time as all
nodes together allow.

With the new cluster-wide limit in the datacenter configuration, every
guest backup also needs a slot in a table on the cluster file system
before it starts. The table is only changed under a cluster lock, and
waiting guest backups get their slots in the order they queued up. A
job has at most one guest waiting for a slot, so waiting does not tie
up the whole node, and waits at most the lock wait time in total.

A guest backup gives back its slot when it is done. If that fails, for
example because the backup process got killed, the scheduler of the
node removes the entry once that guest backup is no longer running.
Slots of offline nodes stay taken, as their guest backups might still
run and keep the shared systems busy. A guest that waits for a slot
names these nodes in the task log and, if it gives up, in its skip
reason. Entries of nodes removed from the cluster are dropped. Granting
slots needs quorum; guest backups that already run continue without
it.

A table that cannot be parsed gets replaced by an empty one, with a
warning that includes its content. Guest backups that run at that
point then no longer count against the limits until they end, which is
better than skipping every guest backup that needs a slot until an
admin notices. The scheduler of each node does this within a minute, so
it also happens without waiting backups. A table that cannot be read at
all, for example because of a pmxcfs problem, stops granting instead,
as that is likely to pass.

The limit is only enforced once all other online nodes run a version
that takes part, as told by the version info each node broadcasts.
Older nodes would not count their guest backups, and a node that gets
downgraded broadcasts its older version again.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- check node versions with the helper from pve-cluster [Fiona]; the
  pvecfg_min_version() of qemu-server used before is gone since
  qemu-server 9.2.15, which failed jobs with a cluster-wide or shared
  storage limit on clusters with more than one node online
- move the slot table out of priv/ [Fiona]
- if a guest lock file exists but cannot be opened, fail the cleanup
  instead of dropping the entry of the guest [Fiona]
- log a slot request error again if it changes [Fiona]
- include up to 1 KiB of the table in parse errors [Fiona]
- replace an unparsable slot table with a warning, instead of skipping
  every guest backup that needs a slot until an admin repairs it
- rely on the lock wait time that the constructor sets [Fiona]
- use v5.36, signatures and PVE::File, also in the test, and no
  multi-line postfix conditions [Fiona]
- pick up changed limits while waiting, like in patch 12
- reject non-ASCII digits and trailing newlines in slot table counts
- name offline nodes that hold slots in the wait message and in the
  skip reason of a guest that waits for a slot
- retry a failed slot request only every 30 seconds, as each attempt
  can block and log for 10 seconds, for example without quorum
- $min_version is now 9.2.31

NOTE: $min_version assumes that this lands with pve-manager 9.2.31;
adapt it to the first release that contains it.

 PVE/Makefile                      |   2 +-
 PVE/Service/pvescheduler.pm       |   4 +
 PVE/VZDump.pm                     | 189 +++++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm        | 247 ++++++++++++++++++++++++++++++
 PVE/VZDump/Makefile               |  15 ++
 test/Makefile                     |   6 +-
 test/vzdump_cluster_slots_test.pl | 205 +++++++++++++++++++++++++
 7 files changed, 655 insertions(+), 13 deletions(-)
 create mode 100644 PVE/VZDump/ClusterSlots.pm
 create mode 100644 PVE/VZDump/Makefile
 create mode 100755 test/vzdump_cluster_slots_test.pl

diff --git a/PVE/Makefile b/PVE/Makefile
index efcb250d0..ffdd3a8fd 100644
--- a/PVE/Makefile
+++ b/PVE/Makefile
@@ -1,6 +1,6 @@
 include ../defines.mk
 
-SUBDIRS=API2 Status CLI Service Ceph Jobs
+SUBDIRS=API2 Status CLI Service Ceph Jobs VZDump
 
 PERLSOURCE = 			\
 	API2.pm			\
diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm
index be33db6ed..2af72328f 100755
--- a/PVE/Service/pvescheduler.pm
+++ b/PVE/Service/pvescheduler.pm
@@ -7,6 +7,7 @@ use POSIX qw(WNOHANG SIGCHLD SIGHUP SIG_BLOCK SIG_SETMASK);
 
 use PVE::Jobs;
 use PVE::SafeSyslog;
+use PVE::VZDump;
 
 use PVE::API2::Replication;
 
@@ -125,6 +126,9 @@ sub run {
         $fork->(
             'jobs',
             sub {
+                eval { PVE::VZDump::cleanup_cluster_slots() };
+                syslog('err', "unable to clean up backup slots - $@") if $@;
+
                 PVE::Jobs::run_jobs($first_run);
             },
         );
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index eef338435..867fe0b8f 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -27,6 +27,7 @@ use PVE::RPCEnvironment;
 use PVE::Storage;
 use PVE::UPID;
 use PVE::VZDump::Common;
+use PVE::VZDump::ClusterSlots;
 use PVE::VZDump::Plugin;
 use PVE::Tools qw(extract_param split_list);
 use PVE::API2Tools;
@@ -1724,6 +1725,22 @@ my sub try_guest_lock {
     return $fh;
 }
 
+my sub guest_backup_running {
+    my ($vmid, $upid) = @_;
+
+    my $path = "$lockdir/guest-$vmid.lock";
+    open(my $fh, '<', $path) or do {
+        die "unable to open '$path' - $!\n" if !$!{ENOENT};
+        return 0;
+    };
+    return 0 if flock($fh, LOCK_SH | LOCK_NB);
+
+    my $holder = <$fh> // '';
+    chomp($holder);
+
+    return $holder eq $upid;
+}
+
 # Guest backups in slots above a lowered limit keep counting against it until they end.
 my sub try_slot {
     my ($prefix, $limit) = @_;
@@ -1795,8 +1812,6 @@ my sub guest_moved_to {
 my sub claim_next_guest {
     my ($self, $queue, $deferred) = @_;
 
-    refresh_cluster_config($self);
-
     my $maxwait = $self->{opts}->{lockwait} * 60;
 
     my $slot;
@@ -1848,24 +1863,165 @@ my sub skip_after_error {
     skip_task($_, "not started due to error - $reason") for splice($queue->@*);
 }
 
+my sub cluster_slot_resources {
+    my ($self) = @_;
+
+    my $limits = PVE::VZDump::ClusterSlots::limits();
+    return undef if !scalar(keys $limits->%*);
+
+    if (!PVE::VZDump::ClusterSlots::supported_by_all_nodes()) {
+        if (!$self->{'cluster-slots-unsupported'}++) {
+            debugmsg(
+                'warn',
+                "cluster-wide backup limits are not enforced - not all online nodes support them",
+            );
+        }
+        return undef;
+    }
+
+    return ['cluster'];
+}
+
+my sub release_cluster_slots {
+    my ($key) = @_;
+
+    return if !defined($key);
+
+    eval { PVE::VZDump::ClusterSlots::release($key) };
+    if (my $err = $@) {
+        die $err if $err =~ m/interrupted by signal$/; # also if wrapped by the cluster lock
+        debugmsg('warn', "unable to release cluster-wide backup slot - $err");
+    }
+}
+
+# Tells why slots may stay taken, with the name needed to release them.
+my sub offline_slot_holders {
+    my ($resources) = @_;
+
+    my $nodes = eval { PVE::VZDump::ClusterSlots::offline_holders($resources) };
+    die $@ if $@ =~ m/interrupted by signal$/;
+    return '' if !scalar(($nodes // [])->@*);
+
+    my $list = join(', ', map { "'$_'" } $nodes->@*);
+    return scalar($nodes->@*) > 1
+        ? ", offline nodes $list hold slots"
+        : ", offline node $list holds slots";
+}
+
+sub cleanup_cluster_slots {
+    PVE::VZDump::ClusterSlots::cleanup_local_entries(\&guest_backup_running);
+}
+
+# A claimed guest that needs cluster-wide slots waits for them as pending guest, only one at a time.
+my sub next_guest_to_start {
+    my ($self, $queue, $schedule) = @_;
+
+    refresh_cluster_config($self);
+
+    my $pending = $schedule->{pending};
+    if (!$pending) {
+        my ($task, $locks) = claim_next_guest($self, $queue, $schedule->{deferred}) or return;
+        my $resources = eval { cluster_slot_resources($self) };
+        if (my $err = $@) {
+            unshift $queue->@*, $task;
+            die $err;
+        }
+        return ($task, $locks) if !$resources;
+
+        $pending = $schedule->{pending} = {
+            task => $task,
+            locks => $locks,
+            resources => $resources,
+            since => time(),
+            refreshed => time(),
+        };
+    }
+
+    my $task = $pending->{task};
+    # A failed request can block and log for seconds, for example without quorum, so retry rarely.
+    if (time() >= ($pending->{'retry-at'} // 0)) {
+        my $granted = eval {
+            $pending->{key} //= PVE::VZDump::ClusterSlots::request(
+                $self->{upid}, $task->{vmid}, $pending->{resources},
+            );
+            if (time() - $pending->{refreshed} >= 30) {
+                PVE::VZDump::ClusterSlots::refresh();
+                $pending->{refreshed} = time();
+            }
+            PVE::VZDump::ClusterSlots::granted($pending->{key});
+        };
+        if (my $err = $@) {
+            die $err if $err =~ m/interrupted by signal$/;
+            chomp($err);
+            if (($pending->{'request-error'} // '') ne $err) {
+                debugmsg('warn', "unable to request cluster-wide backup slot - $err");
+            }
+            $pending->{'request-error'} = $err;
+            $pending->{'retry-at'} = time() + 30;
+        } elsif (!defined($granted)) {
+            $pending->{key} = undef; # vanished, for example if released by an admin
+        } elsif ($granted) {
+            delete $schedule->{pending};
+            $schedule->{'cluster-wait'} += time() - $pending->{since};
+            if (my $node = guest_moved_to($task->{vmid})) {
+                release_cluster_slots($pending->{key});
+                skip_task($task, "guest moved to node '$node'");
+                return;
+            }
+            return ($task, $pending->{locks}, $pending->{key});
+        } else {
+            delete $pending->{'request-error'};
+        }
+    }
+
+    my $waited = $schedule->{'cluster-wait'} + time() - $pending->{since};
+    my $maxwait = $self->{opts}->{lockwait} * 60;
+    if ($waited >= $maxwait) {
+        release_cluster_slots($pending->{key});
+        delete $schedule->{pending};
+        $schedule->{'cluster-wait'} = $waited;
+        my $reason =
+            'timeout waiting for cluster-wide backup slot' . offline_slot_holders(['cluster']);
+        $reason .= " - $pending->{'request-error'}" if defined($pending->{'request-error'});
+        skip_task($task, $reason);
+    } elsif (!$pending->{logged}++) {
+        my $offline = offline_slot_holders(['cluster']);
+        debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot$offline");
+    }
+
+    return;
+}
+
+my sub requeue_pending_guest {
+    my ($queue, $schedule) = @_;
+
+    my $pending = delete $schedule->{pending} or return;
+    release_cluster_slots($pending->{key});
+    unshift $queue->@*, $pending->{task};
+}
+
 my sub exec_serial_backup_tasks {
     my ($self, $tasklist) = @_;
 
     my @queue = $tasklist->@*;
-    my $deferred = {};
+    my $schedule = { deferred => {}, 'cluster-wait' => 0 };
 
     eval {
-        while (scalar(@queue)) {
+        while (scalar(@queue) || $schedule->{pending}) {
             # $locks keeps the guest lock and the slots taken until the guest backup is done
-            my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+            my ($task, $locks, $slot_key) = next_guest_to_start($self, \@queue, $schedule);
             if ($task) {
-                $self->exec_backup_task($task);
-            } elsif (scalar(@queue)) {
+                eval { $self->exec_backup_task($task) };
+                my $err = $@;
+                release_cluster_slots($slot_key);
+                die $err if $err;
+            } else {
                 sleep(1);
             }
         }
     };
     if (my $err = $@) {
+        requeue_pending_guest(\@queue, $schedule);
         skip_after_error(\@queue, $err);
         die $err;
     }
@@ -1875,7 +2031,7 @@ my sub exec_parallel_backup_tasks {
     my ($self, $tasklist, $concurrency) = @_;
 
     my @queue = $tasklist->@*;
-    my $deferred = {};
+    my $schedule = { deferred => {}, 'cluster-wait' => 0 };
     my $running = {};
     my $interrupted = 0;
     my $error;
@@ -1886,9 +2042,9 @@ my sub exec_parallel_backup_tasks {
     local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
         sub { $interrupted = 1; };
 
-    while (scalar(@queue) || scalar(keys $running->%*)) {
+    while (scalar(@queue) || $schedule->{pending} || scalar(keys $running->%*)) {
         while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
-            my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+            my ($task, $locks, $slot_key) = eval { next_guest_to_start($self, \@queue, $schedule) };
             if (my $err = $@) {
                 # let running guest backups finish, they would get aborted otherwise
                 debugmsg('err', "unable to start further guest backups - $err");
@@ -1897,19 +2053,28 @@ my sub exec_parallel_backup_tasks {
             }
             last if !$task;
 
+            if ($interrupted) { # getting the slots can take a while
+                release_cluster_slots($slot_key);
+                unshift @queue, $task;
+                last;
+            }
+
             # the backup process keeps holding the locks through its inherited copies of $locks
             my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
             # a guest backup process signalled while it exits dies into this eval
             POSIX::_exit(1) if $$ != $job_pid;
             if (my $err = $@) {
                 fail_task($task, $err);
+                release_cluster_slots($slot_key);
                 next;
             }
+            $child->{'slot-key'} = $slot_key;
             $running->{ $child->{pid} } = $child;
             unshift @queue, $task if $child->{cancelled};
         }
 
         if ($interrupted || defined($error)) {
+            requeue_pending_guest(\@queue, $schedule);
             last if !scalar(keys $running->%*);
         }
 
@@ -1930,7 +2095,9 @@ my sub exec_parallel_backup_tasks {
 
         for my $pid (keys $running->%*) {
             next if waitpid($pid, POSIX::WNOHANG()) != $pid;
-            finish_guest_backup(delete $running->{$pid}, $?);
+            my $child = delete $running->{$pid};
+            finish_guest_backup($child, $?);
+            release_cluster_slots($child->{'slot-key'});
         }
     }
 
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
new file mode 100644
index 000000000..7ba47a0eb
--- /dev/null
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -0,0 +1,247 @@
+package PVE::VZDump::ClusterSlots;
+
+# Cluster-wide limits for guest backups running at the same time.
+#
+# Each guest backup that counts against such a limit has an entry in a shared table on pmxcfs. It
+# starts out as waiter and becomes holder once all limits it counts against have room. Entries of
+# offline nodes stay, as their guest backups might still be running.
+
+use v5.36;
+
+use JSON;
+
+use PVE::Cluster qw(cfs_read_file);
+use PVE::File;
+use PVE::INotify;
+use PVE::RPCEnvironment;
+
+my $state_file = '/etc/pve/backup-slots.json';
+
+# first release of pve-manager that takes part in cluster-wide limits
+my $min_version = [9, 2, 31];
+
+sub limits() {
+    my $limits = {};
+
+    my $dc_conf = cfs_read_file('datacenter.cfg');
+    if (defined(my $cluster = $dc_conf->{'backup-concurrency'}->{cluster})) {
+        $limits->{cluster} = $cluster;
+    }
+
+    return $limits;
+}
+
+my sub get_members() {
+    my $members = PVE::Cluster::get_members();
+    return $members if $members && scalar(keys $members->%*);
+
+    # entries of other nodes would get dropped as if these left the cluster
+    die "unable to get cluster members\n" if -e '/etc/pve/corosync.conf';
+
+    return { PVE::INotify::nodename() => { online => 1 } };
+}
+
+# Offline nodes cannot start guest backups, and their entries stay anyway.
+sub supported_by_all_nodes() {
+    my $members = get_members();
+    my $nodename = PVE::INotify::nodename();
+
+    for my $node (keys $members->%*) {
+        next if $node eq $nodename || !$members->{$node}->{online};
+        return 0 if !PVE::Cluster::node_pvecfg_version_at_least($node, $min_version->@*);
+    }
+    return 1;
+}
+
+# Checks a copy, so that numbers do not get written back as strings.
+my sub is_count($value) {
+    return defined($value) && !ref($value) && $value =~ m/^[0-9]+\z/;
+}
+
+my sub valid_entry($entry) {
+    return
+        ref($entry) eq 'HASH'
+        && (grep { !ref($entry->{$_}) && length($entry->{$_} // '') } qw(node upid)) == 2
+        && is_count($entry->{vmid})
+        && is_count($entry->{seq})
+        && ref($entry->{resources}) eq 'ARRAY'
+        && !grep { ref($_) || !length($_ // '') } $entry->{resources}->@*;
+}
+
+sub parse_state($raw) {
+    my $content = length($raw) > 1024 ? substr($raw, 0, 1024) . '...' : $raw;
+
+    my $state = eval { decode_json($raw) };
+    die "invalid JSON '$content' - $@" if $@;
+    if (
+        ref($state) ne 'HASH'
+        || (grep { ref($state->{$_}) ne 'HASH' } qw(holders waiters))
+        || !is_count($state->{seq})
+        || grep { !valid_entry($_) } map { values $state->{$_}->%* } qw(holders waiters)
+    ) {
+        die "'$content' is not a backup slot table\n";
+    }
+
+    return $state;
+}
+
+# Fails if the file cannot be read, as that is likely transient and writing back a table that lacks
+# holders would let more guest backups start than the limits allow. With $reset_invalid, a table
+# that cannot be parsed gets replaced though, as otherwise every guest backup that needs a slot gets
+# skipped until an admin repairs it.
+sub read_state($reset_invalid = 0) {
+    open(my $fh, '<', $state_file) or do {
+        die "unable to open '$state_file' - $!\n" if !$!{ENOENT};
+        return { holders => {}, waiters => {}, seq => 0 };
+    };
+    my $raw = PVE::File::safe_read_from($fh, 0, 0, $state_file);
+    my $state = eval { parse_state($raw) };
+    if (my $err = $@) {
+        die "unable to parse '$state_file' - $err" if !$reset_invalid;
+        chomp($err);
+        my $msg = "replacing unparsable '$state_file', guest backups that run now do not count"
+            . " against cluster-wide limits until they end - $err";
+        PVE::RPCEnvironment::get()->warn($msg);
+        return ({ holders => {}, waiters => {}, seq => 0 }, 1);
+    }
+
+    return $state;
+}
+
+my sub used_slots($state) {
+    my $used = {};
+    for my $holder (values $state->{holders}->%*) {
+        $used->{$_}++ for $holder->{resources}->@*;
+    }
+    return $used;
+}
+
+my sub remove_entries($state, $match) {
+    my $count = 0;
+    for my $entries ($state->@{qw(holders waiters)}) {
+        for my $key (keys $entries->%*) {
+            next if !$match->($entries->{$key});
+            delete $entries->{$key};
+            $count++;
+        }
+    }
+    return $count;
+}
+
+# Moves waiters that fit into all limits they count against to the holders, oldest first. Only
+# resources with a limit matter, the others are tracked for information. Waiters of offline nodes
+# neither get granted nor hold up others.
+sub grant_waiters($state, $limits, $members) {
+    remove_entries($state, sub($entry) { !$members->{ $entry->{node} } });
+
+    my $used = used_slots($state);
+
+    my $waiters = $state->{waiters};
+    for my $key (sort { $waiters->{$a}->{seq} <=> $waiters->{$b}->{seq} } keys $waiters->%*) {
+        my $waiter = $waiters->{$key};
+        next if !$members->{ $waiter->{node} }->{online};
+
+        my $limited = [grep { defined($limits->{$_}) } $waiter->{resources}->@*];
+        next if grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+
+        $used->{$_}++ for $waiter->{resources}->@*;
+        $state->{holders}->{$key} = delete $waiters->{$key};
+        $state->{holders}->{$key}->{granted} = time();
+    }
+}
+
+my sub update_state($code) {
+    my $res = PVE::Cluster::cfs_lock_domain(
+        'backup-slots',
+        10,
+        sub {
+            my $json = JSON->new->canonical();
+
+            my ($state, $replaced) = read_state(1);
+            my $previous = $replaced ? '' : $json->encode($state);
+
+            my $res = $code->($state);
+            grant_waiters($state, limits(), get_members());
+
+            my $raw = $json->encode($state);
+            PVE::File::file_set_contents($state_file, $raw) if $raw ne $previous;
+
+            return $res;
+        },
+    );
+    die $@ if $@;
+
+    return $res;
+}
+
+sub request($upid, $vmid, $resources) {
+    my $node = PVE::INotify::nodename();
+    my $key = "$node/$vmid/$upid";
+
+    update_state(sub($state) {
+        return if $state->{holders}->{$key} || $state->{waiters}->{$key};
+
+        $state->{waiters}->{$key} = {
+            node => $node,
+            upid => $upid,
+            vmid => $vmid,
+            resources => $resources,
+            seq => ++$state->{seq},
+            queued => time(),
+        };
+    });
+
+    return $key;
+}
+
+# Returns 1 if granted, 0 if still waiting and undef if the request vanished.
+sub granted($key) {
+    my $state = read_state();
+    return 1 if $state->{holders}->{$key};
+    return 0 if $state->{waiters}->{$key};
+    return undef;
+}
+
+# Offline nodes that hold slots of the given resources, as their guest backups might be gone.
+sub offline_holders($resources) {
+    my $members = get_members();
+    my %wanted = map { $_ => 1 } $resources->@*;
+
+    my $nodes = {};
+    for my $holder (values read_state()->{holders}->%*) {
+        my $member = $members->{ $holder->{node} } or next;
+        next if $member->{online} || !grep { $wanted{$_} } $holder->{resources}->@*;
+        $nodes->{ $holder->{node} } = 1;
+    }
+    return [sort keys $nodes->%*];
+}
+
+# Grants waiters that fit by now, for example after limits got raised.
+sub refresh() {
+    update_state(sub { });
+}
+
+sub release($key) {
+    update_state(sub($state) {
+        delete $state->{$_}->{$key} for qw(holders waiters);
+    });
+}
+
+# Removes the entries of this node whose guest backup is gone, as told by $running->($vmid, $upid).
+sub cleanup_local_entries($running) {
+    my $node = PVE::INotify::nodename();
+
+    my $stale = sub($entry) {
+        return $entry->{node} eq $node && !$running->($entry->{vmid}, $entry->{upid});
+    };
+
+    # avoid the cluster lock if there is nothing to remove, but let it replace an unparsable table
+    my $state = eval { read_state() };
+    return if $state && !remove_entries($state, $stale);
+
+    update_state(sub($state) {
+        remove_entries($state, $stale);
+    });
+}
+
+1;
diff --git a/PVE/VZDump/Makefile b/PVE/VZDump/Makefile
new file mode 100644
index 000000000..6500eb4d4
--- /dev/null
+++ b/PVE/VZDump/Makefile
@@ -0,0 +1,15 @@
+include ../../defines.mk
+
+PERLSOURCE =   \
+	ClusterSlots.pm \
+
+all:
+
+.PHONY: clean
+clean:
+	rm -rf *~
+
+.PHONY: install
+install: $(PERLSOURCE)
+	install -d $(PERLLIBDIR)/PVE/VZDump
+	install -m 0644 $(PERLSOURCE) $(PERLLIBDIR)/PVE/VZDump
diff --git a/test/Makefile b/test/Makefile
index bbe87a7c3..04de559be 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
 	./$<
 
 .PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency test-vzdump-cluster-slots
 
 .PHONY: test-vzdump-guest-included
 test-vzdump-guest-included:
@@ -32,6 +32,10 @@ test-vzdump-new:
 test-vzdump-concurrency:
 	./vzdump_concurrency_test.pl
 
+.PHONY: test-vzdump-cluster-slots
+test-vzdump-cluster-slots:
+	./vzdump_cluster_slots_test.pl
+
 .PHONY: test-osd
 test-osd:
 	./OSD_test.pl
diff --git a/test/vzdump_cluster_slots_test.pl b/test/vzdump_cluster_slots_test.pl
new file mode 100755
index 000000000..51ca8341d
--- /dev/null
+++ b/test/vzdump_cluster_slots_test.pl
@@ -0,0 +1,205 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use lib '..';
+
+use JSON;
+use Storable qw(dclone);
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump::ClusterSlots;
+
+my $members = {
+    node1 => { online => 1 },
+    node2 => { online => 1 },
+    node3 => { online => 0 },
+};
+
+my sub make_state($holders, $waiters) {
+    my $state = { holders => {}, waiters => {}, seq => 0 };
+    for my $holder ($holders->@*) {
+        my ($key, $node, $resources) = $holder->@*;
+        $state->{holders}->{$key} = { node => $node, resources => $resources };
+    }
+    for my $waiter ($waiters->@*) {
+        my ($key, $node, $resources) = $waiter->@*;
+        $state->{waiters}->{$key} = {
+            node => $node,
+            resources => $resources,
+            seq => ++$state->{seq},
+        };
+    }
+    return $state;
+}
+
+my sub granted($state) {
+    return [sort keys $state->{holders}->%*];
+}
+
+my $tests = [
+    {
+        description => 'waiters are granted in order up to the limit',
+        limits => { cluster => 2 },
+        waiters =>
+            [['a', 'node1', ['cluster']], ['b', 'node2', ['cluster']], ['c', 'node1', ['cluster']]],
+        expected => ['a', 'b'],
+    },
+    {
+        description => 'holders count against the limit',
+        limits => { cluster => 2 },
+        holders => [['h', 'node1', ['cluster']]],
+        waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['a', 'h'],
+    },
+    {
+        description => 'resources without limit do not hold up',
+        limits => {},
+        holders => [['h', 'node1', ['cluster']]],
+        waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['a', 'b', 'h'],
+    },
+    {
+        description => 'waiters of offline nodes neither get granted nor hold up others',
+        limits => { cluster => 1 },
+        waiters => [['a', 'node3', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['b'],
+    },
+    {
+        description => 'holders of offline nodes keep counting',
+        limits => { cluster => 1 },
+        holders => [['h', 'node3', ['cluster']]],
+        waiters => [['a', 'node1', ['cluster']]],
+        expected => ['h'],
+    },
+    {
+        description => 'entries of nodes no longer in the cluster are dropped',
+        limits => { cluster => 1 },
+        holders => [['h', 'gone', ['cluster']]],
+        waiters => [['a', 'gone', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['b'],
+        'expected-waiters' => [],
+    },
+];
+
+for my $test ($tests->@*) {
+    my $state = make_state($test->{holders} // [], $test->{waiters} // []);
+
+    PVE::VZDump::ClusterSlots::grant_waiters($state, $test->{limits}, $members);
+
+    is_deeply(granted($state), $test->{expected}, $test->{description});
+
+    if (my $expected_waiters = $test->{'expected-waiters'}) {
+        is_deeply(
+            [sort keys $state->{waiters}->%*],
+            $expected_waiters,
+            "$test->{description} (waiters)",
+        );
+    }
+}
+
+# Request, grant, release and cleanup against an in-memory table.
+my $table = { holders => {}, waiters => {}, seq => 0 };
+my $writes = 0;
+
+my $slots_module = Test::MockModule->new('PVE::VZDump::ClusterSlots');
+$slots_module->mock(
+    read_state => sub { return dclone($table); },
+    limits => sub { return { cluster => 1 }; },
+);
+my $cluster_module = Test::MockModule->new('PVE::Cluster');
+$cluster_module->mock(
+    cfs_lock_domain => sub {
+        my ($domain, $timeout, $code) = @_;
+        return eval { $code->() };
+    },
+    get_members => sub { return $members; },
+);
+my $file_module = Test::MockModule->new('PVE::File');
+$file_module->mock(
+    file_set_contents => sub {
+        my ($filename, $raw) = @_;
+        $table = decode_json($raw);
+        $writes++;
+    },
+);
+my $inotify_module = Test::MockModule->new('PVE::INotify');
+$inotify_module->mock(nodename => sub { return 'node1'; });
+
+$table->{holders}->{other} = { node => 'node2', vmid => 200, upid => 'u0', resources => [] };
+
+my $first = PVE::VZDump::ClusterSlots::request('u1', 100, ['cluster']);
+my $second = PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(PVE::VZDump::ClusterSlots::granted($first), 1, 'first request gets the slot');
+is(PVE::VZDump::ClusterSlots::granted($second), 0, 'second request waits');
+
+PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(scalar(keys $table->{waiters}->%*), 1, 'repeated request does not queue again');
+
+my $count = $writes;
+PVE::VZDump::ClusterSlots::refresh();
+is($writes, $count, 'unchanged table is not written');
+
+PVE::VZDump::ClusterSlots::release($first);
+is(PVE::VZDump::ClusterSlots::granted($second), 1, 'released slot goes to the waiting request');
+is(PVE::VZDump::ClusterSlots::granted($first), undef, 'released request is gone');
+
+PVE::VZDump::ClusterSlots::cleanup_local_entries(sub { return 0; });
+is_deeply([keys $table->{holders}->%*], ['other'], 'cleanup only drops stale entries of the node');
+
+$table->{holders}->{down} =
+    { node => 'node3', vmid => 300, upid => 'u3', resources => ['cluster'] };
+is_deeply(
+    PVE::VZDump::ClusterSlots::offline_holders(['cluster']),
+    ['node3'],
+    'offline node holds a slot',
+);
+is_deeply(
+    PVE::VZDump::ClusterSlots::offline_holders(['storage/pbs']),
+    [],
+    'offline node holds no slot of another limit',
+);
+
+my $entry = '{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":["cluster"]}';
+my $parsed = PVE::VZDump::ClusterSlots::parse_state('{"holders":{},"waiters":{},"seq":3}');
+is(JSON->new->canonical->encode($parsed), '{"holders":{},"seq":3,"waiters":{}}', 'valid table');
+ok(
+    PVE::VZDump::ClusterSlots::parse_state(qq({"holders":{"a":$entry},"waiters":{},"seq":3})),
+    'valid table with an entry',
+);
+for my $raw (
+    'garbage',
+    '[]',
+    '{"holders":null,"waiters":{},"seq":3}',
+    '{"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":[],"seq":3}',
+    '{"holders":{},"waiters":{}}',
+    '{"holders":{},"waiters":{},"seq":"x"}',
+    '{"holders":{"a":{}},"waiters":{},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":"cluster"}},"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":{"a":{"upid":"UPID:x","vmid":100,"seq":1,"resources":[]}},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[null]}},"waiters":{},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[{}]}},"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":{},"seq":"\u0663"}',
+    '{"holders":{},"waiters":{},"seq":"3\n"}',
+) {
+    eval { PVE::VZDump::ClusterSlots::parse_state($raw) };
+    ok($@, "rejects $raw");
+}
+
+my $versions = { node2 => '9.2.31', node3 => '9.2.30' };
+$cluster_module->mock(
+    get_node_kv => sub {
+        my ($key, $node) = @_;
+        return {} if $key ne 'version-info' || !defined($versions->{$node});
+        return { $node => encode_json({ version => $versions->{$node} }) };
+    },
+);
+ok(PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older offline node is ignored');
+$versions->{node2} = '9.2.30';
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older online node blocks');
+delete $versions->{node2};
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'online node without version blocks');
+
+done_testing();
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (15 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
                   ` (2 subsequent siblings)
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Guest backups now also count against the backup concurrency limit of
the storages they use, both the backup target and the storages of the
included guest disks. This protects a backup server or a Ceph cluster
from more parallel backups than it can handle, which a limit for the
whole cluster cannot express once several such systems are in use.

Limits of shared storages count for the whole cluster and use the
cluster-wide slots. Limits of local storages count on each node and
use slot locks like the node limit. A guest whose local storage has no
free slot lets other guests of the job go first.

With several limits, a waiting guest backup can be blocked by a full
storage while younger ones with room in their limits get slots. This
may happen a few times, then a slot of each limit it needs is kept
free for it, so it cannot starve.

The storages are taken from the guest configuration before the guest
gets locked for backup, so waiting for slots does not block changes to
the guest. If its disks moved to another storage with a limit by the
time it got locked, the backup fails instead of running without a slot
for that storage. If the storages cannot be determined, the guest is
skipped for the same reason. Nothing of this happens as long as no
storage has a limit. The task log tells which limit a guest waits for.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- log the wait for a storage slot once, not on every pass, if guests
  wait for different storages
- the warning about limits that are not enforced names the shared
  storage limits too

 PVE/VZDump.pm                     | 113 +++++++++++++++++++++++++++---
 PVE/VZDump/ClusterSlots.pm        |  42 ++++++++++-
 test/vzdump_cluster_slots_test.pl |  79 ++++++++++++++++++++-
 3 files changed, 221 insertions(+), 13 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 867fe0b8f..10d0caf1b 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -951,6 +951,34 @@ sub get_backup_file_list {
     return $bklist;
 }
 
+my sub limited_backup_storages {
+    my ($self, $task) = @_;
+
+    my $storage_cfg = PVE::Storage::config();
+    my $limited = {
+        map { $_ => 1 }
+        grep { defined($storage_cfg->{ids}->{$_}->{'backup-concurrency'}) }
+            keys $storage_cfg->{ids}->%*
+    };
+    return [] if !scalar(keys $limited->%*);
+
+    my $storeids = {};
+    $storeids->{ $self->{opts}->{storage} } = 1 if $self->{opts}->{storage};
+
+    if (my $plugin = $task->{plugin}) {
+        my $config_class = $plugin->type() eq 'qemu' ? 'PVE::QemuConfig' : 'PVE::LXC::Config';
+        my $conf = $config_class->load_config($task->{vmid});
+        for my $volume ($config_class->get_backup_volumes($conf)->@*) {
+            next if !$volume->{included};
+            my $volid = $volume->{volume_config}->{file} // $volume->{volume_config}->{volume};
+            my ($storeid) = PVE::Storage::parse_volume_id($volid, 1);
+            $storeids->{$storeid} = 1 if defined($storeid);
+        }
+    }
+
+    return [sort grep { $limited->{$_} } keys $storeids->%*];
+}
+
 sub exec_backup_task {
     my ($self, $task) = @_;
 
@@ -1134,6 +1162,13 @@ sub exec_backup_task {
 
         $cleanup->{unlock} = 1;
 
+        my $taken = { map { $_ => 1 } $task->{storages}->@* };
+        if (my @missing = grep { !$taken->{$_} } limited_backup_storages($self, $task)->@*) {
+            my $list = join(', ', map { "'$_'" } @missing);
+            die "no backup slot for storage $list - guest disks or storage limits changed while"
+                . " waiting for backup slots\n";
+        }
+
         # prepare
 
         my $mode = $running ? $task->{mode} : 'stop';
@@ -1783,6 +1818,25 @@ my sub try_node_slot {
     return try_slot("$lockdir/slot", node_backup_concurrency());
 }
 
+# Limits of local storages count on each node, shared ones are handled cluster-wide.
+my sub try_local_storage_slots {
+    my ($storeids) = @_;
+
+    my $storage_cfg = PVE::Storage::config();
+
+    my $locks = [];
+    for my $storeid ($storeids->@*) {
+        my $scfg = $storage_cfg->{ids}->{$storeid} or next;
+        my $limit = $scfg->{'backup-concurrency'};
+        next if $scfg->{shared} || !defined($limit);
+
+        my $lock = try_slot("$lockdir/storage-$storeid", $limit) or return (undef, $storeid);
+        push $locks->@*, $lock;
+    }
+
+    return ($locks);
+}
+
 my sub log_wait {
     my ($self, $reason) = @_;
 
@@ -1845,10 +1899,25 @@ my sub claim_next_guest {
             skip_task($task, "guest moved to node '$node'");
             next;
         }
+
+        $task->{storages} = eval { limited_backup_storages($self, $task) };
+        if (my $err = $@) {
+            die $err if $err =~ m/interrupted by signal$/;
+            chomp($err);
+            splice($queue->@*, $i--, 1);
+            skip_task($task, "unable to check storage backup limits - $err");
+            next;
+        }
+
+        my ($storage_locks, $busy_storage) = try_local_storage_slots($task->{storages});
+        if (!$storage_locks) {
+            log_wait($self, "a free backup slot of storage '$busy_storage'");
+            next;
+        }
         delete $self->{'waiting-for'};
 
         splice($queue->@*, $i--, 1);
-        return ($task, [$guest_lock, $slot]);
+        return ($task, [$guest_lock, $slot, $storage_locks->@*]);
     }
 
     return;
@@ -1864,7 +1933,7 @@ my sub skip_after_error {
 }
 
 my sub cluster_slot_resources {
-    my ($self) = @_;
+    my ($self, $task) = @_;
 
     my $limits = PVE::VZDump::ClusterSlots::limits();
     return undef if !scalar(keys $limits->%*);
@@ -1873,13 +1942,31 @@ my sub cluster_slot_resources {
         if (!$self->{'cluster-slots-unsupported'}++) {
             debugmsg(
                 'warn',
-                "cluster-wide backup limits are not enforced - not all online nodes support them",
+                "cluster-wide and shared storage backup limits are not enforced - not all online"
+                    . " nodes support them",
             );
         }
         return undef;
     }
 
-    return ['cluster'];
+    my $storage_cfg = PVE::Storage::config();
+    my $shared = [grep { $storage_cfg->{ids}->{$_}->{shared} } $task->{storages}->@*];
+
+    return ['cluster', map { "storage/$_" } $shared->@*];
+}
+
+my sub render_cluster_limits {
+    my ($resources) = @_;
+    return join(', ', map { m!^storage/(.*)$! ? "storage '$1'" : "the $_" } $resources->@*);
+}
+
+my sub reached_cluster_limits {
+    my ($key) = @_;
+
+    return [] if !defined($key);
+    my $reached = eval { PVE::VZDump::ClusterSlots::reached_limits($key) };
+    die $@ if $@ =~ m/interrupted by signal$/;
+    return $reached // [];
 }
 
 my sub release_cluster_slots {
@@ -1921,7 +2008,7 @@ my sub next_guest_to_start {
     my $pending = $schedule->{pending};
     if (!$pending) {
         my ($task, $locks) = claim_next_guest($self, $queue, $schedule->{deferred}) or return;
-        my $resources = eval { cluster_slot_resources($self) };
+        my $resources = eval { cluster_slot_resources($self, $task) };
         if (my $err = $@) {
             unshift $queue->@*, $task;
             die $err;
@@ -1977,16 +2064,22 @@ my sub next_guest_to_start {
     my $waited = $schedule->{'cluster-wait'} + time() - $pending->{since};
     my $maxwait = $self->{opts}->{lockwait} * 60;
     if ($waited >= $maxwait) {
+        # before the release, as that removes the request
+        my $offline = offline_slot_holders(reached_cluster_limits($pending->{key}));
         release_cluster_slots($pending->{key});
         delete $schedule->{pending};
         $schedule->{'cluster-wait'} = $waited;
-        my $reason =
-            'timeout waiting for cluster-wide backup slot' . offline_slot_holders(['cluster']);
+        my $reason = "timeout waiting for cluster-wide backup slot$offline";
         $reason .= " - $pending->{'request-error'}" if defined($pending->{'request-error'});
         skip_task($task, $reason);
-    } elsif (!$pending->{logged}++) {
-        my $offline = offline_slot_holders(['cluster']);
-        debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot$offline");
+    } elsif (defined($pending->{key}) && !$pending->{logged}++) {
+        my $reached = reached_cluster_limits($pending->{key});
+        my $reason = "queued behind other guest backups";
+        if (scalar($reached->@*)) {
+            my $limits = render_cluster_limits($reached);
+            $reason = "limit reached for $limits" . offline_slot_holders($reached);
+        }
+        debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot, $reason");
     }
 
     return;
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
index 7ba47a0eb..9d199481d 100644
--- a/PVE/VZDump/ClusterSlots.pm
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -14,12 +14,16 @@ use PVE::Cluster qw(cfs_read_file);
 use PVE::File;
 use PVE::INotify;
 use PVE::RPCEnvironment;
+use PVE::Storage;
 
 my $state_file = '/etc/pve/backup-slots.json';
 
 # first release of pve-manager that takes part in cluster-wide limits
 my $min_version = [9, 2, 31];
 
+# How often younger requests may get ahead of a waiting one before its limits are reserved for it.
+use constant MAX_BYPASS => 3;
+
 sub limits() {
     my $limits = {};
 
@@ -28,6 +32,13 @@ sub limits() {
         $limits->{cluster} = $cluster;
     }
 
+    my $storage_cfg = PVE::Storage::config();
+    for my $storeid (keys $storage_cfg->{ids}->%*) {
+        my $scfg = $storage_cfg->{ids}->{$storeid};
+        next if !$scfg->{shared} || !defined($scfg->{'backup-concurrency'});
+        $limits->{"storage/$storeid"} = $scfg->{'backup-concurrency'};
+    }
+
     return $limits;
 }
 
@@ -130,19 +141,36 @@ my sub remove_entries($state, $match) {
 
 # Moves waiters that fit into all limits they count against to the holders, oldest first. Only
 # resources with a limit matter, the others are tracked for information. Waiters of offline nodes
-# neither get granted nor hold up others.
+# neither get granted nor hold up others. A waiter that younger ones got ahead of too often
+# reserves a slot of each limit it counts against, so it cannot starve.
 sub grant_waiters($state, $limits, $members) {
     remove_entries($state, sub($entry) { !$members->{ $entry->{node} } });
 
     my $used = used_slots($state);
 
+    my $passed_over = [];
+
     my $waiters = $state->{waiters};
     for my $key (sort { $waiters->{$a}->{seq} <=> $waiters->{$b}->{seq} } keys $waiters->%*) {
         my $waiter = $waiters->{$key};
         next if !$members->{ $waiter->{node} }->{online};
 
         my $limited = [grep { defined($limits->{$_}) } $waiter->{resources}->@*];
-        next if grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+        my $fits = !grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+
+        if (!$fits) {
+            if (($waiter->{bypassed} // 0) >= MAX_BYPASS) {
+                $used->{$_}++ for $limited->@*;
+            }
+            push $passed_over->@*, [$waiter, $limited];
+            next;
+        }
+
+        my %mine = map { $_ => 1 } $limited->@*;
+        for my $older ($passed_over->@*) {
+            my ($older_waiter, $older_limited) = $older->@*;
+            $older_waiter->{bypassed}++ if grep { $mine{$_} } $older_limited->@*;
+        }
 
         $used->{$_}++ for $waiter->{resources}->@*;
         $state->{holders}->{$key} = delete $waiters->{$key};
@@ -216,6 +244,16 @@ sub offline_holders($resources) {
     return [sort keys $nodes->%*];
 }
 
+sub reached_limits($key) {
+    my $state = read_state();
+    my $waiter = $state->{waiters}->{$key} or return [];
+
+    my $used = used_slots($state);
+    my $limits = limits();
+    return [grep { defined($limits->{$_}) && ($used->{$_} // 0) >= $limits->{$_} }
+        $waiter->{resources}->@*];
+}
+
 # Grants waiters that fit by now, for example after limits got raised.
 sub refresh() {
     update_state(sub { });
diff --git a/test/vzdump_cluster_slots_test.pl b/test/vzdump_cluster_slots_test.pl
index 51ca8341d..786bbe455 100755
--- a/test/vzdump_cluster_slots_test.pl
+++ b/test/vzdump_cluster_slots_test.pl
@@ -24,11 +24,12 @@ my sub make_state($holders, $waiters) {
         $state->{holders}->{$key} = { node => $node, resources => $resources };
     }
     for my $waiter ($waiters->@*) {
-        my ($key, $node, $resources) = $waiter->@*;
+        my ($key, $node, $resources, $bypassed) = $waiter->@*;
         $state->{waiters}->{$key} = {
             node => $node,
             resources => $resources,
             seq => ++$state->{seq},
+            bypassed => $bypassed // 0,
         };
     }
     return $state;
@@ -81,6 +82,54 @@ my $tests = [
         expected => ['b'],
         'expected-waiters' => [],
     },
+    {
+        description => 'younger waiter with free resources gets ahead of a blocked one',
+        limits => { x => 1, y => 1 },
+        holders => [['h', 'node1', ['x']]],
+        waiters => [['a', 'node1', ['x', 'y']], ['b', 'node2', ['y']]],
+        expected => ['b', 'h'],
+        bypassed => { a => 1 },
+    },
+    {
+        description => 'waiter passed over too often reserves its resources',
+        limits => { x => 1, y => 1 },
+        holders => [['h', 'node1', ['x']]],
+        waiters => [['a', 'node1', ['x', 'y'], 3], ['b', 'node2', ['y']]],
+        expected => ['h'],
+    },
+    {
+        description => 'waiter passed over too often reserves only one slot of each resource',
+        limits => { cluster => 3, x => 1 },
+        holders => [['h', 'node1', ['cluster', 'x']]],
+        waiters => [
+            ['a', 'node1', ['cluster', 'x'], 3],
+            ['b', 'node2', ['cluster']],
+            ['c', 'node2', ['cluster']],
+        ],
+        expected => ['b', 'h'],
+    },
+    {
+        description => 'reserved resources do not hold up unrelated waiters',
+        limits => { x => 1, y => 1, z => 1 },
+        holders => [['h', 'node1', ['x']]],
+        waiters => [['a', 'node1', ['x', 'y'], 3], ['b', 'node2', ['z']]],
+        expected => ['b', 'h'],
+    },
+    {
+        description => 'resources without limit do not count as passed over',
+        limits => { x => 1 },
+        holders => [['h', 'node1', ['cluster', 'x']]],
+        waiters => [['a', 'node1', ['cluster', 'x']], ['b', 'node2', ['cluster']]],
+        expected => ['b', 'h'],
+        bypassed => { a => 0 },
+    },
+    {
+        description => 'resources without limit do not get reserved',
+        limits => { x => 1 },
+        holders => [['h', 'node1', ['cluster', 'x']]],
+        waiters => [['a', 'node1', ['cluster', 'x'], 3], ['b', 'node2', ['cluster', 'y']]],
+        expected => ['b', 'h'],
+    },
 ];
 
 for my $test ($tests->@*) {
@@ -97,6 +146,34 @@ for my $test ($tests->@*) {
             "$test->{description} (waiters)",
         );
     }
+    for my $key (sort keys(($test->{bypassed} // {})->%*)) {
+        is(
+            $state->{waiters}->{$key}->{bypassed},
+            $test->{bypassed}->{$key},
+            "$test->{description} ($key bypassed)",
+        );
+    }
+}
+
+{
+    my $limits = { cluster => 2, x => 1 };
+    my $state = make_state([['h', 'node1', ['cluster', 'x']]], [['a', 'node1', ['cluster', 'x']]]);
+    my $add_younger = sub {
+        my ($key) = @_;
+        $state->{waiters}->{$key} =
+            { node => 'node2', resources => ['cluster'], seq => ++$state->{seq} };
+        PVE::VZDump::ClusterSlots::grant_waiters($state, $limits, $members);
+    };
+    for my $younger (qw(b c d)) {
+        $add_younger->($younger);
+        delete $state->{holders}->{$younger};
+    }
+    $add_younger->('e');
+    is_deeply(granted($state), ['h'], 'younger waiters stop getting ahead after a few times');
+
+    delete $state->{holders}->{h};
+    PVE::VZDump::ClusterSlots::grant_waiters($state, $limits, $members);
+    is_deeply(granted($state), ['a', 'e'], 'passed over waiter gets its slots once they are free');
 }
 
 # Request, grant, release and cleanup against an in-memory table.
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (16 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

List the guest backups that hold or wait for cluster-wide backup
slots, to see where the slots went when guest backups wait.

Slots of an offline node stay taken, as its guest backups might still
be running. Allow releasing them once it is clear that these backups
stopped, for example because the node is powered off. Otherwise the
node releases them itself when it is back online.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- list slots without going through pvedaemon, the table is no longer in
  priv/ [Fiona]
- use signatures and no multi-line postfix condition, like patch 16
- say which permission the guest list needs, and the unit of 'since'

 PVE/API2/Cluster/BackupInfo.pm | 106 ++++++++++++++++++++++++++++++++-
 PVE/VZDump/ClusterSlots.pm     |  11 ++++
 2 files changed, 115 insertions(+), 2 deletions(-)

diff --git a/PVE/API2/Cluster/BackupInfo.pm b/PVE/API2/Cluster/BackupInfo.pm
index 4ef59ac3d..df487c7c5 100644
--- a/PVE/API2/Cluster/BackupInfo.pm
+++ b/PVE/API2/Cluster/BackupInfo.pm
@@ -9,10 +9,11 @@ use PVE::Tools qw(extract_param);
 use PVE::Cluster qw(cfs_lock_file cfs_read_file cfs_write_file);
 use PVE::RESTHandler;
 use PVE::RPCEnvironment;
-use PVE::JSONSchema;
+use PVE::JSONSchema qw(get_standard_option);
 use PVE::Storage;
 use PVE::Exception qw(raise_param_exc);
 use PVE::VZDump;
+use PVE::VZDump::ClusterSlots;
 use PVE::VZDump::Common;
 
 use base qw(PVE::RESTHandler);
@@ -57,11 +58,112 @@ __PACKAGE__->register_method({
     },
     code => sub {
         return [
-            { subdir => 'not-backed-up' },
+            { subdir => 'not-backed-up' }, { subdir => 'slots' },
         ];
     },
 });
 
+__PACKAGE__->register_method({
+    name => 'get_backup_slots',
+    path => 'slots',
+    method => 'GET',
+    description => "Shows guest backups that hold or wait for cluster-wide backup slots.",
+    permissions => {
+        description =>
+            "Only lists guests where you have 'VM.Audit' permissions on '/vms/<vmid>'.",
+        check => ['perm', '/', ['Sys.Audit']],
+    },
+    parameters => {
+        additionalProperties => 0,
+        properties => {},
+    },
+    returns => {
+        type => 'array',
+        items => {
+            type => 'object',
+            properties => {
+                node => get_standard_option('pve-node'),
+                vmid => get_standard_option('pve-vmid'),
+                upid => {
+                    type => 'string',
+                    description => 'Backup task the guest backup belongs to.',
+                },
+                state => {
+                    type => 'string',
+                    enum => ['running', 'waiting'],
+                    description =>
+                        'Whether the guest backup holds its slots or waits for them.',
+                },
+                since => {
+                    type => 'integer',
+                    description =>
+                        'When the guest backup got its slots or started waiting for them (UNIX epoch).',
+                },
+                storages => {
+                    type => 'array',
+                    items => get_standard_option('pve-storage-id'),
+                    description =>
+                        'Shared storages with a backup concurrency limit that the guest'
+                        . ' backup uses.',
+                },
+            },
+        },
+    },
+    code => sub {
+        my $rpcenv = PVE::RPCEnvironment::get();
+        my $user = $rpcenv->get_user();
+
+        my $state = PVE::VZDump::ClusterSlots::read_state();
+
+        my $result = [];
+        for my $kind (qw(holders waiters)) {
+            for my $entry (values $state->{$kind}->%*) {
+                next if !$rpcenv->check($user, "/vms/$entry->{vmid}", ['VM.Audit'], 1);
+
+                push $result->@*,
+                    {
+                        node => $entry->{node},
+                        vmid => int($entry->{vmid}),
+                        upid => $entry->{upid},
+                        state => $kind eq 'holders' ? 'running' : 'waiting',
+                        since => int($kind eq 'holders' ? $entry->{granted} : $entry->{queued}),
+                        storages => [map { m!^storage/(.+)$! ? $1 : () } $entry->{resources}->@*],
+                    };
+            }
+        }
+
+        return $result;
+    },
+});
+
+__PACKAGE__->register_method({
+    name => 'release_backup_slots',
+    path => 'slots/{node}',
+    method => 'DELETE',
+    protected => 1,
+    description =>
+        "Release the cluster-wide backup slots of an offline node. They stay taken because its"
+        . " guest backups might still be running, so only release them once you are sure that"
+        . " these stopped, for example because the node is powered off.",
+    permissions => {
+        check => ['perm', '/', ['Sys.Modify']],
+    },
+    parameters => {
+        additionalProperties => 0,
+        properties => {
+            node => get_standard_option('pve-node'),
+        },
+    },
+    returns => { type => 'null' },
+    code => sub {
+        my ($param) = @_;
+
+        PVE::VZDump::ClusterSlots::release_node($param->{node});
+
+        return;
+    },
+});
+
 __PACKAGE__->register_method({
     name => 'get_guests_not_in_backup',
     path => 'not-backed-up',
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
index 9d199481d..02895c127 100644
--- a/PVE/VZDump/ClusterSlots.pm
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -282,4 +282,15 @@ sub cleanup_local_entries($running) {
     });
 }
 
+sub release_node($node) {
+    update_state(sub($state) {
+        # checked under the lock, so the node cannot rejoin and take new slots meanwhile
+        if (get_members()->{$node}->{online}) {
+            die "cannot release backup slots of node '$node' - it is online\n";
+        }
+
+        remove_entries($state, sub($entry) { $entry->{node} eq $node });
+    });
+}
+
 1;
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (17 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Expose the cluster-wide limit next to the per-node one in the
datacenter options, and the limit of a storage in the advanced section
of the storage edit window, so admins can protect shared systems from
too many parallel backups without editing configuration files.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- use title case for the 'Cluster-Wide' label

 www/manager6/dc/OptionView.js |  8 ++++++++
 www/manager6/storage/Base.js  | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/www/manager6/dc/OptionView.js b/www/manager6/dc/OptionView.js
index 4a7e9692a..8f10cba2b 100644
--- a/www/manager6/dc/OptionView.js
+++ b/www/manager6/dc/OptionView.js
@@ -372,6 +372,14 @@ Ext.define('PVE.dc.OptionView', {
                     minValue: 1,
                     submitValue: true,
                 },
+                {
+                    xtype: 'proxmoxintegerfield',
+                    name: 'cluster',
+                    fieldLabel: gettext('Cluster-Wide'),
+                    emptyText: gettext('Unlimited'),
+                    minValue: 1,
+                    submitValue: true,
+                },
             ],
         });
         me.add_inputpanel_row('next-id', gettext('Next Free Guest ID'), {
diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index 439cfeb88..753e9406b 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -109,6 +109,23 @@ Ext.define('PVE.panel.StorageBase', {
             }
         };
 
+        if (me.type !== 'esxi') {
+            addAdvancedWidget({
+                xtype: 'proxmoxintegerfield',
+                name: 'backup-concurrency',
+                fieldLabel: gettext('Backup Concurrency'),
+                emptyText: gettext('Unlimited'),
+                minValue: 1,
+                deleteEmpty: !me.isCreate,
+                autoEl: {
+                    tag: 'div',
+                    'data-qtip': gettext(
+                        'Guest backups using this storage at the same time, as target or for guest disks. Counted for the whole cluster if the storage is shared, otherwise on each node.',
+                    ),
+                },
+            });
+        }
+
         const qemuImgStorageTypes = ['dir', 'btrfs', 'nfs', 'cifs'];
 
         if (qemuImgStorageTypes.includes(me.type)) {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits
  2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
                   ` (18 preceding siblings ...)
  2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
@ 2026-11-18 13:18 ` Thomas Lamprecht
  19 siblings, 0 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

Describe how many guests a node backs up at the same time and that the
bandwidth limit applies to each of them, how jobs can run alongside
each other, and the cluster-wide and storage limits, including how to
size them and what happens with slots of an offline node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- jobs to an external backup provider ignore 'concurrency' and 'overlap'
- a job starts no other guests while one waits for cluster-wide slots
- say what happens once a wait ends, which limits need all nodes to be
  upgraded, and that overlapping jobs share the node limit

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..241c38c 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,122 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_concurrency]]
+Concurrent Backups
+------------------
+
+By default, a node runs one backup job at a time and backs up its guests one
+after the other. A job that is due while another one runs waits for it, up to
+the `lockwait` time. As each node backs up its own guests, how long a job takes
+and how much load it puts on the backup target depend on how its guests are
+spread over the nodes. The settings described here allow backing up several
+guests of a node at the same time, running jobs alongside each other, and
+limiting the load on shared storages and backup servers.
+
+Node Limit and Job Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backup concurrency of a node limits how many guests it backs up at the same
+time, across all jobs. Set the default for all nodes in the datacenter options
+(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in
+their options (`backup-concurrency` in the node configuration):
+
+----
+# pvesh set /cluster/options --backup-concurrency 4
+# pvenode config set --backup-concurrency 8
+----
+
+A job backs up as many guests at the same time as its node allows. With the
+`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use
+fewer, but not more.
+
+Size the node limit by what the node itself can handle, that is, its CPU,
+memory, network connection, and local disks. Each parallel guest backup runs its
+own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to
+each guest backup on its own, so the bandwidth of a node can add up to the
+configured limit times the number of guests backed up at the same time.
+
+Jobs Running Alongside Each Other
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same
+time as other jobs that have it enabled, for example, a short hourly job
+alongside a long nightly one. Together, they stay within the backup concurrency
+of the node, so with the default of one, still only one guest is backed up at a
+time. A job without the option still runs alone, and while it waits for running
+jobs to finish, overlapping jobs that are due later do not start before it.
+Waiting for a free backup slot of the node is not limited by `lockwait`.
+
+If two running jobs include the same guest, the second one backs it up once the
+first one is done with it, or skips it after waiting `lockwait` minutes.
+
+Jobs to an external backup provider ignore the `concurrency` and `overlap`
+options. They back up one guest at a time and always run alone.
+
+NOTE: The job phases of hook scripts can overlap for overlapping jobs. For
+example, a hook script that mounts a disk in the `job-start` phase and unmounts
+it in the `job-end` phase must not be used for overlapping jobs. With a job
+concurrency above one, the phases of different guests of a job overlap as well.
+
+Cluster-wide and Storage Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each node runs its part of a job on its own, so in a cluster, shared storages
+and backup servers get the backups of all nodes at the same time. To protect
+them, you can limit how many guest backups run at the same time:
+
+* for the whole cluster, with the `cluster` key of `backup-concurrency` in the
+  datacenter options ('Cluster-Wide' in the UI),
+* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in
+  the advanced storage settings). It counts guest backups that use the storage
+  as backup target or for included guest disks, but not for fleecing or
+  temporary files. For a shared storage, the limit applies to the whole cluster,
+  for a local storage to each node.
+
+----
+# pvesh set /cluster/options --backup-concurrency node=8,cluster=12
+# pvesm set pbs-main --backup-concurrency 6
+----
+
+A guest backup only starts once all limits it counts against have room. Size
+the cluster-wide and storage limits by what the shared systems can handle,
+independent of the node limits. With such limits in place, raising the node
+limits cannot overload the shared systems.
+
+TIP: Several storage entries can refer to the same system, for example,
+namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits
+count separately, so use the cluster-wide limit to bound their sum.
+
+The cluster-wide limit and the limits of shared storages are counted in
+cluster-wide slots, which guest backups get in the order they started waiting.
+One that waits for a storage at its limit lets others with room go first, but
+only a few times. A job has at most one guest waiting for cluster-wide slots at
+a time and starts no other guests meanwhile. It waits for these slots at most
+`lockwait` minutes in total and skips guests that did not get them by then. Like
+for node slots, waiting for free slots of a local storage is not limited by
+`lockwait`. The task log tells which limit a guest waits for.
+
+To see which guest backups hold or wait for cluster-wide slots, use:
+
+----
+# pvesh get /cluster/backup-info/slots
+----
+
+Granting cluster-wide slots requires quorum, and the cluster-wide and shared
+storage limits are only enforced once all online nodes of the cluster run a
+version that supports them. Until then, the task log warns that they are not
+enforced. Guest backups that started before that, or while neither a
+cluster-wide limit nor a limit on a shared storage was set, do not count against
+them. Slots of a node that goes offline stay taken, as its guest backups might
+still be running, and the node releases them once it is back. Guest backups that
+wait for such slots name the node in the task log. Once you made sure that its
+backups stopped, for example because the node is powered off, you can release
+its slots manually:
+
+----
+# pvesh delete /cluster/backup-info/slots/NODE
+----
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2026-11-18 13:33 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal