From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 storage 03/20] add backup concurrency option to storages
Date: Wed, 18 Nov 2026 14:18:27 +0100 [thread overview]
Message-ID: <20261118132947.1093069-4-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>
Allows limiting how many guest backups use a storage at the same time,
be it as backup target or because guest disks live on it. A backup
server or a Ceph cluster can thus be protected from more backup load
than it can handle, independent of how many nodes and jobs take part.
For shared storages the limit counts for the whole cluster, for local
storages on each node. The ESXi import storage is not a backup source
or target and does not get the option. The option is declared in the
base plugin, as the backup code reads it from storages of any type, so
plugins that isolate their properties can offer it as well. Third-party
plugins need to list it in their options() to support it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
changes v1 -> v2:
- rebase onto the property isolation changes; the option is now in the
pinned lists of the plugin property test
src/PVE/Storage/BTRFSPlugin.pm | 1 +
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
src/PVE/Storage/ISCSIPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 1 +
src/PVE/Storage/LvmThinPlugin.pm | 1 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/PBSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 9 +++++++++
src/PVE/Storage/RBDPlugin.pm | 1 +
src/PVE/Storage/ZFSPlugin.pm | 1 +
src/PVE/Storage/ZFSPoolPlugin.pm | 1 +
src/test/plugin_properties_test.pm | 5 +++--
15 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index 72cb39b7..3f5181fd 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -83,6 +83,7 @@ sub options {
return {
path => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CIFSPlugin.pm b/src/PVE/Storage/CIFSPlugin.pm
index 15d5a22d..594ec1de 100644
--- a/src/PVE/Storage/CIFSPlugin.pm
+++ b/src/PVE/Storage/CIFSPlugin.pm
@@ -189,6 +189,7 @@ sub options {
share => { fixed => 1 },
subdir => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/CephFSPlugin.pm b/src/PVE/Storage/CephFSPlugin.pm
index 39b9686b..19a1f7ab 100644
--- a/src/PVE/Storage/CephFSPlugin.pm
+++ b/src/PVE/Storage/CephFSPlugin.pm
@@ -164,6 +164,7 @@ sub options {
'content-dirs' => { optional => 1 },
monhost => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
subdir => { optional => 1 },
disable => { optional => 1 },
options => { optional => 1 },
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 397d4fe7..56c5d785 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -76,6 +76,7 @@ sub options {
path => { fixed => 1 },
'content-dirs' => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm
index d57bc704..f1d133ba 100644
--- a/src/PVE/Storage/ISCSIDirectPlugin.pm
+++ b/src/PVE/Storage/ISCSIDirectPlugin.pm
@@ -77,6 +77,7 @@ sub options {
portal => { fixed => 1 },
target => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm
index 1e349cde..9c7b20a4 100644
--- a/src/PVE/Storage/ISCSIPlugin.pm
+++ b/src/PVE/Storage/ISCSIPlugin.pm
@@ -416,6 +416,7 @@ sub options {
portal => { fixed => 1 },
target => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 3cdab9e4..60bc8743 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -949,6 +949,7 @@ sub options {
return {
vgname => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
shared => { optional => 1 },
disable => { optional => 1 },
saferemove => { optional => 1 },
diff --git a/src/PVE/Storage/LvmThinPlugin.pm b/src/PVE/Storage/LvmThinPlugin.pm
index 67f7bb2f..74817e77 100644
--- a/src/PVE/Storage/LvmThinPlugin.pm
+++ b/src/PVE/Storage/LvmThinPlugin.pm
@@ -63,6 +63,7 @@ sub options {
thinpool => { fixed => 1 },
vgname => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/PVE/Storage/NFSPlugin.pm b/src/PVE/Storage/NFSPlugin.pm
index fdc32c01..cf42b619 100644
--- a/src/PVE/Storage/NFSPlugin.pm
+++ b/src/PVE/Storage/NFSPlugin.pm
@@ -119,6 +119,7 @@ sub options {
server => { fixed => 1 },
export => { fixed => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
'prune-backups' => { optional => 1 },
'max-protected-backups' => { optional => 1 },
diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
index cf826237..01f8c936 100644
--- a/src/PVE/Storage/PBSPlugin.pm
+++ b/src/PVE/Storage/PBSPlugin.pm
@@ -85,6 +85,7 @@ sub options {
namespace => { optional => 1 },
port => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
username => { optional => 1 },
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 724c2d35..2095e7eb 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -217,6 +217,15 @@ my $defaultData = {
default =>
"Unlimited for users with Datastore.Allocate privilege, 5 for other users",
},
+ 'backup-concurrency' => {
+ description =>
+ "Maximal number of guest backups using this storage at the same time,"
+ . " either as backup target or for guest disks. Counted for the whole cluster if"
+ . " the storage is shared, for each node otherwise.",
+ type => 'integer',
+ minimum => 1,
+ optional => 1,
+ },
shared => {
title => "Shared",
description =>
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index 26a1cf81..f4bc702f 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -482,6 +482,7 @@ sub properties {
sub options {
return {
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
monhost => { optional => 1 },
pool => { optional => 1 },
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 732069e0..69316e7d 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -247,6 +247,7 @@ sub properties {
sub options {
return {
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
portal => { fixed => 1 },
target => { fixed => 1 },
diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlugin.pm
index 1085db94..6c47dc48 100644
--- a/src/PVE/Storage/ZFSPoolPlugin.pm
+++ b/src/PVE/Storage/ZFSPoolPlugin.pm
@@ -106,6 +106,7 @@ sub options {
blocksize => { optional => 1 },
sparse => { optional => 1 },
nodes => { optional => 1 },
+ 'backup-concurrency' => { optional => 1 },
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
diff --git a/src/test/plugin_properties_test.pm b/src/test/plugin_properties_test.pm
index d8e336d1..25c1a023 100644
--- a/src/test/plugin_properties_test.pm
+++ b/src/test/plugin_properties_test.pm
@@ -12,6 +12,7 @@ use PVE::Storage::Plugin;
# none can be removed, renamed, narrowed or left unexposed by an in-tree plugin that isolates.
my $available = {
authsupported => 'string',
+ 'backup-concurrency' => 'integer minimum=1',
base => 'string:pve-volume-id',
blocksize => 'string:pve-storage-zfs-blocksize',
bwlimit => 'string:{clone,default,migration,move,restore}',
@@ -137,8 +138,8 @@ for my $name (sort keys $available->%*) {
# adding one can break third-party plugins. A property belongs here only if methods of the base
# plugin or code outside of the plugins read it from storages of any type.
my $base = [qw(
- base bwlimit content content-dirs create-base-path create-subdirs disable format
- max-protected-backups mkdir nodes path preallocation prune-backups shared
+ backup-concurrency base bwlimit content content-dirs create-base-path create-subdirs disable
+ format max-protected-backups mkdir nodes path preallocation prune-backups shared
snapshot-as-volume-chain storage type
)];
my $declared = PVE::Storage::Plugin->base_properties();
--
2.47.3
next prev parent reply other threads:[~2026-11-18 13:32 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118132947.1093069-4-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.