public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH guest-common 04/19] vzdump: add options for parallel guest backups and overlapping jobs
Date: Sun,  1 Nov 2026 23:58:31 +0100	[thread overview]
Message-ID: <20261101225849.884587-5-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261101225849.884587-1-t.lamprecht@proxmox.com>

A node only ever runs one backup job, and that job backs up its guests
one after the other. How long a job takes thus depends on how its
guests happen to be spread over the nodes, and a short job has to wait
until a long one finished, failing once the lock wait time is
exceeded.

The new 'concurrency' option allows backing up several guests of a job
at the same time, up to the backup concurrency of the node. The
bandwidth limit then applies to each of them, as its description now
says. The 'overlap' option lets a job run alongside other jobs that
allow it as well. It is off by default, as existing hook scripts may
rely on job phases never overlapping.

With overlapping jobs the same guest can be part of two running
backups, and with cluster-wide limits a guest backup can wait for a
slot, so the lock wait now also bounds these waits.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 67a4cf4..f98c9ae 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -297,7 +297,7 @@ my $confdesc = {
     },
     bwlimit => {
         type => 'integer',
-        description => "Limit I/O bandwidth (in KiB/s).",
+        description => "Limit I/O bandwidth of each guest backup (in KiB/s).",
         optional => 1,
         minimum => 0,
         default => 0,
@@ -327,11 +327,27 @@ my $confdesc = {
     },
     lockwait => {
         type => 'integer',
-        description => "Maximal time to wait for the global lock (minutes).",
+        description =>
+            "Maximal time to wait for the global lock, for guests that another backup"
+            . " is currently processing, and for cluster-wide backup slots (minutes).",
         optional => 1,
         minimum => 0,
         default => 3 * 60, # 3 hours
     },
+    concurrency => {
+        type => 'integer',
+        description => "Maximal number of guests to back up at the same time on each node."
+            . " Defaults to, and is limited by, the backup concurrency configured for the node.",
+        optional => 1,
+        minimum => 1,
+    },
+    overlap => {
+        type => 'boolean',
+        description => "Allow running at the same time as other backups on the same node that"
+            . " also allow it. The job phases of their hook scripts can then overlap.",
+        optional => 1,
+        default => 0,
+    },
     stopwait => {
         type => 'integer',
         description => "Maximal time to wait until a guest system is stopped (minutes).",
-- 
2.47.3





  parent reply	other threads:[~2026-11-01 23:00 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-01 22:58 [PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 01/19] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-01 22:58 ` [PATCH cluster 02/19] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-01 22:58 ` [PATCH storage 03/19] add backup concurrency option to storages Thomas Lamprecht
2026-11-01 22:58 ` Thomas Lamprecht [this message]
2026-11-01 22:58 ` [PATCH container 05/19] vzdump: honor bandwidth limit for backups to Proxmox Backup Server Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 06/19] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 07/19] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 08/19] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 09/19] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 10/19] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 11/19] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 12/19] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 13/19] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 14/19] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 15/19] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 16/19] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 17/19] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-01 22:58 ` [PATCH manager 18/19] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-01 22:58 ` [PATCH docs 19/19] vzdump: document concurrent backups and their limits Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261101225849.884587-5-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal