all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits
Date: Wed, 18 Nov 2026 14:18:44 +0100	[thread overview]
Message-ID: <20261118132947.1093069-21-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>

Describe how many guests a node backs up at the same time and that the
bandwidth limit applies to each of them, how jobs can run alongside
each other, and the cluster-wide and storage limits, including how to
size them and what happens with slots of an offline node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- jobs to an external backup provider ignore 'concurrency' and 'overlap'
- a job starts no other guests while one waits for cluster-wide slots
- say what happens once a wait ends, which limits need all nodes to be
  upgraded, and that overlapping jobs share the node limit

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..241c38c 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,122 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_concurrency]]
+Concurrent Backups
+------------------
+
+By default, a node runs one backup job at a time and backs up its guests one
+after the other. A job that is due while another one runs waits for it, up to
+the `lockwait` time. As each node backs up its own guests, how long a job takes
+and how much load it puts on the backup target depend on how its guests are
+spread over the nodes. The settings described here allow backing up several
+guests of a node at the same time, running jobs alongside each other, and
+limiting the load on shared storages and backup servers.
+
+Node Limit and Job Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backup concurrency of a node limits how many guests it backs up at the same
+time, across all jobs. Set the default for all nodes in the datacenter options
+(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in
+their options (`backup-concurrency` in the node configuration):
+
+----
+# pvesh set /cluster/options --backup-concurrency 4
+# pvenode config set --backup-concurrency 8
+----
+
+A job backs up as many guests at the same time as its node allows. With the
+`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use
+fewer, but not more.
+
+Size the node limit by what the node itself can handle, that is, its CPU,
+memory, network connection, and local disks. Each parallel guest backup runs its
+own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to
+each guest backup on its own, so the bandwidth of a node can add up to the
+configured limit times the number of guests backed up at the same time.
+
+Jobs Running Alongside Each Other
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same
+time as other jobs that have it enabled, for example, a short hourly job
+alongside a long nightly one. Together, they stay within the backup concurrency
+of the node, so with the default of one, still only one guest is backed up at a
+time. A job without the option still runs alone, and while it waits for running
+jobs to finish, overlapping jobs that are due later do not start before it.
+Waiting for a free backup slot of the node is not limited by `lockwait`.
+
+If two running jobs include the same guest, the second one backs it up once the
+first one is done with it, or skips it after waiting `lockwait` minutes.
+
+Jobs to an external backup provider ignore the `concurrency` and `overlap`
+options. They back up one guest at a time and always run alone.
+
+NOTE: The job phases of hook scripts can overlap for overlapping jobs. For
+example, a hook script that mounts a disk in the `job-start` phase and unmounts
+it in the `job-end` phase must not be used for overlapping jobs. With a job
+concurrency above one, the phases of different guests of a job overlap as well.
+
+Cluster-wide and Storage Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each node runs its part of a job on its own, so in a cluster, shared storages
+and backup servers get the backups of all nodes at the same time. To protect
+them, you can limit how many guest backups run at the same time:
+
+* for the whole cluster, with the `cluster` key of `backup-concurrency` in the
+  datacenter options ('Cluster-Wide' in the UI),
+* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in
+  the advanced storage settings). It counts guest backups that use the storage
+  as backup target or for included guest disks, but not for fleecing or
+  temporary files. For a shared storage, the limit applies to the whole cluster,
+  for a local storage to each node.
+
+----
+# pvesh set /cluster/options --backup-concurrency node=8,cluster=12
+# pvesm set pbs-main --backup-concurrency 6
+----
+
+A guest backup only starts once all limits it counts against have room. Size
+the cluster-wide and storage limits by what the shared systems can handle,
+independent of the node limits. With such limits in place, raising the node
+limits cannot overload the shared systems.
+
+TIP: Several storage entries can refer to the same system, for example,
+namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits
+count separately, so use the cluster-wide limit to bound their sum.
+
+The cluster-wide limit and the limits of shared storages are counted in
+cluster-wide slots, which guest backups get in the order they started waiting.
+One that waits for a storage at its limit lets others with room go first, but
+only a few times. A job has at most one guest waiting for cluster-wide slots at
+a time and starts no other guests meanwhile. It waits for these slots at most
+`lockwait` minutes in total and skips guests that did not get them by then. Like
+for node slots, waiting for free slots of a local storage is not limited by
+`lockwait`. The task log tells which limit a guest waits for.
+
+To see which guest backups hold or wait for cluster-wide slots, use:
+
+----
+# pvesh get /cluster/backup-info/slots
+----
+
+Granting cluster-wide slots requires quorum, and the cluster-wide and shared
+storage limits are only enforced once all online nodes of the cluster run a
+version that supports them. Until then, the task log warns that they are not
+enforced. Guest backups that started before that, or while neither a
+cluster-wide limit nor a limit on a shared storage was set, do not count against
+them. Slots of a node that goes offline stay taken, as its guest backups might
+still be running, and the node releases them once it is back. Guest backups that
+wait for such slots name the node in the task log. Once you made sure that its
+backups stopped, for example because the node is powered off, you can release
+its slots manually:
+
+----
+# pvesh delete /cluster/backup-info/slots/NODE
+----
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





      parent reply	other threads:[~2026-11-18 13:32 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118132947.1093069-21-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal