public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits
Date: Wed, 18 Nov 2026 14:18:44 +0100	[thread overview]
Message-ID: <20261118132947.1093069-21-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>

Describe how many guests a node backs up at the same time and that the
bandwidth limit applies to each of them, how jobs can run alongside
each other, and the cluster-wide and storage limits, including how to
size them and what happens with slots of an offline node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- jobs to an external backup provider ignore 'concurrency' and 'overlap'
- a job starts no other guests while one waits for cluster-wide slots
- say what happens once a wait ends, which limits need all nodes to be
  upgraded, and that overlapping jobs share the node limit

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..241c38c 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,122 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_concurrency]]
+Concurrent Backups
+------------------
+
+By default, a node runs one backup job at a time and backs up its guests one
+after the other. A job that is due while another one runs waits for it, up to
+the `lockwait` time. As each node backs up its own guests, how long a job takes
+and how much load it puts on the backup target depend on how its guests are
+spread over the nodes. The settings described here allow backing up several
+guests of a node at the same time, running jobs alongside each other, and
+limiting the load on shared storages and backup servers.
+
+Node Limit and Job Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backup concurrency of a node limits how many guests it backs up at the same
+time, across all jobs. Set the default for all nodes in the datacenter options
+(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in
+their options (`backup-concurrency` in the node configuration):
+
+----
+# pvesh set /cluster/options --backup-concurrency 4
+# pvenode config set --backup-concurrency 8
+----
+
+A job backs up as many guests at the same time as its node allows. With the
+`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use
+fewer, but not more.
+
+Size the node limit by what the node itself can handle, that is, its CPU,
+memory, network connection, and local disks. Each parallel guest backup runs its
+own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to
+each guest backup on its own, so the bandwidth of a node can add up to the
+configured limit times the number of guests backed up at the same time.
+
+Jobs Running Alongside Each Other
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same
+time as other jobs that have it enabled, for example, a short hourly job
+alongside a long nightly one. Together, they stay within the backup concurrency
+of the node, so with the default of one, still only one guest is backed up at a
+time. A job without the option still runs alone, and while it waits for running
+jobs to finish, overlapping jobs that are due later do not start before it.
+Waiting for a free backup slot of the node is not limited by `lockwait`.
+
+If two running jobs include the same guest, the second one backs it up once the
+first one is done with it, or skips it after waiting `lockwait` minutes.
+
+Jobs to an external backup provider ignore the `concurrency` and `overlap`
+options. They back up one guest at a time and always run alone.
+
+NOTE: The job phases of hook scripts can overlap for overlapping jobs. For
+example, a hook script that mounts a disk in the `job-start` phase and unmounts
+it in the `job-end` phase must not be used for overlapping jobs. With a job
+concurrency above one, the phases of different guests of a job overlap as well.
+
+Cluster-wide and Storage Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each node runs its part of a job on its own, so in a cluster, shared storages
+and backup servers get the backups of all nodes at the same time. To protect
+them, you can limit how many guest backups run at the same time:
+
+* for the whole cluster, with the `cluster` key of `backup-concurrency` in the
+  datacenter options ('Cluster-Wide' in the UI),
+* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in
+  the advanced storage settings). It counts guest backups that use the storage
+  as backup target or for included guest disks, but not for fleecing or
+  temporary files. For a shared storage, the limit applies to the whole cluster,
+  for a local storage to each node.
+
+----
+# pvesh set /cluster/options --backup-concurrency node=8,cluster=12
+# pvesm set pbs-main --backup-concurrency 6
+----
+
+A guest backup only starts once all limits it counts against have room. Size
+the cluster-wide and storage limits by what the shared systems can handle,
+independent of the node limits. With such limits in place, raising the node
+limits cannot overload the shared systems.
+
+TIP: Several storage entries can refer to the same system, for example,
+namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits
+count separately, so use the cluster-wide limit to bound their sum.
+
+The cluster-wide limit and the limits of shared storages are counted in
+cluster-wide slots, which guest backups get in the order they started waiting.
+One that waits for a storage at its limit lets others with room go first, but
+only a few times. A job has at most one guest waiting for cluster-wide slots at
+a time and starts no other guests meanwhile. It waits for these slots at most
+`lockwait` minutes in total and skips guests that did not get them by then. Like
+for node slots, waiting for free slots of a local storage is not limited by
+`lockwait`. The task log tells which limit a guest waits for.
+
+To see which guest backups hold or wait for cluster-wide slots, use:
+
+----
+# pvesh get /cluster/backup-info/slots
+----
+
+Granting cluster-wide slots requires quorum, and the cluster-wide and shared
+storage limits are only enforced once all online nodes of the cluster run a
+version that supports them. Until then, the task log warns that they are not
+enforced. Guest backups that started before that, or while neither a
+cluster-wide limit nor a limit on a shared storage was set, do not count against
+them. Slots of a node that goes offline stay taken, as its guest backups might
+still be running, and the node releases them once it is back. Guest backups that
+wait for such slots name the node in the task log. Once you made sure that its
+backups stopped, for example because the node is powered off, you can release
+its slots manually:
+
+----
+# pvesh delete /cluster/backup-info/slots/NODE
+----
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





      parent reply	other threads:[~2026-11-18 13:32 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118132947.1093069-21-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal