[PATCH docs 19/19] vzdump: document concurrent backups and their limits

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Describe how many guests a node backs up at the same time and that the
bandwidth limit applies to each of them, how jobs can run alongside
each other, and the cluster-wide and storage limits, including how to
size them and what happens with slots of an offline node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 108 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..6e65c29 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,114 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_concurrency]]
+Concurrent Backups
+------------------
+
+By default, a node runs one backup job at a time and backs up its guests one
+after the other. A job that is due while another one runs waits for it, up to
+the `lockwait` time. As each node backs up its own guests, how long a job takes
+and how much load it puts on the backup target depend on how its guests are
+spread over the nodes. The settings described here allow backing up several
+guests of a node at the same time, running jobs alongside each other, and
+limiting the load on shared storages and backup servers.
+
+Node Limit and Job Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backup concurrency of a node limits how many guests it backs up at the same
+time, across all jobs. Set the default for all nodes in the datacenter options
+(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in
+their options (`backup-concurrency` in the node configuration):
+
+----
+# pvesh set /cluster/options --backup-concurrency 4
+# pvenode config set --backup-concurrency 8
+----
+
+A job backs up as many guests at the same time as its node allows. With the
+`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use
+fewer, but not more.
+
+Size the node limit by what the node itself can handle, that is, its CPU,
+memory, network connection, and local disks. Each parallel guest backup runs its
+own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to
+each guest backup on its own, so the bandwidth of a node can add up to the
+configured limit times the number of guests backed up at the same time.
+
+Jobs Running Alongside Each Other
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same
+time as other jobs that have it enabled, for example, a short hourly job
+alongside a long nightly one. Together, they stay within the backup concurrency
+of the node. A job without the option still runs alone, and while it waits for
+running jobs to finish, overlapping jobs that are due later do not start before
+it. Waiting for a free backup slot of the node is not limited by `lockwait`.
+
+If two running jobs include the same guest, the second one backs it up once the
+first one is done with it, waiting at most `lockwait` minutes.
+
+NOTE: The job phases of hook scripts can overlap for overlapping jobs. For
+example, a hook script that mounts a disk in the `job-start` phase and unmounts
+it in the `job-end` phase must not be used for overlapping jobs. With a job
+concurrency above one, the phases of different guests of a job overlap as well.
+
+Cluster-wide and Storage Limits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each node runs its part of a job on its own, so in a cluster, shared storages
+and backup servers get the backups of all nodes at the same time. To protect
+them, you can limit how many guest backups run at the same time:
+
+* for the whole cluster, with the `cluster` key of `backup-concurrency` in the
+  datacenter options ('Cluster-wide' in the UI),
+* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in
+  the advanced storage settings). It counts guest backups that use the storage
+  as backup target or for included guest disks, but not for fleecing or
+  temporary files. For a shared storage, the limit applies to the whole cluster,
+  for a local storage to each node.
+
+----
+# pvesh set /cluster/options --backup-concurrency node=8,cluster=12
+# pvesm set pbs-main --backup-concurrency 6
+----
+
+A guest backup only starts once all limits it counts against have room. Size
+the cluster-wide and storage limits by what the shared systems can handle,
+independent of the node limits. With such limits in place, raising the node
+limits cannot overload the shared systems, so the node limits only need to fit
+what each node can handle.
+
+TIP: Several storage entries can refer to the same system, for example,
+namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits
+count separately, so use the cluster-wide limit to bound their sum.
+
+Guest backups get cluster-wide slots in the order they started waiting. One that
+waits for a storage at its limit lets others with room go first, but only a few
+times. A job has at most one guest waiting for cluster-wide slots at a time, and
+waits for them at most `lockwait` minutes in total. Like for node slots, waiting
+for free slots of a local storage is not limited by `lockwait`. The task log
+tells which limit a guest waits for.
+To see which guest backups hold or wait for cluster-wide slots, use:
+
+----
+# pvesh get /cluster/backup-info/slots
+----
+
+Granting cluster-wide slots requires quorum, and the limits are only enforced
+once all online nodes of the cluster run a version that supports them. Guest
+backups that started before that, or while neither a cluster-wide limit nor a
+limit on a shared storage was set, do not count against them. Slots of a node
+that goes offline stay taken, as its guest backups might still be running, and
+the node releases them once it is back. Once you made sure that these backups
+stopped, for example because the node is powered off, you can release its slots
+manually:
+
+----
+# pvesh delete /cluster/backup-info/slots/NODE
+----
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3