From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id AFB531FF0C0 for ; Wed, 18 Nov 2026 14:32:09 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1E7A321774; Wed, 18 Nov 2026 14:30:31 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Date: Wed, 18 Nov 2026 14:18:44 +0100 Message-ID: <20261118132947.1093069-21-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com> References: <20261118132947.1093069-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795008593318 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.021 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: XF3E6NR2NVOH52RNIKFZKJJYHKPV2ZLE X-Message-ID-Hash: XF3E6NR2NVOH52RNIKFZKJJYHKPV2ZLE X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Describe how many guests a node backs up at the same time and that the bandwidth limit applies to each of them, how jobs can run alongside each other, and the cluster-wide and storage limits, including how to size them and what happens with slots of an offline node. Signed-off-by: Thomas Lamprecht --- changes v1 -> v2: - jobs to an external backup provider ignore 'concurrency' and 'overlap' - a job starts no other guests while one waits for cluster-wide slots - say what happens once a wait ends, which limits need all nodes to be upgraded, and that overlapping jobs share the node limit vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 44db027..241c38c 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -308,6 +308,122 @@ the `performance` setting, `max-workers` (affects VM backups only) and `pbs-entries-max` (affects container backups only). See the xref:vzdump_configuration[configuration options] for details. +[[vzdump_concurrency]] +Concurrent Backups +------------------ + +By default, a node runs one backup job at a time and backs up its guests one +after the other. A job that is due while another one runs waits for it, up to +the `lockwait` time. As each node backs up its own guests, how long a job takes +and how much load it puts on the backup target depend on how its guests are +spread over the nodes. The settings described here allow backing up several +guests of a node at the same time, running jobs alongside each other, and +limiting the load on shared storages and backup servers. + +Node Limit and Job Concurrency +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The backup concurrency of a node limits how many guests it backs up at the same +time, across all jobs. Set the default for all nodes in the datacenter options +(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in +their options (`backup-concurrency` in the node configuration): + +---- +# pvesh set /cluster/options --backup-concurrency 4 +# pvenode config set --backup-concurrency 8 +---- + +A job backs up as many guests at the same time as its node allows. With the +`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use +fewer, but not more. + +Size the node limit by what the node itself can handle, that is, its CPU, +memory, network connection, and local disks. Each parallel guest backup runs its +own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to +each guest backup on its own, so the bandwidth of a node can add up to the +configured limit times the number of guests backed up at the same time. + +Jobs Running Alongside Each Other +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same +time as other jobs that have it enabled, for example, a short hourly job +alongside a long nightly one. Together, they stay within the backup concurrency +of the node, so with the default of one, still only one guest is backed up at a +time. A job without the option still runs alone, and while it waits for running +jobs to finish, overlapping jobs that are due later do not start before it. +Waiting for a free backup slot of the node is not limited by `lockwait`. + +If two running jobs include the same guest, the second one backs it up once the +first one is done with it, or skips it after waiting `lockwait` minutes. + +Jobs to an external backup provider ignore the `concurrency` and `overlap` +options. They back up one guest at a time and always run alone. + +NOTE: The job phases of hook scripts can overlap for overlapping jobs. For +example, a hook script that mounts a disk in the `job-start` phase and unmounts +it in the `job-end` phase must not be used for overlapping jobs. With a job +concurrency above one, the phases of different guests of a job overlap as well. + +Cluster-wide and Storage Limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each node runs its part of a job on its own, so in a cluster, shared storages +and backup servers get the backups of all nodes at the same time. To protect +them, you can limit how many guest backups run at the same time: + +* for the whole cluster, with the `cluster` key of `backup-concurrency` in the + datacenter options ('Cluster-Wide' in the UI), +* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in + the advanced storage settings). It counts guest backups that use the storage + as backup target or for included guest disks, but not for fleecing or + temporary files. For a shared storage, the limit applies to the whole cluster, + for a local storage to each node. + +---- +# pvesh set /cluster/options --backup-concurrency node=8,cluster=12 +# pvesm set pbs-main --backup-concurrency 6 +---- + +A guest backup only starts once all limits it counts against have room. Size +the cluster-wide and storage limits by what the shared systems can handle, +independent of the node limits. With such limits in place, raising the node +limits cannot overload the shared systems. + +TIP: Several storage entries can refer to the same system, for example, +namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits +count separately, so use the cluster-wide limit to bound their sum. + +The cluster-wide limit and the limits of shared storages are counted in +cluster-wide slots, which guest backups get in the order they started waiting. +One that waits for a storage at its limit lets others with room go first, but +only a few times. A job has at most one guest waiting for cluster-wide slots at +a time and starts no other guests meanwhile. It waits for these slots at most +`lockwait` minutes in total and skips guests that did not get them by then. Like +for node slots, waiting for free slots of a local storage is not limited by +`lockwait`. The task log tells which limit a guest waits for. + +To see which guest backups hold or wait for cluster-wide slots, use: + +---- +# pvesh get /cluster/backup-info/slots +---- + +Granting cluster-wide slots requires quorum, and the cluster-wide and shared +storage limits are only enforced once all online nodes of the cluster run a +version that supports them. Until then, the task log warns that they are not +enforced. Guest backups that started before that, or while neither a +cluster-wide limit nor a limit on a shared storage was set, do not count against +them. Slots of a node that goes offline stay taken, as its guest backups might +still be running, and the node releases them once it is back. Guest backups that +wait for such slots name the node in the task log. Once you made sure that its +backups stopped, for example because the node is powered off, you can release +its slots manually: + +---- +# pvesh delete /cluster/backup-info/slots/NODE +---- + [[vzdump_retention]] Backup Retention ---------------- -- 2.47.3