From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id D0B261FF09D for ; Mon, 02 Nov 2026 00:01:12 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4812E2186F; Sun, 01 Nov 2026 23:59:24 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH docs 19/19] vzdump: document concurrent backups and their limits Date: Sun, 1 Nov 2026 23:58:46 +0100 Message-ID: <20261101225849.884587-20-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261101225849.884587-1-t.lamprecht@proxmox.com> References: <20261101225849.884587-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793573948958 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.236 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: K3LYQSQTLSMENKQI3IFRIODVJTG7LF6M X-Message-ID-Hash: K3LYQSQTLSMENKQI3IFRIODVJTG7LF6M X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Describe how many guests a node backs up at the same time and that the bandwidth limit applies to each of them, how jobs can run alongside each other, and the cluster-wide and storage limits, including how to size them and what happens with slots of an offline node. Signed-off-by: Thomas Lamprecht --- vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 44db027..6e65c29 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -308,6 +308,114 @@ the `performance` setting, `max-workers` (affects VM backups only) and `pbs-entries-max` (affects container backups only). See the xref:vzdump_configuration[configuration options] for details. +[[vzdump_concurrency]] +Concurrent Backups +------------------ + +By default, a node runs one backup job at a time and backs up its guests one +after the other. A job that is due while another one runs waits for it, up to +the `lockwait` time. As each node backs up its own guests, how long a job takes +and how much load it puts on the backup target depend on how its guests are +spread over the nodes. The settings described here allow backing up several +guests of a node at the same time, running jobs alongside each other, and +limiting the load on shared storages and backup servers. + +Node Limit and Job Concurrency +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The backup concurrency of a node limits how many guests it backs up at the same +time, across all jobs. Set the default for all nodes in the datacenter options +(`backup-concurrency` in `datacenter.cfg`), and override it for single nodes in +their options (`backup-concurrency` in the node configuration): + +---- +# pvesh set /cluster/options --backup-concurrency 4 +# pvenode config set --backup-concurrency 8 +---- + +A job backs up as many guests at the same time as its node allows. With the +`concurrency` job option ('Concurrency' in the 'Advanced' tab), a job can use +fewer, but not more. + +Size the node limit by what the node itself can handle, that is, its CPU, +memory, network connection, and local disks. Each parallel guest backup runs its +own compressor or Proxmox Backup Server client. The `bwlimit` setting applies to +each guest backup on its own, so the bandwidth of a node can add up to the +configured limit times the number of guests backed up at the same time. + +Jobs Running Alongside Each Other +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Jobs with the `overlap` option ('Overlap' in the 'Advanced' tab) run at the same +time as other jobs that have it enabled, for example, a short hourly job +alongside a long nightly one. Together, they stay within the backup concurrency +of the node. A job without the option still runs alone, and while it waits for +running jobs to finish, overlapping jobs that are due later do not start before +it. Waiting for a free backup slot of the node is not limited by `lockwait`. + +If two running jobs include the same guest, the second one backs it up once the +first one is done with it, waiting at most `lockwait` minutes. + +NOTE: The job phases of hook scripts can overlap for overlapping jobs. For +example, a hook script that mounts a disk in the `job-start` phase and unmounts +it in the `job-end` phase must not be used for overlapping jobs. With a job +concurrency above one, the phases of different guests of a job overlap as well. + +Cluster-wide and Storage Limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each node runs its part of a job on its own, so in a cluster, shared storages +and backup servers get the backups of all nodes at the same time. To protect +them, you can limit how many guest backups run at the same time: + +* for the whole cluster, with the `cluster` key of `backup-concurrency` in the + datacenter options ('Cluster-wide' in the UI), +* for a storage, with its `backup-concurrency` option ('Backup Concurrency' in + the advanced storage settings). It counts guest backups that use the storage + as backup target or for included guest disks, but not for fleecing or + temporary files. For a shared storage, the limit applies to the whole cluster, + for a local storage to each node. + +---- +# pvesh set /cluster/options --backup-concurrency node=8,cluster=12 +# pvesm set pbs-main --backup-concurrency 6 +---- + +A guest backup only starts once all limits it counts against have room. Size +the cluster-wide and storage limits by what the shared systems can handle, +independent of the node limits. With such limits in place, raising the node +limits cannot overload the shared systems, so the node limits only need to fit +what each node can handle. + +TIP: Several storage entries can refer to the same system, for example, +namespaces of a Proxmox Backup Server or pools of a Ceph cluster. Their limits +count separately, so use the cluster-wide limit to bound their sum. + +Guest backups get cluster-wide slots in the order they started waiting. One that +waits for a storage at its limit lets others with room go first, but only a few +times. A job has at most one guest waiting for cluster-wide slots at a time, and +waits for them at most `lockwait` minutes in total. Like for node slots, waiting +for free slots of a local storage is not limited by `lockwait`. The task log +tells which limit a guest waits for. +To see which guest backups hold or wait for cluster-wide slots, use: + +---- +# pvesh get /cluster/backup-info/slots +---- + +Granting cluster-wide slots requires quorum, and the limits are only enforced +once all online nodes of the cluster run a version that supports them. Guest +backups that started before that, or while neither a cluster-wide limit nor a +limit on a shared storage was set, do not count against them. Slots of a node +that goes offline stay taken, as its guest backups might still be running, and +the node releases them once it is back. Once you made sure that these backups +stopped, for example because the node is powered off, you can release its slots +manually: + +---- +# pvesh delete /cluster/backup-info/slots/NODE +---- + [[vzdump_retention]] Backup Retention ---------------- -- 2.47.3