public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
Date: Wed, 18 Nov 2026 14:18:24 +0100	[thread overview]
Message-ID: <20261118132947.1093069-1-t.lamprecht@proxmox.com> (raw)

This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.

changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
  shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
  than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
  quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
  also explain the review points not followed

[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/


Original cover letter
=====================

A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.

This series adds:

 - parallel guest backups within a job, up to a per-node backup
   concurrency, set as datacenter default or per node; a job can lower
   it with its new 'concurrency' option,
 - the 'overlap' job option to run alongside other such jobs, together
   within the node's backup concurrency,
 - cluster-wide and per-storage limits that protect shared storages and
   backup servers from too many guest backups at the same time.

The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.

The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.

Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.

Repo (build-)dependencies:
 pve-manager -> pve-guest-common, pve-cluster, pve-storage

A separate series on top adds a start window option for backup jobs.

Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.

pve-cluster:

Thomas Lamprecht (2):
      datacenter config: add backup concurrency setting
      datacenter config: backup concurrency: add cluster-wide limit

 src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

pve-storage:

Thomas Lamprecht (1):
      add backup concurrency option to storages

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 src/test/plugin_properties_test.pm   | 5 +++--
 15 files changed, 25 insertions(+), 2 deletions(-)

pve-guest-common:

Thomas Lamprecht (1):
      vzdump: add options for parallel guest backups and overlapping jobs

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

pve-manager:

Thomas Lamprecht (15):
      vzdump: drop never-set fallback for the lock wait time
      vzdump: notification: capture guest logs right after each backup
      vzdump: stop all running backup jobs of the node
      vzdump: show the error if a job could not get the global lock
      vzdump: run job-init hook and set up target once the job may start
      vzdump: list guests as skipped if the job could not start
      vzdump: back up guests of a job in parallel
      vzdump: allow jobs to run alongside each other
      ui: options: add backup concurrency for datacenter and nodes
      ui: backup job: add concurrency and overlap options
      ui: backup now: add overlap option
      vzdump: limit concurrent guest backups cluster-wide
      vzdump: honor backup concurrency limits of storages
      api: backup info: add endpoints for cluster-wide backup slots
      ui: add cluster-wide and storage backup concurrency limits

 PVE/API2/Cluster/BackupInfo.pm              | 106 ++-
 PVE/API2/VZDump.pm                          |  14 +-
 PVE/Makefile                                |   2 +-
 PVE/NodeConfig.pm                           |   7 +
 PVE/Service/pvescheduler.pm                 |   4 +
 PVE/VZDump.pm                               | 893 ++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm                  | 296 +++++++
 PVE/VZDump/Makefile                         |  15 +
 debian/tmpfiles                             |   5 +-
 test/Makefile                               |  10 +-
 test/vzdump_cluster_slots_test.pl           | 282 +++++++
 test/vzdump_concurrency_test.pl             |  89 ++
 test/vzdump_new_test.pl                     |   1 +
 www/manager6/dc/OptionView.js               |  22 +
 www/manager6/node/NodeOptionsView.js        |   8 +
 www/manager6/panel/BackupAdvancedOptions.js |  40 +-
 www/manager6/storage/Base.js                |  17 +
 www/manager6/window/Backup.js               |  28 +-
 18 files changed, 1773 insertions(+), 66 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      vzdump: document concurrent backups and their limits

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

-- 
2.47.3





             reply	other threads:[~2026-11-18 13:31 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 13:18 Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118132947.1093069-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal