public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
@ 2026-11-18 13:18 Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
                   ` (19 more replies)
  0 siblings, 20 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.

changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
  shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
  than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
  quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
  also explain the review points not followed

[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/


Original cover letter
=====================

A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.

This series adds:

 - parallel guest backups within a job, up to a per-node backup
   concurrency, set as datacenter default or per node; a job can lower
   it with its new 'concurrency' option,
 - the 'overlap' job option to run alongside other such jobs, together
   within the node's backup concurrency,
 - cluster-wide and per-storage limits that protect shared storages and
   backup servers from too many guest backups at the same time.

The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.

The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.

Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.

Repo (build-)dependencies:
 pve-manager -> pve-guest-common, pve-cluster, pve-storage

A separate series on top adds a start window option for backup jobs.

Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.

pve-cluster:

Thomas Lamprecht (2):
      datacenter config: add backup concurrency setting
      datacenter config: backup concurrency: add cluster-wide limit

 src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

pve-storage:

Thomas Lamprecht (1):
      add backup concurrency option to storages

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 src/test/plugin_properties_test.pm   | 5 +++--
 15 files changed, 25 insertions(+), 2 deletions(-)

pve-guest-common:

Thomas Lamprecht (1):
      vzdump: add options for parallel guest backups and overlapping jobs

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

pve-manager:

Thomas Lamprecht (15):
      vzdump: drop never-set fallback for the lock wait time
      vzdump: notification: capture guest logs right after each backup
      vzdump: stop all running backup jobs of the node
      vzdump: show the error if a job could not get the global lock
      vzdump: run job-init hook and set up target once the job may start
      vzdump: list guests as skipped if the job could not start
      vzdump: back up guests of a job in parallel
      vzdump: allow jobs to run alongside each other
      ui: options: add backup concurrency for datacenter and nodes
      ui: backup job: add concurrency and overlap options
      ui: backup now: add overlap option
      vzdump: limit concurrent guest backups cluster-wide
      vzdump: honor backup concurrency limits of storages
      api: backup info: add endpoints for cluster-wide backup slots
      ui: add cluster-wide and storage backup concurrency limits

 PVE/API2/Cluster/BackupInfo.pm              | 106 ++-
 PVE/API2/VZDump.pm                          |  14 +-
 PVE/Makefile                                |   2 +-
 PVE/NodeConfig.pm                           |   7 +
 PVE/Service/pvescheduler.pm                 |   4 +
 PVE/VZDump.pm                               | 893 ++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm                  | 296 +++++++
 PVE/VZDump/Makefile                         |  15 +
 debian/tmpfiles                             |   5 +-
 test/Makefile                               |  10 +-
 test/vzdump_cluster_slots_test.pl           | 282 +++++++
 test/vzdump_concurrency_test.pl             |  89 ++
 test/vzdump_new_test.pl                     |   1 +
 www/manager6/dc/OptionView.js               |  22 +
 www/manager6/node/NodeOptionsView.js        |   8 +
 www/manager6/panel/BackupAdvancedOptions.js |  40 +-
 www/manager6/storage/Base.js                |  17 +
 www/manager6/window/Backup.js               |  28 +-
 18 files changed, 1773 insertions(+), 66 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      vzdump: document concurrent backups and their limits

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

-- 
2.47.3





^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2026-11-18 13:33 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal