all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits
@ 2026-11-18 13:18 Thomas Lamprecht
  2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
                   ` (19 more replies)
  0 siblings, 20 replies; 21+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 13:18 UTC (permalink / raw)
  To: pve-devel

This is v2 of the series [0], addressing Fiona's review (thanks!) and
some additional findings since v1. Patch 16 now uses
node_pvecfg_version_at_least() from pve-cluster [1], which got applied
in the meantime.

changes v1 -> v2:
- drop the container patch, it got applied
- new patches 5 and 15
- fix the node version check, which failed jobs with a cluster-wide or
  shared storage limit on clusters with qemu-server 9.2.15 or newer
- after lowering a limit, guest backups start again as soon as fewer
  than the new limit run, and waiting jobs pick up changed limits
- less noise in the task log while waiting, and better handling of
  quorum loss, offline nodes and an unparsable slot table
- rebase onto current master
- details are in the per-patch changelogs, those of patches 6 and 11
  also explain the review points not followed

[0]: https://lore.proxmox.com/pve-devel/20261101225849.884587-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261103124831.139399-3-f.ebner@proxmox.com/


Original cover letter
=====================

A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.

This series adds:

 - parallel guest backups within a job, up to a per-node backup
   concurrency, set as datacenter default or per node; a job can lower
   it with its new 'concurrency' option,
 - the 'overlap' job option to run alongside other such jobs, together
   within the node's backup concurrency,
 - cluster-wide and per-storage limits that protect shared storages and
   backup servers from too many guest backups at the same time.

The new settings are opt-in: without them, each node still backs up one
guest at a time and jobs do not overlap.

The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. The slot table is only used once all online nodes run a version
that supports it; until then, the cluster-wide and shared storage limits
are not enforced and the task log warns.

Patches 5 to 10 fix or improve existing behavior, for example stopping
all backup jobs of a node, and naming the error and the skipped guests
of a job that could not start. They could be applied ahead of the rest.

Repo (build-)dependencies:
 pve-manager -> pve-guest-common, pve-cluster, pve-storage

A separate series on top adds a start window option for backup jobs.

Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.

pve-cluster:

Thomas Lamprecht (2):
      datacenter config: add backup concurrency setting
      datacenter config: backup concurrency: add cluster-wide limit

 src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

pve-storage:

Thomas Lamprecht (1):
      add backup concurrency option to storages

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 src/test/plugin_properties_test.pm   | 5 +++--
 15 files changed, 25 insertions(+), 2 deletions(-)

pve-guest-common:

Thomas Lamprecht (1):
      vzdump: add options for parallel guest backups and overlapping jobs

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

pve-manager:

Thomas Lamprecht (15):
      vzdump: drop never-set fallback for the lock wait time
      vzdump: notification: capture guest logs right after each backup
      vzdump: stop all running backup jobs of the node
      vzdump: show the error if a job could not get the global lock
      vzdump: run job-init hook and set up target once the job may start
      vzdump: list guests as skipped if the job could not start
      vzdump: back up guests of a job in parallel
      vzdump: allow jobs to run alongside each other
      ui: options: add backup concurrency for datacenter and nodes
      ui: backup job: add concurrency and overlap options
      ui: backup now: add overlap option
      vzdump: limit concurrent guest backups cluster-wide
      vzdump: honor backup concurrency limits of storages
      api: backup info: add endpoints for cluster-wide backup slots
      ui: add cluster-wide and storage backup concurrency limits

 PVE/API2/Cluster/BackupInfo.pm              | 106 ++-
 PVE/API2/VZDump.pm                          |  14 +-
 PVE/Makefile                                |   2 +-
 PVE/NodeConfig.pm                           |   7 +
 PVE/Service/pvescheduler.pm                 |   4 +
 PVE/VZDump.pm                               | 893 ++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm                  | 296 +++++++
 PVE/VZDump/Makefile                         |  15 +
 debian/tmpfiles                             |   5 +-
 test/Makefile                               |  10 +-
 test/vzdump_cluster_slots_test.pl           | 282 +++++++
 test/vzdump_concurrency_test.pl             |  89 ++
 test/vzdump_new_test.pl                     |   1 +
 www/manager6/dc/OptionView.js               |  22 +
 www/manager6/node/NodeOptionsView.js        |   8 +
 www/manager6/panel/BackupAdvancedOptions.js |  40 +-
 www/manager6/storage/Base.js                |  17 +
 www/manager6/window/Backup.js               |  28 +-
 18 files changed, 1773 insertions(+), 66 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      vzdump: document concurrent backups and their limits

 vzdump.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

-- 
2.47.3





^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2026-11-18 13:33 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal