[PATCH cluster,storage,guest-common,container,manager,docs 00/19] parallel guest backups and backup concurrency limits

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

A node runs one backup job at a time, and that job backs up its guests one
after the other. This leads to odd load distributions depending on how the
guests of a job are distributed in a cluster. The extreme cases: If all are
on one node all get backed up in series, adding potentially a lot of delay
but being rather lenient on the backup target (storage/server). If all are
spread such that each guest is on a different node all will get backed up
concurrently, being potentially much faster but also putting a lot of load
towards the target. There is no real control over that so depending on
current guest distribution a job might run slow, ok or fast, overload or
underuse targets.

This series adds:

 - parallel guest backups within a job, up to a per-node backup
   concurrency, set as datacenter default or per node; a job can lower
   it with its new 'concurrency' option,
 - the 'overlap' job option to run alongside other such jobs, together
   within the node's backup concurrency,
 - cluster-wide and per-storage limits that protect shared storages and
   backup servers from too many guest backups at the same time.

Without configuration nothing changes: each node backs up one guest at
a time and jobs do not overlap.

The node limit and the limits of local storages use file locks on each
node. The cluster-wide limit and those of shared storages use a slot
table on pmxcfs, changed under a cluster lock. Waiting guest backups
get their slots in order, and one held up by a full storage cannot
starve. These limits are only enforced once all online nodes run a
version that supports them; until then the task log warns.

Patches 5 to 10 fix or improve existing behavior and could be applied
ahead of the rest, for example stopping all backup jobs of a node, and
naming the error and the skipped guests of a job that could not start.

Repo (build-)dependencies:
 pve-manager -> pve-guest-common, pve-cluster, pve-storage

A separate series on top adds a start window option for backup jobs.

Besides the unit tests, I tested this end to end on nested clusters
and on a three node cluster with Ceph and HA, also with PBS and NFS
targets, guests that keep writing during their backup, node failures,
quorum loss, stopped jobs and a rolling upgrade.

pve-cluster:

Thomas Lamprecht (2):
      datacenter config: add backup concurrency setting
      datacenter config: backup concurrency: add cluster-wide limit

 src/PVE/DataCenterConfig.pm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

pve-storage:

Thomas Lamprecht (1):
      add backup concurrency option to storages

 src/PVE/Storage/BTRFSPlugin.pm       | 1 +
 src/PVE/Storage/CIFSPlugin.pm        | 1 +
 src/PVE/Storage/CephFSPlugin.pm      | 1 +
 src/PVE/Storage/DirPlugin.pm         | 1 +
 src/PVE/Storage/ISCSIDirectPlugin.pm | 1 +
 src/PVE/Storage/ISCSIPlugin.pm       | 1 +
 src/PVE/Storage/LVMPlugin.pm         | 1 +
 src/PVE/Storage/LvmThinPlugin.pm     | 1 +
 src/PVE/Storage/NFSPlugin.pm         | 1 +
 src/PVE/Storage/PBSPlugin.pm         | 1 +
 src/PVE/Storage/Plugin.pm            | 9 +++++++++
 src/PVE/Storage/RBDPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPlugin.pm         | 1 +
 src/PVE/Storage/ZFSPoolPlugin.pm     | 1 +
 14 files changed, 22 insertions(+)

pve-guest-common:

Thomas Lamprecht (1):
      vzdump: add options for parallel guest backups and overlapping jobs

 src/PVE/VZDump/Common.pm | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

pve-container:

Thomas Lamprecht (1):
      vzdump: honor bandwidth limit for backups to Proxmox Backup Server

 src/PVE/VZDump/LXC.pm | 1 +
 1 file changed, 1 insertion(+)

pve-manager:

Thomas Lamprecht (13):
      vzdump: notification: capture guest logs right after each backup
      vzdump: stop all backup jobs of the node
      vzdump: show the error if a job could not get the global lock
      vzdump: run job-init hook and set up target once the job may start
      vzdump: list guests as skipped if the job could not start
      vzdump: back up guests of a job in parallel
      vzdump: allow jobs to run alongside each other
      ui: options: add backup concurrency for datacenter and nodes
      ui: backup job: add concurrency and overlap options
      vzdump: limit concurrent guest backups cluster-wide
      vzdump: honor backup concurrency limits of storages
      api: backup info: add endpoints for cluster-wide backup slots
      ui: add cluster-wide and storage backup concurrency limits

 PVE/API2/Cluster/BackupInfo.pm              | 106 ++-
 PVE/API2/VZDump.pm                          |  10 +-
 PVE/Makefile                                |   2 +-
 PVE/NodeConfig.pm                           |   7 +
 PVE/Service/pvescheduler.pm                 |   4 +
 PVE/VZDump.pm                               | 783 ++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm                  | 299 ++++++++
 PVE/VZDump/Makefile                         |  15 +
 test/Makefile                               |  10 +-
 test/vzdump_cluster_slots_test.pl           | 271 +++++++
 test/vzdump_concurrency_test.pl             |  89 +++
 test/vzdump_new_test.pl                     |   1 +
 www/manager6/dc/OptionView.js               |  22 +
 www/manager6/node/NodeOptionsView.js        |   8 +
 www/manager6/panel/BackupAdvancedOptions.js |  40 +-
 www/manager6/storage/Base.js                |  17 +
 16 files changed, 1623 insertions(+), 61 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      vzdump: document concurrent backups and their limits

 vzdump.adoc | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 108 insertions(+)

-- 
2.47.3