all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide
Date: Wed, 18 Nov 2026 14:18:40 +0100	[thread overview]
Message-ID: <20261118132947.1093069-17-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>

Each node runs its part of a backup job on its own, so shared storages
and backup servers get as many guest backups at the same time as all
nodes together allow.

With the new cluster-wide limit in the datacenter configuration, every
guest backup also needs a slot in a table on the cluster file system
before it starts. The table is only changed under a cluster lock, and
waiting guest backups get their slots in the order they queued up. A
job has at most one guest waiting for a slot, so waiting does not tie
up the whole node, and waits at most the lock wait time in total.

A guest backup gives back its slot when it is done. If that fails, for
example because the backup process got killed, the scheduler of the
node removes the entry once that guest backup is no longer running.
Slots of offline nodes stay taken, as their guest backups might still
run and keep the shared systems busy. A guest that waits for a slot
names these nodes in the task log and, if it gives up, in its skip
reason. Entries of nodes removed from the cluster are dropped. Granting
slots needs quorum; guest backups that already run continue without
it.

A table that cannot be parsed gets replaced by an empty one, with a
warning that includes its content. Guest backups that run at that
point then no longer count against the limits until they end, which is
better than skipping every guest backup that needs a slot until an
admin notices. The scheduler of each node does this within a minute, so
it also happens without waiting backups. A table that cannot be read at
all, for example because of a pmxcfs problem, stops granting instead,
as that is likely to pass.

The limit is only enforced once all other online nodes run a version
that takes part, as told by the version info each node broadcasts.
Older nodes would not count their guest backups, and a node that gets
downgraded broadcasts its older version again.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
- check node versions with the helper from pve-cluster [Fiona]; the
  pvecfg_min_version() of qemu-server used before is gone since
  qemu-server 9.2.15, which failed jobs with a cluster-wide or shared
  storage limit on clusters with more than one node online
- move the slot table out of priv/ [Fiona]
- if a guest lock file exists but cannot be opened, fail the cleanup
  instead of dropping the entry of the guest [Fiona]
- log a slot request error again if it changes [Fiona]
- include up to 1 KiB of the table in parse errors [Fiona]
- replace an unparsable slot table with a warning, instead of skipping
  every guest backup that needs a slot until an admin repairs it
- rely on the lock wait time that the constructor sets [Fiona]
- use v5.36, signatures and PVE::File, also in the test, and no
  multi-line postfix conditions [Fiona]
- pick up changed limits while waiting, like in patch 12
- reject non-ASCII digits and trailing newlines in slot table counts
- name offline nodes that hold slots in the wait message and in the
  skip reason of a guest that waits for a slot
- retry a failed slot request only every 30 seconds, as each attempt
  can block and log for 10 seconds, for example without quorum
- $min_version is now 9.2.31

NOTE: $min_version assumes that this lands with pve-manager 9.2.31;
adapt it to the first release that contains it.

 PVE/Makefile                      |   2 +-
 PVE/Service/pvescheduler.pm       |   4 +
 PVE/VZDump.pm                     | 189 +++++++++++++++++++++--
 PVE/VZDump/ClusterSlots.pm        | 247 ++++++++++++++++++++++++++++++
 PVE/VZDump/Makefile               |  15 ++
 test/Makefile                     |   6 +-
 test/vzdump_cluster_slots_test.pl | 205 +++++++++++++++++++++++++
 7 files changed, 655 insertions(+), 13 deletions(-)
 create mode 100644 PVE/VZDump/ClusterSlots.pm
 create mode 100644 PVE/VZDump/Makefile
 create mode 100755 test/vzdump_cluster_slots_test.pl

diff --git a/PVE/Makefile b/PVE/Makefile
index efcb250d0..ffdd3a8fd 100644
--- a/PVE/Makefile
+++ b/PVE/Makefile
@@ -1,6 +1,6 @@
 include ../defines.mk
 
-SUBDIRS=API2 Status CLI Service Ceph Jobs
+SUBDIRS=API2 Status CLI Service Ceph Jobs VZDump
 
 PERLSOURCE = 			\
 	API2.pm			\
diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm
index be33db6ed..2af72328f 100755
--- a/PVE/Service/pvescheduler.pm
+++ b/PVE/Service/pvescheduler.pm
@@ -7,6 +7,7 @@ use POSIX qw(WNOHANG SIGCHLD SIGHUP SIG_BLOCK SIG_SETMASK);
 
 use PVE::Jobs;
 use PVE::SafeSyslog;
+use PVE::VZDump;
 
 use PVE::API2::Replication;
 
@@ -125,6 +126,9 @@ sub run {
         $fork->(
             'jobs',
             sub {
+                eval { PVE::VZDump::cleanup_cluster_slots() };
+                syslog('err', "unable to clean up backup slots - $@") if $@;
+
                 PVE::Jobs::run_jobs($first_run);
             },
         );
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index eef338435..867fe0b8f 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -27,6 +27,7 @@ use PVE::RPCEnvironment;
 use PVE::Storage;
 use PVE::UPID;
 use PVE::VZDump::Common;
+use PVE::VZDump::ClusterSlots;
 use PVE::VZDump::Plugin;
 use PVE::Tools qw(extract_param split_list);
 use PVE::API2Tools;
@@ -1724,6 +1725,22 @@ my sub try_guest_lock {
     return $fh;
 }
 
+my sub guest_backup_running {
+    my ($vmid, $upid) = @_;
+
+    my $path = "$lockdir/guest-$vmid.lock";
+    open(my $fh, '<', $path) or do {
+        die "unable to open '$path' - $!\n" if !$!{ENOENT};
+        return 0;
+    };
+    return 0 if flock($fh, LOCK_SH | LOCK_NB);
+
+    my $holder = <$fh> // '';
+    chomp($holder);
+
+    return $holder eq $upid;
+}
+
 # Guest backups in slots above a lowered limit keep counting against it until they end.
 my sub try_slot {
     my ($prefix, $limit) = @_;
@@ -1795,8 +1812,6 @@ my sub guest_moved_to {
 my sub claim_next_guest {
     my ($self, $queue, $deferred) = @_;
 
-    refresh_cluster_config($self);
-
     my $maxwait = $self->{opts}->{lockwait} * 60;
 
     my $slot;
@@ -1848,24 +1863,165 @@ my sub skip_after_error {
     skip_task($_, "not started due to error - $reason") for splice($queue->@*);
 }
 
+my sub cluster_slot_resources {
+    my ($self) = @_;
+
+    my $limits = PVE::VZDump::ClusterSlots::limits();
+    return undef if !scalar(keys $limits->%*);
+
+    if (!PVE::VZDump::ClusterSlots::supported_by_all_nodes()) {
+        if (!$self->{'cluster-slots-unsupported'}++) {
+            debugmsg(
+                'warn',
+                "cluster-wide backup limits are not enforced - not all online nodes support them",
+            );
+        }
+        return undef;
+    }
+
+    return ['cluster'];
+}
+
+my sub release_cluster_slots {
+    my ($key) = @_;
+
+    return if !defined($key);
+
+    eval { PVE::VZDump::ClusterSlots::release($key) };
+    if (my $err = $@) {
+        die $err if $err =~ m/interrupted by signal$/; # also if wrapped by the cluster lock
+        debugmsg('warn', "unable to release cluster-wide backup slot - $err");
+    }
+}
+
+# Tells why slots may stay taken, with the name needed to release them.
+my sub offline_slot_holders {
+    my ($resources) = @_;
+
+    my $nodes = eval { PVE::VZDump::ClusterSlots::offline_holders($resources) };
+    die $@ if $@ =~ m/interrupted by signal$/;
+    return '' if !scalar(($nodes // [])->@*);
+
+    my $list = join(', ', map { "'$_'" } $nodes->@*);
+    return scalar($nodes->@*) > 1
+        ? ", offline nodes $list hold slots"
+        : ", offline node $list holds slots";
+}
+
+sub cleanup_cluster_slots {
+    PVE::VZDump::ClusterSlots::cleanup_local_entries(\&guest_backup_running);
+}
+
+# A claimed guest that needs cluster-wide slots waits for them as pending guest, only one at a time.
+my sub next_guest_to_start {
+    my ($self, $queue, $schedule) = @_;
+
+    refresh_cluster_config($self);
+
+    my $pending = $schedule->{pending};
+    if (!$pending) {
+        my ($task, $locks) = claim_next_guest($self, $queue, $schedule->{deferred}) or return;
+        my $resources = eval { cluster_slot_resources($self) };
+        if (my $err = $@) {
+            unshift $queue->@*, $task;
+            die $err;
+        }
+        return ($task, $locks) if !$resources;
+
+        $pending = $schedule->{pending} = {
+            task => $task,
+            locks => $locks,
+            resources => $resources,
+            since => time(),
+            refreshed => time(),
+        };
+    }
+
+    my $task = $pending->{task};
+    # A failed request can block and log for seconds, for example without quorum, so retry rarely.
+    if (time() >= ($pending->{'retry-at'} // 0)) {
+        my $granted = eval {
+            $pending->{key} //= PVE::VZDump::ClusterSlots::request(
+                $self->{upid}, $task->{vmid}, $pending->{resources},
+            );
+            if (time() - $pending->{refreshed} >= 30) {
+                PVE::VZDump::ClusterSlots::refresh();
+                $pending->{refreshed} = time();
+            }
+            PVE::VZDump::ClusterSlots::granted($pending->{key});
+        };
+        if (my $err = $@) {
+            die $err if $err =~ m/interrupted by signal$/;
+            chomp($err);
+            if (($pending->{'request-error'} // '') ne $err) {
+                debugmsg('warn', "unable to request cluster-wide backup slot - $err");
+            }
+            $pending->{'request-error'} = $err;
+            $pending->{'retry-at'} = time() + 30;
+        } elsif (!defined($granted)) {
+            $pending->{key} = undef; # vanished, for example if released by an admin
+        } elsif ($granted) {
+            delete $schedule->{pending};
+            $schedule->{'cluster-wait'} += time() - $pending->{since};
+            if (my $node = guest_moved_to($task->{vmid})) {
+                release_cluster_slots($pending->{key});
+                skip_task($task, "guest moved to node '$node'");
+                return;
+            }
+            return ($task, $pending->{locks}, $pending->{key});
+        } else {
+            delete $pending->{'request-error'};
+        }
+    }
+
+    my $waited = $schedule->{'cluster-wait'} + time() - $pending->{since};
+    my $maxwait = $self->{opts}->{lockwait} * 60;
+    if ($waited >= $maxwait) {
+        release_cluster_slots($pending->{key});
+        delete $schedule->{pending};
+        $schedule->{'cluster-wait'} = $waited;
+        my $reason =
+            'timeout waiting for cluster-wide backup slot' . offline_slot_holders(['cluster']);
+        $reason .= " - $pending->{'request-error'}" if defined($pending->{'request-error'});
+        skip_task($task, $reason);
+    } elsif (!$pending->{logged}++) {
+        my $offline = offline_slot_holders(['cluster']);
+        debugmsg('info', "VM $task->{vmid} waits for a cluster-wide backup slot$offline");
+    }
+
+    return;
+}
+
+my sub requeue_pending_guest {
+    my ($queue, $schedule) = @_;
+
+    my $pending = delete $schedule->{pending} or return;
+    release_cluster_slots($pending->{key});
+    unshift $queue->@*, $pending->{task};
+}
+
 my sub exec_serial_backup_tasks {
     my ($self, $tasklist) = @_;
 
     my @queue = $tasklist->@*;
-    my $deferred = {};
+    my $schedule = { deferred => {}, 'cluster-wait' => 0 };
 
     eval {
-        while (scalar(@queue)) {
+        while (scalar(@queue) || $schedule->{pending}) {
             # $locks keeps the guest lock and the slots taken until the guest backup is done
-            my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+            my ($task, $locks, $slot_key) = next_guest_to_start($self, \@queue, $schedule);
             if ($task) {
-                $self->exec_backup_task($task);
-            } elsif (scalar(@queue)) {
+                eval { $self->exec_backup_task($task) };
+                my $err = $@;
+                release_cluster_slots($slot_key);
+                die $err if $err;
+            } else {
                 sleep(1);
             }
         }
     };
     if (my $err = $@) {
+        requeue_pending_guest(\@queue, $schedule);
         skip_after_error(\@queue, $err);
         die $err;
     }
@@ -1875,7 +2031,7 @@ my sub exec_parallel_backup_tasks {
     my ($self, $tasklist, $concurrency) = @_;
 
     my @queue = $tasklist->@*;
-    my $deferred = {};
+    my $schedule = { deferred => {}, 'cluster-wait' => 0 };
     my $running = {};
     my $interrupted = 0;
     my $error;
@@ -1886,9 +2042,9 @@ my sub exec_parallel_backup_tasks {
     local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
         sub { $interrupted = 1; };
 
-    while (scalar(@queue) || scalar(keys $running->%*)) {
+    while (scalar(@queue) || $schedule->{pending} || scalar(keys $running->%*)) {
         while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
-            my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+            my ($task, $locks, $slot_key) = eval { next_guest_to_start($self, \@queue, $schedule) };
             if (my $err = $@) {
                 # let running guest backups finish, they would get aborted otherwise
                 debugmsg('err', "unable to start further guest backups - $err");
@@ -1897,19 +2053,28 @@ my sub exec_parallel_backup_tasks {
             }
             last if !$task;
 
+            if ($interrupted) { # getting the slots can take a while
+                release_cluster_slots($slot_key);
+                unshift @queue, $task;
+                last;
+            }
+
             # the backup process keeps holding the locks through its inherited copies of $locks
             my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
             # a guest backup process signalled while it exits dies into this eval
             POSIX::_exit(1) if $$ != $job_pid;
             if (my $err = $@) {
                 fail_task($task, $err);
+                release_cluster_slots($slot_key);
                 next;
             }
+            $child->{'slot-key'} = $slot_key;
             $running->{ $child->{pid} } = $child;
             unshift @queue, $task if $child->{cancelled};
         }
 
         if ($interrupted || defined($error)) {
+            requeue_pending_guest(\@queue, $schedule);
             last if !scalar(keys $running->%*);
         }
 
@@ -1930,7 +2095,9 @@ my sub exec_parallel_backup_tasks {
 
         for my $pid (keys $running->%*) {
             next if waitpid($pid, POSIX::WNOHANG()) != $pid;
-            finish_guest_backup(delete $running->{$pid}, $?);
+            my $child = delete $running->{$pid};
+            finish_guest_backup($child, $?);
+            release_cluster_slots($child->{'slot-key'});
         }
     }
 
diff --git a/PVE/VZDump/ClusterSlots.pm b/PVE/VZDump/ClusterSlots.pm
new file mode 100644
index 000000000..7ba47a0eb
--- /dev/null
+++ b/PVE/VZDump/ClusterSlots.pm
@@ -0,0 +1,247 @@
+package PVE::VZDump::ClusterSlots;
+
+# Cluster-wide limits for guest backups running at the same time.
+#
+# Each guest backup that counts against such a limit has an entry in a shared table on pmxcfs. It
+# starts out as waiter and becomes holder once all limits it counts against have room. Entries of
+# offline nodes stay, as their guest backups might still be running.
+
+use v5.36;
+
+use JSON;
+
+use PVE::Cluster qw(cfs_read_file);
+use PVE::File;
+use PVE::INotify;
+use PVE::RPCEnvironment;
+
+my $state_file = '/etc/pve/backup-slots.json';
+
+# first release of pve-manager that takes part in cluster-wide limits
+my $min_version = [9, 2, 31];
+
+sub limits() {
+    my $limits = {};
+
+    my $dc_conf = cfs_read_file('datacenter.cfg');
+    if (defined(my $cluster = $dc_conf->{'backup-concurrency'}->{cluster})) {
+        $limits->{cluster} = $cluster;
+    }
+
+    return $limits;
+}
+
+my sub get_members() {
+    my $members = PVE::Cluster::get_members();
+    return $members if $members && scalar(keys $members->%*);
+
+    # entries of other nodes would get dropped as if these left the cluster
+    die "unable to get cluster members\n" if -e '/etc/pve/corosync.conf';
+
+    return { PVE::INotify::nodename() => { online => 1 } };
+}
+
+# Offline nodes cannot start guest backups, and their entries stay anyway.
+sub supported_by_all_nodes() {
+    my $members = get_members();
+    my $nodename = PVE::INotify::nodename();
+
+    for my $node (keys $members->%*) {
+        next if $node eq $nodename || !$members->{$node}->{online};
+        return 0 if !PVE::Cluster::node_pvecfg_version_at_least($node, $min_version->@*);
+    }
+    return 1;
+}
+
+# Checks a copy, so that numbers do not get written back as strings.
+my sub is_count($value) {
+    return defined($value) && !ref($value) && $value =~ m/^[0-9]+\z/;
+}
+
+my sub valid_entry($entry) {
+    return
+        ref($entry) eq 'HASH'
+        && (grep { !ref($entry->{$_}) && length($entry->{$_} // '') } qw(node upid)) == 2
+        && is_count($entry->{vmid})
+        && is_count($entry->{seq})
+        && ref($entry->{resources}) eq 'ARRAY'
+        && !grep { ref($_) || !length($_ // '') } $entry->{resources}->@*;
+}
+
+sub parse_state($raw) {
+    my $content = length($raw) > 1024 ? substr($raw, 0, 1024) . '...' : $raw;
+
+    my $state = eval { decode_json($raw) };
+    die "invalid JSON '$content' - $@" if $@;
+    if (
+        ref($state) ne 'HASH'
+        || (grep { ref($state->{$_}) ne 'HASH' } qw(holders waiters))
+        || !is_count($state->{seq})
+        || grep { !valid_entry($_) } map { values $state->{$_}->%* } qw(holders waiters)
+    ) {
+        die "'$content' is not a backup slot table\n";
+    }
+
+    return $state;
+}
+
+# Fails if the file cannot be read, as that is likely transient and writing back a table that lacks
+# holders would let more guest backups start than the limits allow. With $reset_invalid, a table
+# that cannot be parsed gets replaced though, as otherwise every guest backup that needs a slot gets
+# skipped until an admin repairs it.
+sub read_state($reset_invalid = 0) {
+    open(my $fh, '<', $state_file) or do {
+        die "unable to open '$state_file' - $!\n" if !$!{ENOENT};
+        return { holders => {}, waiters => {}, seq => 0 };
+    };
+    my $raw = PVE::File::safe_read_from($fh, 0, 0, $state_file);
+    my $state = eval { parse_state($raw) };
+    if (my $err = $@) {
+        die "unable to parse '$state_file' - $err" if !$reset_invalid;
+        chomp($err);
+        my $msg = "replacing unparsable '$state_file', guest backups that run now do not count"
+            . " against cluster-wide limits until they end - $err";
+        PVE::RPCEnvironment::get()->warn($msg);
+        return ({ holders => {}, waiters => {}, seq => 0 }, 1);
+    }
+
+    return $state;
+}
+
+my sub used_slots($state) {
+    my $used = {};
+    for my $holder (values $state->{holders}->%*) {
+        $used->{$_}++ for $holder->{resources}->@*;
+    }
+    return $used;
+}
+
+my sub remove_entries($state, $match) {
+    my $count = 0;
+    for my $entries ($state->@{qw(holders waiters)}) {
+        for my $key (keys $entries->%*) {
+            next if !$match->($entries->{$key});
+            delete $entries->{$key};
+            $count++;
+        }
+    }
+    return $count;
+}
+
+# Moves waiters that fit into all limits they count against to the holders, oldest first. Only
+# resources with a limit matter, the others are tracked for information. Waiters of offline nodes
+# neither get granted nor hold up others.
+sub grant_waiters($state, $limits, $members) {
+    remove_entries($state, sub($entry) { !$members->{ $entry->{node} } });
+
+    my $used = used_slots($state);
+
+    my $waiters = $state->{waiters};
+    for my $key (sort { $waiters->{$a}->{seq} <=> $waiters->{$b}->{seq} } keys $waiters->%*) {
+        my $waiter = $waiters->{$key};
+        next if !$members->{ $waiter->{node} }->{online};
+
+        my $limited = [grep { defined($limits->{$_}) } $waiter->{resources}->@*];
+        next if grep { ($used->{$_} // 0) >= $limits->{$_} } $limited->@*;
+
+        $used->{$_}++ for $waiter->{resources}->@*;
+        $state->{holders}->{$key} = delete $waiters->{$key};
+        $state->{holders}->{$key}->{granted} = time();
+    }
+}
+
+my sub update_state($code) {
+    my $res = PVE::Cluster::cfs_lock_domain(
+        'backup-slots',
+        10,
+        sub {
+            my $json = JSON->new->canonical();
+
+            my ($state, $replaced) = read_state(1);
+            my $previous = $replaced ? '' : $json->encode($state);
+
+            my $res = $code->($state);
+            grant_waiters($state, limits(), get_members());
+
+            my $raw = $json->encode($state);
+            PVE::File::file_set_contents($state_file, $raw) if $raw ne $previous;
+
+            return $res;
+        },
+    );
+    die $@ if $@;
+
+    return $res;
+}
+
+sub request($upid, $vmid, $resources) {
+    my $node = PVE::INotify::nodename();
+    my $key = "$node/$vmid/$upid";
+
+    update_state(sub($state) {
+        return if $state->{holders}->{$key} || $state->{waiters}->{$key};
+
+        $state->{waiters}->{$key} = {
+            node => $node,
+            upid => $upid,
+            vmid => $vmid,
+            resources => $resources,
+            seq => ++$state->{seq},
+            queued => time(),
+        };
+    });
+
+    return $key;
+}
+
+# Returns 1 if granted, 0 if still waiting and undef if the request vanished.
+sub granted($key) {
+    my $state = read_state();
+    return 1 if $state->{holders}->{$key};
+    return 0 if $state->{waiters}->{$key};
+    return undef;
+}
+
+# Offline nodes that hold slots of the given resources, as their guest backups might be gone.
+sub offline_holders($resources) {
+    my $members = get_members();
+    my %wanted = map { $_ => 1 } $resources->@*;
+
+    my $nodes = {};
+    for my $holder (values read_state()->{holders}->%*) {
+        my $member = $members->{ $holder->{node} } or next;
+        next if $member->{online} || !grep { $wanted{$_} } $holder->{resources}->@*;
+        $nodes->{ $holder->{node} } = 1;
+    }
+    return [sort keys $nodes->%*];
+}
+
+# Grants waiters that fit by now, for example after limits got raised.
+sub refresh() {
+    update_state(sub { });
+}
+
+sub release($key) {
+    update_state(sub($state) {
+        delete $state->{$_}->{$key} for qw(holders waiters);
+    });
+}
+
+# Removes the entries of this node whose guest backup is gone, as told by $running->($vmid, $upid).
+sub cleanup_local_entries($running) {
+    my $node = PVE::INotify::nodename();
+
+    my $stale = sub($entry) {
+        return $entry->{node} eq $node && !$running->($entry->{vmid}, $entry->{upid});
+    };
+
+    # avoid the cluster lock if there is nothing to remove, but let it replace an unparsable table
+    my $state = eval { read_state() };
+    return if $state && !remove_entries($state, $stale);
+
+    update_state(sub($state) {
+        remove_entries($state, $stale);
+    });
+}
+
+1;
diff --git a/PVE/VZDump/Makefile b/PVE/VZDump/Makefile
new file mode 100644
index 000000000..6500eb4d4
--- /dev/null
+++ b/PVE/VZDump/Makefile
@@ -0,0 +1,15 @@
+include ../../defines.mk
+
+PERLSOURCE =   \
+	ClusterSlots.pm \
+
+all:
+
+.PHONY: clean
+clean:
+	rm -rf *~
+
+.PHONY: install
+install: $(PERLSOURCE)
+	install -d $(PERLLIBDIR)/PVE/VZDump
+	install -m 0644 $(PERLSOURCE) $(PERLLIBDIR)/PVE/VZDump
diff --git a/test/Makefile b/test/Makefile
index bbe87a7c3..04de559be 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
 	./$<
 
 .PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency test-vzdump-cluster-slots
 
 .PHONY: test-vzdump-guest-included
 test-vzdump-guest-included:
@@ -32,6 +32,10 @@ test-vzdump-new:
 test-vzdump-concurrency:
 	./vzdump_concurrency_test.pl
 
+.PHONY: test-vzdump-cluster-slots
+test-vzdump-cluster-slots:
+	./vzdump_cluster_slots_test.pl
+
 .PHONY: test-osd
 test-osd:
 	./OSD_test.pl
diff --git a/test/vzdump_cluster_slots_test.pl b/test/vzdump_cluster_slots_test.pl
new file mode 100755
index 000000000..51ca8341d
--- /dev/null
+++ b/test/vzdump_cluster_slots_test.pl
@@ -0,0 +1,205 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use lib '..';
+
+use JSON;
+use Storable qw(dclone);
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump::ClusterSlots;
+
+my $members = {
+    node1 => { online => 1 },
+    node2 => { online => 1 },
+    node3 => { online => 0 },
+};
+
+my sub make_state($holders, $waiters) {
+    my $state = { holders => {}, waiters => {}, seq => 0 };
+    for my $holder ($holders->@*) {
+        my ($key, $node, $resources) = $holder->@*;
+        $state->{holders}->{$key} = { node => $node, resources => $resources };
+    }
+    for my $waiter ($waiters->@*) {
+        my ($key, $node, $resources) = $waiter->@*;
+        $state->{waiters}->{$key} = {
+            node => $node,
+            resources => $resources,
+            seq => ++$state->{seq},
+        };
+    }
+    return $state;
+}
+
+my sub granted($state) {
+    return [sort keys $state->{holders}->%*];
+}
+
+my $tests = [
+    {
+        description => 'waiters are granted in order up to the limit',
+        limits => { cluster => 2 },
+        waiters =>
+            [['a', 'node1', ['cluster']], ['b', 'node2', ['cluster']], ['c', 'node1', ['cluster']]],
+        expected => ['a', 'b'],
+    },
+    {
+        description => 'holders count against the limit',
+        limits => { cluster => 2 },
+        holders => [['h', 'node1', ['cluster']]],
+        waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['a', 'h'],
+    },
+    {
+        description => 'resources without limit do not hold up',
+        limits => {},
+        holders => [['h', 'node1', ['cluster']]],
+        waiters => [['a', 'node2', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['a', 'b', 'h'],
+    },
+    {
+        description => 'waiters of offline nodes neither get granted nor hold up others',
+        limits => { cluster => 1 },
+        waiters => [['a', 'node3', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['b'],
+    },
+    {
+        description => 'holders of offline nodes keep counting',
+        limits => { cluster => 1 },
+        holders => [['h', 'node3', ['cluster']]],
+        waiters => [['a', 'node1', ['cluster']]],
+        expected => ['h'],
+    },
+    {
+        description => 'entries of nodes no longer in the cluster are dropped',
+        limits => { cluster => 1 },
+        holders => [['h', 'gone', ['cluster']]],
+        waiters => [['a', 'gone', ['cluster']], ['b', 'node1', ['cluster']]],
+        expected => ['b'],
+        'expected-waiters' => [],
+    },
+];
+
+for my $test ($tests->@*) {
+    my $state = make_state($test->{holders} // [], $test->{waiters} // []);
+
+    PVE::VZDump::ClusterSlots::grant_waiters($state, $test->{limits}, $members);
+
+    is_deeply(granted($state), $test->{expected}, $test->{description});
+
+    if (my $expected_waiters = $test->{'expected-waiters'}) {
+        is_deeply(
+            [sort keys $state->{waiters}->%*],
+            $expected_waiters,
+            "$test->{description} (waiters)",
+        );
+    }
+}
+
+# Request, grant, release and cleanup against an in-memory table.
+my $table = { holders => {}, waiters => {}, seq => 0 };
+my $writes = 0;
+
+my $slots_module = Test::MockModule->new('PVE::VZDump::ClusterSlots');
+$slots_module->mock(
+    read_state => sub { return dclone($table); },
+    limits => sub { return { cluster => 1 }; },
+);
+my $cluster_module = Test::MockModule->new('PVE::Cluster');
+$cluster_module->mock(
+    cfs_lock_domain => sub {
+        my ($domain, $timeout, $code) = @_;
+        return eval { $code->() };
+    },
+    get_members => sub { return $members; },
+);
+my $file_module = Test::MockModule->new('PVE::File');
+$file_module->mock(
+    file_set_contents => sub {
+        my ($filename, $raw) = @_;
+        $table = decode_json($raw);
+        $writes++;
+    },
+);
+my $inotify_module = Test::MockModule->new('PVE::INotify');
+$inotify_module->mock(nodename => sub { return 'node1'; });
+
+$table->{holders}->{other} = { node => 'node2', vmid => 200, upid => 'u0', resources => [] };
+
+my $first = PVE::VZDump::ClusterSlots::request('u1', 100, ['cluster']);
+my $second = PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(PVE::VZDump::ClusterSlots::granted($first), 1, 'first request gets the slot');
+is(PVE::VZDump::ClusterSlots::granted($second), 0, 'second request waits');
+
+PVE::VZDump::ClusterSlots::request('u2', 101, ['cluster']);
+is(scalar(keys $table->{waiters}->%*), 1, 'repeated request does not queue again');
+
+my $count = $writes;
+PVE::VZDump::ClusterSlots::refresh();
+is($writes, $count, 'unchanged table is not written');
+
+PVE::VZDump::ClusterSlots::release($first);
+is(PVE::VZDump::ClusterSlots::granted($second), 1, 'released slot goes to the waiting request');
+is(PVE::VZDump::ClusterSlots::granted($first), undef, 'released request is gone');
+
+PVE::VZDump::ClusterSlots::cleanup_local_entries(sub { return 0; });
+is_deeply([keys $table->{holders}->%*], ['other'], 'cleanup only drops stale entries of the node');
+
+$table->{holders}->{down} =
+    { node => 'node3', vmid => 300, upid => 'u3', resources => ['cluster'] };
+is_deeply(
+    PVE::VZDump::ClusterSlots::offline_holders(['cluster']),
+    ['node3'],
+    'offline node holds a slot',
+);
+is_deeply(
+    PVE::VZDump::ClusterSlots::offline_holders(['storage/pbs']),
+    [],
+    'offline node holds no slot of another limit',
+);
+
+my $entry = '{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":["cluster"]}';
+my $parsed = PVE::VZDump::ClusterSlots::parse_state('{"holders":{},"waiters":{},"seq":3}');
+is(JSON->new->canonical->encode($parsed), '{"holders":{},"seq":3,"waiters":{}}', 'valid table');
+ok(
+    PVE::VZDump::ClusterSlots::parse_state(qq({"holders":{"a":$entry},"waiters":{},"seq":3})),
+    'valid table with an entry',
+);
+for my $raw (
+    'garbage',
+    '[]',
+    '{"holders":null,"waiters":{},"seq":3}',
+    '{"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":[],"seq":3}',
+    '{"holders":{},"waiters":{}}',
+    '{"holders":{},"waiters":{},"seq":"x"}',
+    '{"holders":{"a":{}},"waiters":{},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":"cluster"}},"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":{"a":{"upid":"UPID:x","vmid":100,"seq":1,"resources":[]}},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[null]}},"waiters":{},"seq":3}',
+    '{"holders":{"a":{"node":"node1","upid":"UPID:x","vmid":100,"seq":1,"resources":[{}]}},"waiters":{},"seq":3}',
+    '{"holders":{},"waiters":{},"seq":"\u0663"}',
+    '{"holders":{},"waiters":{},"seq":"3\n"}',
+) {
+    eval { PVE::VZDump::ClusterSlots::parse_state($raw) };
+    ok($@, "rejects $raw");
+}
+
+my $versions = { node2 => '9.2.31', node3 => '9.2.30' };
+$cluster_module->mock(
+    get_node_kv => sub {
+        my ($key, $node) = @_;
+        return {} if $key ne 'version-info' || !defined($versions->{$node});
+        return { $node => encode_json({ version => $versions->{$node} }) };
+    },
+);
+ok(PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older offline node is ignored');
+$versions->{node2} = '9.2.30';
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'older online node blocks');
+delete $versions->{node2};
+ok(!PVE::VZDump::ClusterSlots::supported_by_all_nodes(), 'online node without version blocks');
+
+done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-18 13:33 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118132947.1093069-17-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal