From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other
Date: Wed, 18 Nov 2026 14:18:36 +0100 [thread overview]
Message-ID: <20261118132947.1093069-13-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>
A node ran one backup job at a time. A short job, for example an
hourly one to a nearby target, had to wait until a long nightly job
finished, possibly for hours, and failed once the lock wait time ran
out.
Jobs with the new 'overlap' option share the global lock and run at
the same time as other such jobs. Every guest backup on the node, no
matter which job it belongs to, holds one of the node's backup slots
while running, so overlapping jobs together stay within the node's
backup concurrency. When the concurrency gets lowered, slots above
the new limit keep counting until their guest backups end. Jobs
without the option keep running alone, and a job waiting for that
takes precedence over overlapping jobs starting later, so these cannot
starve it. Jobs to external backup providers always run alone.
A guest that another job is backing up is deferred until that backup
finishes, waiting at most the lock wait time. Before starting a guest
backup, the job also checks whether the guest moved to another node in
the meantime.
The global lock, the node slots and the guest locks are file locks
held by the processes doing the work, so they get released whenever
such a process ends. Jobs that do not overlap still take the global
lock exclusively, which keeps it working for the boot-time cleanup of
stale backup locks and against older versions during an upgrade.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
changes v1 -> v2:
- after lowering a limit, reuse free slots once fewer than the limit are
occupied, even if slots above it are still in use [Fiona]; only slots
that were used before get probed for that
- take the node slot before the guest lock, so a job waiting for a slot
does not lock guests of other jobs for a moment
- pick up a changed datacenter default of the node limit while waiting
- warn and run alone if checking for an external backup provider fails,
but still end on a stop request [Fiona]
- preserve the global lock error across unlocking the gate [Fiona]
- create /run/vzdump through tmpfiles.d [Fiona]
- rely on the lock wait time that the constructor sets [Fiona]
- use PVE::File [Fiona], match numbers with [0-9]
PVE/API2/VZDump.pm | 2 +-
PVE/VZDump.pm | 244 +++++++++++++++++++++++++++++++++++++++++++--
debian/tmpfiles | 5 +-
3 files changed, 241 insertions(+), 10 deletions(-)
diff --git a/PVE/API2/VZDump.pm b/PVE/API2/VZDump.pm
index 97b1797f8..42017cf9e 100644
--- a/PVE/API2/VZDump.pm
+++ b/PVE/API2/VZDump.pm
@@ -137,7 +137,7 @@ __PACKAGE__->register_method({
my $vzdump = PVE::VZDump->new($cmdline, $param, $skiplist);
my $LOCK_FH = eval {
- my $lock_fh = $vzdump->getlock($upid); # only one process allowed
+ my $lock_fh = $vzdump->getlock($upid);
$vzdump->init_job();
$lock_fh;
};
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 054665430..eef338435 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -17,6 +17,7 @@ use Time::Local;
use PVE::Cluster qw(cfs_read_file);
use PVE::DataCenterConfig;
use PVE::Exception qw(raise_param_exc);
+use PVE::File;
use PVE::HA::Config;
use PVE::HA::Env::PVE2;
use PVE::JSONSchema qw(get_standard_option);
@@ -47,6 +48,7 @@ PVE::JSONSchema::register_standard_option(
my @posix_filesystems = qw(ext3 ext4 nfs nfs4 reiserfs xfs);
my $lockfile = '/var/run/vzdump.lock';
+my $lockdir = '/run/vzdump';
my $logdir = '/var/log/vzdump';
my @plugins = qw();
@@ -688,6 +690,25 @@ sub new {
$self->{'default-prune-backups'} = $defaults->{'prune-backups'};
+ $self->{exclusive} = !$opts->{overlap};
+ if (!$self->{exclusive} && $opts->{storage}) {
+ # external providers are not known to cope with other backups running at the same time
+ $self->{exclusive} = eval {
+ PVE::Storage::storage_has_feature(
+ PVE::Storage::config(), $opts->{storage}, 'backup-provider',
+ );
+ };
+ if (my $err = $@) {
+ die $err if $err =~ m/^interrupted by signal$/;
+ debugmsg(
+ 'warn',
+ "unable to check if storage '$opts->{storage}' uses a backup provider, so not"
+ . " running at the same time as other backups - $err",
+ );
+ $self->{exclusive} = 1;
+ }
+ }
+
return $self;
}
@@ -787,14 +808,32 @@ sub getlock {
my $maxwait = $self->{opts}->{lockwait};
die "missing UPID" if !$upid; # should not happen
+ $self->{upid} = $upid;
my $SERVER_FLCK;
if (!open($SERVER_FLCK, '>>', "$lockfile")) {
debugmsg('err', "can't open lock on file '$lockfile' - $!", undef, 1);
die "can't open lock on file '$lockfile' - $!";
}
+ open(my $gate, '>>', "$lockdir/gate.lock")
+ or die "can't open lock on file '$lockdir/gate.lock' - $!\n";
- if (!flock($SERVER_FLCK, LOCK_EX | LOCK_NB)) {
+ # Jobs that may overlap share the global lock. A job waiting to get it exclusively holds the
+ # gate meanwhile, so overlapping jobs starting later cannot keep it waiting forever.
+ my $mode = $self->{exclusive} ? LOCK_EX : LOCK_SH;
+ my $acquire = sub {
+ my ($flags) = @_;
+
+ flock($gate, $mode | $flags) or return (0, "$!");
+ my $locked = flock($SERVER_FLCK, $mode | $flags);
+ my $err = "$!";
+ flock($gate, LOCK_UN);
+
+ return ($locked, $err);
+ };
+
+ my ($locked) = $acquire->(LOCK_NB);
+ if (!$locked) {
if (!$maxwait) {
debugmsg('err', "can't acquire lock '$lockfile' (wait = 0)", undef, 1);
die "can't acquire lock '$lockfile' (wait = 0)";
@@ -806,8 +845,8 @@ sub getlock {
local $SIG{ALRM} = sub { alarm(0); die "got timeout\n"; };
- if (!flock($SERVER_FLCK, LOCK_EX)) {
- my $err = $!;
+ ($locked, my $err) = $acquire->(0);
+ if (!$locked) {
close($SERVER_FLCK);
alarm(0);
die "$err\n";
@@ -1664,12 +1703,182 @@ my sub finish_guest_backup {
}
}
+my sub try_lock_file {
+ my ($path) = @_;
+
+ open(my $fh, '>>', $path) or die "unable to open lock file '$path' - $!\n";
+ return $fh if flock($fh, LOCK_EX | LOCK_NB);
+
+ close($fh);
+ return undef;
+}
+
+my sub try_guest_lock {
+ my ($self, $vmid) = @_;
+
+ my $fh = try_lock_file("$lockdir/guest-$vmid.lock") or return undef;
+ truncate($fh, 0);
+ print $fh "$self->{upid}\n";
+ $fh->flush();
+
+ return $fh;
+}
+
+# Guest backups in slots above a lowered limit keep counting against it until they end.
+my sub try_slot {
+ my ($prefix, $limit) = @_;
+
+ # Otherwise, processes counting the slots in use at the same time could all take the last one.
+ my $select_lock = "$lockdir/select-slot.lock";
+ open(my $selecting, '>>', $select_lock)
+ or die "unable to open lock file '$select_lock' - $!\n";
+ flock($selecting, LOCK_EX) or die "unable to lock '$select_lock' - $!\n";
+
+ # only slots that were used before have a lock file
+ my ($busy, $free, $exists) = (0, undef, {});
+ for my $path (glob("$prefix-*.lock")) {
+ my ($slot) = $path =~ m/^\Q$prefix\E-([0-9]+)\.lock$/ or next;
+ $exists->{$slot} = 1;
+ # rebuilt from the number, as names from glob() are tainted and vzdump runs with -T
+ my $fh = try_lock_file("$prefix-$slot.lock");
+ if (!$fh) {
+ $busy++;
+ } elsif ($slot <= $limit && !$free) {
+ $free = $fh;
+ } else {
+ close($fh);
+ }
+ }
+
+ if ($busy >= $limit) {
+ close($free) if $free;
+ return undef;
+ }
+ return $free if $free;
+
+ # fewer slots are busy than the limit allows, so the first one without lock file is within it
+ my $slot = 1;
+ $slot++ while $exists->{$slot};
+ return try_lock_file("$prefix-$slot.lock");
+}
+
+my sub try_node_slot {
+ return try_slot("$lockdir/slot", node_backup_concurrency());
+}
+
+my sub log_wait {
+ my ($self, $reason) = @_;
+
+ return if $self->{'waiting-for'}->{$reason}++;
+ debugmsg('info', "waiting for $reason");
+}
+
+# Config files from pmxcfs are cached until cfs_update(), refresh them to pick up changed limits.
+my sub refresh_cluster_config {
+ my ($self) = @_;
+
+ return if time() - ($self->{'cluster-config-refreshed'} // 0) < 5;
+
+ PVE::Cluster::cfs_update();
+ $self->{'cluster-config-refreshed'} = time();
+}
+
+my sub guest_moved_to {
+ my ($vmid) = @_;
+
+ PVE::Cluster::cfs_update();
+ my $node = PVE::Cluster::get_vmlist()->{ids}->{$vmid}->{node};
+
+ return defined($node) && $node ne PVE::INotify::nodename() ? $node : undef;
+}
+
+my sub claim_next_guest {
+ my ($self, $queue, $deferred) = @_;
+
+ refresh_cluster_config($self);
+
+ my $maxwait = $self->{opts}->{lockwait} * 60;
+
+ my $slot;
+ for (my $i = 0; $i < scalar($queue->@*); $i++) {
+ my $task = $queue->[$i];
+ my $vmid = $task->{vmid};
+
+ # first, so that a job waiting for a slot does not lock guests of other jobs for a moment
+ $slot //= try_node_slot();
+ if (!$slot) {
+ log_wait($self, "a free backup slot on the node");
+ return;
+ }
+
+ my $guest_lock = try_guest_lock($self, $vmid);
+ if (!$guest_lock) {
+ my $holder = PVE::File::file_read_first_line("$lockdir/guest-$vmid.lock") // '';
+ if (!defined($deferred->{$vmid})) {
+ $deferred->{$vmid} = time();
+ debugmsg('info', "VM $vmid is being backed up by '$holder', deferring");
+ } elsif (time() - $deferred->{$vmid} >= $maxwait) {
+ splice($queue->@*, $i--, 1);
+ skip_task($task, "still being backed up by '$holder'");
+ }
+ next;
+ }
+ delete $deferred->{$vmid};
+
+ if (my $node = guest_moved_to($vmid)) {
+ splice($queue->@*, $i--, 1);
+ skip_task($task, "guest moved to node '$node'");
+ next;
+ }
+ delete $self->{'waiting-for'};
+
+ splice($queue->@*, $i--, 1);
+ return ($task, [$guest_lock, $slot]);
+ }
+
+ return;
+}
+
+my sub skip_after_error {
+ my ($queue, $err) = @_;
+
+ return if $err =~ m/interrupted by signal$/;
+
+ chomp(my $reason = $err);
+ skip_task($_, "not started due to error - $reason") for splice($queue->@*);
+}
+
+my sub exec_serial_backup_tasks {
+ my ($self, $tasklist) = @_;
+
+ my @queue = $tasklist->@*;
+ my $deferred = {};
+
+ eval {
+ while (scalar(@queue)) {
+ # $locks keeps the guest lock and the slots taken until the guest backup is done
+ my ($task, $locks) = claim_next_guest($self, \@queue, $deferred);
+ if ($task) {
+ $self->exec_backup_task($task);
+ } elsif (scalar(@queue)) {
+ sleep(1);
+ }
+ }
+ };
+ if (my $err = $@) {
+ skip_after_error(\@queue, $err);
+ die $err;
+ }
+}
+
my sub exec_parallel_backup_tasks {
my ($self, $tasklist, $concurrency) = @_;
my @queue = $tasklist->@*;
+ my $deferred = {};
my $running = {};
my $interrupted = 0;
+ my $error;
my $job_pid = $$;
# Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
@@ -1678,8 +1887,17 @@ my sub exec_parallel_backup_tasks {
sub { $interrupted = 1; };
while (scalar(@queue) || scalar(keys $running->%*)) {
- while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
- my $task = shift @queue;
+ while (!$interrupted && !defined($error) && scalar(keys $running->%*) < $concurrency) {
+ my ($task, $locks) = eval { claim_next_guest($self, \@queue, $deferred) };
+ if (my $err = $@) {
+ # let running guest backups finish, they would get aborted otherwise
+ debugmsg('err', "unable to start further guest backups - $err");
+ $error = $err;
+ last;
+ }
+ last if !$task;
+
+ # the backup process keeps holding the locks through its inherited copies of $locks
my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
# a guest backup process signalled while it exits dies into this eval
POSIX::_exit(1) if $$ != $job_pid;
@@ -1691,7 +1909,14 @@ my sub exec_parallel_backup_tasks {
unshift @queue, $task if $child->{cancelled};
}
- last if $interrupted && !scalar(keys $running->%*);
+ if ($interrupted || defined($error)) {
+ last if !scalar(keys $running->%*);
+ }
+
+ if (!scalar(keys $running->%*)) {
+ sleep(1);
+ next;
+ }
my $handles = {};
for my $child (values $running->%*) {
@@ -1710,6 +1935,11 @@ my sub exec_parallel_backup_tasks {
}
die "interrupted by signal\n" if $interrupted;
+
+ if (defined($error)) {
+ skip_after_error(\@queue, $error);
+ die $error;
+ }
}
sub exec_backup {
@@ -1745,7 +1975,7 @@ sub exec_backup {
debugmsg('info', "running up to $concurrency guest backups at the same time");
exec_parallel_backup_tasks($self, $tasklist, $concurrency);
} else {
- $self->exec_backup_task($_) for $tasklist->@*;
+ exec_serial_backup_tasks($self, $tasklist);
}
$self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
diff --git a/debian/tmpfiles b/debian/tmpfiles
index 1263300fb..1703c789d 100644
--- a/debian/tmpfiles
+++ b/debian/tmpfiles
@@ -1,2 +1,3 @@
-#Type Path Mode User Group Age Argument
-d /run/pve 0750 root www-data - -
+#Type Path Mode User Group Age Argument
+d /run/pve 0750 root www-data - -
+d /run/vzdump 0700 root root - -
--
2.47.3
next prev parent reply other threads:[~2026-11-18 13:32 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118132947.1093069-13-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.