From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 manager 11/20] vzdump: back up guests of a job in parallel
Date: Wed, 18 Nov 2026 14:18:35 +0100 [thread overview]
Message-ID: <20261118132947.1093069-12-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118132947.1093069-1-t.lamprecht@proxmox.com>
Each node backed up the guests of a job strictly one after the other.
How long a job took thus depended on how its guests were spread over
the nodes: guests all on one node were backed up in series, even if
that node and the backup target could handle more.
Allow backing up several guests at the same time, up to the backup
concurrency of the node. It comes from the new node configuration
option or the datacenter-wide default, and stays at one guest unless
configured. A job can lower it with its 'concurrency' option.
Each guest backup then runs in a forked process. The job process
collects the results, prefixes the guest output in the task log with
the guest ID, keeps the warning count, and sends a single notification
as before. A guest backup process that ends without a result is
reported as failed.
Stopping the task signals all its processes, so the guest backups stop
on their own while the job process waits for them. It does not signal
them again, which could interrupt their cleanup. A stop request that
arrives while a guest backup process is being forked does not reach
that process. The new process therefore starts its backup only after
the job process confirmed that no stop request came in; otherwise the
guest counts as not started. A signal that only reached the job
process keeps it from starting further guests.
A guest listed more than once is backed up once, as its backups would
otherwise run at the same time.
Backups to external providers stay sequential, as providers are not
known to cope with backups from multiple processes. With a concurrency
of one, guests are backed up in the job process as before.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
changes v1 -> v2:
- comment why a backup process without start byte ends as interrupted
[Fiona]
- flush the output of a guest backup only once [Fiona]
- report a guest backup process that fails before its backup begins as
failed with the error, it was left as not started
Not changed [Fiona]:
- no separate message for a canceled start: end-of-file means do not
start, both after a stop request while forking and if the job process
is gone, and in the latter case nothing reads the output anymore
- no separate exit status check. The result is written once the backup
and its cleanup finished, so a non-zero exit with a result is either
an interrupted backup that the result records as failed, or a process
killed while exiting after a finished backup
PVE/NodeConfig.pm | 7 +
PVE/VZDump.pm | 257 +++++++++++++++++++++++++++++++-
test/Makefile | 6 +-
test/vzdump_concurrency_test.pl | 89 +++++++++++
4 files changed, 353 insertions(+), 6 deletions(-)
create mode 100755 test/vzdump_concurrency_test.pl
diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm
index cc1a2a296..fbd7ecf87 100644
--- a/PVE/NodeConfig.pm
+++ b/PVE/NodeConfig.pm
@@ -108,6 +108,13 @@ my $confdesc = {
default => 80,
optional => 1,
},
+ 'backup-concurrency' => {
+ description => "Maximal number of guests backed up at the same time on this node."
+ . " Overrides the datacenter-wide default.",
+ type => 'integer',
+ minimum => 1,
+ optional => 1,
+ },
};
my $wakeonlan_desc = {
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index fd2a74ab9..054665430 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -11,6 +11,7 @@ use IO::File;
use IO::Select;
use IPC::Open3;
use POSIX qw(strftime);
+use Storable;
use Time::Local;
use PVE::Cluster qw(cfs_read_file);
@@ -19,6 +20,7 @@ use PVE::Exception qw(raise_param_exc);
use PVE::HA::Config;
use PVE::HA::Env::PVE2;
use PVE::JSONSchema qw(get_standard_option);
+use PVE::NodeConfig;
use PVE::Notify;
use PVE::RPCEnvironment;
use PVE::Storage;
@@ -289,10 +291,11 @@ sub check_vmids {
my (@vmids) = @_;
my $res = [];
+ my $seen = {};
for my $vmid (sort { $a <=> $b } @vmids) {
die "ERROR: strange VM ID '${vmid}'\n" if $vmid !~ m/^\d+$/;
$vmid = int($vmid); # remove leading zeros
- next if !$vmid;
+ next if !$vmid || $seen->{$vmid}++;
push @$res, $vmid;
}
@@ -1470,6 +1473,245 @@ sub skip_all_guests {
return $tasklist;
}
+my sub fail_task {
+ my ($task, $err) = @_;
+
+ $task->{state} = 'err';
+ $task->{msg} = $err;
+ debugmsg('err', "Backup of VM $task->{vmid} failed - $err");
+}
+
+sub node_backup_concurrency {
+ my $node_conf = PVE::NodeConfig::load_config(PVE::INotify::nodename());
+
+ return $node_conf->{'backup-concurrency'}
+ // cfs_read_file('datacenter.cfg')->{'backup-concurrency'}->{node} // 1;
+}
+
+sub guest_backup_concurrency {
+ my ($self) = @_;
+
+ # external providers are not known to cope with backups from multiple processes
+ return 1 if $self->{'backup-provider'};
+
+ my $node_limit = node_backup_concurrency();
+ my $limit = $self->{opts}->{concurrency} // $node_limit;
+
+ return $limit < $node_limit ? $limit : $node_limit;
+}
+
+my $guest_result_keys = [qw(backuptime hostname log msg size state target tmplog vmtype)];
+
+my sub fork_guest_backup {
+ my ($self, $task, $running, $interrupted) = @_;
+
+ pipe(my $output_reader, my $output_writer) or die "unable to create pipe - $!\n";
+ pipe(my $result_reader, my $result_writer) or die "unable to create pipe - $!\n";
+ pipe(my $start_reader, my $start_writer) or die "unable to create pipe - $!\n";
+
+ # Until the child set up its own handlers, a signal would only reach the inherited one of the
+ # job process and get lost.
+ my $signals = POSIX::SigSet->new(
+ POSIX::SIGINT(),
+ POSIX::SIGTERM(),
+ POSIX::SIGQUIT(),
+ POSIX::SIGHUP(),
+ POSIX::SIGPIPE(),
+ );
+ my $old_mask = POSIX::SigSet->new();
+ POSIX::sigprocmask(POSIX::SIG_BLOCK(), $signals, $old_mask)
+ or die "unable to block signals - $!\n";
+
+ my $pid = fork();
+ if (!defined($pid)) {
+ my $err = $!;
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+ die "unable to fork guest backup - $err\n";
+ }
+
+ if (!$pid) {
+ my $interrupted = 0;
+ $SIG{INT} = $SIG{TERM} = $SIG{QUIT} = $SIG{HUP} = $SIG{PIPE} = sub {
+ # a second signal must not abort the cleanup the first one started
+ die "interrupted by signal\n" if !$interrupted++;
+ };
+
+ my $exitcode = 1;
+ eval {
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+ close($output_reader);
+ close($result_reader);
+ close($start_writer);
+ # only the job process reads the output of the other guest backups
+ for my $sibling (values $running->%*) {
+ close($_) for grep { defined } $sibling->@{qw(output result)};
+ }
+ open(STDOUT, '>&', $output_writer) or die "unable to redirect STDOUT - $!\n";
+ open(STDERR, '>&', $output_writer) or die "unable to redirect STDERR - $!\n";
+ close($output_writer);
+
+ # The job process closes the pipe without writing if it got a stop request while
+ # forking, which this process missed, so end as if it got that signal.
+ sysread($start_reader, my $start, 1) or die "interrupted by signal\n";
+ close($start_reader);
+
+ PVE::INotify::inotify_close();
+
+ eval { $self->exec_backup_task($task) };
+ my $err = $@;
+ # errors of the backup itself are handled there, so this one came up before it began
+ if ($err && $err !~ m/^interrupted by signal$/ && $task->{state} eq 'todo') {
+ chomp($err);
+ fail_task($task, $err);
+ undef $err;
+ }
+
+ my $result = { map { $_ => $task->{$_} } $guest_result_keys->@* };
+ print $result_writer Storable::nfreeze($result);
+ close($result_writer);
+
+ die $err if $err;
+ $exitcode = 0;
+ };
+ warn $@ if $@ && $@ !~ m/^interrupted by signal$/;
+ STDOUT->flush();
+ POSIX::_exit($exitcode);
+ }
+
+ POSIX::sigprocmask(POSIX::SIG_SETMASK(), $old_mask);
+
+ close($output_writer);
+ close($result_writer);
+ close($start_reader);
+
+ # A stop request that came in while forking reached only the job process, so the backup
+ # process must not start then.
+ my $cancelled = $$interrupted;
+ if (!$cancelled) {
+ local $SIG{PIPE} = 'IGNORE';
+ syswrite($start_writer, '1');
+ }
+ close($start_writer);
+
+ return {
+ pid => $pid,
+ cancelled => $cancelled,
+ task => $task,
+ output => $output_reader,
+ result => $result_reader,
+ 'output-buffer' => '',
+ 'result-data' => '',
+ };
+}
+
+my sub forward_guest_output {
+ my ($child, $data, $final) = @_;
+
+ my $vmid = $child->{task}->{vmid};
+
+ $child->{'output-buffer'} .= $data;
+ $child->{'output-buffer'} .= "\n" if $final && length($child->{'output-buffer'});
+
+ while ($child->{'output-buffer'} =~ s/^(.*)\n//) {
+ my $line = $1;
+ # a guest backup process counts its own warnings, and that count is lost when it exits
+ PVE::RPCEnvironment::get()->{warning_count}++ if $line =~ m/^WARN: /;
+ print STDERR "$vmid: $line\n";
+ }
+}
+
+my sub close_guest_backup_pipe {
+ my ($child, $kind) = @_;
+
+ close(delete $child->{$kind});
+ forward_guest_output($child, '', 1) if $kind eq 'output';
+}
+
+my sub read_guest_backup_pipe {
+ my ($child, $kind) = @_;
+
+ my $count = sysread($child->{$kind}, my $data, 64 * 1024);
+ return if !defined($count) && ($!{EINTR} || $!{EAGAIN});
+
+ if (!$count) {
+ close_guest_backup_pipe($child, $kind);
+ } elsif ($kind eq 'output') {
+ forward_guest_output($child, $data);
+ } else {
+ $child->{'result-data'} .= $data;
+ }
+}
+
+my sub finish_guest_backup {
+ my ($child, $status) = @_;
+
+ # the backup process is gone, but helpers it started might still hold the output pipe
+ for my $kind (qw(result output)) {
+ while ($child->{$kind} && IO::Select->new($child->{$kind})->can_read(0)) {
+ read_guest_backup_pipe($child, $kind);
+ }
+ close_guest_backup_pipe($child, $kind) if $child->{$kind};
+ }
+ return if $child->{cancelled};
+
+ my $task = $child->{task};
+ my $result = eval { Storable::thaw($child->{'result-data'}) };
+ if (ref($result) eq 'HASH') {
+ $task->{$_} = $result->{$_} for keys $result->%*;
+ } else {
+ my $reason = $status & 127 ? "signal " . ($status & 127) : "exit code " . ($status >> 8);
+ fail_task($task, "backup process ended unexpectedly ($reason)");
+ }
+}
+
+my sub exec_parallel_backup_tasks {
+ my ($self, $tasklist, $concurrency) = @_;
+
+ my @queue = $tasklist->@*;
+ my $running = {};
+ my $interrupted = 0;
+ my $job_pid = $$;
+
+ # Stopping a task signals all its processes, so the guest backups stop on their own. Signalling
+ # them again could interrupt their cleanup.
+ local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} = local $SIG{PIPE} =
+ sub { $interrupted = 1; };
+
+ while (scalar(@queue) || scalar(keys $running->%*)) {
+ while (!$interrupted && scalar(@queue) && scalar(keys $running->%*) < $concurrency) {
+ my $task = shift @queue;
+ my $child = eval { fork_guest_backup($self, $task, $running, \$interrupted) };
+ # a guest backup process signalled while it exits dies into this eval
+ POSIX::_exit(1) if $$ != $job_pid;
+ if (my $err = $@) {
+ fail_task($task, $err);
+ next;
+ }
+ $running->{ $child->{pid} } = $child;
+ unshift @queue, $task if $child->{cancelled};
+ }
+
+ last if $interrupted && !scalar(keys $running->%*);
+
+ my $handles = {};
+ for my $child (values $running->%*) {
+ $handles->{ fileno($child->{$_}) } = [$child, $_]
+ for grep { $child->{$_} } qw(output result);
+ }
+ my $select = IO::Select->new(map { $_->[0]->{ $_->[1] } } values $handles->%*);
+ for my $fh ($select->can_read(1)) {
+ read_guest_backup_pipe($handles->{ fileno($fh) }->@*);
+ }
+
+ for my $pid (keys $running->%*) {
+ next if waitpid($pid, POSIX::WNOHANG()) != $pid;
+ finish_guest_backup(delete $running->{$pid}, $?);
+ }
+ }
+
+ die "interrupted by signal\n" if $interrupted;
+}
+
sub exec_backup {
my ($self, $rpcenv, $authuser) = @_;
@@ -1490,22 +1732,27 @@ sub exec_backup {
open my $job_start_fd, '>', \$job_start_log;
open my $job_end_fd, '>', \$job_end_log;
+ my $concurrency = $self->guest_backup_concurrency();
+ $concurrency = scalar($tasklist->@*) if scalar($tasklist->@*) < $concurrency;
+
my $starttime = time();
- my $errcount = 0;
eval {
$self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
$self->run_hook_script('job-start', undef, $job_start_fd);
- foreach my $task (@$tasklist) {
- $self->exec_backup_task($task);
- $errcount += 1 if $task->{state} ne 'ok';
+ if ($concurrency > 1) {
+ debugmsg('info', "running up to $concurrency guest backups at the same time");
+ exec_parallel_backup_tasks($self, $tasklist, $concurrency);
+ } else {
+ $self->exec_backup_task($_) for $tasklist->@*;
}
$self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
$self->run_hook_script('job-end', undef, $job_end_fd);
};
my $err = $@;
+ my $errcount = grep { $_->{state} ne 'ok' } $tasklist->@*;
if ($err) {
if ($self->{'backup-provider'}) {
diff --git a/test/Makefile b/test/Makefile
index 91761579d..bbe87a7c3 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -18,7 +18,7 @@ replication%.t: replication_test%.pl
./$<
.PHONY: test-vzdump
-test-vzdump: test-vzdump-guest-included test-vzdump-new
+test-vzdump: test-vzdump-guest-included test-vzdump-new test-vzdump-concurrency
.PHONY: test-vzdump-guest-included
test-vzdump-guest-included:
@@ -28,6 +28,10 @@ test-vzdump-guest-included:
test-vzdump-new:
./vzdump_new_test.pl
+.PHONY: test-vzdump-concurrency
+test-vzdump-concurrency:
+ ./vzdump_concurrency_test.pl
+
.PHONY: test-osd
test-osd:
./OSD_test.pl
diff --git a/test/vzdump_concurrency_test.pl b/test/vzdump_concurrency_test.pl
new file mode 100755
index 000000000..46d679490
--- /dev/null
+++ b/test/vzdump_concurrency_test.pl
@@ -0,0 +1,89 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib '..';
+
+use Test::More;
+use Test::MockModule;
+
+use PVE::VZDump;
+
+my $node_config = {};
+my $datacenter_config = {};
+
+my $node_config_module = Test::MockModule->new('PVE::NodeConfig');
+$node_config_module->mock(load_config => sub { return $node_config; });
+
+my $vzdump_module = Test::MockModule->new('PVE::VZDump');
+$vzdump_module->mock(
+ cfs_read_file => sub {
+ my ($filename) = @_;
+ die "unexpected file '$filename'\n" if $filename ne 'datacenter.cfg';
+ return $datacenter_config;
+ },
+);
+
+my $tests = [
+ {
+ description => 'defaults keep backing up one guest at a time',
+ expected => 1,
+ },
+ {
+ description => 'datacenter default applies to the node',
+ datacenter => 4,
+ expected => 4,
+ },
+ {
+ description => 'node configuration overrides datacenter default',
+ datacenter => 4,
+ node => 2,
+ expected => 2,
+ },
+ {
+ description => 'node configuration applies without datacenter default',
+ node => 3,
+ expected => 3,
+ },
+ {
+ description => 'job limit lowers node limit',
+ node => 4,
+ job => 2,
+ expected => 2,
+ },
+ {
+ description => 'job limit cannot raise node limit',
+ node => 2,
+ job => 8,
+ expected => 2,
+ },
+ {
+ description => 'job limit does not raise the default',
+ job => 8,
+ expected => 1,
+ },
+ {
+ description => 'external backup providers back up one guest at a time',
+ node => 4,
+ job => 4,
+ provider => 1,
+ expected => 1,
+ },
+];
+
+for my $test ($tests->@*) {
+ $datacenter_config =
+ defined($test->{datacenter})
+ ? { 'backup-concurrency' => { node => $test->{datacenter} } }
+ : {};
+ $node_config = defined($test->{node}) ? { 'backup-concurrency' => $test->{node} } : {};
+
+ my $self = bless { opts => {} }, 'PVE::VZDump';
+ $self->{opts}->{concurrency} = $test->{job} if defined($test->{job});
+ $self->{'backup-provider'} = {} if $test->{provider};
+
+ is($self->guest_backup_concurrency(), $test->{expected}, $test->{description});
+}
+
+done_testing();
--
2.47.3
next prev parent reply other threads:[~2026-11-18 13:32 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 13:18 [PATCH v2 cluster,storage,guest-common,manager,docs 00/20] parallel guest backups and backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 01/20] datacenter config: add backup concurrency setting Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 cluster 02/20] datacenter config: backup concurrency: add cluster-wide limit Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 storage 03/20] add backup concurrency option to storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 guest-common 04/20] vzdump: add options for parallel guest backups and overlapping jobs Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 05/20] vzdump: drop never-set fallback for the lock wait time Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 06/20] vzdump: notification: capture guest logs right after each backup Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 07/20] vzdump: stop all running backup jobs of the node Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 08/20] vzdump: show the error if a job could not get the global lock Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 09/20] vzdump: run job-init hook and set up target once the job may start Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 10/20] vzdump: list guests as skipped if the job could not start Thomas Lamprecht
2026-11-18 13:18 ` Thomas Lamprecht [this message]
2026-11-18 13:18 ` [PATCH v2 manager 12/20] vzdump: allow jobs to run alongside each other Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 13/20] ui: options: add backup concurrency for datacenter and nodes Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 14/20] ui: backup job: add concurrency and overlap options Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 15/20] ui: backup now: add overlap option Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 16/20] vzdump: limit concurrent guest backups cluster-wide Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 17/20] vzdump: honor backup concurrency limits of storages Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 18/20] api: backup info: add endpoints for cluster-wide backup slots Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 manager 19/20] ui: add cluster-wide and storage backup concurrency limits Thomas Lamprecht
2026-11-18 13:18 ` [PATCH v2 docs 20/20] vzdump: document concurrent backups and their limits Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118132947.1093069-12-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox