* [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies
@ 2026-11-20 16:05 Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.
As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
processed, then retries them in the order they failed up to a
configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.
The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.
If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.
Since a guest can now go through backup_init() and backup_cleanup()
several times within one job, the backup provider API documentation
and the directory example provider are updated accordingly.
The mobile UI does not offer the retry options yet. Manual backups
started there use the node defaults from /etc/vzdump.conf. Adding the
fields there could be done as a follow-up.
Testing info:
To simulate a transient failure, lock a guest before starting the
backup and unlock it while the job waits for the retry:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`
Tested: no retries with 'ignore', successful retry with 'retry' and
'retry-immediately' (including ordering and the warning status),
exhausted retries with backoff, no retry after a failing backup-end
hook, and the --stdout cases (explicit retry strategy rejected,
max-retries alone accepted, retry strategy from vzdump.conf ignored
with a warning).
Changes since v3:
- backup provider API: document that backup_init() and backup_cleanup()
can be called more than once for the same guest within a job
- backup provider example: reset per-guest state in backup_init(), so
a retry does not inherit the 'failed' flag or the backing image of
the failed attempt (preceded by a perltidy-only commit)
- clarify in the commit message and docs that the notification subject
shows the number of retried guests, and the guest table the retries
per guest
- cover letter: note the missing retry options in the mobile UI
Changes since v2 (thanks @Jonas):
- don't retry guests whose backup was already created (e.g. failing
backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
for retried guests
- show the number of retried guests in the notification subject and
each guest's retry count in the guest table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
and UI defaults
Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
window, which has no advanced section
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464
pve-storage-plugin-examples:
Jakob Klocker (2):
backup provider example: run proxmox-perltidy
backup provider example: reset per-guest state in backup_init
.../BackupProvider/Plugin/DirectoryExample.pm | 633 +++++++++---------
1 file changed, 328 insertions(+), 305 deletions(-)
pve-storage:
Jakob Klocker (1):
backup provider api: document repeated backup_init() calls on retries
src/PVE/BackupProvider/Plugin/Base.pm | 12 ++++++++++++
1 file changed, 12 insertions(+)
pve-guest-common:
Jakob Klocker (1):
vzdump: add error-behavior and max-retries options
src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
pve-manager:
Jakob Klocker (2):
fix #6464: vzdump: add retry strategies for failed guest backups
ui: backup: add error behavior and retry options
PVE/VZDump.pm | 219 +++++++++++++++++++-
www/manager6/dc/Backup.js | 2 +-
www/manager6/panel/BackupAdvancedOptions.js | 52 +++++
www/manager6/window/Backup.js | 56 ++++-
4 files changed, 323 insertions(+), 6 deletions(-)
pve-docs:
Jakob Klocker (1):
vzdump: document backup retry options
vzdump.adoc | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
Summary over all repositories:
8 files changed, 747 insertions(+), 311 deletions(-)
--
Generated by murpp 0.12.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
No functional change intended.
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
.../BackupProvider/Plugin/DirectoryExample.pm | 629 +++++++++---------
1 file changed, 324 insertions(+), 305 deletions(-)
diff --git a/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm b/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
index 365ce40..f61198b 100644
--- a/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
+++ b/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
@@ -45,76 +45,79 @@ my sub bind_next_free_dev_nbd_node {
# This avoids race conditions between allocation and use.
die "file '/sys/module/nbd' does not exist - 'nbd' kernel module not loaded?"
- if !-e "/sys/module/nbd";
+ if !-e "/sys/module/nbd";
my $line = PVE::Tools::file_read_firstline("/sys/module/nbd/parameters/nbds_max")
- or die "could not read 'nbds_max' parameter file for 'nbd' kernel module\n";
+ or die "could not read 'nbds_max' parameter file for 'nbd' kernel module\n";
my ($nbds_max) = ($line =~ m/(\d+)/)
- or die "could not determine 'nbds_max' parameter for 'nbd' kernel module\n";
+ or die "could not determine 'nbds_max' parameter for 'nbd' kernel module\n";
my $filename = "/run/qemu-server/reserved-dev-nbd-nodes";
my $code = sub {
- my $expiretime = 60;
- my $ctime = time();
-
- my $used = {};
- my $latest = [0, 0];
-
- if (my $fh = IO::File->new ($filename, "r")) {
- while (my $line = <$fh>) {
- if ($line =~ m/^(\d+)\s(\d+)$/) {
- my ($n, $timestamp) = ($1, $2);
-
- $latest = [$n, $timestamp] if $latest->[1] <= $timestamp;
-
- if (($timestamp + $expiretime) > $ctime) {
- $used->{$n} = $timestamp; # not expired
- }
- }
- }
- }
-
- my $new_n;
- for (my $count = 0; $count < $nbds_max; $count++) {
- my $n = ($latest->[0] + $count) % $nbds_max;
- my $block_device = "/dev/nbd${n}";
- next if $used->{$n}; # reserved
- next if !-e $block_device;
-
- my $st = File::stat::stat("/run/lock/qemu-nbd-nbd${n}");
- next if defined($st) && S_ISSOCK($st->mode) && $st->uid == 0; # in use
-
- # Used to avoid looping if there are other issues then the NBD node being in use
- my $socket_error = 0;
- eval {
- my $errfunc = sub {
- my ($line) = @_;
- $socket_error = 1 if $line =~ m/^qemu-nbd: Failed to set NBD socket$/;
- log_warn($line);
- };
- run_command(["qemu-nbd", "-c", $block_device, $options->@*], errfunc => $errfunc);
- };
- if (my $err = $@) {
- die $err if !$socket_error;
- log_warn("unable to bind $block_device - trying next one");
- next;
- }
- $used->{$n} = $ctime;
- $new_n = $n;
- last;
- }
-
- my $data = "";
- $data .= "$_ $used->{$_}\n" for keys $used->%*;
-
- PVE::Tools::file_set_contents($filename, $data);
-
- return defined($new_n) ? "/dev/nbd${new_n}" : undef;
+ my $expiretime = 60;
+ my $ctime = time();
+
+ my $used = {};
+ my $latest = [0, 0];
+
+ if (my $fh = IO::File->new($filename, "r")) {
+ while (my $line = <$fh>) {
+ if ($line =~ m/^(\d+)\s(\d+)$/) {
+ my ($n, $timestamp) = ($1, $2);
+
+ $latest = [$n, $timestamp] if $latest->[1] <= $timestamp;
+
+ if (($timestamp + $expiretime) > $ctime) {
+ $used->{$n} = $timestamp; # not expired
+ }
+ }
+ }
+ }
+
+ my $new_n;
+ for (my $count = 0; $count < $nbds_max; $count++) {
+ my $n = ($latest->[0] + $count) % $nbds_max;
+ my $block_device = "/dev/nbd${n}";
+ next if $used->{$n}; # reserved
+ next if !-e $block_device;
+
+ my $st = File::stat::stat("/run/lock/qemu-nbd-nbd${n}");
+ next if defined($st) && S_ISSOCK($st->mode) && $st->uid == 0; # in use
+
+ # Used to avoid looping if there are other issues then the NBD node being in use
+ my $socket_error = 0;
+ eval {
+ my $errfunc = sub {
+ my ($line) = @_;
+ $socket_error = 1 if $line =~ m/^qemu-nbd: Failed to set NBD socket$/;
+ log_warn($line);
+ };
+ run_command(
+ ["qemu-nbd", "-c", $block_device, $options->@*],
+ errfunc => $errfunc,
+ );
+ };
+ if (my $err = $@) {
+ die $err if !$socket_error;
+ log_warn("unable to bind $block_device - trying next one");
+ next;
+ }
+ $used->{$n} = $ctime;
+ $new_n = $n;
+ last;
+ }
+
+ my $data = "";
+ $data .= "$_ $used->{$_}\n" for keys $used->%*;
+
+ PVE::Tools::file_set_contents($filename, $data);
+
+ return defined($new_n) ? "/dev/nbd${new_n}" : undef;
};
my $block_device =
- PVE::Tools::lock_file('/run/lock/qemu-server/reserved-dev-nbd-nodes.lock', 10, $code);
+ PVE::Tools::lock_file('/run/lock/qemu-server/reserved-dev-nbd-nodes.lock', 10, $code);
die $@ if $@;
die "unable to find free /dev/nbdX block device node\n" if !$block_device;
@@ -128,10 +131,10 @@ sub new {
my ($class, $storage_plugin, $scfg, $storeid, $log_function) = @_;
my $self = bless {
- scfg => $scfg,
- storeid => $storeid,
- 'storage-plugin' => $storage_plugin,
- 'log-function' => $log_function,
+ scfg => $scfg,
+ storeid => $storeid,
+ 'storage-plugin' => $storage_plugin,
+ 'log-function' => $log_function,
}, $class;
return $self;
@@ -151,8 +154,8 @@ sub job_init {
log_info($self, "job init called");
if (!-e '/sys/module/nbd/parameters') {
- die "required 'nbd' kernel module not loaded - use 'modprobe nbd nbds_max=128' to load it"
- ." manually\n";
+ die "required 'nbd' kernel module not loaded - use 'modprobe nbd nbds_max=128' to load it"
+ . " manually\n";
}
log_info($self, "backup provider initialized successfully for new job $start_time");
@@ -203,46 +206,43 @@ my sub update_previous_info {
my $previous_info_file = "$self->{scfg}->{path}/$vmid/previous-info";
if (defined(my $info = $self->{$vmid}->{previous})) {
- file_set_contents($previous_info_file, to_json($info));
+ file_set_contents($previous_info_file, to_json($info));
} else {
- unlink($previous_info_file);
+ unlink($previous_info_file);
}
}
-
sub backup_cleanup {
my ($self, $vmid, $vmtype, $success, $info) = @_;
if ($success) {
- log_info($self, "backup cleanup called - success");
- eval {
- update_previous_info($self, $vmid, $self->{$vmid}->{previous});
- };
- if (my $err = $@) {
- log_error($self, "failed to update previous-info file: $err");
- }
- my $size = 0;
- my $backup_dir = $self->{$vmid}->{'backup-dir'};
- my @backup_files = glob("$backup_dir/*");
- $size += -s $_ for @backup_files;
- my $stats = { 'archive-size' => $size };
- return { 'stats' => $stats };
+ log_info($self, "backup cleanup called - success");
+ eval { update_previous_info($self, $vmid, $self->{$vmid}->{previous}); };
+ if (my $err = $@) {
+ log_error($self, "failed to update previous-info file: $err");
+ }
+ my $size = 0;
+ my $backup_dir = $self->{$vmid}->{'backup-dir'};
+ my @backup_files = glob("$backup_dir/*");
+ $size += -s $_ for @backup_files;
+ my $stats = { 'archive-size' => $size };
+ return { 'stats' => $stats };
} else {
- log_info($self, "backup cleanup called - failure");
+ log_info($self, "backup cleanup called - failure");
- $self->{$vmid}->{failed} = 1;
+ $self->{$vmid}->{failed} = 1;
- if (my $dir = $self->{$vmid}->{'backup-dir'}) {
- eval { remove_tree($dir) };
- log_warning($self, "unable to clean up $dir - $@") if $@;
- }
+ if (my $dir = $self->{$vmid}->{'backup-dir'}) {
+ eval { remove_tree($dir) };
+ log_warning($self, "unable to clean up $dir - $@") if $@;
+ }
- # Restore old previous-info so next attempt can re-use bitmap again
- if (my $info = $self->{$vmid}->{'old-previous-info'}) {
- my $previous_info_dir = "$self->{scfg}->{path}/$vmid/";
- my $previous_info_file = "$previous_info_dir/previous-info";
- file_set_contents($previous_info_file, $info);
- }
+ # Restore old previous-info so next attempt can re-use bitmap again
+ if (my $info = $self->{$vmid}->{'old-previous-info'}) {
+ my $previous_info_dir = "$self->{scfg}->{path}/$vmid/";
+ my $previous_info_file = "$previous_info_dir/previous-info";
+ file_set_contents($previous_info_file, $info);
+ }
}
}
@@ -270,20 +270,20 @@ sub backup_vm_query_incremental {
my $info = get_previous_info_tainted($self, $vmid);
for my $device_name (keys $volumes->%*) {
- $out->{$device_name} = 'new';
+ $out->{$device_name} = 'new';
- my $prev_file = $info->{$device_name};
- next if !defined $prev_file;
- # it's type-time/disk.qcow2
- next if $prev_file !~ m!^([^/]+/[^/]+\.qcow2)$!;
- $prev_file = $1; # untaint
+ my $prev_file = $info->{$device_name};
+ next if !defined $prev_file;
+ # it's type-time/disk.qcow2
+ next if $prev_file !~ m!^([^/]+/[^/]+\.qcow2)$!;
+ $prev_file = $1; # untaint
- my $full_path = "$self->{scfg}->{path}/$vmid/$prev_file";
+ my $full_path = "$self->{scfg}->{path}/$vmid/$prev_file";
- if (-e $full_path) {
- $self->{$vmid}->{previous}->{$device_name} = $prev_file;
- $out->{$device_name} = 'use';
- }
+ if (-e $full_path) {
+ $self->{$vmid}->{previous}->{$device_name} = $prev_file;
+ $out->{$device_name} = 'use';
+ }
}
return $out;
@@ -303,7 +303,7 @@ sub backup_handle_log_file {
my $log_dir = $self->{$vmid}->{'backup-dir'};
if ($self->{$vmid}->{failed}) {
- $log_dir .= ".failed";
+ $log_dir .= ".failed";
}
make_path($log_dir);
die "unable to create directory $log_dir\n" if !-d $log_dir;
@@ -314,7 +314,16 @@ sub backup_handle_log_file {
}
my sub backup_file {
- my ($self, $vmid, $device_name, $size, $in_fh, $bitmap_mode, $next_dirty_region, $bandwidth_limit) = @_;
+ my (
+ $self,
+ $vmid,
+ $device_name,
+ $size,
+ $in_fh,
+ $bitmap_mode,
+ $next_dirty_region,
+ $bandwidth_limit,
+ ) = @_;
# TODO honor bandwidth_limit
@@ -322,78 +331,82 @@ my sub backup_file {
my $create_cmd = ["qemu-img", "create", "-f", "qcow2", $target, $size];
if (my $previous_file = $self->{$vmid}->{previous}->{$device_name}) {
- my $target_base = "../$previous_file";
- push $create_cmd->@*, "-b", $target_base, "-F", "qcow2";
+ my $target_base = "../$previous_file";
+ push $create_cmd->@*, "-b", $target_base, "-F", "qcow2";
}
run_command($create_cmd);
my $nbd_node;
eval {
- # allows to easily write to qcow2 target
- $nbd_node = bind_next_free_dev_nbd_node([$target, '--format=qcow2']);
- # FIXME use nbdfuse like in qemu-server rather than qemu-nbd. Seems like there is a race and
- # sysseek() can fail with "Invalid argument" if done too early...
- sleep 1;
-
- my $block_size = 4 * 1024 * 1024; # 4 MiB
-
- my $out_fh = IO::File->new($nbd_node, "r+")
- or die "unable to open NBD backup target - $!\n";
-
- my $buffer = '';
- my $skip_discard;
-
- while (scalar((my $region_offset, my $region_length) = $next_dirty_region->())) {
- sysseek($in_fh, $region_offset, SEEK_SET)
- // die "unable to seek '$region_offset' in NBD backup source - $!\n";
- sysseek($out_fh, $region_offset, SEEK_SET)
- // die "unable to seek '$region_offset' in NBD backup target - $!\n";
-
- my $local_offset = 0; # within the region
- while ($local_offset < $region_length) {
- my $remaining = $region_length - $local_offset;
- my $request_size = $remaining < $block_size ? $remaining : $block_size;
- my $offset = $region_offset + $local_offset;
-
- my $read = sysread($in_fh, $buffer, $request_size);
- die "failed to read from backup source - $!\n" if !defined($read);
- die "premature EOF while reading backup source\n" if $read == 0;
-
- my $written = 0;
- while ($written < $read) {
- my $res = syswrite($out_fh, $buffer, $request_size - $written, $written);
- die "failed to write to backup target - $!\n" if !defined($res);
- die "unable to progress writing to backup target\n" if $res == 0;
- $written += $res;
- }
-
- if (!$skip_discard) {
- eval { PVE::Storage::Common::deallocate($in_fh, $offset, $request_size); };
- if (my $err = $@) {
- # Just assume that if one request didn't work, others won't either.
- log_warning(
- $self, "discard source failed (skipping further discards) - $err");
- $skip_discard = 1;
- }
- }
-
- $local_offset += $request_size;
- }
- }
- $out_fh->sync();
+ # allows to easily write to qcow2 target
+ $nbd_node = bind_next_free_dev_nbd_node([$target, '--format=qcow2']);
+ # FIXME use nbdfuse like in qemu-server rather than qemu-nbd. Seems like there is a race and
+ # sysseek() can fail with "Invalid argument" if done too early...
+ sleep 1;
+
+ my $block_size = 4 * 1024 * 1024; # 4 MiB
+
+ my $out_fh = IO::File->new($nbd_node, "r+")
+ or die "unable to open NBD backup target - $!\n";
+
+ my $buffer = '';
+ my $skip_discard;
+
+ while (scalar((my $region_offset, my $region_length) = $next_dirty_region->())) {
+ sysseek($in_fh, $region_offset, SEEK_SET)
+ // die "unable to seek '$region_offset' in NBD backup source - $!\n";
+ sysseek($out_fh, $region_offset, SEEK_SET)
+ // die "unable to seek '$region_offset' in NBD backup target - $!\n";
+
+ my $local_offset = 0; # within the region
+ while ($local_offset < $region_length) {
+ my $remaining = $region_length - $local_offset;
+ my $request_size = $remaining < $block_size ? $remaining : $block_size;
+ my $offset = $region_offset + $local_offset;
+
+ my $read = sysread($in_fh, $buffer, $request_size);
+ die "failed to read from backup source - $!\n" if !defined($read);
+ die "premature EOF while reading backup source\n" if $read == 0;
+
+ my $written = 0;
+ while ($written < $read) {
+ my $res = syswrite($out_fh, $buffer, $request_size - $written, $written);
+ die "failed to write to backup target - $!\n" if !defined($res);
+ die "unable to progress writing to backup target\n" if $res == 0;
+ $written += $res;
+ }
+
+ if (!$skip_discard) {
+ eval { PVE::Storage::Common::deallocate($in_fh, $offset, $request_size); };
+ if (my $err = $@) {
+ # Just assume that if one request didn't work, others won't either.
+ log_warning(
+ $self,
+ "discard source failed (skipping further discards) - $err",
+ );
+ $skip_discard = 1;
+ }
+ }
+
+ $local_offset += $request_size;
+ }
+ }
+ $out_fh->sync();
};
my $err = $@;
- $self->{$vmid}->{previous}->{$device_name} = "$self->{$vmid}->{'archive-subdir'}/${device_name}.qcow2";
+ $self->{$vmid}->{previous}->{$device_name} =
+ "$self->{$vmid}->{'archive-subdir'}/${device_name}.qcow2";
- eval { run_command(['qemu-nbd', '-d', $nbd_node ]); };
+ eval { run_command(['qemu-nbd', '-d', $nbd_node]); };
log_warning($self, "unable to disconnect NBD backup target - $@") if $@;
die $err if $err;
}
my sub backup_nbd {
- my ($self, $vmid, $device_name, $size, $nbd_path, $bitmap_mode, $bitmap_name, $bandwidth_limit) = @_;
+ my ($self, $vmid, $device_name, $size, $nbd_path, $bitmap_mode, $bitmap_name, $bandwidth_limit)
+ = @_;
# TODO honor bandwidth_limit
@@ -411,69 +424,70 @@ my sub backup_nbd {
# set when the bitmap is dirty, see QEMU's docs/interop/nbd.txt
my $dirty_bitmap = [];
if ($bitmap_mode ne 'none') {
- my $input = IO::File->new();
- my $info = IO::File->new();
- $error_fh = IO::File->new();
- my $nbdinfo_cmd = ["nbdinfo", $nbd_info_uri, "--map=qemu:dirty-bitmap:${bitmap_name}"];
- $cpid = open3($input, $info, $error_fh, $nbdinfo_cmd->@*)
- or die "failed to spawn nbdinfo child - $!\n";
-
- $next_dirty_region = sub {
- my ($offset, $length, $type);
- do {
- my $line = <$info>;
- return if !$line;
- die "unexpected output from nbdinfo - $line\n"
- if $line !~ m/^\s*(\d+)\s*(\d+)\s*(\d+)/; # also untaints
- ($offset, $length, $type) = ($1, $2, $3);
- } while (($type & 0x1) == 0); # not dirty
- return ($offset, $length);
- };
+ my $input = IO::File->new();
+ my $info = IO::File->new();
+ $error_fh = IO::File->new();
+ my $nbdinfo_cmd = ["nbdinfo", $nbd_info_uri, "--map=qemu:dirty-bitmap:${bitmap_name}"];
+ $cpid = open3($input, $info, $error_fh, $nbdinfo_cmd->@*)
+ or die "failed to spawn nbdinfo child - $!\n";
+
+ $next_dirty_region = sub {
+ my ($offset, $length, $type);
+ do {
+ my $line = <$info>;
+ return if !$line;
+ die "unexpected output from nbdinfo - $line\n"
+ if $line !~ m/^\s*(\d+)\s*(\d+)\s*(\d+)/; # also untaints
+ ($offset, $length, $type) = ($1, $2, $3);
+ } while (($type & 0x1) == 0); # not dirty
+ return ($offset, $length);
+ };
} else {
- my $done = 0;
- $next_dirty_region = sub {
- return if $done;
- $done = 1;
- return (0, $size);
- };
+ my $done = 0;
+ $next_dirty_region = sub {
+ return if $done;
+ $done = 1;
+ return (0, $size);
+ };
}
my $nbd_node;
eval {
- $nbd_node = bind_next_free_dev_nbd_node([$qemu_nbd_uri, "--format=raw", "--discard=on"]);
-
- my $in_fh = IO::File->new($nbd_node, 'r+')
- or die "unable to open NBD backup source '$nbd_node' - $!\n";
-
- backup_file(
- $self,
- $vmid,
- $device_name,
- $size,
- $in_fh,
- $bitmap_mode,
- $next_dirty_region,
- $bandwidth_limit,
- );
+ $nbd_node =
+ bind_next_free_dev_nbd_node([$qemu_nbd_uri, "--format=raw", "--discard=on"]);
+
+ my $in_fh = IO::File->new($nbd_node, 'r+')
+ or die "unable to open NBD backup source '$nbd_node' - $!\n";
+
+ backup_file(
+ $self,
+ $vmid,
+ $device_name,
+ $size,
+ $in_fh,
+ $bitmap_mode,
+ $next_dirty_region,
+ $bandwidth_limit,
+ );
};
my $err = $@;
- eval { run_command(["qemu-nbd", "-d", $nbd_node ]); };
+ eval { run_command(["qemu-nbd", "-d", $nbd_node]); };
log_warning($self, "unable to disconnect NBD backup source - $@") if $@;
if ($cpid) {
- my $waited;
- my $wait_limit = 5;
- for ($waited = 0; $waited < $wait_limit && waitpid($cpid, POSIX::WNOHANG) == 0; $waited++) {
- kill 15, $cpid if $waited == 0;
- sleep 1;
- }
- if ($waited == $wait_limit) {
- kill 9, $cpid;
- sleep 1;
- log_warning($self, "unable to collect nbdinfo child process")
- if waitpid($cpid, POSIX::WNOHANG) == 0;
- }
+ my $waited;
+ my $wait_limit = 5;
+ for ($waited = 0; $waited < $wait_limit && waitpid($cpid, POSIX::WNOHANG) == 0; $waited++) {
+ kill 15, $cpid if $waited == 0;
+ sleep 1;
+ }
+ if ($waited == $wait_limit) {
+ kill 9, $cpid;
+ sleep 1;
+ log_warning($self, "unable to collect nbdinfo child process")
+ if waitpid($cpid, POSIX::WNOHANG) == 0;
+ }
}
die $err if $err;
@@ -485,29 +499,29 @@ my sub backup_vm_volume {
my $backup_mechanism = $self->{'storage-plugin'}->get_vm_backup_mechanism($self->{scfg});
if ($backup_mechanism eq 'nbd') {
- backup_nbd(
- $self,
- $vmid,
- $device_name,
- $info->{size},
- $info->{'nbd-path'},
- $info->{'bitmap-mode'},
- $info->{'bitmap-name'},
- $bandwidth_limit,
- );
+ backup_nbd(
+ $self,
+ $vmid,
+ $device_name,
+ $info->{size},
+ $info->{'nbd-path'},
+ $info->{'bitmap-mode'},
+ $info->{'bitmap-name'},
+ $bandwidth_limit,
+ );
} elsif ($backup_mechanism eq 'file-handle') {
- backup_file(
- $self,
- $vmid,
- $device_name,
- $info->{size},
- $info->{'file-handle'},
- $info->{'bitmap-mode'},
- $info->{'next-dirty-region'},
- $bandwidth_limit,
- );
+ backup_file(
+ $self,
+ $vmid,
+ $device_name,
+ $info->{size},
+ $info->{'file-handle'},
+ $info->{'bitmap-mode'},
+ $info->{'next-dirty-region'},
+ $bandwidth_limit,
+ );
} else {
- die "internal error - unknown VM backup mechansim '$backup_mechanism'\n";
+ die "internal error - unknown VM backup mechansim '$backup_mechanism'\n";
}
}
@@ -518,13 +532,18 @@ sub backup_vm {
file_set_contents($target, $guest_config);
if (my $firewall_config = $info->{'firewall-config'}) {
- $target = "$self->{$vmid}->{'backup-dir'}/firewall.conf";
- file_set_contents($target, $firewall_config);
+ $target = "$self->{$vmid}->{'backup-dir'}/firewall.conf";
+ file_set_contents($target, $firewall_config);
}
for my $device_name (sort keys $volumes->%*) {
- backup_vm_volume(
- $self, $vmid, $device_name, $volumes->{$device_name}, $info->{'bandwidth-limit'});
+ backup_vm_volume(
+ $self,
+ $vmid,
+ $device_name,
+ $volumes->{$device_name},
+ $info->{'bandwidth-limit'},
+ );
}
}
@@ -535,10 +554,10 @@ my sub backup_directory_tar {
# copied from PVE::Storage::Plugin::COMMON_TAR_FLAGS
my @tar_flags = qw(
- --one-file-system
- -p --sparse --numeric-owner --acls
- --xattrs --xattrs-include=user.* --xattrs-include=security.capability
- --warning=no-file-ignored --warning=no-xattr-write
+ --one-file-system
+ -p --sparse --numeric-owner --acls
+ --xattrs --xattrs-include=user.* --xattrs-include=security.capability
+ --warning=no-file-ignored --warning=no-xattr-write
);
my $tar = ['tar', 'cpf', '-', '--totals', @tar_flags];
@@ -548,11 +567,11 @@ my sub backup_directory_tar {
my @exclude_no_anchored = ();
my @exclude_anchored = ();
for my $pattern ($exclude_patterns->@*) {
- if ($pattern !~ m|^/|) {
- push @exclude_no_anchored, $pattern;
- } else {
- push @exclude_anchored, $pattern;
- }
+ if ($pattern !~ m|^/|) {
+ push @exclude_no_anchored, $pattern;
+ } else {
+ push @exclude_anchored, $pattern;
+ }
}
push @$tar, '--no-anchored';
@@ -564,22 +583,22 @@ my sub backup_directory_tar {
push @$tar, '--', $sources->@*;
- my $cmd = [ $tar ];
+ my $cmd = [$tar];
- push @$cmd, [ 'cstream', '-t', $bandwidth_limit * 1024 ] if $bandwidth_limit;
+ push @$cmd, ['cstream', '-t', $bandwidth_limit * 1024] if $bandwidth_limit;
my $target = "$self->{$vmid}->{'backup-dir'}/archive.tar";
- push @{$cmd->[-1]}, \(">" . PVE::Tools::shellquote($target));
+ push @{ $cmd->[-1] }, \(">" . PVE::Tools::shellquote($target));
my $logfunc = sub {
- my $line = shift;
- log_info($self, "tar: $line");
+ my $line = shift;
+ log_info($self, "tar: $line");
};
PVE::Tools::run_command($cmd, logfunc => $logfunc);
return;
-};
+}
# NOTE This only serves as an example to illustrate the 'directory' restore mechanism. It is not
# fleshed out properly, e.g. I didn't check if exclusion is compatible with
@@ -594,26 +613,26 @@ my sub backup_directory_squashfs {
push $mksquashfs->@*, '-wildcards';
for my $pattern ($exclude_patterns->@*) {
- if ($pattern !~ m|^/|) { # non-anchored
- push $mksquashfs->@*, '-e', "... $pattern";
- } else { # anchored
- push $mksquashfs->@*, '-e', substr($pattern, 1); # need to strip leading slash
- }
+ if ($pattern !~ m|^/|) { # non-anchored
+ push $mksquashfs->@*, '-e', "... $pattern";
+ } else { # anchored
+ push $mksquashfs->@*, '-e', substr($pattern, 1); # need to strip leading slash
+ }
}
- my $cmd = [ $mksquashfs ];
+ my $cmd = [$mksquashfs];
- push @$cmd, [ 'cstream', '-t', $bandwidth_limit * 1024 ] if $bandwidth_limit;
+ push @$cmd, ['cstream', '-t', $bandwidth_limit * 1024] if $bandwidth_limit;
my $logfunc = sub {
- my $line = shift;
- log_info($self, "mksquashfs: $line");
+ my $line = shift;
+ log_info($self, "mksquashfs: $line");
};
PVE::Tools::run_command($cmd, logfunc => $logfunc);
return;
-};
+}
sub backup_container {
my ($self, $vmid, $guest_config, $exclude_patterns, $info) = @_;
@@ -622,30 +641,30 @@ sub backup_container {
file_set_contents($target, $guest_config);
if (my $firewall_config = $info->{'firewall-config'}) {
- $target = "$self->{$vmid}->{'backup-dir'}/firewall.conf";
- file_set_contents($target, $firewall_config);
+ $target = "$self->{$vmid}->{'backup-dir'}/firewall.conf";
+ file_set_contents($target, $firewall_config);
}
my $backup_mode = $self->{'storage-plugin'}->get_lxc_backup_mode($self->{scfg});
if ($backup_mode eq 'tar') {
- backup_directory_tar(
- $self,
- $vmid,
- $info->{directory},
- $exclude_patterns,
- $info->{sources},
- $info->{'bandwidth-limit'},
- );
+ backup_directory_tar(
+ $self,
+ $vmid,
+ $info->{directory},
+ $exclude_patterns,
+ $info->{sources},
+ $info->{'bandwidth-limit'},
+ );
} elsif ($backup_mode eq 'squashfs') {
- backup_directory_squashfs(
- $self,
- $vmid,
- $info->{directory},
- $exclude_patterns,
- $info->{'bandwidth-limit'},
- );
+ backup_directory_squashfs(
+ $self,
+ $vmid,
+ $info->{directory},
+ $exclude_patterns,
+ $info->{'bandwidth-limit'},
+ );
} else {
- die "got unexpected backup mode '$backup_mode' from storage plugin\n";
+ die "got unexpected backup mode '$backup_mode' from storage plugin\n";
}
}
@@ -660,19 +679,19 @@ sub restore_get_mechanism {
return ('qemu-img', $vmtype) if $vmtype eq 'qemu';
if ($vmtype eq 'lxc') {
- my (undef, $relative_backup_dir) = $self->{'storage-plugin'}->parse_volname($volname);
+ my (undef, $relative_backup_dir) = $self->{'storage-plugin'}->parse_volname($volname);
- if (-e "$self->{scfg}->{path}/${relative_backup_dir}/archive.tar") {
- $self->{'restore-mechanisms'}->{$volname} = 'tar';
- return ('tar', $vmtype);
- }
+ if (-e "$self->{scfg}->{path}/${relative_backup_dir}/archive.tar") {
+ $self->{'restore-mechanisms'}->{$volname} = 'tar';
+ return ('tar', $vmtype);
+ }
- if (-e "$self->{scfg}->{path}/${relative_backup_dir}/archive.sqfs") {
- $self->{'restore-mechanisms'}->{$volname} = 'directory';
- return ('directory', $vmtype)
- }
+ if (-e "$self->{scfg}->{path}/${relative_backup_dir}/archive.sqfs") {
+ $self->{'restore-mechanisms'}->{$volname} = 'directory';
+ return ('directory', $vmtype);
+ }
- die "unable to find archive '$volname'\n";
+ die "unable to find archive '$volname'\n";
}
die "cannot restore unexpected guest type '$vmtype'\n";
@@ -708,9 +727,9 @@ sub restore_vm_init {
my @backup_files = glob("$backup_dir/*");
for my $backup_file (@backup_files) {
- next if $backup_file !~ m!^(.*/(.*)\.qcow2)$!;
- $backup_file = $1; # untaint
- $res->{$2}->{size} = PVE::Storage::Plugin::file_size_info($backup_file, undef, 'qcow2');
+ next if $backup_file !~ m!^(.*/(.*)\.qcow2)$!;
+ $backup_file = $1; # untaint
+ $res->{$2}->{size} = PVE::Storage::Plugin::file_size_info($backup_file, undef, 'qcow2');
}
return $res;
@@ -735,7 +754,7 @@ sub restore_vm_volume_init {
my $nbd_node = bind_next_free_dev_nbd_node([$image]);
$self->{"${volname}/${device_name}"}->{'nbd-node'} = $nbd_node;
return {
- 'qemu-img-path' => $nbd_node,
+ 'qemu-img-path' => $nbd_node,
};
}
@@ -743,7 +762,7 @@ sub restore_vm_volume_cleanup {
my ($self, $volname, $device_name, $info) = @_;
if (my $nbd_node = delete($self->{"${volname}/${device_name}"}->{'nbd-node'})) {
- PVE::Tools::run_command(['qemu-nbd', '-d', $nbd_node]);
+ PVE::Tools::run_command(['qemu-nbd', '-d', $nbd_node]);
}
return;
@@ -786,11 +805,11 @@ sub restore_container_init {
my ($self, $volname, $info) = @_;
if ($self->{'restore-mechanisms'}->{$volname} eq 'tar') {
- return restore_tar_init($self, $volname);
+ return restore_tar_init($self, $volname);
} elsif ($self->{'restore-mechanisms'}->{$volname} eq 'directory') {
- return restore_directory_init($self, $volname);
+ return restore_directory_init($self, $volname);
} else {
- die "no restore mechanism set for '$volname'\n";
+ die "no restore mechanism set for '$volname'\n";
}
}
@@ -798,11 +817,11 @@ sub restore_container_cleanup {
my ($self, $volname, $info) = @_;
if ($self->{'restore-mechanisms'}->{$volname} eq 'tar') {
- return; # nothing to do
+ return; # nothing to do
} elsif ($self->{'restore-mechanisms'}->{$volname} eq 'directory') {
- return restore_directory_cleanup($self, $volname);
+ return restore_directory_cleanup($self, $volname);
} else {
- die "no restore mechanism set for '$volname'\n";
+ die "no restore mechanism set for '$volname'\n";
}
}
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
With backup retries, backup_init() can be called again for the same
guest within one job. Reset the per-guest state, so a retry does not
inherit the 'failed' flag or the removed backing image of the failed
attempt.
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
.../src/PVE/BackupProvider/Plugin/DirectoryExample.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm b/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
index f61198b..2a53956 100644
--- a/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
+++ b/backup-provider-directory/src/PVE/BackupProvider/Plugin/DirectoryExample.pm
@@ -174,6 +174,10 @@ sub job_cleanup {
sub backup_init {
my ($self, $vmid, $vmtype, $backup_time) = @_;
+ # Start with fresh per-guest state. A failed attempt of the same job may have left
+ # state behind, like the 'failed' flag or 'previous' pointing to its removed directory.
+ $self->{$vmid} = {};
+
my $archive_subdir = "${vmtype}-${backup_time}";
my $archive = "${vmid}/${archive_subdir}";
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
vzdump can retry failed guest backups within the same job, depending
on the new 'error-behavior' option. As a result, backup_init() and
backup_cleanup() can be called multiple times for the same guest on
the same provider instance.
Document that providers must not carry over per-guest state from a
failed attempt.
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
src/PVE/BackupProvider/Plugin/Base.pm | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/src/PVE/BackupProvider/Plugin/Base.pm b/src/PVE/BackupProvider/Plugin/Base.pm
index 9358fc6..c693db0 100644
--- a/src/PVE/BackupProvider/Plugin/Base.pm
+++ b/src/PVE/BackupProvider/Plugin/Base.pm
@@ -48,6 +48,11 @@ backup_container_prepare() call while still privileged. The actual
backup_container() call happens as the (unprivileged) container root user, so
that the file owner and group IDs match the container's perspective.
+If the backup of a guest fails, the job may retry it later, depending on the
+C<error-behavior> backup option. Each attempt goes through its own
+backup_init() to backup_cleanup() cycle on the same backup provider instance,
+so these methods can be called more than once for the same guest within a job.
+
=head3 1.1 Backup Mechanisms
VM:
@@ -254,6 +259,13 @@ C</> and colon C<:>.
Use C<$self> to remember it for the C<backup_container()> or C<backup_vm()>
method that will be called later.
+If a previous attempt for the same guest failed and the job retries it,
+C<backup_init()> is called again for that guest within the same job, after
+C<backup_cleanup()> was called with C<$success> being false. Each call starts a
+new attempt, so do not carry over per-guest state from an earlier attempt. For
+example, reset what was remembered in C<$self> for this guest, and do not base
+an incremental backup on data of the failed attempt.
+
Parameters:
=over
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
` (2 preceding siblings ...)
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
Add two new backup job options to control what happens when a guest
backup fails:
- 'error-behavior' selects the behavior. 'ignore' keeps the current
behavior and is the default. 'retry' queues the failed guest and
retries it after the remaining guests have been backed up.
'retry-immediately' retries the failed guest right away, before
continuing with the next one.
- 'max-retries' limits how many times a failed guest backup is retried
(1-10, default 5).
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 2cb48a3..2babd11 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -384,6 +384,24 @@ my $confdesc = {
optional => 1,
enum => ['legacy', 'data', 'metadata'],
},
+ 'error-behavior' => {
+ type => 'string',
+ description => "How to handle guest backup failures. 'ignore' does not retry,"
+ . " 'retry' retries failed guests after all other guests have been processed,"
+ . " 'retry-immediately' retries a failed guest before continuing with the next one.",
+ enum => ['ignore', 'retry', 'retry-immediately'],
+ optional => 1,
+ default => 'ignore',
+ },
+ 'max-retries' => {
+ type => 'integer',
+ description => "Maximum number of times a failed guest backup is retried."
+ . " Only used if 'error-behavior' is not 'ignore'.",
+ minimum => 1,
+ maximum => 10,
+ optional => 1,
+ default => 5,
+ },
};
sub get_confdesc {
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
` (3 preceding siblings ...)
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
Add the `error-behavior` and `max-retries` options to automatically
retry failed guest backups, which helps with transient failures.
Supported retry strategies:
- `ignore`: Preserves current behavior (no retries, default).
- `retry-immediately`: Retries the guest inline after failure.
- `retry`: Defers failed guests until all other guests have been
processed, then retries them in the order they failed.
A guest is retried at most `max-retries` times. Between attempts, the
delay starts at 30 seconds and doubles with each retry, capped at 30
minutes as suggested on bugzilla [0]. If a guest still fails after the
last retry, it is counted as an error and the job finishes with errors.
If a retry succeeds, the task finishes with a warning and the
notification is sent with severity `warning`. Its subject shows how
many guests needed retries, and the guest table shows how many retries
each of them needed.
Guests whose archive was already created (e.g. failing backup-end hook)
are not retried, to avoid duplicate archives and repeated pruning.
Retries are not possible with --stdout, as a retry would corrupt the
stream. Setting `error-behavior` to a retry strategy together with
--stdout is rejected, and a retry strategy from /etc/vzdump.conf is
ignored with a warning.
The task log is kept across retries and contains all attempts.
[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
PVE/VZDump.pm | 219 +++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 215 insertions(+), 4 deletions(-)
diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index d2861f9e..d0f98a13 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -10,6 +10,7 @@ use File::Path;
use IO::File;
use IO::Select;
use IPC::Open3;
+use List::Util qw(min);
use POSIX qw(strftime);
use Time::Local;
@@ -20,6 +21,7 @@ use PVE::HA::Config;
use PVE::HA::Env::PVE2;
use PVE::JSONSchema qw(get_standard_option);
use PVE::Notify;
+use PVE::RESTEnvironment qw(log_warn);
use PVE::RPCEnvironment;
use PVE::Storage;
use PVE::VZDump::Common;
@@ -395,6 +397,11 @@ my sub read_backup_task_logs {
return $task_logs;
}
+my sub retries_text {
+ my ($count) = @_;
+ return $count == 1 ? '1 retry' : "$count retries";
+}
+
my sub build_guest_table {
my ($task_list) = @_;
@@ -436,11 +443,18 @@ my sub build_guest_table {
my $successful = $task->{state} eq 'ok';
my $size = $successful ? $task->{size} : 0;
my $filename = $successful ? $task->{target} : undef;
+
+ my $status = $task->{state};
+ my $retries = $task->{retries};
+ if ($successful && $retries) {
+ $status .= " (after " . retries_text($retries) . ")";
+ }
+
push @{ $table->{data} },
{
"vmid" => int($task->{vmid}),
"name" => $task->{hostname},
- "status" => $task->{state},
+ "status" => $status,
"time" => int($task->{backuptime}),
"size" => int($size),
"filename" => $filename,
@@ -502,8 +516,11 @@ sub send_notification {
my ($error_count, $total_size) = aggregate_task_statistics($tasklist);
my $failed = ($error_count || $err);
+ my $retried_count = grep { $_->{state} eq 'ok' && $_->{retries} } @$tasklist;
my $status_text = $failed ? 'backup failed' : 'backup successful';
+ $status_text .= " ($retried_count guest(s) needed retries)"
+ if !$failed && $retried_count;
if ($err) {
if ($err =~ /\n/) {
@@ -540,7 +557,7 @@ sub send_notification {
# Add backup-job metadata field in case this is a backup job.
$fields->{'job-id'} = $job_id if $job_id;
- my $severity = $failed ? "error" : "info";
+ my $severity = $failed ? "error" : $retried_count ? "warning" : "info";
my $email_configured = $mailto && scalar(@$mailto);
if (($mode eq 'auto' && $email_configured) || $mode eq 'legacy-sendmail') {
@@ -1048,7 +1065,9 @@ sub exec_backup_task {
-d $task->{tmpdir}
|| die "unable to create temporary directory '$task->{tmpdir}'";
- $logfd = IO::File->new(">$tmplog")
+ # keep the log of earlier attempts of this job
+ my $logmode = defined($task->{tmplog}) ? '>>' : '>';
+ $logfd = IO::File->new("${logmode}${tmplog}")
|| die "unable to create log file '$tmplog'";
$task->{dumpdir} = $opts->{dumpdir};
@@ -1216,6 +1235,9 @@ sub exec_backup_task {
debugmsg('info', "archive file size: $cs", $logfd);
}
+ # The backup exists from here on, later failures must not trigger a retry
+ $task->{committed} = 1;
+
# Mark as protected before pruning.
if (my $storeid = $opts->{storage}) {
my $volname =
@@ -1445,14 +1467,193 @@ sub exec_backup {
my $starttime = time();
my $errcount = 0;
+ my $retried_count = 0;
+
+ my $error_behavior = $opts->{'error-behavior'} // 'ignore';
+ if ($opts->{stdout} && $error_behavior ne 'ignore') {
+ log_warn(
+ "ignoring 'error-behavior' from vzdump.conf - retries are not possible with 'stdout'");
+ $error_behavior = 'ignore';
+ }
+
+ my $max_retries = $opts->{'max-retries'} // 5;
+ my $min_between_retry = 30;
+ my $max_wait = 60 * 30;
+
+ my $get_backoff = sub {
+ my ($retry) = @_;
+ return min($min_between_retry * 2**($retry - 1), $max_wait);
+ };
+
+ my $log_committed_failure = sub {
+ my ($vmid) = @_;
+ debugmsg(
+ 'err',
+ "Backup of guest $vmid was created, but a later step failed - not retrying",
+ undef,
+ 1,
+ );
+ };
+
+ my $log_retried_success = sub {
+ my ($vmid, $retries) = @_;
+ log_warn("Backup of guest $vmid succeeded after " . retries_text($retries));
+ $retried_count++;
+ };
+
eval {
$self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
$self->run_hook_script('job-start', undef, $job_start_fd);
+ my $failed_tasks = {};
+
foreach my $task (@$tasklist) {
+ my $vmid = $task->{vmid};
+ my $initial_task = {%$task};
+
+ $task->{retries} = 0;
$self->exec_backup_task($task);
- $errcount += 1 if $task->{state} ne 'ok';
+
+ next if $task->{state} eq 'ok';
+
+ if ($error_behavior ne 'ignore' && $task->{committed}) {
+ $log_committed_failure->($vmid);
+ $errcount++;
+ next;
+ }
+
+ if ($error_behavior eq 'retry-immediately') {
+ for my $retry (1 .. $max_retries) {
+ my $backoff = $get_backoff->($retry);
+
+ debugmsg(
+ 'info',
+ "Retrying backup for guest $vmid (retry $retry/$max_retries)"
+ . " after ${backoff}s delay...",
+ undef,
+ 1,
+ );
+ sleep($backoff);
+
+ %$task = (%$initial_task, retries => $retry, tmplog => $task->{tmplog});
+ $self->exec_backup_task($task);
+
+ if ($task->{state} eq 'ok') {
+ $log_retried_success->($vmid, $retry);
+ last;
+ }
+ last if $task->{committed};
+ }
+
+ } elsif ($error_behavior eq 'retry') {
+ $failed_tasks->{$vmid} = {
+ task => $task,
+ initial_task => $initial_task,
+ fail_time => time(),
+ retries => 0,
+ };
+ next;
+ }
+
+ if ($task->{state} ne 'ok') {
+ if ($error_behavior eq 'retry-immediately') {
+ if ($task->{committed}) {
+ $log_committed_failure->($vmid);
+ } else {
+ debugmsg(
+ 'err',
+ "Guest $vmid failed after " . retries_text($max_retries),
+ undef,
+ 1,
+ );
+ }
+ }
+ $errcount++;
+ }
+ }
+
+ if ($error_behavior eq 'retry') {
+ while (%$failed_tasks) {
+ my $now = time();
+ my $min_sleep = $max_wait;
+ my $ready_to_run = [];
+
+ my @sorted_vmids = sort {
+ $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time}
+ || $a <=> $b
+ } keys %$failed_tasks;
+
+ for my $vmid (@sorted_vmids) {
+ my $fail_info = $failed_tasks->{$vmid};
+
+ if ($fail_info->{retries} >= $max_retries) {
+ debugmsg(
+ 'err',
+ "Guest $vmid failed after " . retries_text($max_retries),
+ undef,
+ 1,
+ );
+ $errcount++;
+ delete $failed_tasks->{$vmid};
+ next;
+ }
+
+ my $delay = $get_backoff->($fail_info->{retries} + 1);
+ my $target_time = $fail_info->{fail_time} + $delay;
+ my $wait_time = $target_time - $now;
+
+ if ($wait_time > 0) {
+ $min_sleep = $wait_time if $wait_time < $min_sleep;
+ } else {
+ push @$ready_to_run, $vmid;
+ }
+ }
+
+ if (scalar(@$ready_to_run) == 0 && %$failed_tasks) {
+ debugmsg(
+ 'info',
+ "Waiting $min_sleep seconds before next backup retry...",
+ undef,
+ 1,
+ );
+ sleep($min_sleep);
+ next;
+ }
+
+ for my $vmid (@$ready_to_run) {
+ my $fail_info = $failed_tasks->{$vmid};
+
+ $fail_info->{retries}++;
+ debugmsg(
+ 'info',
+ "Retrying backup for guest $vmid"
+ . " (retry $fail_info->{retries}/$max_retries)",
+ undef,
+ 1,
+ );
+
+ my $task = $fail_info->{task};
+ my $tmplog = $task->{tmplog};
+ %$task = (
+ %{ $fail_info->{initial_task} },
+ retries => $fail_info->{retries},
+ tmplog => $tmplog,
+ );
+ $self->exec_backup_task($task);
+
+ if ($task->{state} eq 'ok') {
+ $log_retried_success->($vmid, $fail_info->{retries});
+ delete $failed_tasks->{$vmid};
+ } elsif ($task->{committed}) {
+ $log_committed_failure->($vmid);
+ $errcount++;
+ delete $failed_tasks->{$vmid};
+ } else {
+ $fail_info->{fail_time} = time();
+ }
+ }
+ }
}
$self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
@@ -1472,6 +1673,13 @@ sub exec_backup {
} else {
if ($errcount) {
debugmsg('info', "Backup job finished with errors", undef, 1);
+ } elsif ($retried_count) {
+ debugmsg(
+ 'info',
+ "Backup job finished with warnings - $retried_count guest(s) needed retries",
+ undef,
+ 1,
+ );
} else {
debugmsg('info', "Backup job finished successfully", undef, 1);
}
@@ -1547,6 +1755,9 @@ sub verify_vzdump_parameters {
raise_param_exc({ pool => "option conflicts with option 'vmid'" })
if $param->{pool} && $param->{vmid};
+ raise_param_exc({ 'error-behavior' => "retries are not possible with option 'stdout'" })
+ if $param->{stdout} && ($param->{'error-behavior'} // 'ignore') ne 'ignore';
+
$parse_prune_backups->($param, 'CLI parameters');
parse_fleecing($param);
parse_performance($param);
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
` (4 preceding siblings ...)
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
Expose the `error-behavior` and `max-retries` vzdump parameters in the
scheduled datacenter backup and in the manual backup window.
New backup jobs preselect `retry`, as suggested in the bug report,
while existing jobs keep their configuration when edited. The schema
default stays `ignore`, so existing jobs, CLI and API usage keep their
current behavior. The manual backup window uses the node's defaults
from /etc/vzdump.conf.
The manual backup window only offers `retry` and `ignore`, since it
backs up a single guest, where `retry` and `retry-immediately` behave
the same. The width of the scheduled backup job window is increased
from 720 to 750 pixels so the new fields fit without cutting off their
labels.
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
www/manager6/dc/Backup.js | 2 +-
www/manager6/panel/BackupAdvancedOptions.js | 52 +++++++++++++++++++
www/manager6/window/Backup.js | 56 ++++++++++++++++++++-
3 files changed, 108 insertions(+), 2 deletions(-)
diff --git a/www/manager6/dc/Backup.js b/www/manager6/dc/Backup.js
index d3db9d9c..b640fa1a 100644
--- a/www/manager6/dc/Backup.js
+++ b/www/manager6/dc/Backup.js
@@ -7,7 +7,7 @@ Ext.define('PVE.dc.BackupEdit', {
defaultFocus: undefined,
subject: gettext('Backup Job'),
- width: 720,
+ width: 750,
bodyPadding: 0,
url: '/api2/extjs/cluster/backup',
diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf..5023cdad 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -24,6 +24,11 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
me.lookup('fleecingStorage').setDisabled(!value);
},
+ errorBehaviorChange: function (field, value) {
+ let me = this;
+ me.lookup('maxRetriesField').setDisabled(value === 'ignore');
+ },
+
control: {
'proxmoxcheckbox[reference=fleecingEnabled]': {
change: 'toggleFleecing',
@@ -117,6 +122,53 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
},
items: [
+ {
+ xtype: 'pveTwoColumnContainer',
+ startColumn: {
+ xtype: 'proxmoxKVComboBox',
+ fieldLabel: gettext('Error Behavior'),
+ name: 'error-behavior',
+ comboItems: [
+ ['__default__', Proxmox.Utils.defaultText],
+ ['retry', gettext('Retry at end of job')],
+ ['retry-immediately', gettext('Retry immediately')],
+ ['ignore', gettext('Ignore')],
+ ],
+ cbind: {
+ value: (get) => (get('isCreate') ? 'retry' : '__default__'),
+ deleteEmpty: '{!isCreate}',
+ },
+ listeners: {
+ change: 'errorBehaviorChange',
+ },
+ },
+ endFlex: 2,
+ endColumn: {
+ xtype: 'displayfield',
+ value: `${gettext('How to handle guest backup failures.')} ${Ext.String.format(gettext('Schema default: {0}'), gettext('Ignore'))}`,
+ },
+ },
+ {
+ xtype: 'pveTwoColumnContainer',
+ startColumn: {
+ xtype: 'proxmoxintegerfield',
+ fieldLabel: gettext('Max Retries'),
+ name: 'max-retries',
+ minValue: 1,
+ maxValue: 10,
+ reference: 'maxRetriesField',
+ allowBlank: true,
+ emptyText: Proxmox.Utils.defaultText,
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ },
+ endFlex: 2,
+ endColumn: {
+ xtype: 'displayfield',
+ value: `${gettext('Maximum number of retries before giving up on a guest.')} ${Ext.String.format(gettext('Schema default: {0}'), 5)}`,
+ },
+ },
{
xtype: 'pveTwoColumnContainer',
startColumn: {
diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
index 0caada65..3ce1d69b 100644
--- a/www/manager6/window/Backup.js
+++ b/www/manager6/window/Backup.js
@@ -55,6 +55,34 @@ Ext.define('PVE.window.Backup', {
},
});
+ let maxRetriesField = Ext.create({
+ xtype: 'proxmoxintegerfield',
+ fieldLabel: gettext('Max Retries'),
+ name: 'max-retries',
+ minValue: 1,
+ maxValue: 10,
+ allowBlank: true,
+ emptyText: '5',
+ disabled: true,
+ });
+
+ let errorBehaviorSelector = Ext.create({
+ xtype: 'proxmoxKVComboBox',
+ fieldLabel: gettext('Error Behavior'),
+ name: 'error-behavior',
+ value: 'ignore',
+ deleteEmpty: false,
+ comboItems: [
+ ['retry', gettext('Retry')],
+ ['ignore', gettext('Ignore')],
+ ],
+ listeners: {
+ change: function (field, value) {
+ maxRetriesField.setDisabled(value === 'ignore');
+ },
+ },
+ });
+
let pbsChangeDetectionModeSelector = Ext.create({
xtype: 'proxmoxKVComboBox',
flex: 1,
@@ -196,6 +224,17 @@ Ext.define('PVE.window.Backup', {
);
}
+ if (!initialDefaults && data['error-behavior'] !== undefined) {
+ // both retry strategies behave the same for a single guest
+ let behavior = data['error-behavior'];
+ errorBehaviorSelector.setValue(
+ behavior === 'retry-immediately' ? 'retry' : behavior,
+ );
+ }
+ if (!initialDefaults && data['max-retries'] !== undefined) {
+ maxRetriesField.setValue(data['max-retries']);
+ }
+
initialDefaults = true;
// always update storage dependent properties
@@ -250,7 +289,14 @@ Ext.define('PVE.window.Backup', {
bodyPadding: 10,
border: false,
column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
- column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
+ column2: [
+ compressionSelector,
+ notificationModeSelector,
+ mailtoField,
+ removeCheckbox,
+ errorBehaviorSelector,
+ maxRetriesField,
+ ],
columnB: [
{
xtype: 'textareafield',
@@ -342,6 +388,14 @@ Ext.define('PVE.window.Backup', {
);
}
+ if (values['error-behavior']) {
+ params['error-behavior'] = values['error-behavior'];
+ }
+
+ if (values['max-retries'] !== undefined && values['max-retries'] !== '') {
+ params['max-retries'] = values['max-retries'];
+ }
+
Proxmox.Utils.API2Request({
url: '/nodes/' + me.nodename + '/vzdump',
params: params,
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH pve-docs v4 7/7] vzdump: document backup retry options
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
` (5 preceding siblings ...)
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
@ 2026-11-20 16:05 ` Jakob Klocker
6 siblings, 0 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
To: pve-devel
Describe the new `error-behavior` and `max-retries` options, the
exponential backoff between attempts, and which failures are retried.
Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
vzdump.adoc | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..2b5a0ed 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,72 @@ the `performance` setting, `max-workers` (affects VM backups only) and
`pbs-entries-max` (affects container backups only). See the
xref:vzdump_configuration[configuration options] for details.
+[[vzdump_retry]]
+Backup Retries
+--------------
+
+The backup of a single guest can fail because of a temporary problem, for
+example when the guest is locked by another operation, or when the connection
+to the backup server is interrupted during the backup. To avoid a failed backup
+in such cases, the `error-behavior` option allows a job to retry failed guest
+backups automatically:
+
+`ignore` ::
+Does not retry failed guests. This is the default.
+
+`retry` ::
+Defers failed guests until all other guests have been processed and then
+retries them in the order in which they failed.
+
+`retry-immediately` ::
+Retries a failed guest right away, before continuing with the next guest.
+
+The `max-retries` option (1 to 10, default 5) limits how often a guest is
+retried. Between attempts, the job uses exponential backoff: the delay starts
+at 30 seconds, doubles with every retry and is capped at 30 minutes. With
+`retry`, the delay is counted from the last failed attempt of that guest, so
+other guests can be retried in the meantime.
+
+If a retry succeeds, the guest does not count as an error, but the task
+finishes with a warning and the notification is sent with severity `warning`.
+Its subject shows how many guests needed retries, and the guest table shows
+how many retries each of them needed. This way, recurring problems stay
+visible even if all backups eventually succeed. If the last retry fails as
+well, the job finishes with errors. The task log of the guest contains all
+attempts.
+
+For example, to back up three guests and retry each failed guest at most three
+times at the end of the job, run:
+
+ # vzdump 101 102 103 --error-behavior retry --max-retries 3
+
+Both options can also be set as node-wide defaults in `/etc/vzdump.conf`, see
+xref:vzdump_configuration[configuration].
+
+In the web interface, the retry strategy for backup jobs can be set in the
+'Advanced' tab of the backup job window. New backup jobs preselect `retry`,
+existing jobs keep their configuration. For a manual backup, the retry strategy
+can be chosen directly in the backup window ('Backup now'), which preselects the
+node's default from `/etc/vzdump.conf`. It only offers `retry` and `ignore`,
+as both retry strategies behave the same for a single guest.
+
+Only failures that happen before the backup archive of a guest has been created
+are retried. If the archive was created, but a later step failed, for example a
+failing `backup-end` hook script, the guest is not retried. This avoids
+creating multiple backups of the same guest and pruning older backups
+repeatedly. Errors that prevent the whole job from starting, for example a
+backup storage that cannot be activated, are not retried either.
+
+Retries are not possible when writing the backup to standard output with
+`--stdout`, as a retry would append a second archive to the data that was
+already written. Setting `error-behavior` to a retry strategy together with
+`--stdout` is rejected, and a retry strategy from `/etc/vzdump.conf` is
+ignored with a warning.
+
+NOTE: Retries can make a backup job take considerably longer. With the default
+of 5 retries, the delays alone add up to more than 15 minutes for a guest that
+keeps failing, and about 2.5 hours with the maximum of 10 retries.
+
[[vzdump_retention]]
Backup Retention
----------------
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-20 16:07 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox