public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies
@ 2026-11-16 10:13 Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed up to a
  configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.

Testing info:
When testing I'd suggest to fail the backup by locking and
unlocking the guest with following commands:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Changes since v2 (thanks @Jonas)
- don't retry guests whose backup was already created (e.g. failing
  backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
  one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
  for retried guests
- show the number of retries in the notification subject and guest
  table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
  description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
  and UI defaults

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 219 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 +++++
 www/manager6/window/Backup.js               |  56 ++++-
 4 files changed, 323 insertions(+), 6 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)


Summary over all repositories:
  6 files changed, 406 insertions(+), 6 deletions(-)

-- 
Generated by murpp 0.12.0




^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Add two new backup job options to control what happens when a guest
backup fails:

- 'error-behavior' selects the behavior. 'ignore' keeps the current
  behavior and is the default. 'retry' queues the failed guest and
  retries it after the remaining guests have been backed up.
  'retry-immediately' retries the failed guest right away, before
  continuing with the next one.
- 'max-retries' limits how many times a failed guest backup is retried
  (1-10, default 5).

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 2cb48a3..2babd11 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -384,6 +384,24 @@ my $confdesc = {
         optional => 1,
         enum => ['legacy', 'data', 'metadata'],
     },
+    'error-behavior' => {
+        type => 'string',
+        description => "How to handle guest backup failures. 'ignore' does not retry,"
+            . " 'retry' retries failed guests after all other guests have been processed,"
+            . " 'retry-immediately' retries a failed guest before continuing with the next one.",
+        enum => ['ignore', 'retry', 'retry-immediately'],
+        optional => 1,
+        default => 'ignore',
+    },
+    'max-retries' => {
+        type => 'integer',
+        description => "Maximum number of times a failed guest backup is retried."
+            . " Only used if 'error-behavior' is not 'ignore'.",
+        minimum => 1,
+        maximum => 10,
+        optional => 1,
+        default => 5,
+    },
 };
 
 sub get_confdesc {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Add the `error-behavior` and `max-retries` options to automatically
retry failed guest backups, which helps with transient failures.

Supported retry strategies:
- `ignore`: Preserves current behavior (no retries, default).
- `retry-immediately`: Retries the guest inline after failure.
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed.

A guest is retried at most `max-retries` times. Between attempts, the
delay starts at 30 seconds and doubles with each retry, capped at 30
minutes as suggested on bugzilla [0]. If a guest still fails after the
last retry, it is counted as an error and the job finishes with errors.
If a retry succeeds, the task finishes with a warning and the
notification is sent with severity `warning`, showing the number of
retries in the subject and guest table.

Guests whose archive was already created (e.g. failing backup-end hook)
are not retried, to avoid duplicate archives and repeated pruning.

Retries are not possible with --stdout, as a retry would corrupt the
stream. Setting `error-behavior` to a retry strategy together with
--stdout is rejected, and a retry strategy from /etc/vzdump.conf is
ignored with a warning.

The task log is kept across retries and contains all attempts.

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 PVE/VZDump.pm | 219 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 215 insertions(+), 4 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index d2861f9e..d0f98a13 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -10,6 +10,7 @@ use File::Path;
 use IO::File;
 use IO::Select;
 use IPC::Open3;
+use List::Util qw(min);
 use POSIX qw(strftime);
 use Time::Local;
 
@@ -20,6 +21,7 @@ use PVE::HA::Config;
 use PVE::HA::Env::PVE2;
 use PVE::JSONSchema qw(get_standard_option);
 use PVE::Notify;
+use PVE::RESTEnvironment qw(log_warn);
 use PVE::RPCEnvironment;
 use PVE::Storage;
 use PVE::VZDump::Common;
@@ -395,6 +397,11 @@ my sub read_backup_task_logs {
     return $task_logs;
 }
 
+my sub retries_text {
+    my ($count) = @_;
+    return $count == 1 ? '1 retry' : "$count retries";
+}
+
 my sub build_guest_table {
     my ($task_list) = @_;
 
@@ -436,11 +443,18 @@ my sub build_guest_table {
         my $successful = $task->{state} eq 'ok';
         my $size = $successful ? $task->{size} : 0;
         my $filename = $successful ? $task->{target} : undef;
+
+        my $status = $task->{state};
+        my $retries = $task->{retries};
+        if ($successful && $retries) {
+            $status .= " (after " . retries_text($retries) . ")";
+        }
+
         push @{ $table->{data} },
             {
                 "vmid" => int($task->{vmid}),
                 "name" => $task->{hostname},
-                "status" => $task->{state},
+                "status" => $status,
                 "time" => int($task->{backuptime}),
                 "size" => int($size),
                 "filename" => $filename,
@@ -502,8 +516,11 @@ sub send_notification {
     my ($error_count, $total_size) = aggregate_task_statistics($tasklist);
 
     my $failed = ($error_count || $err);
+    my $retried_count = grep { $_->{state} eq 'ok' && $_->{retries} } @$tasklist;
 
     my $status_text = $failed ? 'backup failed' : 'backup successful';
+    $status_text .= " ($retried_count guest(s) needed retries)"
+        if !$failed && $retried_count;
 
     if ($err) {
         if ($err =~ /\n/) {
@@ -540,7 +557,7 @@ sub send_notification {
     # Add backup-job metadata field in case this is a backup job.
     $fields->{'job-id'} = $job_id if $job_id;
 
-    my $severity = $failed ? "error" : "info";
+    my $severity = $failed ? "error" : $retried_count ? "warning" : "info";
     my $email_configured = $mailto && scalar(@$mailto);
 
     if (($mode eq 'auto' && $email_configured) || $mode eq 'legacy-sendmail') {
@@ -1048,7 +1065,9 @@ sub exec_backup_task {
         -d $task->{tmpdir}
             || die "unable to create temporary directory '$task->{tmpdir}'";
 
-        $logfd = IO::File->new(">$tmplog")
+        # keep the log of earlier attempts of this job
+        my $logmode = defined($task->{tmplog}) ? '>>' : '>';
+        $logfd = IO::File->new("${logmode}${tmplog}")
             || die "unable to create log file '$tmplog'";
 
         $task->{dumpdir} = $opts->{dumpdir};
@@ -1216,6 +1235,9 @@ sub exec_backup_task {
             debugmsg('info', "archive file size: $cs", $logfd);
         }
 
+        # The backup exists from here on, later failures must not trigger a retry
+        $task->{committed} = 1;
+
         # Mark as protected before pruning.
         if (my $storeid = $opts->{storage}) {
             my $volname =
@@ -1445,14 +1467,193 @@ sub exec_backup {
 
     my $starttime = time();
     my $errcount = 0;
+    my $retried_count = 0;
+
+    my $error_behavior = $opts->{'error-behavior'} // 'ignore';
+    if ($opts->{stdout} && $error_behavior ne 'ignore') {
+        log_warn(
+            "ignoring 'error-behavior' from vzdump.conf - retries are not possible with 'stdout'");
+        $error_behavior = 'ignore';
+    }
+
+    my $max_retries = $opts->{'max-retries'} // 5;
+    my $min_between_retry = 30;
+    my $max_wait = 60 * 30;
+
+    my $get_backoff = sub {
+        my ($retry) = @_;
+        return min($min_between_retry * 2**($retry - 1), $max_wait);
+    };
+
+    my $log_committed_failure = sub {
+        my ($vmid) = @_;
+        debugmsg(
+            'err',
+            "Backup of guest $vmid was created, but a later step failed - not retrying",
+            undef,
+            1,
+        );
+    };
+
+    my $log_retried_success = sub {
+        my ($vmid, $retries) = @_;
+        log_warn("Backup of guest $vmid succeeded after " . retries_text($retries));
+        $retried_count++;
+    };
+
     eval {
 
         $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
         $self->run_hook_script('job-start', undef, $job_start_fd);
 
+        my $failed_tasks = {};
+
         foreach my $task (@$tasklist) {
+            my $vmid = $task->{vmid};
+            my $initial_task = {%$task};
+
+            $task->{retries} = 0;
             $self->exec_backup_task($task);
-            $errcount += 1 if $task->{state} ne 'ok';
+
+            next if $task->{state} eq 'ok';
+
+            if ($error_behavior ne 'ignore' && $task->{committed}) {
+                $log_committed_failure->($vmid);
+                $errcount++;
+                next;
+            }
+
+            if ($error_behavior eq 'retry-immediately') {
+                for my $retry (1 .. $max_retries) {
+                    my $backoff = $get_backoff->($retry);
+
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid (retry $retry/$max_retries)"
+                            . " after ${backoff}s delay...",
+                        undef,
+                        1,
+                    );
+                    sleep($backoff);
+
+                    %$task = (%$initial_task, retries => $retry, tmplog => $task->{tmplog});
+                    $self->exec_backup_task($task);
+
+                    if ($task->{state} eq 'ok') {
+                        $log_retried_success->($vmid, $retry);
+                        last;
+                    }
+                    last if $task->{committed};
+                }
+
+            } elsif ($error_behavior eq 'retry') {
+                $failed_tasks->{$vmid} = {
+                    task => $task,
+                    initial_task => $initial_task,
+                    fail_time => time(),
+                    retries => 0,
+                };
+                next;
+            }
+
+            if ($task->{state} ne 'ok') {
+                if ($error_behavior eq 'retry-immediately') {
+                    if ($task->{committed}) {
+                        $log_committed_failure->($vmid);
+                    } else {
+                        debugmsg(
+                            'err',
+                            "Guest $vmid failed after " . retries_text($max_retries),
+                            undef,
+                            1,
+                        );
+                    }
+                }
+                $errcount++;
+            }
+        }
+
+        if ($error_behavior eq 'retry') {
+            while (%$failed_tasks) {
+                my $now = time();
+                my $min_sleep = $max_wait;
+                my $ready_to_run = [];
+
+                my @sorted_vmids = sort {
+                    $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time}
+                        || $a <=> $b
+                } keys %$failed_tasks;
+
+                for my $vmid (@sorted_vmids) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    if ($fail_info->{retries} >= $max_retries) {
+                        debugmsg(
+                            'err',
+                            "Guest $vmid failed after " . retries_text($max_retries),
+                            undef,
+                            1,
+                        );
+                        $errcount++;
+                        delete $failed_tasks->{$vmid};
+                        next;
+                    }
+
+                    my $delay = $get_backoff->($fail_info->{retries} + 1);
+                    my $target_time = $fail_info->{fail_time} + $delay;
+                    my $wait_time = $target_time - $now;
+
+                    if ($wait_time > 0) {
+                        $min_sleep = $wait_time if $wait_time < $min_sleep;
+                    } else {
+                        push @$ready_to_run, $vmid;
+                    }
+                }
+
+                if (scalar(@$ready_to_run) == 0 && %$failed_tasks) {
+                    debugmsg(
+                        'info',
+                        "Waiting $min_sleep seconds before next backup retry...",
+                        undef,
+                        1,
+                    );
+                    sleep($min_sleep);
+                    next;
+                }
+
+                for my $vmid (@$ready_to_run) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    $fail_info->{retries}++;
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid"
+                            . " (retry $fail_info->{retries}/$max_retries)",
+                        undef,
+                        1,
+                    );
+
+                    my $task = $fail_info->{task};
+                    my $tmplog = $task->{tmplog};
+                    %$task = (
+                        %{ $fail_info->{initial_task} },
+                        retries => $fail_info->{retries},
+                        tmplog => $tmplog,
+                    );
+                    $self->exec_backup_task($task);
+
+                    if ($task->{state} eq 'ok') {
+                        $log_retried_success->($vmid, $fail_info->{retries});
+                        delete $failed_tasks->{$vmid};
+                    } elsif ($task->{committed}) {
+                        $log_committed_failure->($vmid);
+                        $errcount++;
+                        delete $failed_tasks->{$vmid};
+                    } else {
+                        $fail_info->{fail_time} = time();
+                    }
+                }
+            }
         }
 
         $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
@@ -1472,6 +1673,13 @@ sub exec_backup {
     } else {
         if ($errcount) {
             debugmsg('info', "Backup job finished with errors", undef, 1);
+        } elsif ($retried_count) {
+            debugmsg(
+                'info',
+                "Backup job finished with warnings - $retried_count guest(s) needed retries",
+                undef,
+                1,
+            );
         } else {
             debugmsg('info', "Backup job finished successfully", undef, 1);
         }
@@ -1547,6 +1755,9 @@ sub verify_vzdump_parameters {
     raise_param_exc({ pool => "option conflicts with option 'vmid'" })
         if $param->{pool} && $param->{vmid};
 
+    raise_param_exc({ 'error-behavior' => "retries are not possible with option 'stdout'" })
+        if $param->{stdout} && ($param->{'error-behavior'} // 'ignore') ne 'ignore';
+
     $parse_prune_backups->($param, 'CLI parameters');
     parse_fleecing($param);
     parse_performance($param);
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Expose the `error-behavior` and `max-retries` vzdump parameters in the
scheduled datacenter backup and in the manual backup window.

New backup jobs preselect `retry`, as suggested in the bug report,
while existing jobs keep their configuration when edited. The schema
default stays `ignore`, so existing jobs, CLI and API usage keep their
current behavior. The manual backup window uses the node's defaults
from /etc/vzdump.conf.

The manual backup window only offers `retry` and `ignore`, since it
backs up a single guest, where `retry` and `retry-immediately` behave
the same. The width of the scheduled backup job window is increased
from 720 to 750 pixels so the new fields fit without cutting off their
labels.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 www/manager6/dc/Backup.js                   |  2 +-
 www/manager6/panel/BackupAdvancedOptions.js | 52 +++++++++++++++++++
 www/manager6/window/Backup.js               | 56 ++++++++++++++++++++-
 3 files changed, 108 insertions(+), 2 deletions(-)

diff --git a/www/manager6/dc/Backup.js b/www/manager6/dc/Backup.js
index d3db9d9c..b640fa1a 100644
--- a/www/manager6/dc/Backup.js
+++ b/www/manager6/dc/Backup.js
@@ -7,7 +7,7 @@ Ext.define('PVE.dc.BackupEdit', {
     defaultFocus: undefined,
 
     subject: gettext('Backup Job'),
-    width: 720,
+    width: 750,
     bodyPadding: 0,
 
     url: '/api2/extjs/cluster/backup',
diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf..5023cdad 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -24,6 +24,11 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
             me.lookup('fleecingStorage').setDisabled(!value);
         },
 
+        errorBehaviorChange: function (field, value) {
+            let me = this;
+            me.lookup('maxRetriesField').setDisabled(value === 'ignore');
+        },
+
         control: {
             'proxmoxcheckbox[reference=fleecingEnabled]': {
                 change: 'toggleFleecing',
@@ -117,6 +122,53 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
     },
 
     items: [
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxKVComboBox',
+                fieldLabel: gettext('Error Behavior'),
+                name: 'error-behavior',
+                comboItems: [
+                    ['__default__', Proxmox.Utils.defaultText],
+                    ['retry', gettext('Retry at end of job')],
+                    ['retry-immediately', gettext('Retry immediately')],
+                    ['ignore', gettext('Ignore')],
+                ],
+                cbind: {
+                    value: (get) => (get('isCreate') ? 'retry' : '__default__'),
+                    deleteEmpty: '{!isCreate}',
+                },
+                listeners: {
+                    change: 'errorBehaviorChange',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: `${gettext('How to handle guest backup failures.')} ${Ext.String.format(gettext('Schema default: {0}'), gettext('Ignore'))}`,
+            },
+        },
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxintegerfield',
+                fieldLabel: gettext('Max Retries'),
+                name: 'max-retries',
+                minValue: 1,
+                maxValue: 10,
+                reference: 'maxRetriesField',
+                allowBlank: true,
+                emptyText: Proxmox.Utils.defaultText,
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: `${gettext('Maximum number of retries before giving up on a guest.')} ${Ext.String.format(gettext('Schema default: {0}'), 5)}`,
+            },
+        },
         {
             xtype: 'pveTwoColumnContainer',
             startColumn: {
diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
index 0caada65..3ce1d69b 100644
--- a/www/manager6/window/Backup.js
+++ b/www/manager6/window/Backup.js
@@ -55,6 +55,34 @@ Ext.define('PVE.window.Backup', {
             },
         });
 
+        let maxRetriesField = Ext.create({
+            xtype: 'proxmoxintegerfield',
+            fieldLabel: gettext('Max Retries'),
+            name: 'max-retries',
+            minValue: 1,
+            maxValue: 10,
+            allowBlank: true,
+            emptyText: '5',
+            disabled: true,
+        });
+
+        let errorBehaviorSelector = Ext.create({
+            xtype: 'proxmoxKVComboBox',
+            fieldLabel: gettext('Error Behavior'),
+            name: 'error-behavior',
+            value: 'ignore',
+            deleteEmpty: false,
+            comboItems: [
+                ['retry', gettext('Retry')],
+                ['ignore', gettext('Ignore')],
+            ],
+            listeners: {
+                change: function (field, value) {
+                    maxRetriesField.setDisabled(value === 'ignore');
+                },
+            },
+        });
+
         let pbsChangeDetectionModeSelector = Ext.create({
             xtype: 'proxmoxKVComboBox',
             flex: 1,
@@ -196,6 +224,17 @@ Ext.define('PVE.window.Backup', {
                                 );
                             }
 
+                            if (!initialDefaults && data['error-behavior'] !== undefined) {
+                                // both retry strategies behave the same for a single guest
+                                let behavior = data['error-behavior'];
+                                errorBehaviorSelector.setValue(
+                                    behavior === 'retry-immediately' ? 'retry' : behavior,
+                                );
+                            }
+                            if (!initialDefaults && data['max-retries'] !== undefined) {
+                                maxRetriesField.setValue(data['max-retries']);
+                            }
+
                             initialDefaults = true;
 
                             // always update storage dependent properties
@@ -250,7 +289,14 @@ Ext.define('PVE.window.Backup', {
             bodyPadding: 10,
             border: false,
             column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
-            column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
+            column2: [
+                compressionSelector,
+                notificationModeSelector,
+                mailtoField,
+                removeCheckbox,
+                errorBehaviorSelector,
+                maxRetriesField,
+            ],
             columnB: [
                 {
                     xtype: 'textareafield',
@@ -342,6 +388,14 @@ Ext.define('PVE.window.Backup', {
                     );
                 }
 
+                if (values['error-behavior']) {
+                    params['error-behavior'] = values['error-behavior'];
+                }
+
+                if (values['max-retries'] !== undefined && values['max-retries'] !== '') {
+                    params['max-retries'] = values['max-retries'];
+                }
+
                 Proxmox.Utils.API2Request({
                     url: '/nodes/' + me.nodename + '/vzdump',
                     params: params,
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-docs v3 4/4] vzdump: document backup retry options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
                   ` (2 preceding siblings ...)
  2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Describe the new `error-behavior` and `max-retries` options, the
exponential backoff between attempts, and which failures are retried.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..856d3d3 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,71 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_retry]]
+Backup Retries
+--------------
+
+The backup of a single guest can fail because of a temporary problem, for
+example when the guest is locked by another operation, or when the connection
+to the backup server is interrupted during the backup. To avoid a failed backup
+in such cases, the `error-behavior` option allows a job to retry failed guest
+backups automatically:
+
+`ignore` ::
+Does not retry failed guests. This is the default.
+
+`retry` ::
+Defers failed guests until all other guests have been processed and then
+retries them in the order in which they failed.
+
+`retry-immediately` ::
+Retries a failed guest right away, before continuing with the next guest.
+
+The `max-retries` option (1 to 10, default 5) limits how often a guest is
+retried. Between attempts, the job uses exponential backoff: the delay starts
+at 30 seconds, doubles with every retry and is capped at 30 minutes. With
+`retry`, the delay is counted from the last failed attempt of that guest, so
+other guests can be retried in the meantime.
+
+If a retry succeeds, the guest does not count as an error, but the task
+finishes with a warning and the notification is sent with severity `warning`
+and shows how many retries were needed. This way, recurring problems stay
+visible even if all backups eventually succeed. If the last retry fails as
+well, the job finishes with errors. The task log of the guest contains all
+attempts.
+
+For example, to back up three guests and retry each failed guest at most three
+times at the end of the job, run:
+
+ # vzdump 101 102 103 --error-behavior retry --max-retries 3
+
+Both options can also be set as node-wide defaults in `/etc/vzdump.conf`, see
+xref:vzdump_configuration[configuration].
+
+In the web interface, the retry strategy for backup jobs can be set in the
+'Advanced' tab of the backup job window. New backup jobs preselect `retry`,
+existing jobs keep their configuration. For a manual backup, the retry strategy
+can be chosen directly in the backup window ('Backup now'), which preselects the
+node's default from `/etc/vzdump.conf`. It only offers `retry` and `ignore`,
+as both retry strategies behave the same for a single guest.
+
+Only failures that happen before the backup archive of a guest has been created
+are retried. If the archive was created, but a later step failed, for example a
+failing `backup-end` hook script, the guest is not retried. This avoids
+creating multiple backups of the same guest and pruning older backups
+repeatedly. Errors that prevent the whole job from starting, for example a
+backup storage that cannot be activated, are not retried either.
+
+Retries are not possible when writing the backup to standard output with
+`--stdout`, as a retry would append a second archive to the data that was
+already written. Setting `error-behavior` to a retry strategy together with
+`--stdout` is rejected, and a retry strategy from `/etc/vzdump.conf` is
+ignored with a warning.
+
+NOTE: Retries can make a backup job take considerably longer. With the default
+of 5 retries, the delays alone add up to more than 15 minutes for a guest that
+keeps failing, and about 2.5 hours with the maximum of 10 retries.
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-16 10:15 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal