all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies
@ 2026-11-16 10:13 Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed up to a
  configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.

Testing info:
When testing I'd suggest to fail the backup by locking and
unlocking the guest with following commands:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Changes since v2 (thanks @Jonas)
- don't retry guests whose backup was already created (e.g. failing
  backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
  one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
  for retried guests
- show the number of retries in the notification subject and guest
  table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
  description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
  and UI defaults

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 219 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 +++++
 www/manager6/window/Backup.js               |  56 ++++-
 4 files changed, 323 insertions(+), 6 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)


Summary over all repositories:
  6 files changed, 406 insertions(+), 6 deletions(-)

-- 
Generated by murpp 0.12.0




^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Add two new backup job options to control what happens when a guest
backup fails:

- 'error-behavior' selects the behavior. 'ignore' keeps the current
  behavior and is the default. 'retry' queues the failed guest and
  retries it after the remaining guests have been backed up.
  'retry-immediately' retries the failed guest right away, before
  continuing with the next one.
- 'max-retries' limits how many times a failed guest backup is retried
  (1-10, default 5).

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 2cb48a3..2babd11 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -384,6 +384,24 @@ my $confdesc = {
         optional => 1,
         enum => ['legacy', 'data', 'metadata'],
     },
+    'error-behavior' => {
+        type => 'string',
+        description => "How to handle guest backup failures. 'ignore' does not retry,"
+            . " 'retry' retries failed guests after all other guests have been processed,"
+            . " 'retry-immediately' retries a failed guest before continuing with the next one.",
+        enum => ['ignore', 'retry', 'retry-immediately'],
+        optional => 1,
+        default => 'ignore',
+    },
+    'max-retries' => {
+        type => 'integer',
+        description => "Maximum number of times a failed guest backup is retried."
+            . " Only used if 'error-behavior' is not 'ignore'.",
+        minimum => 1,
+        maximum => 10,
+        optional => 1,
+        default => 5,
+    },
 };
 
 sub get_confdesc {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Add the `error-behavior` and `max-retries` options to automatically
retry failed guest backups, which helps with transient failures.

Supported retry strategies:
- `ignore`: Preserves current behavior (no retries, default).
- `retry-immediately`: Retries the guest inline after failure.
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed.

A guest is retried at most `max-retries` times. Between attempts, the
delay starts at 30 seconds and doubles with each retry, capped at 30
minutes as suggested on bugzilla [0]. If a guest still fails after the
last retry, it is counted as an error and the job finishes with errors.
If a retry succeeds, the task finishes with a warning and the
notification is sent with severity `warning`, showing the number of
retries in the subject and guest table.

Guests whose archive was already created (e.g. failing backup-end hook)
are not retried, to avoid duplicate archives and repeated pruning.

Retries are not possible with --stdout, as a retry would corrupt the
stream. Setting `error-behavior` to a retry strategy together with
--stdout is rejected, and a retry strategy from /etc/vzdump.conf is
ignored with a warning.

The task log is kept across retries and contains all attempts.

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 PVE/VZDump.pm | 219 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 215 insertions(+), 4 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index d2861f9e..d0f98a13 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -10,6 +10,7 @@ use File::Path;
 use IO::File;
 use IO::Select;
 use IPC::Open3;
+use List::Util qw(min);
 use POSIX qw(strftime);
 use Time::Local;
 
@@ -20,6 +21,7 @@ use PVE::HA::Config;
 use PVE::HA::Env::PVE2;
 use PVE::JSONSchema qw(get_standard_option);
 use PVE::Notify;
+use PVE::RESTEnvironment qw(log_warn);
 use PVE::RPCEnvironment;
 use PVE::Storage;
 use PVE::VZDump::Common;
@@ -395,6 +397,11 @@ my sub read_backup_task_logs {
     return $task_logs;
 }
 
+my sub retries_text {
+    my ($count) = @_;
+    return $count == 1 ? '1 retry' : "$count retries";
+}
+
 my sub build_guest_table {
     my ($task_list) = @_;
 
@@ -436,11 +443,18 @@ my sub build_guest_table {
         my $successful = $task->{state} eq 'ok';
         my $size = $successful ? $task->{size} : 0;
         my $filename = $successful ? $task->{target} : undef;
+
+        my $status = $task->{state};
+        my $retries = $task->{retries};
+        if ($successful && $retries) {
+            $status .= " (after " . retries_text($retries) . ")";
+        }
+
         push @{ $table->{data} },
             {
                 "vmid" => int($task->{vmid}),
                 "name" => $task->{hostname},
-                "status" => $task->{state},
+                "status" => $status,
                 "time" => int($task->{backuptime}),
                 "size" => int($size),
                 "filename" => $filename,
@@ -502,8 +516,11 @@ sub send_notification {
     my ($error_count, $total_size) = aggregate_task_statistics($tasklist);
 
     my $failed = ($error_count || $err);
+    my $retried_count = grep { $_->{state} eq 'ok' && $_->{retries} } @$tasklist;
 
     my $status_text = $failed ? 'backup failed' : 'backup successful';
+    $status_text .= " ($retried_count guest(s) needed retries)"
+        if !$failed && $retried_count;
 
     if ($err) {
         if ($err =~ /\n/) {
@@ -540,7 +557,7 @@ sub send_notification {
     # Add backup-job metadata field in case this is a backup job.
     $fields->{'job-id'} = $job_id if $job_id;
 
-    my $severity = $failed ? "error" : "info";
+    my $severity = $failed ? "error" : $retried_count ? "warning" : "info";
     my $email_configured = $mailto && scalar(@$mailto);
 
     if (($mode eq 'auto' && $email_configured) || $mode eq 'legacy-sendmail') {
@@ -1048,7 +1065,9 @@ sub exec_backup_task {
         -d $task->{tmpdir}
             || die "unable to create temporary directory '$task->{tmpdir}'";
 
-        $logfd = IO::File->new(">$tmplog")
+        # keep the log of earlier attempts of this job
+        my $logmode = defined($task->{tmplog}) ? '>>' : '>';
+        $logfd = IO::File->new("${logmode}${tmplog}")
             || die "unable to create log file '$tmplog'";
 
         $task->{dumpdir} = $opts->{dumpdir};
@@ -1216,6 +1235,9 @@ sub exec_backup_task {
             debugmsg('info', "archive file size: $cs", $logfd);
         }
 
+        # The backup exists from here on, later failures must not trigger a retry
+        $task->{committed} = 1;
+
         # Mark as protected before pruning.
         if (my $storeid = $opts->{storage}) {
             my $volname =
@@ -1445,14 +1467,193 @@ sub exec_backup {
 
     my $starttime = time();
     my $errcount = 0;
+    my $retried_count = 0;
+
+    my $error_behavior = $opts->{'error-behavior'} // 'ignore';
+    if ($opts->{stdout} && $error_behavior ne 'ignore') {
+        log_warn(
+            "ignoring 'error-behavior' from vzdump.conf - retries are not possible with 'stdout'");
+        $error_behavior = 'ignore';
+    }
+
+    my $max_retries = $opts->{'max-retries'} // 5;
+    my $min_between_retry = 30;
+    my $max_wait = 60 * 30;
+
+    my $get_backoff = sub {
+        my ($retry) = @_;
+        return min($min_between_retry * 2**($retry - 1), $max_wait);
+    };
+
+    my $log_committed_failure = sub {
+        my ($vmid) = @_;
+        debugmsg(
+            'err',
+            "Backup of guest $vmid was created, but a later step failed - not retrying",
+            undef,
+            1,
+        );
+    };
+
+    my $log_retried_success = sub {
+        my ($vmid, $retries) = @_;
+        log_warn("Backup of guest $vmid succeeded after " . retries_text($retries));
+        $retried_count++;
+    };
+
     eval {
 
         $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
         $self->run_hook_script('job-start', undef, $job_start_fd);
 
+        my $failed_tasks = {};
+
         foreach my $task (@$tasklist) {
+            my $vmid = $task->{vmid};
+            my $initial_task = {%$task};
+
+            $task->{retries} = 0;
             $self->exec_backup_task($task);
-            $errcount += 1 if $task->{state} ne 'ok';
+
+            next if $task->{state} eq 'ok';
+
+            if ($error_behavior ne 'ignore' && $task->{committed}) {
+                $log_committed_failure->($vmid);
+                $errcount++;
+                next;
+            }
+
+            if ($error_behavior eq 'retry-immediately') {
+                for my $retry (1 .. $max_retries) {
+                    my $backoff = $get_backoff->($retry);
+
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid (retry $retry/$max_retries)"
+                            . " after ${backoff}s delay...",
+                        undef,
+                        1,
+                    );
+                    sleep($backoff);
+
+                    %$task = (%$initial_task, retries => $retry, tmplog => $task->{tmplog});
+                    $self->exec_backup_task($task);
+
+                    if ($task->{state} eq 'ok') {
+                        $log_retried_success->($vmid, $retry);
+                        last;
+                    }
+                    last if $task->{committed};
+                }
+
+            } elsif ($error_behavior eq 'retry') {
+                $failed_tasks->{$vmid} = {
+                    task => $task,
+                    initial_task => $initial_task,
+                    fail_time => time(),
+                    retries => 0,
+                };
+                next;
+            }
+
+            if ($task->{state} ne 'ok') {
+                if ($error_behavior eq 'retry-immediately') {
+                    if ($task->{committed}) {
+                        $log_committed_failure->($vmid);
+                    } else {
+                        debugmsg(
+                            'err',
+                            "Guest $vmid failed after " . retries_text($max_retries),
+                            undef,
+                            1,
+                        );
+                    }
+                }
+                $errcount++;
+            }
+        }
+
+        if ($error_behavior eq 'retry') {
+            while (%$failed_tasks) {
+                my $now = time();
+                my $min_sleep = $max_wait;
+                my $ready_to_run = [];
+
+                my @sorted_vmids = sort {
+                    $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time}
+                        || $a <=> $b
+                } keys %$failed_tasks;
+
+                for my $vmid (@sorted_vmids) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    if ($fail_info->{retries} >= $max_retries) {
+                        debugmsg(
+                            'err',
+                            "Guest $vmid failed after " . retries_text($max_retries),
+                            undef,
+                            1,
+                        );
+                        $errcount++;
+                        delete $failed_tasks->{$vmid};
+                        next;
+                    }
+
+                    my $delay = $get_backoff->($fail_info->{retries} + 1);
+                    my $target_time = $fail_info->{fail_time} + $delay;
+                    my $wait_time = $target_time - $now;
+
+                    if ($wait_time > 0) {
+                        $min_sleep = $wait_time if $wait_time < $min_sleep;
+                    } else {
+                        push @$ready_to_run, $vmid;
+                    }
+                }
+
+                if (scalar(@$ready_to_run) == 0 && %$failed_tasks) {
+                    debugmsg(
+                        'info',
+                        "Waiting $min_sleep seconds before next backup retry...",
+                        undef,
+                        1,
+                    );
+                    sleep($min_sleep);
+                    next;
+                }
+
+                for my $vmid (@$ready_to_run) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    $fail_info->{retries}++;
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid"
+                            . " (retry $fail_info->{retries}/$max_retries)",
+                        undef,
+                        1,
+                    );
+
+                    my $task = $fail_info->{task};
+                    my $tmplog = $task->{tmplog};
+                    %$task = (
+                        %{ $fail_info->{initial_task} },
+                        retries => $fail_info->{retries},
+                        tmplog => $tmplog,
+                    );
+                    $self->exec_backup_task($task);
+
+                    if ($task->{state} eq 'ok') {
+                        $log_retried_success->($vmid, $fail_info->{retries});
+                        delete $failed_tasks->{$vmid};
+                    } elsif ($task->{committed}) {
+                        $log_committed_failure->($vmid);
+                        $errcount++;
+                        delete $failed_tasks->{$vmid};
+                    } else {
+                        $fail_info->{fail_time} = time();
+                    }
+                }
+            }
         }
 
         $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
@@ -1472,6 +1673,13 @@ sub exec_backup {
     } else {
         if ($errcount) {
             debugmsg('info', "Backup job finished with errors", undef, 1);
+        } elsif ($retried_count) {
+            debugmsg(
+                'info',
+                "Backup job finished with warnings - $retried_count guest(s) needed retries",
+                undef,
+                1,
+            );
         } else {
             debugmsg('info', "Backup job finished successfully", undef, 1);
         }
@@ -1547,6 +1755,9 @@ sub verify_vzdump_parameters {
     raise_param_exc({ pool => "option conflicts with option 'vmid'" })
         if $param->{pool} && $param->{vmid};
 
+    raise_param_exc({ 'error-behavior' => "retries are not possible with option 'stdout'" })
+        if $param->{stdout} && ($param->{'error-behavior'} // 'ignore') ne 'ignore';
+
     $parse_prune_backups->($param, 'CLI parameters');
     parse_fleecing($param);
     parse_performance($param);
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Expose the `error-behavior` and `max-retries` vzdump parameters in the
scheduled datacenter backup and in the manual backup window.

New backup jobs preselect `retry`, as suggested in the bug report,
while existing jobs keep their configuration when edited. The schema
default stays `ignore`, so existing jobs, CLI and API usage keep their
current behavior. The manual backup window uses the node's defaults
from /etc/vzdump.conf.

The manual backup window only offers `retry` and `ignore`, since it
backs up a single guest, where `retry` and `retry-immediately` behave
the same. The width of the scheduled backup job window is increased
from 720 to 750 pixels so the new fields fit without cutting off their
labels.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 www/manager6/dc/Backup.js                   |  2 +-
 www/manager6/panel/BackupAdvancedOptions.js | 52 +++++++++++++++++++
 www/manager6/window/Backup.js               | 56 ++++++++++++++++++++-
 3 files changed, 108 insertions(+), 2 deletions(-)

diff --git a/www/manager6/dc/Backup.js b/www/manager6/dc/Backup.js
index d3db9d9c..b640fa1a 100644
--- a/www/manager6/dc/Backup.js
+++ b/www/manager6/dc/Backup.js
@@ -7,7 +7,7 @@ Ext.define('PVE.dc.BackupEdit', {
     defaultFocus: undefined,
 
     subject: gettext('Backup Job'),
-    width: 720,
+    width: 750,
     bodyPadding: 0,
 
     url: '/api2/extjs/cluster/backup',
diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf..5023cdad 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -24,6 +24,11 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
             me.lookup('fleecingStorage').setDisabled(!value);
         },
 
+        errorBehaviorChange: function (field, value) {
+            let me = this;
+            me.lookup('maxRetriesField').setDisabled(value === 'ignore');
+        },
+
         control: {
             'proxmoxcheckbox[reference=fleecingEnabled]': {
                 change: 'toggleFleecing',
@@ -117,6 +122,53 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
     },
 
     items: [
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxKVComboBox',
+                fieldLabel: gettext('Error Behavior'),
+                name: 'error-behavior',
+                comboItems: [
+                    ['__default__', Proxmox.Utils.defaultText],
+                    ['retry', gettext('Retry at end of job')],
+                    ['retry-immediately', gettext('Retry immediately')],
+                    ['ignore', gettext('Ignore')],
+                ],
+                cbind: {
+                    value: (get) => (get('isCreate') ? 'retry' : '__default__'),
+                    deleteEmpty: '{!isCreate}',
+                },
+                listeners: {
+                    change: 'errorBehaviorChange',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: `${gettext('How to handle guest backup failures.')} ${Ext.String.format(gettext('Schema default: {0}'), gettext('Ignore'))}`,
+            },
+        },
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxintegerfield',
+                fieldLabel: gettext('Max Retries'),
+                name: 'max-retries',
+                minValue: 1,
+                maxValue: 10,
+                reference: 'maxRetriesField',
+                allowBlank: true,
+                emptyText: Proxmox.Utils.defaultText,
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: `${gettext('Maximum number of retries before giving up on a guest.')} ${Ext.String.format(gettext('Schema default: {0}'), 5)}`,
+            },
+        },
         {
             xtype: 'pveTwoColumnContainer',
             startColumn: {
diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
index 0caada65..3ce1d69b 100644
--- a/www/manager6/window/Backup.js
+++ b/www/manager6/window/Backup.js
@@ -55,6 +55,34 @@ Ext.define('PVE.window.Backup', {
             },
         });
 
+        let maxRetriesField = Ext.create({
+            xtype: 'proxmoxintegerfield',
+            fieldLabel: gettext('Max Retries'),
+            name: 'max-retries',
+            minValue: 1,
+            maxValue: 10,
+            allowBlank: true,
+            emptyText: '5',
+            disabled: true,
+        });
+
+        let errorBehaviorSelector = Ext.create({
+            xtype: 'proxmoxKVComboBox',
+            fieldLabel: gettext('Error Behavior'),
+            name: 'error-behavior',
+            value: 'ignore',
+            deleteEmpty: false,
+            comboItems: [
+                ['retry', gettext('Retry')],
+                ['ignore', gettext('Ignore')],
+            ],
+            listeners: {
+                change: function (field, value) {
+                    maxRetriesField.setDisabled(value === 'ignore');
+                },
+            },
+        });
+
         let pbsChangeDetectionModeSelector = Ext.create({
             xtype: 'proxmoxKVComboBox',
             flex: 1,
@@ -196,6 +224,17 @@ Ext.define('PVE.window.Backup', {
                                 );
                             }
 
+                            if (!initialDefaults && data['error-behavior'] !== undefined) {
+                                // both retry strategies behave the same for a single guest
+                                let behavior = data['error-behavior'];
+                                errorBehaviorSelector.setValue(
+                                    behavior === 'retry-immediately' ? 'retry' : behavior,
+                                );
+                            }
+                            if (!initialDefaults && data['max-retries'] !== undefined) {
+                                maxRetriesField.setValue(data['max-retries']);
+                            }
+
                             initialDefaults = true;
 
                             // always update storage dependent properties
@@ -250,7 +289,14 @@ Ext.define('PVE.window.Backup', {
             bodyPadding: 10,
             border: false,
             column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
-            column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
+            column2: [
+                compressionSelector,
+                notificationModeSelector,
+                mailtoField,
+                removeCheckbox,
+                errorBehaviorSelector,
+                maxRetriesField,
+            ],
             columnB: [
                 {
                     xtype: 'textareafield',
@@ -342,6 +388,14 @@ Ext.define('PVE.window.Backup', {
                     );
                 }
 
+                if (values['error-behavior']) {
+                    params['error-behavior'] = values['error-behavior'];
+                }
+
+                if (values['max-retries'] !== undefined && values['max-retries'] !== '') {
+                    params['max-retries'] = values['max-retries'];
+                }
+
                 Proxmox.Utils.API2Request({
                     url: '/nodes/' + me.nodename + '/vzdump',
                     params: params,
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH pve-docs v3 4/4] vzdump: document backup retry options
  2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
                   ` (2 preceding siblings ...)
  2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
@ 2026-11-16 10:13 ` Jakob Klocker
  3 siblings, 0 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

Describe the new `error-behavior` and `max-retries` options, the
exponential backoff between attempts, and which failures are retried.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..856d3d3 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,71 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_retry]]
+Backup Retries
+--------------
+
+The backup of a single guest can fail because of a temporary problem, for
+example when the guest is locked by another operation, or when the connection
+to the backup server is interrupted during the backup. To avoid a failed backup
+in such cases, the `error-behavior` option allows a job to retry failed guest
+backups automatically:
+
+`ignore` ::
+Does not retry failed guests. This is the default.
+
+`retry` ::
+Defers failed guests until all other guests have been processed and then
+retries them in the order in which they failed.
+
+`retry-immediately` ::
+Retries a failed guest right away, before continuing with the next guest.
+
+The `max-retries` option (1 to 10, default 5) limits how often a guest is
+retried. Between attempts, the job uses exponential backoff: the delay starts
+at 30 seconds, doubles with every retry and is capped at 30 minutes. With
+`retry`, the delay is counted from the last failed attempt of that guest, so
+other guests can be retried in the meantime.
+
+If a retry succeeds, the guest does not count as an error, but the task
+finishes with a warning and the notification is sent with severity `warning`
+and shows how many retries were needed. This way, recurring problems stay
+visible even if all backups eventually succeed. If the last retry fails as
+well, the job finishes with errors. The task log of the guest contains all
+attempts.
+
+For example, to back up three guests and retry each failed guest at most three
+times at the end of the job, run:
+
+ # vzdump 101 102 103 --error-behavior retry --max-retries 3
+
+Both options can also be set as node-wide defaults in `/etc/vzdump.conf`, see
+xref:vzdump_configuration[configuration].
+
+In the web interface, the retry strategy for backup jobs can be set in the
+'Advanced' tab of the backup job window. New backup jobs preselect `retry`,
+existing jobs keep their configuration. For a manual backup, the retry strategy
+can be chosen directly in the backup window ('Backup now'), which preselects the
+node's default from `/etc/vzdump.conf`. It only offers `retry` and `ignore`,
+as both retry strategies behave the same for a single guest.
+
+Only failures that happen before the backup archive of a guest has been created
+are retried. If the archive was created, but a later step failed, for example a
+failing `backup-end` hook script, the guest is not retried. This avoids
+creating multiple backups of the same guest and pruning older backups
+repeatedly. Errors that prevent the whole job from starting, for example a
+backup storage that cannot be activated, are not retried either.
+
+Retries are not possible when writing the backup to standard output with
+`--stdout`, as a retry would append a second archive to the data that was
+already written. Setting `error-behavior` to a retry strategy together with
+`--stdout` is rejected, and a retry strategy from `/etc/vzdump.conf` is
+ignored with a warning.
+
+NOTE: Retries can make a backup job take considerably longer. With the default
+of 5 retries, the delays alone add up to more than 15 minutes for a guest that
+keeps failing, and about 2.5 hours with the maximum of 10 retries.
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-16 10:15 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal