[PATCH docs/guest-common/manager v2 0/4] fix #6464: backup: add backup retry strategies

[PATCH docs/guest-common/manager v2 0/4] fix #6464: backup: add backup retry strategies

[Jakob Klocker]

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `retry`: Defers failed guests and retries them at the end of the job
  up to the configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, it is reported as successful in the
notification and no error is counted. If it still fails after the last
retry, the job finishes with errors as before. Task logging has been
switched to append mode so the log contains the complete transcript of
all attempts.

I kept the retry options in the advanced tab of the backup job window,
but I'm not sure whether they should be moved to the general tab.
Opinions on this are appreciated.

Testing info:
When testing I'd suggest to fail the backup by locking and
unlocking the guest with following commands:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 131 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 ++++++++
 www/manager6/window/Backup.js               |  45 ++++++-
 4 files changed, 226 insertions(+), 4 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)


Summary over all repositories:
  6 files changed, 290 insertions(+), 4 deletions(-)

-- 
Generated by murpp 0.12.0

[PATCH pve-guest-common v2 1/4] vzdump: add error-behavior and max-retries options

[Jakob Klocker]

Add two new backup job options to control what happens when a guest
backup fails:

- 'error-behavior' selects the behavior. 'ignore' keeps the current
  behavior and is the default. 'retry' queues the failed guest and
  retries it after the remaining guests have been backed up.
  'retry-immediately' retries the failed guest right away, before
  continuing with the next one.
- 'max-retries' limits how many times a failed guest backup is retried
  (1-10, default 5).

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 src/PVE/VZDump/Common.pm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/src/PVE/VZDump/Common.pm b/src/PVE/VZDump/Common.pm
index 2cb48a3..6ca773a 100644
--- a/src/PVE/VZDump/Common.pm
+++ b/src/PVE/VZDump/Common.pm
@@ -384,6 +384,21 @@ my $confdesc = {
         optional => 1,
         enum => ['legacy', 'data', 'metadata'],
     },
+    'error-behavior' => {
+        type => 'string',
+        description => "Behavior when a guest backup fails.",
+        enum => ['ignore', 'retry', 'retry-immediately'],
+        optional => 1,
+        default => 'ignore',
+    },
+    'max-retries' => {
+        type => 'integer',
+        description => "Maximum number of times to retry a failed guest backup.",
+        minimum => 1,
+        maximum => 10,
+        optional => 1,
+        default => 5,
+    },
 };
 
 sub get_confdesc {
-- 
2.47.3

[PATCH pve-manager v2 2/4] fix #6464: vzdump: add retry strategies for failed guest backups

[Jakob Klocker]

Add the `error-behavior` and `max-retries` options to automatically
retry failed guest backups, which helps with transient failures.

Supported retry strategies:
- `ignore`: Preserves current behavior (no retries, default).
- `retry-immediately`: Retries the guest inline after
  failure.
- `retry`: Defers failed guests until all other guests have been
processed, then retries them in the order they failed.

A guest is retried at most `max-retries` times. Between attempts, the
delay starts at 30 seconds and doubles with each retry, capped at 30
minutes as suggested on bugzilla [0]. If a guest still fails after the
last retry, it is counted as an error and the job finishes with errors.
If a retry succeeds, the guest is reported as successful in the
notifications and no error is counted.

The task log is opened in append mode for retries, so the log contains
the complete transcript of all attempts.

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 PVE/VZDump.pm | 131 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 129 insertions(+), 2 deletions(-)

diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
index 56d2aea3..e4aa303d 100644
--- a/PVE/VZDump.pm
+++ b/PVE/VZDump.pm
@@ -10,6 +10,7 @@ use File::Path;
 use IO::File;
 use IO::Select;
 use IPC::Open3;
+use List::Util qw(min);
 use POSIX qw(strftime);
 use Time::Local;
 
@@ -1048,7 +1049,8 @@ sub exec_backup_task {
         -d $task->{tmpdir}
             || die "unable to create temporary directory '$task->{tmpdir}'";
 
-        $logfd = IO::File->new(">$tmplog")
+        my $logmode = $task->{retries} ? '>>' : '>';
+        $logfd = IO::File->new("${logmode}${tmplog}")
             || die "unable to create log file '$tmplog'";
 
         $task->{dumpdir} = $opts->{dumpdir};
@@ -1435,14 +1437,139 @@ sub exec_backup {
 
     my $starttime = time();
     my $errcount = 0;
+
+    my $error_behavior = $opts->{'error-behavior'} // 'ignore';
+    my $max_retries = $opts->{'max-retries'} // 5;
+    my $min_between_retry = 30;
+    my $max_wait = 60 * 30;
+
+    my $get_backoff = sub {
+        my ($retry) = @_;
+        return min($min_between_retry * 2**($retry - 1), $max_wait);
+    };
+
     eval {
 
         $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
         $self->run_hook_script('job-start', undef, $job_start_fd);
 
+        my $failed_tasks = {};
+
         foreach my $task (@$tasklist) {
+            my $vmid = $task->{vmid};
+            my $initial_task = {%$task};
+
+            $task->{retries} = 0;
             $self->exec_backup_task($task);
-            $errcount += 1 if $task->{state} ne 'ok';
+
+            next if $task->{state} eq 'ok';
+
+            if ($error_behavior eq 'retry-immediately') {
+                for my $retry (1 .. $max_retries) {
+                    my $backoff = $get_backoff->($retry);
+
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid (attempt $retry/$max_retries) after ${backoff}s delay...",
+                        undef,
+                        1,
+                    );
+                    sleep($backoff);
+
+                    %$task = (%$initial_task, retries => $retry);
+                    $self->exec_backup_task($task);
+
+                    last if $task->{state} eq 'ok';
+                }
+
+            } elsif ($error_behavior eq 'retry') {
+                $failed_tasks->{$vmid} = {
+                    task => $task,
+                    initial_task => $initial_task,
+                    fail_time => time(),
+                    retries => 0,
+                };
+                next;
+            }
+
+            if ($task->{state} ne 'ok') {
+                debugmsg('err', "Guest $vmid failed after $max_retries retries.", undef, 1)
+                    if $error_behavior eq 'retry-immediately';
+                $errcount++;
+            }
+        }
+
+        if ($error_behavior eq 'retry') {
+            while (keys %$failed_tasks > 0) {
+                my $now = time();
+                my $min_sleep = $max_wait;
+                my $ready_to_run = [];
+
+                my @sorted_vmids = sort {
+                    $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time}
+                        || $a <=> $b
+                } keys %$failed_tasks;
+
+                foreach my $vmid (@sorted_vmids) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    if ($fail_info->{retries} >= $max_retries) {
+                        debugmsg(
+                            'err',
+                            "Guest $vmid failed after $max_retries retries.",
+                            undef,
+                            1,
+                        );
+                        $errcount++;
+                        delete $failed_tasks->{$vmid};
+                        next;
+                    }
+
+                    my $delay = $get_backoff->($fail_info->{retries} + 1);
+                    my $target_time = $fail_info->{fail_time} + $delay;
+                    my $wait_time = $target_time - $now;
+
+                    if ($wait_time > 0) {
+                        $min_sleep = $wait_time if $wait_time < $min_sleep;
+                    } else {
+                        push @$ready_to_run, $vmid;
+                    }
+                }
+
+                if (scalar(@$ready_to_run) == 0 && keys %$failed_tasks > 0) {
+                    debugmsg(
+                        'info',
+                        "Waiting $min_sleep seconds before next backup retry...",
+                        undef,
+                        1,
+                    );
+                    sleep($min_sleep);
+                    next;
+                }
+
+                foreach my $vmid (@$ready_to_run) {
+                    my $fail_info = $failed_tasks->{$vmid};
+
+                    $fail_info->{retries}++;
+                    debugmsg(
+                        'info',
+                        "Retrying backup for guest $vmid (attempt $fail_info->{retries}/$max_retries)",
+                        undef,
+                        1,
+                    );
+
+                    my $task = $fail_info->{task};
+                    %$task =
+                        (%{ $fail_info->{initial_task} }, retries => $fail_info->{retries});
+                    $self->exec_backup_task($task);
+
+                    if ($task->{state} eq 'ok') {
+                        delete $failed_tasks->{$vmid};
+                    } else {
+                        $fail_info->{fail_time} = time();
+                    }
+                }
+            }
         }
 
         $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
-- 
2.47.3

Re: [PATCH pve-manager v2 2/4] fix #6464: vzdump: add retry strategies for failed guest backups

[Jonas Theisen]

On 11/11/26 10:35, Jakob Klocker wrote:
> <snip>
>
> diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
> index 56d2aea3..e4aa303d 100644
> --- a/PVE/VZDump.pm
> +++ b/PVE/VZDump.pm
> @@ -10,6 +10,7 @@ use File::Path;
>   use IO::File;
>   use IO::Select;
>   use IPC::Open3;
> +use List::Util qw(min);
>   use POSIX qw(strftime);
>   use Time::Local;
>   
> @@ -1048,7 +1049,8 @@ sub exec_backup_task {
>           -d $task->{tmpdir}
>               || die "unable to create temporary directory '$task->{tmpdir}'";
>   
> -        $logfd = IO::File->new(">$tmplog")
> +        my $logmode = $task->{retries} ? '>>' : '>';
> +        $logfd = IO::File->new("${logmode}${tmplog}")

In case the first backup run fails before opening its backup log
the retry's log gets appended to an older previous log file.
Maybe cleanup the $tmplog once per job to avoid appending
to a remaining log file?
>               || die "unable to create log file '$tmplog'";
>   
>           $task->{dumpdir} = $opts->{dumpdir};
>
> <snip>

[PATCH pve-manager v2 3/4] ui: backup: add error behavior and retry options

[Jakob Klocker]

Expose the `error-behavior` and `max-retries` vzdump parameters in the
scheduled datacenter backup and in the manual backup window.

When creating a new backup job or starting a manual backup, the UI
defaults to `retry` with a maximum of 5 retries.

The manual backup window only offers `retry` and `ignore`, since it
backs up a single guest, where `retry` and `retry-immediately` behave
the same. The width of the scheduled backup job window is increased
from 720 to 750 pixels so the new fields fit without cutting off their
labels.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 www/manager6/dc/Backup.js                   |  2 +-
 www/manager6/panel/BackupAdvancedOptions.js | 52 +++++++++++++++++++++
 www/manager6/window/Backup.js               | 45 +++++++++++++++++-
 3 files changed, 97 insertions(+), 2 deletions(-)

diff --git a/www/manager6/dc/Backup.js b/www/manager6/dc/Backup.js
index d3db9d9c..b640fa1a 100644
--- a/www/manager6/dc/Backup.js
+++ b/www/manager6/dc/Backup.js
@@ -7,7 +7,7 @@ Ext.define('PVE.dc.BackupEdit', {
     defaultFocus: undefined,
 
     subject: gettext('Backup Job'),
-    width: 720,
+    width: 750,
     bodyPadding: 0,
 
     url: '/api2/extjs/cluster/backup',
diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
index fe8cc3bf..b06011fe 100644
--- a/www/manager6/panel/BackupAdvancedOptions.js
+++ b/www/manager6/panel/BackupAdvancedOptions.js
@@ -24,6 +24,11 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
             me.lookup('fleecingStorage').setDisabled(!value);
         },
 
+        errorBehaviorChange: function (field, value) {
+            let me = this;
+            me.lookup('maxRetriesField').setDisabled(value === 'ignore');
+        },
+
         control: {
             'proxmoxcheckbox[reference=fleecingEnabled]': {
                 change: 'toggleFleecing',
@@ -117,6 +122,53 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
     },
 
     items: [
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxKVComboBox',
+                fieldLabel: gettext('Error Behavior'),
+                name: 'error-behavior',
+                value: 'retry',
+                comboItems: [
+                    ['retry', gettext('Retry at end of job')],
+                    ['retry-immediately', gettext('Retry immediately')],
+                    ['ignore', gettext('Ignore')],
+                ],
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+                listeners: {
+                    change: 'errorBehaviorChange',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: gettext('How to handle a guest backup failure.'),
+            },
+        },
+        {
+            xtype: 'pveTwoColumnContainer',
+            startColumn: {
+                xtype: 'proxmoxintegerfield',
+                fieldLabel: gettext('Max Retries'),
+                name: 'max-retries',
+                minValue: 1,
+                maxValue: 10,
+                reference: 'maxRetriesField',
+                value: 5,
+                allowBlank: true,
+                emptyText: '5',
+                cbind: {
+                    deleteEmpty: '{!isCreate}',
+                },
+            },
+            endFlex: 2,
+            endColumn: {
+                xtype: 'displayfield',
+                value: gettext('Maximum number of retries before giving up on a guest.'),
+            },
+        },
         {
             xtype: 'pveTwoColumnContainer',
             startColumn: {
diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
index 0caada65..5552620b 100644
--- a/www/manager6/window/Backup.js
+++ b/www/manager6/window/Backup.js
@@ -55,6 +55,34 @@ Ext.define('PVE.window.Backup', {
             },
         });
 
+        let maxRetriesField = Ext.create({
+            xtype: 'proxmoxintegerfield',
+            fieldLabel: gettext('Max Retries'),
+            name: 'max-retries',
+            minValue: 1,
+            maxValue: 10,
+            value: 5,
+            allowBlank: true,
+            emptyText: '5',
+        });
+
+        let errorBehaviorSelector = Ext.create({
+            xtype: 'proxmoxKVComboBox',
+            fieldLabel: gettext('Error Behavior'),
+            name: 'error-behavior',
+            value: 'retry',
+            deleteEmpty: false,
+            comboItems: [
+                ['retry', gettext('Retry')],
+                ['ignore', gettext('Ignore')],
+            ],
+            listeners: {
+                change: function (field, value) {
+                    maxRetriesField.setDisabled(value === 'ignore');
+                },
+            },
+        });
+
         let pbsChangeDetectionModeSelector = Ext.create({
             xtype: 'proxmoxKVComboBox',
             flex: 1,
@@ -250,7 +278,14 @@ Ext.define('PVE.window.Backup', {
             bodyPadding: 10,
             border: false,
             column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
-            column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
+            column2: [
+                compressionSelector,
+                notificationModeSelector,
+                mailtoField,
+                removeCheckbox,
+                errorBehaviorSelector,
+                maxRetriesField,
+            ],
             columnB: [
                 {
                     xtype: 'textareafield',
@@ -342,6 +377,14 @@ Ext.define('PVE.window.Backup', {
                     );
                 }
 
+                if (values['error-behavior']) {
+                    params['error-behavior'] = values['error-behavior'];
+                }
+
+                if (values['max-retries'] !== undefined && values['max-retries'] !== '') {
+                    params['max-retries'] = values['max-retries'];
+                }
+
                 Proxmox.Utils.API2Request({
                     url: '/nodes/' + me.nodename + '/vzdump',
                     params: params,
-- 
2.47.3

Re: [PATCH pve-manager v2 3/4] ui: backup: add error behavior and retry options

[Jonas Theisen]

On 11/11/26 10:35, Jakob Klocker wrote:
> Expose the `error-behavior` and `max-retries` vzdump parameters in the
> scheduled datacenter backup and in the manual backup window.
>
> When creating a new backup job or starting a manual backup, the UI
> defaults to `retry` with a maximum of 5 retries.
>
> The manual backup window only offers `retry` and `ignore`, since it
> backs up a single guest, where `retry` and `retry-immediately` behave
> the same. The width of the scheduled backup job window is increased
> from 720 to 750 pixels so the new fields fit without cutting off their
> labels.
>
> Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
> ---
>   www/manager6/dc/Backup.js                   |  2 +-
>   www/manager6/panel/BackupAdvancedOptions.js | 52 +++++++++++++++++++++
>   www/manager6/window/Backup.js               | 45 +++++++++++++++++-
>   3 files changed, 97 insertions(+), 2 deletions(-)
>
> diff --git a/www/manager6/dc/Backup.js b/www/manager6/dc/Backup.js
> index d3db9d9c..b640fa1a 100644
> --- a/www/manager6/dc/Backup.js
> +++ b/www/manager6/dc/Backup.js
> @@ -7,7 +7,7 @@ Ext.define('PVE.dc.BackupEdit', {
>       defaultFocus: undefined,
>   
>       subject: gettext('Backup Job'),
> -    width: 720,
> +    width: 750,
>       bodyPadding: 0,
>   
>       url: '/api2/extjs/cluster/backup',
> diff --git a/www/manager6/panel/BackupAdvancedOptions.js b/www/manager6/panel/BackupAdvancedOptions.js
> index fe8cc3bf..b06011fe 100644
> --- a/www/manager6/panel/BackupAdvancedOptions.js
> +++ b/www/manager6/panel/BackupAdvancedOptions.js
> @@ -24,6 +24,11 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
>               me.lookup('fleecingStorage').setDisabled(!value);
>           },
>   
> +        errorBehaviorChange: function (field, value) {
> +            let me = this;
> +            me.lookup('maxRetriesField').setDisabled(value === 'ignore');
> +        },
> +
>           control: {
>               'proxmoxcheckbox[reference=fleecingEnabled]': {
>                   change: 'toggleFleecing',
> @@ -117,6 +122,53 @@ Ext.define('PVE.panel.BackupAdvancedOptions', {
>       },
>   
>       items: [
> +        {
> +            xtype: 'pveTwoColumnContainer',
> +            startColumn: {
> +                xtype: 'proxmoxKVComboBox',
> +                fieldLabel: gettext('Error Behavior'),
> +                name: 'error-behavior',
> +                value: 'retry',

The default "retry" also causes existing jobs to be silently switched
over to that after an unrelated edit, which is not ideal.
> diff --git a/www/manager6/window/Backup.js b/www/manager6/window/Backup.js
> index 0caada65..5552620b 100644
> --- a/www/manager6/window/Backup.js
> +++ b/www/manager6/window/Backup.js
> @@ -55,6 +55,34 @@ Ext.define('PVE.window.Backup', {
>               },
>           });
>   
> +        let maxRetriesField = Ext.create({
> +            xtype: 'proxmoxintegerfield',
> +            fieldLabel: gettext('Max Retries'),
> +            name: 'max-retries',
> +            minValue: 1,
> +            maxValue: 10,
> +            value: 5,
> +            allowBlank: true,
> +            emptyText: '5',
> +        });
> +
> +        let errorBehaviorSelector = Ext.create({
> +            xtype: 'proxmoxKVComboBox',
> +            fieldLabel: gettext('Error Behavior'),
> +            name: 'error-behavior',
> +            value: 'retry',
> +            deleteEmpty: false,
> +            comboItems: [
> +                ['retry', gettext('Retry')],
> +                ['ignore', gettext('Ignore')],
> +            ],
> +            listeners: {
> +                change: function (field, value) {
> +                    maxRetriesField.setDisabled(value === 'ignore');
> +                },
> +            },
> +        });
> +
>           let pbsChangeDetectionModeSelector = Ext.create({
>               xtype: 'proxmoxKVComboBox',
>               flex: 1,
> @@ -250,7 +278,14 @@ Ext.define('PVE.window.Backup', {
>               bodyPadding: 10,
>               border: false,
>               column1: [storagesel, modeSelector, protectedCheckbox, pbsChangeDetection],
> -            column2: [compressionSelector, notificationModeSelector, mailtoField, removeCheckbox],
> +            column2: [
> +                compressionSelector,
> +                notificationModeSelector,
> +                mailtoField,
> +                removeCheckbox,
> +                errorBehaviorSelector,
> +                maxRetriesField,
> +            ],
>
>               columnB: [
>                   {
>                       xtype: 'textareafield',
> @@ -342,6 +377,14 @@ Ext.define('PVE.window.Backup', {
>                       );
>                   }
>   
> +                if (values['error-behavior']) {
> +                    params['error-behavior'] = values['error-behavior'];
> +                }
> +
> +                if (values['max-retries'] !== undefined && values['max-retries'] !== '') {
> +                    params['max-retries'] = values['max-retries'];
> +                }
> +
>                   Proxmox.Utils.API2Request({
>                       url: '/nodes/' + me.nodename + '/vzdump',
>                       params: params,

The manual backup window does not honor configured defaults for the
two new parameters from /etc/vzdump.conf.
So even if an admin was to set a different value there the GUI values
would overwrite them.

[PATCH pve-docs v2 4/4] vzdump: document backup retry options

[Jakob Klocker]

Describe the new `error-behavior` and `max-retries` options, the
exponential backoff between attempts, and which failures are retried.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 vzdump.adoc | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..872ff16 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,55 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_retry]]
+Backup Retries
+--------------
+
+The backup of a single guest can fail because of a temporary problem, for
+example when the guest is locked by another operation, or when the connection
+to the backup server is interrupted during the backup. To avoid a failed backup
+in such cases, the `error-behavior` option allows a job to retry failed guest
+backups automatically:
+
+`ignore` ::
+Does not retry failed guests. This is the default.
+
+`retry` ::
+Defers failed guests until all other guests have been processed and then
+retries them in the order in which they failed.
+
+`retry-immediately` ::
+Retries a failed guest right away, before continuing with the next guest.
+
+The `max-retries` option (1 to 10, default 5) limits how often a guest is
+retried. Between attempts, the job uses exponential backoff: the delay starts
+at 30 seconds, doubles with every retry and is capped at 30 minutes. With
+`retry`, the delay is counted from the last failed attempt of that guest, so
+other guests can be retried in the meantime.
+
+If a retry succeeds, the guest is reported as successful in the notification
+and does not count as an error. If the last retry fails as well, the job
+finishes with errors. The task log of the guest contains all attempts.
+
+For example, to back up three guests and retry each failed guest at most three
+times at the end of the job, run:
+
+ # vzdump 101 102 103 --error-behavior retry --max-retries 3
+
+In the UI, the options for backup jobs can be set in the 'Advanced' tab of the
+backup job window. For a manual backup, they are available directly in the
+backup window ('Backup now'). For new backup jobs and manual backups, the UI
+preselects `retry` with 5 retries. The manual backup window only offers `retry`
+and `ignore`, as both retry strategies behave the same for a single guest.
+
+Only failures while backing up a guest are retried. Errors that prevent the
+whole job from starting, for example a backup storage that cannot be activated,
+are not retried.
+
+NOTE: Retries can make a backup job take considerably longer. With the default
+of 5 retries, the delays alone add up to more than 15 minutes for a guest that
+keeps failing, and about 2.5 hours with the maximum of 10 retries.
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3

Re: [PATCH docs/guest-common/manager v2 0/4] fix #6464: backup: add backup retry strategies

[Jonas Theisen]

On 11/11/26 10:35, Jakob Klocker wrote:
> <snip>
>
> If a guest succeeds on a retry, it is reported as successful in the
> notification and no error is counted. If it still fails after the last
> retry, the job finishes with errors as before. Task logging has been
> switched to append mode so the log contains the complete transcript of
> all attempts.

I personally would prefer if a transient failure would still be
reported as a warning. Retries are good to not miss the backup
completely but it still in most cases means something is wrong.
> I kept the retry options in the advanced tab of the backup job window,
> but I'm not sure whether they should be moved to the general tab.
> Opinions on this are appreciated.

A more general issue is the combination of retries with the --stdout
parameter, where the tar gets written to a stdout instead of a file.
The resulting stream gets corrupted by a retry so probably the CLI
should just bail in that combination.
>
> Testing info:
> When testing I'd suggest to fail the backup by locking and
> unlocking the guest with following commands:
> `qm set <VMID> -lock backup`
> `qm unlock <VMID>`

I also tested this in combination with an unreachable PBS by just
yanking out the connection on the PBS side.
This caused the backup task to fail even before the retries since
it could not activate the storage.
Since i assume this is one of the more common failure modes
i think that should be covered by a retry.
> <snip>