public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: Jakob Klocker <j.klocker@proxmox.com>, pve-devel@lists.proxmox.com
Subject: Re: [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups
Date: Sat, 21 Nov 2026 22:42:50 +0100	[thread overview]
Message-ID: <53857bec-e147-4911-91ea-b61553194a52@proxmox.com> (raw)
In-Reply-To: <20261120160510.311796-6-j.klocker@proxmox.com>

Am 20.11.26 um 17:07 schrieb Jakob Klocker:
> Add the `error-behavior` and `max-retries` options to automatically
> retry failed guest backups, which helps with transient failures.
> 
> Supported retry strategies:
> - `ignore`: Preserves current behavior (no retries, default).
> - `retry-immediately`: Retries the guest inline after failure.
> - `retry`: Defers failed guests until all other guests have been
>   processed, then retries them in the order they failed.
> 
> A guest is retried at most `max-retries` times. Between attempts, the
> delay starts at 30 seconds and doubles with each retry, capped at 30
> minutes as suggested on bugzilla [0]. If a guest still fails after the
> last retry, it is counted as an error and the job finishes with errors.
> If a retry succeeds, the task finishes with a warning and the
> notification is sent with severity `warning`. Its subject shows how
> many guests needed retries, and the guest table shows how many retries
> each of them needed.
> 
> Guests whose archive was already created (e.g. failing backup-end hook)
> are not retried, to avoid duplicate archives and repeated pruning.
> 
> Retries are not possible with --stdout, as a retry would corrupt the
> stream. Setting `error-behavior` to a retry strategy together with
> --stdout is rejected, and a retry strategy from /etc/vzdump.conf is
> ignored with a warning.
> 
> The task log is kept across retries and contains all attempts.
> 
> [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
> Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464
> 
> Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
> ---
>  PVE/VZDump.pm | 219 +++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 215 insertions(+), 4 deletions(-)
> 
> diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm
> index d2861f9e..d0f98a13 100644
> --- a/PVE/VZDump.pm
> +++ b/PVE/VZDump.pm
> @@ -10,6 +10,7 @@ use File::Path;
>  use IO::File;
>  use IO::Select;
>  use IPC::Open3;
> +use List::Util qw(min);
>  use POSIX qw(strftime);
>  use Time::Local;
>  
> @@ -20,6 +21,7 @@ use PVE::HA::Config;
>  use PVE::HA::Env::PVE2;
>  use PVE::JSONSchema qw(get_standard_option);
>  use PVE::Notify;
> +use PVE::RESTEnvironment qw(log_warn);

debugmsg already calls this when passing 'warn' as level and also has
some other special handling for how vzdump is used, in any case
this should not be required here.

>  use PVE::RPCEnvironment;
>  use PVE::Storage;
>  use PVE::VZDump::Common;
> @@ -395,6 +397,11 @@ my sub read_backup_task_logs {
>      return $task_logs;
>  }
>  
> +my sub retries_text {
> +    my ($count) = @_;
> +    return $count == 1 ? '1 retry' : "$count retries";
> +}

I don't want to add such a helper for every plural text there might be...

Iff, then check if we can use pre-existing ones or do at least a generic
one, like:

sub plural($count, $singular, $plural) {
    return $count == 1 ? $singular : $plural;
}

Else this is gonna end in bloat.

> +
>  my sub build_guest_table {
>      my ($task_list) = @_;
>  
> @@ -436,11 +443,18 @@ my sub build_guest_table {
>          my $successful = $task->{state} eq 'ok';
>          my $size = $successful ? $task->{size} : 0;
>          my $filename = $successful ? $task->{target} : undef;
> +
> +        my $status = $task->{state};
> +        my $retries = $task->{retries};

could be

my ($status, $retries) = $task->{'state', 'retries'};

also $task->{'state'} is used a bit above already at my $successful.

but tbh. not a big fan of such "useless" (i.e. 1:1 reference) intermediate
variables for straight forward single-level hash element access. They just
add complexity as one needs to check where they are set while not winning much.

> +        if ($successful && $retries) {
> +            $status .= " (after " . retries_text($retries) . ")";
> +        }
> +
>          push @{ $table->{data} },
>              {
>                  "vmid" => int($task->{vmid}),
>                  "name" => $task->{hostname},
> -                "status" => $task->{state},
> +                "status" => $status,
>                  "time" => int($task->{backuptime}),
>                  "size" => int($size),
>                  "filename" => $filename,
> @@ -502,8 +516,11 @@ sub send_notification {
>      my ($error_count, $total_size) = aggregate_task_statistics($tasklist);
>  
>      my $failed = ($error_count || $err);
> +    my $retried_count = grep { $_->{state} eq 'ok' && $_->{retries} } @$tasklist;
>  
>      my $status_text = $failed ? 'backup failed' : 'backup successful';
> +    $status_text .= " ($retried_count guest(s) needed retries)"
> +        if !$failed && $retried_count;

here now plural difference? would do this consistently if we do.

>  
>      if ($err) {
>          if ($err =~ /\n/) {
> @@ -540,7 +557,7 @@ sub send_notification {
>      # Add backup-job metadata field in case this is a backup job.
>      $fields->{'job-id'} = $job_id if $job_id;
>  
> -    my $severity = $failed ? "error" : "info";
> +    my $severity = $failed ? "error" : $retried_count ? "warning" : "info";
>      my $email_configured = $mailto && scalar(@$mailto);
>  
>      if (($mode eq 'auto' && $email_configured) || $mode eq 'legacy-sendmail') {
> @@ -1048,7 +1065,9 @@ sub exec_backup_task {
>          -d $task->{tmpdir}
>              || die "unable to create temporary directory '$task->{tmpdir}'";
>  
> -        $logfd = IO::File->new(">$tmplog")
> +        # keep the log of earlier attempts of this job
> +        my $logmode = defined($task->{tmplog}) ? '>>' : '>';

meh, not a fan. rather would keep this one log per job and add a retry suffix
there. Say "-r$task->{retries}.log" (or whatever file ending this uses).

Else these get very long and hard to browse, harder than they are now.

> +        $logfd = IO::File->new("${logmode}${tmplog}")
>              || die "unable to create log file '$tmplog'";
>  
>          $task->{dumpdir} = $opts->{dumpdir};
> @@ -1216,6 +1235,9 @@ sub exec_backup_task {
>              debugmsg('info', "archive file size: $cs", $logfd);
>          }
>  
> +        # The backup exists from here on, later failures must not trigger a retry
> +        $task->{committed} = 1;
> +
>          # Mark as protected before pruning.
>          if (my $storeid = $opts->{storage}) {
>              my $volname =
> @@ -1445,14 +1467,193 @@ sub exec_backup {
>  
>      my $starttime = time();
>      my $errcount = 0;
> +    my $retried_count = 0;
> +
> +    my $error_behavior = $opts->{'error-behavior'} // 'ignore';
> +    if ($opts->{stdout} && $error_behavior ne 'ignore') {
> +        log_warn(
> +            "ignoring 'error-behavior' from vzdump.conf - retries are not possible with 'stdout'");
> +        $error_behavior = 'ignore';
> +    }
> +
> +    my $max_retries = $opts->{'max-retries'} // 5;
> +    my $min_between_retry = 30;
> +    my $max_wait = 60 * 30;
> +
> +    my $get_backoff = sub {
> +        my ($retry) = @_;
> +        return min($min_between_retry * 2**($retry - 1), $max_wait);

fwiw this still calculates 2^X even if we're clearly out of the window then for one with
a very high max-retries count:

Ad this whole code hunk here bloats exec_backup quite a bit up, and it's not exactly
small to begin width.. would be nice to either move some stuff out or avoid it in the
first place if simple or - a bigger rework is out of scope here.

> +    };
> +
> +    my $log_committed_failure = sub {
> +        my ($vmid) = @_;
> +        debugmsg(
> +            'err',
> +            "Backup of guest $vmid was created, but a later step failed - not retrying",
> +            undef,
> +            1,
> +        );
> +    };
> +
> +    my $log_retried_success = sub {
> +        my ($vmid, $retries) = @_;
> +        log_warn("Backup of guest $vmid succeeded after " . retries_text($retries));
> +        $retried_count++;
> +    };
> +
>      eval {
>  
>          $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'};
>          $self->run_hook_script('job-start', undef, $job_start_fd);
>  
> +        my $failed_tasks = {};
> +
>          foreach my $task (@$tasklist) {
> +            my $vmid = $task->{vmid};
> +            my $initial_task = {%$task};
> +
> +            $task->{retries} = 0;
>              $self->exec_backup_task($task);
> -            $errcount += 1 if $task->{state} ne 'ok';
> +
> +            next if $task->{state} eq 'ok';
> +
> +            if ($error_behavior ne 'ignore' && $task->{committed}) {
> +                $log_committed_failure->($vmid);
> +                $errcount++;
> +                next;
> +            }
> +
> +            if ($error_behavior eq 'retry-immediately') {
> +                for my $retry (1 .. $max_retries) {
> +                    my $backoff = $get_backoff->($retry);
> +
> +                    debugmsg(
> +                        'info',
> +                        "Retrying backup for guest $vmid (retry $retry/$max_retries)"
> +                            . " after ${backoff}s delay...",
> +                        undef,
> +                        1,
> +                    );
> +                    sleep($backoff);

this can be interrupted before we $backoff duration has gone by.
Normally one does something like (untested and not saying its perfectly
right for this usecase here as proposed):

my $current = time();
my $target = $current + $backoff;
while (sleep($target - $current)) {
    die "failed to wait for timeout - $!\n" if $! && !$!{EINTR};
    $current = time();
    last if $current >= $target;
}

bit overkill naturally, one can also just not care for simpler cases
(also not saying this is such one, seems borderline) but then comment
that.


> +
> +                    %$task = (%$initial_task, retries => $retry, tmplog => $task->{tmplog});
> +                    $self->exec_backup_task($task);
> +
> +                    if ($task->{state} eq 'ok') {
> +                        $log_retried_success->($vmid, $retry);
> +                        last;
> +                    }
> +                    last if $task->{committed};
> +                }
> +
> +            } elsif ($error_behavior eq 'retry') {
> +                $failed_tasks->{$vmid} = {
> +                    task => $task,
> +                    initial_task => $initial_task,
> +                    fail_time => time(),
> +                    retries => 0,
> +                };
> +                next;
> +            }
> +
> +            if ($task->{state} ne 'ok') {
> +                if ($error_behavior eq 'retry-immediately') {
> +                    if ($task->{committed}) {
> +                        $log_committed_failure->($vmid);
> +                    } else {
> +                        debugmsg(
> +                            'err',
> +                            "Guest $vmid failed after " . retries_text($max_retries),
> +                            undef,
> +                            1,
> +                        );
> +                    }
> +                }
> +                $errcount++;
> +            }
> +        }
> +
> +        if ($error_behavior eq 'retry') {

hmm, I think we probably do need to rework whit all a bit to prepare it for doing the
retires without having to bloat this up to zeppelin size class (to exaggerate).

With a modularizing this first retrying should come much more natural.

> +            while (%$failed_tasks) {
> +                my $now = time();
> +                my $min_sleep = $max_wait;
> +                my $ready_to_run = [];
> +
> +                my @sorted_vmids = sort {
> +                    $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time}
> +                        || $a <=> $b
> +                } keys %$failed_tasks;
> +
> +                for my $vmid (@sorted_vmids) {
> +                    my $fail_info = $failed_tasks->{$vmid};
> +
> +                    if ($fail_info->{retries} >= $max_retries) {
> +                        debugmsg(
> +                            'err',
> +                            "Guest $vmid failed after " . retries_text($max_retries),
> +                            undef,
> +                            1,
> +                        );
> +                        $errcount++;
> +                        delete $failed_tasks->{$vmid};
> +                        next;
> +                    }
> +
> +                    my $delay = $get_backoff->($fail_info->{retries} + 1);
> +                    my $target_time = $fail_info->{fail_time} + $delay;
> +                    my $wait_time = $target_time - $now;
> +
> +                    if ($wait_time > 0) {
> +                        $min_sleep = $wait_time if $wait_time < $min_sleep;
> +                    } else {
> +                        push @$ready_to_run, $vmid;
> +                    }
> +                }
> +
> +                if (scalar(@$ready_to_run) == 0 && %$failed_tasks) {
> +                    debugmsg(
> +                        'info',
> +                        "Waiting $min_sleep seconds before next backup retry...",
> +                        undef,
> +                        1,
> +                    );
> +                    sleep($min_sleep);
> +                    next;
> +                }
> +
> +                for my $vmid (@$ready_to_run) {
> +                    my $fail_info = $failed_tasks->{$vmid};
> +
> +                    $fail_info->{retries}++;
> +                    debugmsg(
> +                        'info',
> +                        "Retrying backup for guest $vmid"
> +                            . " (retry $fail_info->{retries}/$max_retries)",
> +                        undef,
> +                        1,
> +                    );
> +
> +                    my $task = $fail_info->{task};
> +                    my $tmplog = $task->{tmplog};
> +                    %$task = (
> +                        %{ $fail_info->{initial_task} },
> +                        retries => $fail_info->{retries},
> +                        tmplog => $tmplog,
> +                    );
> +                    $self->exec_backup_task($task);
> +
> +                    if ($task->{state} eq 'ok') {
> +                        $log_retried_success->($vmid, $fail_info->{retries});
> +                        delete $failed_tasks->{$vmid};
> +                    } elsif ($task->{committed}) {
> +                        $log_committed_failure->($vmid);
> +                        $errcount++;
> +                        delete $failed_tasks->{$vmid};
> +                    } else {
> +                        $fail_info->{fail_time} = time();
> +                    }
> +                }
> +            }
>          }
>  
>          $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'};
> @@ -1472,6 +1673,13 @@ sub exec_backup {
>      } else {
>          if ($errcount) {
>              debugmsg('info', "Backup job finished with errors", undef, 1);
> +        } elsif ($retried_count) {
> +            debugmsg(
> +                'info',
> +                "Backup job finished with warnings - $retried_count guest(s) needed retries",
> +                undef,
> +                1,
> +            );
>          } else {
>              debugmsg('info', "Backup job finished successfully", undef, 1);
>          }
> @@ -1547,6 +1755,9 @@ sub verify_vzdump_parameters {
>      raise_param_exc({ pool => "option conflicts with option 'vmid'" })
>          if $param->{pool} && $param->{vmid};
>  
> +    raise_param_exc({ 'error-behavior' => "retries are not possible with option 'stdout'" })
> +        if $param->{stdout} && ($param->{'error-behavior'} // 'ignore') ne 'ignore';
> +
>      $parse_prune_backups->($param, 'CLI parameters');
>      parse_fleecing($param);
>      parse_performance($param);





  reply	other threads:[~2026-11-21 21:42 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-21 21:42   ` Thomas Lamprecht [this message]
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=53857bec-e147-4911-91ea-b61553194a52@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=j.klocker@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal