From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 6194A1FF0E4 for ; Sat, 21 Nov 2026 22:42:59 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D7F572153C; Sat, 21 Nov 2026 22:42:56 +0100 (CET) Message-ID: <53857bec-e147-4911-91ea-b61553194a52@proxmox.com> Date: Sat, 21 Nov 2026 22:42:50 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups To: Jakob Klocker , pve-devel@lists.proxmox.com References: <20261120160510.311796-1-j.klocker@proxmox.com> <20261120160510.311796-6-j.klocker@proxmox.com> Content-Language: en-US From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20261120160510.311796-6-j.klocker@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795297370963 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.146 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PR2I6VVUT452MO4CUQYYKVNCKA7JAHU2 X-Message-ID-Hash: PR2I6VVUT452MO4CUQYYKVNCKA7JAHU2 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 20.11.26 um 17:07 schrieb Jakob Klocker: > Add the `error-behavior` and `max-retries` options to automatically > retry failed guest backups, which helps with transient failures. > > Supported retry strategies: > - `ignore`: Preserves current behavior (no retries, default). > - `retry-immediately`: Retries the guest inline after failure. > - `retry`: Defers failed guests until all other guests have been > processed, then retries them in the order they failed. > > A guest is retried at most `max-retries` times. Between attempts, the > delay starts at 30 seconds and doubles with each retry, capped at 30 > minutes as suggested on bugzilla [0]. If a guest still fails after the > last retry, it is counted as an error and the job finishes with errors. > If a retry succeeds, the task finishes with a warning and the > notification is sent with severity `warning`. Its subject shows how > many guests needed retries, and the guest table shows how many retries > each of them needed. > > Guests whose archive was already created (e.g. failing backup-end hook) > are not retried, to avoid duplicate archives and repeated pruning. > > Retries are not possible with --stdout, as a retry would corrupt the > stream. Setting `error-behavior` to a retry strategy together with > --stdout is rejected, and a retry strategy from /etc/vzdump.conf is > ignored with a warning. > > The task log is kept across retries and contains all attempts. > > [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5 > Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464 > > Signed-off-by: Jakob Klocker > --- > PVE/VZDump.pm | 219 +++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 215 insertions(+), 4 deletions(-) > > diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm > index d2861f9e..d0f98a13 100644 > --- a/PVE/VZDump.pm > +++ b/PVE/VZDump.pm > @@ -10,6 +10,7 @@ use File::Path; > use IO::File; > use IO::Select; > use IPC::Open3; > +use List::Util qw(min); > use POSIX qw(strftime); > use Time::Local; > > @@ -20,6 +21,7 @@ use PVE::HA::Config; > use PVE::HA::Env::PVE2; > use PVE::JSONSchema qw(get_standard_option); > use PVE::Notify; > +use PVE::RESTEnvironment qw(log_warn); debugmsg already calls this when passing 'warn' as level and also has some other special handling for how vzdump is used, in any case this should not be required here. > use PVE::RPCEnvironment; > use PVE::Storage; > use PVE::VZDump::Common; > @@ -395,6 +397,11 @@ my sub read_backup_task_logs { > return $task_logs; > } > > +my sub retries_text { > + my ($count) = @_; > + return $count == 1 ? '1 retry' : "$count retries"; > +} I don't want to add such a helper for every plural text there might be... Iff, then check if we can use pre-existing ones or do at least a generic one, like: sub plural($count, $singular, $plural) { return $count == 1 ? $singular : $plural; } Else this is gonna end in bloat. > + > my sub build_guest_table { > my ($task_list) = @_; > > @@ -436,11 +443,18 @@ my sub build_guest_table { > my $successful = $task->{state} eq 'ok'; > my $size = $successful ? $task->{size} : 0; > my $filename = $successful ? $task->{target} : undef; > + > + my $status = $task->{state}; > + my $retries = $task->{retries}; could be my ($status, $retries) = $task->{'state', 'retries'}; also $task->{'state'} is used a bit above already at my $successful. but tbh. not a big fan of such "useless" (i.e. 1:1 reference) intermediate variables for straight forward single-level hash element access. They just add complexity as one needs to check where they are set while not winning much. > + if ($successful && $retries) { > + $status .= " (after " . retries_text($retries) . ")"; > + } > + > push @{ $table->{data} }, > { > "vmid" => int($task->{vmid}), > "name" => $task->{hostname}, > - "status" => $task->{state}, > + "status" => $status, > "time" => int($task->{backuptime}), > "size" => int($size), > "filename" => $filename, > @@ -502,8 +516,11 @@ sub send_notification { > my ($error_count, $total_size) = aggregate_task_statistics($tasklist); > > my $failed = ($error_count || $err); > + my $retried_count = grep { $_->{state} eq 'ok' && $_->{retries} } @$tasklist; > > my $status_text = $failed ? 'backup failed' : 'backup successful'; > + $status_text .= " ($retried_count guest(s) needed retries)" > + if !$failed && $retried_count; here now plural difference? would do this consistently if we do. > > if ($err) { > if ($err =~ /\n/) { > @@ -540,7 +557,7 @@ sub send_notification { > # Add backup-job metadata field in case this is a backup job. > $fields->{'job-id'} = $job_id if $job_id; > > - my $severity = $failed ? "error" : "info"; > + my $severity = $failed ? "error" : $retried_count ? "warning" : "info"; > my $email_configured = $mailto && scalar(@$mailto); > > if (($mode eq 'auto' && $email_configured) || $mode eq 'legacy-sendmail') { > @@ -1048,7 +1065,9 @@ sub exec_backup_task { > -d $task->{tmpdir} > || die "unable to create temporary directory '$task->{tmpdir}'"; > > - $logfd = IO::File->new(">$tmplog") > + # keep the log of earlier attempts of this job > + my $logmode = defined($task->{tmplog}) ? '>>' : '>'; meh, not a fan. rather would keep this one log per job and add a retry suffix there. Say "-r$task->{retries}.log" (or whatever file ending this uses). Else these get very long and hard to browse, harder than they are now. > + $logfd = IO::File->new("${logmode}${tmplog}") > || die "unable to create log file '$tmplog'"; > > $task->{dumpdir} = $opts->{dumpdir}; > @@ -1216,6 +1235,9 @@ sub exec_backup_task { > debugmsg('info', "archive file size: $cs", $logfd); > } > > + # The backup exists from here on, later failures must not trigger a retry > + $task->{committed} = 1; > + > # Mark as protected before pruning. > if (my $storeid = $opts->{storage}) { > my $volname = > @@ -1445,14 +1467,193 @@ sub exec_backup { > > my $starttime = time(); > my $errcount = 0; > + my $retried_count = 0; > + > + my $error_behavior = $opts->{'error-behavior'} // 'ignore'; > + if ($opts->{stdout} && $error_behavior ne 'ignore') { > + log_warn( > + "ignoring 'error-behavior' from vzdump.conf - retries are not possible with 'stdout'"); > + $error_behavior = 'ignore'; > + } > + > + my $max_retries = $opts->{'max-retries'} // 5; > + my $min_between_retry = 30; > + my $max_wait = 60 * 30; > + > + my $get_backoff = sub { > + my ($retry) = @_; > + return min($min_between_retry * 2**($retry - 1), $max_wait); fwiw this still calculates 2^X even if we're clearly out of the window then for one with a very high max-retries count: Ad this whole code hunk here bloats exec_backup quite a bit up, and it's not exactly small to begin width.. would be nice to either move some stuff out or avoid it in the first place if simple or - a bigger rework is out of scope here. > + }; > + > + my $log_committed_failure = sub { > + my ($vmid) = @_; > + debugmsg( > + 'err', > + "Backup of guest $vmid was created, but a later step failed - not retrying", > + undef, > + 1, > + ); > + }; > + > + my $log_retried_success = sub { > + my ($vmid, $retries) = @_; > + log_warn("Backup of guest $vmid succeeded after " . retries_text($retries)); > + $retried_count++; > + }; > + > eval { > > $self->{'backup-provider'}->job_init($starttime) if $self->{'backup-provider'}; > $self->run_hook_script('job-start', undef, $job_start_fd); > > + my $failed_tasks = {}; > + > foreach my $task (@$tasklist) { > + my $vmid = $task->{vmid}; > + my $initial_task = {%$task}; > + > + $task->{retries} = 0; > $self->exec_backup_task($task); > - $errcount += 1 if $task->{state} ne 'ok'; > + > + next if $task->{state} eq 'ok'; > + > + if ($error_behavior ne 'ignore' && $task->{committed}) { > + $log_committed_failure->($vmid); > + $errcount++; > + next; > + } > + > + if ($error_behavior eq 'retry-immediately') { > + for my $retry (1 .. $max_retries) { > + my $backoff = $get_backoff->($retry); > + > + debugmsg( > + 'info', > + "Retrying backup for guest $vmid (retry $retry/$max_retries)" > + . " after ${backoff}s delay...", > + undef, > + 1, > + ); > + sleep($backoff); this can be interrupted before we $backoff duration has gone by. Normally one does something like (untested and not saying its perfectly right for this usecase here as proposed): my $current = time(); my $target = $current + $backoff; while (sleep($target - $current)) { die "failed to wait for timeout - $!\n" if $! && !$!{EINTR}; $current = time(); last if $current >= $target; } bit overkill naturally, one can also just not care for simpler cases (also not saying this is such one, seems borderline) but then comment that. > + > + %$task = (%$initial_task, retries => $retry, tmplog => $task->{tmplog}); > + $self->exec_backup_task($task); > + > + if ($task->{state} eq 'ok') { > + $log_retried_success->($vmid, $retry); > + last; > + } > + last if $task->{committed}; > + } > + > + } elsif ($error_behavior eq 'retry') { > + $failed_tasks->{$vmid} = { > + task => $task, > + initial_task => $initial_task, > + fail_time => time(), > + retries => 0, > + }; > + next; > + } > + > + if ($task->{state} ne 'ok') { > + if ($error_behavior eq 'retry-immediately') { > + if ($task->{committed}) { > + $log_committed_failure->($vmid); > + } else { > + debugmsg( > + 'err', > + "Guest $vmid failed after " . retries_text($max_retries), > + undef, > + 1, > + ); > + } > + } > + $errcount++; > + } > + } > + > + if ($error_behavior eq 'retry') { hmm, I think we probably do need to rework whit all a bit to prepare it for doing the retires without having to bloat this up to zeppelin size class (to exaggerate). With a modularizing this first retrying should come much more natural. > + while (%$failed_tasks) { > + my $now = time(); > + my $min_sleep = $max_wait; > + my $ready_to_run = []; > + > + my @sorted_vmids = sort { > + $failed_tasks->{$a}->{fail_time} <=> $failed_tasks->{$b}->{fail_time} > + || $a <=> $b > + } keys %$failed_tasks; > + > + for my $vmid (@sorted_vmids) { > + my $fail_info = $failed_tasks->{$vmid}; > + > + if ($fail_info->{retries} >= $max_retries) { > + debugmsg( > + 'err', > + "Guest $vmid failed after " . retries_text($max_retries), > + undef, > + 1, > + ); > + $errcount++; > + delete $failed_tasks->{$vmid}; > + next; > + } > + > + my $delay = $get_backoff->($fail_info->{retries} + 1); > + my $target_time = $fail_info->{fail_time} + $delay; > + my $wait_time = $target_time - $now; > + > + if ($wait_time > 0) { > + $min_sleep = $wait_time if $wait_time < $min_sleep; > + } else { > + push @$ready_to_run, $vmid; > + } > + } > + > + if (scalar(@$ready_to_run) == 0 && %$failed_tasks) { > + debugmsg( > + 'info', > + "Waiting $min_sleep seconds before next backup retry...", > + undef, > + 1, > + ); > + sleep($min_sleep); > + next; > + } > + > + for my $vmid (@$ready_to_run) { > + my $fail_info = $failed_tasks->{$vmid}; > + > + $fail_info->{retries}++; > + debugmsg( > + 'info', > + "Retrying backup for guest $vmid" > + . " (retry $fail_info->{retries}/$max_retries)", > + undef, > + 1, > + ); > + > + my $task = $fail_info->{task}; > + my $tmplog = $task->{tmplog}; > + %$task = ( > + %{ $fail_info->{initial_task} }, > + retries => $fail_info->{retries}, > + tmplog => $tmplog, > + ); > + $self->exec_backup_task($task); > + > + if ($task->{state} eq 'ok') { > + $log_retried_success->($vmid, $fail_info->{retries}); > + delete $failed_tasks->{$vmid}; > + } elsif ($task->{committed}) { > + $log_committed_failure->($vmid); > + $errcount++; > + delete $failed_tasks->{$vmid}; > + } else { > + $fail_info->{fail_time} = time(); > + } > + } > + } > } > > $self->{'backup-provider'}->job_cleanup() if $self->{'backup-provider'}; > @@ -1472,6 +1673,13 @@ sub exec_backup { > } else { > if ($errcount) { > debugmsg('info', "Backup job finished with errors", undef, 1); > + } elsif ($retried_count) { > + debugmsg( > + 'info', > + "Backup job finished with warnings - $retried_count guest(s) needed retries", > + undef, > + 1, > + ); > } else { > debugmsg('info', "Backup job finished successfully", undef, 1); > } > @@ -1547,6 +1755,9 @@ sub verify_vzdump_parameters { > raise_param_exc({ pool => "option conflicts with option 'vmid'" }) > if $param->{pool} && $param->{vmid}; > > + raise_param_exc({ 'error-behavior' => "retries are not possible with option 'stdout'" }) > + if $param->{stdout} && ($param->{'error-behavior'} // 'ignore') ne 'ignore'; > + > $parse_prune_backups->($param, 'CLI parameters'); > parse_fleecing($param); > parse_performance($param);