public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies
@ 2026-11-16 10:13 Jakob Klocker
  2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Jakob Klocker @ 2026-11-16 10:13 UTC (permalink / raw)
  To: pve-devel

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed up to a
  configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.

Testing info:
When testing I'd suggest to fail the backup by locking and
unlocking the guest with following commands:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Changes since v2 (thanks @Jonas)
- don't retry guests whose backup was already created (e.g. failing
  backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
  one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
  for retried guests
- show the number of retries in the notification subject and guest
  table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
  description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
  and UI defaults

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 219 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 +++++
 www/manager6/window/Backup.js               |  56 ++++-
 4 files changed, 323 insertions(+), 6 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)


Summary over all repositories:
  6 files changed, 406 insertions(+), 6 deletions(-)

-- 
Generated by murpp 0.12.0




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-16 10:15 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal