all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies
@ 2026-11-20 16:05 Jakob Klocker
  2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Jakob Klocker @ 2026-11-20 16:05 UTC (permalink / raw)
  To: pve-devel

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed up to a
  configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.

Since a guest can now go through backup_init() and backup_cleanup()
several times within one job, the backup provider API documentation
and the directory example provider are updated accordingly.

The mobile UI does not offer the retry options yet. Manual backups
started there use the node defaults from /etc/vzdump.conf. Adding the
fields there could be done as a follow-up.

Testing info:
To simulate a transient failure, lock a guest before starting the
backup and unlock it while the job waits for the retry:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Tested: no retries with 'ignore', successful retry with 'retry' and
'retry-immediately' (including ordering and the warning status),
exhausted retries with backoff, no retry after a failing backup-end
hook, and the --stdout cases (explicit retry strategy rejected,
max-retries alone accepted, retry strategy from vzdump.conf ignored
with a warning).

Changes since v3:
- backup provider API: document that backup_init() and backup_cleanup()
  can be called more than once for the same guest within a job
- backup provider example: reset per-guest state in backup_init(), so
  a retry does not inherit the 'failed' flag or the backing image of
  the failed attempt (preceded by a perltidy-only commit)
- clarify in the commit message and docs that the notification subject
  shows the number of retried guests, and the guest table the retries
  per guest
- cover letter: note the missing retry options in the mobile UI

Changes since v2 (thanks @Jonas):
- don't retry guests whose backup was already created (e.g. failing
  backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
  one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
  for retried guests
- show the number of retried guests in the notification subject and
  each guest's retry count in the guest table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
  description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
  and UI defaults

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-storage-plugin-examples:

Jakob Klocker (2):
  backup provider example: run proxmox-perltidy
  backup provider example: reset per-guest state in backup_init

 .../BackupProvider/Plugin/DirectoryExample.pm | 633 +++++++++---------
 1 file changed, 328 insertions(+), 305 deletions(-)


pve-storage:

Jakob Klocker (1):
  backup provider api: document repeated backup_init() calls on retries

 src/PVE/BackupProvider/Plugin/Base.pm | 12 ++++++++++++
 1 file changed, 12 insertions(+)


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 219 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 +++++
 www/manager6/window/Backup.js               |  56 ++++-
 4 files changed, 323 insertions(+), 6 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)


Summary over all repositories:
  8 files changed, 747 insertions(+), 311 deletions(-)

-- 
Generated by murpp 0.12.0




^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-20 16:07 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 16:05 [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal