From: Jakob Klocker <j.klocker@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies
Date: Fri, 20 Nov 2026 17:05:03 +0100 [thread overview]
Message-ID: <20261120160510.311796-1-j.klocker@proxmox.com> (raw)
This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.
As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
processed, then retries them in the order they failed up to a
configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.
The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.
If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.
Since a guest can now go through backup_init() and backup_cleanup()
several times within one job, the backup provider API documentation
and the directory example provider are updated accordingly.
The mobile UI does not offer the retry options yet. Manual backups
started there use the node defaults from /etc/vzdump.conf. Adding the
fields there could be done as a follow-up.
Testing info:
To simulate a transient failure, lock a guest before starting the
backup and unlock it while the job waits for the retry:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`
Tested: no retries with 'ignore', successful retry with 'retry' and
'retry-immediately' (including ordering and the warning status),
exhausted retries with backoff, no retry after a failing backup-end
hook, and the --stdout cases (explicit retry strategy rejected,
max-retries alone accepted, retry strategy from vzdump.conf ignored
with a warning).
Changes since v3:
- backup provider API: document that backup_init() and backup_cleanup()
can be called more than once for the same guest within a job
- backup provider example: reset per-guest state in backup_init(), so
a retry does not inherit the 'failed' flag or the backing image of
the failed attempt (preceded by a perltidy-only commit)
- clarify in the commit message and docs that the notification subject
shows the number of retried guests, and the guest table the retries
per guest
- cover letter: note the missing retry options in the mobile UI
Changes since v2 (thanks @Jonas):
- don't retry guests whose backup was already created (e.g. failing
backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
for retried guests
- show the number of retried guests in the notification subject and
each guest's retry count in the guest table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
and UI defaults
Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
window, which has no advanced section
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464
pve-storage-plugin-examples:
Jakob Klocker (2):
backup provider example: run proxmox-perltidy
backup provider example: reset per-guest state in backup_init
.../BackupProvider/Plugin/DirectoryExample.pm | 633 +++++++++---------
1 file changed, 328 insertions(+), 305 deletions(-)
pve-storage:
Jakob Klocker (1):
backup provider api: document repeated backup_init() calls on retries
src/PVE/BackupProvider/Plugin/Base.pm | 12 ++++++++++++
1 file changed, 12 insertions(+)
pve-guest-common:
Jakob Klocker (1):
vzdump: add error-behavior and max-retries options
src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
pve-manager:
Jakob Klocker (2):
fix #6464: vzdump: add retry strategies for failed guest backups
ui: backup: add error behavior and retry options
PVE/VZDump.pm | 219 +++++++++++++++++++-
www/manager6/dc/Backup.js | 2 +-
www/manager6/panel/BackupAdvancedOptions.js | 52 +++++
www/manager6/window/Backup.js | 56 ++++-
4 files changed, 323 insertions(+), 6 deletions(-)
pve-docs:
Jakob Klocker (1):
vzdump: document backup retry options
vzdump.adoc | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
Summary over all repositories:
8 files changed, 747 insertions(+), 311 deletions(-)
--
Generated by murpp 0.12.0
next reply other threads:[~2026-11-20 16:07 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 16:05 Jakob Klocker [this message]
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 1/7] backup provider example: run proxmox-perltidy Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage-plugin-examples v4 2/7] backup provider example: reset per-guest state in backup_init Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-storage v4 3/7] backup provider api: document repeated backup_init() calls on retries Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-guest-common v4 4/7] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 5/7] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-manager v4 6/7] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-20 16:05 ` [PATCH pve-docs v4 7/7] vzdump: document backup " Jakob Klocker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120160510.311796-1-j.klocker@proxmox.com \
--to=j.klocker@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox