public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Jakob Klocker <j.klocker@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies
Date: Mon, 16 Nov 2026 11:13:40 +0100	[thread overview]
Message-ID: <20261116101344.140523-1-j.klocker@proxmox.com> (raw)

This series introduces automatic backup retries for guest backups in
vzdump to better handle transient failures.

As suggested by @Thomas [0], three strategies were added:
- `retry`: Defers failed guests until all other guests have been
  processed, then retries them in the order they failed up to a
  configurable maximum retry count.
- `retry-immediately`: Immediately retries failed guests up to a
  configurable maximum retry count.
- `ignore`: Retains the existing behavior by ignoring failures.

The number of retries is set with the new `max-retries` option (1-10,
default 5). Between attempts, a backoff delay is applied that starts at
30 seconds, doubles with each retry and is capped at 30 minutes. For
`retry`, the delay is measured from the last failure of that guest. The
interval and cap are intentionally not configurable for now.

If a guest succeeds on a retry, no error is counted, but the task
finishes with a warning and the notification is sent with severity
`warning`. If it still fails after the last retry, the job finishes
with errors as before. The task log of a guest contains all attempts
of the current job.

Testing info:
When testing I'd suggest to fail the backup by locking and
unlocking the guest with following commands:
`qm set <VMID> -lock backup`
`qm unlock <VMID>`

Changes since v2 (thanks @Jonas)
- don't retry guests whose backup was already created (e.g. failing
  backup-end hook)
- with --stdout, reject an explicitly set retry strategy and ignore
  one inherited from vzdump.conf with a warning
- don't append the task log to a stale log of a previous job
- finish the task with warnings and notify with severity 'warning'
  for retried guests
- show the number of retries in the notification subject and guest
  table
- UI: only preselect `retry` for new jobs
- UI: don't store max-retries unless set, show schema defaults in
  description
- UI: use node defaults from vzdump.conf in the manual backup window
- docs: document non-retried failures, --stdout restriction, warnings
  and UI defaults

Changes since v1 (thanks @Shan):
- add documentation for the new retry options to pve-docs
- fix typo in pve-manager commit message
- drop the `advanced` flag from the retry fields in the manual backup
  window, which has no advanced section

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464


pve-guest-common:

Jakob Klocker (1):
  vzdump: add error-behavior and max-retries options

 src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)


pve-manager:

Jakob Klocker (2):
  fix #6464: vzdump: add retry strategies for failed guest backups
  ui: backup: add error behavior and retry options

 PVE/VZDump.pm                               | 219 +++++++++++++++++++-
 www/manager6/dc/Backup.js                   |   2 +-
 www/manager6/panel/BackupAdvancedOptions.js |  52 +++++
 www/manager6/window/Backup.js               |  56 ++++-
 4 files changed, 323 insertions(+), 6 deletions(-)


pve-docs:

Jakob Klocker (1):
  vzdump: document backup retry options

 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)


Summary over all repositories:
  6 files changed, 406 insertions(+), 6 deletions(-)

-- 
Generated by murpp 0.12.0




             reply	other threads:[~2026-11-16 10:14 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 10:13 Jakob Klocker [this message]
2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-docs v3 4/4] vzdump: document backup " Jakob Klocker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116101344.140523-1-j.klocker@proxmox.com \
    --to=j.klocker@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal