public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Jakob Klocker <j.klocker@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-docs v3 4/4] vzdump: document backup retry options
Date: Mon, 16 Nov 2026 11:13:44 +0100	[thread overview]
Message-ID: <20261116101344.140523-5-j.klocker@proxmox.com> (raw)
In-Reply-To: <20261116101344.140523-1-j.klocker@proxmox.com>

Describe the new `error-behavior` and `max-retries` options, the
exponential backoff between attempts, and which failures are retried.

Signed-off-by: Jakob Klocker <j.klocker@proxmox.com>
---
 vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/vzdump.adoc b/vzdump.adoc
index 44db027..856d3d3 100644
--- a/vzdump.adoc
+++ b/vzdump.adoc
@@ -308,6 +308,71 @@ the `performance` setting, `max-workers` (affects VM backups only) and
 `pbs-entries-max` (affects container backups only). See the
 xref:vzdump_configuration[configuration options] for details.
 
+[[vzdump_retry]]
+Backup Retries
+--------------
+
+The backup of a single guest can fail because of a temporary problem, for
+example when the guest is locked by another operation, or when the connection
+to the backup server is interrupted during the backup. To avoid a failed backup
+in such cases, the `error-behavior` option allows a job to retry failed guest
+backups automatically:
+
+`ignore` ::
+Does not retry failed guests. This is the default.
+
+`retry` ::
+Defers failed guests until all other guests have been processed and then
+retries them in the order in which they failed.
+
+`retry-immediately` ::
+Retries a failed guest right away, before continuing with the next guest.
+
+The `max-retries` option (1 to 10, default 5) limits how often a guest is
+retried. Between attempts, the job uses exponential backoff: the delay starts
+at 30 seconds, doubles with every retry and is capped at 30 minutes. With
+`retry`, the delay is counted from the last failed attempt of that guest, so
+other guests can be retried in the meantime.
+
+If a retry succeeds, the guest does not count as an error, but the task
+finishes with a warning and the notification is sent with severity `warning`
+and shows how many retries were needed. This way, recurring problems stay
+visible even if all backups eventually succeed. If the last retry fails as
+well, the job finishes with errors. The task log of the guest contains all
+attempts.
+
+For example, to back up three guests and retry each failed guest at most three
+times at the end of the job, run:
+
+ # vzdump 101 102 103 --error-behavior retry --max-retries 3
+
+Both options can also be set as node-wide defaults in `/etc/vzdump.conf`, see
+xref:vzdump_configuration[configuration].
+
+In the web interface, the retry strategy for backup jobs can be set in the
+'Advanced' tab of the backup job window. New backup jobs preselect `retry`,
+existing jobs keep their configuration. For a manual backup, the retry strategy
+can be chosen directly in the backup window ('Backup now'), which preselects the
+node's default from `/etc/vzdump.conf`. It only offers `retry` and `ignore`,
+as both retry strategies behave the same for a single guest.
+
+Only failures that happen before the backup archive of a guest has been created
+are retried. If the archive was created, but a later step failed, for example a
+failing `backup-end` hook script, the guest is not retried. This avoids
+creating multiple backups of the same guest and pruning older backups
+repeatedly. Errors that prevent the whole job from starting, for example a
+backup storage that cannot be activated, are not retried either.
+
+Retries are not possible when writing the backup to standard output with
+`--stdout`, as a retry would append a second archive to the data that was
+already written. Setting `error-behavior` to a retry strategy together with
+`--stdout` is rejected, and a retry strategy from `/etc/vzdump.conf` is
+ignored with a warning.
+
+NOTE: Retries can make a backup job take considerably longer. With the default
+of 5 retries, the delays alone add up to more than 15 minutes for a guest that
+keeps failing, and about 2.5 hours with the maximum of 10 retries.
+
 [[vzdump_retention]]
 Backup Retention
 ----------------
-- 
2.47.3





      parent reply	other threads:[~2026-11-16 10:14 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 10:13 [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-guest-common v3 1/4] vzdump: add error-behavior and max-retries options Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 2/4] fix #6464: vzdump: add retry strategies for failed guest backups Jakob Klocker
2026-11-16 10:13 ` [PATCH pve-manager v3 3/4] ui: backup: add error behavior and retry options Jakob Klocker
2026-11-16 10:13 ` Jakob Klocker [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116101344.140523-5-j.klocker@proxmox.com \
    --to=j.klocker@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal