From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 23B3F1FF0BC for ; Mon, 16 Nov 2026 11:14:29 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7A5D82171A; Mon, 16 Nov 2026 11:14:24 +0100 (CET) From: Jakob Klocker To: pve-devel@lists.proxmox.com Subject: [PATCH pve-docs v3 4/4] vzdump: document backup retry options Date: Mon, 16 Nov 2026 11:13:44 +0100 Message-ID: <20261116101344.140523-5-j.klocker@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261116101344.140523-1-j.klocker@proxmox.com> References: <20261116101344.140523-1-j.klocker@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794824059771 X-SPAM-LEVEL: Spam detection results: 0 AWL 1.280 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: OK363OPLT4HLJTN3B2HXB3JKP35UFMOM X-Message-ID-Hash: OK363OPLT4HLJTN3B2HXB3JKP35UFMOM X-MailFrom: j.klocker@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Describe the new `error-behavior` and `max-retries` options, the exponential backoff between attempts, and which failures are retried. Signed-off-by: Jakob Klocker --- vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 44db027..856d3d3 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -308,6 +308,71 @@ the `performance` setting, `max-workers` (affects VM backups only) and `pbs-entries-max` (affects container backups only). See the xref:vzdump_configuration[configuration options] for details. +[[vzdump_retry]] +Backup Retries +-------------- + +The backup of a single guest can fail because of a temporary problem, for +example when the guest is locked by another operation, or when the connection +to the backup server is interrupted during the backup. To avoid a failed backup +in such cases, the `error-behavior` option allows a job to retry failed guest +backups automatically: + +`ignore` :: +Does not retry failed guests. This is the default. + +`retry` :: +Defers failed guests until all other guests have been processed and then +retries them in the order in which they failed. + +`retry-immediately` :: +Retries a failed guest right away, before continuing with the next guest. + +The `max-retries` option (1 to 10, default 5) limits how often a guest is +retried. Between attempts, the job uses exponential backoff: the delay starts +at 30 seconds, doubles with every retry and is capped at 30 minutes. With +`retry`, the delay is counted from the last failed attempt of that guest, so +other guests can be retried in the meantime. + +If a retry succeeds, the guest does not count as an error, but the task +finishes with a warning and the notification is sent with severity `warning` +and shows how many retries were needed. This way, recurring problems stay +visible even if all backups eventually succeed. If the last retry fails as +well, the job finishes with errors. The task log of the guest contains all +attempts. + +For example, to back up three guests and retry each failed guest at most three +times at the end of the job, run: + + # vzdump 101 102 103 --error-behavior retry --max-retries 3 + +Both options can also be set as node-wide defaults in `/etc/vzdump.conf`, see +xref:vzdump_configuration[configuration]. + +In the web interface, the retry strategy for backup jobs can be set in the +'Advanced' tab of the backup job window. New backup jobs preselect `retry`, +existing jobs keep their configuration. For a manual backup, the retry strategy +can be chosen directly in the backup window ('Backup now'), which preselects the +node's default from `/etc/vzdump.conf`. It only offers `retry` and `ignore`, +as both retry strategies behave the same for a single guest. + +Only failures that happen before the backup archive of a guest has been created +are retried. If the archive was created, but a later step failed, for example a +failing `backup-end` hook script, the guest is not retried. This avoids +creating multiple backups of the same guest and pruning older backups +repeatedly. Errors that prevent the whole job from starting, for example a +backup storage that cannot be activated, are not retried either. + +Retries are not possible when writing the backup to standard output with +`--stdout`, as a retry would append a second archive to the data that was +already written. Setting `error-behavior` to a retry strategy together with +`--stdout` is rejected, and a retry strategy from `/etc/vzdump.conf` is +ignored with a warning. + +NOTE: Retries can make a backup job take considerably longer. With the default +of 5 retries, the delays alone add up to more than 15 minutes for a guest that +keeps failing, and about 2.5 hours with the maximum of 10 retries. + [[vzdump_retention]] Backup Retention ---------------- -- 2.47.3