From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 013E51FF0BC for ; Mon, 16 Nov 2026 11:14:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A6407217E6; Mon, 16 Nov 2026 11:14:25 +0100 (CET) From: Jakob Klocker To: pve-devel@lists.proxmox.com Subject: [PATCH docs/guest-common/manager v3 0/4] fix #6464: backup: add backup retry strategies Date: Mon, 16 Nov 2026 11:13:40 +0100 Message-ID: <20261116101344.140523-1-j.klocker@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794824059477 X-SPAM-LEVEL: Spam detection results: 0 AWL 1.289 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KK3DC54TDVM5BWYV4AOIXZ62JHFGGOXG X-Message-ID-Hash: KK3DC54TDVM5BWYV4AOIXZ62JHFGGOXG X-MailFrom: j.klocker@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series introduces automatic backup retries for guest backups in vzdump to better handle transient failures. As suggested by @Thomas [0], three strategies were added: - `retry`: Defers failed guests until all other guests have been processed, then retries them in the order they failed up to a configurable maximum retry count. - `retry-immediately`: Immediately retries failed guests up to a configurable maximum retry count. - `ignore`: Retains the existing behavior by ignoring failures. The number of retries is set with the new `max-retries` option (1-10, default 5). Between attempts, a backoff delay is applied that starts at 30 seconds, doubles with each retry and is capped at 30 minutes. For `retry`, the delay is measured from the last failure of that guest. The interval and cap are intentionally not configurable for now. If a guest succeeds on a retry, no error is counted, but the task finishes with a warning and the notification is sent with severity `warning`. If it still fails after the last retry, the job finishes with errors as before. The task log of a guest contains all attempts of the current job. Testing info: When testing I'd suggest to fail the backup by locking and unlocking the guest with following commands: `qm set -lock backup` `qm unlock ` Changes since v2 (thanks @Jonas) - don't retry guests whose backup was already created (e.g. failing backup-end hook) - with --stdout, reject an explicitly set retry strategy and ignore one inherited from vzdump.conf with a warning - don't append the task log to a stale log of a previous job - finish the task with warnings and notify with severity 'warning' for retried guests - show the number of retries in the notification subject and guest table - UI: only preselect `retry` for new jobs - UI: don't store max-retries unless set, show schema defaults in description - UI: use node defaults from vzdump.conf in the manual backup window - docs: document non-retried failures, --stdout restriction, warnings and UI defaults Changes since v1 (thanks @Shan): - add documentation for the new retry options to pve-docs - fix typo in pve-manager commit message - drop the `advanced` flag from the retry fields in the manual backup window, which has no advanced section [0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5 Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464 pve-guest-common: Jakob Klocker (1): vzdump: add error-behavior and max-retries options src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) pve-manager: Jakob Klocker (2): fix #6464: vzdump: add retry strategies for failed guest backups ui: backup: add error behavior and retry options PVE/VZDump.pm | 219 +++++++++++++++++++- www/manager6/dc/Backup.js | 2 +- www/manager6/panel/BackupAdvancedOptions.js | 52 +++++ www/manager6/window/Backup.js | 56 ++++- 4 files changed, 323 insertions(+), 6 deletions(-) pve-docs: Jakob Klocker (1): vzdump: document backup retry options vzdump.adoc | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) Summary over all repositories: 6 files changed, 406 insertions(+), 6 deletions(-) -- Generated by murpp 0.12.0