From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 65BCA1FF0AF for ; Fri, 20 Nov 2026 17:07:02 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4BDA021837; Fri, 20 Nov 2026 17:06:23 +0100 (CET) From: Jakob Klocker To: pve-devel@lists.proxmox.com Subject: [PATCH docs/guest-common/manager/storage/storage-plugin-examples v4 0/7] fix #6464: backup: add backup retry strategies Date: Fri, 20 Nov 2026 17:05:03 +0100 Message-ID: <20261120160510.311796-1-j.klocker@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795190776316 X-SPAM-LEVEL: Spam detection results: 0 AWL 1.137 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RAMKRSOG56DZA6SQUK6MD252NRDGL5PZ X-Message-ID-Hash: RAMKRSOG56DZA6SQUK6MD252NRDGL5PZ X-MailFrom: j.klocker@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series introduces automatic backup retries for guest backups in vzdump to better handle transient failures. As suggested by @Thomas [0], three strategies were added: - `retry`: Defers failed guests until all other guests have been processed, then retries them in the order they failed up to a configurable maximum retry count. - `retry-immediately`: Immediately retries failed guests up to a configurable maximum retry count. - `ignore`: Retains the existing behavior by ignoring failures. The number of retries is set with the new `max-retries` option (1-10, default 5). Between attempts, a backoff delay is applied that starts at 30 seconds, doubles with each retry and is capped at 30 minutes. For `retry`, the delay is measured from the last failure of that guest. The interval and cap are intentionally not configurable for now. If a guest succeeds on a retry, no error is counted, but the task finishes with a warning and the notification is sent with severity `warning`. If it still fails after the last retry, the job finishes with errors as before. The task log of a guest contains all attempts of the current job. Since a guest can now go through backup_init() and backup_cleanup() several times within one job, the backup provider API documentation and the directory example provider are updated accordingly. The mobile UI does not offer the retry options yet. Manual backups started there use the node defaults from /etc/vzdump.conf. Adding the fields there could be done as a follow-up. Testing info: To simulate a transient failure, lock a guest before starting the backup and unlock it while the job waits for the retry: `qm set -lock backup` `qm unlock ` Tested: no retries with 'ignore', successful retry with 'retry' and 'retry-immediately' (including ordering and the warning status), exhausted retries with backoff, no retry after a failing backup-end hook, and the --stdout cases (explicit retry strategy rejected, max-retries alone accepted, retry strategy from vzdump.conf ignored with a warning). Changes since v3: - backup provider API: document that backup_init() and backup_cleanup() can be called more than once for the same guest within a job - backup provider example: reset per-guest state in backup_init(), so a retry does not inherit the 'failed' flag or the backing image of the failed attempt (preceded by a perltidy-only commit) - clarify in the commit message and docs that the notification subject shows the number of retried guests, and the guest table the retries per guest - cover letter: note the missing retry options in the mobile UI Changes since v2 (thanks @Jonas): - don't retry guests whose backup was already created (e.g. failing backup-end hook) - with --stdout, reject an explicitly set retry strategy and ignore one inherited from vzdump.conf with a warning - don't append the task log to a stale log of a previous job - finish the task with warnings and notify with severity 'warning' for retried guests - show the number of retried guests in the notification subject and each guest's retry count in the guest table - UI: only preselect `retry` for new jobs - UI: don't store max-retries unless set, show schema defaults in description - UI: use node defaults from vzdump.conf in the manual backup window - docs: document non-retried failures, --stdout restriction, warnings and UI defaults Changes since v1 (thanks @Shan): - add documentation for the new retry options to pve-docs - fix typo in pve-manager commit message - drop the `advanced` flag from the retry fields in the manual backup window, which has no advanced section [0] https://bugzilla.proxmox.com/show_bug.cgi?id=6464#c5 Link: https://bugzilla.proxmox.com/show_bug.cgi?id=6464 pve-storage-plugin-examples: Jakob Klocker (2): backup provider example: run proxmox-perltidy backup provider example: reset per-guest state in backup_init .../BackupProvider/Plugin/DirectoryExample.pm | 633 +++++++++--------- 1 file changed, 328 insertions(+), 305 deletions(-) pve-storage: Jakob Klocker (1): backup provider api: document repeated backup_init() calls on retries src/PVE/BackupProvider/Plugin/Base.pm | 12 ++++++++++++ 1 file changed, 12 insertions(+) pve-guest-common: Jakob Klocker (1): vzdump: add error-behavior and max-retries options src/PVE/VZDump/Common.pm | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) pve-manager: Jakob Klocker (2): fix #6464: vzdump: add retry strategies for failed guest backups ui: backup: add error behavior and retry options PVE/VZDump.pm | 219 +++++++++++++++++++- www/manager6/dc/Backup.js | 2 +- www/manager6/panel/BackupAdvancedOptions.js | 52 +++++ www/manager6/window/Backup.js | 56 ++++- 4 files changed, 323 insertions(+), 6 deletions(-) pve-docs: Jakob Klocker (1): vzdump: document backup retry options vzdump.adoc | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) Summary over all repositories: 8 files changed, 747 insertions(+), 311 deletions(-) -- Generated by murpp 0.12.0