From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B5BB81FF0DF for ; Fri, 28 Aug 2026 15:31:38 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 167CE215EE; Fri, 28 Aug 2026 15:31:12 +0200 (CEST) From: Shannon Sterz To: pve-devel@lists.proxmox.com Subject: [RFC cluster/common/container/docs/installer/manager 00/21] add rudimentary host backup mechanism Date: Fri, 28 Aug 2026 15:30:09 +0200 Message-ID: <20260828133030.351140-1-s.sterz@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1787923824041 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.904 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: BLGRTWC2IMQV67B7CNSKTLCCRXBMNPX3 X-Message-ID-Hash: BLGRTWC2IMQV67B7CNSKTLCCRXBMNPX3 X-MailFrom: s.sterz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is still very early stages and rough around the edges. I am sending this now, because the rough mechanism seems to work well enough to start gathering some input. Further clean up will follow, I am mostly looking for feedback on whether this is the right direction to go. All changes outlined here could definitively benefit from more thorough testing. Especially with more complex and exotic setups. The goal of this series is to add a host backup mechanism for Proxmox VE. Such a mechanism should enable users to automate backups and help in a disaster recovery scenario. This is accomplished by backing up `/etc` and several bits of information of a running system via the pbs client. For restoring there are two options: - File restore for single config files. - Restoring via the installer from a clean slate (currently only intended for disaster recovery). For now this mostly works properly if the hardware is identical, but should be extended to be more forgiving (see open work). The patches for pmxcfs extend it to allow for live-backups without having to interrupt pmxcfs operations. Secondly patches for pve-manager add the ui, api endpoints and job setup for the basic backups. They allow for custom hook scripts and additional files that need to be backed up for more flexibility. Finally, the patches for the installer add a restore mode for the TUI installer to allow restoring when setting up a new system. How to Apply & Bump ------------------- * patches 02-03/18 implement the core live backup mechanism for pmxcfs and are for the patches in pve-manager * patch 04/18 to is required to be applied and the libpve-common-perl dependency needs to be bumped in pve-manager and pve-container before patches 05-06/18 can be applied Backup Consistency ------------------ Looking over options to improve backup consistency there were several discarded ideas I've had. A selection: - [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html): + Basic idea: Overlay the current root file system, read from the underlay and then merge back the overlay. Writes during the backup happen only on the overlay, so the view we get should be consistent. + Merging the overlaid changes back down would require an unmount operation ("Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed." [2]). This will likely interrupt the system and possibly cause inconsistencies. + Where do we put the overlay? * If we put it in RAM via a tmpfs, any write that happens during the backup may be easily lost if changes are not properly merged back down. * Putting it on another file system backed by a physical device either means we need to reserve a lot of space on setup or require additional hardware. + We probably would want to create the overlay on boot, because otherwise running processes may be left in an inconsistent state leading to I/O errors. For example, because the underlay they had open file handles to is now read-only or if they try to write to the underlay they break overlayfs requirements.). - [fsfreeze](https://manpages.debian.org/trixie/util-linux/fsfreeze.8.en.html): + We could try to sync out data, freeze the filesystem, sync remaining data and then resume. + Still interrupts currently running systems and could lead to issues. For example, if the backup fails without cleaning up the freeze, recovering from that is tricky. SSH nor Web UI can be used as establishing new connections via both are blocked until the filesystem is unfrozen. - [blksnap](https://github.com/veeam/blksnap?tab=readme-ov-file) by veeam: + Would require us to ship a kernel module by ourselves, though the repo has instructions on how to package it for Debian. + Last commit is from over 10 months ago, last submission to LKML seems to have happened in 2024 [3], though the repo mentions patches that were/are prepared for a v8. It seems that up-streaming efforts have been stalled [4]. For now I landed on doing snapshots for CoW filesystems (zfs,btrfs) and sqlite3's live backup mechanism. Users can try to improve consistency based hook scripts too (e.g. if LVM thick snapshots are an option). Open Work --------- There is still a lot that is missing here, amongst others: - Better Support for non-identical hardware restore: Currently the installer does not allow progressing if it can't restore the exact disk layout. This should cleaned up to allow properly overriding the root disk layout. There are also some pitfall when modifying the initial disk setup when it comes to restoring the storage.cfg. - Encryption: Currently the reinstall restoration mode does not work for encrypted backups at all. Inputting that information via the TUI was deemed to cumbersome. - Better testing: Currently this missing unit and integration tests. At least a basic set of tests should be implement for things like parsing out the installer state from an already installed system. - Documentation: This is currently lacking documentation more or less entirely. Mainly cause I did not get around to it yet. While I think most parts should be fairly intuitive to most of our users, things like `additional-files` and the `hooks` definitively deserver more explanations. cluster: Shannon Sterz (4): pmxcfs: status: fix formatting of parameters in checked_mkdir() pmxcfs: correctly log message when directory can't be created pmxcfs: add live backup capability pmxcfs: add ability to query backup progress src/PVE/Cluster.pm | 82 +++++++++++++++++++++- src/pmxcfs/cfs-ipc-ops.h | 4 ++ src/pmxcfs/cfs-utils.h | 1 + src/pmxcfs/database.c | 146 +++++++++++++++++++++++++++++++++++++++ src/pmxcfs/memdb.c | 42 +++++++++++ src/pmxcfs/memdb.h | 122 ++++++++++++++++++++++++++++++++ src/pmxcfs/server.c | 105 ++++++++++++++++++++++++++++ src/pmxcfs/status.c | 13 +++- src/pmxcfs/status.h | 2 + 9 files changed, 511 insertions(+), 6 deletions(-) common: Shannon Sterz (1): systemd: move parse_os_release() helper to PVE::Systemd src/PVE/Systemd.pm | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) container: Shannon Sterz (1): setup: use parse_os_release from PVE::Systemd src/PVE/LXC/Setup.pm | 35 ++--------------------------------- 1 file changed, 2 insertions(+), 33 deletions(-) manager: Shannon Sterz (7): jobs/api: add basic host backup job logic api: cluster: add endpoints for manage host backup jobs api: node: add endpoints for listing backups for a node api: host backup: include global, disk and network options for restore api: host backup: add warnings in case zfs snapdir is disabled ui: node: add panel to manage backups of a host ui: dc: add panel for managing host backup jobs PVE/API2/Cluster/HostBackup.pm | 355 +++++++++++++++++++ PVE/API2/Cluster/Jobs.pm | 10 +- PVE/API2/Cluster/Makefile | 1 + PVE/API2/HostBackup.pm | 559 ++++++++++++++++++++++++++++++ PVE/API2/Makefile | 1 + PVE/API2/Nodes.pm | 7 + PVE/HostBackupTools.pm | 452 ++++++++++++++++++++++++ PVE/Jobs.pm | 2 + PVE/Jobs/HostBackup.pm | 122 +++++++ PVE/Jobs/Makefile | 5 +- PVE/Makefile | 1 + www/manager6/Makefile | 2 + www/manager6/Utils.js | 1 + www/manager6/dc/Config.js | 6 + www/manager6/dc/HostBackupJobs.js | 360 +++++++++++++++++++ www/manager6/node/Config.js | 7 + www/manager6/node/HostBackup.js | 292 ++++++++++++++++ 17 files changed, 2180 insertions(+), 3 deletions(-) create mode 100644 PVE/API2/Cluster/HostBackup.pm create mode 100644 PVE/API2/HostBackup.pm create mode 100644 PVE/HostBackupTools.pm create mode 100644 PVE/Jobs/HostBackup.pm create mode 100644 www/manager6/dc/HostBackupJobs.js create mode 100644 www/manager6/node/HostBackup.js installer: Shannon Sterz (7): bump proxmox-installer-types to 0.2 make tidy and clean up whitespace in unconfigured.sh installer-common: add option to verify TLS connections via callback low-level-installer: add support for restoring backups installer-common/tui-installer: implement restore tui unconfigured: add restore mode to unconfigured.sh tui-installer: unmount a potentially mounted backup on abort Cargo.toml | 4 +- Proxmox/Install.pm | 66 ++- Proxmox/Install/Config.pm | 8 +- debian/control | 6 +- .../src/bin/proxmox-auto-installer.rs | 2 +- proxmox-auto-installer/src/utils.rs | 5 +- .../src/fetch_plugins/http.rs | 8 +- proxmox-installer-common/Cargo.toml | 3 +- proxmox-installer-common/src/http.rs | 329 ++++++++--- proxmox-installer-common/src/lib.rs | 1 + proxmox-installer-common/src/options.rs | 132 ++++- proxmox-installer-common/src/restore.rs | 119 ++++ proxmox-installer-common/src/setup.rs | 3 + proxmox-post-hook/src/main.rs | 4 +- proxmox-tui-installer/Cargo.toml | 6 +- proxmox-tui-installer/src/main.rs | 120 +++- proxmox-tui-installer/src/options.rs | 3 + proxmox-tui-installer/src/setup.rs | 13 +- proxmox-tui-installer/src/views/bootdisk.rs | 31 +- proxmox-tui-installer/src/views/mod.rs | 3 + proxmox-tui-installer/src/views/restore.rs | 522 ++++++++++++++++++ unconfigured.sh | 20 +- 22 files changed, 1281 insertions(+), 127 deletions(-) create mode 100644 proxmox-installer-common/src/restore.rs create mode 100644 proxmox-tui-installer/src/views/restore.rs docs: Shannon Sterz (1): examples: add example hook script for host backup jobs Makefile | 1 + examples/host-backup-example-hookscript.pl | 90 ++++++++++++++++++++++ 2 files changed, 91 insertions(+) create mode 100755 examples/host-backup-example-hookscript.pl Summary over all repositories: 52 files changed, 4103 insertions(+), 169 deletions(-) -- Generated by murpp 0.12.0