[RFC cluster/common/container/docs/installer/manager 00/21] add rudimentary host backup mechanism

[Shannon Sterz <s.sterz@proxmox.com>]

This is still very early stages and rough around the edges. I am sending this
now, because the rough mechanism seems to work well enough to start gathering
some input. Further clean up will follow, I am mostly looking for feedback on
whether this is the right direction to go. All changes outlined here could
definitively benefit from more thorough testing. Especially with more complex
and exotic setups.

The goal of this series is to add a host backup mechanism for Proxmox VE. Such
a mechanism should enable users to automate backups and help in a disaster
recovery scenario. This is accomplished by backing up `/etc` and several bits of
information of a running system via the pbs client.

For restoring there are two options:

- File restore for single config files.
- Restoring via the installer from a clean slate (currently only intended for
disaster recovery). For now this mostly works properly if the hardware is
identical, but should be extended to be more forgiving (see open work).

The patches for pmxcfs extend it to allow for live-backups without having to
interrupt pmxcfs operations.

Secondly patches for pve-manager add the ui, api endpoints and job setup for
the basic backups. They allow for custom hook scripts and additional files that
need to be backed up for more flexibility.

Finally, the patches for the installer add a restore mode for the TUI installer
to allow restoring when setting up a new system.

How to Apply & Bump
-------------------

* patches 02-03/18 implement the core live backup mechanism for pmxcfs and are
  for the patches in pve-manager

* patch 04/18 to is required to be applied and the libpve-common-perl
  dependency needs to be bumped in pve-manager and pve-container before patches
  05-06/18 can be applied

Backup Consistency
------------------

Looking over options to improve backup consistency there were several discarded
ideas I've had. A selection:

- [overlayfs](https://docs.kernel.org/filesystems/overlayfs.html):
    + Basic idea: Overlay the current root file system, read from the underlay
    and then merge back the overlay. Writes during the backup happen only on the
    overlay, so the view we get should be consistent.
    + Merging the overlaid changes back down would require an unmount operation
    ("Changes to the underlying filesystems while part of a mounted overlay
    filesystem are not allowed." [2]). This will likely interrupt the system and
    possibly cause inconsistencies.
    + Where do we put the overlay?
        * If we put it in RAM via a tmpfs, any write that happens during the
        backup may be easily lost if changes are not properly merged back down.
        * Putting it on another file system backed by a physical device either
        means we need to reserve a lot of space on setup or require additional
        hardware.
    + We probably would want to create the overlay on boot, because otherwise
    running processes may be left in an inconsistent state leading to I/O
    errors. For example, because the underlay they had open file handles to is
    now read-only or if they try to write to the underlay they break overlayfs
    requirements.).
- [fsfreeze](https://manpages.debian.org/trixie/util-linux/fsfreeze.8.en.html):
    + We could try to sync out data, freeze the filesystem, sync remaining data
    and then resume.
    + Still interrupts currently running systems and could lead to issues. For
    example, if the backup fails without cleaning up the freeze, recovering from
    that is tricky. SSH nor Web UI can be used as establishing new connections
    via both are blocked until the filesystem is unfrozen.
- [blksnap](https://github.com/veeam/blksnap?tab=readme-ov-file) by veeam:
    + Would require us to ship a kernel module by ourselves, though the repo has
    instructions on how to package it for Debian.
    + Last commit is from over 10 months ago, last submission to LKML seems to
    have happened in 2024 [3], though the repo mentions patches that were/are
    prepared for a v8. It seems that up-streaming efforts have been stalled [4].

For now I landed on doing snapshots for CoW filesystems (zfs,btrfs) and
sqlite3's live backup mechanism. Users can try to improve consistency based hook
scripts too (e.g. if LVM thick snapshots are an option).

Open Work
---------

There is still a lot that is missing here, amongst others:

- Better Support for non-identical hardware restore: Currently the installer
does not allow progressing if it can't restore the exact disk layout. This
should cleaned up to allow properly overriding the root disk layout. There are
also some pitfall when modifying the initial disk setup when it comes to
restoring the storage.cfg.

- Encryption: Currently the reinstall restoration mode does not work for
encrypted backups at all. Inputting that information via the TUI was deemed to
cumbersome.

- Better testing: Currently this missing unit and integration tests. At least a
basic set of tests should be implement for things like parsing out the
installer state from an already installed system.

- Documentation: This is currently lacking documentation more or less entirely.
Mainly cause I did not get around to it yet. While I think most parts should be
fairly intuitive to most of our users, things like `additional-files` and the
`hooks` definitively deserver more explanations.


cluster:

Shannon Sterz (4):
  pmxcfs: status: fix formatting of parameters in checked_mkdir()
  pmxcfs: correctly log message when directory can't be created
  pmxcfs: add live backup capability
  pmxcfs: add ability to query backup progress

 src/PVE/Cluster.pm       |  82 +++++++++++++++++++++-
 src/pmxcfs/cfs-ipc-ops.h |   4 ++
 src/pmxcfs/cfs-utils.h   |   1 +
 src/pmxcfs/database.c    | 146 +++++++++++++++++++++++++++++++++++++++
 src/pmxcfs/memdb.c       |  42 +++++++++++
 src/pmxcfs/memdb.h       | 122 ++++++++++++++++++++++++++++++++
 src/pmxcfs/server.c      | 105 ++++++++++++++++++++++++++++
 src/pmxcfs/status.c      |  13 +++-
 src/pmxcfs/status.h      |   2 +
 9 files changed, 511 insertions(+), 6 deletions(-)


common:

Shannon Sterz (1):
  systemd: move parse_os_release() helper to PVE::Systemd

 src/PVE/Systemd.pm | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)


container:

Shannon Sterz (1):
  setup: use parse_os_release from PVE::Systemd

 src/PVE/LXC/Setup.pm | 35 ++---------------------------------
 1 file changed, 2 insertions(+), 33 deletions(-)


manager:

Shannon Sterz (7):
  jobs/api: add basic host backup job logic
  api: cluster: add endpoints for manage host backup jobs
  api: node: add endpoints for listing backups for a node
  api: host backup: include global, disk and network options for restore
  api: host backup: add warnings in case zfs snapdir is disabled
  ui: node: add panel to manage backups of a host
  ui: dc: add panel for managing host backup jobs

 PVE/API2/Cluster/HostBackup.pm    | 355 +++++++++++++++++++
 PVE/API2/Cluster/Jobs.pm          |  10 +-
 PVE/API2/Cluster/Makefile         |   1 +
 PVE/API2/HostBackup.pm            | 559 ++++++++++++++++++++++++++++++
 PVE/API2/Makefile                 |   1 +
 PVE/API2/Nodes.pm                 |   7 +
 PVE/HostBackupTools.pm            | 452 ++++++++++++++++++++++++
 PVE/Jobs.pm                       |   2 +
 PVE/Jobs/HostBackup.pm            | 122 +++++++
 PVE/Jobs/Makefile                 |   5 +-
 PVE/Makefile                      |   1 +
 www/manager6/Makefile             |   2 +
 www/manager6/Utils.js             |   1 +
 www/manager6/dc/Config.js         |   6 +
 www/manager6/dc/HostBackupJobs.js | 360 +++++++++++++++++++
 www/manager6/node/Config.js       |   7 +
 www/manager6/node/HostBackup.js   | 292 ++++++++++++++++
 17 files changed, 2180 insertions(+), 3 deletions(-)
 create mode 100644 PVE/API2/Cluster/HostBackup.pm
 create mode 100644 PVE/API2/HostBackup.pm
 create mode 100644 PVE/HostBackupTools.pm
 create mode 100644 PVE/Jobs/HostBackup.pm
 create mode 100644 www/manager6/dc/HostBackupJobs.js
 create mode 100644 www/manager6/node/HostBackup.js


installer:

Shannon Sterz (7):
  bump proxmox-installer-types to 0.2
  make tidy and clean up whitespace in unconfigured.sh
  installer-common: add option to verify TLS connections via callback
  low-level-installer: add support for restoring backups
  installer-common/tui-installer: implement restore tui
  unconfigured: add restore mode to unconfigured.sh
  tui-installer: unmount a potentially mounted backup on abort

 Cargo.toml                                    |   4 +-
 Proxmox/Install.pm                            |  66 ++-
 Proxmox/Install/Config.pm                     |   8 +-
 debian/control                                |   6 +-
 .../src/bin/proxmox-auto-installer.rs         |   2 +-
 proxmox-auto-installer/src/utils.rs           |   5 +-
 .../src/fetch_plugins/http.rs                 |   8 +-
 proxmox-installer-common/Cargo.toml           |   3 +-
 proxmox-installer-common/src/http.rs          | 329 ++++++++---
 proxmox-installer-common/src/lib.rs           |   1 +
 proxmox-installer-common/src/options.rs       | 132 ++++-
 proxmox-installer-common/src/restore.rs       | 119 ++++
 proxmox-installer-common/src/setup.rs         |   3 +
 proxmox-post-hook/src/main.rs                 |   4 +-
 proxmox-tui-installer/Cargo.toml              |   6 +-
 proxmox-tui-installer/src/main.rs             | 120 +++-
 proxmox-tui-installer/src/options.rs          |   3 +
 proxmox-tui-installer/src/setup.rs            |  13 +-
 proxmox-tui-installer/src/views/bootdisk.rs   |  31 +-
 proxmox-tui-installer/src/views/mod.rs        |   3 +
 proxmox-tui-installer/src/views/restore.rs    | 522 ++++++++++++++++++
 unconfigured.sh                               |  20 +-
 22 files changed, 1281 insertions(+), 127 deletions(-)
 create mode 100644 proxmox-installer-common/src/restore.rs
 create mode 100644 proxmox-tui-installer/src/views/restore.rs


docs:

Shannon Sterz (1):
  examples: add example hook script for host backup jobs

 Makefile                                   |  1 +
 examples/host-backup-example-hookscript.pl | 90 ++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100755 examples/host-backup-example-hookscript.pl


Summary over all repositories:
  52 files changed, 4103 insertions(+), 169 deletions(-)

-- 
Generated by murpp 0.12.0