[pbs-devel] [PATCH v2 00/20] Single file restore for VM images

[Stefan Reiter <s.reiter@proxmox.com>]

Implements CLI-based single file and directory restore for both pxar.didx
archives (containers, hosts) and img.fidx (VMs, raw block devices). The design
for VM restore uses a small virtual machine that the host communicates with via
virtio-vsock.

This is encapsuled into a new package called "proxmox-file-restore", providing a
binary of the same name. A second package is provided in a new git repository
called "proxmox-backup-restore-image", providing a minimal kernel image and a
base initramfs (without the daemon, which is included in proxmox-file-restore).

The first couple patches in the proxmox-backup repo are adapted versions of the
ones Dominik sent to the list a while ago:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html

Dependency bump in proxmox-backup for pxar is required.

Tested with ext4 and NTFS VMs, but theoretically includes support for many more
filesystems.

Known issues/Missing features:
* GUI/PVE support
* PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore
* ZFS/LVM/md/... support
* shell auto-complete for "proxmox-file-restore" doesn't seem to work (and I
  don't know why...)
* some patches might include some sneaky rustfmt/clippy fixes that'd better fit
  to a previous patch, sorry for that, rebasing so many patches is annoying ;)


v2:
* rebase on master
* drop applied patches
* pxar: make contents() call available without tokio-io feature (Wolfgang)
* pxar: drop peek() implementation, rework extractor to cope (Wolfgang)
* only move necessary functions to new key_source.rs (Dietmar)
* implement static ticket-based authentication for VMs, as relying on ports
  <1024 does not guarantee security (Dietmar, Wolfgang)
* allow running proxmox-file-restore as regular user by providing setuid-binary
  to start QEMU VMs (setgid kvm is not enough because of /dev/vhost-vsock)
  (Dietmar, Fabian)
* update debian/* with new proxmox-backup-restore-image naming (Thomas)
* encode zip file directly on the VM, only encode pxar when requested (Dominik)
* use tokio task in watchdog, instead of alarm() (Wolfgang)

...plus some reshuffling of patches because of all the changes from above, hope
I didn't miss anything big ;)


pxar: Stefan Reiter (1):
  decoder/aio: add contents() and content_size() calls

 src/decoder/aio.rs | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

proxmox-backup: Dominik Csapak (1):
  file-restore: add binary and basic commands

Stefan Reiter (18):
  vsock_client: remove wrong comment
  vsock_client: remove some &mut restrictions and rustfmt
  vsock_client: support authorization header
  proxmox_client_tools: move common key related functions to
    key_source.rs
  file-restore: allow specifying output-format
  server/rest: extract auth to seperate module
  server/rest: add ApiAuth trait to make user auth generic
  file-restore-daemon: add binary with virtio-vsock API server
  file-restore-daemon: add watchdog module
  file-restore-daemon: add disk module
  add tools/cpio encoding module
  file-restore: add qemu-helper setuid binary
  file-restore: add basic VM/block device support
  debian/client: add postinst hook to rebuild file-restore initramfs
  file-restore(-daemon): implement list API
  pxar/extract: add sequential variant to extract_sub_dir
  tools/zip: add zip_directory helper
  file-restore: add 'extract' command for VM file restore

 Cargo.toml                                    |   5 +-
 Makefile                                      |  22 +-
 debian/control                                |  14 +
 debian/control.in                             |  11 +
 debian/proxmox-file-restore.bash-completion   |   1 +
 debian/proxmox-file-restore.bc                |   8 +
 debian/proxmox-file-restore.install           |   5 +
 debian/proxmox-file-restore.postinst          |  63 ++
 debian/proxmox-file-restore.triggers          |   1 +
 debian/rules                                  |   9 +-
 docs/Makefile                                 |  10 +-
 docs/command-line-tools.rst                   |   5 +
 docs/proxmox-file-restore/description.rst     |   3 +
 docs/proxmox-file-restore/man1.rst            |  28 +
 src/api2.rs                                   |   2 +-
 src/api2/types/file_restore.rs                |  15 +
 src/api2/types/mod.rs                         |   3 +
 src/bin/proxmox-backup-api.rs                 |  13 +-
 src/bin/proxmox-backup-client.rs              | 453 +-------------
 src/bin/proxmox-backup-proxy.rs               |   7 +-
 src/bin/proxmox-file-restore.rs               | 446 ++++++++++++++
 src/bin/proxmox-restore-daemon.rs             | 124 ++++
 src/bin/proxmox-restore-qemu-helper.rs        | 372 ++++++++++++
 src/bin/proxmox_backup_client/benchmark.rs    |   4 +-
 src/bin/proxmox_backup_client/catalog.rs      |   3 +-
 src/bin/proxmox_backup_client/key.rs          | 112 +---
 src/bin/proxmox_backup_client/mod.rs          |  28 -
 src/bin/proxmox_backup_client/mount.rs        |   4 +-
 src/bin/proxmox_backup_client/snapshot.rs     |   4 +-
 src/bin/proxmox_client_tools/key_source.rs    | 573 ++++++++++++++++++
 src/bin/proxmox_client_tools/mod.rs           |  65 +-
 src/bin/proxmox_file_restore/block_driver.rs  | 206 +++++++
 .../proxmox_file_restore/block_driver_qemu.rs | 362 +++++++++++
 src/bin/proxmox_file_restore/mod.rs           |   5 +
 src/bin/proxmox_restore_daemon/api.rs         | 369 +++++++++++
 src/bin/proxmox_restore_daemon/auth.rs        |  48 ++
 src/bin/proxmox_restore_daemon/disk.rs        | 341 +++++++++++
 src/bin/proxmox_restore_daemon/mod.rs         |  11 +
 src/bin/proxmox_restore_daemon/watchdog.rs    |  41 ++
 src/buildcfg.rs                               |  21 +
 src/client/vsock_client.rs                    |  78 +--
 src/pxar/extract.rs                           | 214 +++++--
 src/pxar/mod.rs                               |   5 +-
 src/server.rs                                 |   2 +
 src/server/auth.rs                            | 124 ++++
 src/server/config.rs                          |  17 +-
 src/server/rest.rs                            | 141 +----
 src/tools.rs                                  |   1 +
 src/tools/cpio.rs                             |  73 +++
 src/tools/zip.rs                              |  77 +++
 zsh-completions/_proxmox-file-restore         |  13 +
 51 files changed, 3734 insertions(+), 828 deletions(-)
 create mode 100644 debian/proxmox-file-restore.bash-completion
 create mode 100644 debian/proxmox-file-restore.bc
 create mode 100644 debian/proxmox-file-restore.install
 create mode 100755 debian/proxmox-file-restore.postinst
 create mode 100644 debian/proxmox-file-restore.triggers
 create mode 100644 docs/proxmox-file-restore/description.rst
 create mode 100644 docs/proxmox-file-restore/man1.rst
 create mode 100644 src/api2/types/file_restore.rs
 create mode 100644 src/bin/proxmox-file-restore.rs
 create mode 100644 src/bin/proxmox-restore-daemon.rs
 create mode 100644 src/bin/proxmox-restore-qemu-helper.rs
 create mode 100644 src/bin/proxmox_client_tools/key_source.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs
 create mode 100644 src/bin/proxmox_file_restore/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/api.rs
 create mode 100644 src/bin/proxmox_restore_daemon/auth.rs
 create mode 100644 src/bin/proxmox_restore_daemon/disk.rs
 create mode 100644 src/bin/proxmox_restore_daemon/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs
 create mode 100644 src/server/auth.rs
 create mode 100644 src/tools/cpio.rs
 create mode 100644 zsh-completions/_proxmox-file-restore

-- 
2.20.1