[pbs-devel] [PATCH 00/22] Single file restore for VM images

[Stefan Reiter <s.reiter@proxmox.com>]

Implements CLI-based single file and directory restore for both pxar.didx
archives (containers, hosts) and img.fidx (VMs, raw block devices). The design
for VM restore uses a small virtual machine that the host communicates with via
virtio-vsock.

This is encapsuled into a new package called "proxmox-file-restore", providing a
binary of the same name. A second package is provided in a new git repository
called "proxmox-restore-vm-data", providing a minimal kernel image and a base
initramfs (without the daemon, which is included in proxmox-file-restore).

Requires my previously sent pxar asyncify series:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html

The first couple patches in the proxmox-backup repo are adapted versions of the
ones Dominik sent to the list a while ago:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html

Dependency bump in proxmox-backup for pxar is required, though best done
together with the changes from the aforementioned seperate series.

Tested with ext4 and NTFS VMs, but theoretically includes support for many more
filesystems (see 'config-base' in the new proxmox-restore-vm-data repository).

Known issues/Missing features:
* GUI/PVE support
* PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore
* ZFS/LVM/md/... support
* shell auto-complete for "proxmox-file-restore" doesn't work (and I don't know
  why...)
* some patches might include some sneaky rustfmt/clippy fixes that'd better fit
  to a previous patch, sorry for that, rebasing so many patches is annoying ;)


pxar: Stefan Reiter (2):
  decoder/aio: add contents() and content_size() calls
  decoder: add peek()

 src/accessor/mod.rs |  3 +++
 src/decoder/aio.rs  | 53 +++++++++++++++++++++++++++++++++++++++++++--
 src/decoder/mod.rs  | 19 ++++++++++++++--
 src/decoder/sync.rs | 10 ++++++++-
 4 files changed, 80 insertions(+), 5 deletions(-)

proxmox-restore-vm-data: Stefan Reiter (1):
  initial commit

proxmox-backup: Dominik Csapak (5):
  api2/admin/datastore: refactor list_dir_content in catalog_reader
  api2/admin/datastore: accept "/" as path for root
  api2/admin/datastore: refactor create_zip into pxar/extract
  pxar/extract: add extract_sub_dir
  file-restore: add binary and basic commands

Stefan Reiter (14):
  pxar/extract: add sequential variants to create_zip, extract_sub_dir
  client: extract common functions to proxmox_client_tools module
  proxmox_client_tools: extract 'key' from client module
  file-restore: allow specifying output-format
  rest: implement tower service for UnixStream
  client: add VsockClient to connect to virtio-vsock VMs
  file-restore-daemon: add binary with virtio-vsock API server
  file-restore-daemon: add watchdog module
  file-restore-daemon: add disk module
  file-restore: add basic VM/block device support
  file-restore: improve logging of VM with logrotate
  debian/client: add postinst hook to rebuild file-restore initramfs
  file-restore(-daemon): implement list API
  file-restore: add 'extract' command for VM file restore

 Cargo.toml                                    |   5 +-
 Makefile                                      |  18 +-
 debian/control                                |  13 +
 debian/control.in                             |  10 +
 debian/proxmox-backup-client.triggers         |   1 +
 debian/proxmox-file-restore.bash-completion   |   1 +
 debian/proxmox-file-restore.bc                |   8 +
 debian/proxmox-file-restore.install           |   4 +
 debian/proxmox-file-restore.postinst          |  63 ++
 debian/proxmox-file-restore.triggers          |   1 +
 debian/rules                                  |   7 +-
 docs/Makefile                                 |  10 +-
 docs/command-line-tools.rst                   |   5 +
 docs/proxmox-file-restore/description.rst     |   4 +
 docs/proxmox-file-restore/man1.rst            |  28 +
 src/api2.rs                                   |   2 +-
 src/api2/admin/datastore.rs                   | 152 +---
 src/api2/helpers.rs                           |  31 +
 src/api2/types/file_restore.rs                |  15 +
 src/api2/types/mod.rs                         |  46 +
 src/backup/catalog.rs                         |  26 +
 src/bin/proxmox-backup-client.rs              | 799 +-----------------
 src/bin/proxmox-file-restore.rs               | 484 +++++++++++
 src/bin/proxmox-restore-daemon.rs             | 124 +++
 src/bin/proxmox_backup_client/catalog.rs      |   4 +-
 src/bin/proxmox_backup_client/mod.rs          |  30 -
 src/bin/proxmox_backup_client/snapshot.rs     |   3 +-
 .../key.rs                                    | 440 +++++++++-
 src/bin/proxmox_client_tools/mod.rs           | 392 +++++++++
 src/bin/proxmox_file_restore/block_driver.rs  | 221 +++++
 .../proxmox_file_restore/block_driver_qemu.rs | 478 +++++++++++
 src/bin/proxmox_file_restore/mod.rs           |   5 +
 src/bin/proxmox_restore_daemon/api.rs         | 316 +++++++
 src/bin/proxmox_restore_daemon/disk.rs        | 341 ++++++++
 src/bin/proxmox_restore_daemon/mod.rs         |   9 +
 src/bin/proxmox_restore_daemon/watchdog.rs    |  63 ++
 src/buildcfg.rs                               |  20 +
 src/client.rs                                 |   3 +
 src/client/vsock_client.rs                    | 259 ++++++
 src/pxar/extract.rs                           | 436 +++++++++-
 src/pxar/mod.rs                               |   5 +-
 src/server/rest.rs                            |  20 +
 www/window/FileBrowser.js                     |   1 +
 zsh-completions/_proxmox-file-restore         |  13 +
 44 files changed, 3940 insertions(+), 976 deletions(-)
 create mode 100644 debian/proxmox-backup-client.triggers
 create mode 100644 debian/proxmox-file-restore.bash-completion
 create mode 100644 debian/proxmox-file-restore.bc
 create mode 100644 debian/proxmox-file-restore.install
 create mode 100755 debian/proxmox-file-restore.postinst
 create mode 100644 debian/proxmox-file-restore.triggers
 create mode 100644 docs/proxmox-file-restore/description.rst
 create mode 100644 docs/proxmox-file-restore/man1.rst
 create mode 100644 src/api2/types/file_restore.rs
 create mode 100644 src/bin/proxmox-file-restore.rs
 create mode 100644 src/bin/proxmox-restore-daemon.rs
 rename src/bin/{proxmox_backup_client => proxmox_client_tools}/key.rs (52%)
 create mode 100644 src/bin/proxmox_client_tools/mod.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs
 create mode 100644 src/bin/proxmox_file_restore/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/api.rs
 create mode 100644 src/bin/proxmox_restore_daemon/disk.rs
 create mode 100644 src/bin/proxmox_restore_daemon/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs
 create mode 100644 src/client/vsock_client.rs
 create mode 100644 zsh-completions/_proxmox-file-restore

-- 
2.20.1