all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH v3 00/20] Single file restore for VM images
@ 2021-03-31 10:21 Stefan Reiter
  2021-03-31 10:21 ` [pbs-devel] [PATCH v3 pxar 01/20] decoder/aio: add contents() and content_size() calls Stefan Reiter
                   ` (20 more replies)
  0 siblings, 21 replies; 32+ messages in thread
From: Stefan Reiter @ 2021-03-31 10:21 UTC (permalink / raw)
  To: pbs-devel

Implements CLI-based single file and directory restore for both pxar.didx
archives (containers, hosts) and img.fidx (VMs, raw block devices). The design
for VM restore uses a small virtual machine that the host communicates with via
virtio-vsock.

This is encapsuled into a new package called "proxmox-file-restore", providing a
binary of the same name. A second package is provided in a new git repository[0]
called "proxmox-backup-restore-image", providing a minimal kernel image and a
base initramfs (without the daemon, which is included in proxmox-file-restore).

Dependency bump in proxmox-backup for pxar is required.

Tested with ext4 and NTFS VMs, but theoretically includes support for many more
filesystems.

Known issues/Missing features:
* GUI/PVE support
* PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore
* ZFS/LVM/md/... support
* shell auto-complete for "proxmox-file-restore" doesn't seem to work (and I
  don't know why...)

[0] now already public at:
    https://git.proxmox.com/?p=proxmox-backup-restore-image.git;a=summary


v3:
* rebase on master
* pxar: fix usage of assume_init (Wolfgang)
* fix crash with '--output-format json' in 'proxmox-file-restore status'
* make ApiAuth a single trait, makes for less generic-creep (Wolfgang)
* redo extract_sub_dir for sequential decoders (Wolfgang)
* fix Filesystems::scan in daemon/disk.rs for zfs (Wolfgang)
* some minor code cleanups


v2:
* rebase on master
* drop applied patches
* pxar: make contents() call available without tokio-io feature (Wolfgang)
* pxar: drop peek() implementation, rework extractor to cope (Wolfgang)
* only move necessary functions to new key_source.rs (Dietmar)
* implement static ticket-based authentication for VMs, as relying on ports
  <1024 does not guarantee security (Dietmar, Wolfgang)
* allow running proxmox-file-restore as regular user by providing setuid-binary
  to start QEMU VMs (setgid kvm is not enough because of /dev/vhost-vsock)
  (Dietmar, Fabian)
* update debian/* with new proxmox-backup-restore-image naming (Thomas)
* encode zip file directly on the VM, only encode pxar when requested (Dominik)
* use tokio task in watchdog, instead of alarm() (Wolfgang)


pxar: Stefan Reiter (1):
  decoder/aio: add contents() and content_size() calls

 src/decoder/aio.rs | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

proxmox-backup: Dominik Csapak (1):
  file-restore: add binary and basic commands

Stefan Reiter (18):
  vsock_client: remove wrong comment
  vsock_client: remove some &mut restrictions and rustfmt
  vsock_client: support authorization header
  proxmox_client_tools: move common key related functions to
    key_source.rs
  file-restore: allow specifying output-format
  server/rest: extract auth to seperate module
  server/rest: add ApiAuth trait to make user auth generic
  file-restore-daemon: add binary with virtio-vsock API server
  file-restore-daemon: add watchdog module
  file-restore-daemon: add disk module
  add tools/cpio encoding module
  file-restore: add qemu-helper setuid binary
  file-restore: add basic VM/block device support
  debian/client: add postinst hook to rebuild file-restore initramfs
  file-restore(-daemon): implement list API
  pxar/extract: add sequential variant of extract_sub_dir
  tools/zip: add zip_directory helper
  file-restore: add 'extract' command for VM file restore

 Cargo.toml                                    |   5 +-
 Makefile                                      |  22 +-
 debian/control                                |  14 +
 debian/control.in                             |  11 +
 debian/proxmox-file-restore.bash-completion   |   1 +
 debian/proxmox-file-restore.bc                |   8 +
 debian/proxmox-file-restore.install           |   5 +
 debian/proxmox-file-restore.postinst          |  63 ++
 debian/proxmox-file-restore.triggers          |   1 +
 debian/rules                                  |   9 +-
 docs/Makefile                                 |  10 +-
 docs/command-line-tools.rst                   |   5 +
 docs/proxmox-file-restore/description.rst     |   3 +
 docs/proxmox-file-restore/man1.rst            |  28 +
 src/api2.rs                                   |   2 +-
 src/api2/types/file_restore.rs                |  15 +
 src/api2/types/mod.rs                         |   3 +
 src/bin/proxmox-backup-api.rs                 |  13 +-
 src/bin/proxmox-backup-client.rs              | 453 +-------------
 src/bin/proxmox-backup-proxy.rs               |   7 +-
 src/bin/proxmox-file-restore.rs               | 456 ++++++++++++++
 src/bin/proxmox-restore-daemon.rs             | 124 ++++
 src/bin/proxmox-restore-qemu-helper.rs        | 372 ++++++++++++
 src/bin/proxmox_backup_client/benchmark.rs    |   4 +-
 src/bin/proxmox_backup_client/catalog.rs      |   3 +-
 src/bin/proxmox_backup_client/key.rs          | 112 +---
 src/bin/proxmox_backup_client/mod.rs          |  28 -
 src/bin/proxmox_backup_client/mount.rs        |   4 +-
 src/bin/proxmox_backup_client/snapshot.rs     |   4 +-
 src/bin/proxmox_client_tools/key_source.rs    | 573 ++++++++++++++++++
 src/bin/proxmox_client_tools/mod.rs           |  65 +-
 src/bin/proxmox_file_restore/block_driver.rs  | 206 +++++++
 .../proxmox_file_restore/block_driver_qemu.rs | 362 +++++++++++
 src/bin/proxmox_file_restore/mod.rs           |   5 +
 src/bin/proxmox_restore_daemon/api.rs         | 369 +++++++++++
 src/bin/proxmox_restore_daemon/auth.rs        |  45 ++
 src/bin/proxmox_restore_daemon/disk.rs        | 329 ++++++++++
 src/bin/proxmox_restore_daemon/mod.rs         |  11 +
 src/bin/proxmox_restore_daemon/watchdog.rs    |  41 ++
 src/buildcfg.rs                               |  21 +
 src/client/vsock_client.rs                    |  78 +--
 src/pxar/extract.rs                           | 316 +++++++---
 src/pxar/mod.rs                               |   5 +-
 src/server.rs                                 |   2 +
 src/server/auth.rs                            | 140 +++++
 src/server/config.rs                          |  13 +-
 src/server/rest.rs                            | 130 +---
 src/tools.rs                                  |   1 +
 src/tools/cpio.rs                             |  73 +++
 src/tools/zip.rs                              |  77 +++
 zsh-completions/_proxmox-file-restore         |  13 +
 51 files changed, 3796 insertions(+), 864 deletions(-)
 create mode 100644 debian/proxmox-file-restore.bash-completion
 create mode 100644 debian/proxmox-file-restore.bc
 create mode 100644 debian/proxmox-file-restore.install
 create mode 100755 debian/proxmox-file-restore.postinst
 create mode 100644 debian/proxmox-file-restore.triggers
 create mode 100644 docs/proxmox-file-restore/description.rst
 create mode 100644 docs/proxmox-file-restore/man1.rst
 create mode 100644 src/api2/types/file_restore.rs
 create mode 100644 src/bin/proxmox-file-restore.rs
 create mode 100644 src/bin/proxmox-restore-daemon.rs
 create mode 100644 src/bin/proxmox-restore-qemu-helper.rs
 create mode 100644 src/bin/proxmox_client_tools/key_source.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs
 create mode 100644 src/bin/proxmox_file_restore/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/api.rs
 create mode 100644 src/bin/proxmox_restore_daemon/auth.rs
 create mode 100644 src/bin/proxmox_restore_daemon/disk.rs
 create mode 100644 src/bin/proxmox_restore_daemon/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs
 create mode 100644 src/server/auth.rs
 create mode 100644 src/tools/cpio.rs
 create mode 100644 zsh-completions/_proxmox-file-restore

-- 
2.20.1




^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2021-04-08 14:44 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-31 10:21 [pbs-devel] [PATCH v3 00/20] Single file restore for VM images Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 pxar 01/20] decoder/aio: add contents() and content_size() calls Stefan Reiter
2021-03-31 11:54   ` [pbs-devel] applied: " Wolfgang Bumiller
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 02/20] vsock_client: remove wrong comment Stefan Reiter
2021-04-01  9:53   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 03/20] vsock_client: remove some &mut restrictions and rustfmt Stefan Reiter
2021-04-01  9:54   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 04/20] vsock_client: support authorization header Stefan Reiter
2021-04-01  9:54   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 05/20] proxmox_client_tools: move common key related functions to key_source.rs Stefan Reiter
2021-04-01  9:54   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 06/20] file-restore: add binary and basic commands Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 07/20] file-restore: allow specifying output-format Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 08/20] server/rest: extract auth to seperate module Stefan Reiter
2021-04-01  9:55   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 09/20] server/rest: add ApiAuth trait to make user auth generic Stefan Reiter
2021-03-31 12:55   ` Wolfgang Bumiller
2021-03-31 14:07     ` Thomas Lamprecht
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 10/20] file-restore-daemon: add binary with virtio-vsock API server Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 11/20] file-restore-daemon: add watchdog module Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 12/20] file-restore-daemon: add disk module Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 13/20] add tools/cpio encoding module Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 14/20] file-restore: add qemu-helper setuid binary Stefan Reiter
2021-03-31 14:15   ` Oguz Bektas
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 15/20] file-restore: add basic VM/block device support Stefan Reiter
2021-04-01 15:43   ` [pbs-devel] [PATCH v4 " Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 16/20] debian/client: add postinst hook to rebuild file-restore initramfs Stefan Reiter
2021-03-31 10:21 ` [pbs-devel] [PATCH v3 proxmox-backup 17/20] file-restore(-daemon): implement list API Stefan Reiter
2021-03-31 10:22 ` [pbs-devel] [PATCH v3 proxmox-backup 18/20] pxar/extract: add sequential variant of extract_sub_dir Stefan Reiter
2021-03-31 10:22 ` [pbs-devel] [PATCH v3 proxmox-backup 19/20] tools/zip: add zip_directory helper Stefan Reiter
2021-03-31 10:22 ` [pbs-devel] [PATCH v3 proxmox-backup 20/20] file-restore: add 'extract' command for VM file restore Stefan Reiter
2021-04-08 14:44 ` [pbs-devel] applied: [PATCH v3 00/20] Single file restore for VM images Thomas Lamprecht

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal