public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH 00/22] Single file restore for VM images
@ 2021-02-16 17:06 Stefan Reiter
  2021-02-16 17:06 ` [pbs-devel] [PATCH pxar 01/22] decoder/aio: add contents() and content_size() calls Stefan Reiter
                   ` (22 more replies)
  0 siblings, 23 replies; 50+ messages in thread
From: Stefan Reiter @ 2021-02-16 17:06 UTC (permalink / raw)
  To: pbs-devel

Implements CLI-based single file and directory restore for both pxar.didx
archives (containers, hosts) and img.fidx (VMs, raw block devices). The design
for VM restore uses a small virtual machine that the host communicates with via
virtio-vsock.

This is encapsuled into a new package called "proxmox-file-restore", providing a
binary of the same name. A second package is provided in a new git repository
called "proxmox-restore-vm-data", providing a minimal kernel image and a base
initramfs (without the daemon, which is included in proxmox-file-restore).

Requires my previously sent pxar asyncify series:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html

The first couple patches in the proxmox-backup repo are adapted versions of the
ones Dominik sent to the list a while ago:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html

Dependency bump in proxmox-backup for pxar is required, though best done
together with the changes from the aforementioned seperate series.

Tested with ext4 and NTFS VMs, but theoretically includes support for many more
filesystems (see 'config-base' in the new proxmox-restore-vm-data repository).

Known issues/Missing features:
* GUI/PVE support
* PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore
* ZFS/LVM/md/... support
* shell auto-complete for "proxmox-file-restore" doesn't work (and I don't know
  why...)
* some patches might include some sneaky rustfmt/clippy fixes that'd better fit
  to a previous patch, sorry for that, rebasing so many patches is annoying ;)


pxar: Stefan Reiter (2):
  decoder/aio: add contents() and content_size() calls
  decoder: add peek()

 src/accessor/mod.rs |  3 +++
 src/decoder/aio.rs  | 53 +++++++++++++++++++++++++++++++++++++++++++--
 src/decoder/mod.rs  | 19 ++++++++++++++--
 src/decoder/sync.rs | 10 ++++++++-
 4 files changed, 80 insertions(+), 5 deletions(-)

proxmox-restore-vm-data: Stefan Reiter (1):
  initial commit

proxmox-backup: Dominik Csapak (5):
  api2/admin/datastore: refactor list_dir_content in catalog_reader
  api2/admin/datastore: accept "/" as path for root
  api2/admin/datastore: refactor create_zip into pxar/extract
  pxar/extract: add extract_sub_dir
  file-restore: add binary and basic commands

Stefan Reiter (14):
  pxar/extract: add sequential variants to create_zip, extract_sub_dir
  client: extract common functions to proxmox_client_tools module
  proxmox_client_tools: extract 'key' from client module
  file-restore: allow specifying output-format
  rest: implement tower service for UnixStream
  client: add VsockClient to connect to virtio-vsock VMs
  file-restore-daemon: add binary with virtio-vsock API server
  file-restore-daemon: add watchdog module
  file-restore-daemon: add disk module
  file-restore: add basic VM/block device support
  file-restore: improve logging of VM with logrotate
  debian/client: add postinst hook to rebuild file-restore initramfs
  file-restore(-daemon): implement list API
  file-restore: add 'extract' command for VM file restore

 Cargo.toml                                    |   5 +-
 Makefile                                      |  18 +-
 debian/control                                |  13 +
 debian/control.in                             |  10 +
 debian/proxmox-backup-client.triggers         |   1 +
 debian/proxmox-file-restore.bash-completion   |   1 +
 debian/proxmox-file-restore.bc                |   8 +
 debian/proxmox-file-restore.install           |   4 +
 debian/proxmox-file-restore.postinst          |  63 ++
 debian/proxmox-file-restore.triggers          |   1 +
 debian/rules                                  |   7 +-
 docs/Makefile                                 |  10 +-
 docs/command-line-tools.rst                   |   5 +
 docs/proxmox-file-restore/description.rst     |   4 +
 docs/proxmox-file-restore/man1.rst            |  28 +
 src/api2.rs                                   |   2 +-
 src/api2/admin/datastore.rs                   | 152 +---
 src/api2/helpers.rs                           |  31 +
 src/api2/types/file_restore.rs                |  15 +
 src/api2/types/mod.rs                         |  46 +
 src/backup/catalog.rs                         |  26 +
 src/bin/proxmox-backup-client.rs              | 799 +-----------------
 src/bin/proxmox-file-restore.rs               | 484 +++++++++++
 src/bin/proxmox-restore-daemon.rs             | 124 +++
 src/bin/proxmox_backup_client/catalog.rs      |   4 +-
 src/bin/proxmox_backup_client/mod.rs          |  30 -
 src/bin/proxmox_backup_client/snapshot.rs     |   3 +-
 .../key.rs                                    | 440 +++++++++-
 src/bin/proxmox_client_tools/mod.rs           | 392 +++++++++
 src/bin/proxmox_file_restore/block_driver.rs  | 221 +++++
 .../proxmox_file_restore/block_driver_qemu.rs | 478 +++++++++++
 src/bin/proxmox_file_restore/mod.rs           |   5 +
 src/bin/proxmox_restore_daemon/api.rs         | 316 +++++++
 src/bin/proxmox_restore_daemon/disk.rs        | 341 ++++++++
 src/bin/proxmox_restore_daemon/mod.rs         |   9 +
 src/bin/proxmox_restore_daemon/watchdog.rs    |  63 ++
 src/buildcfg.rs                               |  20 +
 src/client.rs                                 |   3 +
 src/client/vsock_client.rs                    | 259 ++++++
 src/pxar/extract.rs                           | 436 +++++++++-
 src/pxar/mod.rs                               |   5 +-
 src/server/rest.rs                            |  20 +
 www/window/FileBrowser.js                     |   1 +
 zsh-completions/_proxmox-file-restore         |  13 +
 44 files changed, 3940 insertions(+), 976 deletions(-)
 create mode 100644 debian/proxmox-backup-client.triggers
 create mode 100644 debian/proxmox-file-restore.bash-completion
 create mode 100644 debian/proxmox-file-restore.bc
 create mode 100644 debian/proxmox-file-restore.install
 create mode 100755 debian/proxmox-file-restore.postinst
 create mode 100644 debian/proxmox-file-restore.triggers
 create mode 100644 docs/proxmox-file-restore/description.rst
 create mode 100644 docs/proxmox-file-restore/man1.rst
 create mode 100644 src/api2/types/file_restore.rs
 create mode 100644 src/bin/proxmox-file-restore.rs
 create mode 100644 src/bin/proxmox-restore-daemon.rs
 rename src/bin/{proxmox_backup_client => proxmox_client_tools}/key.rs (52%)
 create mode 100644 src/bin/proxmox_client_tools/mod.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver.rs
 create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs
 create mode 100644 src/bin/proxmox_file_restore/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/api.rs
 create mode 100644 src/bin/proxmox_restore_daemon/disk.rs
 create mode 100644 src/bin/proxmox_restore_daemon/mod.rs
 create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs
 create mode 100644 src/client/vsock_client.rs
 create mode 100644 zsh-completions/_proxmox-file-restore

-- 
2.20.1




^ permalink raw reply	[flat|nested] 50+ messages in thread

end of thread, other threads:[~2021-03-16 15:33 UTC | newest]

Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-16 17:06 [pbs-devel] [PATCH 00/22] Single file restore for VM images Stefan Reiter
2021-02-16 17:06 ` [pbs-devel] [PATCH pxar 01/22] decoder/aio: add contents() and content_size() calls Stefan Reiter
2021-02-17  7:56   ` Wolfgang Bumiller
2021-02-16 17:06 ` [pbs-devel] [PATCH pxar 02/22] decoder: add peek() Stefan Reiter
2021-02-17  8:20   ` Wolfgang Bumiller
2021-02-17  8:38     ` Stefan Reiter
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-restore-vm-data 03/22] initial commit Stefan Reiter
2021-03-15 18:35   ` [pbs-devel] applied: " Thomas Lamprecht
2021-03-16 15:33     ` Stefan Reiter
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 04/22] api2/admin/datastore: refactor list_dir_content in catalog_reader Stefan Reiter
2021-02-17  7:50   ` [pbs-devel] applied: " Thomas Lamprecht
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 05/22] api2/admin/datastore: accept "/" as path for root Stefan Reiter
2021-02-17  7:50   ` [pbs-devel] applied: " Thomas Lamprecht
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 06/22] api2/admin/datastore: refactor create_zip into pxar/extract Stefan Reiter
2021-02-17  7:50   ` [pbs-devel] applied: " Thomas Lamprecht
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 07/22] pxar/extract: add extract_sub_dir Stefan Reiter
2021-02-17  7:51   ` [pbs-devel] applied: " Thomas Lamprecht
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 08/22] pxar/extract: add sequential variants to create_zip, extract_sub_dir Stefan Reiter
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 09/22] client: extract common functions to proxmox_client_tools module Stefan Reiter
2021-02-17  6:49   ` Dietmar Maurer
2021-02-17  7:58     ` Stefan Reiter
2021-02-17  8:50       ` Dietmar Maurer
2021-02-17  9:47         ` Stefan Reiter
2021-02-17 10:12           ` Dietmar Maurer
2021-02-17  9:13   ` [pbs-devel] applied: " Dietmar Maurer
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 10/22] proxmox_client_tools: extract 'key' from client module Stefan Reiter
2021-02-17  9:11   ` Dietmar Maurer
2021-02-16 17:06 ` [pbs-devel] [PATCH proxmox-backup 11/22] file-restore: add binary and basic commands Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 12/22] file-restore: allow specifying output-format Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 13/22] rest: implement tower service for UnixStream Stefan Reiter
2021-02-17  6:52   ` [pbs-devel] applied: " Dietmar Maurer
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 14/22] client: add VsockClient to connect to virtio-vsock VMs Stefan Reiter
2021-02-17  7:24   ` [pbs-devel] applied: " Dietmar Maurer
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 15/22] file-restore-daemon: add binary with virtio-vsock API server Stefan Reiter
2021-02-17 10:17   ` Dietmar Maurer
2021-02-17 10:25   ` Dietmar Maurer
2021-02-17 10:30     ` Stefan Reiter
2021-02-17 11:13       ` Dietmar Maurer
2021-02-17 11:26   ` Dietmar Maurer
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 16/22] file-restore-daemon: add watchdog module Stefan Reiter
2021-02-17 10:52   ` Wolfgang Bumiller
2021-02-17 11:14     ` Stefan Reiter
2021-02-17 11:29       ` Wolfgang Bumiller
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 17/22] file-restore-daemon: add disk module Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 18/22] file-restore: add basic VM/block device support Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 19/22] file-restore: improve logging of VM with logrotate Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 20/22] debian/client: add postinst hook to rebuild file-restore initramfs Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 21/22] file-restore(-daemon): implement list API Stefan Reiter
2021-02-16 17:07 ` [pbs-devel] [PATCH proxmox-backup 22/22] file-restore: add 'extract' command for VM file restore Stefan Reiter
2021-02-16 17:11 ` [pbs-devel] [PATCH 00/22] Single file restore for VM images Stefan Reiter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal