From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A9AEF6CE3C for ; Wed, 31 Mar 2021 12:22:31 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 80310DF1A for ; Wed, 31 Mar 2021 12:22:30 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 31651DDF5 for ; Wed, 31 Mar 2021 12:22:24 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0044D42942 for ; Wed, 31 Mar 2021 12:22:23 +0200 (CEST) From: Stefan Reiter To: pbs-devel@lists.proxmox.com Date: Wed, 31 Mar 2021 12:21:42 +0200 Message-Id: <20210331102202.14767-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.019 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [mod.rs, key.rs, disk.rs, buildcfg.rs, control.in, proxmox-file-restore.rs, cpio.rs, config.rs, benchmark.rs, watchdog.rs, tools.rs, api2.rs, mount.rs, snapshot.rs, rest.rs, zip.rs, proxmox-restore-qemu-helper.rs, proxmox-restore-daemon.rs, catalog.rs, proxmox-backup-client.rs] Subject: [pbs-devel] [PATCH v3 00/20] Single file restore for VM images X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Mar 2021 10:22:31 -0000 Implements CLI-based single file and directory restore for both pxar.didx archives (containers, hosts) and img.fidx (VMs, raw block devices). The design for VM restore uses a small virtual machine that the host communicates with via virtio-vsock. This is encapsuled into a new package called "proxmox-file-restore", providing a binary of the same name. A second package is provided in a new git repository[0] called "proxmox-backup-restore-image", providing a minimal kernel image and a base initramfs (without the daemon, which is included in proxmox-file-restore). Dependency bump in proxmox-backup for pxar is required. Tested with ext4 and NTFS VMs, but theoretically includes support for many more filesystems. Known issues/Missing features: * GUI/PVE support * PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore * ZFS/LVM/md/... support * shell auto-complete for "proxmox-file-restore" doesn't seem to work (and I don't know why...) [0] now already public at: https://git.proxmox.com/?p=proxmox-backup-restore-image.git;a=summary v3: * rebase on master * pxar: fix usage of assume_init (Wolfgang) * fix crash with '--output-format json' in 'proxmox-file-restore status' * make ApiAuth a single trait, makes for less generic-creep (Wolfgang) * redo extract_sub_dir for sequential decoders (Wolfgang) * fix Filesystems::scan in daemon/disk.rs for zfs (Wolfgang) * some minor code cleanups v2: * rebase on master * drop applied patches * pxar: make contents() call available without tokio-io feature (Wolfgang) * pxar: drop peek() implementation, rework extractor to cope (Wolfgang) * only move necessary functions to new key_source.rs (Dietmar) * implement static ticket-based authentication for VMs, as relying on ports <1024 does not guarantee security (Dietmar, Wolfgang) * allow running proxmox-file-restore as regular user by providing setuid-binary to start QEMU VMs (setgid kvm is not enough because of /dev/vhost-vsock) (Dietmar, Fabian) * update debian/* with new proxmox-backup-restore-image naming (Thomas) * encode zip file directly on the VM, only encode pxar when requested (Dominik) * use tokio task in watchdog, instead of alarm() (Wolfgang) pxar: Stefan Reiter (1): decoder/aio: add contents() and content_size() calls src/decoder/aio.rs | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) proxmox-backup: Dominik Csapak (1): file-restore: add binary and basic commands Stefan Reiter (18): vsock_client: remove wrong comment vsock_client: remove some &mut restrictions and rustfmt vsock_client: support authorization header proxmox_client_tools: move common key related functions to key_source.rs file-restore: allow specifying output-format server/rest: extract auth to seperate module server/rest: add ApiAuth trait to make user auth generic file-restore-daemon: add binary with virtio-vsock API server file-restore-daemon: add watchdog module file-restore-daemon: add disk module add tools/cpio encoding module file-restore: add qemu-helper setuid binary file-restore: add basic VM/block device support debian/client: add postinst hook to rebuild file-restore initramfs file-restore(-daemon): implement list API pxar/extract: add sequential variant of extract_sub_dir tools/zip: add zip_directory helper file-restore: add 'extract' command for VM file restore Cargo.toml | 5 +- Makefile | 22 +- debian/control | 14 + debian/control.in | 11 + debian/proxmox-file-restore.bash-completion | 1 + debian/proxmox-file-restore.bc | 8 + debian/proxmox-file-restore.install | 5 + debian/proxmox-file-restore.postinst | 63 ++ debian/proxmox-file-restore.triggers | 1 + debian/rules | 9 +- docs/Makefile | 10 +- docs/command-line-tools.rst | 5 + docs/proxmox-file-restore/description.rst | 3 + docs/proxmox-file-restore/man1.rst | 28 + src/api2.rs | 2 +- src/api2/types/file_restore.rs | 15 + src/api2/types/mod.rs | 3 + src/bin/proxmox-backup-api.rs | 13 +- src/bin/proxmox-backup-client.rs | 453 +------------- src/bin/proxmox-backup-proxy.rs | 7 +- src/bin/proxmox-file-restore.rs | 456 ++++++++++++++ src/bin/proxmox-restore-daemon.rs | 124 ++++ src/bin/proxmox-restore-qemu-helper.rs | 372 ++++++++++++ src/bin/proxmox_backup_client/benchmark.rs | 4 +- src/bin/proxmox_backup_client/catalog.rs | 3 +- src/bin/proxmox_backup_client/key.rs | 112 +--- src/bin/proxmox_backup_client/mod.rs | 28 - src/bin/proxmox_backup_client/mount.rs | 4 +- src/bin/proxmox_backup_client/snapshot.rs | 4 +- src/bin/proxmox_client_tools/key_source.rs | 573 ++++++++++++++++++ src/bin/proxmox_client_tools/mod.rs | 65 +- src/bin/proxmox_file_restore/block_driver.rs | 206 +++++++ .../proxmox_file_restore/block_driver_qemu.rs | 362 +++++++++++ src/bin/proxmox_file_restore/mod.rs | 5 + src/bin/proxmox_restore_daemon/api.rs | 369 +++++++++++ src/bin/proxmox_restore_daemon/auth.rs | 45 ++ src/bin/proxmox_restore_daemon/disk.rs | 329 ++++++++++ src/bin/proxmox_restore_daemon/mod.rs | 11 + src/bin/proxmox_restore_daemon/watchdog.rs | 41 ++ src/buildcfg.rs | 21 + src/client/vsock_client.rs | 78 +-- src/pxar/extract.rs | 316 +++++++--- src/pxar/mod.rs | 5 +- src/server.rs | 2 + src/server/auth.rs | 140 +++++ src/server/config.rs | 13 +- src/server/rest.rs | 130 +--- src/tools.rs | 1 + src/tools/cpio.rs | 73 +++ src/tools/zip.rs | 77 +++ zsh-completions/_proxmox-file-restore | 13 + 51 files changed, 3796 insertions(+), 864 deletions(-) create mode 100644 debian/proxmox-file-restore.bash-completion create mode 100644 debian/proxmox-file-restore.bc create mode 100644 debian/proxmox-file-restore.install create mode 100755 debian/proxmox-file-restore.postinst create mode 100644 debian/proxmox-file-restore.triggers create mode 100644 docs/proxmox-file-restore/description.rst create mode 100644 docs/proxmox-file-restore/man1.rst create mode 100644 src/api2/types/file_restore.rs create mode 100644 src/bin/proxmox-file-restore.rs create mode 100644 src/bin/proxmox-restore-daemon.rs create mode 100644 src/bin/proxmox-restore-qemu-helper.rs create mode 100644 src/bin/proxmox_client_tools/key_source.rs create mode 100644 src/bin/proxmox_file_restore/block_driver.rs create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs create mode 100644 src/bin/proxmox_file_restore/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/api.rs create mode 100644 src/bin/proxmox_restore_daemon/auth.rs create mode 100644 src/bin/proxmox_restore_daemon/disk.rs create mode 100644 src/bin/proxmox_restore_daemon/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs create mode 100644 src/server/auth.rs create mode 100644 src/tools/cpio.rs create mode 100644 zsh-completions/_proxmox-file-restore -- 2.20.1