From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 13FC46A15C for ; Wed, 24 Mar 2021 16:21:09 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4A3C3C2C9 for ; Wed, 24 Mar 2021 16:21:07 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id C687FC1CF for ; Wed, 24 Mar 2021 16:21:01 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 856044647A for ; Wed, 24 Mar 2021 16:21:01 +0100 (CET) From: Stefan Reiter To: pbs-devel@lists.proxmox.com Date: Wed, 24 Mar 2021 16:18:07 +0100 Message-Id: <20210324151827.26200-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.022 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [key.rs, disk.rs, benchmark.rs, config.rs, buildcfg.rs, api2.rs, proxmox-backup-proxy.rs, aio.rs, watchdog.rs, mount.rs, auth.rs, proxmox-backup-api.rs, proxmox-file-restore.rs, catalog.rs, cpio.rs, proxmox-backup-client.rs, tools.rs, zip.rs, server.rs, extract.rs] Subject: [pbs-devel] [PATCH v2 00/20] Single file restore for VM images X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 15:21:09 -0000 Implements CLI-based single file and directory restore for both pxar.didx archives (containers, hosts) and img.fidx (VMs, raw block devices). The design for VM restore uses a small virtual machine that the host communicates with via virtio-vsock. This is encapsuled into a new package called "proxmox-file-restore", providing a binary of the same name. A second package is provided in a new git repository called "proxmox-backup-restore-image", providing a minimal kernel image and a base initramfs (without the daemon, which is included in proxmox-file-restore). The first couple patches in the proxmox-backup repo are adapted versions of the ones Dominik sent to the list a while ago: https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html Dependency bump in proxmox-backup for pxar is required. Tested with ext4 and NTFS VMs, but theoretically includes support for many more filesystems. Known issues/Missing features: * GUI/PVE support * PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore * ZFS/LVM/md/... support * shell auto-complete for "proxmox-file-restore" doesn't seem to work (and I don't know why...) * some patches might include some sneaky rustfmt/clippy fixes that'd better fit to a previous patch, sorry for that, rebasing so many patches is annoying ;) v2: * rebase on master * drop applied patches * pxar: make contents() call available without tokio-io feature (Wolfgang) * pxar: drop peek() implementation, rework extractor to cope (Wolfgang) * only move necessary functions to new key_source.rs (Dietmar) * implement static ticket-based authentication for VMs, as relying on ports <1024 does not guarantee security (Dietmar, Wolfgang) * allow running proxmox-file-restore as regular user by providing setuid-binary to start QEMU VMs (setgid kvm is not enough because of /dev/vhost-vsock) (Dietmar, Fabian) * update debian/* with new proxmox-backup-restore-image naming (Thomas) * encode zip file directly on the VM, only encode pxar when requested (Dominik) * use tokio task in watchdog, instead of alarm() (Wolfgang) ...plus some reshuffling of patches because of all the changes from above, hope I didn't miss anything big ;) pxar: Stefan Reiter (1): decoder/aio: add contents() and content_size() calls src/decoder/aio.rs | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) proxmox-backup: Dominik Csapak (1): file-restore: add binary and basic commands Stefan Reiter (18): vsock_client: remove wrong comment vsock_client: remove some &mut restrictions and rustfmt vsock_client: support authorization header proxmox_client_tools: move common key related functions to key_source.rs file-restore: allow specifying output-format server/rest: extract auth to seperate module server/rest: add ApiAuth trait to make user auth generic file-restore-daemon: add binary with virtio-vsock API server file-restore-daemon: add watchdog module file-restore-daemon: add disk module add tools/cpio encoding module file-restore: add qemu-helper setuid binary file-restore: add basic VM/block device support debian/client: add postinst hook to rebuild file-restore initramfs file-restore(-daemon): implement list API pxar/extract: add sequential variant to extract_sub_dir tools/zip: add zip_directory helper file-restore: add 'extract' command for VM file restore Cargo.toml | 5 +- Makefile | 22 +- debian/control | 14 + debian/control.in | 11 + debian/proxmox-file-restore.bash-completion | 1 + debian/proxmox-file-restore.bc | 8 + debian/proxmox-file-restore.install | 5 + debian/proxmox-file-restore.postinst | 63 ++ debian/proxmox-file-restore.triggers | 1 + debian/rules | 9 +- docs/Makefile | 10 +- docs/command-line-tools.rst | 5 + docs/proxmox-file-restore/description.rst | 3 + docs/proxmox-file-restore/man1.rst | 28 + src/api2.rs | 2 +- src/api2/types/file_restore.rs | 15 + src/api2/types/mod.rs | 3 + src/bin/proxmox-backup-api.rs | 13 +- src/bin/proxmox-backup-client.rs | 453 +------------- src/bin/proxmox-backup-proxy.rs | 7 +- src/bin/proxmox-file-restore.rs | 446 ++++++++++++++ src/bin/proxmox-restore-daemon.rs | 124 ++++ src/bin/proxmox-restore-qemu-helper.rs | 372 ++++++++++++ src/bin/proxmox_backup_client/benchmark.rs | 4 +- src/bin/proxmox_backup_client/catalog.rs | 3 +- src/bin/proxmox_backup_client/key.rs | 112 +--- src/bin/proxmox_backup_client/mod.rs | 28 - src/bin/proxmox_backup_client/mount.rs | 4 +- src/bin/proxmox_backup_client/snapshot.rs | 4 +- src/bin/proxmox_client_tools/key_source.rs | 573 ++++++++++++++++++ src/bin/proxmox_client_tools/mod.rs | 65 +- src/bin/proxmox_file_restore/block_driver.rs | 206 +++++++ .../proxmox_file_restore/block_driver_qemu.rs | 362 +++++++++++ src/bin/proxmox_file_restore/mod.rs | 5 + src/bin/proxmox_restore_daemon/api.rs | 369 +++++++++++ src/bin/proxmox_restore_daemon/auth.rs | 48 ++ src/bin/proxmox_restore_daemon/disk.rs | 341 +++++++++++ src/bin/proxmox_restore_daemon/mod.rs | 11 + src/bin/proxmox_restore_daemon/watchdog.rs | 41 ++ src/buildcfg.rs | 21 + src/client/vsock_client.rs | 78 +-- src/pxar/extract.rs | 214 +++++-- src/pxar/mod.rs | 5 +- src/server.rs | 2 + src/server/auth.rs | 124 ++++ src/server/config.rs | 17 +- src/server/rest.rs | 141 +---- src/tools.rs | 1 + src/tools/cpio.rs | 73 +++ src/tools/zip.rs | 77 +++ zsh-completions/_proxmox-file-restore | 13 + 51 files changed, 3734 insertions(+), 828 deletions(-) create mode 100644 debian/proxmox-file-restore.bash-completion create mode 100644 debian/proxmox-file-restore.bc create mode 100644 debian/proxmox-file-restore.install create mode 100755 debian/proxmox-file-restore.postinst create mode 100644 debian/proxmox-file-restore.triggers create mode 100644 docs/proxmox-file-restore/description.rst create mode 100644 docs/proxmox-file-restore/man1.rst create mode 100644 src/api2/types/file_restore.rs create mode 100644 src/bin/proxmox-file-restore.rs create mode 100644 src/bin/proxmox-restore-daemon.rs create mode 100644 src/bin/proxmox-restore-qemu-helper.rs create mode 100644 src/bin/proxmox_client_tools/key_source.rs create mode 100644 src/bin/proxmox_file_restore/block_driver.rs create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs create mode 100644 src/bin/proxmox_file_restore/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/api.rs create mode 100644 src/bin/proxmox_restore_daemon/auth.rs create mode 100644 src/bin/proxmox_restore_daemon/disk.rs create mode 100644 src/bin/proxmox_restore_daemon/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs create mode 100644 src/server/auth.rs create mode 100644 src/tools/cpio.rs create mode 100644 zsh-completions/_proxmox-file-restore -- 2.20.1