From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6B06C6A50F for ; Tue, 16 Feb 2021 18:07:39 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 71FD118FA4 for ; Tue, 16 Feb 2021 18:07:37 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 0005318DF0 for ; Tue, 16 Feb 2021 18:07:30 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id BD05846263 for ; Tue, 16 Feb 2021 18:07:30 +0100 (CET) From: Stefan Reiter To: pbs-devel@lists.proxmox.com Date: Tue, 16 Feb 2021 18:06:48 +0100 Message-Id: <20210216170710.31767-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.029 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [sync.rs, watchdog.rs, buildcfg.rs, api.rs, snapshot.rs, datastore.rs, api2.rs, key.rs, extract.rs, proxmox-restore-daemon.rs, aio.rs, control.in, rest.rs, proxmox-backup-client.rs, mod.rs, helpers.rs, proxmox-file-restore.rs, client.rs, catalog.rs, disk.rs] Subject: [pbs-devel] [PATCH 00/22] Single file restore for VM images X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Feb 2021 17:07:39 -0000 Implements CLI-based single file and directory restore for both pxar.didx archives (containers, hosts) and img.fidx (VMs, raw block devices). The design for VM restore uses a small virtual machine that the host communicates with via virtio-vsock. This is encapsuled into a new package called "proxmox-file-restore", providing a binary of the same name. A second package is provided in a new git repository called "proxmox-restore-vm-data", providing a minimal kernel image and a base initramfs (without the daemon, which is included in proxmox-file-restore). Requires my previously sent pxar asyncify series: https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html The first couple patches in the proxmox-backup repo are adapted versions of the ones Dominik sent to the list a while ago: https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html Dependency bump in proxmox-backup for pxar is required, though best done together with the changes from the aforementioned seperate series. Tested with ext4 and NTFS VMs, but theoretically includes support for many more filesystems (see 'config-base' in the new proxmox-restore-vm-data repository). Known issues/Missing features: * GUI/PVE support * PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore * ZFS/LVM/md/... support * shell auto-complete for "proxmox-file-restore" doesn't work (and I don't know why...) * some patches might include some sneaky rustfmt/clippy fixes that'd better fit to a previous patch, sorry for that, rebasing so many patches is annoying ;) pxar: Stefan Reiter (2): decoder/aio: add contents() and content_size() calls decoder: add peek() src/accessor/mod.rs | 3 +++ src/decoder/aio.rs | 53 +++++++++++++++++++++++++++++++++++++++++++-- src/decoder/mod.rs | 19 ++++++++++++++-- src/decoder/sync.rs | 10 ++++++++- 4 files changed, 80 insertions(+), 5 deletions(-) proxmox-restore-vm-data: Stefan Reiter (1): initial commit proxmox-backup: Dominik Csapak (5): api2/admin/datastore: refactor list_dir_content in catalog_reader api2/admin/datastore: accept "/" as path for root api2/admin/datastore: refactor create_zip into pxar/extract pxar/extract: add extract_sub_dir file-restore: add binary and basic commands Stefan Reiter (14): pxar/extract: add sequential variants to create_zip, extract_sub_dir client: extract common functions to proxmox_client_tools module proxmox_client_tools: extract 'key' from client module file-restore: allow specifying output-format rest: implement tower service for UnixStream client: add VsockClient to connect to virtio-vsock VMs file-restore-daemon: add binary with virtio-vsock API server file-restore-daemon: add watchdog module file-restore-daemon: add disk module file-restore: add basic VM/block device support file-restore: improve logging of VM with logrotate debian/client: add postinst hook to rebuild file-restore initramfs file-restore(-daemon): implement list API file-restore: add 'extract' command for VM file restore Cargo.toml | 5 +- Makefile | 18 +- debian/control | 13 + debian/control.in | 10 + debian/proxmox-backup-client.triggers | 1 + debian/proxmox-file-restore.bash-completion | 1 + debian/proxmox-file-restore.bc | 8 + debian/proxmox-file-restore.install | 4 + debian/proxmox-file-restore.postinst | 63 ++ debian/proxmox-file-restore.triggers | 1 + debian/rules | 7 +- docs/Makefile | 10 +- docs/command-line-tools.rst | 5 + docs/proxmox-file-restore/description.rst | 4 + docs/proxmox-file-restore/man1.rst | 28 + src/api2.rs | 2 +- src/api2/admin/datastore.rs | 152 +--- src/api2/helpers.rs | 31 + src/api2/types/file_restore.rs | 15 + src/api2/types/mod.rs | 46 + src/backup/catalog.rs | 26 + src/bin/proxmox-backup-client.rs | 799 +----------------- src/bin/proxmox-file-restore.rs | 484 +++++++++++ src/bin/proxmox-restore-daemon.rs | 124 +++ src/bin/proxmox_backup_client/catalog.rs | 4 +- src/bin/proxmox_backup_client/mod.rs | 30 - src/bin/proxmox_backup_client/snapshot.rs | 3 +- .../key.rs | 440 +++++++++- src/bin/proxmox_client_tools/mod.rs | 392 +++++++++ src/bin/proxmox_file_restore/block_driver.rs | 221 +++++ .../proxmox_file_restore/block_driver_qemu.rs | 478 +++++++++++ src/bin/proxmox_file_restore/mod.rs | 5 + src/bin/proxmox_restore_daemon/api.rs | 316 +++++++ src/bin/proxmox_restore_daemon/disk.rs | 341 ++++++++ src/bin/proxmox_restore_daemon/mod.rs | 9 + src/bin/proxmox_restore_daemon/watchdog.rs | 63 ++ src/buildcfg.rs | 20 + src/client.rs | 3 + src/client/vsock_client.rs | 259 ++++++ src/pxar/extract.rs | 436 +++++++++- src/pxar/mod.rs | 5 +- src/server/rest.rs | 20 + www/window/FileBrowser.js | 1 + zsh-completions/_proxmox-file-restore | 13 + 44 files changed, 3940 insertions(+), 976 deletions(-) create mode 100644 debian/proxmox-backup-client.triggers create mode 100644 debian/proxmox-file-restore.bash-completion create mode 100644 debian/proxmox-file-restore.bc create mode 100644 debian/proxmox-file-restore.install create mode 100755 debian/proxmox-file-restore.postinst create mode 100644 debian/proxmox-file-restore.triggers create mode 100644 docs/proxmox-file-restore/description.rst create mode 100644 docs/proxmox-file-restore/man1.rst create mode 100644 src/api2/types/file_restore.rs create mode 100644 src/bin/proxmox-file-restore.rs create mode 100644 src/bin/proxmox-restore-daemon.rs rename src/bin/{proxmox_backup_client => proxmox_client_tools}/key.rs (52%) create mode 100644 src/bin/proxmox_client_tools/mod.rs create mode 100644 src/bin/proxmox_file_restore/block_driver.rs create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs create mode 100644 src/bin/proxmox_file_restore/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/api.rs create mode 100644 src/bin/proxmox_restore_daemon/disk.rs create mode 100644 src/bin/proxmox_restore_daemon/mod.rs create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs create mode 100644 src/client/vsock_client.rs create mode 100644 zsh-completions/_proxmox-file-restore -- 2.20.1