From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <s.reiter@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 6B06C6A50F
for <pbs-devel@lists.proxmox.com>; Tue, 16 Feb 2021 18:07:39 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 71FD118FA4
for <pbs-devel@lists.proxmox.com>; Tue, 16 Feb 2021 18:07:37 +0100 (CET)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[212.186.127.180])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id 0005318DF0
for <pbs-devel@lists.proxmox.com>; Tue, 16 Feb 2021 18:07:30 +0100 (CET)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id BD05846263
for <pbs-devel@lists.proxmox.com>; Tue, 16 Feb 2021 18:07:30 +0100 (CET)
From: Stefan Reiter <s.reiter@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Tue, 16 Feb 2021 18:06:48 +0100
Message-Id: <20210216170710.31767-1-s.reiter@proxmox.com>
X-Mailer: git-send-email 2.20.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.029 Adjusted score from AWL reputation of From: address
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/,
medium trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [sync.rs, watchdog.rs, buildcfg.rs, api.rs, snapshot.rs,
datastore.rs, api2.rs, key.rs, extract.rs, proxmox-restore-daemon.rs, aio.rs,
control.in, rest.rs, proxmox-backup-client.rs, mod.rs, helpers.rs,
proxmox-file-restore.rs, client.rs, catalog.rs, disk.rs]
Subject: [pbs-devel] [PATCH 00/22] Single file restore for VM images
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 16 Feb 2021 17:07:39 -0000
Implements CLI-based single file and directory restore for both pxar.didx
archives (containers, hosts) and img.fidx (VMs, raw block devices). The design
for VM restore uses a small virtual machine that the host communicates with via
virtio-vsock.
This is encapsuled into a new package called "proxmox-file-restore", providing a
binary of the same name. A second package is provided in a new git repository
called "proxmox-restore-vm-data", providing a minimal kernel image and a base
initramfs (without the daemon, which is included in proxmox-file-restore).
Requires my previously sent pxar asyncify series:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html
The first couple patches in the proxmox-backup repo are adapted versions of the
ones Dominik sent to the list a while ago:
https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001788.html
Dependency bump in proxmox-backup for pxar is required, though best done
together with the changes from the aforementioned seperate series.
Tested with ext4 and NTFS VMs, but theoretically includes support for many more
filesystems (see 'config-base' in the new proxmox-restore-vm-data repository).
Known issues/Missing features:
* GUI/PVE support
* PBS_PASSWORD/PBS_FINGERPRINT currently have to be set manually for VM restore
* ZFS/LVM/md/... support
* shell auto-complete for "proxmox-file-restore" doesn't work (and I don't know
why...)
* some patches might include some sneaky rustfmt/clippy fixes that'd better fit
to a previous patch, sorry for that, rebasing so many patches is annoying ;)
pxar: Stefan Reiter (2):
decoder/aio: add contents() and content_size() calls
decoder: add peek()
src/accessor/mod.rs | 3 +++
src/decoder/aio.rs | 53 +++++++++++++++++++++++++++++++++++++++++++--
src/decoder/mod.rs | 19 ++++++++++++++--
src/decoder/sync.rs | 10 ++++++++-
4 files changed, 80 insertions(+), 5 deletions(-)
proxmox-restore-vm-data: Stefan Reiter (1):
initial commit
proxmox-backup: Dominik Csapak (5):
api2/admin/datastore: refactor list_dir_content in catalog_reader
api2/admin/datastore: accept "/" as path for root
api2/admin/datastore: refactor create_zip into pxar/extract
pxar/extract: add extract_sub_dir
file-restore: add binary and basic commands
Stefan Reiter (14):
pxar/extract: add sequential variants to create_zip, extract_sub_dir
client: extract common functions to proxmox_client_tools module
proxmox_client_tools: extract 'key' from client module
file-restore: allow specifying output-format
rest: implement tower service for UnixStream
client: add VsockClient to connect to virtio-vsock VMs
file-restore-daemon: add binary with virtio-vsock API server
file-restore-daemon: add watchdog module
file-restore-daemon: add disk module
file-restore: add basic VM/block device support
file-restore: improve logging of VM with logrotate
debian/client: add postinst hook to rebuild file-restore initramfs
file-restore(-daemon): implement list API
file-restore: add 'extract' command for VM file restore
Cargo.toml | 5 +-
Makefile | 18 +-
debian/control | 13 +
debian/control.in | 10 +
debian/proxmox-backup-client.triggers | 1 +
debian/proxmox-file-restore.bash-completion | 1 +
debian/proxmox-file-restore.bc | 8 +
debian/proxmox-file-restore.install | 4 +
debian/proxmox-file-restore.postinst | 63 ++
debian/proxmox-file-restore.triggers | 1 +
debian/rules | 7 +-
docs/Makefile | 10 +-
docs/command-line-tools.rst | 5 +
docs/proxmox-file-restore/description.rst | 4 +
docs/proxmox-file-restore/man1.rst | 28 +
src/api2.rs | 2 +-
src/api2/admin/datastore.rs | 152 +---
src/api2/helpers.rs | 31 +
src/api2/types/file_restore.rs | 15 +
src/api2/types/mod.rs | 46 +
src/backup/catalog.rs | 26 +
src/bin/proxmox-backup-client.rs | 799 +-----------------
src/bin/proxmox-file-restore.rs | 484 +++++++++++
src/bin/proxmox-restore-daemon.rs | 124 +++
src/bin/proxmox_backup_client/catalog.rs | 4 +-
src/bin/proxmox_backup_client/mod.rs | 30 -
src/bin/proxmox_backup_client/snapshot.rs | 3 +-
.../key.rs | 440 +++++++++-
src/bin/proxmox_client_tools/mod.rs | 392 +++++++++
src/bin/proxmox_file_restore/block_driver.rs | 221 +++++
.../proxmox_file_restore/block_driver_qemu.rs | 478 +++++++++++
src/bin/proxmox_file_restore/mod.rs | 5 +
src/bin/proxmox_restore_daemon/api.rs | 316 +++++++
src/bin/proxmox_restore_daemon/disk.rs | 341 ++++++++
src/bin/proxmox_restore_daemon/mod.rs | 9 +
src/bin/proxmox_restore_daemon/watchdog.rs | 63 ++
src/buildcfg.rs | 20 +
src/client.rs | 3 +
src/client/vsock_client.rs | 259 ++++++
src/pxar/extract.rs | 436 +++++++++-
src/pxar/mod.rs | 5 +-
src/server/rest.rs | 20 +
www/window/FileBrowser.js | 1 +
zsh-completions/_proxmox-file-restore | 13 +
44 files changed, 3940 insertions(+), 976 deletions(-)
create mode 100644 debian/proxmox-backup-client.triggers
create mode 100644 debian/proxmox-file-restore.bash-completion
create mode 100644 debian/proxmox-file-restore.bc
create mode 100644 debian/proxmox-file-restore.install
create mode 100755 debian/proxmox-file-restore.postinst
create mode 100644 debian/proxmox-file-restore.triggers
create mode 100644 docs/proxmox-file-restore/description.rst
create mode 100644 docs/proxmox-file-restore/man1.rst
create mode 100644 src/api2/types/file_restore.rs
create mode 100644 src/bin/proxmox-file-restore.rs
create mode 100644 src/bin/proxmox-restore-daemon.rs
rename src/bin/{proxmox_backup_client => proxmox_client_tools}/key.rs (52%)
create mode 100644 src/bin/proxmox_client_tools/mod.rs
create mode 100644 src/bin/proxmox_file_restore/block_driver.rs
create mode 100644 src/bin/proxmox_file_restore/block_driver_qemu.rs
create mode 100644 src/bin/proxmox_file_restore/mod.rs
create mode 100644 src/bin/proxmox_restore_daemon/api.rs
create mode 100644 src/bin/proxmox_restore_daemon/disk.rs
create mode 100644 src/bin/proxmox_restore_daemon/mod.rs
create mode 100644 src/bin/proxmox_restore_daemon/watchdog.rs
create mode 100644 src/client/vsock_client.rs
create mode 100644 zsh-completions/_proxmox-file-restore
--
2.20.1