From: Stefan Reiter <s.reiter@proxmox.com>
To: pve-devel@lists.proxmox.com, pbs-devel@lists.proxmox.com
Subject: [pve-devel] [PATCH v2 00/11] live-restore for PBS snapshots
Date: Wed, 3 Mar 2021 10:56:01 +0100 [thread overview]
Message-ID: <20210303095612.7475-1-s.reiter@proxmox.com> (raw)
First two patches are unrelated cleanups to QEMU patches-in-patches, patch 3
depends on them though.
v2:
* incorporated review feedback on qemu-server patches
* updated and rebased QEMU patches for 5.2
* fixed live-restore to ceph RBD - note that restoring to user-space Ceph is
still rather slow, I suspect it's the same bug that effects snapshot writing
as well, bigger buffers make it only slightly better
Original cover letter:
"live-restore" allows starting a VM immediately from a backup snapshot, no
waiting for a long restore process. This is made possible with QEMU backing
images, i.e. data is read from the backup which is attached to the VM as a
drive, but new data is written to the destination, while a background process
('block-stream') copies over data in a linear fashion as well.
QEMU backing images are normally only supported for qcow2 images, but since the
destination always starts out empty, we can use a dirty bitmap to achieve the
same effect - this is implemented as the 'alloc-track' driver in the 'qemu' part
of the series.
The Rust part of the equation is adjusted to provide (quiet a lot) more caching,
as mixing random read/write from the guest with the linear reads from the
background process (both of which may use read sizes smaller or bigger than a
single chunk) would thrash performance without large buffers.
I've marked the feature as 'experimental' in the GUI for now, as I'm sure there
are a lot of edge cases I've missed to test, and there's also the possibility of
data loss, since anything the VM writes during the restore is removed if the
restore fails.
pve-qemu: Stefan Reiter (3):
clean up pve/ patches by merging
move bitmap-mirror patches to seperate folder
add alloc-track block driver patch
...-support-for-sync-bitmap-mode-never.patch} | 30 +-
...support-for-conditional-and-always-.patch} | 0
...heck-for-bitmap-mode-without-bitmap.patch} | 4 +-
...to-bdrv_dirty_bitmap_merge_internal.patch} | 0
...-iotests-add-test-for-bitmap-mirror.patch} | 0
...0006-mirror-move-some-checks-to-qmp.patch} | 4 +-
...le-posix-make-locking-optiono-on-cre.patch | 4 +-
...-Backup-add-backup-dump-block-driver.patch | 2 +-
...ckup-proxmox-backup-patches-for-qemu.patch | 671 ++++++-------
...estore-new-command-to-restore-from-p.patch | 18 +-
...rty-bitmap-tracking-for-incremental.patch} | 80 +-
...-coroutines-to-fix-AIO-freeze-cleanu.patch | 914 ------------------
.../pve/0031-PVE-various-PBS-fixes.patch | 218 +++++
...-driver-to-map-backup-archives-into.patch} | 0
...d-query_proxmox_support-QMP-command.patch} | 4 +-
...-add-query-pbs-bitmap-info-QMP-call.patch} | 0
...t-stderr-to-journal-when-daemonized.patch} | 0
...-sequential-job-transaction-support.patch} | 20 +-
...transaction-to-synchronize-job-stat.patch} | 0
...block-on-finishing-and-cleanup-crea.patch} | 245 +++--
...name-incremental-to-use-dirty-bitmap.patch | 126 ---
...grate-dirty-bitmap-state-via-savevm.patch} | 0
.../pve/0039-PVE-fixup-pbs-restore-API.patch | 44 -
...irty-counter-for-non-incremental-bac.patch | 30 -
...irty-bitmap-migrate-other-bitmaps-e.patch} | 0
...ll-back-to-open-iscsi-initiatorname.patch} | 0
...use-proxmox_backup_check_incremental.patch | 36 -
...outine-QMP-for-backup-cancel_backup.patch} | 0
...ckup-add-compress-and-encrypt-option.patch | 103 --
... => 0043-PBS-add-master-key-support.patch} | 0
...st-path-reads-without-allocation-if-.patch | 52 +
...PVE-block-stream-increase-chunk-size.patch | 23 +
...issing-crypt-and-compress-parameters.patch | 43 -
...rite-callback-with-big-blocks-correc.patch | 76 --
...accept-NULL-qiov-in-bdrv_pad_request.patch | 42 +
...-block-handling-to-PBS-dump-callback.patch | 85 --
.../0047-block-add-alloc-track-driver.patch | 380 ++++++++
...n-up-error-handling-for-create_backu.patch | 187 ----
...-multiple-CREATED-jobs-in-sequential.patch | 39 -
debian/patches/series | 54 +-
40 files changed, 1356 insertions(+), 2178 deletions(-)
rename debian/patches/{pve/0031-drive-mirror-add-support-for-sync-bitmap-mode-never.patch => bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch} (96%)
rename debian/patches/{pve/0032-drive-mirror-add-support-for-conditional-and-always-.patch => bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch} (100%)
rename debian/patches/{pve/0033-mirror-add-check-for-bitmap-mode-without-bitmap.patch => bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch} (90%)
rename debian/patches/{pve/0034-mirror-switch-to-bdrv_dirty_bitmap_merge_internal.patch => bitmap-mirror/0004-mirror-switch-to-bdrv_dirty_bitmap_merge_internal.patch} (100%)
rename debian/patches/{pve/0035-iotests-add-test-for-bitmap-mirror.patch => bitmap-mirror/0005-iotests-add-test-for-bitmap-mirror.patch} (100%)
rename debian/patches/{pve/0036-mirror-move-some-checks-to-qmp.patch => bitmap-mirror/0006-mirror-move-some-checks-to-qmp.patch} (99%)
rename debian/patches/pve/{0037-PVE-Backup-Add-dirty-bitmap-tracking-for-incremental.patch => 0030-PVE-Backup-Add-dirty-bitmap-tracking-for-incremental.patch} (88%)
delete mode 100644 debian/patches/pve/0030-PVE-Backup-avoid-coroutines-to-fix-AIO-freeze-cleanu.patch
create mode 100644 debian/patches/pve/0031-PVE-various-PBS-fixes.patch
rename debian/patches/pve/{0043-PVE-Add-PBS-block-driver-to-map-backup-archives-into.patch => 0032-PVE-Add-PBS-block-driver-to-map-backup-archives-into.patch} (100%)
rename debian/patches/pve/{0044-PVE-add-query_proxmox_support-QMP-command.patch => 0033-PVE-add-query_proxmox_support-QMP-command.patch} (94%)
rename debian/patches/pve/{0048-PVE-add-query-pbs-bitmap-info-QMP-call.patch => 0034-PVE-add-query-pbs-bitmap-info-QMP-call.patch} (100%)
rename debian/patches/pve/{0049-PVE-redirect-stderr-to-journal-when-daemonized.patch => 0035-PVE-redirect-stderr-to-journal-when-daemonized.patch} (100%)
rename debian/patches/pve/{0050-PVE-Add-sequential-job-transaction-support.patch => 0036-PVE-Add-sequential-job-transaction-support.patch} (75%)
rename debian/patches/pve/{0051-PVE-Backup-Use-a-transaction-to-synchronize-job-stat.patch => 0037-PVE-Backup-Use-a-transaction-to-synchronize-job-stat.patch} (100%)
rename debian/patches/pve/{0052-PVE-Backup-Use-more-coroutines-and-don-t-block-on-fi.patch => 0038-PVE-Backup-Don-t-block-on-finishing-and-cleanup-crea.patch} (63%)
delete mode 100644 debian/patches/pve/0038-PVE-backup-rename-incremental-to-use-dirty-bitmap.patch
rename debian/patches/pve/{0054-PVE-Migrate-dirty-bitmap-state-via-savevm.patch => 0039-PVE-Migrate-dirty-bitmap-state-via-savevm.patch} (100%)
delete mode 100644 debian/patches/pve/0039-PVE-fixup-pbs-restore-API.patch
delete mode 100644 debian/patches/pve/0040-PVE-always-set-dirty-counter-for-non-incremental-bac.patch
rename debian/patches/pve/{0055-migration-block-dirty-bitmap-migrate-other-bitmaps-e.patch => 0040-migration-block-dirty-bitmap-migrate-other-bitmaps-e.patch} (100%)
rename debian/patches/pve/{0057-PVE-fall-back-to-open-iscsi-initiatorname.patch => 0041-PVE-fall-back-to-open-iscsi-initiatorname.patch} (100%)
delete mode 100644 debian/patches/pve/0041-PVE-use-proxmox_backup_check_incremental.patch
rename debian/patches/pve/{0058-PVE-Use-coroutine-QMP-for-backup-cancel_backup.patch => 0042-PVE-Use-coroutine-QMP-for-backup-cancel_backup.patch} (100%)
delete mode 100644 debian/patches/pve/0042-PVE-fixup-pbs-backup-add-compress-and-encrypt-option.patch
rename debian/patches/pve/{0059-PBS-add-master-key-support.patch => 0043-PBS-add-master-key-support.patch} (100%)
create mode 100644 debian/patches/pve/0044-PVE-block-pbs-fast-path-reads-without-allocation-if-.patch
create mode 100644 debian/patches/pve/0045-PVE-block-stream-increase-chunk-size.patch
delete mode 100644 debian/patches/pve/0045-pbs-fix-missing-crypt-and-compress-parameters.patch
delete mode 100644 debian/patches/pve/0046-PVE-handle-PBS-write-callback-with-big-blocks-correc.patch
create mode 100644 debian/patches/pve/0046-block-io-accept-NULL-qiov-in-bdrv_pad_request.patch
delete mode 100644 debian/patches/pve/0047-PVE-add-zero-block-handling-to-PBS-dump-callback.patch
create mode 100644 debian/patches/pve/0047-block-add-alloc-track-driver.patch
delete mode 100644 debian/patches/pve/0053-PVE-fix-and-clean-up-error-handling-for-create_backu.patch
delete mode 100644 debian/patches/pve/0056-PVE-fix-aborting-multiple-CREATED-jobs-in-sequential.patch
proxmox-backup: Stefan Reiter (1):
RemoteChunkReader: add LRU cached variant
src/bin/proxmox_backup_client/mount.rs | 4 +-
src/client/remote_chunk_reader.rs | 77 ++++++++++++++++++++------
2 files changed, 62 insertions(+), 19 deletions(-)
proxmox-backup-qemu: Stefan Reiter (1):
access: use bigger cache and LRU chunk reader
src/restore.rs | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
qemu-server: Stefan Reiter (5):
make qemu_drive_mirror_monitor more generic
cfg2cmd: allow PBS snapshots as backing files for drives
enable live-restore for PBS
extract register_qmeventd_handle to QemuServer.pm
live-restore: register qmeventd handle
PVE/API2/Qemu.pm | 14 +-
PVE/QemuServer.pm | 297 ++++++++++++++++++++++++++++++++-------
PVE/VZDump/QemuServer.pm | 32 +----
3 files changed, 259 insertions(+), 84 deletions(-)
manager: Stefan Reiter (1):
ui: restore: add live-restore checkbox
www/manager6/grid/BackupView.js | 6 ++++-
www/manager6/storage/BackupView.js | 5 +++-
www/manager6/window/Restore.js | 38 +++++++++++++++++++++++++++++-
3 files changed, 46 insertions(+), 3 deletions(-)
--
2.20.1
next reply other threads:[~2021-03-03 9:57 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-03 9:56 Stefan Reiter [this message]
2021-03-03 9:56 ` [pve-devel] [PATCH v2 pve-qemu 01/11] clean up pve/ patches by merging Stefan Reiter
2021-03-03 16:32 ` [pve-devel] applied: " Thomas Lamprecht
2021-03-03 9:56 ` [pve-devel] [PATCH v2 pve-qemu 02/11] move bitmap-mirror patches to seperate folder Stefan Reiter
2021-03-03 16:32 ` [pve-devel] applied: " Thomas Lamprecht
2021-03-03 9:56 ` [pve-devel] [PATCH v2 pve-qemu 03/11] add alloc-track block driver patch Stefan Reiter
2021-03-15 14:14 ` Wolfgang Bumiller
2021-03-15 15:41 ` [pve-devel] [PATCH pve-qemu v3] " Stefan Reiter
2021-03-16 19:57 ` [pve-devel] applied: " Thomas Lamprecht
2021-03-03 9:56 ` [pve-devel] [PATCH v2 proxmox-backup 04/11] RemoteChunkReader: add LRU cached variant Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 proxmox-backup-qemu 05/11] access: use bigger cache and LRU chunk reader Stefan Reiter
2021-03-16 20:17 ` Thomas Lamprecht
2021-03-17 13:37 ` Stefan Reiter
2021-03-17 13:59 ` Thomas Lamprecht
2021-03-17 16:03 ` [pve-devel] [pbs-devel] " Dietmar Maurer
2021-03-03 9:56 ` [pve-devel] [PATCH v2 qemu-server 06/11] make qemu_drive_mirror_monitor more generic Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 qemu-server 07/11] cfg2cmd: allow PBS snapshots as backing files for drives Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 qemu-server 08/11] enable live-restore for PBS Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 qemu-server 09/11] extract register_qmeventd_handle to QemuServer.pm Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 qemu-server 10/11] live-restore: register qmeventd handle Stefan Reiter
2021-03-03 9:56 ` [pve-devel] [PATCH v2 manager 11/11] ui: restore: add live-restore checkbox Stefan Reiter
2021-04-15 18:34 ` [pve-devel] applied: " Thomas Lamprecht
2021-03-22 11:08 ` [pve-devel] [PATCH v2 00/11] live-restore for PBS snapshots Dominic Jäger
2021-04-06 19:09 ` [pve-devel] partially-applied: " Thomas Lamprecht
2021-04-15 18:35 ` [pve-devel] " Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210303095612.7475-1-s.reiter@proxmox.com \
--to=s.reiter@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox