From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 43A1F69BDE; Wed, 3 Mar 2021 10:57:55 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1FC6B34713; Wed, 3 Mar 2021 10:57:25 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 30BB8346DA; Wed, 3 Mar 2021 10:57:23 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id F3F4D4584C; Wed, 3 Mar 2021 10:57:22 +0100 (CET) From: Stefan Reiter To: pve-devel@lists.proxmox.com, pbs-devel@lists.proxmox.com Date: Wed, 3 Mar 2021 10:56:01 +0100 Message-Id: <20210303095612.7475-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.025 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs, mount.rs, qemuserver.pm, qemu.pm] Subject: [pve-devel] [PATCH v2 00/11] live-restore for PBS snapshots X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2021 09:57:55 -0000 First two patches are unrelated cleanups to QEMU patches-in-patches, patch 3 depends on them though. v2: * incorporated review feedback on qemu-server patches * updated and rebased QEMU patches for 5.2 * fixed live-restore to ceph RBD - note that restoring to user-space Ceph is still rather slow, I suspect it's the same bug that effects snapshot writing as well, bigger buffers make it only slightly better Original cover letter: "live-restore" allows starting a VM immediately from a backup snapshot, no waiting for a long restore process. This is made possible with QEMU backing images, i.e. data is read from the backup which is attached to the VM as a drive, but new data is written to the destination, while a background process ('block-stream') copies over data in a linear fashion as well. QEMU backing images are normally only supported for qcow2 images, but since the destination always starts out empty, we can use a dirty bitmap to achieve the same effect - this is implemented as the 'alloc-track' driver in the 'qemu' part of the series. The Rust part of the equation is adjusted to provide (quiet a lot) more caching, as mixing random read/write from the guest with the linear reads from the background process (both of which may use read sizes smaller or bigger than a single chunk) would thrash performance without large buffers. I've marked the feature as 'experimental' in the GUI for now, as I'm sure there are a lot of edge cases I've missed to test, and there's also the possibility of data loss, since anything the VM writes during the restore is removed if the restore fails. pve-qemu: Stefan Reiter (3): clean up pve/ patches by merging move bitmap-mirror patches to seperate folder add alloc-track block driver patch ...-support-for-sync-bitmap-mode-never.patch} | 30 +- ...support-for-conditional-and-always-.patch} | 0 ...heck-for-bitmap-mode-without-bitmap.patch} | 4 +- ...to-bdrv_dirty_bitmap_merge_internal.patch} | 0 ...-iotests-add-test-for-bitmap-mirror.patch} | 0 ...0006-mirror-move-some-checks-to-qmp.patch} | 4 +- ...le-posix-make-locking-optiono-on-cre.patch | 4 +- ...-Backup-add-backup-dump-block-driver.patch | 2 +- ...ckup-proxmox-backup-patches-for-qemu.patch | 671 ++++++------- ...estore-new-command-to-restore-from-p.patch | 18 +- ...rty-bitmap-tracking-for-incremental.patch} | 80 +- ...-coroutines-to-fix-AIO-freeze-cleanu.patch | 914 ------------------ .../pve/0031-PVE-various-PBS-fixes.patch | 218 +++++ ...-driver-to-map-backup-archives-into.patch} | 0 ...d-query_proxmox_support-QMP-command.patch} | 4 +- ...-add-query-pbs-bitmap-info-QMP-call.patch} | 0 ...t-stderr-to-journal-when-daemonized.patch} | 0 ...-sequential-job-transaction-support.patch} | 20 +- ...transaction-to-synchronize-job-stat.patch} | 0 ...block-on-finishing-and-cleanup-crea.patch} | 245 +++-- ...name-incremental-to-use-dirty-bitmap.patch | 126 --- ...grate-dirty-bitmap-state-via-savevm.patch} | 0 .../pve/0039-PVE-fixup-pbs-restore-API.patch | 44 - ...irty-counter-for-non-incremental-bac.patch | 30 - ...irty-bitmap-migrate-other-bitmaps-e.patch} | 0 ...ll-back-to-open-iscsi-initiatorname.patch} | 0 ...use-proxmox_backup_check_incremental.patch | 36 - ...outine-QMP-for-backup-cancel_backup.patch} | 0 ...ckup-add-compress-and-encrypt-option.patch | 103 -- ... => 0043-PBS-add-master-key-support.patch} | 0 ...st-path-reads-without-allocation-if-.patch | 52 + ...PVE-block-stream-increase-chunk-size.patch | 23 + ...issing-crypt-and-compress-parameters.patch | 43 - ...rite-callback-with-big-blocks-correc.patch | 76 -- ...accept-NULL-qiov-in-bdrv_pad_request.patch | 42 + ...-block-handling-to-PBS-dump-callback.patch | 85 -- .../0047-block-add-alloc-track-driver.patch | 380 ++++++++ ...n-up-error-handling-for-create_backu.patch | 187 ---- ...-multiple-CREATED-jobs-in-sequential.patch | 39 - debian/patches/series | 54 +- 40 files changed, 1356 insertions(+), 2178 deletions(-) rename debian/patches/{pve/0031-drive-mirror-add-support-for-sync-bitmap-mode-never.patch => bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch} (96%) rename debian/patches/{pve/0032-drive-mirror-add-support-for-conditional-and-always-.patch => bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch} (100%) rename debian/patches/{pve/0033-mirror-add-check-for-bitmap-mode-without-bitmap.patch => bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch} (90%) rename debian/patches/{pve/0034-mirror-switch-to-bdrv_dirty_bitmap_merge_internal.patch => bitmap-mirror/0004-mirror-switch-to-bdrv_dirty_bitmap_merge_internal.patch} (100%) rename debian/patches/{pve/0035-iotests-add-test-for-bitmap-mirror.patch => bitmap-mirror/0005-iotests-add-test-for-bitmap-mirror.patch} (100%) rename debian/patches/{pve/0036-mirror-move-some-checks-to-qmp.patch => bitmap-mirror/0006-mirror-move-some-checks-to-qmp.patch} (99%) rename debian/patches/pve/{0037-PVE-Backup-Add-dirty-bitmap-tracking-for-incremental.patch => 0030-PVE-Backup-Add-dirty-bitmap-tracking-for-incremental.patch} (88%) delete mode 100644 debian/patches/pve/0030-PVE-Backup-avoid-coroutines-to-fix-AIO-freeze-cleanu.patch create mode 100644 debian/patches/pve/0031-PVE-various-PBS-fixes.patch rename debian/patches/pve/{0043-PVE-Add-PBS-block-driver-to-map-backup-archives-into.patch => 0032-PVE-Add-PBS-block-driver-to-map-backup-archives-into.patch} (100%) rename debian/patches/pve/{0044-PVE-add-query_proxmox_support-QMP-command.patch => 0033-PVE-add-query_proxmox_support-QMP-command.patch} (94%) rename debian/patches/pve/{0048-PVE-add-query-pbs-bitmap-info-QMP-call.patch => 0034-PVE-add-query-pbs-bitmap-info-QMP-call.patch} (100%) rename debian/patches/pve/{0049-PVE-redirect-stderr-to-journal-when-daemonized.patch => 0035-PVE-redirect-stderr-to-journal-when-daemonized.patch} (100%) rename debian/patches/pve/{0050-PVE-Add-sequential-job-transaction-support.patch => 0036-PVE-Add-sequential-job-transaction-support.patch} (75%) rename debian/patches/pve/{0051-PVE-Backup-Use-a-transaction-to-synchronize-job-stat.patch => 0037-PVE-Backup-Use-a-transaction-to-synchronize-job-stat.patch} (100%) rename debian/patches/pve/{0052-PVE-Backup-Use-more-coroutines-and-don-t-block-on-fi.patch => 0038-PVE-Backup-Don-t-block-on-finishing-and-cleanup-crea.patch} (63%) delete mode 100644 debian/patches/pve/0038-PVE-backup-rename-incremental-to-use-dirty-bitmap.patch rename debian/patches/pve/{0054-PVE-Migrate-dirty-bitmap-state-via-savevm.patch => 0039-PVE-Migrate-dirty-bitmap-state-via-savevm.patch} (100%) delete mode 100644 debian/patches/pve/0039-PVE-fixup-pbs-restore-API.patch delete mode 100644 debian/patches/pve/0040-PVE-always-set-dirty-counter-for-non-incremental-bac.patch rename debian/patches/pve/{0055-migration-block-dirty-bitmap-migrate-other-bitmaps-e.patch => 0040-migration-block-dirty-bitmap-migrate-other-bitmaps-e.patch} (100%) rename debian/patches/pve/{0057-PVE-fall-back-to-open-iscsi-initiatorname.patch => 0041-PVE-fall-back-to-open-iscsi-initiatorname.patch} (100%) delete mode 100644 debian/patches/pve/0041-PVE-use-proxmox_backup_check_incremental.patch rename debian/patches/pve/{0058-PVE-Use-coroutine-QMP-for-backup-cancel_backup.patch => 0042-PVE-Use-coroutine-QMP-for-backup-cancel_backup.patch} (100%) delete mode 100644 debian/patches/pve/0042-PVE-fixup-pbs-backup-add-compress-and-encrypt-option.patch rename debian/patches/pve/{0059-PBS-add-master-key-support.patch => 0043-PBS-add-master-key-support.patch} (100%) create mode 100644 debian/patches/pve/0044-PVE-block-pbs-fast-path-reads-without-allocation-if-.patch create mode 100644 debian/patches/pve/0045-PVE-block-stream-increase-chunk-size.patch delete mode 100644 debian/patches/pve/0045-pbs-fix-missing-crypt-and-compress-parameters.patch delete mode 100644 debian/patches/pve/0046-PVE-handle-PBS-write-callback-with-big-blocks-correc.patch create mode 100644 debian/patches/pve/0046-block-io-accept-NULL-qiov-in-bdrv_pad_request.patch delete mode 100644 debian/patches/pve/0047-PVE-add-zero-block-handling-to-PBS-dump-callback.patch create mode 100644 debian/patches/pve/0047-block-add-alloc-track-driver.patch delete mode 100644 debian/patches/pve/0053-PVE-fix-and-clean-up-error-handling-for-create_backu.patch delete mode 100644 debian/patches/pve/0056-PVE-fix-aborting-multiple-CREATED-jobs-in-sequential.patch proxmox-backup: Stefan Reiter (1): RemoteChunkReader: add LRU cached variant src/bin/proxmox_backup_client/mount.rs | 4 +- src/client/remote_chunk_reader.rs | 77 ++++++++++++++++++++------ 2 files changed, 62 insertions(+), 19 deletions(-) proxmox-backup-qemu: Stefan Reiter (1): access: use bigger cache and LRU chunk reader src/restore.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) qemu-server: Stefan Reiter (5): make qemu_drive_mirror_monitor more generic cfg2cmd: allow PBS snapshots as backing files for drives enable live-restore for PBS extract register_qmeventd_handle to QemuServer.pm live-restore: register qmeventd handle PVE/API2/Qemu.pm | 14 +- PVE/QemuServer.pm | 297 ++++++++++++++++++++++++++++++++------- PVE/VZDump/QemuServer.pm | 32 +---- 3 files changed, 259 insertions(+), 84 deletions(-) manager: Stefan Reiter (1): ui: restore: add live-restore checkbox www/manager6/grid/BackupView.js | 6 ++++- www/manager6/storage/BackupView.js | 5 +++- www/manager6/window/Restore.js | 38 +++++++++++++++++++++++++++++- 3 files changed, 46 insertions(+), 3 deletions(-) -- 2.20.1