From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 77A3B8606B for ; Wed, 22 Dec 2021 14:53:38 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6BEA416D6C for ; Wed, 22 Dec 2021 14:53:08 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 173EA16D29 for ; Wed, 22 Dec 2021 14:53:06 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 919EA44373 for ; Wed, 22 Dec 2021 14:52:59 +0100 (CET) From: =?UTF-8?q?Fabian=20Gr=C3=BCnbichler?= To: pve-devel@lists.proxmox.com Date: Wed, 22 Dec 2021 14:52:36 +0100 Message-Id: <20211222135257.3242938-1-f.gruenbichler@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.238 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH v3 qemu-server++ 0/21] remote migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Dec 2021 13:53:38 -0000 this series adds remote migration for VMs. both live and offline migration including NBD and storage-migrated disks should work. groundwork for extending to pve-container and pvesr already laid. new in v3: lots of refactoring and edge-case handling new in v2: dropped parts already applied, incorporated Fabian's and Dominik's feedback (thanks!) new in v1: explicit remote endpoint specified as part of API call instead of remote.cfg overview over affected repos and changes, see individual patches for more details. proxmox-websocket-tunnel: new tunnel helper tool for forwarding commands and data over websocket connections, required by pve-guest-common on source side pve-guest-common: new/refactored generic WS/SSH tunnel fork/read/.. helpers new storage migration over WS tunnel helpers pve-storage: refactor storage_migrate to make code-reuse possible qemu-server: some refactoring, new mtunnel endpoints, new remote_migration endpoints TODO: handle pending changes and snapshots TODO: proper CLI for remote migration TODO: handle C^c TODO: strict parser for FW config as well? potential TODO: precond endpoint? as usual, some of the patches are best viewed with '-w', especially in qemu-server.. required dependencies are noted pve-guest-common: Fabian Grünbichler (3): migrate: handle migration_network with remote migration add tunnel helper module add storage tunnel module src/Makefile | 2 + debian/control | 1 + src/PVE/AbstractMigrate.pm | 37 ++-- src/PVE/StorageTunnel.pm | 231 ++++++++++++++++++++++++ src/PVE/Tunnel.pm | 356 +++++++++++++++++++++++++++++++++++++ 5 files changed, 611 insertions(+), 16 deletions(-) create mode 100644 src/PVE/StorageTunnel.pm create mode 100644 src/PVE/Tunnel.pm proxmox-websocket-tunnel: Fabian Grünbichler (4): initial commit add tunnel implementation add fingerprint validation add packaging qemu-server: Fabian Grünbichler (10): refactor map_storage to map_id schema: use pve-bridge-id parse_config: optional strict mode update_vm: allow simultaneous setting of boot-order and dev nbd alloc helper: allow passing in explicit format migrate: move tunnel-helpers to pve-guest-common mtunnel: add API endpoints migrate: refactor remote VM/tunnel start migrate: add remote migration handling api: add remote migrate endpoint PVE/API2/Qemu.pm | 721 ++++++++++++++++++++++++- PVE/QemuMigrate.pm | 738 ++++++++++++++++---------- PVE/QemuServer.pm | 100 ++-- debian/control | 2 + test/MigrationTest/QemuMigrateMock.pm | 28 +- 5 files changed, 1240 insertions(+), 349 deletions(-) pve-storage: Fabian Grünbichler (4): volname_for_storage: parse volname before calling storage_migrate: pull out snapshot decision storage_migrate: pull out import/export_prepare add volume_import/export_start helpers PVE/Storage.pm | 205 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 153 insertions(+), 52 deletions(-)