From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DCCC2614B1 for ; Wed, 9 Feb 2022 14:07:54 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D20B61570 for ; Wed, 9 Feb 2022 14:07:54 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 27D89154E for ; Wed, 9 Feb 2022 14:07:53 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E578B46241 for ; Wed, 9 Feb 2022 14:07:52 +0100 (CET) From: =?UTF-8?q?Fabian=20Gr=C3=BCnbichler?= To: pve-devel@lists.proxmox.com Date: Wed, 9 Feb 2022 14:07:29 +0100 Message-Id: <20220209130750.902245-1-f.gruenbichler@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.205 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH-SERIES 0/21] remote migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2022 13:07:54 -0000 this series adds remote migration for VMs (and if the last pve-container patch is applied, CTs ;)). both live and offline migration of VMs including NBD and storage-migrated disks should work, containers don't have any live migration so both offline and restart mode work identical except for the restart part. groundwork for extending to pvesr already laid. uncovered (but not fixed) https://bugzilla.proxmox.com/show_bug.cgi?id=3873 (migration btrfs -> btrfs with snapshots) new in v5: lots of edge cases fixed, PoC for pve-container, some more helper moving for re-use in pve-container without duplication new in v4: lots of small fixes, improved bwlimit handling, `qm` command (thanks Fabian Ebner and Dominik Csapak for the feedback on v3!) new in v3: lots of refactoring and edge-case handling new in v2: dropped parts already applied, incorporated Fabian's and Dominik's feedback (thanks!) new in v1: explicit remote endpoint specified as part of API call instead of remote.cfg overview over affected repos and changes, see individual patches for more details. pve-guest-common: new/refactored generic WS/SSH tunnel fork/read/.. helpers new storage migration over WS tunnel helpers pve-storage: refactor storage_migrate to make code-reuse possible use raw+size for btrfs when no snapshots exist qemu-server: some refactoring, new mtunnel endpoints, new remote_migration endpoints, new `qm` command TODO: handle pending changes and snapshots TODO: improve handling of C^c TODO: strict parser for FW config as well? as usual, some of the patches are best viewed with '-w', especially in qemu-server.. pve-container: --target-storage for regular migration remote migration PoC TODO: handle pending changes and snapshots TODO: strict parser for FW config as well? pve-common: map_id helper moved from qemu-server's map_storage required dependencies are noted, patch overview follows.. pve-common: add 'map_id' helper for ID maps pve-container: fix #1532: add target-storage support to migration config: add strict parser migration: add remote migration qemu-server: move map_storage to PVE::JSONSchema::map_id schema: use pve-bridge-id parse_config: optional strict mode update_vm: allow simultaneous setting of boot-order and dev nbd alloc helper: allow passing in explicit format migrate: move tunnel-helpers to pve-guest-common mtunnel: add API endpoints migrate: refactor remote VM/tunnel start migrate: add remote migration handling api: add remote migrate endpoint qm: add remote-migrate command pve-storage: storage_migrate_snapshot: skip for btrfs without snapshots storage_migrate: pull out import/export_prepare add volume_import/export_start helpers pve-guest-common: migrate: add get_bwlimit helper add tunnel helper module add storage tunnel module