From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 7B5521FF0C0 for ; Wed, 18 Nov 2026 16:42:25 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 18C52217D4; Wed, 18 Nov 2026 16:42:24 +0100 (CET) From: Jonas Theisen To: pve-devel@lists.proxmox.com Subject: [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Date: Wed, 18 Nov 2026 16:40:31 +0100 Message-ID: <20261118154207.424260-1-j.theisen@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795016538569 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.060 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TFTYRZPFK74V6KJKO2BDKSVJL2MNTSFX X-Message-ID-Hash: TFTYRZPFK74V6KJKO2BDKSVJL2MNTSFX X-MailFrom: j.theisen@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Live migrations of running VMs can currently fail late, or seem to succeed and leave the guest unstable, if the target node's CPU cannot run the VM's virtual CPU. This series checks the target's CPU upfront, reports the result in the migration precondition API, and refuses migrations that cannot work. The checks use CPU information each node broadcasts to the cluster. The already existing 'cpuflags-kvm' node key-value entry provides the flags KVM supports. The new 'host-cpu-info' entry, broadcast by pvestatd, provides the CPU vendor and model name, TSC frequency and TSC scaling support. The checks are only done for running VMs, as stopped VMs get a virtual CPU based on the target node, and they operate on the QEMU CPU of the running instance, so they also cover custom CPU models. Checks that block the migration: - 'host'/'max' CPU model, target has a different CPU vendor - fix #2924: 'host'/'max' CPU model, target lacks KVM CPU flags of the source. As the source flags only approximate those of the VM, root can override this with the new 'allow-missing-cpu-flags' option. 'force' is not reused, as the HA stack always sets it. Instead, the new option is forwarded to the HA stack like other migration options. The migration dialog offers it as an "Ignore CPU incompatibility" checkbox for root, also for HA-managed VMs. - abstracted CPU configuration (custom model, 'nested-virt', ...), where the target keeps the running QEMU CPU: KVM on the target does not support an enabled Hyper-V enlightenment (for example the Intel-only 'hv-evmcs' on an AMD target), or any enabled flag with 'enforce' - new 'hv-reenlightenment' flag, target can neither scale the TSC nor match the source's TSC frequency within 250 ppm Checks that only warn, as a task warning and in the precondition API: - fix #6535: live migration between CPU vendors with any other model - 'host'/'max' CPU model, target has a different CPU model name - the target does not broadcast the CPU information a blocking check needs, for example as it runs an older version. Such nodes are not blocked, as it is unknown whether they are compatible. The 'hv-reenlightenment' flag is added as part of this series. It lets nested Hyper-V guests, for example Windows with WSL2 or VBS, keep a stable clock source across live migrations. For it, the TSC frequency is set explicitly to the host's, which is queried via KVM_GET_TSC_KHZ on a temporary VM, as upstream Linux does not export it otherwise. API changes of the migrate precondition endpoint: - 'incompatible-cpu' in 'not_allowed_nodes', listing the type, message and possible override option of each blocking incompatibility - 'cpu-warnings', listing the warnings per node The migration tests are extended to check for expected warnings in the log, and cover each of the checks. Dependencies: pve-manager's pvestatd patch uses PVE::QemuServer::Helpers::get_host_cpu_info(), so it needs a bumped qemu-server. Without it, qemu-server still works and runs the checks based on the already broadcast CPU flags, but warns about the missing CPU information. The 'host-cpu-flags' override option relies on the HA stack accepting forwarded migration options, as introduced for 'with-conntrack-state' and 'skip-config-check'. Testing: The checks can be tested on a cluster with identical CPUs by making the target node broadcast different CPU information. Run the following on the target node. pvestatd broadcasts 'host-cpu-info' again whenever it differs, so stop it first: systemctl stop pvestatd perl -MPVE::Cluster -e 'PVE::Cluster::broadcast_node_kv("host-cpu-info", q({"vendor":"GenuineIntel","model-name":"Fake CPU","tsc-khz":3000000,"tsc-scaling":0}))' The 'cpuflags-kvm' entry is only broadcast again when the QEMU version changes, so it can be changed while pvestatd runs, for example to remove some flags: perl -MPVE::Cluster -MPVE::INotify -e ' PVE::Cluster::cfs_update(); my $node = PVE::INotify::nodename(); my $flags = PVE::Cluster::get_node_kv("cpuflags-kvm", $node)->{$node}; $flags = join(" ", grep { !/^(avx512f|sha-ni)$/ } split(/ /, $flags)); PVE::Cluster::broadcast_node_kv("cpuflags-kvm", $flags); ' Restarting pvestatd broadcasts the real values of both entries again: systemctl restart pvestatd The results show up in the precondition API, in the migration dialog, and in the task log of 'qm migrate --online'. Disk-less VMs suffice for live migration tests: pvesh get /nodes//qemu//migrate --output-format json-pretty What to expect for a running VM: - vendor 'GenuineIntel' on an AMD cluster (or vice versa): blocked for CPU type 'host', a warning for other CPU types - a different 'model-name': a warning for CPU type 'host' - 'tsc-scaling' 0 and a 'tsc-khz' more than 250 ppm off: blocked for VMs with 'flags=+hv-reenlightenment' - flags removed from 'cpuflags-kvm': blocked for CPU type 'host', unless '--allow-missing-cpu-flags' is set, also for HA-managed VMs - 'hv-*' flags removed that a VM with a kept virtual CPU uses, for example 'hv-stimer' for 'x86-64-v2-AES,flags=+hv-reenlightenment' with ostype 'win11': blocked, without override - an empty 'host-cpu-info' value: a 'missing-cpu-info' warning for VMs where a blocking check needs the information qemu-server: Jonas Theisen (11): test: migration: allow checking for warnings and errors in the log migrate: block live migration of host CPU VMs between CPU vendors fix #6535: migrate: warn about live migration between CPU vendors cpu flags: factor out parsing the broadcast CPU flags of a node fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags migrate: warn about live migration of host CPU VMs between CPU models migrate: factor out whether the running virtual CPU is kept migrate: block migration if the target cannot run the kept virtual CPU helpers: add helper to query the host's TSC frequency and scaling support cpu config: add hv-reenlightenment flag and set TSC frequency for it migrate: block migration to nodes with incompatible TSC for re-enlightenment src/PVE/API2/Qemu.pm | 104 ++- src/PVE/QemuMigrate.pm | 39 +- src/PVE/QemuMigrate/Helpers.pm | 329 +++++++++ src/PVE/QemuServer/CPUConfig.pm | 18 + src/PVE/QemuServer/CPUFlags.pm | 29 +- src/PVE/QemuServer/Helpers.pm | 99 +++ src/test/MigrationTest/QemuMigrateMock.pm | 36 + src/test/QemuServerTest/CommandLineMocks.pm | 3 + src/test/cfg2cmd/cpu-hv-reenlightenment.conf | 6 + .../cfg2cmd/cpu-hv-reenlightenment.conf.cmd | 26 + src/test/run_qemu_migrate_tests.pl | 685 ++++++++++++++++++ 11 files changed, 1366 insertions(+), 8 deletions(-) create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd pve-manager: Jonas Theisen (4): pvestatd: broadcast host CPU information relevant for live migration ui: migrate: show why the CPU of the target node is incompatible fix #6535: ui: migrate: show CPU warnings for the target node api: nodes: migrateall: skip VMs whose CPU is incompatible with the target PVE/API2/Nodes.pm | 5 +++ PVE/Service/pvestatd.pm | 21 ++++++++++++ www/manager6/window/Migrate.js | 63 ++++++++++++++++++++++++++++++++++ 3 files changed, 89 insertions(+) Summary over all repositories: 14 files changed, 1455 insertions(+), 8 deletions(-) -- Generated by murpp 0.12.1