From: Jonas Theisen <j.theisen@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM
Date: Wed, 18 Nov 2026 16:40:31 +0100 [thread overview]
Message-ID: <20261118154207.424260-1-j.theisen@proxmox.com> (raw)
Live migrations of running VMs can currently fail late, or seem to
succeed and leave the guest unstable, if the target node's CPU cannot
run the VM's virtual CPU. This series checks the target's CPU upfront,
reports the result in the migration precondition API, and refuses
migrations that cannot work.
The checks use CPU information each node broadcasts to the cluster. The
already existing 'cpuflags-kvm' node key-value entry provides the flags
KVM supports. The new 'host-cpu-info' entry, broadcast by pvestatd,
provides the CPU vendor and model name, TSC frequency and TSC scaling
support. The checks are only done for running VMs, as stopped VMs get
a virtual CPU based on the target node, and they operate on the QEMU
CPU of the running instance, so they also cover custom CPU models.
Checks that block the migration:
- 'host'/'max' CPU model, target has a different CPU vendor
- fix #2924: 'host'/'max' CPU model, target lacks KVM CPU flags of the
source. As the source flags only approximate those of the VM, root
can override this with the new 'allow-missing-cpu-flags' option.
'force' is not reused, as the HA stack always sets it. Instead, the
new option is forwarded to the HA stack like other migration options.
The migration dialog offers it as an "Ignore CPU incompatibility"
checkbox for root, also for HA-managed VMs.
- abstracted CPU configuration (custom model, 'nested-virt', ...), where
the target keeps the running QEMU CPU: KVM on the target does not
support an enabled Hyper-V enlightenment (for example the Intel-only
'hv-evmcs' on an AMD target), or any enabled flag with 'enforce'
- new 'hv-reenlightenment' flag, target can neither scale the TSC nor
match the source's TSC frequency within 250 ppm
Checks that only warn, as a task warning and in the precondition API:
- fix #6535: live migration between CPU vendors with any other model
- 'host'/'max' CPU model, target has a different CPU model name
- the target does not broadcast the CPU information a blocking check
needs, for example as it runs an older version. Such nodes are not
blocked, as it is unknown whether they are compatible.
The 'hv-reenlightenment' flag is added as part of this series. It lets
nested Hyper-V guests, for example Windows with WSL2 or VBS, keep
a stable clock source across live migrations. For it, the TSC
frequency is set explicitly to the host's, which is queried via
KVM_GET_TSC_KHZ on a temporary VM, as upstream Linux does not export it
otherwise.
API changes of the migrate precondition endpoint:
- 'incompatible-cpu' in 'not_allowed_nodes', listing the type, message
and possible override option of each blocking incompatibility
- 'cpu-warnings', listing the warnings per node
The migration tests are extended to check for expected warnings in the
log, and cover each of the checks.
Dependencies: pve-manager's pvestatd patch uses
PVE::QemuServer::Helpers::get_host_cpu_info(), so it needs a bumped
qemu-server. Without it, qemu-server still works and runs the checks
based on the already broadcast CPU flags, but warns about the missing
CPU information.
The 'host-cpu-flags' override option relies on the HA stack accepting
forwarded migration options, as introduced for 'with-conntrack-state'
and 'skip-config-check'.
Testing:
The checks can be tested on a cluster with identical CPUs by making the
target node broadcast different CPU information. Run the following on
the target node.
pvestatd broadcasts 'host-cpu-info' again whenever it differs, so stop
it first:
systemctl stop pvestatd
perl -MPVE::Cluster -e 'PVE::Cluster::broadcast_node_kv("host-cpu-info", q({"vendor":"GenuineIntel","model-name":"Fake CPU","tsc-khz":3000000,"tsc-scaling":0}))'
The 'cpuflags-kvm' entry is only broadcast again when the QEMU version
changes, so it can be changed while pvestatd runs, for example to
remove some flags:
perl -MPVE::Cluster -MPVE::INotify -e '
PVE::Cluster::cfs_update();
my $node = PVE::INotify::nodename();
my $flags = PVE::Cluster::get_node_kv("cpuflags-kvm", $node)->{$node};
$flags = join(" ", grep { !/^(avx512f|sha-ni)$/ } split(/ /, $flags));
PVE::Cluster::broadcast_node_kv("cpuflags-kvm", $flags);
'
Restarting pvestatd broadcasts the real values of both entries again:
systemctl restart pvestatd
The results show up in the precondition API, in the migration dialog,
and in the task log of 'qm migrate <vmid> <target> --online'. Disk-less
VMs suffice for live migration tests:
pvesh get /nodes/<source>/qemu/<vmid>/migrate --output-format json-pretty
What to expect for a running VM:
- vendor 'GenuineIntel' on an AMD cluster (or vice versa): blocked for
CPU type 'host', a warning for other CPU types
- a different 'model-name': a warning for CPU type 'host'
- 'tsc-scaling' 0 and a 'tsc-khz' more than 250 ppm off: blocked for
VMs with 'flags=+hv-reenlightenment'
- flags removed from 'cpuflags-kvm': blocked for CPU type 'host',
unless '--allow-missing-cpu-flags' is set, also for HA-managed VMs
- 'hv-*' flags removed that a VM with a kept virtual CPU uses, for
example 'hv-stimer' for 'x86-64-v2-AES,flags=+hv-reenlightenment'
with ostype 'win11': blocked, without override
- an empty 'host-cpu-info' value: a 'missing-cpu-info' warning for VMs
where a blocking check needs the information
qemu-server:
Jonas Theisen (11):
test: migration: allow checking for warnings and errors in the log
migrate: block live migration of host CPU VMs between CPU vendors
fix #6535: migrate: warn about live migration between CPU vendors
cpu flags: factor out parsing the broadcast CPU flags of a node
fix #2924: migrate: block live migration of host CPU VMs to nodes
lacking flags
migrate: warn about live migration of host CPU VMs between CPU models
migrate: factor out whether the running virtual CPU is kept
migrate: block migration if the target cannot run the kept virtual CPU
helpers: add helper to query the host's TSC frequency and scaling
support
cpu config: add hv-reenlightenment flag and set TSC frequency for it
migrate: block migration to nodes with incompatible TSC for
re-enlightenment
src/PVE/API2/Qemu.pm | 104 ++-
src/PVE/QemuMigrate.pm | 39 +-
src/PVE/QemuMigrate/Helpers.pm | 329 +++++++++
src/PVE/QemuServer/CPUConfig.pm | 18 +
src/PVE/QemuServer/CPUFlags.pm | 29 +-
src/PVE/QemuServer/Helpers.pm | 99 +++
src/test/MigrationTest/QemuMigrateMock.pm | 36 +
src/test/QemuServerTest/CommandLineMocks.pm | 3 +
src/test/cfg2cmd/cpu-hv-reenlightenment.conf | 6 +
.../cfg2cmd/cpu-hv-reenlightenment.conf.cmd | 26 +
src/test/run_qemu_migrate_tests.pl | 685 ++++++++++++++++++
11 files changed, 1366 insertions(+), 8 deletions(-)
create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf
create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd
pve-manager:
Jonas Theisen (4):
pvestatd: broadcast host CPU information relevant for live migration
ui: migrate: show why the CPU of the target node is incompatible
fix #6535: ui: migrate: show CPU warnings for the target node
api: nodes: migrateall: skip VMs whose CPU is incompatible with the
target
PVE/API2/Nodes.pm | 5 +++
PVE/Service/pvestatd.pm | 21 ++++++++++++
www/manager6/window/Migrate.js | 63 ++++++++++++++++++++++++++++++++++
3 files changed, 89 insertions(+)
Summary over all repositories:
14 files changed, 1455 insertions(+), 8 deletions(-)
--
Generated by murpp 0.12.1
next reply other threads:[~2026-11-18 15:42 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:40 Jonas Theisen [this message]
2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 02/15] migrate: block live migration of host CPU VMs between CPU vendors Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 03/15] fix #6535: migrate: warn about live migration " Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 04/15] cpu flags: factor out parsing the broadcast CPU flags of a node Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 06/15] migrate: warn about live migration of host CPU VMs between CPU models Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 07/15] migrate: factor out whether the running virtual CPU is kept Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 08/15] migrate: block migration if the target cannot run the kept virtual CPU Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 09/15] helpers: add helper to query the host's TSC frequency and scaling support Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 11/15] migrate: block migration to nodes with incompatible TSC for re-enlightenment Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 12/15] pvestatd: broadcast host CPU information relevant for live migration Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 13/15] ui: migrate: show why the CPU of the target node is incompatible Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 14/15] fix #6535: ui: migrate: show CPU warnings for the target node Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 15/15] api: nodes: migrateall: skip VMs whose CPU is incompatible with the target Jonas Theisen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118154207.424260-1-j.theisen@proxmox.com \
--to=j.theisen@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox