* Re: [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM
[not found] <38B8F285-72BD-CA4C-AC77-1FB069EF64DE@hxcore.ol>
@ 2026-11-20 14:53 ` Jonas Theisen
0 siblings, 0 replies; 2+ messages in thread
From: Jonas Theisen @ 2026-11-20 14:53 UTC (permalink / raw)
To: crandler@crandland.de; +Cc: pve-devel@lists.proxmox.com
Thank you for the feedback!
Comments inline:
On 11/20/26 01:23, crandler@crandland.de wrote:
> <snip>
>
> 1. Built-in models are not covered. A VM with 'x86-64-v3' that is live
> migrated to a node without AVX2 still fails only when QEMU starts on
> the target ("QEMU exited with code 1"). The target builds the CPU
> again from the config, but the model flags are the same, and
> 'enforce' is set for all models except 'host'. Since the x86-64-v*
> flags show up explicitly in the running QEMU CPU, the flag part of
> the 'kept-cpu-flags' check should work for them as well. The
> migration dialog would then mark such a node as incompatible, and
> 'migrateall' would skip these VMs.
Completely correct and an overlook on my end. I have added / extended
the appropriate check and will include that in v2.
Thanks for noticing!
>
> 2. HA node selection. As far as I can see, the checks also apply to
> migrations started by the HA stack, which is great. The HA manager
> does not know about them when it picks a target, though (maintenance
> mode, auto-rebalance). If it picks a node that cannot run the VM's
> CPU, the migration is refused and the service goes back to 'started'
> on the source without the failure being recorded, so a later round
> can pick the same node again. #8059 describes this for auto-rebalance
> (failed migrations are retried, candidates are not checked for
> migratability). Would it make sense to let the CRM filter its
> candidate nodes with the same check, or should I open a separate bug
> for that?
I think this could be covered in the bug you already mentioned.
I included a pointer to it in the v2 cover letter but maybe a small comment
on the existing bug could help clarify the interconnect of both issues.
>
> Thanks,
> Sven
>
^ permalink raw reply [flat|nested] 2+ messages in thread
* [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM
@ 2026-11-18 15:40 Jonas Theisen
0 siblings, 0 replies; 2+ messages in thread
From: Jonas Theisen @ 2026-11-18 15:40 UTC (permalink / raw)
To: pve-devel
Live migrations of running VMs can currently fail late, or seem to
succeed and leave the guest unstable, if the target node's CPU cannot
run the VM's virtual CPU. This series checks the target's CPU upfront,
reports the result in the migration precondition API, and refuses
migrations that cannot work.
The checks use CPU information each node broadcasts to the cluster. The
already existing 'cpuflags-kvm' node key-value entry provides the flags
KVM supports. The new 'host-cpu-info' entry, broadcast by pvestatd,
provides the CPU vendor and model name, TSC frequency and TSC scaling
support. The checks are only done for running VMs, as stopped VMs get
a virtual CPU based on the target node, and they operate on the QEMU
CPU of the running instance, so they also cover custom CPU models.
Checks that block the migration:
- 'host'/'max' CPU model, target has a different CPU vendor
- fix #2924: 'host'/'max' CPU model, target lacks KVM CPU flags of the
source. As the source flags only approximate those of the VM, root
can override this with the new 'allow-missing-cpu-flags' option.
'force' is not reused, as the HA stack always sets it. Instead, the
new option is forwarded to the HA stack like other migration options.
The migration dialog offers it as an "Ignore CPU incompatibility"
checkbox for root, also for HA-managed VMs.
- abstracted CPU configuration (custom model, 'nested-virt', ...), where
the target keeps the running QEMU CPU: KVM on the target does not
support an enabled Hyper-V enlightenment (for example the Intel-only
'hv-evmcs' on an AMD target), or any enabled flag with 'enforce'
- new 'hv-reenlightenment' flag, target can neither scale the TSC nor
match the source's TSC frequency within 250 ppm
Checks that only warn, as a task warning and in the precondition API:
- fix #6535: live migration between CPU vendors with any other model
- 'host'/'max' CPU model, target has a different CPU model name
- the target does not broadcast the CPU information a blocking check
needs, for example as it runs an older version. Such nodes are not
blocked, as it is unknown whether they are compatible.
The 'hv-reenlightenment' flag is added as part of this series. It lets
nested Hyper-V guests, for example Windows with WSL2 or VBS, keep
a stable clock source across live migrations. For it, the TSC
frequency is set explicitly to the host's, which is queried via
KVM_GET_TSC_KHZ on a temporary VM, as upstream Linux does not export it
otherwise.
API changes of the migrate precondition endpoint:
- 'incompatible-cpu' in 'not_allowed_nodes', listing the type, message
and possible override option of each blocking incompatibility
- 'cpu-warnings', listing the warnings per node
The migration tests are extended to check for expected warnings in the
log, and cover each of the checks.
Dependencies: pve-manager's pvestatd patch uses
PVE::QemuServer::Helpers::get_host_cpu_info(), so it needs a bumped
qemu-server. Without it, qemu-server still works and runs the checks
based on the already broadcast CPU flags, but warns about the missing
CPU information.
The 'host-cpu-flags' override option relies on the HA stack accepting
forwarded migration options, as introduced for 'with-conntrack-state'
and 'skip-config-check'.
Testing:
The checks can be tested on a cluster with identical CPUs by making the
target node broadcast different CPU information. Run the following on
the target node.
pvestatd broadcasts 'host-cpu-info' again whenever it differs, so stop
it first:
systemctl stop pvestatd
perl -MPVE::Cluster -e 'PVE::Cluster::broadcast_node_kv("host-cpu-info", q({"vendor":"GenuineIntel","model-name":"Fake CPU","tsc-khz":3000000,"tsc-scaling":0}))'
The 'cpuflags-kvm' entry is only broadcast again when the QEMU version
changes, so it can be changed while pvestatd runs, for example to
remove some flags:
perl -MPVE::Cluster -MPVE::INotify -e '
PVE::Cluster::cfs_update();
my $node = PVE::INotify::nodename();
my $flags = PVE::Cluster::get_node_kv("cpuflags-kvm", $node)->{$node};
$flags = join(" ", grep { !/^(avx512f|sha-ni)$/ } split(/ /, $flags));
PVE::Cluster::broadcast_node_kv("cpuflags-kvm", $flags);
'
Restarting pvestatd broadcasts the real values of both entries again:
systemctl restart pvestatd
The results show up in the precondition API, in the migration dialog,
and in the task log of 'qm migrate <vmid> <target> --online'. Disk-less
VMs suffice for live migration tests:
pvesh get /nodes/<source>/qemu/<vmid>/migrate --output-format json-pretty
What to expect for a running VM:
- vendor 'GenuineIntel' on an AMD cluster (or vice versa): blocked for
CPU type 'host', a warning for other CPU types
- a different 'model-name': a warning for CPU type 'host'
- 'tsc-scaling' 0 and a 'tsc-khz' more than 250 ppm off: blocked for
VMs with 'flags=+hv-reenlightenment'
- flags removed from 'cpuflags-kvm': blocked for CPU type 'host',
unless '--allow-missing-cpu-flags' is set, also for HA-managed VMs
- 'hv-*' flags removed that a VM with a kept virtual CPU uses, for
example 'hv-stimer' for 'x86-64-v2-AES,flags=+hv-reenlightenment'
with ostype 'win11': blocked, without override
- an empty 'host-cpu-info' value: a 'missing-cpu-info' warning for VMs
where a blocking check needs the information
qemu-server:
Jonas Theisen (11):
test: migration: allow checking for warnings and errors in the log
migrate: block live migration of host CPU VMs between CPU vendors
fix #6535: migrate: warn about live migration between CPU vendors
cpu flags: factor out parsing the broadcast CPU flags of a node
fix #2924: migrate: block live migration of host CPU VMs to nodes
lacking flags
migrate: warn about live migration of host CPU VMs between CPU models
migrate: factor out whether the running virtual CPU is kept
migrate: block migration if the target cannot run the kept virtual CPU
helpers: add helper to query the host's TSC frequency and scaling
support
cpu config: add hv-reenlightenment flag and set TSC frequency for it
migrate: block migration to nodes with incompatible TSC for
re-enlightenment
src/PVE/API2/Qemu.pm | 104 ++-
src/PVE/QemuMigrate.pm | 39 +-
src/PVE/QemuMigrate/Helpers.pm | 329 +++++++++
src/PVE/QemuServer/CPUConfig.pm | 18 +
src/PVE/QemuServer/CPUFlags.pm | 29 +-
src/PVE/QemuServer/Helpers.pm | 99 +++
src/test/MigrationTest/QemuMigrateMock.pm | 36 +
src/test/QemuServerTest/CommandLineMocks.pm | 3 +
src/test/cfg2cmd/cpu-hv-reenlightenment.conf | 6 +
.../cfg2cmd/cpu-hv-reenlightenment.conf.cmd | 26 +
src/test/run_qemu_migrate_tests.pl | 685 ++++++++++++++++++
11 files changed, 1366 insertions(+), 8 deletions(-)
create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf
create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd
pve-manager:
Jonas Theisen (4):
pvestatd: broadcast host CPU information relevant for live migration
ui: migrate: show why the CPU of the target node is incompatible
fix #6535: ui: migrate: show CPU warnings for the target node
api: nodes: migrateall: skip VMs whose CPU is incompatible with the
target
PVE/API2/Nodes.pm | 5 +++
PVE/Service/pvestatd.pm | 21 ++++++++++++
www/manager6/window/Migrate.js | 63 ++++++++++++++++++++++++++++++++++
3 files changed, 89 insertions(+)
Summary over all repositories:
14 files changed, 1455 insertions(+), 8 deletions(-)
--
Generated by murpp 0.12.1
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-20 14:53 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <38B8F285-72BD-CA4C-AC77-1FB069EF64DE@hxcore.ol>
2026-11-20 14:53 ` [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Jonas Theisen
2026-11-18 15:40 Jonas Theisen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox