public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM
@ 2026-11-18 15:40 Jonas Theisen
  2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
                   ` (14 more replies)
  0 siblings, 15 replies; 16+ messages in thread
From: Jonas Theisen @ 2026-11-18 15:40 UTC (permalink / raw)
  To: pve-devel

Live migrations of running VMs can currently fail late, or seem to
succeed and leave the guest unstable, if the target node's CPU cannot
run the VM's virtual CPU. This series checks the target's CPU upfront,
reports the result in the migration precondition API, and refuses
migrations that cannot work.

The checks use CPU information each node broadcasts to the cluster. The
already existing 'cpuflags-kvm' node key-value entry provides the flags
KVM supports. The new 'host-cpu-info' entry, broadcast by pvestatd,
provides the CPU vendor and model name, TSC frequency and TSC scaling
support. The checks are only done for running VMs, as stopped VMs get
a virtual CPU based on the target node, and they operate on the QEMU
CPU of the running instance, so they also cover custom CPU models.

Checks that block the migration:

- 'host'/'max' CPU model, target has a different CPU vendor
- fix #2924: 'host'/'max' CPU model, target lacks KVM CPU flags of the
  source. As the source flags only approximate those of the VM, root
  can override this with the new 'allow-missing-cpu-flags' option.
  'force' is not reused, as the HA stack always sets it. Instead, the
  new option is forwarded to the HA stack like other migration options.
  The migration dialog offers it as an "Ignore CPU incompatibility"
  checkbox for root, also for HA-managed VMs.
- abstracted CPU configuration (custom model, 'nested-virt', ...), where
  the target keeps the running QEMU CPU: KVM on the target does not
  support an enabled Hyper-V enlightenment (for example the Intel-only
  'hv-evmcs' on an AMD target), or any enabled flag with 'enforce'
- new 'hv-reenlightenment' flag, target can neither scale the TSC nor
  match the source's TSC frequency within 250 ppm

Checks that only warn, as a task warning and in the precondition API:

- fix #6535: live migration between CPU vendors with any other model
- 'host'/'max' CPU model, target has a different CPU model name
- the target does not broadcast the CPU information a blocking check
  needs, for example as it runs an older version. Such nodes are not
  blocked, as it is unknown whether they are compatible.

The 'hv-reenlightenment' flag is added as part of this series. It lets
nested Hyper-V guests, for example Windows with WSL2 or VBS, keep
a stable clock source across live migrations. For it, the TSC
frequency is set explicitly to the host's, which is queried via
KVM_GET_TSC_KHZ on a temporary VM, as upstream Linux does not export it
otherwise.

API changes of the migrate precondition endpoint:

- 'incompatible-cpu' in 'not_allowed_nodes', listing the type, message
  and possible override option of each blocking incompatibility
- 'cpu-warnings', listing the warnings per node

The migration tests are extended to check for expected warnings in the
log, and cover each of the checks.

Dependencies: pve-manager's pvestatd patch uses
PVE::QemuServer::Helpers::get_host_cpu_info(), so it needs a bumped
qemu-server. Without it, qemu-server still works and runs the checks
based on the already broadcast CPU flags, but warns about the missing
CPU information.

The 'host-cpu-flags' override option relies on the HA stack accepting
forwarded migration options, as introduced for 'with-conntrack-state'
and 'skip-config-check'.

Testing:

The checks can be tested on a cluster with identical CPUs by making the
target node broadcast different CPU information. Run the following on
the target node.

pvestatd broadcasts 'host-cpu-info' again whenever it differs, so stop
it first:

  systemctl stop pvestatd
  perl -MPVE::Cluster -e 'PVE::Cluster::broadcast_node_kv("host-cpu-info", q({"vendor":"GenuineIntel","model-name":"Fake CPU","tsc-khz":3000000,"tsc-scaling":0}))'

The 'cpuflags-kvm' entry is only broadcast again when the QEMU version
changes, so it can be changed while pvestatd runs, for example to
remove some flags:

  perl -MPVE::Cluster -MPVE::INotify -e '
      PVE::Cluster::cfs_update();
      my $node = PVE::INotify::nodename();
      my $flags = PVE::Cluster::get_node_kv("cpuflags-kvm", $node)->{$node};
      $flags = join(" ", grep { !/^(avx512f|sha-ni)$/ } split(/ /, $flags));
      PVE::Cluster::broadcast_node_kv("cpuflags-kvm", $flags);
  '

Restarting pvestatd broadcasts the real values of both entries again:

  systemctl restart pvestatd

The results show up in the precondition API, in the migration dialog,
and in the task log of 'qm migrate <vmid> <target> --online'. Disk-less
VMs suffice for live migration tests:

  pvesh get /nodes/<source>/qemu/<vmid>/migrate --output-format json-pretty

What to expect for a running VM:

- vendor 'GenuineIntel' on an AMD cluster (or vice versa): blocked for
  CPU type 'host', a warning for other CPU types
- a different 'model-name': a warning for CPU type 'host'
- 'tsc-scaling' 0 and a 'tsc-khz' more than 250 ppm off: blocked for
  VMs with 'flags=+hv-reenlightenment'
- flags removed from 'cpuflags-kvm': blocked for CPU type 'host',
  unless '--allow-missing-cpu-flags' is set, also for HA-managed VMs
- 'hv-*' flags removed that a VM with a kept virtual CPU uses, for
  example 'hv-stimer' for 'x86-64-v2-AES,flags=+hv-reenlightenment'
  with ostype 'win11': blocked, without override
- an empty 'host-cpu-info' value: a 'missing-cpu-info' warning for VMs
  where a blocking check needs the information


qemu-server:

Jonas Theisen (11):
  test: migration: allow checking for warnings and errors in the log
  migrate: block live migration of host CPU VMs between CPU vendors
  fix #6535: migrate: warn about live migration between CPU vendors
  cpu flags: factor out parsing the broadcast CPU flags of a node
  fix #2924: migrate: block live migration of host CPU VMs to nodes
    lacking flags
  migrate: warn about live migration of host CPU VMs between CPU models
  migrate: factor out whether the running virtual CPU is kept
  migrate: block migration if the target cannot run the kept virtual CPU
  helpers: add helper to query the host's TSC frequency and scaling
    support
  cpu config: add hv-reenlightenment flag and set TSC frequency for it
  migrate: block migration to nodes with incompatible TSC for
    re-enlightenment

 src/PVE/API2/Qemu.pm                          | 104 ++-
 src/PVE/QemuMigrate.pm                        |  39 +-
 src/PVE/QemuMigrate/Helpers.pm                | 329 +++++++++
 src/PVE/QemuServer/CPUConfig.pm               |  18 +
 src/PVE/QemuServer/CPUFlags.pm                |  29 +-
 src/PVE/QemuServer/Helpers.pm                 |  99 +++
 src/test/MigrationTest/QemuMigrateMock.pm     |  36 +
 src/test/QemuServerTest/CommandLineMocks.pm   |   3 +
 src/test/cfg2cmd/cpu-hv-reenlightenment.conf  |   6 +
 .../cfg2cmd/cpu-hv-reenlightenment.conf.cmd   |  26 +
 src/test/run_qemu_migrate_tests.pl            | 685 ++++++++++++++++++
 11 files changed, 1366 insertions(+), 8 deletions(-)
 create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf
 create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd


pve-manager:

Jonas Theisen (4):
  pvestatd: broadcast host CPU information relevant for live migration
  ui: migrate: show why the CPU of the target node is incompatible
  fix #6535: ui: migrate: show CPU warnings for the target node
  api: nodes: migrateall: skip VMs whose CPU is incompatible with the
    target

 PVE/API2/Nodes.pm              |  5 +++
 PVE/Service/pvestatd.pm        | 21 ++++++++++++
 www/manager6/window/Migrate.js | 63 ++++++++++++++++++++++++++++++++++
 3 files changed, 89 insertions(+)


Summary over all repositories:
  14 files changed, 1455 insertions(+), 8 deletions(-)

-- 
Generated by murpp 0.12.1




^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2026-11-18 15:44 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 15:40 [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 02/15] migrate: block live migration of host CPU VMs between CPU vendors Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 03/15] fix #6535: migrate: warn about live migration " Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 04/15] cpu flags: factor out parsing the broadcast CPU flags of a node Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 06/15] migrate: warn about live migration of host CPU VMs between CPU models Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 07/15] migrate: factor out whether the running virtual CPU is kept Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 08/15] migrate: block migration if the target cannot run the kept virtual CPU Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 09/15] helpers: add helper to query the host's TSC frequency and scaling support Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 11/15] migrate: block migration to nodes with incompatible TSC for re-enlightenment Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 12/15] pvestatd: broadcast host CPU information relevant for live migration Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 13/15] ui: migrate: show why the CPU of the target node is incompatible Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 14/15] fix #6535: ui: migrate: show CPU warnings for the target node Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 15/15] api: nodes: migrateall: skip VMs whose CPU is incompatible with the target Jonas Theisen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal