From: Jonas Theisen <j.theisen@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags
Date: Wed, 18 Nov 2026 16:40:36 +0100 [thread overview]
Message-ID: <20261118154207.424260-6-j.theisen@proxmox.com> (raw)
In-Reply-To: <20261118154207.424260-1-j.theisen@proxmox.com>
The 'host' and 'max' CPU models pass the flags of the host CPU through
to the guest. If the target node's CPU lacks flags that the guest might
use, the live migration often fails only near its end, with QEMU on the
target failing to set the special registers of the vCPUs, or the guest
runs into problems after the migration.
Compare the CPU flags that KVM supports on the source and the target
node, which pvestatd already broadcasts, and refuse the migration if
the target lacks any. Flags explicitly disabled for the VM and Hyper-V
enlightenments, which are not passed through, are ignored. As the
flags are broadcast independently of the other CPU information, this
also works for target nodes running an older version. The flags are
normalized, as nodes that did not query them again since an upgrade
can still broadcast aliases of some flags.
The flags of the source node are only an approximation of the flags of
the VM, as it might have been started on another node, and the flags
KVM supports also depend on the kernel version. So allow root to
override the check with the new 'allow-missing-cpu-flags' option. The
existing 'force' option is not reused for this, as the HA stack always
sets it, which would skip the check for all HA-managed VMs. Instead,
the new option is forwarded to the HA stack like other migration
options, so that it also applies to HA-managed VMs.
Signed-off-by: Jonas Theisen <j.theisen@proxmox.com>
---
src/PVE/API2/Qemu.pm | 28 ++-
src/PVE/QemuMigrate.pm | 13 +-
src/PVE/QemuMigrate/Helpers.pm | 77 +++++++-
src/test/MigrationTest/QemuMigrateMock.pm | 4 +
src/test/run_qemu_migrate_tests.pl | 212 +++++++++++++++++++++-
5 files changed, 322 insertions(+), 12 deletions(-)
diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 9d7fcf8b..aaaa8760 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -155,6 +155,13 @@ my $migrate_json_properties = {
. ' successfully by the target. Using this flag can lead to dropped'
. ' configuration options if the target is too old to parse them.',
},
+ 'allow-missing-cpu-flags' => {
+ type => 'boolean',
+ optional => 1,
+ default => 0,
+ description => "Allow live migration of VMs with CPU type 'host' or 'max' to nodes"
+ . " lacking CPU flags of this node. Only root may use this option.",
+ },
};
# Properties that will be forwarded via the HA stack to the LRM, which will then use them for its
@@ -163,6 +170,7 @@ my $migrate_json_properties = {
my $ha_migrate_props = {
'with-conntrack-state' => 1,
'skip-config-check' => 1,
+ 'allow-missing-cpu-flags' => 1,
};
sub ha_migrate_json_properties {
@@ -5520,12 +5528,18 @@ __PACKAGE__->register_method({
type => {
type => 'string',
description => "The check that failed.",
- enum => ['cpu-vendor'],
+ enum => ['cpu-vendor', 'host-cpu-flags'],
},
message => {
type => 'string',
description => "Why the node's CPU is incompatible.",
},
+ override => {
+ type => 'string',
+ optional => 1,
+ description => "The migration option that allows"
+ . " ignoring the incompatibility.",
+ },
},
},
},
@@ -5730,8 +5744,13 @@ __PACKAGE__->register_method({
my @cpu_errors =
grep { $_->{severity} eq 'error' } ($cpu_issues_by_node->{$node} // [])->@*;
if (scalar(@cpu_errors)) {
- $res->{not_allowed_nodes}->{$node}->{'incompatible-cpu'} =
- [map { { type => $_->{type}, message => $_->{message} } } @cpu_errors];
+ $res->{not_allowed_nodes}->{$node}->{'incompatible-cpu'} = [
+ map {
+ my $error = { type => $_->{type}, message => $_->{message} };
+ $error->{override} = $_->{override} if defined($_->{override});
+ $error;
+ } @cpu_errors
+ ];
}
my @cpu_warnings =
@@ -5806,6 +5825,9 @@ __PACKAGE__->register_method({
raise_param_exc({ force => "Only root may use this option." })
if $param->{force} && $authuser ne 'root@pam';
+ raise_param_exc({ 'allow-missing-cpu-flags' => "Only root may use this option." })
+ if $param->{'allow-missing-cpu-flags'} && $authuser ne 'root@pam';
+
raise_param_exc({ migration_type => "Only root may use this option." })
if $param->{migration_type} && $authuser ne 'root@pam';
diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm
index 827f8143..bd157845 100644
--- a/src/PVE/QemuMigrate.pm
+++ b/src/PVE/QemuMigrate.pm
@@ -324,15 +324,26 @@ sub prepare {
my @errors;
for my $issue (($cpu_issues->{$node} // [])->@*) {
+ my $override = $issue->{override};
if ($issue->{severity} eq 'warning') {
log_warn("CPU of node '$node' might be incompatible: $issue->{message}");
+ } elsif (defined($override) && $self->{opts}->{$override}) {
+ log_warn("ignoring incompatible CPU of node '$node' due to '$override':"
+ . " $issue->{message}");
} else {
push @errors, $issue;
}
}
if (scalar(@errors)) {
+ my $hint = '';
+ if (!grep { !defined($_->{override}) } @errors) {
+ my %overrides = map { $_->{override} => 1 } @errors;
+ $hint =
+ " - use " . join(' and ', map { "'$_'" } sort keys %overrides) . " to override";
+ }
die "can't migrate VM to '$node' - incompatible CPU: "
- . join('; ', map { $_->{message} } @errors) . "\n";
+ . join('; ', map { $_->{message} } @errors)
+ . "$hint\n";
}
if (scalar($unchecked_nodes->@*)) {
diff --git a/src/PVE/QemuMigrate/Helpers.pm b/src/PVE/QemuMigrate/Helpers.pm
index 4e4e3ae0..b13992d1 100644
--- a/src/PVE/QemuMigrate/Helpers.pm
+++ b/src/PVE/QemuMigrate/Helpers.pm
@@ -4,6 +4,7 @@ use strict;
use warnings;
use JSON;
+use List::Util qw(min);
use PVE::Cluster;
use PVE::INotify;
@@ -128,6 +129,16 @@ my sub get_node_cpu_info {
return ref($info) eq 'HASH' ? $info : {};
}
+# Returns the CPU flags KVM supports on the node, as broadcast by pvestatd, if known.
+my sub get_node_kvm_cpu_flags {
+ my ($node) = @_;
+
+ # an empty value means that pvestatd failed to detect the flags
+ my $raw = PVE::Cluster::get_node_kv('cpuflags-kvm', $node)->{$node} or return;
+
+ return { map { $_ => 1 } PVE::QemuServer::CPUFlags::parse_node_cpu_flags($raw) };
+}
+
# Parses a QEMU '-cpu' argument into the CPU model and its properties. Flags given without a value
# are stored as 'on' or 'off'.
my sub parse_qemu_cpu_string {
@@ -161,13 +172,26 @@ my sub vendors_differ {
return defined($source_vendor) && defined($target_vendor) && $source_vendor ne $target_vendor;
}
+my sub format_flag_list {
+ my ($flags) = @_;
+
+ my $max_listed = 10;
+ my $list = join(', ', $flags->@[0 .. min($max_listed, scalar($flags->@*)) - 1]);
+ $list .= " and " . (scalar($flags->@*) - $max_listed) . " more"
+ if scalar($flags->@*) > $max_listed;
+
+ return $list;
+}
+
# Checks whether the CPU of a target node can run a running VM after a live migration.
#
# 'applies' returns true if the check is relevant for the VM's CPU, as parsed by
# parse_qemu_cpu_string(), with 'kvm' set if the VM uses KVM. 'needs' lists the CPU information
-# of the target node the check requires, nodes lacking any of it are skipped. 'check' returns why
-# the target node is incompatible, if it is. With the 'error' severity, an incompatibility
-# prevents the migration, with 'warning' it is only reported.
+# of the target node the check requires, nodes lacking any of it are skipped. The CPU information
+# is in the format of get_host_cpu_info(), plus the CPU flags KVM supports as 'kvm-flags' hash.
+# 'check' returns why the target node is incompatible, if it is. With the 'error' severity, an
+# incompatibility prevents the migration, with 'warning' it is only reported. 'override' names
+# the migration option that allows ignoring an error.
my $target_cpu_checks = [
{
type => 'cpu-vendor',
@@ -206,6 +230,40 @@ my $target_cpu_checks = [
. " to work, as guests can become unstable, even some time after the migration";
},
},
+ {
+ type => 'host-cpu-flags',
+ severity => 'error',
+ # the flags are only known per node, the VM might have been started on another node
+ override => 'allow-missing-cpu-flags',
+ needs => ['kvm-flags'],
+ applies => sub {
+ my ($cpu) = @_;
+ return is_host_cpu_model($cpu);
+ },
+ check => sub {
+ my ($cpu, $source, $target) = @_;
+
+ # already reported by the 'cpu-vendor' check
+ return if vendors_differ($source, $target);
+
+ my ($source_flags, $target_flags) =
+ ($source->{'kvm-flags'}, $target->{'kvm-flags'});
+ return if !$source_flags;
+
+ my @missing = sort grep {
+ # Hyper-V enlightenments are not passed through, but set explicitly
+ !m/^hv-/
+ && !$target_flags->{$_}
+ && ($cpu->{props}->{$_} // '') ne 'off'
+ } keys $source_flags->%*;
+ return if !scalar(@missing);
+
+ return
+ "CPU model '$cpu->{model}' passes through the flags of the host CPU, but the"
+ . " node lacks: "
+ . format_flag_list(\@missing);
+ },
+ },
];
# Returns the issues of the target node, and whether all checks that can prevent the migration
@@ -221,8 +279,9 @@ my sub get_target_cpu_issues {
}
my $message = $check->{check}->($cpu, $source_info, $target_info) or next;
- push $issues->@*,
- { type => $check->{type}, severity => $check->{severity}, message => $message };
+ my $issue = { type => $check->{type}, severity => $check->{severity}, message => $message };
+ $issue->{override} = $check->{override} if defined($check->{override});
+ push $issues->@*, $issue;
}
return ($issues, $complete);
@@ -241,7 +300,8 @@ Returns a hash reference with the list of issues for each node that has any, and
reference with the nodes that could not be checked for issues preventing the migration, because
they lack the required CPU information, for example as they run an older version. Each issue is
a hash with a C<type> identifying the check, a C<severity> of C<error> if the VM cannot run on
-the node or C<warning> if it might not, and a human-readable C<message>.
+the node or C<warning> if it might not, and a human-readable C<message>. If a migration option
+allows ignoring an error, C<override> names it.
=cut
@@ -258,12 +318,15 @@ sub check_target_cpu_compatibility {
# Use what this node broadcasts, as for the target nodes. The local pvestatd might not
# broadcast it yet, for example if it was not restarted after an upgrade.
- my $source_info = get_node_cpu_info(PVE::INotify::nodename());
+ my $local_node = PVE::INotify::nodename();
+ my $source_info = get_node_cpu_info($local_node);
$source_info = PVE::QemuServer::Helpers::get_host_cpu_info() if !scalar(keys $source_info->%*);
+ $source_info->{'kvm-flags'} = get_node_kvm_cpu_flags($local_node);
my ($issues_by_node, $unchecked_nodes) = ({}, []);
for my $node ($nodes->@*) {
my $target_info = get_node_cpu_info($node);
+ $target_info->{'kvm-flags'} = get_node_kvm_cpu_flags($node);
my ($issues, $complete) = get_target_cpu_issues($cpu, $checks, $source_info, $target_info);
$issues_by_node->{$node} = $issues if scalar($issues->@*);
diff --git a/src/test/MigrationTest/QemuMigrateMock.pm b/src/test/MigrationTest/QemuMigrateMock.pm
index 459a29a1..73e08227 100644
--- a/src/test/MigrationTest/QemuMigrateMock.pm
+++ b/src/test/MigrationTest/QemuMigrateMock.pm
@@ -237,6 +237,10 @@ $MigrationTest::Shared::qemu_migrate_helpers_module->mock(
$MigrationTest::Shared::cluster_module->mock(
get_node_kv => sub {
my ($key, $node) = @_;
+ if ($key eq 'cpuflags-kvm') {
+ my $flags = $test_cpu_info->{'kvm-flags'}->{$node} or return {};
+ return { $node => $flags };
+ }
if ($key eq 'host-cpu-info') {
# the source node is 'pve0', without 'source' get_host_cpu_info() is used
my $info =
diff --git a/src/test/run_qemu_migrate_tests.pl b/src/test/run_qemu_migrate_tests.pl
index 12ec3e90..0e33cfbc 100755
--- a/src/test/run_qemu_migrate_tests.pl
+++ b/src/test/run_qemu_migrate_tests.pl
@@ -531,7 +531,8 @@ my $tests = [
# target_volids - hash of volids on the target at the beginning
# vm_status - hash with running, runningmachine and optionally runningcpu
# cpu_info - hash with the CPU information of the 'source' node and of each
- # other node in 'nodes'
+ # other node in 'nodes', and the KVM CPU flags of each node in
+ # 'kvm-flags'
# expected_calls - hash whose keys are calls which are required
# to be made if the migration gets far enough
# expect_die - expect the migration call to fail, and an error message
@@ -1600,6 +1601,215 @@ my $tests = [
},
},
},
+ {
+ name => '1033_running_host_cpu_missing_flags',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ cpu_info => {
+ source => { vendor => 'GenuineIntel' },
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel' },
+ },
+ 'kvm-flags' => {
+ pve0 => 'avx512f fpu hv-relaxed sse',
+ pve1 => 'fpu sse',
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'host',
+ },
+ expected_calls => {},
+ expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes"
+ . " through the flags of the host CPU, but the node lacks: avx512f - use"
+ . " 'allow-missing-cpu-flags' to override",
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'host',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ },
+ },
+ {
+ # 'force', which the HA stack always sets, must not override the check
+ name => '1033_running_host_cpu_missing_flags_force',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ cpu_info => {
+ source => { vendor => 'GenuineIntel' },
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel' },
+ },
+ 'kvm-flags' => {
+ pve0 => 'avx512f fpu hv-relaxed sse',
+ pve1 => 'fpu sse',
+ },
+ },
+ opts => {
+ online => 1,
+ force => 1,
+ },
+ config_patch => {
+ cpu => 'host',
+ },
+ expected_calls => {},
+ expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes"
+ . " through the flags of the host CPU, but the node lacks: avx512f - use"
+ . " 'allow-missing-cpu-flags' to override",
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'host',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ },
+ },
+ {
+ name => '1033_running_host_cpu_missing_flags_allowed',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ cpu_info => {
+ source => { vendor => 'GenuineIntel' },
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel' },
+ },
+ 'kvm-flags' => {
+ pve0 => 'avx512f fpu hv-relaxed sse',
+ pve1 => 'fpu sse',
+ },
+ },
+ opts => {
+ online => 1,
+ 'allow-missing-cpu-flags' => 1,
+ },
+ config_patch => {
+ cpu => 'host',
+ },
+ expected_calls => $default_expected_calls_online,
+ expect_log => [
+ "ignoring incompatible CPU of node 'pve1' due to 'allow-missing-cpu-flags': CPU"
+ . " model 'host' passes through the flags of the host CPU, but the node lacks:"
+ . " avx512f",
+ ],
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'host',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ },
+ },
+ },
+ {
+ # nodes running an older version broadcast their KVM CPU flags, but no 'host-cpu-info'
+ name => '1033_running_host_cpu_missing_flags_without_cpu_info',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ cpu_info => {
+ source => { vendor => 'GenuineIntel' },
+ 'kvm-flags' => {
+ pve0 => 'avx512f fpu hv-relaxed sse',
+ pve1 => 'fpu sse',
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'host',
+ },
+ expected_calls => {},
+ expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes"
+ . " through the flags of the host CPU, but the node lacks: avx512f - use"
+ . " 'allow-missing-cpu-flags' to override",
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'host',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ },
+ },
+ {
+ # nodes that did not query their flags since an upgrade still broadcast aliases
+ name => '1033_running_host_cpu_flag_aliases',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt',
+ },
+ cpu_info => {
+ source => { vendor => 'GenuineIntel' },
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel' },
+ },
+ 'kvm-flags' => {
+ pve0 => 'fpu lahf-lm lahf_lm sse sse4-2 sse4.2 sse4_2',
+ pve1 => 'fpu lahf-lm sse sse4.2',
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'host',
+ },
+ expected_calls => $default_expected_calls_online,
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'host',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ },
+ },
+ },
{
name => '105_replicated_to_non_replication_target',
target => 'pve1',
--
2.47.3
next prev parent reply other threads:[~2026-11-18 15:43 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:40 [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 02/15] migrate: block live migration of host CPU VMs between CPU vendors Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 03/15] fix #6535: migrate: warn about live migration " Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 04/15] cpu flags: factor out parsing the broadcast CPU flags of a node Jonas Theisen
2026-11-18 15:40 ` Jonas Theisen [this message]
2026-11-18 15:40 ` [PATCH qemu-server 06/15] migrate: warn about live migration of host CPU VMs between CPU models Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 07/15] migrate: factor out whether the running virtual CPU is kept Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 08/15] migrate: block migration if the target cannot run the kept virtual CPU Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 09/15] helpers: add helper to query the host's TSC frequency and scaling support Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 11/15] migrate: block migration to nodes with incompatible TSC for re-enlightenment Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 12/15] pvestatd: broadcast host CPU information relevant for live migration Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 13/15] ui: migrate: show why the CPU of the target node is incompatible Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 14/15] fix #6535: ui: migrate: show CPU warnings for the target node Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 15/15] api: nodes: migrateall: skip VMs whose CPU is incompatible with the target Jonas Theisen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118154207.424260-6-j.theisen@proxmox.com \
--to=j.theisen@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.