From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 0FD121FF0C0 for ; Wed, 18 Nov 2026 16:43:30 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5A034218E5; Wed, 18 Nov 2026 16:42:44 +0100 (CET) From: Jonas Theisen To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags Date: Wed, 18 Nov 2026 16:40:36 +0100 Message-ID: <20261118154207.424260-6-j.theisen@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118154207.424260-1-j.theisen@proxmox.com> References: <20261118154207.424260-1-j.theisen@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795016549020 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.036 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) PROLO_LEO1 0.1 Meta Catches all Leo drug variations so far RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: L7CWCJFBV54IVGQNP2ERCYXP3TJ5OE5D X-Message-ID-Hash: L7CWCJFBV54IVGQNP2ERCYXP3TJ5OE5D X-MailFrom: j.theisen@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The 'host' and 'max' CPU models pass the flags of the host CPU through to the guest. If the target node's CPU lacks flags that the guest might use, the live migration often fails only near its end, with QEMU on the target failing to set the special registers of the vCPUs, or the guest runs into problems after the migration. Compare the CPU flags that KVM supports on the source and the target node, which pvestatd already broadcasts, and refuse the migration if the target lacks any. Flags explicitly disabled for the VM and Hyper-V enlightenments, which are not passed through, are ignored. As the flags are broadcast independently of the other CPU information, this also works for target nodes running an older version. The flags are normalized, as nodes that did not query them again since an upgrade can still broadcast aliases of some flags. The flags of the source node are only an approximation of the flags of the VM, as it might have been started on another node, and the flags KVM supports also depend on the kernel version. So allow root to override the check with the new 'allow-missing-cpu-flags' option. The existing 'force' option is not reused for this, as the HA stack always sets it, which would skip the check for all HA-managed VMs. Instead, the new option is forwarded to the HA stack like other migration options, so that it also applies to HA-managed VMs. Signed-off-by: Jonas Theisen --- src/PVE/API2/Qemu.pm | 28 ++- src/PVE/QemuMigrate.pm | 13 +- src/PVE/QemuMigrate/Helpers.pm | 77 +++++++- src/test/MigrationTest/QemuMigrateMock.pm | 4 + src/test/run_qemu_migrate_tests.pl | 212 +++++++++++++++++++++- 5 files changed, 322 insertions(+), 12 deletions(-) diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index 9d7fcf8b..aaaa8760 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -155,6 +155,13 @@ my $migrate_json_properties = { . ' successfully by the target. Using this flag can lead to dropped' . ' configuration options if the target is too old to parse them.', }, + 'allow-missing-cpu-flags' => { + type => 'boolean', + optional => 1, + default => 0, + description => "Allow live migration of VMs with CPU type 'host' or 'max' to nodes" + . " lacking CPU flags of this node. Only root may use this option.", + }, }; # Properties that will be forwarded via the HA stack to the LRM, which will then use them for its @@ -163,6 +170,7 @@ my $migrate_json_properties = { my $ha_migrate_props = { 'with-conntrack-state' => 1, 'skip-config-check' => 1, + 'allow-missing-cpu-flags' => 1, }; sub ha_migrate_json_properties { @@ -5520,12 +5528,18 @@ __PACKAGE__->register_method({ type => { type => 'string', description => "The check that failed.", - enum => ['cpu-vendor'], + enum => ['cpu-vendor', 'host-cpu-flags'], }, message => { type => 'string', description => "Why the node's CPU is incompatible.", }, + override => { + type => 'string', + optional => 1, + description => "The migration option that allows" + . " ignoring the incompatibility.", + }, }, }, }, @@ -5730,8 +5744,13 @@ __PACKAGE__->register_method({ my @cpu_errors = grep { $_->{severity} eq 'error' } ($cpu_issues_by_node->{$node} // [])->@*; if (scalar(@cpu_errors)) { - $res->{not_allowed_nodes}->{$node}->{'incompatible-cpu'} = - [map { { type => $_->{type}, message => $_->{message} } } @cpu_errors]; + $res->{not_allowed_nodes}->{$node}->{'incompatible-cpu'} = [ + map { + my $error = { type => $_->{type}, message => $_->{message} }; + $error->{override} = $_->{override} if defined($_->{override}); + $error; + } @cpu_errors + ]; } my @cpu_warnings = @@ -5806,6 +5825,9 @@ __PACKAGE__->register_method({ raise_param_exc({ force => "Only root may use this option." }) if $param->{force} && $authuser ne 'root@pam'; + raise_param_exc({ 'allow-missing-cpu-flags' => "Only root may use this option." }) + if $param->{'allow-missing-cpu-flags'} && $authuser ne 'root@pam'; + raise_param_exc({ migration_type => "Only root may use this option." }) if $param->{migration_type} && $authuser ne 'root@pam'; diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm index 827f8143..bd157845 100644 --- a/src/PVE/QemuMigrate.pm +++ b/src/PVE/QemuMigrate.pm @@ -324,15 +324,26 @@ sub prepare { my @errors; for my $issue (($cpu_issues->{$node} // [])->@*) { + my $override = $issue->{override}; if ($issue->{severity} eq 'warning') { log_warn("CPU of node '$node' might be incompatible: $issue->{message}"); + } elsif (defined($override) && $self->{opts}->{$override}) { + log_warn("ignoring incompatible CPU of node '$node' due to '$override':" + . " $issue->{message}"); } else { push @errors, $issue; } } if (scalar(@errors)) { + my $hint = ''; + if (!grep { !defined($_->{override}) } @errors) { + my %overrides = map { $_->{override} => 1 } @errors; + $hint = + " - use " . join(' and ', map { "'$_'" } sort keys %overrides) . " to override"; + } die "can't migrate VM to '$node' - incompatible CPU: " - . join('; ', map { $_->{message} } @errors) . "\n"; + . join('; ', map { $_->{message} } @errors) + . "$hint\n"; } if (scalar($unchecked_nodes->@*)) { diff --git a/src/PVE/QemuMigrate/Helpers.pm b/src/PVE/QemuMigrate/Helpers.pm index 4e4e3ae0..b13992d1 100644 --- a/src/PVE/QemuMigrate/Helpers.pm +++ b/src/PVE/QemuMigrate/Helpers.pm @@ -4,6 +4,7 @@ use strict; use warnings; use JSON; +use List::Util qw(min); use PVE::Cluster; use PVE::INotify; @@ -128,6 +129,16 @@ my sub get_node_cpu_info { return ref($info) eq 'HASH' ? $info : {}; } +# Returns the CPU flags KVM supports on the node, as broadcast by pvestatd, if known. +my sub get_node_kvm_cpu_flags { + my ($node) = @_; + + # an empty value means that pvestatd failed to detect the flags + my $raw = PVE::Cluster::get_node_kv('cpuflags-kvm', $node)->{$node} or return; + + return { map { $_ => 1 } PVE::QemuServer::CPUFlags::parse_node_cpu_flags($raw) }; +} + # Parses a QEMU '-cpu' argument into the CPU model and its properties. Flags given without a value # are stored as 'on' or 'off'. my sub parse_qemu_cpu_string { @@ -161,13 +172,26 @@ my sub vendors_differ { return defined($source_vendor) && defined($target_vendor) && $source_vendor ne $target_vendor; } +my sub format_flag_list { + my ($flags) = @_; + + my $max_listed = 10; + my $list = join(', ', $flags->@[0 .. min($max_listed, scalar($flags->@*)) - 1]); + $list .= " and " . (scalar($flags->@*) - $max_listed) . " more" + if scalar($flags->@*) > $max_listed; + + return $list; +} + # Checks whether the CPU of a target node can run a running VM after a live migration. # # 'applies' returns true if the check is relevant for the VM's CPU, as parsed by # parse_qemu_cpu_string(), with 'kvm' set if the VM uses KVM. 'needs' lists the CPU information -# of the target node the check requires, nodes lacking any of it are skipped. 'check' returns why -# the target node is incompatible, if it is. With the 'error' severity, an incompatibility -# prevents the migration, with 'warning' it is only reported. +# of the target node the check requires, nodes lacking any of it are skipped. The CPU information +# is in the format of get_host_cpu_info(), plus the CPU flags KVM supports as 'kvm-flags' hash. +# 'check' returns why the target node is incompatible, if it is. With the 'error' severity, an +# incompatibility prevents the migration, with 'warning' it is only reported. 'override' names +# the migration option that allows ignoring an error. my $target_cpu_checks = [ { type => 'cpu-vendor', @@ -206,6 +230,40 @@ my $target_cpu_checks = [ . " to work, as guests can become unstable, even some time after the migration"; }, }, + { + type => 'host-cpu-flags', + severity => 'error', + # the flags are only known per node, the VM might have been started on another node + override => 'allow-missing-cpu-flags', + needs => ['kvm-flags'], + applies => sub { + my ($cpu) = @_; + return is_host_cpu_model($cpu); + }, + check => sub { + my ($cpu, $source, $target) = @_; + + # already reported by the 'cpu-vendor' check + return if vendors_differ($source, $target); + + my ($source_flags, $target_flags) = + ($source->{'kvm-flags'}, $target->{'kvm-flags'}); + return if !$source_flags; + + my @missing = sort grep { + # Hyper-V enlightenments are not passed through, but set explicitly + !m/^hv-/ + && !$target_flags->{$_} + && ($cpu->{props}->{$_} // '') ne 'off' + } keys $source_flags->%*; + return if !scalar(@missing); + + return + "CPU model '$cpu->{model}' passes through the flags of the host CPU, but the" + . " node lacks: " + . format_flag_list(\@missing); + }, + }, ]; # Returns the issues of the target node, and whether all checks that can prevent the migration @@ -221,8 +279,9 @@ my sub get_target_cpu_issues { } my $message = $check->{check}->($cpu, $source_info, $target_info) or next; - push $issues->@*, - { type => $check->{type}, severity => $check->{severity}, message => $message }; + my $issue = { type => $check->{type}, severity => $check->{severity}, message => $message }; + $issue->{override} = $check->{override} if defined($check->{override}); + push $issues->@*, $issue; } return ($issues, $complete); @@ -241,7 +300,8 @@ Returns a hash reference with the list of issues for each node that has any, and reference with the nodes that could not be checked for issues preventing the migration, because they lack the required CPU information, for example as they run an older version. Each issue is a hash with a C identifying the check, a C of C if the VM cannot run on -the node or C if it might not, and a human-readable C. +the node or C if it might not, and a human-readable C. If a migration option +allows ignoring an error, C names it. =cut @@ -258,12 +318,15 @@ sub check_target_cpu_compatibility { # Use what this node broadcasts, as for the target nodes. The local pvestatd might not # broadcast it yet, for example if it was not restarted after an upgrade. - my $source_info = get_node_cpu_info(PVE::INotify::nodename()); + my $local_node = PVE::INotify::nodename(); + my $source_info = get_node_cpu_info($local_node); $source_info = PVE::QemuServer::Helpers::get_host_cpu_info() if !scalar(keys $source_info->%*); + $source_info->{'kvm-flags'} = get_node_kvm_cpu_flags($local_node); my ($issues_by_node, $unchecked_nodes) = ({}, []); for my $node ($nodes->@*) { my $target_info = get_node_cpu_info($node); + $target_info->{'kvm-flags'} = get_node_kvm_cpu_flags($node); my ($issues, $complete) = get_target_cpu_issues($cpu, $checks, $source_info, $target_info); $issues_by_node->{$node} = $issues if scalar($issues->@*); diff --git a/src/test/MigrationTest/QemuMigrateMock.pm b/src/test/MigrationTest/QemuMigrateMock.pm index 459a29a1..73e08227 100644 --- a/src/test/MigrationTest/QemuMigrateMock.pm +++ b/src/test/MigrationTest/QemuMigrateMock.pm @@ -237,6 +237,10 @@ $MigrationTest::Shared::qemu_migrate_helpers_module->mock( $MigrationTest::Shared::cluster_module->mock( get_node_kv => sub { my ($key, $node) = @_; + if ($key eq 'cpuflags-kvm') { + my $flags = $test_cpu_info->{'kvm-flags'}->{$node} or return {}; + return { $node => $flags }; + } if ($key eq 'host-cpu-info') { # the source node is 'pve0', without 'source' get_host_cpu_info() is used my $info = diff --git a/src/test/run_qemu_migrate_tests.pl b/src/test/run_qemu_migrate_tests.pl index 12ec3e90..0e33cfbc 100755 --- a/src/test/run_qemu_migrate_tests.pl +++ b/src/test/run_qemu_migrate_tests.pl @@ -531,7 +531,8 @@ my $tests = [ # target_volids - hash of volids on the target at the beginning # vm_status - hash with running, runningmachine and optionally runningcpu # cpu_info - hash with the CPU information of the 'source' node and of each - # other node in 'nodes' + # other node in 'nodes', and the KVM CPU flags of each node in + # 'kvm-flags' # expected_calls - hash whose keys are calls which are required # to be made if the migration gets far enough # expect_die - expect the migration call to fail, and an error message @@ -1600,6 +1601,215 @@ my $tests = [ }, }, }, + { + name => '1033_running_host_cpu_missing_flags', + target => 'pve1', + vmid => 1033, + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + cpu_info => { + source => { vendor => 'GenuineIntel' }, + nodes => { + pve1 => { vendor => 'GenuineIntel' }, + }, + 'kvm-flags' => { + pve0 => 'avx512f fpu hv-relaxed sse', + pve1 => 'fpu sse', + }, + }, + opts => { + online => 1, + }, + config_patch => { + cpu => 'host', + }, + expected_calls => {}, + expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes" + . " through the flags of the host CPU, but the node lacks: avx512f - use" + . " 'allow-missing-cpu-flags' to override", + expected => { + source_volids => {}, + target_volids => {}, + vm_config => get_patched_config(1033, { + cpu => 'host', + }), + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + }, + }, + { + # 'force', which the HA stack always sets, must not override the check + name => '1033_running_host_cpu_missing_flags_force', + target => 'pve1', + vmid => 1033, + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + cpu_info => { + source => { vendor => 'GenuineIntel' }, + nodes => { + pve1 => { vendor => 'GenuineIntel' }, + }, + 'kvm-flags' => { + pve0 => 'avx512f fpu hv-relaxed sse', + pve1 => 'fpu sse', + }, + }, + opts => { + online => 1, + force => 1, + }, + config_patch => { + cpu => 'host', + }, + expected_calls => {}, + expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes" + . " through the flags of the host CPU, but the node lacks: avx512f - use" + . " 'allow-missing-cpu-flags' to override", + expected => { + source_volids => {}, + target_volids => {}, + vm_config => get_patched_config(1033, { + cpu => 'host', + }), + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + }, + }, + { + name => '1033_running_host_cpu_missing_flags_allowed', + target => 'pve1', + vmid => 1033, + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + cpu_info => { + source => { vendor => 'GenuineIntel' }, + nodes => { + pve1 => { vendor => 'GenuineIntel' }, + }, + 'kvm-flags' => { + pve0 => 'avx512f fpu hv-relaxed sse', + pve1 => 'fpu sse', + }, + }, + opts => { + online => 1, + 'allow-missing-cpu-flags' => 1, + }, + config_patch => { + cpu => 'host', + }, + expected_calls => $default_expected_calls_online, + expect_log => [ + "ignoring incompatible CPU of node 'pve1' due to 'allow-missing-cpu-flags': CPU" + . " model 'host' passes through the flags of the host CPU, but the node lacks:" + . " avx512f", + ], + expected => { + source_volids => {}, + target_volids => {}, + vm_config => get_patched_config(1033, { + cpu => 'host', + }), + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + }, + }, + }, + { + # nodes running an older version broadcast their KVM CPU flags, but no 'host-cpu-info' + name => '1033_running_host_cpu_missing_flags_without_cpu_info', + target => 'pve1', + vmid => 1033, + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + cpu_info => { + source => { vendor => 'GenuineIntel' }, + 'kvm-flags' => { + pve0 => 'avx512f fpu hv-relaxed sse', + pve1 => 'fpu sse', + }, + }, + opts => { + online => 1, + }, + config_patch => { + cpu => 'host', + }, + expected_calls => {}, + expect_die => "can't migrate VM to 'pve1' - incompatible CPU: CPU model 'host' passes" + . " through the flags of the host CPU, but the node lacks: avx512f - use" + . " 'allow-missing-cpu-flags' to override", + expected => { + source_volids => {}, + target_volids => {}, + vm_config => get_patched_config(1033, { + cpu => 'host', + }), + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + }, + }, + { + # nodes that did not query their flags since an upgrade still broadcast aliases + name => '1033_running_host_cpu_flag_aliases', + target => 'pve1', + vmid => 1033, + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + runningcpu => 'host,+kvm_pv_eoi,+kvm_pv_unhalt', + }, + cpu_info => { + source => { vendor => 'GenuineIntel' }, + nodes => { + pve1 => { vendor => 'GenuineIntel' }, + }, + 'kvm-flags' => { + pve0 => 'fpu lahf-lm lahf_lm sse sse4-2 sse4.2 sse4_2', + pve1 => 'fpu lahf-lm sse sse4.2', + }, + }, + opts => { + online => 1, + }, + config_patch => { + cpu => 'host', + }, + expected_calls => $default_expected_calls_online, + expected => { + source_volids => {}, + target_volids => {}, + vm_config => get_patched_config(1033, { + cpu => 'host', + }), + vm_status => { + running => 1, + runningmachine => 'pc-q35-5.0+pve0', + }, + }, + }, { name => '105_replicated_to_non_replication_target', target => 'pve1', -- 2.47.3