From: Jonas Theisen <j.theisen@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu-server 11/15] migrate: block migration to nodes with incompatible TSC for re-enlightenment
Date: Wed, 18 Nov 2026 16:40:42 +0100 [thread overview]
Message-ID: <20261118154207.424260-12-j.theisen@proxmox.com> (raw)
In-Reply-To: <20261118154207.424260-1-j.theisen@proxmox.com>
A VM with 'hv-reenlightenment' runs with the TSC frequency of the host
it was started on. QEMU on the target node can only use it if KVM
there supports TSC scaling, or if the target's TSC frequency is within
250 ppm, which KVM accepts without scaling. QEMU hard-codes this
bound [0], matching the default of KVM's 'tsc_tolerance_ppm' module
parameter [1]. Otherwise the target instance fails to start and the
migration is aborted only after its preparation, for example after
local disks were already allocated on the target.
Broadcast the TSC frequency and scaling support as part of the node's
CPU information, and check this upfront. Target nodes that do not
broadcast this information yet are not blocked, but a warning is
logged, as for the other checks.
[0]: https://gitlab.com/qemu-project/qemu/-/blob/v11.1.0/target/i386/kvm/kvm.c#L886
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/x86.c?h=v7.0#n164
Signed-off-by: Jonas Theisen <j.theisen@proxmox.com>
---
src/PVE/API2/Qemu.pm | 8 +-
src/PVE/QemuMigrate/Helpers.pm | 30 +++++
src/PVE/QemuServer/Helpers.pm | 12 ++
src/test/run_qemu_migrate_tests.pl | 183 +++++++++++++++++++++++++++++
4 files changed, 231 insertions(+), 2 deletions(-)
diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 5e14a1bb..17911368 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -5528,8 +5528,12 @@ __PACKAGE__->register_method({
type => {
type => 'string',
description => "The check that failed.",
- enum =>
- ['cpu-vendor', 'host-cpu-flags', 'kept-cpu-flags'],
+ enum => [
+ 'cpu-vendor',
+ 'host-cpu-flags',
+ 'kept-cpu-flags',
+ 'tsc-frequency',
+ ],
},
message => {
type => 'string',
diff --git a/src/PVE/QemuMigrate/Helpers.pm b/src/PVE/QemuMigrate/Helpers.pm
index 37fa10d4..df23d605 100644
--- a/src/PVE/QemuMigrate/Helpers.pm
+++ b/src/PVE/QemuMigrate/Helpers.pm
@@ -323,6 +323,36 @@ my $target_cpu_checks = [
. format_flag_list(\@missing);
},
},
+ {
+ type => 'tsc-frequency',
+ severity => 'error',
+ needs => ['tsc-scaling'],
+ applies => sub {
+ my ($cpu) = @_;
+ my $props = $cpu->{props};
+ return PVE::JSONSchema::parse_boolean($props->{'hv-reenlightenment'} // 'off')
+ && $props->{'tsc-frequency'};
+ },
+ check => sub {
+ my ($cpu, $source, $target) = @_;
+
+ return if $target->{'tsc-scaling'};
+
+ my $vm_khz = int($cpu->{props}->{'tsc-frequency'} / 1000);
+ my $requirement = "'hv-reenlightenment' requires TSC scaling support or a TSC"
+ . " frequency within 250 ppm of the VM's $vm_khz kHz";
+
+ my $target_khz = $target->{'tsc-khz'}
+ or return "$requirement, but the node's TSC frequency is unknown";
+
+ # without TSC scaling, KVM can only set frequencies within 250 ppm, which QEMU
+ # considers to be within NTP correction bounds
+ my $tolerance = int($target_khz * 250 / 1_000_000);
+ return if abs($vm_khz - $target_khz) <= $tolerance;
+
+ return "$requirement, but the node has $target_khz kHz without TSC scaling";
+ },
+ },
];
# Returns the issues of the target node, and whether all checks that can prevent the migration
diff --git a/src/PVE/QemuServer/Helpers.pm b/src/PVE/QemuServer/Helpers.pm
index 098d71eb..8b5f0cd8 100644
--- a/src/PVE/QemuServer/Helpers.pm
+++ b/src/PVE/QemuServer/Helpers.pm
@@ -426,6 +426,10 @@ encoded as JSON. The returned hash contains:
=item C<model-name>: the CPU model name, for example C<AMD EPYC 7302 16-Core Processor>
+=item C<tsc-khz>: the TSC frequency in kHz that KVM uses for new vCPUs
+
+=item C<tsc-scaling>: whether KVM can run vCPUs with a different TSC frequency
+
=back
Properties that cannot be determined are missing. Consumers of the broadcast information must
@@ -443,6 +447,14 @@ sub get_host_cpu_info {
$info->{vendor} = $cpuinfo->{vendor} if $known->($cpuinfo->{vendor});
$info->{'model-name'} = $cpuinfo->{model} if $known->($cpuinfo->{model});
+ my $tsc_info = eval { get_host_tsc_info() };
+ warn "unable to query TSC information - $@" if $@;
+
+ if ($tsc_info) {
+ $info->{'tsc-khz'} = $tsc_info->{khz} if defined($tsc_info->{khz});
+ $info->{'tsc-scaling'} = $tsc_info->{scaling} ? 1 : 0;
+ }
+
return $info;
}
diff --git a/src/test/run_qemu_migrate_tests.pl b/src/test/run_qemu_migrate_tests.pl
index 57aa59b1..538a55f3 100755
--- a/src/test/run_qemu_migrate_tests.pl
+++ b/src/test/run_qemu_migrate_tests.pl
@@ -1944,6 +1944,189 @@ my $tests = [
},
},
},
+ {
+ name => '1033_running_hv_reenlightenment_tsc_within_bounds',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ cpu_info => {
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel', 'tsc-khz' => 3000500, 'tsc-scaling' => 0 },
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ },
+ expected_calls => $default_expected_calls_online,
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ },
+ },
+ {
+ name => '1033_running_hv_reenlightenment_tsc_scaling',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ cpu_info => {
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel', 'tsc-khz' => 2400000, 'tsc-scaling' => 1 },
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ },
+ expected_calls => $default_expected_calls_online,
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ },
+ },
+ {
+ name => '1033_running_hv_reenlightenment_tsc_mismatch',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ cpu_info => {
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel', 'tsc-khz' => 2400000, 'tsc-scaling' => 0 },
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ },
+ expected_calls => {},
+ expect_die => "can't migrate VM to 'pve1' - incompatible CPU: 'hv-reenlightenment'"
+ . " requires TSC scaling support or a TSC frequency within 250 ppm of the VM's"
+ . " 2999999 kHz, but the node has 2400000 kHz without TSC scaling",
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ },
+ },
+ {
+ name => '1033_running_hv_reenlightenment_tsc_unknown',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ cpu_info => {
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel', 'tsc-scaling' => 0 },
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ },
+ expected_calls => {},
+ expect_die => "can't migrate VM to 'pve1' - incompatible CPU: 'hv-reenlightenment'"
+ . " requires TSC scaling support or a TSC frequency within 250 ppm of the VM's"
+ . " 2999999 kHz, but the node's TSC frequency is unknown",
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ },
+ },
+ {
+ name => '1033_running_hv_reenlightenment_target_without_tsc_info',
+ target => 'pve1',
+ vmid => 1033,
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ cpu_info => {
+ nodes => {
+ pve1 => { vendor => 'GenuineIntel' },
+ },
+ 'kvm-flags' => {
+ pve0 => 'fpu hv-reenlightenment sse',
+ pve1 => 'fpu hv-reenlightenment sse',
+ },
+ },
+ opts => {
+ online => 1,
+ },
+ config_patch => {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ },
+ expected_calls => $default_expected_calls_online,
+ expect_log => [
+ "unable to check CPU compatibility of node 'pve1' - CPU information not available",
+ ],
+ expected => {
+ source_volids => {},
+ target_volids => {},
+ vm_config => get_patched_config(1033, {
+ cpu => 'kvm64,flags=+hv-reenlightenment',
+ }),
+ vm_status => {
+ running => 1,
+ runningmachine => 'pc-q35-5.0+pve0',
+ runningcpu => 'kvm64,+hv-reenlightenment,tsc-frequency=2999999000',
+ },
+ },
+ },
{
name => '105_replicated_to_non_replication_target',
target => 'pve1',
--
2.47.3
next prev parent reply other threads:[~2026-11-18 15:44 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 15:40 [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 02/15] migrate: block live migration of host CPU VMs between CPU vendors Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 03/15] fix #6535: migrate: warn about live migration " Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 04/15] cpu flags: factor out parsing the broadcast CPU flags of a node Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 06/15] migrate: warn about live migration of host CPU VMs between CPU models Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 07/15] migrate: factor out whether the running virtual CPU is kept Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 08/15] migrate: block migration if the target cannot run the kept virtual CPU Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 09/15] helpers: add helper to query the host's TSC frequency and scaling support Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it Jonas Theisen
2026-11-18 15:40 ` Jonas Theisen [this message]
2026-11-18 15:40 ` [PATCH manager 12/15] pvestatd: broadcast host CPU information relevant for live migration Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 13/15] ui: migrate: show why the CPU of the target node is incompatible Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 14/15] fix #6535: ui: migrate: show CPU warnings for the target node Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 15/15] api: nodes: migrateall: skip VMs whose CPU is incompatible with the target Jonas Theisen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118154207.424260-12-j.theisen@proxmox.com \
--to=j.theisen@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.