public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Jonas Theisen <j.theisen@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it
Date: Wed, 18 Nov 2026 16:40:41 +0100	[thread overview]
Message-ID: <20261118154207.424260-11-j.theisen@proxmox.com> (raw)
In-Reply-To: <20261118154207.424260-1-j.theisen@proxmox.com>

Re-enlightenment notifications let nested Hyper-V guests, for example
Windows with WSL2 or VBS enabled, pass a stable clock source on to
their own guests across live migrations.

KVM does not emulate the old TSC frequency after a migration, so QEMU
refuses to load the migration stream of a guest that enabled these
notifications if no TSC frequency was set explicitly. Set it to the
host's TSC frequency, as reported by KVM, whenever the flag is enabled
for the VM or by a custom CPU model.

Treat the CPU configuration as abstracted if the flag is set, so that
the TSC frequency of the running instance is kept for live migration,
snapshots with RAM and hibernation, as the frequency of the host a VM
is resumed on can differ.

Signed-off-by: Jonas Theisen <j.theisen@proxmox.com>
---
 src/PVE/QemuServer/CPUConfig.pm               | 18 +++++++++++++
 src/PVE/QemuServer/CPUFlags.pm                |  8 ++++++
 src/test/QemuServerTest/CommandLineMocks.pm   |  3 +++
 src/test/cfg2cmd/cpu-hv-reenlightenment.conf  |  6 +++++
 .../cfg2cmd/cpu-hv-reenlightenment.conf.cmd   | 26 +++++++++++++++++++
 5 files changed, 61 insertions(+)
 create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf
 create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd

diff --git a/src/PVE/QemuServer/CPUConfig.pm b/src/PVE/QemuServer/CPUConfig.pm
index 414682a6..d629a2c6 100644
--- a/src/PVE/QemuServer/CPUConfig.pm
+++ b/src/PVE/QemuServer/CPUConfig.pm
@@ -718,6 +718,8 @@ determined from the configuration alone. Possible abstractions:
 
 =item abstract flag: for example, nested-virt changes with the host
 
+=item host-dependent setting: for example, hv-reenlightenment sets the TSC frequency of the host
+
 =back
 
 =cut
@@ -732,6 +734,7 @@ sub is_abstracted($cpu_property_string) {
     for my $flag (split(";", $cpu_conf->{flags})) {
         if ($flag =~ $cpu_flag_supported_re) {
             return 1 if $2 eq 'nested-virt';
+            return 1 if $2 eq 'hv-reenlightenment' && $1 eq '+';
         }
     }
     return;
@@ -965,6 +968,21 @@ sub get_cpu_options(
         $vm_flags,
         $pve_forced_flags,
     );
+
+    # Without an explicit TSC frequency, QEMU refuses to migrate a guest that enabled
+    # re-enlightenment notifications, as KVM does not emulate the old frequency afterwards.
+    my $reenlightenment = $resolved_flags->{'hv-reenlightenment'};
+    if ($kvm && $arch eq 'x86_64' && $reenlightenment && $reenlightenment->{op} ne '-') {
+        my $tsc_khz = PVE::QemuServer::Helpers::get_host_tsc_info()->{khz}
+            or die "unable to determine the host TSC frequency required for"
+            . " 'hv-reenlightenment'\n";
+        $resolved_flags->{'tsc-frequency'} = {
+            op => '',
+            value => $tsc_khz * 1000,
+            reason => "required by hv-reenlightenment",
+        };
+    }
+
     $cpu_str .= print_cpu_flags($resolved_flags);
 
     my $using_phys_bits_above_40;
diff --git a/src/PVE/QemuServer/CPUFlags.pm b/src/PVE/QemuServer/CPUFlags.pm
index 0ba2e115..bfdee207 100644
--- a/src/PVE/QemuServer/CPUFlags.pm
+++ b/src/PVE/QemuServer/CPUFlags.pm
@@ -76,6 +76,14 @@ my $supported_vm_specific_cpu_flags_by_arch = {
             description =>
                 "Improve performance for nested virtualization. Only supported on Intel" . " CPUs.",
         },
+        {
+            name => 'hv-reenlightenment',
+            description =>
+                "Allow nested Hyper-V guests, for example with WSL2 or VBS, to keep a stable"
+                . " clock source across live migrations. Sets the TSC frequency of the VM to"
+                . " the frequency of the host it is started on, so live migration only works"
+                . " to hosts with TSC scaling support or a TSC frequency within 250 ppm.",
+        },
         {
             name => 'aes',
             description => "Activate AES instruction set for HW acceleration.",
diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm
index 7fd4cc23..d0cd31c2 100644
--- a/src/test/QemuServerTest/CommandLineMocks.pm
+++ b/src/test/QemuServerTest/CommandLineMocks.pm
@@ -395,6 +395,9 @@ $qemu_server_helpers->mock(
     get_host_phys_address_bits => sub {
         return 46;
     },
+    get_host_tsc_info => sub {
+        return { khz => 2999999, scaling => 1 };
+    },
     kvm_user_version => sub {
         return get_test_qemu_version();
     },
diff --git a/src/test/cfg2cmd/cpu-hv-reenlightenment.conf b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf
new file mode 100644
index 00000000..7e8150df
--- /dev/null
+++ b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf
@@ -0,0 +1,6 @@
+# TEST: Test that enabling hv-reenlightenment sets the TSC frequency of the host
+cpu: x86-64-v2-AES,flags=+hv-reenlightenment
+memory: 768
+numa: 0
+ostype: win11
+sockets: 1
diff --git a/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd
new file mode 100644
index 00000000..c3d8e49c
--- /dev/null
+++ b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd
@@ -0,0 +1,26 @@
+/usr/bin/kvm \
+  -id 8006 \
+  -name vm8006 \
+  -no-shutdown \
+  -chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' \
+  -object 'monitor-qmp,chardev=qmp,id=monitor-qmp' \
+  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \
+  -object 'monitor-qmp,chardev=qmp-event,id=monitor-qmp-event' \
+  -pidfile /var/run/qemu-server/8006.pid \
+  -daemonize \
+  -smp '1,sockets=1,cores=1,maxcpus=1' \
+  -nodefaults \
+  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
+  -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \
+  -global 'kvm-pit.lost_tick_policy=discard' \
+  -cpu 'qemu64,+aes,-cet-ibt,-cet-ss,enforce,+hv-reenlightenment,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,tsc-frequency=2999999000' \
+  -m 768 \
+  -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
+  -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
+  -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
+  -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
+  -device 'VGA,id=vga,bus=pci.0,addr=0x2,edid=off' \
+  -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
+  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \
+  -rtc 'driftfix=slew,base=localtime' \
+  -machine 'hpet=off,type=pc-i440fx-5.1+pve0'
-- 
2.47.3





  parent reply	other threads:[~2026-11-18 15:44 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 15:40 [PATCH manager/qemu-server 00/15] check the target node's CPU before live migrating a VM Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 01/15] test: migration: allow checking for warnings and errors in the log Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 02/15] migrate: block live migration of host CPU VMs between CPU vendors Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 03/15] fix #6535: migrate: warn about live migration " Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 04/15] cpu flags: factor out parsing the broadcast CPU flags of a node Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 05/15] fix #2924: migrate: block live migration of host CPU VMs to nodes lacking flags Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 06/15] migrate: warn about live migration of host CPU VMs between CPU models Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 07/15] migrate: factor out whether the running virtual CPU is kept Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 08/15] migrate: block migration if the target cannot run the kept virtual CPU Jonas Theisen
2026-11-18 15:40 ` [PATCH qemu-server 09/15] helpers: add helper to query the host's TSC frequency and scaling support Jonas Theisen
2026-11-18 15:40 ` Jonas Theisen [this message]
2026-11-18 15:40 ` [PATCH qemu-server 11/15] migrate: block migration to nodes with incompatible TSC for re-enlightenment Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 12/15] pvestatd: broadcast host CPU information relevant for live migration Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 13/15] ui: migrate: show why the CPU of the target node is incompatible Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 14/15] fix #6535: ui: migrate: show CPU warnings for the target node Jonas Theisen
2026-11-18 15:40 ` [PATCH manager 15/15] api: nodes: migrateall: skip VMs whose CPU is incompatible with the target Jonas Theisen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118154207.424260-11-j.theisen@proxmox.com \
    --to=j.theisen@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal