From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 3C7841FF0C0 for ; Wed, 18 Nov 2026 16:44:07 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BC3682195E; Wed, 18 Nov 2026 16:42:57 +0100 (CET) From: Jonas Theisen To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server 10/15] cpu config: add hv-reenlightenment flag and set TSC frequency for it Date: Wed, 18 Nov 2026 16:40:41 +0100 Message-ID: <20261118154207.424260-11-j.theisen@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118154207.424260-1-j.theisen@proxmox.com> References: <20261118154207.424260-1-j.theisen@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795016559469 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.086 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: I7CN2JWJKX2EF6BXYR7AJZNFU3NT7KOR X-Message-ID-Hash: I7CN2JWJKX2EF6BXYR7AJZNFU3NT7KOR X-MailFrom: j.theisen@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Re-enlightenment notifications let nested Hyper-V guests, for example Windows with WSL2 or VBS enabled, pass a stable clock source on to their own guests across live migrations. KVM does not emulate the old TSC frequency after a migration, so QEMU refuses to load the migration stream of a guest that enabled these notifications if no TSC frequency was set explicitly. Set it to the host's TSC frequency, as reported by KVM, whenever the flag is enabled for the VM or by a custom CPU model. Treat the CPU configuration as abstracted if the flag is set, so that the TSC frequency of the running instance is kept for live migration, snapshots with RAM and hibernation, as the frequency of the host a VM is resumed on can differ. Signed-off-by: Jonas Theisen --- src/PVE/QemuServer/CPUConfig.pm | 18 +++++++++++++ src/PVE/QemuServer/CPUFlags.pm | 8 ++++++ src/test/QemuServerTest/CommandLineMocks.pm | 3 +++ src/test/cfg2cmd/cpu-hv-reenlightenment.conf | 6 +++++ .../cfg2cmd/cpu-hv-reenlightenment.conf.cmd | 26 +++++++++++++++++++ 5 files changed, 61 insertions(+) create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf create mode 100644 src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd diff --git a/src/PVE/QemuServer/CPUConfig.pm b/src/PVE/QemuServer/CPUConfig.pm index 414682a6..d629a2c6 100644 --- a/src/PVE/QemuServer/CPUConfig.pm +++ b/src/PVE/QemuServer/CPUConfig.pm @@ -718,6 +718,8 @@ determined from the configuration alone. Possible abstractions: =item abstract flag: for example, nested-virt changes with the host +=item host-dependent setting: for example, hv-reenlightenment sets the TSC frequency of the host + =back =cut @@ -732,6 +734,7 @@ sub is_abstracted($cpu_property_string) { for my $flag (split(";", $cpu_conf->{flags})) { if ($flag =~ $cpu_flag_supported_re) { return 1 if $2 eq 'nested-virt'; + return 1 if $2 eq 'hv-reenlightenment' && $1 eq '+'; } } return; @@ -965,6 +968,21 @@ sub get_cpu_options( $vm_flags, $pve_forced_flags, ); + + # Without an explicit TSC frequency, QEMU refuses to migrate a guest that enabled + # re-enlightenment notifications, as KVM does not emulate the old frequency afterwards. + my $reenlightenment = $resolved_flags->{'hv-reenlightenment'}; + if ($kvm && $arch eq 'x86_64' && $reenlightenment && $reenlightenment->{op} ne '-') { + my $tsc_khz = PVE::QemuServer::Helpers::get_host_tsc_info()->{khz} + or die "unable to determine the host TSC frequency required for" + . " 'hv-reenlightenment'\n"; + $resolved_flags->{'tsc-frequency'} = { + op => '', + value => $tsc_khz * 1000, + reason => "required by hv-reenlightenment", + }; + } + $cpu_str .= print_cpu_flags($resolved_flags); my $using_phys_bits_above_40; diff --git a/src/PVE/QemuServer/CPUFlags.pm b/src/PVE/QemuServer/CPUFlags.pm index 0ba2e115..bfdee207 100644 --- a/src/PVE/QemuServer/CPUFlags.pm +++ b/src/PVE/QemuServer/CPUFlags.pm @@ -76,6 +76,14 @@ my $supported_vm_specific_cpu_flags_by_arch = { description => "Improve performance for nested virtualization. Only supported on Intel" . " CPUs.", }, + { + name => 'hv-reenlightenment', + description => + "Allow nested Hyper-V guests, for example with WSL2 or VBS, to keep a stable" + . " clock source across live migrations. Sets the TSC frequency of the VM to" + . " the frequency of the host it is started on, so live migration only works" + . " to hosts with TSC scaling support or a TSC frequency within 250 ppm.", + }, { name => 'aes', description => "Activate AES instruction set for HW acceleration.", diff --git a/src/test/QemuServerTest/CommandLineMocks.pm b/src/test/QemuServerTest/CommandLineMocks.pm index 7fd4cc23..d0cd31c2 100644 --- a/src/test/QemuServerTest/CommandLineMocks.pm +++ b/src/test/QemuServerTest/CommandLineMocks.pm @@ -395,6 +395,9 @@ $qemu_server_helpers->mock( get_host_phys_address_bits => sub { return 46; }, + get_host_tsc_info => sub { + return { khz => 2999999, scaling => 1 }; + }, kvm_user_version => sub { return get_test_qemu_version(); }, diff --git a/src/test/cfg2cmd/cpu-hv-reenlightenment.conf b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf new file mode 100644 index 00000000..7e8150df --- /dev/null +++ b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf @@ -0,0 +1,6 @@ +# TEST: Test that enabling hv-reenlightenment sets the TSC frequency of the host +cpu: x86-64-v2-AES,flags=+hv-reenlightenment +memory: 768 +numa: 0 +ostype: win11 +sockets: 1 diff --git a/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd new file mode 100644 index 00000000..c3d8e49c --- /dev/null +++ b/src/test/cfg2cmd/cpu-hv-reenlightenment.conf.cmd @@ -0,0 +1,26 @@ +/usr/bin/kvm \ + -id 8006 \ + -name vm8006 \ + -no-shutdown \ + -chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' \ + -object 'monitor-qmp,chardev=qmp,id=monitor-qmp' \ + -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \ + -object 'monitor-qmp,chardev=qmp-event,id=monitor-qmp-event' \ + -pidfile /var/run/qemu-server/8006.pid \ + -daemonize \ + -smp '1,sockets=1,cores=1,maxcpus=1' \ + -nodefaults \ + -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \ + -vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' \ + -global 'kvm-pit.lost_tick_policy=discard' \ + -cpu 'qemu64,+aes,-cet-ibt,-cet-ss,enforce,+hv-reenlightenment,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,tsc-frequency=2999999000' \ + -m 768 \ + -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \ + -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \ + -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \ + -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \ + -device 'VGA,id=vga,bus=pci.0,addr=0x2,edid=off' \ + -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \ + -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' \ + -rtc 'driftfix=slew,base=localtime' \ + -machine 'hpet=off,type=pc-i440fx-5.1+pve0' -- 2.47.3