[pve-devel] [PATCH-SERIES qemu-server 0/3] migration: conntrack: fix two edge cases

[pve-devel] [PATCH-SERIES qemu-server 0/3] migration: conntrack: fix two edge cases

[Fiona Ebner]

Because of a systemd issue [0], when a service that's 'partOf' a scope
fails, the scope itself might end up being left-over, even after all
processes in the scope exit. In particular, this can happen for the
'$vmid.scope' when the 'pve-dbus-vmstate@$vmid.service' fails.

As reported in the community forum [1], it might happen that the
dbus-vmstate object is not added (quickly enough) to the target QEMU
instance, before the migration state is loaded. This would result in
a crash of the target instance.

[0]: https://github.com/systemd/systemd/issues/39141
[1]: https://forum.proxmox.com/threads/172588/

qemu-server:

Fiona Ebner (3):
  dbus vmstate: add missing includes
  migration: conntrack: work around systemd issue where scope for VM
    might become blocked
  migration: conntrack: avoid crash when dbus-vmstate object cannot be
    added (quickly enough)

 src/PVE/QemuServer.pm             |  6 ++++++
 src/PVE/QemuServer/DBusVMState.pm | 15 +++++++++++++++
 2 files changed, 21 insertions(+)


Summary over all repositories:
  2 files changed, 21 insertions(+), 0 deletions(-)

-- 
Generated by git-murpp 0.5.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 1/3] dbus vmstate: add missing includes

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/DBusVMState.pm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index 36ff8168..a72d6dd2 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -3,10 +3,15 @@ package PVE::QemuServer::DBusVMState;
 use strict;
 use warnings;
 
+use Net::DBus;
+use Net::DBus::RemoteService;
+
 use PVE::SafeSyslog;
 use PVE::Systemd;
 use PVE::Tools;
 
+use PVE::QemuServer::Helpers;
+
 use constant {
     DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
 };
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 2/3] migration: conntrack: work around systemd issue where scope for VM might become blocked

[Fiona Ebner]

Because of a systemd issue [0], when a service that's 'partOf' a scope
fails, the scope itself might end up being left-over, even after all
processes in the scope exit. In particular, this can happen for the
'$vmid.scope' when the 'pve-dbus-vmstate@$vmid.service' fails.

Doing a 'reset-failed' of the failed 'partOf' service leads to the
left-over scope being cleaned up too. Without that users in that
situation would get a difficult-to-make-sense-of "timeout waiting on
systemd" error message.

[0]: https://github.com/systemd/systemd/issues/39141

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 7d5ab718..8e2f03dc 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -5802,6 +5802,12 @@ sub vm_start_nolock {
     }
 
     my %silence_std_outs = (outfunc => sub { }, errfunc => sub { });
+    eval { # See systemd GH #39141, need to reset failed PartOf units too, or scope might be blocked
+        run_command(
+            ['/bin/systemctl', 'reset-failed', "pve-dbus-vmstate\@$vmid.service"],
+            %silence_std_outs,
+        );
+    };
     eval { run_command(['/bin/systemctl', 'reset-failed', "$vmid.scope"], %silence_std_outs) };
     eval { run_command(['/bin/systemctl', 'stop', "$vmid.scope"], %silence_std_outs) };
     # Issues with the above 'stop' not being fully completed are extremely rare, a very low
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 3/3] migration: conntrack: avoid crash when dbus-vmstate object cannot be added (quickly enough)

[Fiona Ebner]

As reported in the community forum [0], it might happen that the
dbus-vmstate object is not added (quickly enough) to the target QEMU
instance, before the migration state is loaded. This would result in
a crash of the target instance:

> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate'
> 0. Make sure that your current VM setup matches your saved VM setup,
> including any hotplugged devices
> kvm: load of migration failed: Invalid argument

This is after the configuration is already moved and thus there also
is no source instance running anymore.

Make sure that the qemu_add_dbus_vmstate() function only returns
success when the object is successfully added.

[0]: https://forum.proxmox.com/threads/172588/

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/DBusVMState.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index a72d6dd2..354602a7 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -11,6 +11,7 @@ use PVE::Systemd;
 use PVE::Tools;
 
 use PVE::QemuServer::Helpers;
+use PVE::QemuServer::Monitor qw(mon_cmd);
 
 use constant {
     DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
@@ -59,6 +60,15 @@ sub qemu_add_dbus_vmstate {
     if (my $err = $@) {
         die "failed to start DBus VMState service for VM $vmid: $err\n";
     }
+
+    # Need to wait for the object to be added by the started service.
+    for (my $i = 0; $i < 5; $i++) {
+        my $qom_list = mon_cmd($vmid, 'qom-list', path => '/objects');
+        return if grep { $_->{name} eq 'pve-vmstate' } $qom_list->@*;
+        sleep 1;
+    }
+
+    die "timeout waiting for pve-vmstate object to show up\n";
 }
 
 # Stops the dbus-vmstate helper D-Bus service daemon and removes the associated
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server 3/3] migration: conntrack: avoid crash when dbus-vmstate object cannot be added (quickly enough)

[Fabian Grünbichler]

On September 29, 2025 2:24 pm, Fiona Ebner wrote:
> As reported in the community forum [0], it might happen that the
> dbus-vmstate object is not added (quickly enough) to the target QEMU
> instance, before the migration state is loaded. This would result in
> a crash of the target instance:
> 
>> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate'
>> 0. Make sure that your current VM setup matches your saved VM setup,
>> including any hotplugged devices
>> kvm: load of migration failed: Invalid argument
> 
> This is after the configuration is already moved and thus there also
> is no source instance running anymore.
> 
> Make sure that the qemu_add_dbus_vmstate() function only returns
> success when the object is successfully added.
> 
> [0]: https://forum.proxmox.com/threads/172588/
> 
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> ---
>  src/PVE/QemuServer/DBusVMState.pm | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
> index a72d6dd2..354602a7 100644
> --- a/src/PVE/QemuServer/DBusVMState.pm
> +++ b/src/PVE/QemuServer/DBusVMState.pm
> @@ -11,6 +11,7 @@ use PVE::Systemd;
>  use PVE::Tools;
>  
>  use PVE::QemuServer::Helpers;
> +use PVE::QemuServer::Monitor qw(mon_cmd);
>  
>  use constant {
>      DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
> @@ -59,6 +60,15 @@ sub qemu_add_dbus_vmstate {
>      if (my $err = $@) {
>          die "failed to start DBus VMState service for VM $vmid: $err\n";
>      }
> +
> +    # Need to wait for the object to be added by the started service.
> +    for (my $i = 0; $i < 5; $i++) {
> +        my $qom_list = mon_cmd($vmid, 'qom-list', path => '/objects');
> +        return if grep { $_->{name} eq 'pve-vmstate' } $qom_list->@*;
> +        sleep 1;
> +    }
> +
> +    die "timeout waiting for pve-vmstate object to show up\n";

wouldn't this be better handled by making the service started here
type=notify?

>  }
>  
>  # Stops the dbus-vmstate helper D-Bus service daemon and removes the associated
> -- 
> 2.47.3
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server 3/3] migration: conntrack: avoid crash when dbus-vmstate object cannot be added (quickly enough)

[Fiona Ebner]

Am 03.10.25 um 1:42 PM schrieb Fabian Grünbichler:
> On September 29, 2025 2:24 pm, Fiona Ebner wrote:
>> As reported in the community forum [0], it might happen that the
>> dbus-vmstate object is not added (quickly enough) to the target QEMU
>> instance, before the migration state is loaded. This would result in
>> a crash of the target instance:
>>
>>> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate'
>>> 0. Make sure that your current VM setup matches your saved VM setup,
>>> including any hotplugged devices
>>> kvm: load of migration failed: Invalid argument
>>
>> This is after the configuration is already moved and thus there also
>> is no source instance running anymore.
>>
>> Make sure that the qemu_add_dbus_vmstate() function only returns
>> success when the object is successfully added.
>>
>> [0]: https://forum.proxmox.com/threads/172588/
>>
>> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
>> ---
>>  src/PVE/QemuServer/DBusVMState.pm | 10 ++++++++++
>>  1 file changed, 10 insertions(+)
>>
>> diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
>> index a72d6dd2..354602a7 100644
>> --- a/src/PVE/QemuServer/DBusVMState.pm
>> +++ b/src/PVE/QemuServer/DBusVMState.pm
>> @@ -11,6 +11,7 @@ use PVE::Systemd;
>>  use PVE::Tools;
>>  
>>  use PVE::QemuServer::Helpers;
>> +use PVE::QemuServer::Monitor qw(mon_cmd);
>>  
>>  use constant {
>>      DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
>> @@ -59,6 +60,15 @@ sub qemu_add_dbus_vmstate {
>>      if (my $err = $@) {
>>          die "failed to start DBus VMState service for VM $vmid: $err\n";
>>      }
>> +
>> +    # Need to wait for the object to be added by the started service.
>> +    for (my $i = 0; $i < 5; $i++) {
>> +        my $qom_list = mon_cmd($vmid, 'qom-list', path => '/objects');
>> +        return if grep { $_->{name} eq 'pve-vmstate' } $qom_list->@*;
>> +        sleep 1;
>> +    }
>> +
>> +    die "timeout waiting for pve-vmstate object to show up\n";
> 
> wouldn't this be better handled by making the service started here
> type=notify?

Yes, good idea! Will send a v2.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] partially-applied: [PATCH-SERIES qemu-server 0/3] migration: conntrack: fix two edge cases

[Fabian Grünbichler]

On Mon, 29 Sep 2025 14:24:45 +0200, Fiona Ebner wrote:
> Because of a systemd issue [0], when a service that's 'partOf' a scope
> fails, the scope itself might end up being left-over, even after all
> processes in the scope exit. In particular, this can happen for the
> '$vmid.scope' when the 'pve-dbus-vmstate@$vmid.service' fails.
> 
> As reported in the community forum [1], it might happen that the
> dbus-vmstate object is not added (quickly enough) to the target QEMU
> instance, before the migration state is loaded. This would result in
> a crash of the target instance.
> 
> [...]

[1/3] dbus vmstate: add missing includes
      commit: 5595b5af806a328778e94b05776ee5c6edd56d74
[2/3] migration: conntrack: work around systemd issue where scope for VM might become blocked
      commit: 26717ab3b8796c6f4aa0eedd9a7537091ed73d17

Applied these two, thanks!

[3/3] migration: conntrack: avoid crash when dbus-vmstate object cannot be added (quickly enough)
      (no commit info)

Comment for that sent in a separate mail!

Best regards,
-- 
Fabian Grünbichler <f.gruenbichler@proxmox.com>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel