[pve-devel] [PATCH-SERIES v3 qemu-server 0/2] dbus-vmstate: fix method call on dbus object resolving to wrong instance

[pve-devel] [PATCH-SERIES v3 qemu-server 0/2] dbus-vmstate: fix method call on dbus object resolving to wrong instance

[Fiona Ebner]

Changes in v3:
* split patch
* replace dbus_get_property() with a wrapper around dbus_call_method()

Similar to the dbus_get_property(), calling methods on a dbus object
also needs to be done with care to respect the owner when there are
multiple instances. This was not done for the 'Quit' method when
migrating leading to crashes with unlucky timing.

Fiona Ebner (2):
  dbus-vmstate: introduce dbus_call_method() helper
  dbus-vmstate: fix method call on dbus object resolving to wrong
    instance

 src/PVE/QemuServer/DBusVMState.pm | 42 ++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 14 deletions(-)

-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v3 qemu-server 1/2] dbus-vmstate: introduce dbus_call_method() helper

[Fiona Ebner]

Similar to the already existing dbus_get_property() helper, the owner
is respected even if there are multiple (queued) owners on the DBus.

Have the dbus_get_property() helper use the new function to avoid
duplication.

In preparation to fix the call to 'Quit'.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v3:
* split patch
* replace dbus_get_property() with a wrapper around dbus_call_method()

 src/PVE/QemuServer/DBusVMState.pm | 39 ++++++++++++++++++++-----------
 1 file changed, 26 insertions(+), 13 deletions(-)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index a72d6dd2..05dd1bcd 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -16,6 +16,30 @@ use constant {
     DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
 };
 
+# Call a method for an object from a specific interface name.
+# In contrast to calling the method directly by using $obj->Method(), this
+# actually respects the owner of the object and thus can be used for interfaces
+# with might have multiple (queued) owners on the DBus.
+my sub dbus_call_method {
+    my ($obj, $interface, $method, $params, $timeout) = @_;
+
+    $timeout = 10 if !$timeout;
+
+    my $con = $obj->{service}->get_bus()->get_connection();
+
+    my $call = $con->make_method_call_message(
+        $obj->{service}->get_service_name(),
+        $obj->{object_path},
+        $interface,
+        $method,
+    );
+
+    $call->set_destination($obj->get_service()->get_owner_name());
+    $call->append_args_list($params->@*) if $params;
+
+    return $con->send_with_reply_and_block($call, $timeout * 1000)->get_args_list();
+}
+
 # Retrieves a property from an object from a specific interface name.
 # In contrast to accessing the property directly by using $obj->Property, this
 # actually respects the owner of the object and thus can be used for interfaces
@@ -23,19 +47,8 @@ use constant {
 my sub dbus_get_property {
     my ($obj, $interface, $name) = @_;
 
-    my $con = $obj->{service}->get_bus()->get_connection();
-
-    my $call = $con->make_method_call_message(
-        $obj->{service}->get_service_name(),
-        $obj->{object_path},
-        'org.freedesktop.DBus.Properties',
-        'Get',
-    );
-
-    $call->set_destination($obj->get_service()->get_owner_name());
-    $call->append_args_list($interface, $name);
-
-    my @reply = $con->send_with_reply_and_block($call, 10 * 1000)->get_args_list();
+    my @reply =
+        dbus_call_method($obj, 'org.freedesktop.DBus.Properties', 'Get', [$interface, $name]);
     return $reply[0];
 }
 
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v3 qemu-server 2/2] dbus-vmstate: fix method call on dbus object resolving to wrong instance

[Fiona Ebner]

As reported in the community forum [0] and then later by Thomas,
who provided the relevant system logs, parallel migration with
'--with-conntrack-state' of multiple VMs may currently lead to a
crash upon handover:

> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate' 0.
> Make sure that your current VM setup matches your saved VM setup,
> including any hotplugged devices
> kvm: load of migration failed: Invalid argument

In particular, the following sequence (on my test node)

pvesh create /nodes/pve9a1/qemu/104/dbus-vmstate --action start
pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action start
pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action stop

results in the wrong service being shut down (note the unexpected ID
in the last line!):

Dec 10 10:07:40 pve9a1 pvesh[30453]: starting dbus-vmstate helper for VM 104
Dec 10 10:07:40 pve9a1 systemd[1]: Starting pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104)...
Dec 10 10:07:41 pve9a1 dbus-vmstate[30456]: pve-vmstate-104 listening on :1.55
Dec 10 10:07:41 pve9a1 systemd[1]: Started pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104).
Dec 10 10:07:44 pve9a1 pvesh[30511]: starting dbus-vmstate helper for VM 105
Dec 10 10:07:44 pve9a1 systemd[1]: Starting pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105)...
Dec 10 10:07:45 pve9a1 dbus-vmstate[30573]: pve-vmstate-105 listening on :1.58
Dec 10 10:07:45 pve9a1 systemd[1]: Started pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105).
Dec 10 10:07:48 pve9a1 pvesh[30595]: stopping dbus-vmstate helper for VM 105
Dec 10 10:07:48 pve9a1 dbus-vmstate[30456]: shutting down gracefully ..
Dec 10 10:07:48 pve9a1 systemd[1]: pve-dbus-vmstate@104.service: Deactivated successfully.

So the dbus-vmstate object is removed from the wrong VM before loading
the migration state. Note that the crash is still racy, because if the
dbus-vmstate is removed on the source side for the same wrong VM before
the migration handover, the QEMU objects for both instances will still
match.

To fix the issue, use the dbus_call_method() helper. Like, this the
owner is respected even if there are multiple (queued) owners on the
DBus.

[0]: https://forum.proxmox.com/threads/176821/post-820775

Fixes: dc76a590 ("fix #5180: migrate: integrate helper for live-migrating conntrack info")
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v3:
* split patch.

 src/PVE/QemuServer/DBusVMState.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index 05dd1bcd..480d9f70 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -127,7 +127,8 @@ sub qemu_del_dbus_vmstate {
             $num_entries = eval {
                 dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries');
             };
-            eval { $object->Quit() };
+            # Quit() does QMP object-del which has a timeout of 60 seconds
+            eval { dbus_call_method($object, 'com.proxmox.VMStateHelper', 'Quit', [], 70); };
             if (my $err = $@) {
                 syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n")
                     if !$params{quiet};
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v3 qemu-server 3/3] migrate: remove left-over dbus-vmstate instance when migrating without conntrack state

[Fiona Ebner]

As reported in the enterprise support, if something went wrong when
removing the dbus-vmstate object earlier, a following migrating using
'--with-conntrack-state' will lead to a crash.

Try to remove any leftover when migrating without the conntrack state.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Reported by Friedrich just after I sent the v3. Technically
orthogonal, and should get much less likely after patches 1+2 here,
but still good for future-proofing.

 src/PVE/QemuMigrate.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm
index 8fa84080..b3ddc34e 100644
--- a/src/PVE/QemuMigrate.pm
+++ b/src/PVE/QemuMigrate.pm
@@ -254,6 +254,12 @@ sub prepare {
                 'conntrack state migration not supported or disabled, '
                     . 'active connections might get dropped',
             );
+
+            # In case some leftover instance is running, stop it. The target QEMU instance won't
+            # have the 'dbus-vmstate' object, so the source must not have it either.
+            if (defined(PVE::QemuServer::DBusVMState::qemu_del_dbus_vmstate($vmid, quiet => 1))) {
+                $self->log('warn', "stopped left-over dbus-vmstate helper for VM $vmid");
+            }
         }
     }
 
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH v3 qemu-server 3/3] migrate: remove left-over dbus-vmstate instance when migrating without conntrack state

[Fiona Ebner]

Am 10.12.25 um 2:29 PM schrieb Fiona Ebner:
> As reported in the enterprise support, if something went wrong when
> removing the dbus-vmstate object earlier, a following migrating using

Sorry, this should be 'not using' here.

> '--with-conntrack-state' will lead to a crash.
> 
> Try to remove any leftover when migrating without the conntrack state.
> 
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> ---
> 
> Reported by Friedrich just after I sent the v3. Technically
> orthogonal, and should get much less likely after patches 1+2 here,
> but still good for future-proofing.
> 
>  src/PVE/QemuMigrate.pm | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm
> index 8fa84080..b3ddc34e 100644
> --- a/src/PVE/QemuMigrate.pm
> +++ b/src/PVE/QemuMigrate.pm
> @@ -254,6 +254,12 @@ sub prepare {
>                  'conntrack state migration not supported or disabled, '
>                      . 'active connections might get dropped',
>              );
> +
> +            # In case some leftover instance is running, stop it. The target QEMU instance won't
> +            # have the 'dbus-vmstate' object, so the source must not have it either.
> +            if (defined(PVE::QemuServer::DBusVMState::qemu_del_dbus_vmstate($vmid, quiet => 1))) {
> +                $self->log('warn', "stopped left-over dbus-vmstate helper for VM $vmid");
> +            }
>          }
>      }
>  



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH-SERIES v3 qemu-server 0/2] dbus-vmstate: fix method call on dbus object resolving to wrong instance

[Thomas Lamprecht]

On Wed, 10 Dec 2025 13:57:06 +0100, Fiona Ebner wrote:
> Changes in v3:
> * split patch
> * replace dbus_get_property() with a wrapper around dbus_call_method()
> 
> Similar to the dbus_get_property(), calling methods on a dbus object
> also needs to be done with care to respect the owner when there are
> multiple instances. This was not done for the 'Quit' method when
> migrating leading to crashes with unlucky timing.
> 
> [...]

Applied, thanks!

[1/2] dbus-vmstate: introduce dbus_call_method() helper
      commit: 05428a46147ea88c8648ce0f613b528704bec2f8
[2/2] dbus-vmstate: fix method call on dbus object resolving to wrong instance
      commit: b82c2578e7a452dd5119ca31b8847f75f22fe842
[3/3] migrate: remove left-over dbus-vmstate instance when migrating without conntrack state
      commit: 1f2c5146ac1f9cb1c7a9969283dbdc531e4091ea


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel