[pve-devel] [PATCH qemu-server] dbus-vmstate: workaround method call on dbus object resolving to wrong instance

[pve-devel] [PATCH qemu-server] dbus-vmstate: workaround method call on dbus object resolving to wrong instance

[Fiona Ebner]

As reported in the community forum [0] and then later by Thomas,
who provided the relevant system logs, parallel migration with
'--with-conntrack-state' of multiple VMs may currently lead to a
crash upon handover:

> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate' 0.
> Make sure that your current VM setup matches your saved VM setup,
> including any hotplugged devices
> kvm: load of migration failed: Invalid argument

In particular, the following sequence (on my test node)

pvesh create /nodes/pve9a1/qemu/104/dbus-vmstate --action start
pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action start
pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action stop

results in the wrong service being shut down (note the unexpected ID
in the last line!):

Dec 10 10:07:40 pve9a1 pvesh[30453]: starting dbus-vmstate helper for VM 104
Dec 10 10:07:40 pve9a1 systemd[1]: Starting pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104)...
Dec 10 10:07:41 pve9a1 dbus-vmstate[30456]: pve-vmstate-104 listening on :1.55
Dec 10 10:07:41 pve9a1 systemd[1]: Started pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104).
Dec 10 10:07:44 pve9a1 pvesh[30511]: starting dbus-vmstate helper for VM 105
Dec 10 10:07:44 pve9a1 systemd[1]: Starting pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105)...
Dec 10 10:07:45 pve9a1 dbus-vmstate[30573]: pve-vmstate-105 listening on :1.58
Dec 10 10:07:45 pve9a1 systemd[1]: Started pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105).
Dec 10 10:07:48 pve9a1 pvesh[30595]: stopping dbus-vmstate helper for VM 105
Dec 10 10:07:48 pve9a1 dbus-vmstate[30456]: shutting down gracefully ..
Dec 10 10:07:48 pve9a1 systemd[1]: pve-dbus-vmstate@104.service: Deactivated successfully.

So the dbus-vmstate object is removed from the wrong VM before loading
the migration state. Note that the crash is still racy, because if the
dbus-vmstate is removed on the source side for the same wrong VM before
the migration handover, the QEMU objects for both instances will still
match.

To work around the issue, use 'systemctl' to shut down the correct
instance. The root cause of why the 'Quit' method of the wrong
instance is called, despite matching the 'Id' property of the DBus
object first, still needs to be determined.

[0]: https://forum.proxmox.com/threads/176821/post-820775

Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/DBusVMState.pm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index a72d6dd2..823a75ba 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -114,9 +114,12 @@ sub qemu_del_dbus_vmstate {
             $num_entries = eval {
                 dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries');
             };
-            eval { $object->Quit() };
+            # NOTE: for some reason, $object->Quit() does not resolve to the correct handler even
+            # though we matched the 'Id' property. As a stop-gap until this is figured out, do it
+            # via systemctl.
+            eval { PVE::Tools::run_command(['systemctl', 'stop', "pve-dbus-vmstate\@$vmid"]) };
             if (my $err = $@) {
-                syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n")
+                syslog('warn', "failed to stop pve-dbus-vmstate service for VM $vmid: $err\n")
                     if !$params{quiet};
             }
 
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server] dbus-vmstate: workaround method call on dbus object resolving to wrong instance

[Fiona Ebner]

Am 10.12.25 um 11:10 AM schrieb Fiona Ebner:
> diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
> index a72d6dd2..823a75ba 100644
> --- a/src/PVE/QemuServer/DBusVMState.pm
> +++ b/src/PVE/QemuServer/DBusVMState.pm
> @@ -114,9 +114,12 @@ sub qemu_del_dbus_vmstate {
>              $num_entries = eval {
>                  dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries');
>              };
> -            eval { $object->Quit() };
> +            # NOTE: for some reason, $object->Quit() does not resolve to the correct handler even
> +            # though we matched the 'Id' property. As a stop-gap until this is figured out, do it
> +            # via systemctl.
> +            eval { PVE::Tools::run_command(['systemctl', 'stop', "pve-dbus-vmstate\@$vmid"]) };
>              if (my $err = $@) {
> -                syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n")
> +                syslog('warn', "failed to stop pve-dbus-vmstate service for VM $vmid: $err\n")
>                      if !$params{quiet};
>              }
>  

Hmm, maybe this is by design. We already have a helper for accessing the
properties, probably we need something similar for calling methods:

> # Retrieves a property from an object from a specific interface name.
> # In contrast to accessing the property directly by using $obj->Property, this
> # actually respects the owner of the object and thus can be used for interfaces
> # with might have multiple (queued) owners on the DBus.
> my sub dbus_get_property {
>     my ($obj, $interface, $name) = @_;
> 
>     my $con = $obj->{service}->get_bus()->get_connection();
> 
>     my $call = $con->make_method_call_message(
>         $obj->{service}->get_service_name(),
>         $obj->{object_path},
>         'org.freedesktop.DBus.Properties',
>         'Get',
>     );
> 
>     $call->set_destination($obj->get_service()->get_owner_name());
>     $call->append_args_list($interface, $name);
> 
>     my @reply = $con->send_with_reply_and_block($call, 10 * 1000)->get_args_list();
>     return $reply[0];
> }



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] superseded: [PATCH qemu-server] dbus-vmstate: workaround method call on dbus object resolving to wrong instance

[Fiona Ebner]

superseded-by:
https://lore.proxmox.com/pve-devel/20251210122009.105567-1-f.ebner@proxmox.com/T/



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel