From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 62EF61FF179 for ; Wed, 10 Dec 2025 11:10:04 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C832C17433; Wed, 10 Dec 2025 11:10:42 +0100 (CET) From: Fiona Ebner To: pve-devel@lists.proxmox.com Date: Wed, 10 Dec 2025 11:05:48 +0100 Message-ID: <20251210101006.37654-1-f.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1765361402618 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.167 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH qemu-server] dbus-vmstate: workaround method call on dbus object resolving to wrong instance X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" As reported in the community forum [0] and then later by Thomas, who provided the relevant system logs, parallel migration with '--with-conntrack-state' of multiple VMs may currently lead to a crash upon handover: > kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate' 0. > Make sure that your current VM setup matches your saved VM setup, > including any hotplugged devices > kvm: load of migration failed: Invalid argument In particular, the following sequence (on my test node) pvesh create /nodes/pve9a1/qemu/104/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action stop results in the wrong service being shut down (note the unexpected ID in the last line!): Dec 10 10:07:40 pve9a1 pvesh[30453]: starting dbus-vmstate helper for VM 104 Dec 10 10:07:40 pve9a1 systemd[1]: Starting pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104)... Dec 10 10:07:41 pve9a1 dbus-vmstate[30456]: pve-vmstate-104 listening on :1.55 Dec 10 10:07:41 pve9a1 systemd[1]: Started pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104). Dec 10 10:07:44 pve9a1 pvesh[30511]: starting dbus-vmstate helper for VM 105 Dec 10 10:07:44 pve9a1 systemd[1]: Starting pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105)... Dec 10 10:07:45 pve9a1 dbus-vmstate[30573]: pve-vmstate-105 listening on :1.58 Dec 10 10:07:45 pve9a1 systemd[1]: Started pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105). Dec 10 10:07:48 pve9a1 pvesh[30595]: stopping dbus-vmstate helper for VM 105 Dec 10 10:07:48 pve9a1 dbus-vmstate[30456]: shutting down gracefully .. Dec 10 10:07:48 pve9a1 systemd[1]: pve-dbus-vmstate@104.service: Deactivated successfully. So the dbus-vmstate object is removed from the wrong VM before loading the migration state. Note that the crash is still racy, because if the dbus-vmstate is removed on the source side for the same wrong VM before the migration handover, the QEMU objects for both instances will still match. To work around the issue, use 'systemctl' to shut down the correct instance. The root cause of why the 'Quit' method of the wrong instance is called, despite matching the 'Id' property of the DBus object first, still needs to be determined. [0]: https://forum.proxmox.com/threads/176821/post-820775 Reported-by: Thomas Lamprecht Signed-off-by: Fiona Ebner --- src/PVE/QemuServer/DBusVMState.pm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm index a72d6dd2..823a75ba 100644 --- a/src/PVE/QemuServer/DBusVMState.pm +++ b/src/PVE/QemuServer/DBusVMState.pm @@ -114,9 +114,12 @@ sub qemu_del_dbus_vmstate { $num_entries = eval { dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries'); }; - eval { $object->Quit() }; + # NOTE: for some reason, $object->Quit() does not resolve to the correct handler even + # though we matched the 'Id' property. As a stop-gap until this is figured out, do it + # via systemctl. + eval { PVE::Tools::run_command(['systemctl', 'stop', "pve-dbus-vmstate\@$vmid"]) }; if (my $err = $@) { - syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n") + syslog('warn', "failed to stop pve-dbus-vmstate service for VM $vmid: $err\n") if !$params{quiet}; } -- 2.47.3 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel