From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 455751FF179 for ; Wed, 10 Dec 2025 13:19:38 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CB3F01AE33; Wed, 10 Dec 2025 13:20:16 +0100 (CET) From: Fiona Ebner To: pve-devel@lists.proxmox.com Date: Wed, 10 Dec 2025 13:19:11 +0100 Message-ID: <20251210122009.105567-1-f.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1765369205878 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.168 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [dbusvmstate.pm, proxmox.com] Subject: [pve-devel] [PATCH v2 qemu-server] dbus-vmstate: fix method call on dbus object resolving to wrong instance X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" As reported in the community forum [0] and then later by Thomas, who provided the relevant system logs, parallel migration with '--with-conntrack-state' of multiple VMs may currently lead to a crash upon handover: > kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate' 0. > Make sure that your current VM setup matches your saved VM setup, > including any hotplugged devices > kvm: load of migration failed: Invalid argument In particular, the following sequence (on my test node) pvesh create /nodes/pve9a1/qemu/104/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action stop results in the wrong service being shut down (note the unexpected ID in the last line!): Dec 10 10:07:40 pve9a1 pvesh[30453]: starting dbus-vmstate helper for VM 104 Dec 10 10:07:40 pve9a1 systemd[1]: Starting pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104)... Dec 10 10:07:41 pve9a1 dbus-vmstate[30456]: pve-vmstate-104 listening on :1.55 Dec 10 10:07:41 pve9a1 systemd[1]: Started pve-dbus-vmstate@104.service - PVE DBus VMState Helper (VM 104). Dec 10 10:07:44 pve9a1 pvesh[30511]: starting dbus-vmstate helper for VM 105 Dec 10 10:07:44 pve9a1 systemd[1]: Starting pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105)... Dec 10 10:07:45 pve9a1 dbus-vmstate[30573]: pve-vmstate-105 listening on :1.58 Dec 10 10:07:45 pve9a1 systemd[1]: Started pve-dbus-vmstate@105.service - PVE DBus VMState Helper (VM 105). Dec 10 10:07:48 pve9a1 pvesh[30595]: stopping dbus-vmstate helper for VM 105 Dec 10 10:07:48 pve9a1 dbus-vmstate[30456]: shutting down gracefully .. Dec 10 10:07:48 pve9a1 systemd[1]: pve-dbus-vmstate@104.service: Deactivated successfully. So the dbus-vmstate object is removed from the wrong VM before loading the migration state. Note that the crash is still racy, because if the dbus-vmstate is removed on the source side for the same wrong VM before the migration handover, the QEMU objects for both instances will still match. To fix the issue, introduce a dbus_call_method() helper similar to the already existing dbus_get_property() one. Like, this the owner is respected even if there are multiple (queued) owners on the DBus. [0]: https://forum.proxmox.com/threads/176821/post-820775 Reported-by: Thomas Lamprecht Signed-off-by: Fiona Ebner --- Changes in v2: * Introduce a helper for calling dbus methods which respects the owner src/PVE/QemuServer/DBusVMState.pm | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm index a72d6dd2..f1766035 100644 --- a/src/PVE/QemuServer/DBusVMState.pm +++ b/src/PVE/QemuServer/DBusVMState.pm @@ -39,6 +39,30 @@ my sub dbus_get_property { return $reply[0]; } +# Call a method for an object from a specific interface name. +# In contrast to calling the method directly by using $obj->Method(), this +# actually respects the owner of the object and thus can be used for interfaces +# with might have multiple (queued) owners on the DBus. +my sub dbus_call_method { + my ($obj, $interface, $method, $params, $timeout) = @_; + + $timeout = 10 if !$timeout; + + my $con = $obj->{service}->get_bus()->get_connection(); + + my $call = $con->make_method_call_message( + $obj->{service}->get_service_name(), + $obj->{object_path}, + $interface, + $method, + ); + + $call->set_destination($obj->get_service()->get_owner_name()); + $call->append_args_list($params->@*) if $params; + + return $con->send_with_reply_and_block($call, $timeout * 1000)->get_args_list(); +} + # Starts the dbus-vmstate helper D-Bus service daemon and adds the needed # object to the appropriate QEMU instance for the specified VM. sub qemu_add_dbus_vmstate { @@ -114,7 +138,8 @@ sub qemu_del_dbus_vmstate { $num_entries = eval { dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries'); }; - eval { $object->Quit() }; + # Quit() does QMP object-del which has a timeout of 60 seconds + eval { dbus_call_method($object, 'com.proxmox.VMStateHelper', 'Quit', [], 70); }; if (my $err = $@) { syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n") if !$params{quiet}; -- 2.47.3 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel