[pve-devel] [PATCH-SERIES qemu-server 0/3] migration: conntrack: fix two edge cases

[pve-devel] [PATCH-SERIES qemu-server 0/3] migration: conntrack: fix two edge cases

[Fiona Ebner]

Because of a systemd issue [0], when a service that's 'partOf' a scope
fails, the scope itself might end up being left-over, even after all
processes in the scope exit. In particular, this can happen for the
'$vmid.scope' when the 'pve-dbus-vmstate@$vmid.service' fails.

As reported in the community forum [1], it might happen that the
dbus-vmstate object is not added (quickly enough) to the target QEMU
instance, before the migration state is loaded. This would result in
a crash of the target instance.

[0]: https://github.com/systemd/systemd/issues/39141
[1]: https://forum.proxmox.com/threads/172588/

qemu-server:

Fiona Ebner (3):
  dbus vmstate: add missing includes
  migration: conntrack: work around systemd issue where scope for VM
    might become blocked
  migration: conntrack: avoid crash when dbus-vmstate object cannot be
    added (quickly enough)

 src/PVE/QemuServer.pm             |  6 ++++++
 src/PVE/QemuServer/DBusVMState.pm | 15 +++++++++++++++
 2 files changed, 21 insertions(+)


Summary over all repositories:
  2 files changed, 21 insertions(+), 0 deletions(-)

-- 
Generated by git-murpp 0.5.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 1/3] dbus vmstate: add missing includes

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/DBusVMState.pm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index 36ff8168..a72d6dd2 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -3,10 +3,15 @@ package PVE::QemuServer::DBusVMState;
 use strict;
 use warnings;
 
+use Net::DBus;
+use Net::DBus::RemoteService;
+
 use PVE::SafeSyslog;
 use PVE::Systemd;
 use PVE::Tools;
 
+use PVE::QemuServer::Helpers;
+
 use constant {
     DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
 };
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 2/3] migration: conntrack: work around systemd issue where scope for VM might become blocked

[Fiona Ebner]

Because of a systemd issue [0], when a service that's 'partOf' a scope
fails, the scope itself might end up being left-over, even after all
processes in the scope exit. In particular, this can happen for the
'$vmid.scope' when the 'pve-dbus-vmstate@$vmid.service' fails.

Doing a 'reset-failed' of the failed 'partOf' service leads to the
left-over scope being cleaned up too. Without that users in that
situation would get a difficult-to-make-sense-of "timeout waiting on
systemd" error message.

[0]: https://github.com/systemd/systemd/issues/39141

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 7d5ab718..8e2f03dc 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -5802,6 +5802,12 @@ sub vm_start_nolock {
     }
 
     my %silence_std_outs = (outfunc => sub { }, errfunc => sub { });
+    eval { # See systemd GH #39141, need to reset failed PartOf units too, or scope might be blocked
+        run_command(
+            ['/bin/systemctl', 'reset-failed', "pve-dbus-vmstate\@$vmid.service"],
+            %silence_std_outs,
+        );
+    };
     eval { run_command(['/bin/systemctl', 'reset-failed', "$vmid.scope"], %silence_std_outs) };
     eval { run_command(['/bin/systemctl', 'stop', "$vmid.scope"], %silence_std_outs) };
     # Issues with the above 'stop' not being fully completed are extremely rare, a very low
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server 3/3] migration: conntrack: avoid crash when dbus-vmstate object cannot be added (quickly enough)

[Fiona Ebner]

As reported in the community forum [0], it might happen that the
dbus-vmstate object is not added (quickly enough) to the target QEMU
instance, before the migration state is loaded. This would result in
a crash of the target instance:

> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate'
> 0. Make sure that your current VM setup matches your saved VM setup,
> including any hotplugged devices
> kvm: load of migration failed: Invalid argument

This is after the configuration is already moved and thus there also
is no source instance running anymore.

Make sure that the qemu_add_dbus_vmstate() function only returns
success when the object is successfully added.

[0]: https://forum.proxmox.com/threads/172588/

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/QemuServer/DBusVMState.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm
index a72d6dd2..354602a7 100644
--- a/src/PVE/QemuServer/DBusVMState.pm
+++ b/src/PVE/QemuServer/DBusVMState.pm
@@ -11,6 +11,7 @@ use PVE::Systemd;
 use PVE::Tools;
 
 use PVE::QemuServer::Helpers;
+use PVE::QemuServer::Monitor qw(mon_cmd);
 
 use constant {
     DBUS_VMSTATE_EXE => '/usr/libexec/qemu-server/dbus-vmstate',
@@ -59,6 +60,15 @@ sub qemu_add_dbus_vmstate {
     if (my $err = $@) {
         die "failed to start DBus VMState service for VM $vmid: $err\n";
     }
+
+    # Need to wait for the object to be added by the started service.
+    for (my $i = 0; $i < 5; $i++) {
+        my $qom_list = mon_cmd($vmid, 'qom-list', path => '/objects');
+        return if grep { $_->{name} eq 'pve-vmstate' } $qom_list->@*;
+        sleep 1;
+    }
+
+    die "timeout waiting for pve-vmstate object to show up\n";
 }
 
 # Stops the dbus-vmstate helper D-Bus service daemon and removes the associated
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel