[RFC PATCH-SERIES qemu-server 0/1] fix #7053: allow setting additional HA migration parameters

[RFC PATCH-SERIES qemu-server 0/1] fix #7053: allow setting additional HA migration parameters

[Daniel Kral]

Bugzilla #7053 reports that even though 'with-conntrack-state' is
checked, the VM will always migrate without conntrack state in the end.

In fact, any parameters from the migrate_vm API endpoint but the $vmid
and $node are not passed on to the HA stack at all. This was likely
caught now, because the conntrack state is the only optional parameter
visible in the web interface and set by default.



Currently, the resource motion crm command is matched from ^ to $:

    if ($cmd =~ m/^(migrate|relocate)\s+(\S+)\s+(\S+)$/) {

We could extend that crm command to something like:

    if ($cmd =~ m/^(migrate|relocate)\s+(\S+)\s+(\S+)(?:\s+(\S.*))?$/) {

but this would need the newer `ha-manager {migrate,relocate} ...`
API/CLI endpoint to append both the standard and extended version for
some period as older HA Manager versions wouldn't be able to parse the
extended version but only the standard versions. Newer HA Manager
versions would be fine though, as first the standard version would be
parsed and afterwards the extended version would overwrite the request
from the standard version.

The downside from this though is that the migration parameters are not
the same for VMs and CTs (and possible future resource types) and would
therefore expose quite a lot of resource-specific data structures to the
more generic HA Manager code.

Additionally, both the node with the active HA Manager as well as the
node's LRM where the to-be-moved HA resource is on need to have the
newer pve-ha-manager version to correctly relay the migration
parameters.



As the migrate_vm API request is proxied to the node where the HA
resource is assigned to, this RFC patch series puts the responsibility
to handle the additional migration parameters at the caller's side,
where these are saved while the request is relayed through the HA stack
until the LRM on the node calls migrate_vm again.

The implementation is not fully fleshed out (e.g. cleaning up the
migration params file on a crashed/stopped VM or rejected migration
requests, etc.), but I wanted to get more feedback whether this solution
has any merit and if not decide on another possible solution.

If it does have merit, this could be generalized for both qemu-server
and pve-container if it useful for containers as well.


qemu-server:

Daniel Kral (1):
  fix #7053: api: migrate: save and restore migration params for HA
    managed VMs

 src/PVE/API2/Qemu.pm | 54 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)


Summary over all repositories:
  1 files changed, 54 insertions(+), 0 deletions(-)

-- 
Generated by murpp 0.9.0

[RFC qemu-server 1/1] fix #7053: api: migrate: save and restore migration params for HA managed VMs

[Daniel Kral]

If a HA-managed VM's migrate API endpoint is called from a web API or
CLI environment, it is first relayed to the HA Manager by queueing a
'migrate' CRM command with `ha-manager migrate vm:$vmid $target_node`.
This command doesn't take any additional migration parameters though.

As soon as the HA Manager reads the CRM command in the next HA Manager
round, it passes the migration request - if valid - to the HA resource
state. This migration request is then picked up by the LRM, where the HA
resource is assigned to and calls the migrate_vm API endpoint, which
will then initial the VM migration.

As the migrate_vm API request is proxied to the node, where the HA
resource is assigned to, this allows the migrate parameters to stored
locally while the migration request is passed through the HA stack.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/API2/Qemu.pm | 54 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 1f0864f5..027896a3 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -3,6 +3,7 @@ package PVE::API2::Qemu;
 use strict;
 use warnings;
 use Cwd 'abs_path';
+use File::stat qw();
 use Net::SSLeay;
 use IO::Socket::IP;
 use IO::Socket::UNIX;
@@ -5367,6 +5368,52 @@ __PACKAGE__->register_method({
     },
 });
 
+my sub migrate_params_filename {
+    my ($vmid) = @_;
+    return "/run/qemu-server/$vmid.migrate_params";
+}
+
+my sub save_migrate_params {
+    my ($vmid, $params) = @_;
+
+    my $migrate_params_file = migrate_params_filename($vmid);
+
+    warn "existing migration parameters file for '$vmid' will be overwritten\n"
+        if -f $migrate_params_file;
+
+    PVE::Tools::file_set_contents($migrate_params_file, encode_json($params), 0640);
+}
+
+my sub try_to_restore_migrate_params {
+    my ($vmid, $params) = @_;
+
+    my $migrate_params_file = migrate_params_filename($vmid);
+    my @migrate_params_denylist = qw(node vmid target online force with-local-disks targetstorage);
+
+    if (-f $migrate_params_file) {
+        my $stat = File::stat::lstat($migrate_params_file);
+        # prevent that non-root users could write the migrate parameters file
+        my $has_correct_perms = $stat->uid == 0 && ($stat->mode & 037) == 0;
+
+        if (PVE::HA::Config::vm_is_ha_managed($vmid) && $has_correct_perms) {
+            my $migration_params = {};
+            eval {
+                my $data = PVE::Tools::file_get_contents($migrate_params_file);
+                $migration_params = decode_json($data) // {};
+            };
+            for my $key (keys %$migration_params) {
+                next if grep { $key eq $_ } @migrate_params_denylist;
+
+                $params->{$key} = $migration_params->{$key};
+            }
+        } else {
+            warn "remove orphan migration parameters file\n";
+        }
+
+        unlink $migrate_params_file;
+    }
+}
+
 __PACKAGE__->register_method({
     name => 'migrate_vm',
     path => '{vmid}/migrate',
@@ -5464,6 +5511,13 @@ __PACKAGE__->register_method({
 
         my $vmid = extract_param($param, 'vmid');
 
+        if (PVE::HA::Config::vm_is_ha_managed($vmid) && $rpcenv->{type} ne 'ha') {
+            save_migrate_params($vmid, $param);
+        } else {
+            # always try to restore to remove oprhaned migration parameters files
+            try_to_restore_migrate_params($vmid, $param);
+        }
+
         raise_param_exc({ force => "Only root may use this option." })
             if $param->{force} && $authuser ne 'root@pam';
 
-- 
2.47.3