[pve-devel] [PATCH v2 pve-container/qemu-server/pve-docs/pve-guest-common 0/8] Add pre/post-migrate hooks

[pve-devel] [PATCH v2 pve-container/qemu-server/pve-docs/pve-guest-common 0/8] Add pre/post-migrate hooks

[Stefan Hanreich]

This series adds pre/post-migrate hooks to the migration process of containers
and VMs.

Additionally to testing the API/CLI migration from updated nodes, I have also
made sure to keep backwards compatibility with old nodes. This means that
migration to a node that has not yet been upgraded to support the migration hook
functionality works flawlessly. The behaviour in this case is that a warning
is printed and the migration process then continues as usual.

Changes from v1:
- remove the newly added CLI commands
- remove the abstract methods from pve-guest-common
- Add migrate-hook and query-migrate-hook commands to qm mtunnel
- Add migrate-hook functionality to Qemu backend
- Add pct mtunnel command
- Add migrate-hook, query-migrate-hook and quit commands to pct mtunnel
- Add migrate-hook functionality to LXC backend
- Add documentation for migrate-hooks to VM Hookscript documentation
- Add section for Hookscripts to the Container documentation
- Extend exec_hookscript functionality to include run parameters for run_command


pve-container:

Stefan Hanreich (2):
  add pct mtunnel command to the CLI
  add migration hooks to container migration process

 src/PVE/CLI/pct.pm     | 230 +++++++++++++++++++++++++++++++++++++++++
 src/PVE/LXC/Migrate.pm | 119 +++++++++++++++++++++
 2 files changed, 349 insertions(+)


qemu-server:

Stefan Hanreich (2):
  add migrate-hook and query-migrate-hook commands to CLI
  add migration hooks to VM migration process

 PVE/CLI/qm.pm                         | 109 ++++++++++++++++++++++++++
 PVE/QemuMigrate.pm                    | 108 +++++++++++++++++++++++++
 test/MigrationTest/QemuMigrateMock.pm |  11 ++-
 3 files changed, 227 insertions(+), 1 deletion(-)


pve-docs:

Stefan Hanreich (3):
  Add pre/post-migrate events to hookscript example
  Add hookscript section to container documentation
  Add pre/post-migrate section to VM hookscript documentation

 examples/guest-example-hookscript.pl | 20 ++++++++++++++++++++
 pct.adoc                             | 23 +++++++++++++++++++++++
 qm.adoc                              | 10 ++++++++++
 3 files changed, 53 insertions(+)


pve-guest-common:

Stefan Hanreich (1):
  Add run_params to exec_hookscript function

 src/PVE/GuestHelpers.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
2.30.2

[pve-devel] [PATCH v2 pve-guest-common 1/1] Add run_params to exec_hookscript function

[Stefan Hanreich]

This enables us to pass parameters to the run_command call so the
hookscripts have a more fine-grained control over how the script should
get executed.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 src/PVE/GuestHelpers.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/GuestHelpers.pm b/src/PVE/GuestHelpers.pm
index 0fe3fd6..79c02e2 100644
--- a/src/PVE/GuestHelpers.pm
+++ b/src/PVE/GuestHelpers.pm
@@ -101,7 +101,7 @@ sub check_hookscript {
 }
 
 sub exec_hookscript {
-    my ($conf, $vmid, $phase, $stop_on_error) = @_;
+    my ($conf, $vmid, $phase, $stop_on_error, $run_params) = @_;
 
     return if !$conf->{hookscript};
 
@@ -109,7 +109,7 @@ sub exec_hookscript {
 	my $hookscript = check_hookscript($conf->{hookscript});
 	die $@ if $@;
 
-	PVE::Tools::run_command([$hookscript, $vmid, $phase]);
+	PVE::Tools::run_command([$hookscript, $vmid, $phase], %$run_params);
     };
     if (my $err = $@) {
 	my $errmsg = "hookscript error for $vmid on $phase: $err\n";
-- 
2.30.2

[pve-devel] [PATCH v2 pve-container 1/2] add pct mtunnel command to the CLI

[Stefan Hanreich]

Analogous to the qm mtunnel commands, pct mtunnel creates a tunnel
between the two migration nodes, that can be used to execute various
commands during the migration process.

This tunnel has been implemented using v2 of our tunnel protocol. It
supports the migrate-hook as well as the query-migrate-hook command.

The migrate-hook command executes the respective hook script by forking
from the tunnel process. This enables us to evade timeouts resulting
from long running hook scripts, as well as hookscripts doing weird stuff
with STDOUT.

query-migrate-hook can be used to get information about the currently
running migration-hook. It returns the output of the command in the case
of a successful run / error. If the migrate-hook is still running then
it returns information about the running migration-hook.

In future patches it might be wise to move some basic funtionality used
in the tunnel to its own class, since this functionality is used in more
places already and could be shared. This seemed out of scope for this
patch though.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 src/PVE/CLI/pct.pm | 230 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 230 insertions(+)

diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm
index 23793ee..4adb155 100755
--- a/src/PVE/CLI/pct.pm
+++ b/src/PVE/CLI/pct.pm
@@ -6,6 +6,7 @@ use warnings;
 use Fcntl;
 use File::Copy 'copy';
 use POSIX;
+use JSON;
 
 use PVE::CLIHandler;
 use PVE::Cluster;
@@ -803,6 +804,233 @@ __PACKAGE__->register_method ({
 	return undef;
     }});
 
+__PACKAGE__->register_method ({
+    name => 'mtunnel',
+    path => 'mtunnel',
+    method => 'POST',
+    description => "Used by qmigrate - do not use manually.",
+    parameters => {
+	additionalProperties => 0,
+	properties => {},
+    },
+    returns => { type => 'null'},
+    code => sub {
+	my ($param) = @_;
+
+	if (!PVE::Cluster::check_cfs_quorum(1)) {
+	    print "no quorum\n";
+	    return;
+	}
+
+	my $tunnel_write = sub {
+	    my $text = shift;
+	    chomp $text;
+	    print "$text\n";
+	    *STDOUT->flush();
+	};
+
+	$tunnel_write->("tunnel online");
+	$tunnel_write->("ver 2");
+
+	my $state = {
+	    quit => 0,
+	};
+
+	my $cmd_desc = {
+	    quit                 => {},
+	    'query-migrate-hook' => {},
+	    'migrate-hook'       => {
+		properties => {
+		    vmid   => get_standard_option('pve-vmid'),
+		    source => get_standard_option('pve-node'),
+		    target => get_standard_option('pve-node'),
+		    phase  => {
+			type        => 'string',
+			description => 'The phase of the hook (either pre or post)',
+		    },
+		}
+	    },
+	};
+
+	my $cmd_handlers = {
+	    'quit' => sub {
+		$state->{quit} = 1;
+		return;
+	    },
+	    'query-migrate-hook' => sub {
+		if (!$state->{migrate_hook}) {
+		    die "No migration hook running!"
+		}
+
+		if (!waitpid($state->{migrate_hook}->{pid}, POSIX::WNOHANG)) {
+		    return {
+			status => 'running',
+			pid  => $state->{migrate_hook}->{pid},
+		    }
+		}
+
+		my $reader = $state->{migrate_hook}->{output};
+		my $output = "";
+
+		while (my $line = <$reader>) {
+		    $output .= $line;
+		}
+
+		close $state->{migrate_hook}->{output};
+		delete $state->{migrate_hook};
+
+		my $status = ($? == 0)
+		    ? 'finished'
+		    : 'error';
+
+		return {
+		    status => $status,
+		    output => $output,
+		};
+	    },
+	    'migrate-hook' => sub {
+		if ($state->{migrate_hook}) {
+		    die "Migrate Hook is already running!";
+		}
+
+		my $params = shift;
+
+		my $vmid = $params->{vmid};
+		my $phase = $params->{phase};
+		my $source = $params->{source};
+		my $target = $params->{target};
+
+		my $config_node = ($phase eq 'pre')
+		    ? $source
+		    : $target;
+
+		eval {
+		    my $conf = PVE::LXC::Config->load_config($vmid, $config_node);
+
+		    pipe(my $reader, my $writer);
+
+		    my $pid = fork();
+		    die "Could not fork new process!" if !defined $pid;
+
+		    if ($pid == 0) {
+			# child
+			close $reader;
+
+			$ENV{PVE_MIGRATED_FROM} = $source;
+
+			eval {
+			    PVE::GuestHelpers::exec_hookscript(
+				$conf,
+				$vmid,
+				"$phase-migrate",
+				1,
+				{
+				    output => ">&" . fileno($writer),
+				    errfunc => sub {
+					my $line = shift;
+					print $writer "STDERR: " . $line;
+				    },
+				}
+			    );
+			};
+			my $err = $@;
+
+			close $writer;
+
+			if ($err) {
+			    POSIX::_exit(1);
+			}
+
+			POSIX::_exit(0);
+		    }
+
+		    close $writer;
+
+		    $state->{migrate_hook} = {
+			output => $reader,
+			pid    => $pid,
+		    };
+		};
+		if ($@) {
+		    chomp $@;
+		    die "ERR: $phase-migrate hook failed - $@";
+		} else {
+		    return {}
+		}
+	    },
+	};
+
+	my $reply_err = sub {
+	    my ($msg) = @_;
+
+	    my $reply = JSON::encode_json({
+		success => JSON::false,
+		msg     => $msg,
+	    });
+
+	    $tunnel_write->($reply);
+	};
+
+	my $reply_ok = sub {
+	    my ($res) = @_;
+
+	    $res->{success} = JSON::true;
+
+	    my $reply = JSON::encode_json($res);
+
+	    $tunnel_write->($reply);
+	};
+
+	while (my $line = <STDIN>) {
+	    if ($state->{quit}) {
+		if ($state->{migrate_hook}->{output}) {
+		    close $state->{migrate_hook}->{output};
+		}
+
+		last;
+	    }
+
+	    chomp $line;
+
+	    # untaint, we validate below if needed
+	    ($line) = $line =~ /^(.*)$/;
+	    my $parsed = eval {JSON::decode_json($line)};
+	    if ($@) {
+		$reply_err->("failed to parse command - $@");
+		next;
+	    }
+
+	    my $cmd = delete $parsed->{cmd};
+
+	    if (!defined($cmd)) {
+		$reply_err->("'cmd' missing");
+	    } elsif (my $handler = $cmd_handlers->{$cmd}) {
+		if (!$cmd_desc->{$cmd}) {
+		    $reply_err->("unknown command '$cmd' given");
+		    next;
+		}
+
+		eval {
+		    PVE::JSONSchema::validate($parsed, $cmd_desc->{$cmd});
+		};
+		if ($@) {
+		    $reply_err->("invalid payload format for $cmd' command - $@");
+		    next;
+		}
+
+		eval {
+		    my $res = $handler->($parsed);
+		    $reply_ok->($res);
+		};
+		$reply_err->("failed to handle '$cmd' command - $@") if $@;
+	    } else {
+		$reply_err->("unknown command '$cmd' given");
+	    }
+	}
+
+	return;
+    }});
+
 our $cmddef = {
     list=> [ 'PVE::API2::LXC', 'vmlist', [], { node => $nodename }, sub {
 	my $res = shift;
@@ -874,6 +1102,8 @@ our $cmddef = {
     rescan  => [ __PACKAGE__, 'rescan', []],
     cpusets => [ __PACKAGE__, 'cpusets', []],
     fstrim => [ __PACKAGE__, 'fstrim', ['vmid']],
+
+    mtunnel => [ __PACKAGE__, 'mtunnel', []],
 };
 
 1;
-- 
2.30.2

[pve-devel] [PATCH v2 pve-container 2/2] add migration hooks to container migration process

[Stefan Hanreich]

Using the pct commands implemented in the previous commit, this commit
adds running the migration-hooks during the container migration process.

I am redirecting STDERR from the pct mtunnel to /dev/null since it is
not captured by the current fork_ssh_tunnel function and can then
pollute the output of the migration task. The STDERR of the
migrate-hookscript is not affected, since it is captured by the remote
mtunnel command and gets transferred via the query-migrate-hook command.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 src/PVE/LXC/Migrate.pm | 119 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)

diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm
index 2ef1cce..fdde180 100644
--- a/src/PVE/LXC/Migrate.pm
+++ b/src/PVE/LXC/Migrate.pm
@@ -89,6 +89,8 @@ sub prepare {
 	    if !$target_scfg->{content}->{rootdir};
     });
 
+    $self->migration_hook($vmid, 'pre');
+
     # todo: test if VM uses local resources
 
     # test ssh connection
@@ -384,6 +386,15 @@ sub phase3 {
     }
 }
 
+#  only called when phase1 was successful
+sub phase3_cleanup {
+    my ($self, $vmid, $err) = @_;
+
+    if (!$self->{errors}) {
+	$self->migration_hook($vmid, 'post');
+    }
+}
+
 sub final_cleanup {
     my ($self, $vmid) = @_;
 
@@ -413,7 +424,115 @@ sub final_cleanup {
 	    $self->cmd($cmd);
 	}
     }
+}
+
+sub fork_tunnel {
+    my ($self, $ssh_forward_info) = @_;
+
+    my $cmd = ['/usr/sbin/pct', 'mtunnel', '2>', '/dev/null'];
+    my $log = sub {
+	my ($level, $msg) = @_;
+	$self->log($level, $msg);
+    };
+
+    return PVE::Tunnel::fork_ssh_tunnel($self->{rem_ssh}, $cmd, $ssh_forward_info, $log);
+}
+
+sub migration_hook {
+    my ($self, $vmid, $phase) = @_;
+
+    if (!$self->{vmconf}->{hookscript}) {
+	return;
+    }
+
+    my $stop_on_error = $phase eq 'pre';
+
+    PVE::GuestHelpers::exec_hookscript(
+	$self->{vmconf},
+	$vmid,
+	"$phase-migrate",
+	$stop_on_error,
+    );
+
+    my $tunnel;
+
+    eval {
+	$tunnel = $self->{tunnel} // $self->fork_tunnel();
+    };
+    if ($@ =~ /can't open tunnel/) {
+	$self->log('warn', 'Target node does not support mtunnel. Not running hookscript on target, but still continuing with migration.');
+	return;
+    } elsif ($@) {
+	die $@;
+    }
+
+    my $close_tunnel = sub {
+	if (!$self->{tunnel}) {
+	    eval {
+		$self->log('info', "closing tunnel for migration hook");
+		PVE::Tunnel::finish_tunnel($tunnel);
+	    };
+	    if ($@) {
+		$self->log('warn', 'could not close tunnel to remote host');
+	    }
+	}
+    };
+
+    my $result;
+
+    eval {
+	$self->log('info', "starting hook $phase-migrate on target");
+
+	$result = PVE::Tunnel::write_tunnel($tunnel, 30, "migrate-hook", {
+	    vmid => $vmid,
+	    phase => $phase,
+	    source => PVE::INotify::nodename(),
+	    target => $self->{node},
+	});
+    };
+    my $err = $@;
+
+    if ($err) {
+	$close_tunnel->();
+	die $err;
+    }
+
+    $self->log('info', "successfully started hook $phase-migrate on target");
+
+    my $running = 1;
+
+    while ($running) {
+	eval {
+	    $result = PVE::Tunnel::write_tunnel($tunnel, 30, "query-migrate-hook");
+
+	    if (!exists $result->{status}) {
+		die "Invalid response!";
+	    } elsif ($result->{status} eq 'running') {
+		sleep(5);
+	    } elsif ($result->{status} eq 'finished') {
+		$self->log('info', "$phase-migrate hook ran successfully on target:\n$result->{output}");
+	    } elsif ($result->{status} eq 'error') {
+		my $msg = "An error occured during running the hookscript:\n" . $result->{output};
+
+		if ($stop_on_error) {
+		    die $msg;
+		} else {
+		    $self->log('warn', $msg)
+		}
+	    } else {
+		die "Invalid response!";
+	    }
+
+	    $running = $result->{status} eq 'running';
+	};
+	if ($@) {
+	    $err = $@;
+	    last;
+	}
+    }
 
+    $close_tunnel->();
+    die $err if $err;
 }
 
 1;
-- 
2.30.2

[pve-devel] [PATCH v2 pve-docs 1/3] Add pre/post-migrate events to hookscript example

[Stefan Hanreich]

Also added an example on how to use the passed environment variable to
discern between target/source

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 examples/guest-example-hookscript.pl | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/examples/guest-example-hookscript.pl b/examples/guest-example-hookscript.pl
index adeed59..96b0ff2 100755
--- a/examples/guest-example-hookscript.pl
+++ b/examples/guest-example-hookscript.pl
@@ -54,6 +54,26 @@ if ($phase eq 'pre-start') {
 
     print "$vmid stopped. Doing cleanup.\n";
 
+} elsif ($phase eq 'pre-migrate') {
+    # The environment variable PVE_MIGRATED_FROM will contain the name of the
+    # source node, if this script is run on the target node.
+    my $node_type = ($ENV{PVE_MIGRATED_FROM}) ? 'target' : 'source';
+
+    # Phase 'pre-migrate' will be run before a migration on the source/target
+    # node of the VM/CT
+
+    print "preparing $vmid for migration on $node_type.\n";
+
+} elsif ($phase eq 'post-migrate') {
+    # The environment variable PVE_MIGRATED_FROM will contain the name of the
+    # source node, if this script is run on the target node.
+    my $node_type = ($ENV{PVE_MIGRATED_FROM}) ? 'target' : 'source';
+
+    # Phase 'post-migrate' will be run after a migration on the source/target
+    # node of the VM/CT
+
+    print "finished migrating $vmid on $node_type.\n";
+
 } else {
     die "got unknown phase '$phase'\n";
 }
-- 
2.30.2

[pve-devel] [PATCH v2 pve-docs 2/3] Add hookscript section to container documentation

[Stefan Hanreich]

Since it has been missing so far, I extended the documentation to
contain information about container hookscripts. I added some additional
notes for how to use the pre/post-migrate scripts properly.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 pct.adoc | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/pct.adoc b/pct.adoc
index ebae3e9..57f2206 100644
--- a/pct.adoc
+++ b/pct.adoc
@@ -1260,3 +1260,26 @@ Configuration file for the container '<CTID>'.
 
 include::pve-copyright.adoc[]
 endif::manvolnum[]
+
+Hookscripts
+-----------
+
+You can add a hook script to CTs with the config property `hookscript`.
+
+----
+# pct set 100 --hookscript local:snippets/hookscript.pl
+----
+
+It will be called during various phases of the guests lifetime.
+For an example and documentation see the example script under
+`/usr/share/pve-docs/examples/guest-example-hookscript.pl`.
+
+pre/post-migrate
+~~~~~~~~~~~~~~~~
+Those hooks run before/after a CT migrates from one host to another. This
+hook is a bit special, since it gets executed on the source node as well as the
+target node. In order for this hookscript to function properly, the configured
+script file needs to be available on both nodes. If it is a local script, this
+means storing it on both server under the same path. The preferred option
+would be storing the hookscript on a shared storage, so it is accessible from
+both nodes.
-- 
2.30.2

[pve-devel] [PATCH v2 pve-docs 3/3] Add pre/post-migrate section to VM hookscript documentation

[Stefan Hanreich]

Add a short information about how the hookscripts should be stored as
well as a short part about the preferred way of storing the migration
hookscripts.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 qm.adoc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/qm.adoc b/qm.adoc
index 4d0c7c4..3f5c2cc 100644
--- a/qm.adoc
+++ b/qm.adoc
@@ -1410,6 +1410,16 @@ It will be called during various phases of the guests lifetime.
 For an example and documentation see the example script under
 `/usr/share/pve-docs/examples/guest-example-hookscript.pl`.
 
+pre/post-migrate
+~~~~~~~~~~~~~~~~
+Those hooks run before/after a VM migrates from one host to another. This
+hook is a bit special, since it gets executed on the source node as well as the
+target node. In order for this hookscript to function properly, the configured
+script file needs to be available on both nodes. If it is a local script, this
+means storing it on both server under the same path. The preferred option
+would be storing the hookscript on a shared storage, so it is accessible from
+both nodes.
+
 [[qm_hibernate]]
 Hibernation
 -----------
-- 
2.30.2

[pve-devel] [PATCH v2 qemu-server 1/2] add migrate-hook and query-migrate-hook commands to CLI

[Stefan Hanreich]

migrate-hook runs the migrate hook by forking a new process and
attaching to its STDOUT. This helps us deal with long running
hookscripts, that could otherwise block. Additionally it safeguards us
from hookscripts doing weird stuff with output.

query-migrate-hook can be used to query the current state of the running
hookscript. After the migrate-hook has finished it also transfers the
output back to the source node. When an error occurs, the
query-migrate-hook command informs the caller and returns the respective
output as well.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 PVE/CLI/qm.pm | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)

diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm
index ca5d25f..5ac091b 100755
--- a/PVE/CLI/qm.pm
+++ b/PVE/CLI/qm.pm
@@ -306,6 +306,8 @@ __PACKAGE__->register_method ({
 	$tunnel_write->("tunnel online");
 	$tunnel_write->("ver 1");
 
+	my $state = {};
+
 	while (my $line = <STDIN>) {
 	    chomp $line;
 	    if ($line =~ /^quit$/) {
@@ -323,6 +325,113 @@ __PACKAGE__->register_method ({
 		} else {
 		    $tunnel_write->("ERR: resume failed - VM $vmid not running");
 		}
+	    } elsif ($line =~ /^migrate-hook (\d+) (pre|post) ([\S]+) ([\S]+)$/) {
+		if ($state->{migrate_hook}) {
+		    $tunnel_write->("ERR: migrate-hook is already running!");
+		    next;
+		}
+
+		my $vmid = $1;
+		my $phase = $2;
+		my $source = $3;
+		my $target = $4;
+
+		if (!PVE::JSONSchema::pve_verify_node_name($source, 1)) {
+		    $tunnel_write->("ERR: Invalid name for source node");
+		    next;
+		}
+
+		if (!PVE::JSONSchema::pve_verify_node_name($target, 1)) {
+		    $tunnel_write->("ERR: Invalid name for target node");
+		    next;
+		}
+
+		my $config_node = ($phase eq 'pre')
+		    ? $source
+		    : $target;
+
+		eval {
+		    my $conf = PVE::QemuConfig->load_config($vmid, $config_node);
+
+		    pipe(my $reader, my $writer);
+
+		    my $pid = fork();
+		    die "Could not fork new process!" if !defined $pid;
+
+		    if ($pid == 0) {
+			# child
+			close $reader;
+
+			$ENV{PVE_MIGRATED_FROM} = $source;
+
+			eval {
+			    PVE::GuestHelpers::exec_hookscript(
+				$conf,
+				$vmid,
+				"$phase-migrate",
+				1,
+				{
+				    output  => ">&" . fileno($writer),
+				    errfunc => sub {
+					my $error_line = shift;
+					print $writer "STDERR: " . $error_line;
+				    },
+				}
+			    );
+			};
+			my $err = $@;
+
+			close $writer;
+
+			if ($err) {
+			    POSIX::_exit(1);
+			}
+
+			POSIX::_exit(0);
+		    }
+
+		    close $writer;
+
+		    $state->{migrate_hook} = {
+			output => $reader,
+			pid    => $pid,
+		    };
+		};
+		if ($@) {
+		    chomp $@;
+		    $tunnel_write->("ERR: $phase-migrate hook failed - $@");
+		} else {
+		    $tunnel_write->("OK");
+		}
+	    } elsif ($line =~ /^query-migrate-hook$/) {
+		if (!$state->{migrate_hook}) {
+		    $tunnel_write->("ERR: No migration hook running!");
+		    next;
+		}
+
+		if (!waitpid($state->{migrate_hook}->{pid}, POSIX::WNOHANG)) {
+		    $tunnel_write->("OK");
+		    $tunnel_write->("running");
+		    next;
+		}
+
+		my $reader = $state->{migrate_hook}->{output};
+		my $output = "";
+
+		while (my $line = <$reader>) {
+		    $output .= $line;
+		}
+
+		close $state->{migrate_hook}->{output};
+		delete $state->{migrate_hook};
+
+		my $status = ($? == 0)
+		    ? 'finished'
+		    : 'error';
+
+		$tunnel_write->("OK");
+		$tunnel_write->("$status");
+		$tunnel_write->(MIME::Base64::encode($output, ''));
 	    }
 	}
 
-- 
2.30.2

[pve-devel] [PATCH v2 qemu-server 2/2] add migration hooks to VM migration process

[Stefan Hanreich]

This calls the newly added commands in the previous commit to run the
migrate-hooks during the migration process. When a tunnel already
exists, the tunnel gets reused otherwise it creates a new ad-hoc tunnel
that is used for running the migration-hook.

Additionally I added some mock methods to the QemuMigrateMock class, so
the test class supports the newly added commands as well.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 PVE/QemuMigrate.pm                    | 108 ++++++++++++++++++++++++++
 test/MigrationTest/QemuMigrateMock.pm |  11 ++-
 2 files changed, 118 insertions(+), 1 deletion(-)

diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm
index d52dc8d..42cf7d0 100644
--- a/PVE/QemuMigrate.pm
+++ b/PVE/QemuMigrate.pm
@@ -203,6 +203,8 @@ sub prepare {
     eval { $self->cmd_quiet($cmd); };
     die "Can't connect to destination address using public key\n" if $@;
 
+    $self->migration_hook($vmid, 'pre');
+
     return $running;
 }
 
@@ -1216,6 +1218,10 @@ sub phase3_cleanup {
 	}
     }
 
+    if (!$self->{errors}) {
+	$self->migration_hook($vmid, 'post');
+    }
+
     # close tunnel on successful migration, on error phase2_cleanup closed it
     if ($tunnel) {
 	eval { PVE::Tunnel::finish_tunnel($tunnel); };
@@ -1284,4 +1290,106 @@ sub round_powerof2 {
     return 2 << int(log($_[0]-1)/log(2));
 }
 
+sub migration_hook {
+    my ($self, $vmid, $phase) = @_;
+
+    if (!$self->{vmconf}->{hookscript}) {
+	return;
+    }
+
+    my $stop_on_error = $phase eq 'pre';
+
+    PVE::GuestHelpers::exec_hookscript(
+	$self->{vmconf},
+	$vmid,
+	"$phase-migrate",
+	$stop_on_error,
+    );
+
+    my $tunnel;
+
+    eval {
+	$tunnel = $self->{tunnel} // $self->fork_tunnel();
+    };
+    die $@ if $@;
+
+    my $close_tunnel = sub {
+	if (!$self->{tunnel}) {
+	    $self->log('info', "closing tunnel for migration hook");
+	    PVE::Tunnel::finish_tunnel($tunnel);
+	}
+    };
+
+    my $source = PVE::INotify::nodename();
+    my $target = $self->{node};
+
+    eval {
+	$self->log('info', "running hook $phase-migrate on target");
+	PVE::Tunnel::write_tunnel($tunnel, 30, "migrate-hook $vmid $phase $source $target");
+    };
+    my $err = $@;
+
+    if ($err =~ /no reply to command/) {
+	eval {
+	    $close_tunnel->();
+	};
+	if ($@) {
+	    die $err;
+	} else {
+	    $self->log('warn', 'Got timeout when trying to run migrate-hook. Target doesn\'t support migrate hooks (old version?). Still continuing with migration.');
+	    return;
+	}
+    } elsif ($err) {
+	$close_tunnel->();
+	die $err;
+    }
+
+    $self->log('info', "successfully started hook $phase-migrate on target");
+
+    my $running = 1;
+
+    while ($running) {
+	eval {
+	    PVE::Tunnel::write_tunnel($tunnel, 30, "query-migrate-hook");
+	    my $status = PVE::Tunnel::read_tunnel($tunnel, 30);
+
+	    if ($status eq 'running') {
+		sleep(5);
+	    } elsif ($status eq 'finished') {
+		my $output = MIME::Base64::decode(
+		    PVE::Tunnel::read_tunnel($tunnel, 30)
+		);
+
+		$self->log('info', "$phase-migrate hook ran successfully on target:\n" . $output);
+	    } elsif ($status eq 'error') {
+		my $output = MIME::Base64::decode(
+		    PVE::Tunnel::read_tunnel($tunnel, 30)
+		);
+
+		my $msg = "An error occured during running the hookscript:\n" . $output;
+
+		if ($stop_on_error) {
+		    die $msg;
+		} else {
+		    $self->log('warn', $msg)
+		}
+	    } else {
+		die "Invalid response!"
+	    }
+
+	    $running = $status eq 'running';
+	};
+	if ($@) {
+	    $err = $@;
+	    last;
+	}
+    }
+
+    eval {
+	$close_tunnel->();
+    };
+    die $err if $err; # use the initial error if it exists
+    die $@ if $@;
+}
+
 1;
diff --git a/test/MigrationTest/QemuMigrateMock.pm b/test/MigrationTest/QemuMigrateMock.pm
index f2c0281..e33a284 100644
--- a/test/MigrationTest/QemuMigrateMock.pm
+++ b/test/MigrationTest/QemuMigrateMock.pm
@@ -64,6 +64,8 @@ $tunnel_module->mock(
 	    my $vmid = $1;
 	    die "resuming wrong VM '$vmid'\n" if $vmid ne $test_vmid;
 	    return;
+	} elsif ($command =~ /^migrate-hook.*/) {
+	    return;
 	}
 	die "write_tunnel (mocked) - implement me: $command\n";
     },
@@ -72,7 +74,12 @@ $tunnel_module->mock(
 my $qemu_migrate_module = Test::MockModule->new("PVE::QemuMigrate");
 $qemu_migrate_module->mock(
     fork_tunnel => sub {
-	die "fork_tunnel (mocked) - implement me\n"; # currently no call should lead here
+	return {
+	    writer => "mocked",
+	    reader => "mocked",
+	    pid => 123456,
+	    version => 1,
+	};
     },
     read_tunnel => sub {
 	die "read_tunnel (mocked) - implement me\n"; # currently no call should lead here
@@ -298,6 +305,8 @@ $MigrationTest::Shared::tools_module->mock(
 			return 0;
 		    } elsif ($cmd eq 'stop') {
 			return 0;
+		    } elsif ($cmd eq 'mtunnel') {
+			return 0;
 		    }
 		    die "run_command (mocked) ssh qm command - implement me: ${cmd_msg}";
 		} elsif ($cmd eq 'pvesm') {
-- 
2.30.2