From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH container v3 26/38] test: remote migration: check the cleanup of the target after failures
Date: Wed, 18 Nov 2026 05:37:38 +0100 [thread overview]
Message-ID: <20261118043817.1659212-27-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com>
A config left behind on the target of a failed remote migration blocks
its VMID there, and with guest identities it would also keep the uid of
the container in the target cluster.
Check with a mocked tunnel that the source asks the target to clean up
if the tunnel versions do not match or the container does not stop for a
restart migration. Check on the target that it removes its reservation
if the tunnel closes before the config arrived, keeps a config that
arrived, and logs the error of a failed tunnel command.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
changes v2 -> v3:
* follow the fixed wait for a volume transfer in the cleanup of the
target, which is bounded by time: check a transfer that exits late and
one that cannot be collected
changes v1 -> v2:
* was 'migrate: remote: clean up the target on a tunnel version
mismatch', its fix got applied separately together with the cleanup
after a failed shutdown
* more tests: failed shutdown of a restart migration, tunnel that closes
before the config arrived (with and without a running volume
transfer), logging of a failed tunnel command
src/test/lifecycle_tests.pl | 175 ++++++++++++++++++++++++++++++++++++
1 file changed, 175 insertions(+)
diff --git a/src/test/lifecycle_tests.pl b/src/test/lifecycle_tests.pl
index 7606d3e..ff42c5f 100755
--- a/src/test/lifecycle_tests.pl
+++ b/src/test/lifecycle_tests.pl
@@ -1890,6 +1890,181 @@ subtest 'restore over a container of another guest needs an override' => sub {
);
};
+subtest 'remote restart migration cleans up the target if the container does not stop' => sub {
+ reset_configs();
+ my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+ write_raw_config(200, $raw);
+
+ my $log = '';
+ my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+ $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+ my $cleanup;
+ my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+ $tunnel_module->mock(
+ fork_websocket_tunnel => sub {
+ return { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION, age => 0 };
+ },
+ write_tunnel => sub { return {} },
+ finish_tunnel => sub { (undef, $cleanup) = @_ },
+ );
+ $lxc_module->mock(
+ check_running => sub { return 1 },
+ vm_stop => sub { die "container did not stop\n" },
+ );
+
+ my $err = run_task(sub {
+ PVE::LXC::Migrate->migrate(
+ 'remote-node',
+ '192.0.2.1',
+ 200,
+ {
+ remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } },
+ restart => 1,
+ },
+ );
+ });
+ $lxc_module->mock(check_running => sub { return 0 });
+ $lxc_module->unmock('vm_stop');
+
+ like($err, qr/migration aborted/, 'migration fails');
+ like($log, qr/^err: .*container did not stop/m, 'failed shutdown is reported');
+ is($cleanup, 1, 'target is asked to clean up');
+ is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
+subtest 'migration tunnel logs failed commands and cleans up after an early end' => sub {
+ reset_configs();
+ my ($err, $replies) = run_mtunnel(300);
+ like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends without a command fails');
+ like(
+ $task_log,
+ qr/^WARN: tunnel closed before the config arrived, cleaning up$/m,
+ 'cleanup is logged',
+ );
+ ok(!defined(read_raw_config(300)), 'reserved config is removed');
+
+ reset_configs();
+ my $broken = $INCOMING_CONFIG =~ s/^rootfs: local:/rootfs: iso:/mr;
+ ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $broken });
+ like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after a failed config fails');
+ ok(!$replies->[0]->{success}, 'config command fails');
+ like(
+ $task_log,
+ qr/^WARN: failed to handle 'config' command - /m,
+ 'failed command is logged',
+ );
+ ok(!defined(read_raw_config(300)), 'reserved config is removed after a failed config');
+
+ reset_configs();
+ ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $INCOMING_CONFIG });
+ like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after the config fails');
+ unlike($task_log, qr/cleaning up/, 'no cleanup once the config arrived');
+ is(PVE::LXC::Config->load_config(300)->{lock}, 'migrate', 'config that arrived is kept');
+
+ # the volume of a transfer is only known once the transfer is done
+ my $import_cmd = {
+ cmd => 'disk-import',
+ volname => 'vm-300-disk-0',
+ format => 'raw',
+ export_formats => 'raw+size',
+ storage => 'local',
+ };
+ my ($queries, $ends_after, $query_error, $freed) = (0, 2, undef, []);
+ my $storage_tunnel_module = Test::MockModule->new('PVE::StorageTunnel');
+ $storage_tunnel_module->mock(
+ handle_disk_import => sub {
+ my ($state) = @_;
+ $state->{disk_import} = { pid => 1 };
+ return {};
+ },
+ handle_query_disk_import => sub {
+ my ($state) = @_;
+ die $query_error if $query_error;
+ return { status => 'pending' } if ++$queries < $ends_after;
+ delete $state->{disk_import};
+ $state->{cleanup}->{volumes}->{'local:300/vm-300-disk-0.raw'} = 1;
+ return { status => 'complete' };
+ },
+ );
+ $storage_module->mock(vdisk_free => sub { push $freed->@*, $_[1] });
+
+ reset_configs();
+ ($err, $replies) = run_mtunnel(300, $import_cmd);
+ ok($replies->[0]->{success}, 'volume transfer starts') or diag(explain($replies));
+ is_deeply(
+ $freed,
+ ['local:300/vm-300-disk-0.raw'],
+ 'volume of a transfer that ended is freed',
+ );
+ ok(!defined(read_raw_config(300)), 'reserved config is removed after the transfer ended');
+
+ # a transfer that closed its output can be queried many times before it has exited
+ reset_configs();
+ ($queries, $ends_after, $freed) = (0, 20, []);
+ ($err, $replies) = run_mtunnel(300, $import_cmd);
+ is_deeply($freed, ['local:300/vm-300-disk-0.raw'], 'volume of a transfer that exits late');
+ ok(!defined(read_raw_config(300)), 'reserved config is removed after a late exit');
+
+ reset_configs();
+ ($queries, $query_error, $freed) = (0, "unable to query the transfer\n", []);
+ ($err, $replies) = run_mtunnel(300, $import_cmd);
+ like(
+ $task_log,
+ qr/^WARN: cleanup failed - a volume transfer is still running$/m,
+ 'transfer that cannot be collected is reported',
+ );
+ is_deeply($freed, [], 'nothing is freed while a transfer is not collected');
+ ok(
+ defined(read_raw_config(300)),
+ 'reserved config is kept while a transfer is not collected',
+ );
+ $storage_module->unmock('vdisk_free');
+};
+
+subtest 'remote migration cleans up a target with an incompatible tunnel version' => sub {
+ reset_configs();
+ my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+ write_raw_config(200, $raw);
+
+ my $log = '';
+ my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+ $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+ my $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION - 1, age => 0 };
+ my ($commands, $cleanup) = (0);
+ my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+ $tunnel_module->mock(
+ fork_websocket_tunnel => sub { return $remote_version },
+ write_tunnel => sub { $commands++ },
+ finish_tunnel => sub { (undef, $cleanup) = @_ },
+ );
+
+ my $migrate = sub {
+ ($log, $commands, $cleanup) = ('', 0);
+ return run_task(sub {
+ PVE::LXC::Migrate->migrate(
+ 'remote-node',
+ '192.0.2.1',
+ 200,
+ { remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } } },
+ );
+ });
+ };
+
+ like($migrate->(), qr/migration aborted/, 'migration to an older target fails');
+ like($log, qr/^err: .*Remote tunnel endpoint too old/m, 'target is reported as too old');
+ is($cleanup, 1, 'older target is asked to clean up');
+ is($commands, 0, 'nothing else is sent to the target');
+ is(read_raw_config(200), $raw, 'config of the source is unchanged');
+
+ $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION + 1, age => 0 };
+ like($migrate->(), qr/migration aborted/, 'migration to an incompatible target fails');
+ like($log, qr/^err: .*Remote tunnel endpoint not compatible/m, 'target is reported');
+ is($cleanup, 1, 'incompatible target is asked to clean up');
+ is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
remove_tree($WORKDIR);
done_testing();
--
2.47.3
next prev parent reply other threads:[~2026-11-18 4:42 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 4:37 [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 01/38] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 02/38] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 03/38] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 04/38] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 05/38] pmxcfs: confdb: call an optional listener when the nodes get published Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 06/38] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 07/38] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-18 4:37 ` [PATCH cluster v3 08/38] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-18 4:37 ` [PATCH guest-common v3 09/38] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-18 4:37 ` [PATCH guest-common v3 10/38] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-18 4:37 ` [PATCH guest-common v3 11/38] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-18 4:37 ` [PATCH guest-common v3 12/38] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-18 4:37 ` [PATCH guest-common v3 13/38] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 15/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 16/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 17/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 18/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 19/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18 4:37 ` [PATCH qemu-server v3 20/38] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 21/38] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 22/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 23/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 24/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 25/38] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-18 4:37 ` Thomas Lamprecht [this message]
2026-11-18 4:37 ` [PATCH container v3 27/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18 4:37 ` [PATCH container v3 28/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 29/38] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 30/38] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 31/38] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 32/38] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 33/38] ui: guest summary: show " Thomas Lamprecht
2026-11-18 4:37 ` [PATCH manager v3 34/38] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-18 4:37 ` [PATCH docs v3 35/38] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-18 4:37 ` [PATCH docs v3 36/38] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-18 4:37 ` [PATCH docs v3 37/38] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-18 4:37 ` [PATCH docs v3 38/38] pct: " Thomas Lamprecht
2026-11-18 13:49 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Dominik Csapak
2026-11-18 14:13 ` Thomas Lamprecht
2026-11-19 10:38 ` superseded: " Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118043817.1659212-27-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox