public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH container v2 24/36] test: remote migration: check the cleanup of the target after failures
Date: Tue, 17 Nov 2026 23:34:34 +0100	[thread overview]
Message-ID: <20261117223945.862914-25-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com>

A config left behind on the target of a failed remote migration blocks
its VMID there, and with guest identities it would also keep the uid of
the container in the target cluster.

Check with a mocked tunnel that the source asks the target to clean up
if the tunnel versions do not match or the container does not stop for a
restart migration. Check on the target that it removes its reservation
if the tunnel closes before the config arrived, keeps a config that
arrived, and logs the error of a failed tunnel command.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
* was 'migrate: remote: clean up the target on a tunnel version
  mismatch', its fix got applied separately together with the cleanup
  after a failed shutdown
* more tests: failed shutdown of a restart migration, tunnel that closes
  before the config arrived (with and without a running volume
  transfer), logging of a failed tunnel command

 src/test/lifecycle_tests.pl | 164 ++++++++++++++++++++++++++++++++++++
 1 file changed, 164 insertions(+)

diff --git a/src/test/lifecycle_tests.pl b/src/test/lifecycle_tests.pl
index 235169f..930fa4f 100755
--- a/src/test/lifecycle_tests.pl
+++ b/src/test/lifecycle_tests.pl
@@ -1558,6 +1558,170 @@ subtest 'restore over a container of another guest needs an override' => sub {
     $lxc_module->mock(delete_ifaces_ipams_ips => sub { });
 };
 
+subtest 'remote restart migration cleans up the target if the container does not stop' => sub {
+    reset_configs();
+    my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+    write_raw_config(200, $raw);
+
+    my $log = '';
+    my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+    $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+    my $cleanup;
+    my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+    $tunnel_module->mock(
+        fork_websocket_tunnel => sub {
+            return { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION, age => 0 };
+        },
+        write_tunnel => sub { return {} },
+        finish_tunnel => sub { (undef, $cleanup) = @_ },
+    );
+    $lxc_module->mock(
+        check_running => sub { return 1 },
+        vm_stop => sub { die "container did not stop\n" },
+    );
+
+    my $err = run_task(sub {
+        PVE::LXC::Migrate->migrate(
+            'remote-node',
+            '192.0.2.1',
+            200,
+            {
+                remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } },
+                restart => 1,
+            },
+        );
+    });
+    $lxc_module->mock(check_running => sub { return 0 });
+    $lxc_module->unmock('vm_stop');
+
+    like($err, qr/migration aborted/, 'migration fails');
+    like($log, qr/^err: .*container did not stop/m, 'failed shutdown is reported');
+    is($cleanup, 1, 'target is asked to clean up');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
+subtest 'migration tunnel logs failed commands and cleans up after an early end' => sub {
+    reset_configs();
+    my ($err, $replies) = run_mtunnel(300);
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends without a command fails');
+    like(
+        $task_log,
+        qr/^WARN: tunnel closed before the config arrived, cleaning up$/m,
+        'cleanup is logged',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed');
+
+    reset_configs();
+    my $broken = $INCOMING_CONFIG =~ s/^rootfs: local:/rootfs: iso:/mr;
+    ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $broken });
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after a failed config fails');
+    ok(!$replies->[0]->{success}, 'config command fails');
+    like(
+        $task_log,
+        qr/^WARN: failed to handle 'config' command - /m,
+        'failed command is logged',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed after a failed config');
+
+    reset_configs();
+    ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $INCOMING_CONFIG });
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after the config fails');
+    unlike($task_log, qr/cleaning up/, 'no cleanup once the config arrived');
+    is(PVE::LXC::Config->load_config(300)->{lock}, 'migrate', 'config that arrived is kept');
+
+    # the volume of a transfer is only known once the transfer is done
+    my $import_cmd = {
+        cmd => 'disk-import',
+        volname => 'vm-300-disk-0',
+        format => 'raw',
+        export_formats => 'raw+size',
+        storage => 'local',
+    };
+    my ($queries, $ends_after, $freed) = (0, 2, []);
+    my $storage_tunnel_module = Test::MockModule->new('PVE::StorageTunnel');
+    $storage_tunnel_module->mock(
+        handle_disk_import => sub {
+            my ($state) = @_;
+            $state->{disk_import} = { pid => 1 };
+            return {};
+        },
+        handle_query_disk_import => sub {
+            my ($state) = @_;
+            return { status => 'pending' } if ++$queries < $ends_after;
+            delete $state->{disk_import};
+            $state->{cleanup}->{volumes}->{'local:300/vm-300-disk-0.raw'} = 1;
+            return { status => 'complete' };
+        },
+    );
+    $storage_module->mock(vdisk_free => sub { push $freed->@*, $_[1] });
+
+    reset_configs();
+    ($err, $replies) = run_mtunnel(300, $import_cmd);
+    ok($replies->[0]->{success}, 'volume transfer starts') or diag(explain($replies));
+    is_deeply(
+        $freed,
+        ['local:300/vm-300-disk-0.raw'],
+        'volume of a transfer that ended is freed',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed after the transfer ended');
+
+    reset_configs();
+    ($queries, $ends_after, $freed) = (0, 100, []);
+    ($err, $replies) = run_mtunnel(300, $import_cmd);
+    like(
+        $task_log,
+        qr/^WARN: cleanup failed - a volume transfer is still running$/m,
+        'transfer that does not end is reported',
+    );
+    is_deeply($freed, [], 'nothing is freed while a transfer runs');
+    ok(defined(read_raw_config(300)), 'reserved config is kept while a transfer runs');
+    $storage_module->unmock('vdisk_free');
+};
+
+subtest 'remote migration cleans up a target with an incompatible tunnel version' => sub {
+    reset_configs();
+    my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+    write_raw_config(200, $raw);
+
+    my $log = '';
+    my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+    $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+    my $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION - 1, age => 0 };
+    my ($commands, $cleanup) = (0);
+    my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+    $tunnel_module->mock(
+        fork_websocket_tunnel => sub { return $remote_version },
+        write_tunnel => sub { $commands++ },
+        finish_tunnel => sub { (undef, $cleanup) = @_ },
+    );
+
+    my $migrate = sub {
+        ($log, $commands, $cleanup) = ('', 0);
+        return run_task(sub {
+            PVE::LXC::Migrate->migrate(
+                'remote-node',
+                '192.0.2.1',
+                200,
+                { remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } } },
+            );
+        });
+    };
+
+    like($migrate->(), qr/migration aborted/, 'migration to an older target fails');
+    like($log, qr/^err: .*Remote tunnel endpoint too old/m, 'target is reported as too old');
+    is($cleanup, 1, 'older target is asked to clean up');
+    is($commands, 0, 'nothing else is sent to the target');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+
+    $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION + 1, age => 0 };
+    like($migrate->(), qr/migration aborted/, 'migration to an incompatible target fails');
+    like($log, qr/^err: .*Remote tunnel endpoint not compatible/m, 'target is reported');
+    is($cleanup, 1, 'incompatible target is asked to clean up');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
 remove_tree($WORKDIR);
 
 done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-17 22:42 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 22:34 [PATCH cluster/guest-common/qemu-server/container/manager/docs v2 00/36] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 01/36] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 02/36] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 03/36] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 04/36] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 05/36] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 06/36] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 07/36] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 08/36] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 09/36] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 10/36] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 11/36] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 12/36] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 13/36] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 14/36] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 15/36] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 16/36] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 17/36] api: status: return the guest uid Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 18/36] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 19/36] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 20/36] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 21/36] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 22/36] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 23/36] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-17 22:34 ` Thomas Lamprecht [this message]
2026-11-17 22:34 ` [PATCH container v2 25/36] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 26/36] api: status: return the guest uid Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 27/36] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 28/36] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 29/36] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 30/36] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 31/36] ui: guest summary: show " Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 32/36] api: migrate all: refuse targets that would drop " Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 33/36] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 34/36] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 35/36] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 36/36] pct: " Thomas Lamprecht
2026-11-18  5:57 ` superseded: [PATCH cluster/guest-common/qemu-server/container/manager/docs v2 00/36] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117223945.862914-25-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal