public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH container v3 26/38] test: remote migration: check the cleanup of the target after failures
Date: Wed, 18 Nov 2026 05:37:38 +0100	[thread overview]
Message-ID: <20261118043817.1659212-27-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com>

A config left behind on the target of a failed remote migration blocks
its VMID there, and with guest identities it would also keep the uid of
the container in the target cluster.

Check with a mocked tunnel that the source asks the target to clean up
if the tunnel versions do not match or the container does not stop for a
restart migration. Check on the target that it removes its reservation
if the tunnel closes before the config arrived, keeps a config that
arrived, and logs the error of a failed tunnel command.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v2 -> v3:
* follow the fixed wait for a volume transfer in the cleanup of the
  target, which is bounded by time: check a transfer that exits late and
  one that cannot be collected

changes v1 -> v2:
* was 'migrate: remote: clean up the target on a tunnel version
  mismatch', its fix got applied separately together with the cleanup
  after a failed shutdown
* more tests: failed shutdown of a restart migration, tunnel that closes
  before the config arrived (with and without a running volume
  transfer), logging of a failed tunnel command

 src/test/lifecycle_tests.pl | 175 ++++++++++++++++++++++++++++++++++++
 1 file changed, 175 insertions(+)

diff --git a/src/test/lifecycle_tests.pl b/src/test/lifecycle_tests.pl
index 7606d3e..ff42c5f 100755
--- a/src/test/lifecycle_tests.pl
+++ b/src/test/lifecycle_tests.pl
@@ -1890,6 +1890,181 @@ subtest 'restore over a container of another guest needs an override' => sub {
     );
 };
 
+subtest 'remote restart migration cleans up the target if the container does not stop' => sub {
+    reset_configs();
+    my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+    write_raw_config(200, $raw);
+
+    my $log = '';
+    my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+    $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+    my $cleanup;
+    my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+    $tunnel_module->mock(
+        fork_websocket_tunnel => sub {
+            return { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION, age => 0 };
+        },
+        write_tunnel => sub { return {} },
+        finish_tunnel => sub { (undef, $cleanup) = @_ },
+    );
+    $lxc_module->mock(
+        check_running => sub { return 1 },
+        vm_stop => sub { die "container did not stop\n" },
+    );
+
+    my $err = run_task(sub {
+        PVE::LXC::Migrate->migrate(
+            'remote-node',
+            '192.0.2.1',
+            200,
+            {
+                remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } },
+                restart => 1,
+            },
+        );
+    });
+    $lxc_module->mock(check_running => sub { return 0 });
+    $lxc_module->unmock('vm_stop');
+
+    like($err, qr/migration aborted/, 'migration fails');
+    like($log, qr/^err: .*container did not stop/m, 'failed shutdown is reported');
+    is($cleanup, 1, 'target is asked to clean up');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
+subtest 'migration tunnel logs failed commands and cleans up after an early end' => sub {
+    reset_configs();
+    my ($err, $replies) = run_mtunnel(300);
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends without a command fails');
+    like(
+        $task_log,
+        qr/^WARN: tunnel closed before the config arrived, cleaning up$/m,
+        'cleanup is logged',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed');
+
+    reset_configs();
+    my $broken = $INCOMING_CONFIG =~ s/^rootfs: local:/rootfs: iso:/mr;
+    ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $broken });
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after a failed config fails');
+    ok(!$replies->[0]->{success}, 'config command fails');
+    like(
+        $task_log,
+        qr/^WARN: failed to handle 'config' command - /m,
+        'failed command is logged',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed after a failed config');
+
+    reset_configs();
+    ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $INCOMING_CONFIG });
+    like($err, qr/mtunnel exited unexpectedly/, 'tunnel that ends after the config fails');
+    unlike($task_log, qr/cleaning up/, 'no cleanup once the config arrived');
+    is(PVE::LXC::Config->load_config(300)->{lock}, 'migrate', 'config that arrived is kept');
+
+    # the volume of a transfer is only known once the transfer is done
+    my $import_cmd = {
+        cmd => 'disk-import',
+        volname => 'vm-300-disk-0',
+        format => 'raw',
+        export_formats => 'raw+size',
+        storage => 'local',
+    };
+    my ($queries, $ends_after, $query_error, $freed) = (0, 2, undef, []);
+    my $storage_tunnel_module = Test::MockModule->new('PVE::StorageTunnel');
+    $storage_tunnel_module->mock(
+        handle_disk_import => sub {
+            my ($state) = @_;
+            $state->{disk_import} = { pid => 1 };
+            return {};
+        },
+        handle_query_disk_import => sub {
+            my ($state) = @_;
+            die $query_error if $query_error;
+            return { status => 'pending' } if ++$queries < $ends_after;
+            delete $state->{disk_import};
+            $state->{cleanup}->{volumes}->{'local:300/vm-300-disk-0.raw'} = 1;
+            return { status => 'complete' };
+        },
+    );
+    $storage_module->mock(vdisk_free => sub { push $freed->@*, $_[1] });
+
+    reset_configs();
+    ($err, $replies) = run_mtunnel(300, $import_cmd);
+    ok($replies->[0]->{success}, 'volume transfer starts') or diag(explain($replies));
+    is_deeply(
+        $freed,
+        ['local:300/vm-300-disk-0.raw'],
+        'volume of a transfer that ended is freed',
+    );
+    ok(!defined(read_raw_config(300)), 'reserved config is removed after the transfer ended');
+
+    # a transfer that closed its output can be queried many times before it has exited
+    reset_configs();
+    ($queries, $ends_after, $freed) = (0, 20, []);
+    ($err, $replies) = run_mtunnel(300, $import_cmd);
+    is_deeply($freed, ['local:300/vm-300-disk-0.raw'], 'volume of a transfer that exits late');
+    ok(!defined(read_raw_config(300)), 'reserved config is removed after a late exit');
+
+    reset_configs();
+    ($queries, $query_error, $freed) = (0, "unable to query the transfer\n", []);
+    ($err, $replies) = run_mtunnel(300, $import_cmd);
+    like(
+        $task_log,
+        qr/^WARN: cleanup failed - a volume transfer is still running$/m,
+        'transfer that cannot be collected is reported',
+    );
+    is_deeply($freed, [], 'nothing is freed while a transfer is not collected');
+    ok(
+        defined(read_raw_config(300)),
+        'reserved config is kept while a transfer is not collected',
+    );
+    $storage_module->unmock('vdisk_free');
+};
+
+subtest 'remote migration cleans up a target with an incompatible tunnel version' => sub {
+    reset_configs();
+    my $raw = existing_config('ctime=1500000000,rtime=1500000000000,uid=existingexistingexis');
+    write_raw_config(200, $raw);
+
+    my $log = '';
+    my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate');
+    $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" });
+
+    my $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION - 1, age => 0 };
+    my ($commands, $cleanup) = (0);
+    my $tunnel_module = Test::MockModule->new('PVE::Tunnel');
+    $tunnel_module->mock(
+        fork_websocket_tunnel => sub { return $remote_version },
+        write_tunnel => sub { $commands++ },
+        finish_tunnel => sub { (undef, $cleanup) = @_ },
+    );
+
+    my $migrate = sub {
+        ($log, $commands, $cleanup) = ('', 0);
+        return run_task(sub {
+            PVE::LXC::Migrate->migrate(
+                'remote-node',
+                '192.0.2.1',
+                200,
+                { remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } } },
+            );
+        });
+    };
+
+    like($migrate->(), qr/migration aborted/, 'migration to an older target fails');
+    like($log, qr/^err: .*Remote tunnel endpoint too old/m, 'target is reported as too old');
+    is($cleanup, 1, 'older target is asked to clean up');
+    is($commands, 0, 'nothing else is sent to the target');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+
+    $remote_version = { version => $PVE::LXC::Migrate::WS_TUNNEL_VERSION + 1, age => 0 };
+    like($migrate->(), qr/migration aborted/, 'migration to an incompatible target fails');
+    like($log, qr/^err: .*Remote tunnel endpoint not compatible/m, 'target is reported');
+    is($cleanup, 1, 'incompatible target is asked to clean up');
+    is(read_raw_config(200), $raw, 'config of the source is unchanged');
+};
+
 remove_tree($WORKDIR);
 
 done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-18  4:42 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  4:37 [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 01/38] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 02/38] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 03/38] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 04/38] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 05/38] pmxcfs: confdb: call an optional listener when the nodes get published Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 06/38] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 07/38] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 08/38] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 09/38] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 10/38] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 11/38] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 12/38] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 13/38] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 15/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 16/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 17/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 18/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 19/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 20/38] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 21/38] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 22/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 23/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 24/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 25/38] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-18  4:37 ` Thomas Lamprecht [this message]
2026-11-18  4:37 ` [PATCH container v3 27/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 28/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 29/38] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 30/38] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 31/38] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 32/38] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 33/38] ui: guest summary: show " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 34/38] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 35/38] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 36/38] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 37/38] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 38/38] pct: " Thomas Lamprecht
2026-11-18 13:49 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Dominik Csapak
2026-11-18 14:13   ` Thomas Lamprecht
2026-11-19 10:38 ` superseded: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118043817.1659212-27-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal