From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 533D31FF0AA for ; Tue, 17 Nov 2026 23:45:23 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D64B021B37; Tue, 17 Nov 2026 23:41:10 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH container v2 25/36] api: remote migration: take over the guest uid from the source Date: Tue, 17 Nov 2026 23:34:35 +0100 Message-ID: <20261117223945.862914-26-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com> References: <20261117223945.862914-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794955192476 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.196 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YLQCQO2ZT3LADCMKR3WQCXKV3QF7RK4E X-Message-ID-Hash: YLQCQO2ZT3LADCMKR3WQCXKV3QF7RK4E X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A container migrated to another cluster should keep its uid. Send the uid to the target right after the tunnel version check, before the container gets stopped for a restart migration and before any volume is transferred. The target takes it over for the config that reserves the VMID, and the config writes of the migration keep it. The target accepts the uid only once and only before the config arrives. The registration time is the time of that take-over. If another guest in the target cluster already has the uid, refuse the migration. A migration moves a guest, so a new uid without notice would leave the identity of the container with the other guest. The new option 'allow-new-uid' lets the migration go on with a new uid. A failure to take over the uid, for example because the cluster-wide lock cannot be taken, stops the migration at that point as well, and the source asks the target to clean up. It also does so if stopping the container for a restart migration fails afterwards, as the config on the target then has the uid already. The source waits up to 60 seconds for the reply to the uid, to cover the wait for that lock. Report the result in the task logs of both sides. A refusal only shows in the task log of the source, like any failed tunnel command. If a newer target reports a result that the source does not know, the source only logs a warning. A container without uid and a source with the previous tunnel version send no uid, so the container keeps the uid it got when the target reserved its VMID. Bump the tunnel version, so that older targets are refused before volumes are transferred, and not only when their strict parser rejects 'meta'. Keep accepting the previous version from older sources. Signed-off-by: Thomas Lamprecht --- changes v1 -> v2: * take the uid over at the start of the migration, through a new tunnel command, instead of with the config * refuse the migration if another guest in the target cluster has the uid, unless the new option 'allow-new-uid' is set (Dominik) * log the result on the source without a prefix, as for VMs src/PVE/API2/LXC.pm | 40 ++++- src/PVE/CLI/pct.pm | 7 + src/PVE/LXC/Migrate.pm | 31 +++- src/test/lifecycle_tests.pl | 339 ++++++++++++++++++++++++++++++++++++ 4 files changed, 414 insertions(+), 3 deletions(-) diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm index 67a38af..1a33c49 100644 --- a/src/PVE/API2/LXC.pm +++ b/src/PVE/API2/LXC.pm @@ -1468,6 +1468,13 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster" + . " already has the uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, }, }, returns => { @@ -3291,7 +3298,7 @@ __PACKAGE__->register_method({ my $socket_addr = "/run/pve/ct-$vmid.mtunnel"; my $lock = 'create'; - eval { PVE::LXC::Config->create_and_lock_config($vmid, 0, $lock); }; + my $reservation = eval { PVE::LXC::Config->create_and_lock_config($vmid, 0, $lock) }; raise_param_exc({ vmid => "unable to create empty CT config - $@" }) if $@; @@ -3321,6 +3328,18 @@ __PACKAGE__->register_method({ }; my $cmd_desc = { + identity => { + uid => { + type => 'string', + description => 'uid of the source guest', + }, + 'allow-new-uid' => { + type => 'boolean', + description => 'go on with a new uid if another guest has the uid', + optional => 1, + default => 0, + }, + }, config => { conf => { type => 'string', @@ -3375,9 +3394,26 @@ __PACKAGE__->register_method({ # bump/bump for opt-in changes return { api => $PVE::LXC::Migrate::WS_TUNNEL_VERSION, - age => 0, + age => 1, }; }, + 'identity' => sub { + my ($params) = @_; + + die "the uid can only be taken over once, before the config arrives\n" + if ($state->{lock} // '') ne 'create' || $state->{'identity-handled'}; + $state->{'identity-handled'} = 1; + + my $res = PVE::GuestIdentity::adopt_from_source( + 'PVE::LXC::Config', + $state->{vmid}, + $params->{uid}, + $reservation, + $params->{'allow-new-uid'}, + ); + + return { identity => $res }; + }, 'config' => sub { my ($params) = @_; diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm index 792ae11..da69eed 100755 --- a/src/PVE/CLI/pct.pm +++ b/src/PVE/CLI/pct.pm @@ -980,6 +980,13 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster" + . " already has the uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, }, }, returns => { diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm index 84c5139..aeadcb3 100644 --- a/src/PVE/LXC/Migrate.pm +++ b/src/PVE/LXC/Migrate.pm @@ -8,6 +8,7 @@ use File::Copy; # fixme: remove use PVE::Cluster; use PVE::GuestID; +use PVE::GuestIdentity; use PVE::INotify; use PVE::JSONSchema; use PVE::Replication; @@ -27,7 +28,8 @@ use PVE::AbstractMigrate; use base qw(PVE::AbstractMigrate); # compared against remote end's minimum version -our $WS_TUNNEL_VERSION = 2; +# 3: the 'identity' command, and 'meta' with the guest identity in the config +our $WS_TUNNEL_VERSION = 3; sub lock_vm { my ($self, $vmid, $code, @param) = @_; @@ -155,6 +157,11 @@ sub prepare { $err = "Remote tunnel endpoint too old, upgrade required\n" if $WS_TUNNEL_VERSION > $tunnel->{version}; + if (!$err) { + eval { $self->take_over_identity_on_target($tunnel) }; + $err = $@; + } + if ($err) { # the target already reserved the VMID with a locked config eval { PVE::Tunnel::finish_tunnel($tunnel, 1); }; @@ -193,6 +200,28 @@ sub prepare { return $running; } +# Lets the target of a remote migration take over the uid of the container, before anything gets +# transferred there. The target refuses a uid that another guest there has, unless the migration +# allows a new uid. A container without uid gets the one that the target reserved for it. +sub take_over_identity_on_target { + my ($self, $tunnel) = @_; + + my $identity = PVE::LXC::Config->get_identity($self->{vmconf}) or return; + my $params = { + uid => $identity->{uid}, + 'allow-new-uid' => $self->{opts}->{'allow-new-uid'} ? 1 : 0, + }; + + # the target waits for the cluster-wide guest identity lock + my $res = PVE::Tunnel::write_tunnel($tunnel, 60, 'identity', $params); + if (my $result = $res->{identity}) { + my ($level, $msg) = PVE::GuestIdentity::remote_adopt_result_log($result); + $self->log($level, $msg); + } + + return; +} + sub phase1 { my ($self, $vmid) = @_; diff --git a/src/test/lifecycle_tests.pl b/src/test/lifecycle_tests.pl index 930fa4f..0614d44 100755 --- a/src/test/lifecycle_tests.pl +++ b/src/test/lifecycle_tests.pl @@ -35,6 +35,7 @@ use PVE::CLI::pct; use PVE::LXC; use PVE::LXC::Config; use PVE::LXC::Create; +use PVE::LXC::Migrate; my $WORKDIR = 'lifecycle-working'; my $NODENAME = PVE::INotify::nodename(); @@ -94,6 +95,11 @@ my $storage_cfg = { shared => 1, content => { rootdir => 1, images => 1 }, }, + iso => { + type => 'dir', + path => '/mnt/iso', + content => { iso => 1 }, + }, }, }; @@ -1722,6 +1728,339 @@ subtest 'remote migration cleans up a target with an incompatible tunnel version is(read_raw_config(200), $raw, 'config of the source is unchanged'); }; +subtest 'remote migration takes over the guest identity of the source' => sub { + my $source_meta = 'ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso'; + my $incoming = $INCOMING_CONFIG =~ s/^meta: .*$/meta: $source_meta/mr; + # like the source sends it + my $identity_cmd = + { cmd => 'identity', uid => 'sourcesourcesourceso', 'allow-new-uid' => 0 }; + my $config_cmd = { cmd => 'config', conf => $incoming }; + my $quit_cmd = { cmd => 'quit', cleanup => 0 }; + my $cleanup_cmd = { cmd => 'quit', cleanup => 1 }; + $now_ms = 1_700_000_000_000; + + my $migrate = sub { + my ($name, $setup, $allow_new_uid, $expected_result, $expected_uid) = @_; + + reset_configs(); + $setup->(); + my ($err, $replies) = run_mtunnel( + 300, + { $identity_cmd->%*, 'allow-new-uid' => $allow_new_uid }, + $config_cmd, + $quit_cmd, + ); + is($err, '', "$name: tunnel exits normally") or diag(explain($replies), $task_log); + ok($replies->[0]->{success}, "$name: identity command succeeds"); + ok($replies->[1]->{success}, "$name: config command succeeds"); + ok(!exists($replies->[1]->{identity}), "$name: config command takes over nothing"); + + my $identity = $replies->[0]->{identity}; + is($identity->{result}, $expected_result, "$name: result is returned"); + my $stored = identity_of(300); + is($identity->{uid}, $stored->{uid}, "$name: returned uid is the stored one"); + if (defined($expected_uid)) { + is($stored->{uid}, $expected_uid, "$name: uid"); + } else { + isnt( + $stored->{uid}, + 'sourcesourcesourceso', + "$name: uid is not the one of the source", + ); + } + is($stored->{rtime}, $now_ms, "$name: rtime is the time of the migration"); + is(meta_of(300)->{ctime}, 1600000000, "$name: ctime of the source is kept"); + like( + $task_log, + qr/^guest identity: .*$identity->{uid}/m, + "$name: result is logged on the target", + ); + + return $identity; + }; + + $migrate->('free uid', sub { }, 0, 'adopted', 'sourcesourcesourceso'); + my $reserved_write = $written->{300}->[1]; + ok( + $reserved_write =~ m/^lock: create$/m + && $reserved_write =~ m/^meta: .*uid=sourcesourcesourceso$/m, + 'free uid: taken over right after the reservation', + ); + + my $in_use_by_ct = + sub { write_raw_config(150, existing_config('uid=sourcesourcesourceso')) }; + my $in_use_by_vm = + sub { write_raw_vm_config(151, "memory: 512\nmeta: uid=sourcesourcesourceso\n") }; + $migrate->('uid used by another container, allow-new-uid', $in_use_by_ct, 1, 'in-use'); + $migrate->('uid used by a VM, allow-new-uid', $in_use_by_vm, 1, 'in-use'); + + for my $case ( + ['uid used by another container', $in_use_by_ct, 150], + ['uid used by a VM', $in_use_by_vm, 151], + ) { + my ($name, $setup, $holder) = $case->@*; + reset_configs(); + $setup->(); + my ($err, $replies) = run_mtunnel(300, $identity_cmd, $cleanup_cmd); + is($err, '', "$name: tunnel exits normally") or diag($task_log); + ok(!$replies->[0]->{success}, "$name: identity command fails"); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - guest $holder in the target cluster already" + . " has the uid 'sourcesourcesourceso' of this guest - use 'allow-new-uid' to" + . " migrate it with a new uid\n", + "$name: migration is refused", + ); + is(scalar($written->{300}->@*), 1, "$name: nothing written besides the reservation"); + ok($replies->[1]->{success}, "$name: cleanup succeeds"); + ok(!defined(read_raw_config(300)), "$name: reservation is removed"); + } + + for my $case ( + ['source without identity command', $incoming], + ['source without uid', $INCOMING_CONFIG], + ) { + my ($name, $conf) = $case->@*; + reset_configs(); + my ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $conf }, $quit_cmd); + is($err, '', "$name: tunnel exits normally") or diag($task_log); + ok($replies->[0]->{success}, "$name: config command succeeds"); + my $uid = identity_of(300)->{uid}; + like($uid, qr/^[a-z2-7]{20}$/, "$name: container has a uid"); + isnt($uid, 'sourcesourcesourceso', "$name: it is a new one"); + } + + my $real_adopt = \&PVE::GuestIdentity::adopt; + my ($lock, $locked); + $identity_module->mock( + adopt => sub { + my ($class, $vmid) = @_; + $lock = PVE::LXC::Config->load_config($vmid)->{lock}; + open(my $fh, '>>', "$WORKDIR/lock/pve-config-$vmid.lock") + or die "open failed - $!\n"; + $locked = !flock($fh, LOCK_EX | LOCK_NB); + close($fh); + return $real_adopt->(@_); + }, + ); + $migrate->('order of the writes', sub { }, 0, 'adopted', 'sourcesourcesourceso'); + $identity_module->unmock('adopt'); + is($lock, 'create', 'identity is taken over before the config of the source is written'); + ok($locked, 'config lock is held while taking over the identity'); + + reset_configs(); + $domain_lock_error = "cfs-lock 'domain-guest-identity' error: got lock request timeout\n"; + my ($err, $replies) = run_mtunnel(300, $identity_cmd, $cleanup_cmd); + $domain_lock_error = undef; + is($err, '', 'identity lock failure: tunnel exits normally') or diag($task_log); + ok(!$replies->[0]->{success}, 'identity lock failure: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - unable to adopt the uid of the source guest -" + . " cfs-lock 'domain-guest-identity' error: got lock request timeout\n", + 'identity lock failure: error is returned', + ); + ok($replies->[1]->{success}, 'identity lock failure: cleanup succeeds'); + ok(!defined(read_raw_config(300)), 'identity lock failure: target config is removed'); + + reset_configs(); + ($err, $replies) = + run_mtunnel(300, $identity_cmd, $identity_cmd, $config_cmd, $identity_cmd, $quit_cmd); + is($err, '', 'repeated identity: tunnel exits normally'); + ok($replies->[0]->{success}, 'first identity command succeeds'); + ok($replies->[2]->{success}, 'config command succeeds'); + for my $i (1, 3) { + like( + $replies->[$i]->{msg}, + qr/the uid can only be taken over once, before the config arrives/, + "identity command at position $i fails", + ); + } + + reset_configs(); + my $broken = $incoming =~ s/^rootfs: local:/rootfs: iso:/mr; + ($err, $replies) = run_mtunnel( + 300, + $identity_cmd, + { cmd => 'config', conf => $broken }, + $config_cmd, + $config_cmd, + $quit_cmd, + ); + is($err, '', 'repeated config: tunnel exits normally'); + is($replies->[0]->{identity}->{result}, 'adopted', 'uid is taken over at the start'); + ok(!$replies->[1]->{success}, 'failing config command fails'); + ok($replies->[2]->{success}, 'retry of the config command succeeds'); + ok(!$replies->[3]->{success}, 'config command after success fails'); + is_deeply( + identity_of(300), + { uid => 'sourcesourcesourceso', rtime => $now_ms }, + 'identity taken over at the start is kept', + ); +}; + +subtest 'remote migration sends the guest identity on the source' => sub { + my $source_raw = <<~'EOF'; + arch: amd64 + hostname: source + memory: 512 + meta: ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso + ostype: debian + unprivileged: 1 + EOF + reset_configs(); + write_raw_config(987, $source_raw); + + my $log = ''; + my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" }); + + my ($commands, $reply) = ([]); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + write_tunnel => sub { + my ($tunnel, $timeout, $command, $params) = @_; + push $commands->@*, { command => $command, timeout => $timeout, params => $params }; + return $reply; + }, + ); + + my $new_migration = sub($opts = {}) { + return bless { + opts => { remote => { vmid => 300 }, $opts->%* }, + vmid => 987, + node => 'remote-node', + nodeip => '192.0.2.1', + storecfg => $storage_cfg, + vmconf => PVE::LXC::Config->load_config(987), + tunnel => {}, + }, + 'PVE::LXC::Migrate'; + }; + my $take_over = sub($self) { + ($commands, $log) = ([], ''); + return run_task(sub { $self->take_over_identity_on_target({}) }); + }; + + $reply = { + success => 1, + identity => { + result => 'adopted', + uid => 'sourcesourcesourceso', + requested => 'sourcesourcesourceso', + }, + }; + is($take_over->($new_migration->()), '', 'identity command succeeds'); + is_deeply( + $commands, + [{ + command => 'identity', + timeout => 60, + params => { uid => 'sourcesourcesourceso', 'allow-new-uid' => 0 }, + }], + 'uid is sent, the source waits for the identity lock of the target', + ); + my $adopted = 'guest identity: adopted uid sourcesourcesourceso from the source guest'; + like($log, qr/^info: \Q$adopted\E$/m, 'result of the target is logged'); + + is($take_over->($new_migration->({ 'allow-new-uid' => 1 })), '', 'allow-new-uid'); + is($commands->[0]->{params}->{'allow-new-uid'}, 1, 'allow-new-uid is sent'); + + # a target with a newer version has accepted the uid at this point + $reply = + { success => 1, identity => { result => 'future', uid => 'sourcesourcesourceso' } }; + is($take_over->($new_migration->()), '', 'outcome unknown here does not fail'); + like( + $log, + qr/^warn: unknown result of guest identity adoption 'future'$/m, + 'unknown outcome is a warning', + ); + + write_raw_config(987, $source_raw =~ s/^meta: .*\n//mr); + is($take_over->($new_migration->()), '', 'source without uid'); + is_deeply($commands, [], 'source without uid: nothing is sent'); + write_raw_config(987, $source_raw); + + $reply = { success => 1 }; + ($commands, $log) = ([], ''); + is(run_task(sub { $new_migration->()->phase1(987) }), '', 'phase 1 succeeds') + or diag($log, $task_log); + is($commands->[0]->{command}, 'config', 'phase 1 sends the config'); + like( + $commands->[0]->{params}->{conf}, + qr/^meta: .*uid=sourcesourcesourceso/m, + 'config is sent with the identity', + ); + is($commands->[0]->{timeout}, 10, 'config command does not wait for the identity lock'); + unlike($log, qr/guest identity/, 'nothing is logged about the identity'); + is(identity_of(987)->{uid}, 'sourcesourcesourceso', 'source keeps its uid'); + PVE::LXC::Config->remove_lock(987, 'migrate'); +}; + +subtest 'remote migration keeps the source if the target refuses the identity' => sub { + reset_configs(); + write_raw_config(987, <<~'EOF'); + arch: amd64 + hostname: source + memory: 512 + meta: ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso + ostype: debian + unprivileged: 1 + EOF + + my $log = ''; + my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" }); + + my ($commands, $cleanup) = ([]); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + fork_websocket_tunnel => sub { return { version => 3, age => 1 } }, + write_tunnel => sub { + my ($tunnel, $timeout, $command) = @_; + push $commands->@*, $command; + die "error - tunnel command '$command' failed - failed to handle 'identity' command" + . " - guest 150 in the target cluster already has the uid 'sourcesourcesourceso'" + . " of this guest - use 'allow-new-uid' to migrate it with a new uid\n"; + }, + finish_tunnel => sub { (undef, $cleanup) = @_ }, + ); + + my ($running, $stopped, $destroyed) = (1, 0, 0); + my $destroy_ct = \&PVE::LXC::destroy_lxc_container; + $lxc_module->mock( + check_running => sub { return $running }, + vm_stop => sub { $stopped = 1 }, + destroy_lxc_container => sub { $destroyed = 1 }, + ); + + my $err = run_task(sub { + PVE::LXC::Migrate->migrate( + 'remote-node', + '192.0.2.1', + 987, + { + remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } }, + restart => 1, + delete => 1, + }, + ); + }); + + $lxc_module->mock(check_running => sub { return 0 }, destroy_lxc_container => $destroy_ct); + $lxc_module->unmock('vm_stop'); + + like($err, qr/migration aborted/, 'migration fails'); + like($log, qr/^err: .*already has the uid 'sourcesourcesourceso'/m, 'refusal is logged'); + is_deeply($commands, ['identity'], 'nothing else is sent to the target'); + is($cleanup, 1, 'target gets cleaned up'); + ok(!$stopped, 'source is not stopped for the restart migration'); + ok(!$destroyed, 'source is not destroyed'); + my $conf = PVE::LXC::Config->load_config(987); + ok(!defined($conf->{lock}), 'source is not locked'); + is(identity_of(987)->{uid}, 'sourcesourcesourceso', 'source keeps its uid'); +}; + remove_tree($WORKDIR); done_testing(); -- 2.47.3