From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 66BE81FF0DA for ; Thu, 19 Nov 2026 20:25:39 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7F37421C0D; Thu, 19 Nov 2026 20:20:25 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH container v5 26/41] api: remote migration: adopt the guest uid from the source Date: Thu, 19 Nov 2026 20:18:58 +0100 Message-ID: <20261119191914.2336934-27-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119191914.2336934-1-t.lamprecht@proxmox.com> References: <20261119191914.2336934-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795115960698 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.155 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YTGORCP2RN3FOD43N5FHQSKP7FRLWVXX X-Message-ID-Hash: YTGORCP2RN3FOD43N5FHQSKP7FRLWVXX X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A container migrated to another cluster should keep its uid. Send the uid to the target right after the tunnel version check. At that point, the container is not stopped yet for a restart migration and no volume is transferred. The target adopts the uid for the config that reserves the VMID, and the later config writes of the migration keep it. The target accepts the uid only once and only before the config arrives. The registration time is the time at which the target adopted the uid. The command schema, the target handler and the sender come from pve-guest-common and are shared with VMs. If another guest in the target cluster already has the uid, refuse the migration. A migration moves a guest. If the migrated container silently got a new uid, the uid that tools know for it would lead them to the other guest. The new option 'allow-new-uid' lets the migration go on with a new uid. Any other failure to adopt the uid, for example because the cluster-wide lock cannot be taken, stops the migration at that point as well, and the source asks the target to clean up. The source also does so if stopping the container for a restart migration fails afterwards, because the config on the target has the uid already by then. The source waits up to 60 seconds for the reply to the uid, to cover the wait for that lock. Report the result in the task logs of both sides. If a newer target reports a result that the source does not know, the source only logs a warning. A source with the previous tunnel version sends no uid, so the container on the target keeps the uid that it got when the target reserved its VMID. The target logs that then, and that a migration back to such a cluster gets refused. On the source, a container without a uid gets one before its uid is sent. Bump the tunnel version. Older targets then get refused before volumes are transferred, and not only when their strict parser rejects 'meta'. Keep accepting the previous version from older sources. Signed-off-by: Thomas Lamprecht Reviewed-by: Dominik Csapak Tested-by: Dominik Csapak --- changes v4 -> v5: * the target task logs if the source sent no uid, with the new uid of the container and that a migration back to such a cluster gets refused (Dominik) * 'adopt' instead of 'take over' for the uid, in the subject, the commit message and the tests, so that the task logs, the docs and the code use one word for the step changes v3 -> v4: * 'allow-new-uid' is the standard option of pve-guest-common, and the sender is called adopt_identity_on_target * the source logs the result in its own words, see the pve-guest-common patch * commit message: which side assigns a missing uid * commit message: plainer wording, same content changes v2 -> v3: * use the shared helpers of pve-guest-common for the identity command * a container without uid gets one before the source sends it; before, source and target got unrelated uids changes v1 -> v2: * take the uid over at the start of the migration, through a new tunnel command, instead of with the config * refuse the migration if another guest in the target cluster has the uid, unless the new option 'allow-new-uid' is set (Dominik) * log the result on the source without a prefix, as for VMs src/PVE/API2/LXC.pm | 15 +- src/PVE/CLI/pct.pm | 2 + src/PVE/LXC/Migrate.pm | 8 +- src/test/lifecycle_tests.pl | 361 ++++++++++++++++++++++++++++++++++++ 4 files changed, 383 insertions(+), 3 deletions(-) diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm index 3f7d427..2e2f260 100644 --- a/src/PVE/API2/LXC.pm +++ b/src/PVE/API2/LXC.pm @@ -1543,6 +1543,7 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => get_standard_option('pve-guest-allow-new-uid'), }, }, returns => { @@ -3351,7 +3352,7 @@ __PACKAGE__->register_method({ my $socket_addr = PVE::LXC::mtunnel_socket($vmid); my $lock = 'create'; - eval { PVE::LXC::Config->create_and_lock_config($vmid, 0, $lock); }; + my $reservation = eval { PVE::LXC::Config->create_and_lock_config($vmid, 0, $lock) }; raise_param_exc({ vmid => "unable to create empty CT config - $@" }) if $@; @@ -3381,6 +3382,7 @@ __PACKAGE__->register_method({ }; my $cmd_desc = { + identity => PVE::GuestIdentity::tunnel_identity_cmd_schema(), config => { conf => { type => 'string', @@ -3435,12 +3437,21 @@ __PACKAGE__->register_method({ # bump/bump for opt-in changes return { api => $PVE::LXC::Migrate::WS_TUNNEL_VERSION, - age => 0, + age => 1, }; }, + 'identity' => sub { + my ($params) = @_; + + return PVE::GuestIdentity::handle_tunnel_identity( + 'PVE::LXC::Config', $state, $reservation, $params, + ); + }, 'config' => sub { my ($params) = @_; + PVE::GuestIdentity::log_tunnel_identity_not_sent($state, $reservation); + # parse and write out VM FW config if given if (my $fw_conf = $params->{'firewall-config'}) { my ($path, $fh) = PVE::Tools::tempfile_contents($fw_conf, 700); diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm index 36ad3f3..1f34c0c 100755 --- a/src/PVE/CLI/pct.pm +++ b/src/PVE/CLI/pct.pm @@ -14,6 +14,7 @@ use PVE::CpuSet; use PVE::Exception qw(raise_param_exc); use PVE::File; use PVE::GuestHelpers; +use PVE::GuestIdentity; use PVE::INotify; use PVE::JSONSchema qw(get_standard_option); use PVE::RPCEnvironment; @@ -981,6 +982,7 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => get_standard_option('pve-guest-allow-new-uid'), }, }, returns => { diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm index b3f92e7..1870271 100644 --- a/src/PVE/LXC/Migrate.pm +++ b/src/PVE/LXC/Migrate.pm @@ -27,7 +27,8 @@ use PVE::AbstractMigrate; use base qw(PVE::AbstractMigrate); # compared against remote end's minimum version -our $WS_TUNNEL_VERSION = 2; +# 3: the 'identity' command, and 'meta' with the guest identity in the config +our $WS_TUNNEL_VERSION = 3; sub fork_tunnel { my ($self, $ssh_forward_info) = @_; @@ -167,6 +168,11 @@ sub prepare { $err = "Remote tunnel endpoint too old, upgrade required\n" if $WS_TUNNEL_VERSION > $tunnel->{version}; + if (!$err) { + eval { $self->adopt_identity_on_target('PVE::LXC::Config', $tunnel) }; + $err = $@; + } + if ($err) { # the target already reserved the VMID with a locked config eval { PVE::Tunnel::finish_tunnel($tunnel, 1); }; diff --git a/src/test/lifecycle_tests.pl b/src/test/lifecycle_tests.pl index 226941b..8f6a1f6 100755 --- a/src/test/lifecycle_tests.pl +++ b/src/test/lifecycle_tests.pl @@ -75,6 +75,12 @@ my sub write_raw_config($vmid, $raw, $node = undef) { file_set_contents($file, $raw); } +my sub write_raw_vm_config($vmid, $raw) { + my $dir = cfs_file("nodes/$NODENAME/qemu-server"); + make_path($dir); + file_set_contents("$dir/$vmid.conf", $raw); +} + my sub read_raw_config($vmid, $node = undef) { my $file = cfs_file(ct_config_path($vmid, $node)); return -f $file ? file_get_contents($file) : undef; @@ -2328,6 +2334,361 @@ subtest 'restore over the container left behind by a restore of a backup' => sub is(pending_uid_of(200), undef, 'pending uid is removed'); }; +subtest 'remote migration adopts the guest identity of the source' => sub { + my $source_meta = 'ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso'; + my $incoming = $INCOMING_CONFIG =~ s/^meta: .*$/meta: $source_meta/mr; + # like the source sends it + my $identity_cmd = + { cmd => 'identity', uid => 'sourcesourcesourceso', 'allow-new-uid' => 0 }; + my $config_cmd = { cmd => 'config', conf => $incoming }; + my $quit_cmd = { cmd => 'quit', cleanup => 0 }; + my $cleanup_cmd = { cmd => 'quit', cleanup => 1 }; + $now_ms = 1_700_000_000_000; + + my $migrate = sub { + my ($name, $setup, $allow_new_uid, $expected_result, $expected_uid) = @_; + + reset_configs(); + $setup->(); + my ($err, $replies) = run_mtunnel( + 300, + { $identity_cmd->%*, 'allow-new-uid' => $allow_new_uid }, + $config_cmd, + $quit_cmd, + ); + is($err, '', "$name: tunnel exits normally") or diag(explain($replies), $task_log); + ok($replies->[0]->{success}, "$name: identity command succeeds"); + ok($replies->[1]->{success}, "$name: config command succeeds"); + ok(!exists($replies->[1]->{identity}), "$name: config command adopts nothing"); + + my $identity = $replies->[0]->{identity}; + is($identity->{result}, $expected_result, "$name: result is returned"); + my $stored = identity_of(300); + is($identity->{uid}, $stored->{uid}, "$name: returned uid is the stored one"); + if (defined($expected_uid)) { + is($stored->{uid}, $expected_uid, "$name: uid"); + } else { + isnt( + $stored->{uid}, + 'sourcesourcesourceso', + "$name: uid is not the one of the source", + ); + } + is($stored->{rtime}, $now_ms, "$name: rtime is the time of the migration"); + is(meta_of(300)->{ctime}, 1600000000, "$name: ctime of the source is kept"); + like( + $task_log, + qr/^guest identity: .*$identity->{uid}/m, + "$name: result is logged on the target", + ); + + return $identity; + }; + + $migrate->('free uid', sub { }, 0, 'adopted', 'sourcesourcesourceso'); + my $reserved_write = $written->{300}->[1]; + ok( + $reserved_write =~ m/^lock: create$/m + && $reserved_write =~ m/^meta: .*uid=sourcesourcesourceso$/m, + 'free uid: adopted right after the reservation', + ); + + my $in_use_by_ct = + sub { write_raw_config(150, existing_config('uid=sourcesourcesourceso')) }; + my $in_use_by_vm = + sub { write_raw_vm_config(151, "memory: 512\nmeta: uid=sourcesourcesourceso\n") }; + $migrate->('uid used by another container, allow-new-uid', $in_use_by_ct, 1, 'in-use'); + $migrate->('uid used by a VM, allow-new-uid', $in_use_by_vm, 1, 'in-use'); + + for my $case ( + ['uid used by another container', $in_use_by_ct, 150], + ['uid used by a VM', $in_use_by_vm, 151], + ) { + my ($name, $setup, $holder) = $case->@*; + reset_configs(); + $setup->(); + my ($err, $replies) = run_mtunnel(300, $identity_cmd, $cleanup_cmd); + is($err, '', "$name: tunnel exits normally") or diag($task_log); + ok(!$replies->[0]->{success}, "$name: identity command fails"); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - guest $holder in the target cluster already" + . " has the uid 'sourcesourcesourceso' of this guest - use 'allow-new-uid' to" + . " migrate it with a new uid\n", + "$name: migration is refused", + ); + is(scalar($written->{300}->@*), 1, "$name: nothing written besides the reservation"); + ok($replies->[1]->{success}, "$name: cleanup succeeds"); + ok(!defined(read_raw_config(300)), "$name: reservation is removed"); + } + + for my $case ( + ['source without identity command', $incoming], + ['source without identity command and uid', $INCOMING_CONFIG], + ) { + my ($name, $conf) = $case->@*; + reset_configs(); + my ($err, $replies) = run_mtunnel(300, { cmd => 'config', conf => $conf }, $quit_cmd); + is($err, '', "$name: tunnel exits normally") or diag($task_log); + ok($replies->[0]->{success}, "$name: config command succeeds"); + my $uid = identity_of(300)->{uid}; + like($uid, qr/^[a-z2-7]{20}$/, "$name: container has a uid"); + isnt($uid, 'sourcesourcesourceso', "$name: it is a new one"); + like( + $task_log, + qr/^guest identity: the source sent no uid, .* has the new uid $uid$/m, + "$name: target task logs the new uid", + ); + like( + $task_log, + qr/^guest identity: .* a migration back there gets refused$/m, + "$name: and that the container cannot migrate back", + ); + } + + my $real_adopt = \&PVE::GuestIdentity::adopt; + my ($lock, $locked); + $identity_module->mock( + adopt => sub { + my ($class, $vmid) = @_; + $lock = PVE::LXC::Config->load_config($vmid)->{lock}; + open(my $fh, '>>', "$WORKDIR/lock/pve-config-$vmid.lock") + or die "open failed - $!\n"; + $locked = !flock($fh, LOCK_EX | LOCK_NB); + close($fh); + return $real_adopt->(@_); + }, + ); + $migrate->('order of the writes', sub { }, 0, 'adopted', 'sourcesourcesourceso'); + $identity_module->unmock('adopt'); + is($lock, 'create', 'identity is adopted before the config of the source is written'); + ok($locked, 'config lock is held while adopting the identity'); + + reset_configs(); + $domain_lock_error = "cfs-lock 'domain-guest-identity' error: got lock request timeout\n"; + my ($err, $replies) = run_mtunnel(300, $identity_cmd, $cleanup_cmd); + $domain_lock_error = undef; + is($err, '', 'identity lock failure: tunnel exits normally') or diag($task_log); + ok(!$replies->[0]->{success}, 'identity lock failure: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - unable to adopt the uid of the source guest -" + . " cfs-lock 'domain-guest-identity' error: got lock request timeout\n", + 'identity lock failure: error is returned', + ); + ok($replies->[1]->{success}, 'identity lock failure: cleanup succeeds'); + ok(!defined(read_raw_config(300)), 'identity lock failure: target config is removed'); + + reset_configs(); + ($err, $replies) = + run_mtunnel(300, $identity_cmd, $identity_cmd, $config_cmd, $identity_cmd, $quit_cmd); + is($err, '', 'repeated identity: tunnel exits normally'); + ok($replies->[0]->{success}, 'first identity command succeeds'); + ok($replies->[2]->{success}, 'config command succeeds'); + unlike($task_log, qr/the source sent no uid/, 'no note about a source without uid'); + for my $i (1, 3) { + like( + $replies->[$i]->{msg}, + qr/the uid can only be adopted once, before the config arrives/, + "identity command at position $i fails", + ); + } + + reset_configs(); + my $broken = $incoming =~ s/^rootfs: local:/rootfs: iso:/mr; + ($err, $replies) = run_mtunnel( + 300, + $identity_cmd, + { cmd => 'config', conf => $broken }, + $config_cmd, + $config_cmd, + $quit_cmd, + ); + is($err, '', 'repeated config: tunnel exits normally'); + is($replies->[0]->{identity}->{result}, 'adopted', 'uid is adopted at the start'); + ok(!$replies->[1]->{success}, 'failing config command fails'); + ok($replies->[2]->{success}, 'retry of the config command succeeds'); + ok(!$replies->[3]->{success}, 'config command after success fails'); + is_deeply( + identity_of(300), + { uid => 'sourcesourcesourceso', rtime => $now_ms }, + 'identity adopted at the start is kept', + ); +}; + +subtest 'remote migration sends the guest identity on the source' => sub { + my $source_raw = <<~'EOF'; + arch: amd64 + hostname: source + memory: 512 + meta: ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso + ostype: debian + unprivileged: 1 + EOF + reset_configs(); + write_raw_config(987, $source_raw); + + my $log = ''; + my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" }); + + my ($commands, $reply) = ([]); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + write_tunnel => sub { + my ($tunnel, $timeout, $command, $params) = @_; + push $commands->@*, { command => $command, timeout => $timeout, params => $params }; + return $reply; + }, + ); + + my $new_migration = sub($opts = {}) { + return bless { + opts => { remote => { vmid => 300 }, $opts->%* }, + vmid => 987, + node => 'remote-node', + nodeip => '192.0.2.1', + storecfg => $storage_cfg, + vmconf => PVE::LXC::Config->load_config(987), + tunnel => {}, + }, + 'PVE::LXC::Migrate'; + }; + my $take_over = sub($self) { + ($commands, $log) = ([], ''); + return run_task(sub { $self->adopt_identity_on_target('PVE::LXC::Config', {}) }); + }; + + $reply = { + success => 1, + identity => { + result => 'adopted', + uid => 'sourcesourcesourceso', + requested => 'sourcesourcesourceso', + }, + }; + is($take_over->($new_migration->()), '', 'identity command succeeds'); + is_deeply( + $commands, + [{ + command => 'identity', + timeout => 60, + params => { uid => 'sourcesourcesourceso', 'allow-new-uid' => 0 }, + }], + 'uid is sent, the source waits for the identity lock of the target', + ); + my $adopted = 'guest identity: the migrated guest keeps uid sourcesourcesourceso'; + like($log, qr/^info: \Q$adopted\E$/m, 'result of the target is logged'); + + is($take_over->($new_migration->({ 'allow-new-uid' => 1 })), '', 'allow-new-uid'); + is($commands->[0]->{params}->{'allow-new-uid'}, 1, 'allow-new-uid is sent'); + + # a target with a newer version has accepted the uid at this point + $reply = + { success => 1, identity => { result => 'future', uid => 'sourcesourcesourceso' } }; + is($take_over->($new_migration->()), '', 'outcome unknown here does not fail'); + like( + $log, + qr/^warn: guest identity: unknown result 'future' of the target$/m, + 'unknown outcome is a warning', + ); + + write_raw_config(987, $source_raw =~ s/^meta: .*\n//mr); + $reply = { success => 1 }; + is($take_over->($new_migration->()), '', 'source without uid'); + my $uid = identity_of(987)->{uid}; + like($uid, qr/^[a-z2-7]{20}$/, 'source without uid: it gets a uid first'); + is_deeply( + $commands, + [{ + command => 'identity', + timeout => 60, + params => { uid => $uid, 'allow-new-uid' => 0 }, + }], + 'source without uid: that uid is sent', + ); + write_raw_config(987, $source_raw); + + $reply = { success => 1 }; + ($commands, $log) = ([], ''); + is(run_task(sub { $new_migration->()->phase1(987) }), '', 'phase 1 succeeds') + or diag($log, $task_log); + is($commands->[0]->{command}, 'config', 'phase 1 sends the config'); + like( + $commands->[0]->{params}->{conf}, + qr/^meta: .*uid=sourcesourcesourceso/m, + 'config is sent with the identity', + ); + is($commands->[0]->{timeout}, 10, 'config command does not wait for the identity lock'); + unlike($log, qr/guest identity/, 'nothing is logged about the identity'); + is(identity_of(987)->{uid}, 'sourcesourcesourceso', 'source keeps its uid'); + PVE::LXC::Config->remove_lock(987, 'migrate'); +}; + +subtest 'remote migration keeps the source if the target refuses the identity' => sub { + reset_configs(); + write_raw_config(987, <<~'EOF'); + arch: amd64 + hostname: source + memory: 512 + meta: ctime=1600000000,rtime=1600000000000,uid=sourcesourcesourceso + ostype: debian + unprivileged: 1 + EOF + + my $log = ''; + my $migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n" }); + + my ($commands, $cleanup) = ([]); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + fork_websocket_tunnel => sub { return { version => 3, age => 1 } }, + write_tunnel => sub { + my ($tunnel, $timeout, $command) = @_; + push $commands->@*, $command; + die "error - tunnel command '$command' failed - failed to handle 'identity' command" + . " - guest 150 in the target cluster already has the uid 'sourcesourcesourceso'" + . " of this guest - use 'allow-new-uid' to migrate it with a new uid\n"; + }, + finish_tunnel => sub { (undef, $cleanup) = @_ }, + ); + + my ($running, $stopped, $destroyed) = (1, 0, 0); + my $destroy_ct = \&PVE::LXC::destroy_lxc_container; + $lxc_module->mock( + check_running => sub { return $running }, + vm_stop => sub { $stopped = 1 }, + destroy_lxc_container => sub { $destroyed = 1 }, + ); + + my $err = run_task(sub { + PVE::LXC::Migrate->migrate( + 'remote-node', + '192.0.2.1', + 987, + { + remote => { vmid => 300, conn => { host => '192.0.2.1', port => 8006 } }, + restart => 1, + delete => 1, + }, + ); + }); + + $lxc_module->mock(check_running => sub { return 0 }, destroy_lxc_container => $destroy_ct); + $lxc_module->unmock('vm_stop'); + + like($err, qr/migration aborted/, 'migration fails'); + like($log, qr/^err: .*already has the uid 'sourcesourcesourceso'/m, 'refusal is logged'); + is_deeply($commands, ['identity'], 'nothing else is sent to the target'); + is($cleanup, 1, 'target gets cleaned up'); + ok(!$stopped, 'source is not stopped for the restart migration'); + ok(!$destroyed, 'source is not destroyed'); + my $conf = PVE::LXC::Config->load_config(987); + ok(!defined($conf->{lock}), 'source is not locked'); + is(identity_of(987)->{uid}, 'sourcesourcesourceso', 'source keeps its uid'); +}; + remove_tree($WORKDIR); done_testing(); -- 2.47.3