From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 2BDD51FF0AA for ; Tue, 17 Nov 2026 23:43:45 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4A72E219DE; Tue, 17 Nov 2026 23:41:00 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v2 16/36] api: remote migration: take over the guest uid from the source Date: Tue, 17 Nov 2026 23:34:26 +0100 Message-ID: <20261117223945.862914-17-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com> References: <20261117223945.862914-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794955191655 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.196 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: EFJOJIU3R6FRJECLFBHZNK75QHA7CWXM X-Message-ID-Hash: EFJOJIU3R6FRJECLFBHZNK75QHA7CWXM X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A VM migrated to another cluster should keep its uid. Send the uid to the target right after the tunnel version check, before anything is allocated or transferred there. The target takes it over through the cluster-wide check of pve-guest-common, for the config that reserves the VMID, and the config writes of the migration keep it. The target accepts the uid only once and only before the config arrives. The other members of 'meta', including the creation information, come from the source as before. If another guest in the target cluster already has the uid, refuse the migration. A migration moves a guest, so a new uid without notice would leave the identity of the VM with the other guest. The new option 'allow-new-uid' lets the migration go on with a new uid. A failure to take over the uid, for example because the cluster-wide lock cannot be taken, stops the migration at that point as well. The source then asks the target to clean up, as on a tunnel version mismatch. Report the result in the task logs of both sides. A refusal only shows in the task log of the source, like any failed tunnel command. If a newer target reports a result that the source does not know, the source only logs a warning. A VM without uid and a source with the previous tunnel version send no uid, so the VM keeps the uid it got when the target reserved its VMID. Bump the tunnel version, so that older targets are refused before disks are allocated or transferred, and not only when their strict config parser rejects the identity. Keep accepting the previous tunnel version from older sources. Signed-off-by: Thomas Lamprecht --- changes v1 -> v2: * take the uid over at the start of the migration, through a new tunnel command, instead of with the config * refuse the migration if another guest in the target cluster has the uid, unless the new option 'allow-new-uid' is set (Dominik) src/PVE/API2/Qemu.pm | 40 +++- src/PVE/CLI/qm.pm | 7 + src/PVE/QemuMigrate.pm | 38 ++- src/test/run_guest_identity_tests.pl | 334 +++++++++++++++++++++++++++ 4 files changed, 411 insertions(+), 8 deletions(-) diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index 7b35702c..b564839e 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -5863,6 +5863,13 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster" + . " already has the uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, }, }, returns => { @@ -6873,7 +6880,7 @@ __PACKAGE__->register_method({ PVE::Cluster::check_cfs_quorum(); my $lock = 'create'; - eval { PVE::QemuConfig->create_and_lock_config($vmid, 0, $lock); }; + my $reservation = eval { PVE::QemuConfig->create_and_lock_config($vmid, 0, $lock); }; raise_param_exc({ vmid => "unable to create empty VM config - $@" }) if $@; @@ -6903,6 +6910,18 @@ __PACKAGE__->register_method({ }; my $cmd_desc = { + identity => { + uid => { + type => 'string', + description => 'uid of the source guest', + }, + 'allow-new-uid' => { + type => 'boolean', + description => 'go on with a new uid if another guest has the uid', + optional => 1, + default => 0, + }, + }, config => { conf => { type => 'string', @@ -7027,9 +7046,26 @@ __PACKAGE__->register_method({ # bump/bump for opt-in changes return { api => $PVE::QemuMigrate::WS_TUNNEL_VERSION, - age => 0, + age => 1, }; }, + 'identity' => sub { + my ($params) = @_; + + die "the uid can only be taken over once, before the config arrives\n" + if ($state->{lock} // '') ne 'create' || $state->{'identity-handled'}; + $state->{'identity-handled'} = 1; + + my $res = PVE::GuestIdentity::adopt_from_source( + 'PVE::QemuConfig', + $state->{vmid}, + $params->{uid}, + $reservation, + $params->{'allow-new-uid'}, + ); + + return { identity => $res }; + }, 'config' => sub { my ($params) = @_; diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm index c8b352d6..355063a9 100755 --- a/src/PVE/CLI/qm.pm +++ b/src/PVE/CLI/qm.pm @@ -225,6 +225,13 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster" + . " already has the uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, }, }, returns => { diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm index 4477966f..98f173b2 100644 --- a/src/PVE/QemuMigrate.pm +++ b/src/PVE/QemuMigrate.pm @@ -15,6 +15,7 @@ use PVE::Format qw(render_bytes); use PVE::Firewall::Helpers; use PVE::GuestHelpers qw(safe_boolean_ne safe_string_ne); use PVE::GuestID; +use PVE::GuestIdentity; use PVE::INotify; use PVE::JSONSchema; use PVE::RPCEnvironment; @@ -47,7 +48,8 @@ use PVE::AbstractMigrate; use base qw(PVE::AbstractMigrate); # compared against remote end's minimum version -our $WS_TUNNEL_VERSION = 2; +# 3: the 'identity' command, and 'meta' with the guest identity in the config +our $WS_TUNNEL_VERSION = 3; sub fork_tunnel { my ($self, $ssh_forward_info) = @_; @@ -350,17 +352,22 @@ sub prepare { $self->log('info', "local WS tunnel version: $WS_TUNNEL_VERSION"); $self->log('info', "remote WS tunnel version: $tunnel->{version}"); $self->log('info', "minimum required WS tunnel version: $min_version"); - my $version_err; - $version_err = "Remote tunnel endpoint not compatible, upgrade required\n" + my $err; + $err = "Remote tunnel endpoint not compatible, upgrade required\n" if $WS_TUNNEL_VERSION < $min_version; - $version_err = "Remote tunnel endpoint too old, upgrade required\n" + $err = "Remote tunnel endpoint too old, upgrade required\n" if $WS_TUNNEL_VERSION > $tunnel->{version}; - if ($version_err) { + if (!$err) { + eval { $self->take_over_identity_on_target($tunnel) }; + $err = $@; + } + + if ($err) { # the target already reserved the VMID with a locked config eval { PVE::Tunnel::finish_tunnel($tunnel, 1); }; $self->log('warn', "cleanup of the target failed - $@") if $@; - die $version_err; + die $err; } print "websocket tunnel started\n"; @@ -375,6 +382,25 @@ sub prepare { return $running; } +# Lets the target of a remote migration take over the uid of the VM, before anything gets allocated +# or transferred there. The target refuses a uid that another guest there has, unless the migration +# allows a new uid. A VM without uid gets the one that the target reserved for it. +sub take_over_identity_on_target { + my ($self, $tunnel) = @_; + + my $identity = PVE::QemuConfig->get_identity($self->{vmconf}) or return; + my $params = { + uid => $identity->{uid}, + 'allow-new-uid' => $self->{opts}->{'allow-new-uid'} ? 1 : 0, + }; + + # the target waits for the cluster-wide guest identity lock + my $res = PVE::Tunnel::write_tunnel($tunnel, 60, 'identity', $params); + $self->log(PVE::GuestIdentity::remote_adopt_result_log($res->{identity})) if $res->{identity}; + + return; +} + sub scan_local_volumes { my ($self, $vmid) = @_; diff --git a/src/test/run_guest_identity_tests.pl b/src/test/run_guest_identity_tests.pl index 42ebec74..f755e1b6 100755 --- a/src/test/run_guest_identity_tests.pl +++ b/src/test/run_guest_identity_tests.pl @@ -8,6 +8,9 @@ use v5.36; use lib qw(.. .); use File::Temp qw(tempdir); +use IO::Handle; +use JSON; +use Socket; use Test::MockModule; use Test::More; @@ -15,6 +18,7 @@ use PVE::API2::Qemu; use PVE::JSONSchema; use PVE::QemuConfig; use PVE::QemuConfig::NoWrite; +use PVE::QemuMigrate; use PVE::QemuServer; use PVE::QemuServer::MetaInfo; use PVE::RPCEnvironment; @@ -331,4 +335,334 @@ subtest 'importovf' => sub { ); }; +# Runs the migration tunnel endpoint for $vmid with the commands @$commands from the source side +# and returns the decoded replies, the task output and the error of the task. +sub run_mtunnel($vmid, $commands) { + socketpair(my $client, my $server, AF_UNIX, SOCK_STREAM, PF_UNSPEC) + or die "socketpair failed - $!\n"; + $client->autoflush(1); + + print $client encode_json($_) . "\n" for $commands->@*; + shutdown($client, 1); + + my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment'); + $rpcenv_module->mock( + fork_worker => sub { + my ($self, $type, $id, $user, $code) = @_; + $code->('UPID:test'); + return 'UPID:test'; + }, + ); + my $access_control_module = Test::MockModule->new('PVE::AccessControl'); + $access_control_module->mock(assemble_tunnel_ticket => sub { return 'ticket'; }); + my $socket_module = Test::MockModule->new('IO::Socket::UNIX'); + $socket_module->mock(new => sub { return bless { conn => $server }, 'TestListenSocket'; }); + + my ($output, $err) = ('', ''); + { + open(my $out_fh, '>', \$output); + my $oldfh = select($out_fh); + eval { PVE::API2::Qemu->mtunnel({ node => 'localhost', vmid => $vmid }) }; + $err = $@; + select($oldfh); + close($out_fh); + } + close($server); + + my $replies = []; + while (defined(my $line = <$client>)) { + push $replies->@*, decode_json($line); + } + + return ($replies, $output, $err); +} + +package TestListenSocket { + + sub accept($self) { + return $self->{conn}; + } +} + +subtest 'remote migration, target side' => sub { + QemuServerTest::FakeCFS::reset(); + + my $cluster_module = Test::MockModule->new('PVE::Cluster'); + $cluster_module->mock(check_cfs_quorum => sub { return 1; }); + my $storage_module = Test::MockModule->new('PVE::Storage'); + $storage_module->mock(config => sub { return {}; }); + my $replication_config_module = Test::MockModule->new('PVE::ReplicationConfig'); + $replication_config_module->mock( + new => sub { return bless {}, 'PVE::ReplicationConfig'; }, + check_for_existing_jobs => sub { return; }, + ); + my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); + $qemu_server_module->mock(check_running => sub { return 0; }); + + my $source_conf = "cores: 2\nmemory: 1024\nmeta: creation-qemu=9.2.0,ctime=1600000000," + . "rtime=1600000000000,uid=$UID_A\nname: source\n"; + # like the source sends it + my $identity_cmd = { cmd => 'identity', uid => $UID_A, 'allow-new-uid' => 0 }; + my $config_cmd = { cmd => 'config', conf => $source_conf }; + my $quit_cmd = { cmd => 'quit', cleanup => 0 }; + my $cleanup_cmd = { cmd => 'quit', cleanup => 1 }; + + $QemuServerTest::FakeCFS::now_ms = 1800000000000; + my ($replies, $output, $err) = + run_mtunnel(100, [$identity_cmd, $config_cmd, $config_cmd, $identity_cmd, $quit_cmd]); + is($err, '', 'tunnel exits normally'); + like( + $replies->[3]->{msg}, + qr/the uid can only be taken over once, before the config arrives/, + 'identity command after the config fails', + ); + + ok($replies->[0]->{success}, 'identity command succeeds'); + is_deeply( + $replies->[0]->{identity}, + { result => 'adopted', uid => $UID_A, requested => $UID_A }, + 'identity command returns the adopted identity', + ); + like( + $output, + qr/^guest identity: adopted uid $UID_A from the source guest$/m, + 'target task logs it', + ); + my $writes = $QemuServerTest::FakeCFS::written->{100}; + ok( + $writes->[1] =~ m/^lock: create$/m && $writes->[1] =~ m/^meta: .*uid=$UID_A$/m, + 'uid is taken over right after the reservation', + ); + ok($replies->[1]->{success}, 'config command succeeds'); + ok(!exists($replies->[1]->{identity}), 'config command takes over nothing'); + ok(!$replies->[2]->{success}, 'repeated config command fails'); + is_deeply( + load_meta(100), + { + 'creation-qemu' => '9.2.0', + ctime => 1600000000, + rtime => 1800000000000, + uid => $UID_A, + }, + 'config write keeps the uid, other members come from the source', + ); + ok($replies->[4]->{success}, 'tunnel quits'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::other_guests->{300} = "rtime=1,uid=$UID_A"; + ($replies, $output, $err) = run_mtunnel(101, [$identity_cmd, $cleanup_cmd]); + is($err, '', 'uid in use: tunnel exits normally'); + ok(!$replies->[0]->{success}, 'uid in use: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - guest 300 in the target cluster already has the" + . " uid '$UID_A' of this guest - use 'allow-new-uid' to migrate it with a new uid\n", + 'uid in use: error explains the refusal', + ); + is( + scalar($QemuServerTest::FakeCFS::written->{101}->@*), + 1, + 'uid in use: nothing written besides the reservation', + ); + ok($replies->[1]->{success}, 'uid in use: cleanup on quit succeeds'); + ok(!defined(configs()->{101}), 'uid in use: reservation removed by the cleanup'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::other_guests->{300} = "rtime=1,uid=$UID_A"; + ($replies, $output, $err) = run_mtunnel( + 101, [{ $identity_cmd->%*, 'allow-new-uid' => 1 }, $config_cmd, $quit_cmd], + ); + is($err, '', 'uid in use, allow-new-uid: tunnel exits normally'); + is_deeply( + $replies->[0]->{identity}, + { + result => 'in-use', + uid => generated_uid(1), + requested => $UID_A, + 'in-use-by' => 300, + }, + 'uid in use, allow-new-uid: result is returned', + ); + my $kept = generated_uid(1); + like( + $output, + qr/^guest identity: source guest uid $UID_A is in use by guest 300, keeping uid $kept$/m, + 'uid in use, allow-new-uid: target task logs it', + ); + ok($replies->[1]->{success}, 'uid in use, allow-new-uid: config command succeeds'); + is(stored_identity(101)->{uid}, $kept, 'uid in use, allow-new-uid: new uid stays'); + + QemuServerTest::FakeCFS::reset(); + ($replies) = run_mtunnel(102, [$config_cmd, $quit_cmd]); + ok($replies->[0]->{success}, 'source without identity command: config command succeeds'); + is( + stored_identity(102)->{uid}, + generated_uid(1), + 'source without identity command: new uid, not the one of the config', + ); + + QemuServerTest::FakeCFS::reset(); + my $config_without_uid = + { cmd => 'config', conf => $source_conf =~ s/,rtime=\d+,uid=\w+//r }; + ($replies) = run_mtunnel(102, [$config_without_uid, $quit_cmd]); + ok($replies->[0]->{success}, 'source without uid: config command succeeds'); + is(stored_identity(102)->{uid}, generated_uid(1), 'source without uid: new uid'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::lock_domain_error = + "cfs-lock 'domain-guest-identity' error: no quorum!\n"; + ($replies, $output, $err) = run_mtunnel(103, [$identity_cmd, $cleanup_cmd]); + is($err, '', 'adopting fails: tunnel exits normally'); + ok(!$replies->[0]->{success}, 'adopting fails: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - unable to adopt the uid of the source guest" + . " - cfs-lock 'domain-guest-identity' error: no quorum!\n", + 'adopting fails: error explains the failure', + ); + ok($replies->[1]->{success}, 'adopting fails: cleanup on quit succeeds'); + ok(!defined(configs()->{103}), 'adopting fails: reservation removed by the cleanup'); +}; + +subtest 'remote migration, source side' => sub { + QemuServerTest::FakeCFS::reset(); + + # the source reads the firewall config of the VM from /etc/pve, use a VMID the host lacks + my $vmid = 999999999; + my $source_conf = "cores: 2\nmemory: 1024\nmeta: rtime=1600000000000,uid=$UID_A\n"; + configs()->{$vmid} = $source_conf; + + my ($commands, $reply, $cleanup, $log) = ([], undef, undef, ''); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + write_tunnel => sub { + my ($tunnel, $timeout, $command, $params) = @_; + push $commands->@*, { command => $command, timeout => $timeout, params => $params }; + die $reply if !ref($reply); + return $reply; + }, + finish_tunnel => sub { + (undef, $cleanup) = @_; + }, + ); + my $migrate_module = Test::MockModule->new('PVE::QemuMigrate'); + $migrate_module->mock(fork_websocket_tunnel => sub { return { version => 3, age => 1 }; }); + my $abstract_migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $abstract_migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n"; }); + $QemuServerTest::FakeCFS::qemu_config_module->mock(cleanup_fleecing_images => sub { }); + my $helpers_module = Test::MockModule->new('PVE::QemuMigrate::Helpers'); + $helpers_module->mock(check_local_resources => sub { return ([], {}, {}); }); + my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); + $qemu_server_module->mock(check_running => sub { return 0; }); + my $storage_module = Test::MockModule->new('PVE::Storage'); + $storage_module->mock(config => sub { return {}; }); + my $replication_config_module = Test::MockModule->new('PVE::ReplicationConfig'); + $replication_config_module->mock( + new => sub { return bless {}, 'PVE::ReplicationConfig'; }, + find_local_replication_job => sub { return; }, + check_for_existing_jobs => sub { return; }, + ); + + my $new_migration = sub($opts = {}) { + return bless { + opts => { remote => { vmid => 300 }, $opts->%* }, + node => 'remote-node', + vmid => $vmid, + local_volumes => {}, + volume_map => {}, + storecfg => {}, + }, + 'PVE::QemuMigrate'; + }; + my $prepare = sub($self) { + ($commands, $cleanup, $log) = ([], undef, ''); + my $output = ''; + open(my $out_fh, '>', \$output); + my $oldfh = select($out_fh); + eval { $self->prepare($vmid) }; + my $err = $@; + select($oldfh); + close($out_fh); + return $err; + }; + + $reply = + { success => 1, identity => { result => 'adopted', uid => $UID_A, requested => $UID_A } }; + my $self = $new_migration->(); + is($prepare->($self), '', 'uid free on the target: prepare succeeds'); + is_deeply( + $commands, + [{ + command => 'identity', + timeout => 60, + params => { uid => $UID_A, 'allow-new-uid' => 0 }, + }], + 'uid free on the target: only the uid is sent before anything else', + ); + like( + $log, + qr/^info: guest identity: adopted uid $UID_A from the source guest$/m, + 'uid free on the target: source task logs the outcome', + ); + ok($self->{tunnel}, 'uid free on the target: tunnel is kept for the migration'); + ok(!defined($cleanup), 'uid free on the target: no cleanup'); + + $reply = { + success => 1, + identity => + { result => 'in-use', uid => $UID_B, requested => $UID_A, 'in-use-by' => 300 }, + }; + $self = $new_migration->({ 'allow-new-uid' => 1 }); + is($prepare->($self), '', 'allow-new-uid: prepare succeeds'); + is($commands->[0]->{params}->{'allow-new-uid'}, 1, 'allow-new-uid: option is sent'); + my $in_use = + "guest identity: source guest uid $UID_A is in use by guest 300, keeping uid $UID_B"; + like($log, qr/^info: \Q$in_use\E$/m, 'allow-new-uid: source task logs the new uid'); + + $reply = + "error - tunnel command 'identity' failed - failed to handle 'identity' command -" + . " guest 300 in the target cluster already has the uid '$UID_A' of this guest - use" + . " 'allow-new-uid' to migrate it with a new uid\n"; + $self = $new_migration->(); + is($prepare->($self), $reply, 'uid in use: prepare fails with the error of the target'); + is(scalar($commands->@*), 1, 'uid in use: nothing else is sent'); + is($cleanup, 1, 'uid in use: target is asked to clean up'); + ok(!$self->{tunnel}, 'uid in use: tunnel is not kept'); + + # a target with a newer version has accepted the uid at this point + $reply = { success => 1, identity => { result => 'future', uid => $UID_A } }; + $self = $new_migration->(); + is($prepare->($self), '', 'unknown outcome does not fail the migration'); + like($log, qr/^warn: unknown result of guest identity adoption 'future'$/m, 'it is logged'); + + configs()->{$vmid} = "cores: 2\nmemory: 1024\n"; + $self = $new_migration->(); + is($prepare->($self), '', 'source without uid: prepare succeeds'); + is_deeply($commands, [], 'source without uid: no identity command'); + + configs()->{$vmid} = $source_conf; + $reply = { success => 1 }; + $self = $new_migration->(); + $self->{tunnel} = {}; + ($commands, $log) = ([], ''); + my $output = ''; + { + open(my $out_fh, '>', \$output); + my $oldfh = select($out_fh); + eval { $self->phase1_remote($vmid) }; + is($@, '', 'config sent'); + select($oldfh); + close($out_fh); + } + is($commands->[0]->{command}, 'config', 'config command'); + like( + $commands->[0]->{params}->{conf}, + qr/^meta: rtime=1600000000000,uid=$UID_A$/m, + 'source config sent with its uid', + ); + unlike($log, qr/guest identity/, 'config reply has no identity to log'); + + $QemuServerTest::FakeCFS::qemu_config_module->unmock('cleanup_fleecing_images'); +}; + done_testing(); -- 2.47.3