From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 311E11FF0DE for ; Thu, 19 Nov 2026 11:39:34 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3D24E219AD; Thu, 19 Nov 2026 11:36:59 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v4 18/41] api: remote migration: take over the guest uid from the source Date: Thu, 19 Nov 2026 11:34:04 +0100 Message-ID: <20261119103551.1666964-19-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119103551.1666964-1-t.lamprecht@proxmox.com> References: <20261119103551.1666964-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795084556309 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.159 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FMBUYIGSZUUOQGDTLJJ6SHJBGYRFNGVR X-Message-ID-Hash: FMBUYIGSZUUOQGDTLJJ6SHJBGYRFNGVR X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A VM migrated to another cluster should keep its uid. Send the uid to the target right after the tunnel version check, before anything is allocated or transferred there. The target takes the uid over for the config that reserves the VMID, through the cluster-wide check of pve-guest-common. The later config writes of the migration keep it. The target accepts the uid only once and only before the config arrives. The other members of 'meta', including the creation information, come from the source as before. The command schema, the target handler and the sender are shared with containers through pve-guest-common. If another guest in the target cluster already has the uid, refuse the migration. A migration moves a guest. If the migrated VM silently got a new uid, the uid that tools know for it would lead them to the other guest. The new option 'allow-new-uid' lets the migration go on with a new uid. Any other failure to take over the uid, for example because the cluster-wide lock cannot be taken, stops the migration at that point as well. The source then asks the target to clean up, as it does on a tunnel version mismatch. Report the result in the task logs of both sides. If a newer target reports a result that the source does not know, the source only logs a warning. A source with the previous tunnel version sends no uid, so the VM keeps the uid that it got when the target reserved its VMID. A source with this version gives a VM without a uid one before it sends the uid. Bump the tunnel version. Older targets then get refused before disks are allocated or transferred, and not only when their strict config parser rejects the identity. Keep accepting the previous tunnel version from older sources. Signed-off-by: Thomas Lamprecht Reviewed-by: Dominik Csapak Tested-by: Dominik Csapak --- changes v3 -> v4: * 'allow-new-uid' is the standard option of pve-guest-common, and the sender is called adopt_identity_on_target * the source logs the result in its own words, see the pve-guest-common patch * tests run the tunnel endpoint through the helper of an independent test commit before the series, with a socket path of their own * commit message: which side assigns a missing uid * commit message: plainer wording, same content changes v2 -> v3: * use the shared helpers of pve-guest-common for the identity command * a VM without uid gets one before the source sends it, and a stored uid is sent even if the parser dropped 'meta'; before, source and target got unrelated uids * tests for targets with an older and a newer tunnel version, and for the version the target reports changes v1 -> v2: * take the uid over at the start of the migration, through a new tunnel command, instead of with the config * refuse the migration if another guest in the target cluster has the uid, unless the new option 'allow-new-uid' is set (Dominik) src/PVE/API2/Qemu.pm | 13 +- src/PVE/CLI/qm.pm | 1 + src/PVE/QemuMigrate.pm | 18 +- src/test/run_guest_identity_tests.pl | 324 +++++++++++++++++++++++++++ 4 files changed, 348 insertions(+), 8 deletions(-) diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index ea85a5da..87185c3b 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -5991,6 +5991,7 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => get_standard_option('pve-guest-allow-new-uid'), }, }, returns => { @@ -7030,7 +7031,7 @@ __PACKAGE__->register_method({ PVE::Cluster::check_cfs_quorum(); my $lock = 'create'; - eval { PVE::QemuConfig->create_and_lock_config($vmid, 0, $lock); }; + my $reservation = eval { PVE::QemuConfig->create_and_lock_config($vmid, 0, $lock); }; raise_param_exc({ vmid => "unable to create empty VM config - $@" }) if $@; @@ -7060,6 +7061,7 @@ __PACKAGE__->register_method({ }; my $cmd_desc = { + identity => PVE::GuestIdentity::tunnel_identity_cmd_schema(), config => { conf => { type => 'string', @@ -7184,9 +7186,16 @@ __PACKAGE__->register_method({ # bump/bump for opt-in changes return { api => $PVE::QemuMigrate::WS_TUNNEL_VERSION, - age => 0, + age => 1, }; }, + 'identity' => sub { + my ($params) = @_; + + return PVE::GuestIdentity::handle_tunnel_identity( + 'PVE::QemuConfig', $state, $reservation, $params, + ); + }, 'config' => sub { my ($params) = @_; diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm index 3e4cb13b..9319aea5 100755 --- a/src/PVE/CLI/qm.pm +++ b/src/PVE/CLI/qm.pm @@ -226,6 +226,7 @@ __PACKAGE__->register_method({ minimum => '0', default => 'migrate limit from datacenter or storage config', }, + 'allow-new-uid' => get_standard_option('pve-guest-allow-new-uid'), }, }, returns => { diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm index 2d362140..d574f0c4 100644 --- a/src/PVE/QemuMigrate.pm +++ b/src/PVE/QemuMigrate.pm @@ -47,7 +47,8 @@ use PVE::AbstractMigrate; use base qw(PVE::AbstractMigrate); # compared against remote end's minimum version -our $WS_TUNNEL_VERSION = 2; +# 3: the 'identity' command, and 'meta' with the guest identity in the config +our $WS_TUNNEL_VERSION = 3; sub fork_tunnel { my ($self, $ssh_forward_info) = @_; @@ -350,17 +351,22 @@ sub prepare { $self->log('info', "local WS tunnel version: $WS_TUNNEL_VERSION"); $self->log('info', "remote WS tunnel version: $tunnel->{version}"); $self->log('info', "minimum required WS tunnel version: $min_version"); - my $version_err; - $version_err = "Remote tunnel endpoint not compatible, upgrade required\n" + my $err; + $err = "Remote tunnel endpoint not compatible, upgrade required\n" if $WS_TUNNEL_VERSION < $min_version; - $version_err = "Remote tunnel endpoint too old, upgrade required\n" + $err = "Remote tunnel endpoint too old, upgrade required\n" if $WS_TUNNEL_VERSION > $tunnel->{version}; - if ($version_err) { + if (!$err) { + eval { $self->adopt_identity_on_target('PVE::QemuConfig', $tunnel) }; + $err = $@; + } + + if ($err) { # the target already reserved the VMID with a locked config eval { PVE::Tunnel::finish_tunnel($tunnel, 1); }; $self->log('warn', "cleanup of the target failed - $@") if $@; - die $version_err; + die $err; } print "websocket tunnel started\n"; diff --git a/src/test/run_guest_identity_tests.pl b/src/test/run_guest_identity_tests.pl index 66bb8c30..60f16be1 100755 --- a/src/test/run_guest_identity_tests.pl +++ b/src/test/run_guest_identity_tests.pl @@ -16,12 +16,14 @@ use PVE::RPCEnvironment; use PVE::QemuConfig; use PVE::QemuConfig::NoWrite; +use PVE::QemuMigrate; use PVE::QemuServer; use PVE::QemuServer::MetaInfo; use PVE::API2::Qemu; use QemuServerTest::FakeCFS; +use QemuServerTest::MTunnel qw(run_mtunnel); use QemuServerTest::Utils qw(capture_output); my $UID_A = 'abcdefghijklmnopqrst'; @@ -287,4 +289,326 @@ subtest 'importovf' => sub { ); }; +subtest 'remote migration, target side' => sub { + QemuServerTest::FakeCFS::reset(); + + my $cluster_module = Test::MockModule->new('PVE::Cluster'); + $cluster_module->mock(check_cfs_quorum => sub { return 1; }); + my $storage_module = Test::MockModule->new('PVE::Storage'); + $storage_module->mock(config => sub { return {}; }); + my $replication_config_module = Test::MockModule->new('PVE::ReplicationConfig'); + $replication_config_module->mock( + new => sub { return bless {}, 'PVE::ReplicationConfig'; }, + check_for_existing_jobs => sub { return; }, + ); + my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); + $qemu_server_module->mock(check_running => sub { return 0; }); + + my $source_conf = "cores: 2\nmemory: 1024\nmeta: creation-qemu=9.2.0,ctime=1600000000," + . "rtime=1600000000000,uid=$UID_A\nname: source\n"; + # like the source sends it + my $identity_cmd = { cmd => 'identity', uid => $UID_A, 'allow-new-uid' => 0 }; + my $config_cmd = { cmd => 'config', conf => $source_conf }; + my $quit_cmd = { cmd => 'quit', cleanup => 0 }; + my $cleanup_cmd = { cmd => 'quit', cleanup => 1 }; + + $QemuServerTest::FakeCFS::now_ms = 1800000000000; + my ($replies, $output, $err) = + run_mtunnel(100, [$identity_cmd, $config_cmd, $config_cmd, $identity_cmd, $quit_cmd]); + is($err, '', 'tunnel exits normally'); + like( + $replies->[3]->{msg}, + qr/the uid can only be adopted once, before the config arrives/, + 'identity command after the config fails', + ); + + ok($replies->[0]->{success}, 'identity command succeeds'); + is_deeply( + $replies->[0]->{identity}, + { result => 'adopted', uid => $UID_A, requested => $UID_A }, + 'identity command returns the adopted identity', + ); + like( + $output, + qr/^guest identity: adopted uid $UID_A from the source guest$/m, + 'target task logs it', + ); + my $writes = $QemuServerTest::FakeCFS::written->{100}; + ok( + $writes->[1] =~ m/^lock: create$/m && $writes->[1] =~ m/^meta: .*uid=$UID_A$/m, + 'uid is taken over right after the reservation', + ); + ok($replies->[1]->{success}, 'config command succeeds'); + ok(!exists($replies->[1]->{identity}), 'config command takes over nothing'); + ok(!$replies->[2]->{success}, 'repeated config command fails'); + is_deeply( + load_meta(100), + { + 'creation-qemu' => '9.2.0', + ctime => 1600000000, + rtime => 1800000000000, + uid => $UID_A, + }, + 'config write keeps the uid, other members come from the source', + ); + ok($replies->[4]->{success}, 'tunnel quits'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::other_guests->{300} = "rtime=1,uid=$UID_A"; + ($replies, $output, $err) = run_mtunnel(101, [$identity_cmd, $cleanup_cmd]); + is($err, '', 'uid in use: tunnel exits normally'); + ok(!$replies->[0]->{success}, 'uid in use: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - guest 300 in the target cluster already has the" + . " uid '$UID_A' of this guest - use 'allow-new-uid' to migrate it with a new uid\n", + 'uid in use: error explains the refusal', + ); + is( + scalar($QemuServerTest::FakeCFS::written->{101}->@*), + 1, + 'uid in use: nothing written besides the reservation', + ); + ok($replies->[1]->{success}, 'uid in use: cleanup on quit succeeds'); + ok(!defined(configs()->{101}), 'uid in use: reservation removed by the cleanup'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::other_guests->{300} = "rtime=1,uid=$UID_A"; + ($replies, $output, $err) = run_mtunnel( + 101, [{ $identity_cmd->%*, 'allow-new-uid' => 1 }, $config_cmd, $quit_cmd], + ); + is($err, '', 'uid in use, allow-new-uid: tunnel exits normally'); + is_deeply( + $replies->[0]->{identity}, + { + result => 'in-use', + uid => generated_uid(1), + requested => $UID_A, + 'in-use-by' => 300, + }, + 'uid in use, allow-new-uid: result is returned', + ); + my $kept = generated_uid(1); + like( + $output, + qr/^guest identity: source guest uid $UID_A is in use by guest 300, keeping uid $kept$/m, + 'uid in use, allow-new-uid: target task logs it', + ); + ok($replies->[1]->{success}, 'uid in use, allow-new-uid: config command succeeds'); + is(stored_identity(101)->{uid}, $kept, 'uid in use, allow-new-uid: new uid stays'); + + QemuServerTest::FakeCFS::reset(); + ($replies) = run_mtunnel(105, [{ cmd => 'version' }, $cleanup_cmd]); + is($replies->[0]->{api}, $PVE::QemuMigrate::WS_TUNNEL_VERSION, 'version: tunnel version'); + is($replies->[0]->{api} - $replies->[0]->{age}, 2, 'version: version 2 accepted'); + + # like a source with tunnel version 2 + QemuServerTest::FakeCFS::reset(); + ($replies) = run_mtunnel(102, [$config_cmd, $quit_cmd]); + ok($replies->[0]->{success}, 'source without identity command: config command succeeds'); + is( + stored_identity(102)->{uid}, + generated_uid(1), + 'source without identity command: new uid, not the one of the config', + ); + + QemuServerTest::FakeCFS::reset(); + my $config_without_uid = + { cmd => 'config', conf => $source_conf =~ s/,rtime=\d+,uid=\w+//r }; + ($replies) = run_mtunnel(102, [$config_without_uid, $quit_cmd]); + ok($replies->[0]->{success}, 'source without uid: config command succeeds'); + is(stored_identity(102)->{uid}, generated_uid(1), 'source without uid: new uid'); + + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::lock_domain_error = + "cfs-lock 'domain-guest-identity' error: no quorum!\n"; + ($replies, $output, $err) = run_mtunnel(103, [$identity_cmd, $cleanup_cmd]); + is($err, '', 'adopting fails: tunnel exits normally'); + ok(!$replies->[0]->{success}, 'adopting fails: identity command fails'); + is( + $replies->[0]->{msg}, + "failed to handle 'identity' command - unable to adopt the uid of the source guest" + . " - cfs-lock 'domain-guest-identity' error: no quorum!\n", + 'adopting fails: error explains the failure', + ); + ok($replies->[1]->{success}, 'adopting fails: cleanup on quit succeeds'); + ok(!defined(configs()->{103}), 'adopting fails: reservation removed by the cleanup'); +}; + +subtest 'remote migration, source side' => sub { + QemuServerTest::FakeCFS::reset(); + + # the source reads the firewall config of the VM from /etc/pve, use a VMID the host lacks + my $vmid = 999999999; + my $source_conf = "cores: 2\nmemory: 1024\nmeta: rtime=1600000000000,uid=$UID_A\n"; + configs()->{$vmid} = $source_conf; + + my ($commands, $reply, $cleanup, $log) = ([], undef, undef, ''); + my $tunnel_module = Test::MockModule->new('PVE::Tunnel'); + $tunnel_module->mock( + write_tunnel => sub { + my ($tunnel, $timeout, $command, $params) = @_; + push $commands->@*, { command => $command, timeout => $timeout, params => $params }; + die $reply if !ref($reply); + return $reply; + }, + finish_tunnel => sub { + (undef, $cleanup) = @_; + }, + ); + my $migrate_module = Test::MockModule->new('PVE::QemuMigrate'); + $migrate_module->mock(fork_websocket_tunnel => sub { return { version => 3, age => 1 }; }); + my $abstract_migrate_module = Test::MockModule->new('PVE::AbstractMigrate'); + $abstract_migrate_module->mock(log => sub { $log .= "$_[1]: $_[2]\n"; }); + $QemuServerTest::FakeCFS::qemu_config_module->mock(cleanup_fleecing_images => sub { }); + my $helpers_module = Test::MockModule->new('PVE::QemuMigrate::Helpers'); + $helpers_module->mock(check_local_resources => sub { return ([], {}, {}); }); + my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); + $qemu_server_module->mock(check_running => sub { return 0; }); + my $storage_module = Test::MockModule->new('PVE::Storage'); + $storage_module->mock(config => sub { return {}; }); + my $replication_config_module = Test::MockModule->new('PVE::ReplicationConfig'); + $replication_config_module->mock( + new => sub { return bless {}, 'PVE::ReplicationConfig'; }, + find_local_replication_job => sub { return; }, + check_for_existing_jobs => sub { return; }, + ); + + my $new_migration = sub($opts = {}) { + return bless { + opts => { remote => { vmid => 300 }, $opts->%* }, + node => 'remote-node', + vmid => $vmid, + local_volumes => {}, + volume_map => {}, + storecfg => {}, + }, + 'PVE::QemuMigrate'; + }; + my $prepare = sub($self) { + ($commands, $cleanup, $log) = ([], undef, ''); + my $output = ''; + open(my $out_fh, '>', \$output); + my $oldfh = select($out_fh); + eval { $self->prepare($vmid) }; + my $err = $@; + select($oldfh); + close($out_fh); + return $err; + }; + + $reply = + { success => 1, identity => { result => 'adopted', uid => $UID_A, requested => $UID_A } }; + my $self = $new_migration->(); + is($prepare->($self), '', 'uid free on the target: prepare succeeds'); + is_deeply( + $commands, + [{ + command => 'identity', + timeout => 60, + params => { uid => $UID_A, 'allow-new-uid' => 0 }, + }], + 'uid free on the target: only the uid is sent before anything else', + ); + like( + $log, + qr/^info: guest identity: the migrated guest keeps uid $UID_A$/m, + 'uid free on the target: source task logs the outcome', + ); + ok($self->{tunnel}, 'uid free on the target: tunnel is kept for the migration'); + ok(!defined($cleanup), 'uid free on the target: no cleanup'); + + $reply = { + success => 1, + identity => + { result => 'in-use', uid => $UID_B, requested => $UID_A, 'in-use-by' => 300 }, + }; + $self = $new_migration->({ 'allow-new-uid' => 1 }); + is($prepare->($self), '', 'allow-new-uid: prepare succeeds'); + is($commands->[0]->{params}->{'allow-new-uid'}, 1, 'allow-new-uid: option is sent'); + my $in_use = "guest identity: guest 300 in the target cluster has uid $UID_A, the migrated" + . " guest gets the new uid $UID_B"; + like($log, qr/^info: \Q$in_use\E$/m, 'allow-new-uid: source task logs the new uid'); + + $reply = + "error - tunnel command 'identity' failed - failed to handle 'identity' command -" + . " guest 300 in the target cluster already has the uid '$UID_A' of this guest - use" + . " 'allow-new-uid' to migrate it with a new uid\n"; + $self = $new_migration->(); + is($prepare->($self), $reply, 'uid in use: prepare fails with the error of the target'); + is(scalar($commands->@*), 1, 'uid in use: nothing else is sent'); + is($cleanup, 1, 'uid in use: target is asked to clean up'); + ok(!$self->{tunnel}, 'uid in use: tunnel is not kept'); + + # a target with a newer version has accepted the uid at this point + $reply = { success => 1, identity => { result => 'future', uid => $UID_A } }; + $self = $new_migration->(); + is($prepare->($self), '', 'unknown outcome does not fail the migration'); + like( + $log, + qr/^warn: guest identity: unknown result 'future' of the target$/m, + 'it is logged', + ); + + my $without_uid = "cores: 2\nmemory: 1024\n"; + configs()->{$vmid} = $without_uid; + for my $case ( + [{ version => 2, age => 0 }, "Remote tunnel endpoint too old, upgrade required\n"], + [ + { version => 4, age => 0 }, + "Remote tunnel endpoint not compatible, upgrade required\n", + ], + ) { + my ($remote, $expected) = $case->@*; + my $what = "target with tunnel version $remote->{version}"; + $migrate_module->mock(fork_websocket_tunnel => sub { return $remote; }); + $self = $new_migration->(); + is($prepare->($self), $expected, "$what: refused"); + is_deeply($commands, [], "$what: nothing is sent"); + is($cleanup, 1, "$what: target is asked to clean up"); + is(configs()->{$vmid}, $without_uid, "$what: source config is unchanged"); + } + $migrate_module->mock(fork_websocket_tunnel => sub { return { version => 3, age => 1 }; }); + + $self = $new_migration->(); + is($prepare->($self), '', 'source without uid: prepare succeeds'); + my $assigned = stored_identity($vmid)->{uid}; + is($assigned, generated_uid(1), 'source without uid: gets one first'); + is($commands->[0]->{params}->{uid}, $assigned, 'source without uid: and sends it'); + + # a member of a newer version makes the parser drop the whole property + configs()->{$vmid} = + "cores: 2\nmemory: 1024\nmeta: future=1,rtime=1600000000000,uid=$UID_A\n"; + $self = $new_migration->(); + { + local $SIG{__WARN__} = sub { }; + is($prepare->($self), '', 'meta the parser drops: prepare succeeds'); + } + is($commands->[0]->{params}->{uid}, $UID_A, 'meta the parser drops: stored uid is sent'); + + configs()->{$vmid} = $source_conf; + $reply = { success => 1 }; + $self = $new_migration->(); + $self->{tunnel} = {}; + ($commands, $log) = ([], ''); + my $output = ''; + { + open(my $out_fh, '>', \$output); + my $oldfh = select($out_fh); + eval { $self->phase1_remote($vmid) }; + is($@, '', 'config sent'); + select($oldfh); + close($out_fh); + } + is($commands->[0]->{command}, 'config', 'config command'); + like( + $commands->[0]->{params}->{conf}, + qr/^meta: rtime=1600000000000,uid=$UID_A$/m, + 'source config sent with its uid', + ); + unlike($log, qr/guest identity/, 'config reply has no identity to log'); + + $QemuServerTest::FakeCFS::qemu_config_module->unmock('cleanup_fleecing_images'); +}; + done_testing(); -- 2.47.3