From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 8BF6B1FF0DE for ; Thu, 19 Nov 2026 11:37:57 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 028E02184F; Thu, 19 Nov 2026 11:36:51 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH guest-common v4 12/41] migrate: add shared helpers for the identity command of the migration tunnel Date: Thu, 19 Nov 2026 11:33:58 +0100 Message-ID: <20261119103551.1666964-13-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119103551.1666964-1-t.lamprecht@proxmox.com> References: <20261119103551.1666964-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795084555941 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.161 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5KUZQQUCKR6KS56Q2FCUGE7OTFZBYRV7 X-Message-ID-Hash: 5KUZQQUCKR6KS56Q2FCUGE7OTFZBYRV7 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: In a remote migration, the source sends the uid of the guest to the target with the 'identity' tunnel command. Without shared code, VMs and containers would each define three things: the command, the rule that the target accepts it only once and only before the config arrives, and the sender with its timeout and log line. All of these would have to be kept in sync by hand with every change of the tunnel. Add the parameter schema and the handler of the target to PVE::GuestIdentity, as PVE::StorageTunnel does for the storage commands. Add the sender to PVE::AbstractMigrate, next to the sender of the 'bwlimit' command. The target adopts the uid right after it reserved the VMID, before anything is transferred. If another guest in the target cluster has the uid, the handler refuses the migration, unless the caller allows a new uid for the migrated guest. That override is a standard option, so that the migration commands of both guest types describe it the same way. The reply of the target is about the guest on the target. The guest on the source keeps its uid in any case, so the source logs the reply in its own words. If a newer target reports a result that the source does not know, the source only logs a warning. If the parsed config of the source has no identity, the sender writes the config first. A guest without a uid gets one with that write. This way, the source keeps the uid that it sends and does not get another one with a later write. The write also brings back a stored uid that the parser of the guest type dropped, for example because of a 'meta' member that it does not know. Callers send the command only after the version check of the tunnel. A target that this check refuses then leaves the source config unchanged. Signed-off-by: Thomas Lamprecht --- changes v3 -> v4: * the source logs the reply in its own words (the migrated guest keeps its uid, or gets a new one), not in those of the target, which are about the guest there * the handler adopts the uid and refuses one in use itself, without adopt_from_source of the previous patch * 'allow-new-uid' is a standard option, for the command schema and for the migration commands of both guest types * the sender is called adopt_identity_on_target * comment and commit message: only a target refused by the tunnel version check leaves the source config unchanged * tests: the log lines of both sides, and a failure to take the identity lock on the target * commit message: plainer wording, same content changes v2 -> v3: * new in v3: command schema, target handler and sender moved here from qemu-server and container * the sender gives a guest without uid one before it sends it, and sends the stored uid if the parser dropped 'meta' src/PVE/AbstractMigrate.pm | 42 ++++++ src/PVE/GuestIdentity.pm | 51 +++++++ src/tests/config-identity-tests.pl | 229 +++++++++++++++++++++++++++++ 3 files changed, 322 insertions(+) diff --git a/src/PVE/AbstractMigrate.pm b/src/PVE/AbstractMigrate.pm index 5d74f53..3c67a50 100644 --- a/src/PVE/AbstractMigrate.pm +++ b/src/PVE/AbstractMigrate.pm @@ -443,6 +443,48 @@ sub get_bwlimit { return $bwlimit; } +# Lets the target of a remote migration adopt the uid of the guest, whose config class is +# $config_class, through the tunnel $tunnel, before anything gets transferred there. The target +# refuses a uid that another guest there has, unless the migration allows a new uid, see +# PVE::GuestIdentity::handle_tunnel_identity. Call it only after the version check of the tunnel, +# so that a target refused by that check leaves the config of the source unchanged. +sub adopt_identity_on_target { + my ($self, $config_class, $tunnel) = @_; + + my $conf = $self->{vmconf}; + my $identity = $config_class->get_identity($conf); + if (!$identity) { + # A guest without uid gets one first, so the source keeps the uid that it sends. The write + # also restores a stored uid that the parser of the guest type dropped. + $config_class->write_config($self->{vmid}, $conf); + $identity = $config_class->get_identity($conf) or return; + } + my $uid = $identity->{uid}; + + my $params = { uid => $uid, 'allow-new-uid' => $self->{opts}->{'allow-new-uid'} ? 1 : 0 }; + + # the target waits for the cluster-wide guest identity lock + my $res = PVE::Tunnel::write_tunnel($tunnel, 60, 'identity', $params); + my $reply = $res->{identity} or return; + + # the guest here keeps its uid in any case, the result is about the guest on the target + my $result = $reply->{result} // ''; + if ($result eq 'adopted' || $result eq 'kept') { + $self->log('info', "guest identity: the migrated guest keeps uid $uid"); + } elsif ($result eq 'in-use') { + $self->log( + 'info', + "guest identity: guest $reply->{'in-use-by'} in the target cluster has uid $uid, the" + . " migrated guest gets the new uid $reply->{uid}", + ); + } else { + # from a newer target, which accepted the uid by then + $self->log('warn', "guest identity: unknown result '$result' of the target"); + } + + return; +} + sub map_storage { my ($self, $scfg, $storeid) = @_; diff --git a/src/PVE/GuestIdentity.pm b/src/PVE/GuestIdentity.pm index 7763ac7..531a7b7 100644 --- a/src/PVE/GuestIdentity.pm +++ b/src/PVE/GuestIdentity.pm @@ -387,6 +387,57 @@ sub adopt_from_backup($class, $vmid, $uid, $reservation) { return; } +PVE::JSONSchema::register_standard_option( + 'pve-guest-allow-new-uid', + { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster already has the" + . " uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, +); + +# Returns the properties of the parameters of the 'identity' command of the migration tunnel, see +# handle_tunnel_identity. Every call returns a new copy. +sub tunnel_identity_cmd_schema() { + return { + uid => { + type => 'string', + description => 'uid of the source guest', + }, + 'allow-new-uid' => PVE::JSONSchema::get_standard_option('pve-guest-allow-new-uid'), + }; +} + +# Handles the 'identity' command with the parameters $params on the target of a remote migration of +# a guest of the config class $class: adopts the uid of the source guest for the guest that the +# tunnel reserved with $reservation, logs the result, and returns the reply for the source. $state +# is the state of the tunnel, with the VMID in 'vmid' and the lock of the guest in 'lock'. +# +# The command is only accepted once, and only before the config arrives, while the guest still has +# the 'create' lock of the reservation. If another guest in the cluster has the uid, the command +# fails, which refuses the migration, unless the parameter 'allow-new-uid' is set, in which case the +# guest keeps the uid of its reservation. +sub handle_tunnel_identity($class, $state, $reservation, $params) { + die "the uid can only be adopted once, before the config arrives\n" + if ($state->{lock} // '') ne 'create' || $state->{'identity-handled'}; + $state->{'identity-handled'} = 1; + + my $uid = $params->{uid}; + my $res = eval { adopt($class, $state->{vmid}, $uid, $reservation) }; + die "unable to adopt the uid of the source guest - $@" if $@; + + if ($res->{result} eq 'in-use' && !$params->{'allow-new-uid'}) { + die "guest $res->{'in-use-by'} in the target cluster already has the uid '$uid' of this" + . " guest - use 'allow-new-uid' to migrate it with a new uid\n"; + } + + print adopt_result_message($res, 'source guest') . "\n"; + + return { identity => $res }; +} + # Returns undef if $node runs a version of Proxmox VE that keeps the identity when writing guest # configs, see MIN_PVE_MANAGER_VERSION. Missing or invalid version information counts as too old. # Otherwise returns a hash with 'required-version', the minimum version, and 'version', the version diff --git a/src/tests/config-identity-tests.pl b/src/tests/config-identity-tests.pl index 8172fc4..df01f53 100755 --- a/src/tests/config-identity-tests.pl +++ b/src/tests/config-identity-tests.pl @@ -11,8 +11,10 @@ use Test::MockModule; use Test::More; use PVE::AbstractConfig; +use PVE::AbstractMigrate; use PVE::GuestHelpers; use PVE::GuestIdentity; +use PVE::JSONSchema; my $tmpdir = tempdir('config-identity-XXXXXX', DIR => '.', CLEANUP => 1); $PVE::GuestHelpers::lockdir = "$tmpdir/lock"; @@ -1126,4 +1128,231 @@ subtest 'drop pending meta' => sub { } }; +# Runs $code with the output to STDOUT captured, returns the error and the output. +my sub capture_output($code) { + my $output = ''; + open(my $fh, '>', \$output) or die "unable to capture output - $!\n"; + my $previous_fh = select($fh); + eval { $code->() }; + my $err = $@; + select($previous_fh); + close($fh); + + return ($err, $output); +} + +subtest 'tunnel identity command on the target' => sub { + my $class = 'PVE::TestConfig'; + + # validates the parameters like the tunnel endpoints do, returns the error, output and reply + my $handle = sub($state, $reservation, $params) { + my $schema = { + type => 'object', + properties => PVE::GuestIdentity::tunnel_identity_cmd_schema(), + additionalProperties => 0, + }; + my $reply; + my ($err, $output) = capture_output(sub { + PVE::JSONSchema::validate($params, $schema); + $reply = + PVE::GuestIdentity::handle_tunnel_identity($class, $state, $reservation, $params); + }); + return ($err, $output, $reply); + }; + my $once = "the uid can only be adopted once, before the config arrives\n"; + + my $uid = new_uid(); + my $reservation = $class->create_and_lock_config(600); + my $state = { vmid => 600, lock => 'create' }; + my ($err, $output, $reply) = $handle->($state, $reservation, { uid => $uid }); + is($err, '', 'free uid: accepted'); + is_deeply( + $reply, + { identity => { result => 'adopted', uid => $uid, requested => $uid } }, + 'free uid: reply', + ); + is($output, "guest identity: adopted uid $uid from the source guest\n", 'free uid: logged'); + is(identity_of($class, 600)->{uid}, $uid, 'free uid: stored'); + + ($err) = $handle->($state, $reservation, { uid => new_uid() }); + is($err, $once, 'second command is refused'); + is(identity_of($class, 600)->{uid}, $uid, 'second command: uid is unchanged'); + + $reservation = $class->create_and_lock_config(601); + my $reserved = identity_of($class, 601); + ($err) = $handle->({ vmid => 601, lock => 'migrate' }, $reservation, { uid => new_uid() }); + is($err, $once, 'command after the config is refused'); + is_deeply(identity_of($class, 601), $reserved, 'after the config: identity is kept'); + + ($err, $output) = + $handle->({ vmid => 601, lock => 'create' }, $reservation, { uid => $uid }); + is( + $err, + "guest 600 in the target cluster already has the uid '$uid' of this guest - use" + . " 'allow-new-uid' to migrate it with a new uid\n", + 'uid in use: refused', + ); + is($output, '', 'uid in use: no result logged'); + is_deeply(identity_of($class, 601), $reserved, 'uid in use: reserved identity stays'); + + ($err, $output, $reply) = $handle->( + { vmid => 601, lock => 'create' }, + $reservation, + { uid => $uid, 'allow-new-uid' => 1 }, + ); + is($err, '', 'uid in use with allow-new-uid: accepted'); + is_deeply( + $reply, + { + identity => { + result => 'in-use', + uid => $reserved->{uid}, + requested => $uid, + 'in-use-by' => 600, + }, + }, + 'uid in use with allow-new-uid: reply', + ); + is( + $output, + "guest identity: source guest uid $uid is in use by guest 600, keeping uid" + . " $reserved->{uid}\n", + 'uid in use with allow-new-uid: logged', + ); + is_deeply(identity_of($class, 601), $reserved, 'uid in use with allow-new-uid: kept'); + + $reservation = $class->create_and_lock_config(602); + { + local $lock_domain_error = "cfs-lock 'domain-guest-identity' error: no quorum!\n"; + ($err) = + $handle->({ vmid => 602, lock => 'create' }, $reservation, { uid => new_uid() }); + } + is( + $err, + "unable to adopt the uid of the source guest - cfs-lock 'domain-guest-identity' error:" + . " no quorum!\n", + 'lock error is passed on', + ); + + $reservation = $class->create_and_lock_config(603); + ($err) = $handle->({ vmid => 603, lock => 'create' }, $reservation, {}); + like($err, qr/^uid: property is missing/m, 'missing uid is refused'); + ($err) = + $handle->({ vmid => 603, lock => 'create' }, $reservation, { uid => $uid, x => 1 }); + like($err, qr/^x: property is not defined in schema/m, 'unknown parameter is refused'); + + delete PVE::GuestIdentity::tunnel_identity_cmd_schema()->{uid}; + ok(PVE::GuestIdentity::tunnel_identity_cmd_schema()->{uid}, 'schema is a new copy'); + + my $option = PVE::JSONSchema::get_standard_option('pve-guest-allow-new-uid'); + is($option->{type}, 'boolean', 'the override is a standard option'); +}; + +subtest 'tunnel identity command on the source' => sub { + my $class = 'PVE::TestConfig'; + + my $tunnel = { version => 3 }; + my ($commands, $reply); + local *PVE::Tunnel::write_tunnel = sub { + my ($used_tunnel, $timeout, $command, $params) = @_; + push $commands->@*, [$used_tunnel, $timeout, $command, $params]; + die $reply if !ref($reply); + return $reply; + }; + + # like the migration of the guest $vmid with the options $opts after its preparation loaded + # the config + my $migration = sub($vmid, $opts = {}) { + PVE::Cluster::cfs_update(); + my $conf = $class->load_config($vmid); + return bless { vmid => $vmid, vmconf => $conf, opts => $opts }, 'PVE::AbstractMigrate'; + }; + # returns the error, the output, and the sent parameters + my $send = sub($name, $migrate) { + $commands = []; + my ($err, $output) = + capture_output(sub { $migrate->adopt_identity_on_target($class, $tunnel) }); + my @sent = map { [$_->@[0 .. 2]] } $commands->@*; + is_deeply(\@sent, [[$tunnel, 60, 'identity']], "$name: one command, up to a minute"); + return ($err, $output, $commands->[0]->[3]); + }; + + $class->write_config(610, { name => 'source' }); + my $source = identity_of($class, 610); + $reply = + { identity => { result => 'adopted', uid => $source->{uid}, requested => $source->{uid} } }; + my $writes_before = $writes; + my ($err, $output, $params) = $send->('source with uid', $migration->(610)); + is($err, '', 'source with uid: no error'); + is_deeply($params, { uid => $source->{uid}, 'allow-new-uid' => 0 }, + 'source with uid: sent'); + is($writes, $writes_before, 'source with uid: nothing is written'); + like( + $output, + qr/^\S+ \S+ guest identity: the migrated guest keeps uid $source->{uid}\n\z/, + 'source with uid: result is logged', + ); + + $reply = { + identity => { + result => 'in-use', + uid => 'targettargettargetta', + requested => $source->{uid}, + 'in-use-by' => 7, + }, + }; + ($err, $output, $params) = + $send->('allow-new-uid', $migration->(610, { 'allow-new-uid' => 1 })); + is($params->{'allow-new-uid'}, 1, 'allow-new-uid is sent'); + my $line = + "guest identity: guest 7 in the target cluster has uid $source->{uid}, the migrated" + . " guest gets the new uid targettargettargetta"; + like($output, qr/^\S+ \S+ \Q$line\E\n\z/, 'uid in use on the target: result is logged'); + + set_file($class, 611, "name: source\n"); + my $migrate = $migration->(611); + $reply = { success => 1 }; + $writes_before = $writes; + ($err, $output, $params) = $send->('source without uid', $migrate); + my $assigned = identity_of($class, 611); + ok($assigned, 'source without uid: gets one'); + is($writes, $writes_before + 1, 'source without uid: written once'); + is($params->{uid}, $assigned->{uid}, 'source without uid: sends the uid it got'); + is_deeply( + $class->get_identity($migrate->{vmconf}), + $assigned, + 'source without uid: loaded config has it', + ); + is($output, '', 'source without uid: reply without result is not logged'); + + my $stored_uid = new_uid(); + set_file($class, 612, "meta: ctime=1,rtime=5,uid=$stored_uid\nname: source\n"); + $migrate = $migration->(612); + # like a parser that drops a 'meta' property with a member it does not know + delete $migrate->{vmconf}->{meta}; + ($err, $output, $params) = $send->('stored uid only', $migrate); + is( + $params->{uid}, $stored_uid, 'stored uid only: sends the stored uid', + ); + is_deeply( + identity_of($class, 612), + { uid => $stored_uid, rtime => 5 }, + 'stored uid only: stored identity is kept', + ); + + $reply = + { identity => { result => 'future', uid => $stored_uid, requested => $stored_uid } }; + ($err, $output) = $send->('unknown result', $migration->(612)); + is($err, '', 'unknown result: no error'); + like( + $output, + qr/ guest identity: unknown result 'future' of the target$/m, + 'unknown result: warning', + ); + + $reply = "error - tunnel command 'identity' failed - refused\n"; + ($err, $output) = $send->('failed command', $migration->(612)); + is($err, $reply, 'failed command: error is passed on'); +}; + done_testing(); -- 2.47.3