From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A67DD1FF0DA for ; Thu, 19 Nov 2026 20:21:33 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4EF79218B4; Thu, 19 Nov 2026 20:20:06 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH guest-common v5 12/41] migrate: add shared helpers for the identity command of the migration tunnel Date: Thu, 19 Nov 2026 20:18:44 +0100 Message-ID: <20261119191914.2336934-13-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119191914.2336934-1-t.lamprecht@proxmox.com> References: <20261119191914.2336934-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795115959827 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.156 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: T7CRO3PWYRBRZFNNCJCJAC3WA446U3HL X-Message-ID-Hash: T7CRO3PWYRBRZFNNCJCJAC3WA446U3HL X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: In a remote migration, the source sends the uid of the guest to the target with the 'identity' tunnel command. Without shared code, VMs and containers would each define three things: the command, the rule that the target accepts it only once and only before the config arrives, and the sender with its timeout and log line. All of these would have to be kept in sync by hand with every change of the tunnel. Add the parameter schema and the handler of the target to PVE::GuestIdentity, as PVE::StorageTunnel does for the storage commands. Add the sender to PVE::AbstractMigrate, next to the sender of the 'bwlimit' command. The target adopts the uid right after it reserved the VMID, before anything is transferred. If another guest in the target cluster has the uid, the handler refuses the migration, unless the caller allows a new uid for the migrated guest. That override is a standard option, so that the migration commands of both guest types describe it the same way. The reply of the target is about the guest on the target. The guest on the source keeps its uid in any case, so the source logs the reply in its own words. If a newer target reports a result that the source does not know, the source only logs a warning. A source without guest identities sends no uid at all. Add a helper for the 'config' command of the target that logs the new uid of the guest in that case, and that a migration back to such a cluster gets refused, as nothing else tells the user. If the parsed config of the source has no identity, the sender writes the config first. A guest without a uid gets one with that write. This way, the source keeps the uid that it sends and does not get another one with a later write. The write also brings back a stored uid that the parser of the guest type dropped, for example because of a 'meta' member that it does not know. Callers send the command only after the version check of the tunnel. A target that this check refuses then leaves the source config unchanged. Signed-off-by: Thomas Lamprecht Reviewed-by: Dominik Csapak Tested-by: Dominik Csapak --- changes v4 -> v5: * new helper for the 'config' command of the target: if the source sent no uid, it logs the new uid of the guest and that a migration back to such a cluster gets refused (Dominik) * one comment reworded changes v3 -> v4: * the source logs the reply in its own words (the migrated guest keeps its uid, or gets a new one), not in those of the target, which are about the guest there * the handler adopts the uid and refuses one in use itself, without adopt_from_source of the previous patch * 'allow-new-uid' is a standard option, for the command schema and for the migration commands of both guest types * the sender is called adopt_identity_on_target * comment and commit message: only a target refused by the tunnel version check leaves the source config unchanged * tests: the log lines of both sides, and a failure to take the identity lock on the target * commit message: plainer wording, same content changes v2 -> v3: * new in v3: command schema, target handler and sender moved here from qemu-server and container * the sender gives a guest without uid one before it sends it, and sends the stored uid if the parser dropped 'meta' src/PVE/AbstractMigrate.pm | 42 +++++ src/PVE/GuestIdentity.pm | 66 ++++++++ src/tests/config-identity-tests.pl | 246 +++++++++++++++++++++++++++++ 3 files changed, 354 insertions(+) diff --git a/src/PVE/AbstractMigrate.pm b/src/PVE/AbstractMigrate.pm index 5d74f53..34273c2 100644 --- a/src/PVE/AbstractMigrate.pm +++ b/src/PVE/AbstractMigrate.pm @@ -443,6 +443,48 @@ sub get_bwlimit { return $bwlimit; } +# Lets the target of a remote migration adopt the uid of the guest, whose config class is +# $config_class, through the tunnel $tunnel, before anything gets transferred there. The target +# refuses a uid that another guest there has, unless the migration allows a new uid, see +# PVE::GuestIdentity::handle_tunnel_identity. Call it only after the version check of the tunnel, +# so that a target refused by that check leaves the config of the source unchanged. +sub adopt_identity_on_target { + my ($self, $config_class, $tunnel) = @_; + + my $conf = $self->{vmconf}; + my $identity = $config_class->get_identity($conf); + if (!$identity) { + # A guest without uid gets one first, so the source keeps the uid that it sends. The write + # also restores a stored uid that the parser of the guest type dropped. + $config_class->write_config($self->{vmid}, $conf); + $identity = $config_class->get_identity($conf) or return; + } + my $uid = $identity->{uid}; + + my $params = { uid => $uid, 'allow-new-uid' => $self->{opts}->{'allow-new-uid'} ? 1 : 0 }; + + # the target waits for the cluster-wide guest identity lock + my $res = PVE::Tunnel::write_tunnel($tunnel, 60, 'identity', $params); + my $reply = $res->{identity} or return; + + # the guest here keeps its uid in any case, the result is about the guest on the target + my $result = $reply->{result} // ''; + if ($result eq 'adopted' || $result eq 'kept') { + $self->log('info', "guest identity: the migrated guest keeps uid $uid"); + } elsif ($result eq 'in-use') { + $self->log( + 'info', + "guest identity: guest $reply->{'in-use-by'} in the target cluster has uid $uid, the" + . " migrated guest gets the new uid $reply->{uid}", + ); + } else { + # only a newer target reports another result; the command succeeded, so it accepted the uid + $self->log('warn', "guest identity: unknown result '$result' of the target"); + } + + return; +} + sub map_storage { my ($self, $scfg, $storeid) = @_; diff --git a/src/PVE/GuestIdentity.pm b/src/PVE/GuestIdentity.pm index cd17c30..635336d 100644 --- a/src/PVE/GuestIdentity.pm +++ b/src/PVE/GuestIdentity.pm @@ -389,6 +389,72 @@ sub adopt_from_backup($class, $vmid, $uid, $reservation) { return; } +PVE::JSONSchema::register_standard_option( + 'pve-guest-allow-new-uid', + { + type => 'boolean', + description => "Allow the migration if another guest in the target cluster already has the" + . " uid of this guest. The migrated guest then gets a new uid.", + optional => 1, + default => 0, + }, +); + +# Returns the properties of the parameters of the 'identity' command of the migration tunnel, see +# handle_tunnel_identity. Every call returns a new copy. +sub tunnel_identity_cmd_schema() { + return { + uid => { + type => 'string', + description => 'uid of the source guest', + }, + 'allow-new-uid' => PVE::JSONSchema::get_standard_option('pve-guest-allow-new-uid'), + }; +} + +# Handles the 'identity' command with the parameters $params on the target of a remote migration of +# a guest of the config class $class: adopts the uid of the source guest for the guest that the +# tunnel reserved with $reservation, logs the result, and returns the reply for the source. $state +# is the state of the tunnel, with the VMID in 'vmid' and the lock of the guest in 'lock'. +# +# The command is only accepted once, and only before the config arrives, while the guest still has +# the 'create' lock of the reservation. If another guest in the cluster has the uid, the command +# fails, which refuses the migration, unless the parameter 'allow-new-uid' is set, in which case the +# guest keeps the uid of its reservation. +sub handle_tunnel_identity($class, $state, $reservation, $params) { + die "the uid can only be adopted once, before the config arrives\n" + if ($state->{lock} // '') ne 'create' || $state->{'identity-handled'}; + $state->{'identity-handled'} = 1; + + my $uid = $params->{uid}; + my $res = eval { adopt($class, $state->{vmid}, $uid, $reservation) }; + die "unable to adopt the uid of the source guest - $@" if $@; + + if ($res->{result} eq 'in-use' && !$params->{'allow-new-uid'}) { + die "guest $res->{'in-use-by'} in the target cluster already has the uid '$uid' of this" + . " guest - use 'allow-new-uid' to migrate it with a new uid\n"; + } + + print adopt_result_message($res, 'source guest') . "\n"; + + return { identity => $res }; +} + +# For the 'config' command on the target of a remote migration, with the state $state of the +# tunnel and the reservation $reservation of the guest: logs that the guest got a new uid if the +# source did not send the 'identity' command. Only a source without guest identities does that, +# and a migration back to such a cluster gets refused. +sub log_tunnel_identity_not_sent($state, $reservation) { + return if $state->{'identity-handled'}; + + my $uid = $reservation->{assigned}->{uid}; + print "guest identity: the source sent no uid, the migrated guest has the new uid $uid\n"; + print "guest identity: the source cluster does not support guest identities yet, so a" + . " migration back there gets refused\n"; + + return; +} + # Returns undef if $node runs a version of Proxmox VE that keeps the identity when writing guest # configs, see MIN_PVE_MANAGER_VERSION. Missing or invalid version information counts as too old. # Otherwise returns a hash with 'required-version', the minimum version, and 'version', the version diff --git a/src/tests/config-identity-tests.pl b/src/tests/config-identity-tests.pl index b9b15b7..bb1e1cd 100755 --- a/src/tests/config-identity-tests.pl +++ b/src/tests/config-identity-tests.pl @@ -10,7 +10,9 @@ use Test::MockModule; use Test::More; use PVE::AbstractConfig; +use PVE::AbstractMigrate; use PVE::GuestIdentity; +use PVE::JSONSchema; my $tmpdir = tempdir('config-identity-XXXXXX', DIR => '.', CLEANUP => 1); @@ -1092,4 +1094,248 @@ subtest 'drop pending meta' => sub { } }; +# Runs $code with the output to STDOUT captured, returns the error and the output. +my sub capture_output($code) { + my $output = ''; + open(my $fh, '>', \$output) or die "unable to capture output - $!\n"; + my $previous_fh = select($fh); + eval { $code->() }; + my $err = $@; + select($previous_fh); + close($fh); + + return ($err, $output); +} + +subtest 'tunnel identity command on the target' => sub { + my $class = 'PVE::TestConfig'; + + # validates the parameters like the tunnel endpoints do, returns the error, output and reply + my $handle = sub($state, $reservation, $params) { + my $schema = { + type => 'object', + properties => PVE::GuestIdentity::tunnel_identity_cmd_schema(), + additionalProperties => 0, + }; + my $reply; + my ($err, $output) = capture_output(sub { + PVE::JSONSchema::validate($params, $schema); + $reply = + PVE::GuestIdentity::handle_tunnel_identity($class, $state, $reservation, $params); + }); + return ($err, $output, $reply); + }; + my $once = "the uid can only be adopted once, before the config arrives\n"; + + my $uid = new_uid(); + my $reservation = $class->create_and_lock_config(600); + my $state = { vmid => 600, lock => 'create' }; + my ($err, $output, $reply) = $handle->($state, $reservation, { uid => $uid }); + is($err, '', 'free uid: accepted'); + is_deeply( + $reply, + { identity => { result => 'adopted', uid => $uid, requested => $uid } }, + 'free uid: reply', + ); + is($output, "guest identity: adopted uid $uid from the source guest\n", 'free uid: logged'); + is(identity_of($class, 600)->{uid}, $uid, 'free uid: stored'); + + ($err) = $handle->($state, $reservation, { uid => new_uid() }); + is($err, $once, 'second command is refused'); + is(identity_of($class, 600)->{uid}, $uid, 'second command: uid is unchanged'); + + $reservation = $class->create_and_lock_config(601); + my $reserved = identity_of($class, 601); + ($err) = $handle->({ vmid => 601, lock => 'migrate' }, $reservation, { uid => new_uid() }); + is($err, $once, 'command after the config is refused'); + is_deeply(identity_of($class, 601), $reserved, 'after the config: identity is kept'); + + ($err, $output) = + $handle->({ vmid => 601, lock => 'create' }, $reservation, { uid => $uid }); + is( + $err, + "guest 600 in the target cluster already has the uid '$uid' of this guest - use" + . " 'allow-new-uid' to migrate it with a new uid\n", + 'uid in use: refused', + ); + is($output, '', 'uid in use: no result logged'); + is_deeply(identity_of($class, 601), $reserved, 'uid in use: reserved identity stays'); + + ($err, $output, $reply) = $handle->( + { vmid => 601, lock => 'create' }, + $reservation, + { uid => $uid, 'allow-new-uid' => 1 }, + ); + is($err, '', 'uid in use with allow-new-uid: accepted'); + is_deeply( + $reply, + { + identity => { + result => 'in-use', + uid => $reserved->{uid}, + requested => $uid, + 'in-use-by' => 600, + }, + }, + 'uid in use with allow-new-uid: reply', + ); + is( + $output, + "guest identity: source guest uid $uid is in use by guest 600, keeping uid" + . " $reserved->{uid}\n", + 'uid in use with allow-new-uid: logged', + ); + is_deeply(identity_of($class, 601), $reserved, 'uid in use with allow-new-uid: kept'); + + $reservation = $class->create_and_lock_config(602); + { + local $lock_domain_error = "cfs-lock 'domain-guest-identity' error: no quorum!\n"; + ($err) = + $handle->({ vmid => 602, lock => 'create' }, $reservation, { uid => new_uid() }); + } + is( + $err, + "unable to adopt the uid of the source guest - cfs-lock 'domain-guest-identity' error:" + . " no quorum!\n", + 'lock error is passed on', + ); + + # what the 'config' command logs if no 'identity' command came before it + $reservation = $class->create_and_lock_config(604); + $state = { vmid => 604, lock => 'create' }; + my $log_not_sent = + sub { PVE::GuestIdentity::log_tunnel_identity_not_sent($state, $reservation) }; + (undef, $output) = capture_output($log_not_sent); + my $assigned = $reservation->{assigned}->{uid}; + like( + $output, + qr/^guest identity: the source sent no uid, .* has the new uid $assigned\n/, + 'no identity command: the new uid is logged', + ); + like($output, qr/ migration back there gets refused\n$/, 'and the consequence'); + $handle->($state, $reservation, { uid => new_uid() }); + (undef, $output) = capture_output($log_not_sent); + is($output, '', 'after an identity command: nothing is logged'); + + $reservation = $class->create_and_lock_config(603); + ($err) = $handle->({ vmid => 603, lock => 'create' }, $reservation, {}); + like($err, qr/^uid: property is missing/m, 'missing uid is refused'); + ($err) = + $handle->({ vmid => 603, lock => 'create' }, $reservation, { uid => $uid, x => 1 }); + like($err, qr/^x: property is not defined in schema/m, 'unknown parameter is refused'); + + delete PVE::GuestIdentity::tunnel_identity_cmd_schema()->{uid}; + ok(PVE::GuestIdentity::tunnel_identity_cmd_schema()->{uid}, 'schema is a new copy'); + + my $option = PVE::JSONSchema::get_standard_option('pve-guest-allow-new-uid'); + is($option->{type}, 'boolean', 'the override is a standard option'); +}; + +subtest 'tunnel identity command on the source' => sub { + my $class = 'PVE::TestConfig'; + + my $tunnel = { version => 3 }; + my ($commands, $reply); + local *PVE::Tunnel::write_tunnel = sub { + my ($used_tunnel, $timeout, $command, $params) = @_; + push $commands->@*, [$used_tunnel, $timeout, $command, $params]; + die $reply if !ref($reply); + return $reply; + }; + + # like the migration of the guest $vmid with the options $opts after its preparation loaded + # the config + my $migration = sub($vmid, $opts = {}) { + PVE::Cluster::cfs_update(); + my $conf = $class->load_config($vmid); + return bless { vmid => $vmid, vmconf => $conf, opts => $opts }, 'PVE::AbstractMigrate'; + }; + # returns the error, the output, and the sent parameters + my $send = sub($name, $migrate) { + $commands = []; + my ($err, $output) = + capture_output(sub { $migrate->adopt_identity_on_target($class, $tunnel) }); + my @sent = map { [$_->@[0 .. 2]] } $commands->@*; + is_deeply(\@sent, [[$tunnel, 60, 'identity']], "$name: one command, up to a minute"); + return ($err, $output, $commands->[0]->[3]); + }; + + $class->write_config(610, { name => 'source' }); + my $source = identity_of($class, 610); + $reply = + { identity => { result => 'adopted', uid => $source->{uid}, requested => $source->{uid} } }; + my $writes_before = $writes; + my ($err, $output, $params) = $send->('source with uid', $migration->(610)); + is($err, '', 'source with uid: no error'); + is_deeply($params, { uid => $source->{uid}, 'allow-new-uid' => 0 }, + 'source with uid: sent'); + is($writes, $writes_before, 'source with uid: nothing is written'); + like( + $output, + qr/^\S+ \S+ guest identity: the migrated guest keeps uid $source->{uid}\n\z/, + 'source with uid: result is logged', + ); + + $reply = { + identity => { + result => 'in-use', + uid => 'targettargettargetta', + requested => $source->{uid}, + 'in-use-by' => 7, + }, + }; + ($err, $output, $params) = + $send->('allow-new-uid', $migration->(610, { 'allow-new-uid' => 1 })); + is($params->{'allow-new-uid'}, 1, 'allow-new-uid is sent'); + my $line = + "guest identity: guest 7 in the target cluster has uid $source->{uid}, the migrated" + . " guest gets the new uid targettargettargetta"; + like($output, qr/^\S+ \S+ \Q$line\E\n\z/, 'uid in use on the target: result is logged'); + + set_file($class, 611, "name: source\n"); + my $migrate = $migration->(611); + $reply = { success => 1 }; + $writes_before = $writes; + ($err, $output, $params) = $send->('source without uid', $migrate); + my $assigned = identity_of($class, 611); + ok($assigned, 'source without uid: gets one'); + is($writes, $writes_before + 1, 'source without uid: written once'); + is($params->{uid}, $assigned->{uid}, 'source without uid: sends the uid it got'); + is_deeply( + $class->get_identity($migrate->{vmconf}), + $assigned, + 'source without uid: loaded config has it', + ); + is($output, '', 'source without uid: reply without result is not logged'); + + my $stored_uid = new_uid(); + set_file($class, 612, "meta: ctime=1,rtime=5,uid=$stored_uid\nname: source\n"); + $migrate = $migration->(612); + # like a parser that drops a 'meta' property with a member it does not know + delete $migrate->{vmconf}->{meta}; + ($err, $output, $params) = $send->('stored uid only', $migrate); + is( + $params->{uid}, $stored_uid, 'stored uid only: sends the stored uid', + ); + is_deeply( + identity_of($class, 612), + { uid => $stored_uid, rtime => 5 }, + 'stored uid only: stored identity is kept', + ); + + $reply = + { identity => { result => 'future', uid => $stored_uid, requested => $stored_uid } }; + ($err, $output) = $send->('unknown result', $migration->(612)); + is($err, '', 'unknown result: no error'); + like( + $output, + qr/ guest identity: unknown result 'future' of the target$/m, + 'unknown result: warning', + ); + + $reply = "error - tunnel command 'identity' failed - refused\n"; + ($err, $output) = $send->('failed command', $migration->(612)); + is($err, $reply, 'failed command: error is passed on'); +}; + done_testing(); -- 2.47.3