From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 677351FF0DE for ; Thu, 19 Nov 2026 11:42:57 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A66DD217F6; Thu, 19 Nov 2026 11:38:31 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v4 17/41] restore: take over the guest uid from the backup where possible Date: Thu, 19 Nov 2026 11:34:03 +0100 Message-ID: <20261119103551.1666964-18-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119103551.1666964-1-t.lamprecht@proxmox.com> References: <20261119103551.1666964-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795084556245 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.159 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ZOHGVW2VSB3NGZ5UOGRAAXFSHXUMNEAO X-Message-ID-Hash: ZOHGVW2VSB3NGZ5UOGRAAXFSHXUMNEAO X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A restore to an unused VMID, or over a VM without an identity, should keep the uid of the backup if no other guest in the cluster uses it. Take over that uid through the cluster-wide check of pve-guest-common. Do so after the data is restored and the final config is written, while holding the config lock. A VM that already had an identity keeps it, and with 'unique' the uid is not taken over. Log the result. If taking over the uid fails, only warn, as the restored VM has an identity of its own. Refuse to restore a backup over an existing VM that has a different uid, unless 'allow-identity-mismatch' is set in the create API or for qmrestore. This is independent of 'force', which permits overwriting the VM. With the override, the VM keeps its identity. The exception is a VM that has the uid of the backup as pending uid from an unfinished restore, as described below. For PBS, backup providers and VMA, compare the uids before the restore resets the replication state, restores the firewall config or replaces data of the target. For VMA this also works with standard input, through the header callback. Read the uid from the config that the restore itself uses, not from the best-effort extraction of the API, and ignore unknown members of 'meta' when doing so. Otherwise, the uid of a backup from a newer version would be lost if the property fails to parse. Take over the uid as the last step, after the live phase and the conversion to a template. A restore that fails must not take the uid, because a retry at another VMID could then no longer get it. With 'unique', this last step comes after the new MAC addresses got registered and only removes the pending uid. A failed restore can leave a VM behind. For a retry over such a VM to work, record the uid of the backup as a pending change of 'meta'. Record it after the uid check, if the restore is one that could take the uid over, also with 'unique'. Keep it in all configs that the restore writes. A VM with that pending uid can be restored over without the override. Without 'unique', the retry takes the uid over if no other guest uses it. A pending uid does not count as in use. Taking over the uid removes the pending uid, whatever the result. Only if the cluster-wide lock cannot be taken, it stays for a retry. Starting the VM for a live restore keeps the pending uid while the 'create' lock is held. In all other cases, applying pending changes drops it. A clone does not get it. Legacy tar archives are older than guest identities, so their restore does not handle a uid. Signed-off-by: Thomas Lamprecht --- changes v3 -> v4: * use prepare_restore of pve-guest-common for the check and the pending uid; restore_check_identity is private and its direct test is gone, pve-guest-common covers those cases * 'allow-identity-mismatch' is the standard option of pve-guest-common * commit message: remove a leftover of a conflict resolution, and say that 'unique' only removes the pending uid * comment: what the identity check has to come before * tests: the clone test also checks that a clone does not get the pending uid of its source, instead of a subtest of its own * commit message: plainer wording, same content changes v2 -> v3: * group the imports of the test script changes v1 -> v2: * use the shared helpers of pve-guest-common, one check helper instead of three * no identity handling for legacy tar archives * describe the effect of 'unique', shorter description and a default for 'allow-identity-mismatch' (Dominik) * starting a VM left behind by a failed restore warns that the pending uid gets dropped src/PVE/API2/Qemu.pm | 17 +- src/PVE/CLI/qmrestore.pm | 5 +- src/PVE/QemuServer.pm | 87 +++- src/test/restore-config-expected/150.conf | 9 + src/test/restore-config-input/150.conf | 10 + src/test/run_api_tests.pl | 13 + src/test/run_guest_identity_tests.pl | 15 + src/test/run_qemu_restore_config_tests.pl | 517 +++++++++++++++++++++- 8 files changed, 649 insertions(+), 24 deletions(-) create mode 100644 src/test/restore-config-expected/150.conf create mode 100644 src/test/restore-config-input/150.conf diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index 41b68961..ea85a5da 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -1329,9 +1329,15 @@ __PACKAGE__->register_method({ unique => { optional => 1, type => 'boolean', - description => "Assign a unique random ethernet address.", + description => + "Assign a unique random ethernet address and do not take over" + . " the guest uid of the backup.", requires => 'archive', }, + 'allow-identity-mismatch' => get_standard_option( + 'pve-guest-allow-identity-mismatch', + { requires => 'archive' }, + ), 'live-restore' => { optional => 1, type => 'boolean', @@ -1400,6 +1406,7 @@ __PACKAGE__->register_method({ my $ha_managed = extract_param($param, 'ha-managed'); my $storage = extract_param($param, 'storage'); my $unique = extract_param($param, 'unique'); + my $allow_identity_mismatch = extract_param($param, 'allow-identity-mismatch'); my $live_restore = extract_param($param, 'live-restore'); my $extraction_storage = extract_param($param, 'import-working-storage'); @@ -1535,6 +1542,8 @@ __PACKAGE__->register_method({ live => $live_restore, override_conf => $param, progress => $restore_progress, + reservation => $reservation, + 'allow-identity-mismatch' => $allow_identity_mismatch, }; if (my $volid = $archive->{volid}) { # best effort, real check is after restoring! @@ -1590,6 +1599,8 @@ __PACKAGE__->register_method({ PVE::QemuServer::Network::create_ifaces_ipams_ips($restored_conf, $vmid) if $unique; + + PVE::QemuServer::restore_adopt_identity($vmid, $restore_options); }; my $restore_locked = sub { @@ -4838,6 +4849,10 @@ __PACKAGE__->register_method({ next if PVE::QemuServer::drive_is_cloudinit($drive); push @$vollist, $volid; } + } elsif ($opt eq 'pending') { + # a pending uid only belongs to the source, see PVE::GuestIdentity + $newconf->{pending} = { $value->%* }; + PVE::GuestIdentity::set_pending_uid($newconf, undef); } else { # copy everything else $newconf->{$opt} = $value; diff --git a/src/PVE/CLI/qmrestore.pm b/src/PVE/CLI/qmrestore.pm index abfa2de7..a14d0cb8 100755 --- a/src/PVE/CLI/qmrestore.pm +++ b/src/PVE/CLI/qmrestore.pm @@ -52,8 +52,11 @@ __PACKAGE__->register_method({ unique => { optional => 1, type => 'boolean', - description => "Assign a unique random ethernet address.", + description => "Assign a unique random ethernet address and do not take over" + . " the guest uid of the backup.", }, + 'allow-identity-mismatch' => + get_standard_option('pve-guest-allow-identity-mismatch'), pool => { optional => 1, type => 'string', diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index d08dc33e..c14b7f51 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -32,6 +32,7 @@ use PVE::DataCenterConfig; use PVE::Exception qw(raise raise_param_exc); use PVE::Format qw(render_duration render_bytes); use PVE::GuestHelpers qw(safe_string_ne safe_num_ne safe_boolean_ne); +use PVE::GuestIdentity; use PVE::Mapping::Dir; use PVE::Mapping::PCI; use PVE::Mapping::USB; @@ -4430,19 +4431,6 @@ for my $opt (keys %$confdesc_cloudinit) { $fast_plug_option->{$opt} = 1; } -# 'meta' is read-only, so only a manual edit or an older version can have queued a change of it. -# Drop such a change instead of applying it. -my sub drop_pending_meta { - my ($conf) = @_; - - my $pending_delete_hash = PVE::QemuConfig->parse_pending_delete($conf->{pending}->{delete}); - return if !defined($conf->{pending}->{meta}) && !$pending_delete_hash->{meta}; - - warn "ignoring pending change of read-only option 'meta'\n"; - delete $conf->{pending}->{meta}; - PVE::QemuConfig->remove_from_pending_delete($conf, 'meta'); -} - # hotplug changes in [PENDING] # $selection hash can be used to only apply specified options, for # example: { cores => 1 } (only apply changed 'cores') @@ -4450,7 +4438,7 @@ my sub drop_pending_meta { sub vmconfig_hotplug_pending { my ($vmid, $conf, $storecfg, $selection, $errors) = @_; - drop_pending_meta($conf); + PVE::QemuConfig->drop_pending_meta($conf); my $defaults = load_defaults(); my $arch = PVE::QemuServer::Helpers::get_vm_arch($conf); @@ -4850,7 +4838,13 @@ sub vmconfig_apply_pending { return if !scalar(keys %{ $conf->{pending} }); - drop_pending_meta($conf); + # a restore keeps its pending uid while it holds the 'create' lock, also when it starts the VM + # for a live restore, and removes it itself, see restore_check_identity + my $restore_uid = + ($conf->{lock} // '') eq 'create' ? PVE::GuestIdentity::get_pending_uid($conf) : undef; + PVE::GuestIdentity::set_pending_uid($conf, undef) if defined($restore_uid); + + PVE::QemuConfig->drop_pending_meta($conf); my $add_apply_error = sub { my ($opt, $msg) = @_; @@ -4959,6 +4953,8 @@ sub vmconfig_apply_pending { } } + PVE::GuestIdentity::set_pending_uid($conf, $restore_uid) if defined($restore_uid); + # write all changes at once to avoid unnecessary i/o PVE::QemuConfig->write_config($vmid, $conf); if ($generate_cloudinit) { @@ -6761,6 +6757,49 @@ sub restore_merge_config { return $backup_conf; } +# Checks the guest identity of the backup against the one of the target VM and prepares adopting its +# uid, see PVE::GuestIdentity::prepare_restore. Must run before the restore resets the replication +# state, restores the firewall config or replaces data of the target VM. +# +# A uid that the restore could adopt at its end becomes the pending uid of the VM until then, so +# that a VM left behind by a failure or an abort of the restore still has it for a retry, see +# restore_write_config. That holds with 'unique' too, which only keeps the restore itself from +# adopting the uid. +my sub restore_check_identity { + my ($vmid, $backup_conf_raw, $opts) = @_; + + my $backup = PVE::GuestIdentity::parse_identity_from_raw_config($backup_conf_raw); + $opts->{'backup-uid'} = $backup ? $backup->{uid} : undef; + $opts->{'pending-uid'} = PVE::GuestIdentity::prepare_restore( + 'PVE::QemuConfig', + $vmid, + $opts->{reservation}, + $opts->{'backup-uid'}, + $opts->{'allow-identity-mismatch'}, + ); +} + +# Writes $conf, a config that the restore assembled itself, like a temporary or the restored one, +# with the pending uid of the restore, see restore_check_identity. +my sub restore_write_config { + my ($vmid, $conf, $opts) = @_; + + PVE::GuestIdentity::set_pending_uid($conf, $opts->{'pending-uid'}); + PVE::QemuConfig->write_config($vmid, $conf); +} + +# Adopts the uid of the backup for the restored VM where the identity rules allow it, see +# PVE::GuestIdentity::adopt, which also removes the pending uid. Must be the last step of a +# successful restore, with the config lock still held, so that a failure before leaves the pending +# uid behind. A failure only warns, as the restored VM keeps the identity of its reservation. +sub restore_adopt_identity { + my ($vmid, $opts) = @_; + + my $uid = $opts->{unique} ? undef : $opts->{'backup-uid'}; + + PVE::GuestIdentity::adopt_from_backup('PVE::QemuConfig', $vmid, $uid, $opts->{reservation}); +} + sub scan_volids { my ($cfg, $vmid) = @_; @@ -6997,6 +7036,8 @@ sub restore_proxmox_backup_archive { $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + restore_check_identity($vmid, PVE::Tools::file_get_contents($cfgfn), $options); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); @@ -7016,7 +7057,7 @@ sub restore_proxmox_backup_archive { # fixme: rate limit? # create empty/temp config - PVE::QemuConfig->write_config($vmid, { memory => 128, lock => 'create' }); + restore_write_config($vmid, { memory => 128, lock => 'create' }, $options); $restore_cleanup_oldconf->($storecfg, $vmid, $oldconf, $virtdev_hash); @@ -7101,7 +7142,7 @@ sub restore_proxmox_backup_archive { my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); - PVE::QemuConfig->write_config($vmid, $new_conf); + restore_write_config($vmid, $new_conf, $options); eval { rescan($vmid, 1); }; warn $@ if $@; @@ -7179,6 +7220,8 @@ sub restore_external_archive { my $data = $backup_provider->archive_get_guest_config($volname) or die "backup provider failed to extract guest configuration\n"; + restore_check_identity($vmid, $data, $options); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); @@ -7195,7 +7238,7 @@ sub restore_external_archive { $parse_backup_hints->($rpcenv, $user, $storecfg, $fh, $devinfo, $options); # create empty/temp config - PVE::QemuConfig->write_config($vmid, { memory => 128, lock => 'create' }); + restore_write_config($vmid, { memory => 128, lock => 'create' }, $options); $restore_cleanup_oldconf->($storecfg, $vmid, $oldconf, $virtdev_hash); @@ -7273,7 +7316,7 @@ sub restore_external_archive { my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); - PVE::QemuConfig->write_config($vmid, $new_conf); + restore_write_config($vmid, $new_conf, $options); eval { rescan($vmid, 1); }; warn $@ if $@; @@ -7594,6 +7637,8 @@ sub restore_vma_archive { my $print_devmap = sub { my $cfgfn = "$tmpdir/qemu-server.conf"; + restore_check_identity($vmid, PVE::Tools::file_get_contents($cfgfn), $opts); + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); @@ -7621,7 +7666,7 @@ sub restore_vma_archive { } # create empty/temp config - PVE::QemuConfig->write_config($vmid, { memory => 128, lock => 'create' }); + restore_write_config($vmid, { memory => 128, lock => 'create' }, $opts); $restore_cleanup_oldconf->($cfg, $vmid, $oldconf, $virtdev_hash); # allocate volumes @@ -7740,7 +7785,7 @@ sub restore_vma_archive { my $new_conf = restore_merge_config($conffile, $new_conf_raw, $opts->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); - PVE::QemuConfig->write_config($vmid, $new_conf); + restore_write_config($vmid, $new_conf, $opts); eval { rescan($vmid, 1); }; warn $@ if $@; diff --git a/src/test/restore-config-expected/150.conf b/src/test/restore-config-expected/150.conf new file mode 100644 index 00000000..a8e8909c --- /dev/null +++ b/src/test/restore-config-expected/150.conf @@ -0,0 +1,9 @@ +# VM with guest identity +boot: order=scsi0 +cores: 1 +memory: 512 +meta: creation-qemu=10.1.2,ctime=1762000000,rtime=1762000000123,uid=abcdefghijklmnopqrst +name: identity +scsi0: target:150/vm-150-disk-0.qcow2,size=4G +scsihw: virtio-scsi-pci +vmgenid: 0 diff --git a/src/test/restore-config-input/150.conf b/src/test/restore-config-input/150.conf new file mode 100644 index 00000000..927be6dc --- /dev/null +++ b/src/test/restore-config-input/150.conf @@ -0,0 +1,10 @@ +# VM with guest identity +boot: order=scsi0 +cores: 1 +memory: 512 +meta: creation-qemu=10.1.2,ctime=1762000000,rtime=1762000000123,uid=abcdefghijklmnopqrst +name: identity +scsi0: mydir:150/vm-150-disk-0.qcow2,size=4G +scsihw: virtio-scsi-pci +vmgenid: 0 +#qmdump#map:scsi0:drive-scsi0:mydir:qcow2: diff --git a/src/test/run_api_tests.pl b/src/test/run_api_tests.pl index 2ca713a9..ea027670 100755 --- a/src/test/run_api_tests.pl +++ b/src/test/run_api_tests.pl @@ -13,6 +13,7 @@ use Test::MockModule; use Test::More; use PVE::API2::Qemu; +use PVE::CLI::qmrestore; use PVE::GuestIdentity; use PVE::RPCEnvironment; @@ -124,6 +125,18 @@ is( 'update_vm: reverting other options passes the parameter checks', ); +is_deeply( + api_error('create_vm', { 'allow-identity-mismatch' => 1 }), + { archive => "missing property required by 'allow-identity-mismatch'" }, + "create_vm: 'allow-identity-mismatch' only for restores", +); +my $qmrestore_param = PVE::CLI::qmrestore->map_method_by_name('qmrestore')->{parameters}; +is( + $qmrestore_param->{properties}->{'allow-identity-mismatch'}->{type}, + 'boolean', + "qmrestore has 'allow-identity-mismatch'", +); + subtest 'pending changes of meta are dropped' => sub { my $meta = 'creation-qemu=10.1.2,ctime=1762000000'; my $stored; diff --git a/src/test/run_guest_identity_tests.pl b/src/test/run_guest_identity_tests.pl index 3eec153f..66bb8c30 100755 --- a/src/test/run_guest_identity_tests.pl +++ b/src/test/run_guest_identity_tests.pl @@ -184,6 +184,8 @@ subtest 'clone' => sub { my $disk = "scsi0: local:100/vm-100-disk-0.raw,size=4G\n"; my $source = "memory: 512\nmeta: creation-qemu=9.2.0,ctime=1600000000," . "rtime=1600000000000,uid=$UID_A\nname: source\n$disk"; + # like a VM left behind by a restore of a backup with uid B, with another pending change + $source .= "\n[PENDING]\nmemory: 1024\nmeta: uid=$UID_B\n"; # a snapshot can carry any identity configs()->{100} = "$source\n[snap]\nmemory: 256\nmeta: rtime=1,uid=$UID_B\n${disk}snaptime: 1600000001\n"; @@ -237,6 +239,19 @@ subtest 'clone' => sub { 'clone of a snapshot has a new uid as well', ); is(stored_identity(100)->{uid}, $UID_A, 'source keeps its uid'); + + my $writes = $QemuServerTest::FakeCFS::written->{300}; + ok(scalar(grep { m/^scsi0: local:300\//m } $writes->@*) >= 2, + 'clone writes the disk first'); + ok(!scalar(grep { m/uid=$UID_B/ } $writes->@*), + 'no write of the clone has the pending uid'); + like( + $output, + qr/pending changes for 'memory', discarding for clone/, + 'other pending changes are discarded as before', + ); + unlike(configs()->{300}, qr/^\[PENDING\]/m, 'clone has no pending section'); + like(configs()->{100}, qr/^meta: uid=$UID_B$/m, 'source keeps its pending uid'); }; subtest 'importovf' => sub { diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl index 4e2bc798..c5116836 100755 --- a/src/test/run_qemu_restore_config_tests.pl +++ b/src/test/run_qemu_restore_config_tests.pl @@ -16,9 +16,11 @@ use File::Basename; use PVE::API2::Qemu; use PVE::GuestHelpers; +use PVE::GuestIdentity; use PVE::QemuConfig; use PVE::QemuServer; use PVE::QemuServer::Helpers; +use PVE::QemuServer::MetaInfo; use PVE::RPCEnvironment; use PVE::Tools qw(dir_glob_foreach file_get_contents file_set_contents); @@ -96,7 +98,7 @@ my $ARCHIVES = { my $BACKUP_FIREWALL = "[OPTIONS]\nenable: 1\n"; our $current_backup; # raw config of the backup -# where the restore fails: 'read-config', 'command', 'data', 'after-restore' or 'live' +# where the restore fails: 'read-config', 'command', 'firewall', 'data', 'after-restore' or 'live' our $fail_at = ''; # the stat file that qmextract leaves: undef for none, 'directory' for one that cannot be read, or # its content @@ -199,11 +201,15 @@ $qemu_server_module->mock( check_running => sub { return 0; }, pbs_live_restore => sub { my ($vmid, $conf, $storecfg) = @_; + # like the start of the VM for the live phase + PVE::QemuServer::vmconfig_apply_pending($vmid, $conf, $storecfg) + if scalar(keys(($conf->{pending} // {})->%*)); die "live restore failed\n" if $fail_at eq 'live'; $calls->{live_restore}++; }, rescan => sub { return; }, restore_firewall_config => sub { + die "unable to restore the firewall config\n" if $fail_at eq 'firewall'; $calls->{firewall}++; return $qemu_server_module->original('restore_firewall_config')->(@_); }, @@ -704,4 +710,513 @@ subtest 'failure after the data was restored keeps the VM' => sub { like($output, qr/error after data was restored/, 'warning about the VM left behind'); }; +# Guest identity of restored VMs + +my $UID_BACKUP = 'bbbbbbbbbbbbbbbbbbbb'; +my $UID_TARGET = 'tttttttttttttttttttt'; + +my $TARGET_CONF_NO_UID = "memory: 1024\nmeta: creation-qemu=8.0.2,ctime=1500000000\nname: target\n"; + +my $NOW = 1800000000000; +my $BACKUP_META = 'creation-qemu=9.2.0,ctime=1600000000'; +my $REFUSED = + "backup belongs to another guest: its uid '$UID_BACKUP' differs from uid '$UID_TARGET' of the" + . " existing VM 100 - use 'allow-identity-mismatch' to overwrite the VM anyway\n"; +my $IGNORED = "guest identity: ignoring backup uid $UID_BACKUP, VM 100 keeps uid $UID_TARGET"; +# captures the uid that the VM keeps +my $ADOPT_FAILED = "WARN: guest identity: unable to adopt backup uid $UID_BACKUP, keeping uid" + . " ([a-z2-7]{20}) - cfs-lock 'domain-guest-identity' error: no quorum!"; +$ADOPT_FAILED = qr/^$ADOPT_FAILED$/m; +my $KEPT_TARGET = qr/^guest identity: kept uid $UID_TARGET$/m; + +# legacy tar archives get no guest identity handling +my @IDENTITY_ARCHIVES = grep { $_ ne 'tar' } sort keys $ARCHIVES->%*; + +my $cases = [ + { + name => 'free VMID, backup uid unused', + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + result => { uid => $UID_BACKUP, rtime => $NOW }, + output => qr/^guest identity: adopted uid $UID_BACKUP from the backup$/m, + }, + { + name => 'free VMID, backup uid in use by another guest', + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + other => "rtime=1,uid=$UID_BACKUP", + result => { uid => 'generated', rtime => $NOW }, + output => qr/: backup uid $UID_BACKUP is in use by guest 300, keeping uid \w{20}$/m, + }, + { + name => 'free VMID, backup without uid', + backup => $BACKUP_META, + result => { uid => 'generated', rtime => $NOW }, + output => qr/^guest identity: new uid aaaaaaaaaaaaaaaaaaab$/m, + }, + { + name => 'free VMID, unique', + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + param => { unique => 1 }, + result => { uid => 'generated', rtime => $NOW }, + output => qr/^guest identity: new uid aaaaaaaaaaaaaaaaaaab$/m, + }, + { + name => 'free VMID, identity lock fails', + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + lock_error => "cfs-lock 'domain-guest-identity' error: no quorum!\n", + result => { uid => 'generated', rtime => $NOW }, + output => $ADOPT_FAILED, + }, + { + name => 'free VMID, backup meta with a member unknown to this version', + backup => "$BACKUP_META,future=1,rtime=1600000000000,uid=$UID_BACKUP", + result => { uid => $UID_BACKUP, rtime => $NOW }, + # the parser drops the whole property of the backup, as it does for other members + meta => { rtime => $NOW, uid => $UID_BACKUP }, + output => qr/^guest identity: adopted uid $UID_BACKUP from the backup$/m, + }, + { + name => 'in place, backup of another guest', + target => $TARGET_CONF, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + error => $REFUSED, + }, + { + name => 'in place, backup of another guest, unique', + target => $TARGET_CONF, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + param => { unique => 1 }, + error => $REFUSED, + }, + { + name => 'in place, backup of another guest, override', + target => $TARGET_CONF, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + param => { 'allow-identity-mismatch' => 1 }, + result => { uid => $UID_TARGET, rtime => 1700000000000 }, + output => qr/^\Q$IGNORED\E\n(?:.*\n)*guest identity: kept uid $UID_TARGET$/m, + }, + { + name => 'in place, backup of the same guest', + target => $TARGET_CONF, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_TARGET", + result => { uid => $UID_TARGET, rtime => 1700000000000 }, + output => $KEPT_TARGET, + }, + { + name => 'in place, backup without uid', + target => $TARGET_CONF, + backup => $BACKUP_META, + result => { uid => $UID_TARGET, rtime => 1700000000000 }, + output => $KEPT_TARGET, + }, + { + name => 'in place, target without uid', + target => $TARGET_CONF_NO_UID, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + result => { uid => $UID_BACKUP, rtime => $NOW }, + output => qr/^guest identity: adopted uid $UID_BACKUP from the backup$/m, + }, + { + name => 'in place, target without uid, unique', + target => $TARGET_CONF_NO_UID, + backup => "$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP", + param => { unique => 1 }, + result => { uid => 'generated', rtime => $NOW }, + output => qr/^guest identity: new uid aaaaaaaaaaaaaaaaaaab$/m, + }, +]; + +sub meta_of { + my ($vmid) = @_; + + my $conf = PVE::QemuConfig->load_config($vmid); + return PVE::QemuServer::MetaInfo::parse_meta_info($conf->{meta}); +} + +for my $type (@IDENTITY_ARCHIVES) { + for my $case ($cases->@*) { + subtest "$type: $case->{name}" => sub { + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::now_ms = $NOW; + QemuServerTest::FakeCFS::set_config(100, $case->{target}) if $case->{target}; + $QemuServerTest::FakeCFS::other_guests->{300} = $case->{other} if $case->{other}; + $QemuServerTest::FakeCFS::lock_domain_error = $case->{lock_error}; + + my $param = { ($case->{param} // {})->%* }; + $param->{force} = 1 if $case->{target}; + my ($err, $output) = restore_api($type, backup_config($case->{backup}), $param); + + if (my $expected_err = $case->{error}) { + is($err, $expected_err, 'restore refused'); + is( + $QemuServerTest::FakeCFS::configs->{100}, + $case->{target}, + 'config unchanged', + ); + is_deeply($calls, {}, 'replication state and firewall config untouched'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM'); + return; + } + + is($err, '', 'restore succeeded') or diag($output); + like($output, $case->{output}, 'task log reports the identity'); + + my $expected = { $case->{result}->%* }; + $expected->{uid} = QemuServerTest::FakeCFS::generated_uid(1) + if $expected->{uid} eq 'generated'; + is_deeply( + QemuServerTest::FakeCFS::stored_identity(100), + $expected, + 'stored identity', + ); + is_deeply( + meta_of(100), + $case->{meta} + // { 'creation-qemu' => '9.2.0', ctime => 1600000000, $expected->%* }, + 'other members of meta come from the backup', + ); + }; + } +} + +subtest 'retry of a failed restore to a free VMID keeps the lineage' => sub { + QemuServerTest::FakeCFS::reset(); + my $backup = backup_config("$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP"); + + $fail_at = 'data'; + my ($err) = restore_api('pbs', $backup, {}); + is($err, "restoring the data failed\n", 'first attempt failed'); + + # the web interface sets 'force' for a retry + $fail_at = ''; + ($err) = restore_api('pbs', $backup, { force => 1 }); + is($err, '', 'retry succeeds'); + is(QemuServerTest::FakeCFS::stored_identity(100)->{uid}, $UID_BACKUP, 'uid adopted'); +}; + +subtest 'live restore adopts the uid of the backup only when complete' => sub { + my $backup = backup_config("$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP"); + my $template = "$backup" . "template: 1\n"; + + for my $failure ('before the live phase', 'during the live phase') { + QemuServerTest::FakeCFS::reset(); + + $fail_at = $failure eq 'during the live phase' ? 'live' : ''; + my ($err, $output) = restore_api( + 'pbs', + $failure eq 'before the live phase' ? $template : $backup, + { 'live-restore' => 1 }, + ); + ok($err, "restore fails $failure"); + isnt( + QemuServerTest::FakeCFS::stored_identity(100)->{uid}, + $UID_BACKUP, + "leftover VM does not have the uid of the backup ($failure)", + ); + unlike($output, qr/^guest identity:/m, "no identity reported ($failure)"); + + $fail_at = ''; + ($err, $output) = restore_api('pbs', $backup, { vmid => 101, 'live-restore' => 1 }); + is($err, '', "retry at another VMID succeeds ($failure)") or diag($output); + is($calls->{live_restore}, 1, 'live phase ran'); + is( + QemuServerTest::FakeCFS::stored_identity(101)->{uid}, + $UID_BACKUP, + "retry adopts the uid of the backup ($failure)", + ); + like( + $output, + qr/^guest identity: adopted uid $UID_BACKUP from the backup$/m, + "adoption reported ($failure)", + ); + } +}; + +# A VM that a restore leaves behind keeps the uid of the backup as pending uid, so that retrying +# the restore over it takes the uid over. + +my $BACKUP_WITH_UID = backup_config("$BACKUP_META,rtime=1600000000000,uid=$UID_BACKUP"); +my $ADOPTED = qr/^guest identity: adopted uid $UID_BACKUP from the backup$/m; + +sub pending_uid_of { + my ($vmid) = @_; + + return PVE::GuestIdentity::get_pending_uid(PVE::QemuConfig->load_config($vmid)); +} + +sub stored_uid_of { + my ($vmid) = @_; + + my $identity = QemuServerTest::FakeCFS::stored_identity($vmid); + return $identity ? $identity->{uid} : undef; +} + +# Fails the live restore of a backup with uid to the free VMID 100 in its live phase. +sub failed_live_restore { + QemuServerTest::FakeCFS::reset(); + $QemuServerTest::FakeCFS::now_ms = $NOW; + + local $fail_at = 'live'; + my ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { 'live-restore' => 1 }); + is($err, "live restore failed\n", 'live restore failed in its live phase'); + unlike($output, qr/dropping pending uid/, 'starting the VM keeps the pending uid'); + isnt(stored_uid_of(100), $UID_BACKUP, 'leftover VM has a uid of its own'); + is(pending_uid_of(100), $UID_BACKUP, 'leftover VM has the uid of the backup as pending uid'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'leftover VM is not locked'); +} + +subtest 'retry in place after a failed live restore' => sub { + for my $retry ({ 'live-restore' => 1 }, {}) { + my $what = $retry->{'live-restore'} ? 'live' : 'regular'; + + failed_live_restore(); + + # the web interface restores from the backup list of the VM with 'force' + my ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { force => 1, $retry->%* }); + is($err, '', "$what retry succeeds") or diag($output); + like($output, $ADOPTED, "$what retry adopts the uid of the backup"); + is(stored_uid_of(100), $UID_BACKUP, "$what retry: stored"); + is(pending_uid_of(100), undef, "$what retry: pending uid is removed"); + } +}; + +subtest 'retry in place after a failure once the restored volumes stay' => sub { + QemuServerTest::FakeCFS::reset(); + my $backup = $BACKUP_WITH_UID . "hostpci0: 0000:01:00.0\n"; + + # the permission check needs the restored config, so it runs once the data is restored + $rpcenv->set_user('user@pve'); + my ($err, $output) = restore_api('vma file', $backup, {}); + $rpcenv->set_user('root@pam'); + is( + $err, + "only root can set 'hostpci0' config for non-mapped devices\n", + 'restore refused after restoring the data', + ); + ok(defined($QemuServerTest::FakeCFS::configs->{100}), 'VM is left behind'); + is(pending_uid_of(100), $UID_BACKUP, + 'leftover VM has the uid of the backup as pending uid'); + + ($err, $output) = restore_api('vma file', $backup, { force => 1 }); + is($err, '', 'retry succeeds') or diag($output); + is(stored_uid_of(100), $UID_BACKUP, 'retry adopts the uid of the backup'); + is(pending_uid_of(100), undef, 'retry: pending uid is removed'); +}; + +my $FAILURES = { + # right after the check, before the restore writes a config itself + firewall => "unable to restore the firewall config\n", + data => "restoring the data failed\n", +}; + +for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { + for my $failure (sort keys $FAILURES->%*) { + my $what = "$type: retry in place after a failed restore over a VM without uid"; + + subtest "$what ($failure)" => sub { + QemuServerTest::FakeCFS::reset(); + QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF_NO_UID); + + local $fail_at = $failure; + my ($err) = restore_api($type, $BACKUP_WITH_UID, { force => 1 }); + is($err, $FAILURES->{$failure}, 'restore failed'); + ok(stored_uid_of(100), 'VM got a uid of its own'); + isnt(stored_uid_of(100), $UID_BACKUP, 'not the one of the backup'); + is(pending_uid_of(100), $UID_BACKUP, 'VM has the uid of the backup as pending uid'); + + $fail_at = ''; + ($err, my $output) = restore_api($type, $BACKUP_WITH_UID, { force => 1 }); + is($err, '', 'retry succeeds') or diag($output); + is(stored_uid_of(100), $UID_BACKUP, 'retry adopts the uid of the backup'); + is(pending_uid_of(100), undef, 'retry: pending uid is removed'); + }; + } +} + +subtest 'retry elsewhere, then over the VM left behind' => sub { + failed_live_restore(); + my $leftover = stored_uid_of(100); + + my ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { vmid => 101 }); + is($err, '', 'restore to another VMID succeeds') or diag($output); + is(stored_uid_of(101), $UID_BACKUP, 'pending uid of the leftover VM is not in use'); + is(pending_uid_of(100), $UID_BACKUP, 'leftover VM keeps its pending uid'); + + ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { force => 1 }); + is($err, '', 'restore over the leftover VM is not refused') or diag($output); + like( + $output, + qr/^guest identity: backup uid $UID_BACKUP is in use by guest 101, keeping uid $leftover$/m, + 'uid of the backup is in use by now', + ); + is(stored_uid_of(100), $leftover, 'leftover VM keeps its uid'); + is(pending_uid_of(100), undef, 'pending uid is removed'); +}; + +subtest 'using the VM left behind drops its pending uid' => sub { + failed_live_restore(); + my $leftover = stored_uid_of(100); + + # like a start, or a config change of the stopped VM + my $warnings = ''; + { + local $SIG{__WARN__} = sub { $warnings .= $_[0]; }; + my $conf = PVE::QemuConfig->load_config(100); + PVE::QemuServer::vmconfig_apply_pending(100, $conf, {}); + } + my $dropped = "guest identity: dropping pending uid $UID_BACKUP of an unfinished restore" + . " - a retry now needs 'allow-identity-mismatch'"; + like($warnings, qr/^\Q$dropped\E$/m, 'dropped'); + is(pending_uid_of(100), undef, 'pending uid is gone'); + + my ($err) = restore_api('pbs', $BACKUP_WITH_UID, { force => 1 }); + like( + $err, + qr/^backup belongs to another guest: its uid .$UID_BACKUP. differs/, + 'retry is refused', + ); + is(stored_uid_of(100), $leftover, 'VM stays its own guest'); +}; + +subtest 'a restore with unique keeps a pending uid, but never takes it over' => sub { + for my $retry ({ unique => 1 }, {}) { + my $what = $retry->{unique} ? 'retry with unique' : 'retry without unique'; + QemuServerTest::FakeCFS::reset(); + + # registering the addresses of the new MACs is the last step before the adoption + local $fail_at = 'after-restore'; + my ($err) = restore_api('pbs', $BACKUP_WITH_UID, { unique => 1 }); + is($err, "unable to register IPs\n", 'restore with unique failed at its end'); + my $leftover = stored_uid_of(100); + is(pending_uid_of(100), $UID_BACKUP, 'VM left behind has the pending uid'); + + $fail_at = ''; + my $output; + ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { force => 1, $retry->%* }); + is($err, '', "$what succeeds") or diag($output); + is(pending_uid_of(100), undef, "$what: pending uid is removed"); + if ($retry->{unique}) { + is(stored_uid_of(100), $leftover, "$what: uid of the backup is not taken over"); + } else { + is(stored_uid_of(100), $UID_BACKUP, "$what: uid of the backup is taken over"); + } + } +}; + +subtest 'a failed adoption keeps the pending uid' => sub { + QemuServerTest::FakeCFS::reset(); + + $QemuServerTest::FakeCFS::lock_domain_error = + "cfs-lock 'domain-guest-identity' error: no quorum!\n"; + my ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, {}); + $QemuServerTest::FakeCFS::lock_domain_error = undef; + is($err, '', 'restore succeeds') or diag($output); + my ($kept) = $output =~ $ADOPT_FAILED; + ok(defined($kept), 'failed adoption is a warning') or diag($output); + isnt(stored_uid_of(100), $UID_BACKUP, 'VM has a uid of its own'); + is($kept, stored_uid_of(100), 'warning names the uid that the VM keeps'); + is(pending_uid_of(100), $UID_BACKUP, 'nothing decided, pending uid stays'); + + ($err, $output) = restore_api('pbs', $BACKUP_WITH_UID, { force => 1 }); + is($err, '', 'restore of the backup again is not refused') or diag($output); + is(stored_uid_of(100), $UID_BACKUP, 'uid of the backup is taken over'); +}; + +subtest 'config of a VM left behind shows its stored identity' => sub { + failed_live_restore(); + my $meta = PVE::QemuConfig->load_config(100)->{meta}; + + my $conf = PVE::API2::Qemu->vm_config({ node => 'localhost', vmid => 100 }); + is($conf->{meta}, $meta, 'config with pending changes applied: stored identity'); + $conf = PVE::API2::Qemu->vm_config({ node => 'localhost', vmid => 100, current => 1 }); + is($conf->{meta}, $meta, 'current config: stored identity'); + + my $pending = PVE::API2::Qemu->vm_pending({ node => 'localhost', vmid => 100 }); + my ($item) = grep { $_->{key} eq 'meta' } $pending->@*; + is($item->{value}, $meta, 'pending view: current value'); + is($item->{pending}, "uid=$UID_BACKUP", 'pending view: pending uid'); +}; + +subtest 'removing the pending uid keeps an identity that the parser cannot read' => sub { + QemuServerTest::FakeCFS::reset(); + + # a member of a newer version makes the parser drop the whole property + my $target_meta = "creation-qemu=10.0.0,ctime=1700000000,future-member=1," + . "rtime=1700000000000,uid=$UID_TARGET"; + QemuServerTest::FakeCFS::set_config( + 100, + "memory: 512\nmeta: $target_meta\n\n[PENDING]\nmeta: uid=$UID_BACKUP\n", + ); + my $reservation = { + existed => 1, + previous => { uid => $UID_TARGET, rtime => 1700000000000 }, + 'previous-pending-uid' => $UID_BACKUP, + assigned => { uid => $UID_TARGET, rtime => 1700000000000 }, + }; + my $res; + { + local $SIG{__WARN__} = sub { }; + $res = + PVE::GuestIdentity::adopt('PVE::QemuConfig', 100, 'cccccccccccccccccccc', $reservation); + } + is($res->{result}, 'kept', 'identity of another backup is not taken over'); + is(stored_uid_of(100), $UID_TARGET, 'stored identity is kept'); + is(pending_uid_of(100), undef, 'pending uid is removed'); +}; + +# vzdump never archives pending changes, a crafted backup must not bring any along +for my $type (sort keys $ARCHIVES->%*) { + subtest "$type: pending changes of the backup are dropped" => sub { + QemuServerTest::FakeCFS::reset(); + + my $crafted = $BACKUP_WITH_UID + . "\n[PENDING]\nargs: -device foo\nhostpci0: 0000:01:00.0\nmeta: uid=$UID_TARGET\n"; + my ($err, $output) = restore_api($type, $crafted, {}); + is($err, '', 'restore succeeds') or diag($output); + my @writes = $QemuServerTest::FakeCFS::written->{100}->@*; + ok(!grep({ m/^(args|hostpci0):|uid=$UID_TARGET/m } @writes), 'no write has them'); + unlike( + $QemuServerTest::FakeCFS::configs->{100}, + qr/^\[PENDING\]/m, + 'no pending section', + ); + }; +} + +# Once a write of the config of VMID 100 has the pending uid of the backup, all later writes of the +# restore have it, so that the VM keeps it whenever the restore stops. Only the adoption removes it. +for my $type (@IDENTITY_ARCHIVES, 'pbs live') { + for my $target (undef, $TARGET_CONF_NO_UID) { + my $what = $type . ($target ? ', in place over a VM without uid' : ', free VMID'); + + subtest "$what: every write of the restore keeps the pending uid" => sub { + QemuServerTest::FakeCFS::reset(); + QemuServerTest::FakeCFS::set_config(100, $target) if $target; + + my $param = { $target ? (force => 1) : () }; + $param->{'live-restore'} = 1 if $type eq 'pbs live'; + my ($err, $output) = restore_api($type =~ s/ live$//r, $BACKUP_WITH_UID, $param); + is($err, '', 'restore succeeds') or diag($output); + like($output, $ADOPTED, 'uid of the backup is adopted'); + + my @writes = $QemuServerTest::FakeCFS::written->{100}->@*; + my $last = pop(@writes); + my $first = 0; + $first++ while $first < @writes && $writes[$first] !~ m/^\[PENDING\]/m; + ok($first < @writes, 'pending uid is written') or return; + is( + scalar( + grep { m/^\[PENDING\]\nmeta: uid=$UID_BACKUP$/m } + @writes[$first .. $#writes] + ), + scalar(@writes) - $first, + 'all later writes keep it', + ); + unlike($last, qr/^\[PENDING\]/m, 'adoption removes it'); + }; + } +} + +$fail_at = ''; + done_testing(); -- 2.47.3