From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 226231FF0E1 for ; Mon, 16 Nov 2026 01:51:21 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B878F219E4; Mon, 16 Nov 2026 01:48:42 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server 12/34] api: restore: release the VMID if a restore to an unused VMID fails Date: Mon, 16 Nov 2026 01:46:24 +0100 Message-ID: <20261116004703.2908294-13-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261116004703.2908294-1-t.lamprecht@proxmox.com> References: <20261116004703.2908294-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794790029455 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.166 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) PROLO_LEO2 0.1 Meta Catches all Leo drug variations so far RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LT2KYFLF4VLD3R2242B7BJMUSUJFL32G X-Message-ID-Hash: LT2KYFLF4VLD3R2242B7BJMUSUJFL32G X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A failed restore to a previously unused VMID left its temporary or empty config behind, even when the restore cleaned up the volumes it had allocated. The VMID stayed taken and a retry needed 'force'. Containers already remove their config in this case. Remove the VM config and the restored firewall config if the VMID was unused before the operation and the restore has not reached the point from which it keeps the restored volumes. Keep the VM after later failures, such as a failure in the live phase of a live restore. Remove the config before releasing the guest config lock. The temporary config of a VMA restore has no lock property, so removing it later could delete the config of another restore that took the VMID in the meantime. Only warn that the VM was not cleaned up if the restore changed the target and left it behind. A failure to read the backup should not produce that warning. Signed-off-by: Thomas Lamprecht --- src/PVE/API2/Qemu.pm | 28 +- src/PVE/QemuServer.pm | 8 + src/test/QemuServerTest/FakeCFS.pm | 194 ++++++++++++ src/test/run_qemu_restore_config_tests.pl | 358 ++++++++++++++++++++-- 4 files changed, 567 insertions(+), 21 deletions(-) create mode 100644 src/test/QemuServerTest/FakeCFS.pm diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index b9634d20..504c9768 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -1413,10 +1413,12 @@ __PACKAGE__->register_method({ my $emsg = $is_restore ? "unable to restore VM $vmid -" : "unable to create VM $vmid -"; - eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) }; + my $reservation = eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) }; die "$emsg $@" if $@; my $restored_data = 0; + # the restore paths record whether they changed the VM and whether restored volumes stay + my $restore_progress = {}; my $restorefn = sub { my $conf = PVE::QemuConfig->load_config($vmid); @@ -1432,6 +1434,7 @@ __PACKAGE__->register_method({ bwlimit => $bwlimit, live => $live_restore, override_conf => $param, + progress => $restore_progress, }; if (my $volid = $archive->{volid}) { # best effort, real check is after restoring! @@ -1489,7 +1492,24 @@ __PACKAGE__->register_method({ if $unique; }; - PVE::QemuConfig->lock_config_full($vmid, 1, $realcmd); + my $restore_locked = sub { + eval { $realcmd->() }; + my $err = $@ or return; + + if (!$reservation->{existed} && !$restore_progress->{'volumes-kept'}) { + # Remove the reservation before releasing the guest lock, so another + # operation cannot take ownership first. + eval { + PVE::QemuConfig->destroy_config($vmid); + $restore_progress->{'reservation-removed'} = 1; + PVE::Firewall::remove_vmfw_conf($vmid); + }; + warn $@ if $@; + } + die $err; + }; + + PVE::QemuConfig->lock_config_full($vmid, 1, $restore_locked); if ($start_after_create && !$live_restore) { print "Execute autostart\n"; @@ -1622,13 +1642,15 @@ __PACKAGE__->register_method({ $code = sub { eval { $restorefn->() }; if (my $err = $@) { + die $err if $restore_progress->{'reservation-removed'}; + eval { PVE::QemuConfig->remove_lock($vmid, 'create') }; warn $@ if $@; if ($restored_data) { warn "error after data was restored, VM disks should be OK but config may " . "require adaptions. VM $vmid state is NOT cleaned up.\n"; - } else { + } elsif ($restore_progress->{'target-changed'}) { warn "error before or during data restore, some or all disks were not " . "completely restored. VM $vmid state is NOT cleaned up.\n"; } diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index dcd439f3..9ffc11e8 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -6948,6 +6948,7 @@ sub restore_proxmox_backup_archive { $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); if ($has_firewall_config) { @@ -7042,6 +7043,7 @@ sub restore_proxmox_backup_archive { $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; if ($options->{live}) { # keep lock during live-restore @@ -7129,6 +7131,7 @@ sub restore_external_archive { my $data = $backup_provider->archive_get_guest_config($volname) or die "backup provider failed to extract guest configuration\n"; + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); PVE::Tools::file_set_contents($cfgfn, $data); @@ -7218,6 +7221,7 @@ sub restore_external_archive { $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7543,6 +7547,7 @@ sub restore_vma_archive { my $print_devmap = sub { my $cfgfn = "$tmpdir/qemu-server.conf"; + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # we can read the config - that is already extracted @@ -7676,6 +7681,7 @@ sub restore_vma_archive { $restore_destroy_volumes->($cfg, $virtdev_hash); die $err; } + $opts->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $opts->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7703,6 +7709,7 @@ sub restore_tar_archive { my $storecfg = PVE::Storage::config(); + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # avoid zombie disks when restoring over an existing VM -> cleanup first @@ -7785,6 +7792,7 @@ sub restore_tar_archive { tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat") if !$opts->{info}; die $err; } + $opts->{progress}->{'volumes-kept'} = 1; rmtree $tmpdir; diff --git a/src/test/QemuServerTest/FakeCFS.pm b/src/test/QemuServerTest/FakeCFS.pm new file mode 100644 index 00000000..dd9dddec --- /dev/null +++ b/src/test/QemuServerTest/FakeCFS.pm @@ -0,0 +1,194 @@ +package QemuServerTest::FakeCFS; + +# Fake cluster filesystem for VM config files, so that tests can use the real config class methods +# of PVE::QemuConfig, including the guest identity handling of write_config, without pmxcfs. +# +# Configs are kept as raw text, indexed by VMID, and an empty file at the path that config_file() +# returns exists for each, for checks of the file's existence. Generated guest identities are +# deterministic: the n-th generated uid is 'a' repeated 13 times followed by the base32 encoding of +# n, and rtime is $now_ms. + +use strict; +use warnings; + +use File::Basename qw(dirname); +use File::Path qw(make_path remove_tree); +use File::Temp qw(tempdir); +use Test::MockModule; + +use PVE::GuestHelpers; +use PVE::GuestIdentity; +use PVE::QemuConfig; +use PVE::QemuServer; + +our $configs = {}; # VMID => raw config, for VMs on the local node +our $other_guests = {}; # VMID => raw 'meta' property of other guests, like on other nodes or CTs +our $now_ms = 1700000000000; +our $generated = 0; +our $lock_domain_error; +our $writes = []; + +my $tmpdir = tempdir('fake-cfs-XXXXXX', DIR => '.', CLEANUP => 1); +make_path("$tmpdir/lock"); +$PVE::GuestHelpers::lockdir = "$tmpdir/lock"; + +sub reset { + remove_tree("$tmpdir/etc-pve"); + $configs = {}; + $other_guests = {}; + $now_ms = 1700000000000; + $generated = 0; + $lock_domain_error = undef; + $writes = []; +} + +my sub vmid_of_path { + my ($path) = @_; + + return $path =~ m|/qemu-server/(\d+)\.conf$| ? $1 : undef; +} + +# Creates or removes the file at the path of the config of $vmid, matching the stored config. +my sub sync_config_file { + my ($vmid) = @_; + + my $path = PVE::QemuConfig->config_file($vmid); + if (defined($configs->{$vmid})) { + make_path(dirname($path)); + open(my $fh, '>', $path) or die "unable to create '$path' - $!\n"; + close($fh); + } else { + unlink($path); + } +} + +# Stores the raw config $raw for $vmid, like a manual edit of the config file. +sub set_config { + my ($vmid, $raw) = @_; + + $configs->{$vmid} = $raw; + sync_config_file($vmid); +} + +# Returns the last 'meta' property of the top-level section of the raw config $raw, like the config +# class reads the published identity. +my sub top_level_meta { + my ($raw) = @_; + + my $meta; + for my $line (split(/\n/, $raw // '')) { + last if $line =~ m/^\[/; + $meta = $1 if $line =~ m/^meta:\s*(.*?)\s*$/; + } + + return $meta; +} + +# Returns the identity published in the raw config of $vmid, undef if there is none. +sub published_identity { + my ($vmid) = @_; + + return PVE::GuestIdentity::parse_identity(top_level_meta($configs->{$vmid})); +} + +# Returns the uid that the $n-th generation of an identity yields. +sub generated_uid { + my ($n) = @_; + + my $alphabet = 'abcdefghijklmnopqrstuvwxyz234567'; + return ('a' x 17) . join('', map { substr($alphabet, ($n >> (5 * $_)) & 31, 1) } (2, 1, 0)); +} + +our $guest_identity_module = Test::MockModule->new('PVE::GuestIdentity'); +$guest_identity_module->mock( + __random_bytes => sub { + my ($len) = @_; + die "unexpected length $len\n" if $len != 13; + $generated++; + # 85 zero bits encode as 17 'a', then the 15 bits of the counter, the last 4 bits are unused + my $counter = substr(unpack('B32', pack('N', $generated)), -15); + return pack('B104', ('0' x 85) . $counter . '0000'); + }, + __time_ms => sub { + return $now_ms; + }, +); + +our $cluster_module = Test::MockModule->new('PVE::Cluster'); +$cluster_module->mock( + get_config => sub { + my ($path) = @_; + my $vmid = vmid_of_path($path) // die "get_config (mocked) - unexpected path '$path'\n"; + return $configs->{$vmid}; + }, + cfs_read_file => sub { + my ($path) = @_; + return {} if $path eq 'datacenter.cfg'; + my $vmid = vmid_of_path($path) + // die "cfs_read_file (mocked) - unexpected path '$path'\n"; + return PVE::QemuServer::parse_vm_config($path, $configs->{$vmid}); + }, + cfs_write_file => sub { + my ($path, $conf) = @_; + my $vmid = vmid_of_path($path) + // die "cfs_write_file (mocked) - unexpected path '$path'\n"; + $configs->{$vmid} = PVE::QemuServer::write_vm_config($path, $conf); + sync_config_file($vmid); + push $writes->@*, $vmid; + return; + }, + cfs_update => sub { + return; + }, + check_vmid_unused => sub { + my ($vmid, $noerr) = @_; + return 1 if !defined($configs->{$vmid}) && !defined($other_guests->{$vmid}); + return undef if $noerr; + die "VM $vmid already exists\n"; + }, + cfs_lock_domain => sub { + my ($domain, $timeout, $code, @param) = @_; + die "cfs_lock_domain (mocked) - unexpected domain '$domain'\n" + if $domain ne 'guest-identity'; + if ($lock_domain_error) { + $@ = $lock_domain_error; + return undef; + } + my $res = eval { $code->(@param) }; + return $res; + }, + get_guest_config_properties => sub { + my ($properties) = @_; + die "get_guest_config_properties (mocked) - unexpected properties\n" + if join(',', $properties->@*) ne 'meta'; + + my $res = {}; + for my $vmid (keys $configs->%*) { + my $meta = top_level_meta($configs->{$vmid}); + $res->{$vmid} = { meta => $meta } if defined($meta); + } + $res->{$_} = { meta => $other_guests->{$_} } for keys $other_guests->%*; + return $res; + }, +); + +our $qemu_config_module = Test::MockModule->new('PVE::QemuConfig'); +$qemu_config_module->mock( + config_file => sub { + my ($class, $vmid, $node) = @_; + return "$tmpdir/etc-pve/" . $class->cfs_config_path($vmid, $node); + }, + config_file_lock => sub { + my ($class, $vmid) = @_; + return "$tmpdir/lock/lock-$vmid.conf"; + }, + destroy_config => sub { + my ($class, $vmid) = @_; + die "failed to remove config file: No such file or directory\n" + if !defined($configs->{$vmid}); + delete $configs->{$vmid}; + sync_config_file($vmid); + }, +); + +1; diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl index 20cb3df0..e6739d66 100755 --- a/src/test/run_qemu_restore_config_tests.pl +++ b/src/test/run_qemu_restore_config_tests.pl @@ -3,35 +3,30 @@ use strict; use warnings; -use lib qw(..); +use lib qw(.. .); +use Fcntl qw(:flock O_RDONLY O_NONBLOCK); +use File::Path qw(make_path); +use File::Temp qw(tempdir); +use JSON; use Test::MockModule; use Test::More; use File::Basename; +use PVE::API2::Qemu; +use PVE::QemuConfig; use PVE::QemuServer; -use PVE::Tools qw(dir_glob_foreach file_get_contents); +use PVE::QemuServer::Helpers; +use PVE::RPCEnvironment; +use PVE::Tools qw(dir_glob_foreach file_get_contents file_set_contents); + +# also provides the 'datacenter.cfg' that restore_update_config_line reads +use QemuServerTest::FakeCFS; my $INPUT_DIR = './restore-config-input'; my $EXPECTED_DIR = './restore-config-expected'; -# NOTE update when you add/remove tests -plan tests => 5; - -my $pve_cluster_module = Test::MockModule->new("PVE::Cluster"); -$pve_cluster_module->mock( - cfs_read_file => sub { - my ($file) = @_; - - if ($file eq 'datacenter.cfg') { - return {}; - } else { - die "'cfs_read_file' called - missing mock?\n"; - } - }, -); - dir_glob_foreach( './restore-config-input', '[0-9]+.conf', @@ -78,4 +73,331 @@ dir_glob_foreach( }, ); +# Restores through the create API and the restore paths for PBS, backup providers, VMA and legacy +# tar archives, against the fake cluster filesystem. Storage, archive tools and other external +# effects are mocked. The backups have no disks, so that no volumes get allocated. + +my $runtime_dir = tempdir('restore-run-XXXXXX', DIR => '.', CLEANUP => 1); +$PVE::QemuServer::Helpers::var_run_tmpdir = $runtime_dir; + +my $rpcenv = PVE::RPCEnvironment->init('cli'); +$rpcenv->set_user('root@pam'); + +my $ARCHIVES = { + pbs => 'pbs:backup/vm/100/2025-01-01T00:00:00Z', + provider => 'provider:backup/vm-100-2025-01-01', + 'vma file' => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.vma', + 'vma stdin' => '-', + tar => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.tar', +}; + +our $current_backup; # raw config of the backup +our $fail_at = ''; # where the restore fails: 'read-config', 'data', 'after-restore' or 'live' +my $calls = {}; # calls with effects on the target VM that tests check + +sub backup_config { + my ($meta) = @_; + + return + "boot: order=scsi0\ncores: 1\nmemory: 512\n" + . (defined($meta) ? "meta: $meta\n" : '') + . "name: backup\n"; +} + +package TestBackupProvider { + + sub new { + my ($class) = @_; + return bless {}, $class; + } + + sub restore_get_mechanism { return ('qemu-img', 'qemu'); } + sub restore_vm_init { return {}; } + + sub archive_get_guest_config { + die "unable to read the backup\n" if $main::fail_at eq 'read-config'; + return $main::current_backup; + } + sub archive_get_firewall_config { return "[OPTIONS]\nenable: 1\n"; } + sub restore_vm_cleanup { return; } +} + +my $replication_state_module = Test::MockModule->new('PVE::ReplicationState'); +$replication_state_module->mock(delete_guest_states => sub { $calls->{delete_guest_states}++; }); + +my $firewall_module = Test::MockModule->new('PVE::Firewall'); +$firewall_module->mock(remove_vmfw_conf => sub { $calls->{remove_vmfw_conf}++; }); + +my $tools_module = Test::MockModule->new('PVE::Tools'); +$tools_module->mock( + file_copy => sub { + my ($src, $dst) = @_; + die "file_copy (mocked) - unexpected destination '$dst'\n" if $dst !~ m|/firewall/|; + $calls->{firewall}++; + }, +); + +my $storage_module = Test::MockModule->new('PVE::Storage'); +$storage_module->mock( + abs_filesystem_path => sub { + my ($cfg, $path) = @_; + return $path; + }, + check_volume_access => sub { return; }, + config => sub { return {}; }, + deactivate_volumes => sub { return 1; }, + extract_vzdump_config => sub { return $current_backup; }, + new_backup_provider => sub { return TestBackupProvider->new(); }, + parse_volname => sub { + return ('backup', 'vm/100/2025-01-01T00:00:00Z', 100, undef, undef, undef, 'pbs-vm'); + }, + parse_volume_id => sub { + my ($volid, $noerr) = @_; + return ($1, $2) if $volid =~ m/^([a-z]+):(.+)$/; + die "unable to parse volume ID '$volid'\n" if !$noerr; + return; + }, + path_to_volume_id => sub { return; }, + storage_config => sub { + my ($cfg, $storeid) = @_; + return { type => 'pbs', server => 'localhost', datastore => 'ds' } if $storeid eq 'pbs'; + return { type => 'dir', path => '/mnt/provider' }; + }, + storage_has_feature => sub { + my ($cfg, $storeid, $feature) = @_; + return $storeid eq 'provider' && $feature eq 'backup-provider'; + }, +); + +my $pbs_plugin_module = Test::MockModule->new('PVE::Storage::PBSPlugin'); +$pbs_plugin_module->mock( + pbs_encryption_key_file_name => sub { return '/nonexistent/key'; }, + pbs_get_password => sub { return 'secret'; }, + run_raw_client_cmd => sub { + my ($scfg, $storeid, $cmd, $param) = @_; + my (undef, $file, $target) = $param->@*; + if ($file eq 'index.json') { + my $files = + [{ filename => 'qemu-server.conf.blob' }, { filename => 'fw.conf.blob' }]; + file_set_contents($target, encode_json({ files => $files })); + } elsif ($file eq 'qemu-server.conf') { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents($target, $current_backup); + } elsif ($file eq 'fw.conf') { + file_set_contents($target, "[OPTIONS]\nenable: 1\n"); + } else { + die "run_raw_client_cmd (mocked) - unexpected file '$file'\n"; + } + }, +); + +my $network_module = Test::MockModule->new('PVE::QemuServer::Network'); +$network_module->mock( + create_ifaces_ipams_ips => sub { + die "unable to register IPs\n" if $fail_at eq 'after-restore'; + }, +); + +my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); +$qemu_server_module->mock( + check_running => sub { return 0; }, + pbs_live_restore => sub { + die "live restore failed\n" if $fail_at eq 'live'; + $calls->{live_restore}++; + }, + rescan => sub { return; }, + restore_update_config_line => sub { + die "restoring the data failed\n" if $fail_at eq 'data'; + return $qemu_server_module->original('restore_update_config_line')->(@_); + }, + tar_archive_read_firstfile => sub { return 'qemu-server.conf'; }, + # like 'tar' with qmextract and 'vma extract': write the config and report the VMA header + run_command => sub { + my ($cmd, %param) = @_; + + if (!ref($cmd) && $cmd =~ m/^zcat -f\|tar xf /) { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents("$ENV{VZDUMP_TMPDIR}/qemu-server.conf", $current_backup); + return 0; + } + + my $vma = $cmd->[-1]; + die "run_command (mocked) - unexpected command\n" + if ref($vma) ne 'ARRAY' || $vma->[0] ne 'vma'; + my ($mapfifo, $tmpdir) = $vma->@[4, 6]; + + die "unable to read the backup\n" if $fail_at eq 'read-config'; + + make_path($tmpdir); + file_set_contents("$tmpdir/qemu-server.conf", $current_backup); + file_set_contents("$tmpdir/qemu-server.fw", "[OPTIONS]\nenable: 1\n"); + + sysopen(my $reader, $mapfifo, O_RDONLY | O_NONBLOCK) + or die "unable to open fifo - $!\n"; + $param{afterfork}->(); + $param{outfunc}->('CTIME: Wed Jan 1 00:00:00 2025'); + close($reader); + + return 0; + }, +); + +my $task_error; +my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment'); +$rpcenv_module->mock( + check => sub { return 1; }, + fork_worker => sub { + my ($self, $type, $id, $user, $code) = @_; + eval { $code->('UPID:test') }; + $task_error = $@; + return 'UPID:test'; + }, +); + +$QemuServerTest::FakeCFS::cluster_module->mock(check_cfs_quorum => sub { return 1; }); + +# Restores a backup with the raw config $backup to VMID 100 through the create API, from an archive +# of type $type with the additional API parameters $param. Returns the error of the API call or the +# task and the task output. +sub restore_api { + my ($type, $backup, $param) = @_; + + $current_backup = $backup; + $calls = {}; + $task_error = ''; + + my ($err, $output) = ('', ''); + { + open(my $out_fh, '>', \$output); + local *STDERR = $out_fh; + my $oldfh = select($out_fh); + + eval { + PVE::API2::Qemu->create_vm({ + node => 'localhost', + vmid => 100, + archive => $ARCHIVES->{$type}, + $param->%*, + }); + }; + $err = $@ || $task_error; + + select($oldfh); + close($out_fh); + } + + return ($err, $output); +} + +my $TARGET_CONF = "memory: 1024\nmeta: creation-qemu=10.0.0,ctime=1700000000\nname: target\n"; +my $NOT_CLEANED_UP = qr/VM 100 state is NOT cleaned up/; + +for my $type (sort keys $ARCHIVES->%*) { + subtest "$type: failed restore to a free VMID" => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'data'; + my ($err, $output) = restore_api($type, backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + is($calls->{remove_vmfw_conf}, 1, 'restored firewall config removed'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); + + $fail_at = ''; + ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + is($err, '', 'retry succeeds') or diag($output); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'VM restored'); + }; +} + +subtest 'failed restore to a free VMID before reading the backup' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "unable to read the backup\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); +}; + +subtest 'another restore right after a failed restore to a free VMID' => sub { + QemuServerTest::FakeCFS::reset(); + + # run a second restore with 'force' as soon as the failed one releases the guest lock + my $lock_config_full = PVE::QemuConfig->can('lock_config_full'); + my ($interposed, $lock_free, $second_err) = (0, 0, ''); + $QemuServerTest::FakeCFS::qemu_config_module->mock( + lock_config_full => sub { + my ($class, $vmid, @param) = @_; + my $res = eval { $lock_config_full->($class, $vmid, @param) }; + my $err = $@; + if ($err && !$interposed++) { + open(my $fh, '>>', PVE::QemuConfig->config_file_lock($vmid)) or die "$!\n"; + $lock_free = flock($fh, LOCK_EX | LOCK_NB); + close($fh); + local $fail_at = ''; + ($second_err) = restore_api('pbs', backup_config(), { force => 1 }); + } + die $err if $err; + return $res; + }, + ); + + # the temporary config of a VMA restore has no lock that would stop the second restore + $fail_at = 'data'; + my ($err) = restore_api('vma file', backup_config(), {}); + $QemuServerTest::FakeCFS::qemu_config_module->unmock('lock_config_full'); + + is($err, "restoring the data failed\n", 'first restore failed'); + ok($lock_free, 'guest lock was free for the second restore'); + is($second_err, '', 'second restore succeeded'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'its config stays'); + is($calls->{firewall}, 1, 'its firewall config was restored'); + ok(!$calls->{remove_vmfw_conf}, 'and stays'); +}; + +for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { + subtest "$type: failed restore over an existing VM" => sub { + QemuServerTest::FakeCFS::reset(); + QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + like($err, qr/unable to read the backup\n$/, 'fails before changing the VM'); + is($QemuServerTest::FakeCFS::configs->{100}, $TARGET_CONF, 'config unchanged'); + is_deeply($calls, {}, 'replication state and firewall config untouched'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM'); + + $fail_at = 'data'; + ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + is($err, "restoring the data failed\n", 'fails while restoring'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^memory: 128$/m, 'temporary config'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, qr/error before or during data restore/, 'warning about the changed VM'); + }; +} + +subtest 'failed live restore to a free VMID keeps the restored config' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'live'; + my ($err, $output) = restore_api('pbs', backup_config(), { 'live-restore' => 1 }); + is($err, "live restore failed\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, $NOT_CLEANED_UP, 'warning about the VM left behind'); +}; + +subtest 'failure after the data was restored keeps the VM' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'after-restore'; + my ($err, $output) = restore_api('pbs', backup_config(), { unique => 1 }); + is($err, "unable to register IPs\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + like($output, qr/error after data was restored/, 'warning about the VM left behind'); +}; + +$fail_at = ''; + done_testing(); -- 2.47.3