From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 89F271FF0AA for ; Tue, 17 Nov 2026 23:44:24 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 60DA521A90; Tue, 17 Nov 2026 23:41:02 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v2 12/36] api: restore: release the VMID if a restore to an unused VMID fails Date: Tue, 17 Nov 2026 23:34:22 +0100 Message-ID: <20261117223945.862914-13-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com> References: <20261117223945.862914-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794955191358 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.195 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: D6VSZE4YCQU3YL6AUOKDZYXRPXVVN2GW X-Message-ID-Hash: D6VSZE4YCQU3YL6AUOKDZYXRPXVVN2GW X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A failed restore to a previously unused VMID left its temporary or empty config behind, even when the restore cleaned up the volumes it had allocated. The VMID stayed taken and a retry needed 'force'. Containers already remove their config in this case. Remove the VM config and the restored firewall config if the VMID was unused before the operation and the restore has not reached the point from which it keeps the restored volumes. Keep the VM after later failures, such as a failure in the live phase of a live restore. If the next-id setting enforces unique guest IDs, the released VMID counts as used, so a retry needs another VMID, as for containers. Remove the config before releasing the guest config lock. The temporary config of a VMA restore has no lock property, so removing it later could delete the config of another restore that took the VMID in the meantime. Remove the firewall config before the VM config for a similar reason: the guest config lock is local to the node, so another node can take the VMID once the VM config is gone. The lock is held, so the VM cannot run unprotected in between. Only warn that the VM was not cleaned up if the restore changed the target and left it behind. A failure to read the backup should not produce that warning. Signed-off-by: Thomas Lamprecht --- changes v1 -> v2: * remove the firewall config before the VM config, with a test for the order * note the effect with enforced unique guest IDs in the commit message * test helper: use v5.36, mock the registration of used guest IDs src/PVE/API2/Qemu.pm | 29 +- src/PVE/QemuServer.pm | 8 + src/test/QemuServerTest/FakeCFS.pm | 185 +++++++++++ src/test/run_qemu_restore_config_tests.pl | 370 ++++++++++++++++++++-- 4 files changed, 571 insertions(+), 21 deletions(-) create mode 100644 src/test/QemuServerTest/FakeCFS.pm diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index 89a3ad9d..be0928ea 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -1421,10 +1421,12 @@ __PACKAGE__->register_method({ my $emsg = $is_restore ? "unable to restore VM $vmid -" : "unable to create VM $vmid -"; - eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) }; + my $reservation = eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) }; die "$emsg $@" if $@; my $restored_data = 0; + # the restore paths record whether they changed the VM and whether restored volumes stay + my $restore_progress = {}; my $restorefn = sub { my $conf = PVE::QemuConfig->load_config($vmid); @@ -1440,6 +1442,7 @@ __PACKAGE__->register_method({ bwlimit => $bwlimit, live => $live_restore, override_conf => $param, + progress => $restore_progress, }; if (my $volid = $archive->{volid}) { # best effort, real check is after restoring! @@ -1497,7 +1500,25 @@ __PACKAGE__->register_method({ if $unique; }; - PVE::QemuConfig->lock_config_full($vmid, 1, $realcmd); + my $restore_locked = sub { + eval { $realcmd->() }; + my $err = $@ or return; + + if (!$reservation->{existed} && !$restore_progress->{'volumes-kept'}) { + # Remove the reservation before releasing the guest lock, so another + # operation cannot take ownership first. The firewall config goes first, + # before the VMID is freed. The lock is held, so the VM cannot run unprotected. + eval { + PVE::Firewall::remove_vmfw_conf($vmid); + PVE::QemuConfig->destroy_config($vmid); + $restore_progress->{'reservation-removed'} = 1; + }; + warn $@ if $@; + } + die $err; + }; + + PVE::QemuConfig->lock_config_full($vmid, 1, $restore_locked); if ($start_after_create && !$live_restore) { print "Execute autostart\n"; @@ -1630,13 +1651,15 @@ __PACKAGE__->register_method({ $code = sub { eval { $restorefn->() }; if (my $err = $@) { + die $err if $restore_progress->{'reservation-removed'}; + eval { PVE::QemuConfig->remove_lock($vmid, 'create') }; warn $@ if $@; if ($restored_data) { warn "error after data was restored, VM disks should be OK but config may " . "require adaptions. VM $vmid state is NOT cleaned up.\n"; - } else { + } elsif ($restore_progress->{'target-changed'}) { warn "error before or during data restore, some or all disks were not " . "completely restored. VM $vmid state is NOT cleaned up.\n"; } diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index cf0067c3..d8077ba0 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -6960,6 +6960,7 @@ sub restore_proxmox_backup_archive { $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); if ($has_firewall_config) { @@ -7052,6 +7053,7 @@ sub restore_proxmox_backup_archive { $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; if ($options->{live}) { # keep lock during live-restore @@ -7139,6 +7141,7 @@ sub restore_external_archive { my $data = $backup_provider->archive_get_guest_config($volname) or die "backup provider failed to extract guest configuration\n"; + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); PVE::Tools::file_set_contents($cfgfn, $data); @@ -7226,6 +7229,7 @@ sub restore_external_archive { $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7551,6 +7555,7 @@ sub restore_vma_archive { my $print_devmap = sub { my $cfgfn = "$tmpdir/qemu-server.conf"; + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # we can read the config - that is already extracted @@ -7680,6 +7685,7 @@ sub restore_vma_archive { $restore_destroy_volumes->($cfg, $virtdev_hash); die $err; } + $opts->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $opts->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7707,6 +7713,7 @@ sub restore_tar_archive { my $storecfg = PVE::Storage::config(); + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # avoid zombie disks when restoring over an existing VM -> cleanup first @@ -7789,6 +7796,7 @@ sub restore_tar_archive { tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat") if !$opts->{info}; die $err; } + $opts->{progress}->{'volumes-kept'} = 1; rmtree $tmpdir; diff --git a/src/test/QemuServerTest/FakeCFS.pm b/src/test/QemuServerTest/FakeCFS.pm new file mode 100644 index 00000000..8eacc736 --- /dev/null +++ b/src/test/QemuServerTest/FakeCFS.pm @@ -0,0 +1,185 @@ +package QemuServerTest::FakeCFS; + +# Fake cluster filesystem for VM config files, so that tests can use the real config class methods +# of PVE::QemuConfig, including the guest identity handling of write_config, without pmxcfs. +# +# Configs are kept as raw text, indexed by VMID, and an empty file at the path that config_file() +# returns exists for each, for checks of the file's existence. Generated guest identities are +# deterministic: the n-th generated uid is 'a' repeated 17 times followed by the base32 encoding of +# n, and rtime is $now_ms. + +use v5.36; + +use File::Basename qw(dirname); +use File::Path qw(make_path remove_tree); +use File::Temp qw(tempdir); +use Test::MockModule; + +use PVE::GuestHelpers; +use PVE::GuestID; +use PVE::GuestIdentity; +use PVE::QemuConfig; +use PVE::QemuServer; + +our $configs = {}; # VMID => raw config, for VMs on the local node +our $other_guests = {}; # VMID => raw 'meta' property of other guests, like on other nodes or CTs +our $now_ms = 1700000000000; +our $generated = 0; +our $lock_domain_error; +our $writes = []; + +my $tmpdir = tempdir('fake-cfs-XXXXXX', DIR => '.', CLEANUP => 1); +make_path("$tmpdir/lock"); +$PVE::GuestHelpers::lockdir = "$tmpdir/lock"; + +sub reset() { + remove_tree("$tmpdir/etc-pve"); + $configs = {}; + $other_guests = {}; + $now_ms = 1700000000000; + $generated = 0; + $lock_domain_error = undef; + $writes = []; +} + +my sub vmid_of_path($path) { + return $path =~ m|/qemu-server/(\d+)\.conf$| ? $1 : undef; +} + +# Creates or removes the file at the path of the config of $vmid, matching the stored config. +my sub sync_config_file($vmid) { + my $path = PVE::QemuConfig->config_file($vmid); + if (defined($configs->{$vmid})) { + make_path(dirname($path)); + open(my $fh, '>', $path) or die "unable to create '$path' - $!\n"; + close($fh); + } else { + unlink($path); + } +} + +# Stores the raw config $raw for $vmid, like a manual edit of the config file. +sub set_config($vmid, $raw) { + $configs->{$vmid} = $raw; + sync_config_file($vmid); +} + +# Returns the last 'meta' property of the top-level section of the raw config $raw, like the config +# class reads the stored identity. +my sub top_level_meta($raw) { + my $meta; + for my $line (split(/\n/, $raw // '')) { + last if $line =~ m/^\[/; + $meta = $1 if $line =~ m/^meta:\s*(.*?)\s*$/; + } + + return $meta; +} + +# Returns the identity stored in the raw config of $vmid, undef if there is none. +sub stored_identity($vmid) { + return PVE::GuestIdentity::parse_identity(top_level_meta($configs->{$vmid})); +} + +# Returns the uid that the $n-th generation of an identity yields. +sub generated_uid($n) { + my $alphabet = 'abcdefghijklmnopqrstuvwxyz234567'; + return ('a' x 17) . join('', map { substr($alphabet, ($n >> (5 * $_)) & 31, 1) } (2, 1, 0)); +} + +# the list of used guest IDs is not part of the fake cluster filesystem +our $guest_id_module = Test::MockModule->new('PVE::GuestID'); +$guest_id_module->mock(register_used_id => sub { return; }); + +our $guest_identity_module = Test::MockModule->new('PVE::GuestIdentity'); +$guest_identity_module->mock( + __random_bytes => sub { + my ($len) = @_; + die "unexpected length $len\n" if $len != 20; + $generated++; + # one byte per uid character, the last three hold 5 bits of the counter each + return ("\0" x 17) . pack('C3', map { ($generated >> $_) & 31 } (10, 5, 0)); + }, + __time_ms => sub { + return $now_ms; + }, +); + +our $cluster_module = Test::MockModule->new('PVE::Cluster'); +$cluster_module->mock( + get_config => sub { + my ($path) = @_; + my $vmid = vmid_of_path($path) // die "get_config (mocked) - unexpected path '$path'\n"; + return $configs->{$vmid}; + }, + cfs_read_file => sub { + my ($path) = @_; + return {} if $path eq 'datacenter.cfg'; + my $vmid = vmid_of_path($path) + // die "cfs_read_file (mocked) - unexpected path '$path'\n"; + return PVE::QemuServer::parse_vm_config($path, $configs->{$vmid}); + }, + cfs_write_file => sub { + my ($path, $conf) = @_; + my $vmid = vmid_of_path($path) + // die "cfs_write_file (mocked) - unexpected path '$path'\n"; + $configs->{$vmid} = PVE::QemuServer::write_vm_config($path, $conf); + sync_config_file($vmid); + push $writes->@*, $vmid; + return; + }, + cfs_update => sub { + return; + }, + check_vmid_unused => sub { + my ($vmid, $noerr) = @_; + return 1 if !defined($configs->{$vmid}) && !defined($other_guests->{$vmid}); + return undef if $noerr; + die "VM $vmid already exists\n"; + }, + cfs_lock_domain => sub { + my ($domain, $timeout, $code, @param) = @_; + die "cfs_lock_domain (mocked) - unexpected domain '$domain'\n" + if $domain ne 'guest-identity'; + if ($lock_domain_error) { + $@ = $lock_domain_error; + return undef; + } + my $res = eval { $code->(@param) }; + return $res; + }, + get_guest_config_properties => sub { + my ($properties) = @_; + die "get_guest_config_properties (mocked) - unexpected properties\n" + if join(',', $properties->@*) ne 'meta'; + + my $res = {}; + for my $vmid (keys $configs->%*) { + my $meta = top_level_meta($configs->{$vmid}); + $res->{$vmid} = { meta => $meta } if defined($meta); + } + $res->{$_} = { meta => $other_guests->{$_} } for keys $other_guests->%*; + return $res; + }, +); + +our $qemu_config_module = Test::MockModule->new('PVE::QemuConfig'); +$qemu_config_module->mock( + config_file => sub { + my ($class, $vmid, $node) = @_; + return "$tmpdir/etc-pve/" . $class->cfs_config_path($vmid, $node); + }, + config_file_lock => sub { + my ($class, $vmid) = @_; + return "$tmpdir/lock/lock-$vmid.conf"; + }, + destroy_config => sub { + my ($class, $vmid) = @_; + die "failed to remove config file: No such file or directory\n" + if !defined($configs->{$vmid}); + delete $configs->{$vmid}; + sync_config_file($vmid); + }, +); + +1; diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl index 20cb3df0..ab4dd887 100755 --- a/src/test/run_qemu_restore_config_tests.pl +++ b/src/test/run_qemu_restore_config_tests.pl @@ -3,35 +3,30 @@ use strict; use warnings; -use lib qw(..); +use lib qw(.. .); +use Fcntl qw(:flock O_RDONLY O_NONBLOCK); +use File::Path qw(make_path); +use File::Temp qw(tempdir); +use JSON; use Test::MockModule; use Test::More; use File::Basename; +use PVE::API2::Qemu; +use PVE::QemuConfig; use PVE::QemuServer; -use PVE::Tools qw(dir_glob_foreach file_get_contents); +use PVE::QemuServer::Helpers; +use PVE::RPCEnvironment; +use PVE::Tools qw(dir_glob_foreach file_get_contents file_set_contents); + +# also provides the 'datacenter.cfg' that restore_update_config_line reads +use QemuServerTest::FakeCFS; my $INPUT_DIR = './restore-config-input'; my $EXPECTED_DIR = './restore-config-expected'; -# NOTE update when you add/remove tests -plan tests => 5; - -my $pve_cluster_module = Test::MockModule->new("PVE::Cluster"); -$pve_cluster_module->mock( - cfs_read_file => sub { - my ($file) = @_; - - if ($file eq 'datacenter.cfg') { - return {}; - } else { - die "'cfs_read_file' called - missing mock?\n"; - } - }, -); - dir_glob_foreach( './restore-config-input', '[0-9]+.conf', @@ -78,4 +73,343 @@ dir_glob_foreach( }, ); +# Restores through the create API and the restore paths for PBS, backup providers, VMA and legacy +# tar archives, against the fake cluster filesystem. Storage, archive tools and other external +# effects are mocked. The backups have no disks, so that no volumes get allocated. + +my $runtime_dir = tempdir('restore-run-XXXXXX', DIR => '.', CLEANUP => 1); +$PVE::QemuServer::Helpers::var_run_tmpdir = $runtime_dir; + +my $rpcenv = PVE::RPCEnvironment->init('cli'); +$rpcenv->set_user('root@pam'); + +my $ARCHIVES = { + pbs => 'pbs:backup/vm/100/2025-01-01T00:00:00Z', + provider => 'provider:backup/vm-100-2025-01-01', + 'vma file' => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.vma', + 'vma stdin' => '-', + tar => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.tar', +}; + +our $current_backup; # raw config of the backup +our $fail_at = ''; # where the restore fails: 'read-config', 'data', 'after-restore' or 'live' +my $calls = {}; # calls with effects on the target VM that tests check + +sub backup_config { + my ($meta) = @_; + + return + "boot: order=scsi0\ncores: 1\nmemory: 512\n" + . (defined($meta) ? "meta: $meta\n" : '') + . "name: backup\n"; +} + +package TestBackupProvider { + + sub new { + my ($class) = @_; + return bless {}, $class; + } + + sub restore_get_mechanism { return ('qemu-img', 'qemu'); } + sub restore_vm_init { return {}; } + + sub archive_get_guest_config { + die "unable to read the backup\n" if $main::fail_at eq 'read-config'; + return $main::current_backup; + } + sub archive_get_firewall_config { return "[OPTIONS]\nenable: 1\n"; } + sub restore_vm_cleanup { return; } +} + +my $replication_state_module = Test::MockModule->new('PVE::ReplicationState'); +$replication_state_module->mock(delete_guest_states => sub { $calls->{delete_guest_states}++; }); + +my $firewall_module = Test::MockModule->new('PVE::Firewall'); +$firewall_module->mock( + remove_vmfw_conf => sub { + my ($vmid) = @_; + $calls->{remove_vmfw_conf}++; + $calls->{'remove_vmfw_conf-reserved'}++ + if defined($QemuServerTest::FakeCFS::configs->{$vmid}); + }, +); + +my $tools_module = Test::MockModule->new('PVE::Tools'); +$tools_module->mock( + file_copy => sub { + my ($src, $dst) = @_; + die "file_copy (mocked) - unexpected destination '$dst'\n" if $dst !~ m|/firewall/|; + $calls->{firewall}++; + }, +); + +my $storage_module = Test::MockModule->new('PVE::Storage'); +$storage_module->mock( + abs_filesystem_path => sub { + my ($cfg, $path) = @_; + return $path; + }, + check_volume_access => sub { return; }, + config => sub { return {}; }, + deactivate_volumes => sub { return 1; }, + extract_vzdump_config => sub { return $current_backup; }, + new_backup_provider => sub { return TestBackupProvider->new(); }, + parse_volname => sub { + return ('backup', 'vm/100/2025-01-01T00:00:00Z', 100, undef, undef, undef, 'pbs-vm'); + }, + parse_volume_id => sub { + my ($volid, $noerr) = @_; + return ($1, $2) if $volid =~ m/^([a-z]+):(.+)$/; + die "unable to parse volume ID '$volid'\n" if !$noerr; + return; + }, + path_to_volume_id => sub { return; }, + storage_config => sub { + my ($cfg, $storeid) = @_; + return { type => 'pbs', server => 'localhost', datastore => 'ds' } if $storeid eq 'pbs'; + return { type => 'dir', path => '/mnt/provider' }; + }, + storage_has_feature => sub { + my ($cfg, $storeid, $feature) = @_; + return $storeid eq 'provider' && $feature eq 'backup-provider'; + }, +); + +my $pbs_plugin_module = Test::MockModule->new('PVE::Storage::PBSPlugin'); +$pbs_plugin_module->mock( + pbs_encryption_key_file_name => sub { return '/nonexistent/key'; }, + pbs_get_password => sub { return 'secret'; }, + run_raw_client_cmd => sub { + my ($scfg, $storeid, $cmd, $param) = @_; + my (undef, $file, $target) = $param->@*; + if ($file eq 'index.json') { + my $files = + [{ filename => 'qemu-server.conf.blob' }, { filename => 'fw.conf.blob' }]; + file_set_contents($target, encode_json({ files => $files })); + } elsif ($file eq 'qemu-server.conf') { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents($target, $current_backup); + } elsif ($file eq 'fw.conf') { + file_set_contents($target, "[OPTIONS]\nenable: 1\n"); + } else { + die "run_raw_client_cmd (mocked) - unexpected file '$file'\n"; + } + }, +); + +my $network_module = Test::MockModule->new('PVE::QemuServer::Network'); +$network_module->mock( + create_ifaces_ipams_ips => sub { + die "unable to register IPs\n" if $fail_at eq 'after-restore'; + }, +); + +my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); +$qemu_server_module->mock( + check_running => sub { return 0; }, + pbs_live_restore => sub { + die "live restore failed\n" if $fail_at eq 'live'; + $calls->{live_restore}++; + }, + rescan => sub { return; }, + restore_firewall_config => sub { + die "unable to restore the firewall config\n" if $fail_at eq 'firewall'; + $calls->{firewall}++; + }, + restore_update_config_line => sub { + die "restoring the data failed\n" if $fail_at eq 'data'; + return $qemu_server_module->original('restore_update_config_line')->(@_); + }, + tar_archive_read_firstfile => sub { return 'qemu-server.conf'; }, + # like 'tar' with qmextract and 'vma extract': write the config and report the VMA header + run_command => sub { + my ($cmd, %param) = @_; + + if (!ref($cmd) && $cmd =~ m/^zcat -f\|tar xf /) { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents("$ENV{VZDUMP_TMPDIR}/qemu-server.conf", $current_backup); + return 0; + } + + my $vma = $cmd->[-1]; + die "run_command (mocked) - unexpected command\n" + if ref($vma) ne 'ARRAY' || $vma->[0] ne 'vma'; + my ($mapfifo, $tmpdir) = $vma->@[4, 6]; + + die "unable to read the backup\n" if $fail_at eq 'read-config'; + + make_path($tmpdir); + file_set_contents("$tmpdir/qemu-server.conf", $current_backup); + file_set_contents("$tmpdir/qemu-server.fw", "[OPTIONS]\nenable: 1\n"); + + sysopen(my $reader, $mapfifo, O_RDONLY | O_NONBLOCK) + or die "unable to open fifo - $!\n"; + $param{afterfork}->(); + $param{outfunc}->('CTIME: Wed Jan 1 00:00:00 2025'); + close($reader); + + return 0; + }, +); + +my $task_error; +my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment'); +$rpcenv_module->mock( + check => sub { return 1; }, + fork_worker => sub { + my ($self, $type, $id, $user, $code) = @_; + eval { $code->('UPID:test') }; + $task_error = $@; + return 'UPID:test'; + }, +); + +$QemuServerTest::FakeCFS::cluster_module->mock(check_cfs_quorum => sub { return 1; }); + +# Restores a backup with the raw config $backup to VMID 100 through the create API, from an archive +# of type $type with the additional API parameters $param. Returns the error of the API call or the +# task and the task output. +sub restore_api { + my ($type, $backup, $param) = @_; + + $current_backup = $backup; + $calls = {}; + $task_error = ''; + + my ($err, $output) = ('', ''); + { + open(my $out_fh, '>', \$output); + local *STDERR = $out_fh; + my $oldfh = select($out_fh); + + eval { + PVE::API2::Qemu->create_vm({ + node => 'localhost', + vmid => 100, + archive => $ARCHIVES->{$type}, + $param->%*, + }); + }; + $err = $@ || $task_error; + + select($oldfh); + close($out_fh); + } + + return ($err, $output); +} + +my $TARGET_CONF = "memory: 1024\nmeta: creation-qemu=10.0.0,ctime=1700000000\nname: target\n"; +my $NOT_CLEANED_UP = qr/VM 100 state is NOT cleaned up/; + +for my $type (sort keys $ARCHIVES->%*) { + subtest "$type: failed restore to a free VMID" => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'data'; + my ($err, $output) = restore_api($type, backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + is($calls->{remove_vmfw_conf}, 1, 'restored firewall config removed'); + is($calls->{'remove_vmfw_conf-reserved'}, 1, 'while the VMID was still reserved'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); + + $fail_at = ''; + ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + is($err, '', 'retry succeeds') or diag($output); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'VM restored'); + }; +} + +subtest 'failed restore to a free VMID before reading the backup' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "unable to read the backup\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); +}; + +subtest 'another restore right after a failed restore to a free VMID' => sub { + QemuServerTest::FakeCFS::reset(); + + # run a second restore with 'force' as soon as the failed one releases the guest lock + my $lock_config_full = PVE::QemuConfig->can('lock_config_full'); + my ($interposed, $lock_free, $second_err) = (0, 0, ''); + $QemuServerTest::FakeCFS::qemu_config_module->mock( + lock_config_full => sub { + my ($class, $vmid, @param) = @_; + my $res = eval { $lock_config_full->($class, $vmid, @param) }; + my $err = $@; + if ($err && !$interposed++) { + open(my $fh, '>>', PVE::QemuConfig->config_file_lock($vmid)) or die "$!\n"; + $lock_free = flock($fh, LOCK_EX | LOCK_NB); + close($fh); + local $fail_at = ''; + ($second_err) = restore_api('pbs', backup_config(), { force => 1 }); + } + die $err if $err; + return $res; + }, + ); + + # the temporary config of a VMA restore has no lock that would stop the second restore + $fail_at = 'data'; + my ($err) = restore_api('vma file', backup_config(), {}); + $QemuServerTest::FakeCFS::qemu_config_module->unmock('lock_config_full'); + + is($err, "restoring the data failed\n", 'first restore failed'); + ok($lock_free, 'guest lock was free for the second restore'); + is($second_err, '', 'second restore succeeded'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'its config stays'); + is($calls->{firewall}, 1, 'its firewall config was restored'); + ok(!$calls->{remove_vmfw_conf}, 'and stays'); +}; + +for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { + subtest "$type: failed restore over an existing VM" => sub { + QemuServerTest::FakeCFS::reset(); + QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + like($err, qr/unable to read the backup\n$/, 'fails before changing the VM'); + is($QemuServerTest::FakeCFS::configs->{100}, $TARGET_CONF, 'config unchanged'); + is_deeply($calls, {}, 'replication state and firewall config untouched'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM'); + + $fail_at = 'data'; + ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + is($err, "restoring the data failed\n", 'fails while restoring'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^memory: 128$/m, 'temporary config'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, qr/error before or during data restore/, 'warning about the changed VM'); + }; +} + +subtest 'failed live restore to a free VMID keeps the restored config' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'live'; + my ($err, $output) = restore_api('pbs', backup_config(), { 'live-restore' => 1 }); + is($err, "live restore failed\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, $NOT_CLEANED_UP, 'warning about the VM left behind'); +}; + +subtest 'failure after the data was restored keeps the VM' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'after-restore'; + my ($err, $output) = restore_api('pbs', backup_config(), { unique => 1 }); + is($err, "unable to register IPs\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + like($output, qr/error after data was restored/, 'warning about the VM left behind'); +}; + +$fail_at = ''; + done_testing(); -- 2.47.3