From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 6CB0B1FF0C0 for ; Wed, 18 Nov 2026 05:43:30 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8DA3A21AEB; Wed, 18 Nov 2026 05:39:36 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Date: Wed, 18 Nov 2026 05:37:26 +0100 Message-ID: <20261118043817.1659212-15-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com> References: <20261118043817.1659212-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794976702429 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.186 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KABKS5LE7GQCMLSUPW7WKQHXM47YL6HW X-Message-ID-Hash: KABKS5LE7GQCMLSUPW7WKQHXM47YL6HW X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A failed restore to a previously unused VMID left its temporary or empty config behind, even when the restore cleaned up the volumes it had allocated. The VMID stayed taken and a retry needed 'force'. Containers already remove their config in this case. Remove the VM config and the firewall config that the restore wrote if the VMID was unused before the operation and the restore has not reached the point from which it keeps the restored volumes. Warn if the removed firewall config had replaced one that existed for the VMID before, for example one made by hand, as that one is gone. Keep the VM if a volume that the restore allocated or the firewall config cannot be removed, so that what is left behind keeps its VMID. Keep it as well after later failures, such as a failure in the live phase of a live restore. If the next-id setting enforces unique guest IDs, the released VMID counts as used, so a retry needs another VMID, as for containers. Remove the config before releasing the guest config lock, so that no other operation can take the VMID in between and lose its config to this cleanup. Remove the firewall config before the VM config for a similar reason: the guest config lock is local to the node, so another node can take the VMID once the VM config is gone. The lock is held, so the VM cannot run unprotected in between. Only warn that the VM was not cleaned up if the restore changed the target and left it behind. A failure to read the backup should not produce that warning. Signed-off-by: Thomas Lamprecht --- changes v2 -> v3: * keep the VM if a restored volume or the restored firewall config cannot be removed * only remove a firewall config that the restore wrote, and warn if that one had replaced a firewall config that existed for the free VMID before * tests: retry without force, and cases for the independent fixes before the series ('force' and a VM created in between, lock of the temporary VMA config) changes v1 -> v2: * remove the firewall config before the VM config, with a test for the order * note the effect with enforced unique guest IDs in the commit message * test helper: use v5.36, mock the registration of used guest IDs src/PVE/API2/Qemu.pm | 45 +- src/PVE/QemuServer.pm | 56 ++- src/test/QemuServerTest/FakeCFS.pm | 186 ++++++++ src/test/run_qemu_restore_config_tests.pl | 509 +++++++++++++++++++++- 4 files changed, 768 insertions(+), 28 deletions(-) create mode 100644 src/test/QemuServerTest/FakeCFS.pm diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index cad8eb06..1a3f2ce9 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -1423,10 +1423,19 @@ __PACKAGE__->register_method({ my $emsg = $is_restore ? "unable to restore VM $vmid -" : "unable to create VM $vmid -"; # 'force' only covers the VM that the checks above saw, not one created in the meantime - eval { PVE::QemuConfig->create_and_lock_config($vmid, $force && $vm_exists) }; + my $reservation = + eval { PVE::QemuConfig->create_and_lock_config($vmid, $force && $vm_exists) }; die "$emsg $@" if $@; my $restored_data = 0; + # the restore paths record whether they changed the VM or wrote its firewall config, and + # whether restored volumes stay + my $restore_progress = {}; + # a free VMID can have a firewall config, for example one made by hand + my $firewall_existed = + $is_restore + && !$reservation->{existed} + && PVE::QemuServer::firewall_config_exists($vmid); my $restorefn = sub { my $conf = PVE::QemuConfig->load_config($vmid); @@ -1442,6 +1451,7 @@ __PACKAGE__->register_method({ bwlimit => $bwlimit, live => $live_restore, override_conf => $param, + progress => $restore_progress, }; if (my $volid = $archive->{volid}) { # best effort, real check is after restoring! @@ -1499,7 +1509,34 @@ __PACKAGE__->register_method({ if $unique; }; - PVE::QemuConfig->lock_config_full($vmid, 1, $realcmd); + my $restore_locked = sub { + eval { $realcmd->() }; + my $err = $@ or return; + + if (!$reservation->{existed} && !$restore_progress->{'volumes-kept'}) { + # Remove the reservation before releasing the guest lock, so another + # operation cannot take ownership first. The firewall config that the restore + # wrote goes first, before the VMID is freed. The lock is held, so the VM cannot + # run unprotected. + eval { + if ($restore_progress->{'firewall-restored'}) { + my $removed = PVE::Firewall::remove_vmfw_conf($vmid); + die "unable to remove the firewall config - $!\n" + if !$removed && !$!{ENOENT}; + if ($removed && $firewall_existed) { + log_warn("removed the firewall config of VM $vmid, the failed" + . " restore had replaced the one that existed before"); + } + } + PVE::QemuConfig->destroy_config($vmid); + $restore_progress->{'reservation-removed'} = 1; + }; + warn $@ if $@; + } + die $err; + }; + + PVE::QemuConfig->lock_config_full($vmid, 1, $restore_locked); if ($start_after_create && !$live_restore) { print "Execute autostart\n"; @@ -1632,13 +1669,15 @@ __PACKAGE__->register_method({ $code = sub { eval { $restorefn->() }; if (my $err = $@) { + die $err if $restore_progress->{'reservation-removed'}; + eval { PVE::QemuConfig->remove_lock($vmid, 'create') }; warn $@ if $@; if ($restored_data) { warn "error after data was restored, VM disks should be OK but config may " . "require adaptions. VM $vmid state is NOT cleaned up.\n"; - } else { + } elsif ($restore_progress->{'target-changed'}) { warn "error before or during data restore, some or all disks were not " . "completely restored. VM $vmid state is NOT cleaned up.\n"; } diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index 54c49b58..855ede59 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -6407,11 +6407,13 @@ sub tar_archive_read_firstfile { return $firstfile; } +# Returns the number of volumes that could not be removed. sub tar_restore_cleanup { my ($storecfg, $statfile) = @_; print STDERR "starting cleanup\n"; + my $left = 0; if (my $fd = IO::File->new($statfile, "r")) { while (defined(my $line = <$fd>)) { if ($line =~ m/vzdump:([^\s:]*):(\S+)$/) { @@ -6424,13 +6426,18 @@ sub tar_restore_cleanup { } print STDERR "temporary volume '$volid' successfully removed\n"; }; - print STDERR "unable to cleanup '$volid' - $@" if $@; + if (my $err = $@) { + print STDERR "unable to cleanup '$volid' - $err"; + $left++; + } } else { print STDERR "unable to parse line in statfile - $line"; } } $fd->close(); } + + return $left; } sub restore_file_archive { @@ -6697,17 +6704,24 @@ my $restore_deactivate_volumes = sub { print STDERR $@ if $@; }; +# Returns the number of volumes that could not be removed. my $restore_destroy_volumes = sub { my ($storecfg, $virtdev_hash) = @_; + my $left = 0; for my $dev (values $virtdev_hash->%*) { my $volid = $dev->{volid} or next; eval { PVE::Storage::vdisk_free($storecfg, $volid); print STDERR "temporary volume '$volid' successfully removed\n"; }; - print STDERR "unable to cleanup '$volid' - $@" if $@; + if (my $err = $@) { + print STDERR "unable to cleanup '$volid' - $err"; + $left++; + } } + + return $left; }; sub restore_merge_config { @@ -6877,6 +6891,13 @@ sub restore_firewall_config { PVE::Tools::file_copy($path, "${pve_firewall_dir}/$vmid.fw"); } +# Returns whether the VM $vmid has a firewall config. +sub firewall_config_exists { + my ($vmid) = @_; + + return -e "/etc/pve/firewall/$vmid.fw"; +} + sub restore_proxmox_backup_archive { my ($archive, $vmid, $user, $options) = @_; @@ -6960,12 +6981,14 @@ sub restore_proxmox_backup_archive { $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); if ($has_firewall_config) { $param = [$pbs_backup_name, "fw.conf", $firewall_config_fn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + $options->{progress}->{'firewall-restored'} = 1; restore_firewall_config($vmid, $firewall_config_fn); } @@ -7049,9 +7072,12 @@ sub restore_proxmox_backup_archive { rmtree $tmpdir; if ($err) { - $restore_destroy_volumes->($storecfg, $virtdev_hash); + # a volume that could not be removed stays with the VM + $options->{progress}->{'volumes-kept'} = 1 + if $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; if ($options->{live}) { # keep lock during live-restore @@ -7139,12 +7165,14 @@ sub restore_external_archive { my $data = $backup_provider->archive_get_guest_config($volname) or die "backup provider failed to extract guest configuration\n"; + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); PVE::Tools::file_set_contents($cfgfn, $data); if ($data = $backup_provider->archive_get_firewall_config($volname)) { PVE::Tools::file_set_contents($firewall_config_fn, $data); + $options->{progress}->{'firewall-restored'} = 1; restore_firewall_config($vmid, $firewall_config_fn); } @@ -7223,9 +7251,12 @@ sub restore_external_archive { rmtree($tmpdir); if ($err) { - $restore_destroy_volumes->($storecfg, $virtdev_hash); + # a volume that could not be removed stays with the VM + $options->{progress}->{'volumes-kept'} = 1 + if $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } + $options->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7551,6 +7582,7 @@ sub restore_vma_archive { my $print_devmap = sub { my $cfgfn = "$tmpdir/qemu-server.conf"; + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # we can read the config - that is already extracted @@ -7558,7 +7590,10 @@ sub restore_vma_archive { || die "unable to read qemu-server.conf - $!\n"; my $fwcfgfn = "$tmpdir/qemu-server.fw"; - restore_firewall_config($vmid, $fwcfgfn) if -f $fwcfgfn; + if (-f $fwcfgfn) { + $opts->{progress}->{'firewall-restored'} = 1; + restore_firewall_config($vmid, $fwcfgfn); + } $virtdev_hash = $parse_backup_hints->($rpcenv, $user, $cfg, $fh, $devinfo, $opts); @@ -7677,9 +7712,12 @@ sub restore_vma_archive { rmtree $tmpdir; if ($err) { - $restore_destroy_volumes->($cfg, $virtdev_hash); + # a volume that could not be removed stays with the VM + $opts->{progress}->{'volumes-kept'} = 1 + if $restore_destroy_volumes->($cfg, $virtdev_hash); die $err; } + $opts->{progress}->{'volumes-kept'} = 1; my $new_conf = restore_merge_config($conffile, $new_conf_raw, $opts->{override_conf}); check_restore_permissions($rpcenv, $user, $new_conf); @@ -7707,6 +7745,7 @@ sub restore_tar_archive { my $storecfg = PVE::Storage::config(); + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # avoid zombie disks when restoring over an existing VM -> cleanup first @@ -7786,9 +7825,12 @@ sub restore_tar_archive { $srcfd->close(); }; if (my $err = $@) { - tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat") if !$opts->{info}; + # a volume that could not be removed stays with the VM + $opts->{progress}->{'volumes-kept'} = 1 + if !$opts->{info} && tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat"); die $err; } + $opts->{progress}->{'volumes-kept'} = 1; rmtree $tmpdir; diff --git a/src/test/QemuServerTest/FakeCFS.pm b/src/test/QemuServerTest/FakeCFS.pm new file mode 100644 index 00000000..99d09f87 --- /dev/null +++ b/src/test/QemuServerTest/FakeCFS.pm @@ -0,0 +1,186 @@ +package QemuServerTest::FakeCFS; + +# Fake cluster filesystem for VM config files, so that tests can use the real config class methods +# of PVE::QemuConfig, including the guest identity handling of write_config, without pmxcfs. +# +# Configs are kept as raw text, indexed by VMID, and an empty file at the path that config_file() +# returns exists for each, for checks of the file's existence. Generated guest identities are +# deterministic: the n-th generated uid is 'a' repeated 17 times followed by the base32 encoding of +# n, and rtime is $now_ms. + +use v5.36; + +use File::Basename qw(dirname); +use File::Path qw(make_path remove_tree); +use File::Temp qw(tempdir); +use Test::MockModule; + +use PVE::GuestHelpers; +use PVE::GuestID; +use PVE::GuestIdentity; +use PVE::QemuConfig; +use PVE::QemuServer; + +our $configs = {}; # VMID => raw config, for VMs on the local node +our $other_guests = {}; # VMID => raw 'meta' property of other guests, like on other nodes or CTs +our $now_ms = 1700000000000; +our $generated = 0; +our $lock_domain_error; +our $writes = []; + +my $tmpdir = tempdir('fake-cfs-XXXXXX', DIR => '.', CLEANUP => 1); +make_path("$tmpdir/lock"); +$PVE::GuestHelpers::lockdir = "$tmpdir/lock"; + +sub reset() { + remove_tree("$tmpdir/etc-pve"); + $configs = {}; + $other_guests = {}; + $now_ms = 1700000000000; + $generated = 0; + $lock_domain_error = undef; + $writes = []; +} + +my sub vmid_of_path($path) { + return $path =~ m|/qemu-server/(\d+)\.conf$| ? $1 : undef; +} + +# Creates or removes the file at the path of the config of $vmid, matching the stored config. +my sub sync_config_file($vmid) { + my $path = PVE::QemuConfig->config_file($vmid); + if (defined($configs->{$vmid})) { + make_path(dirname($path)); + open(my $fh, '>', $path) or die "unable to create '$path' - $!\n"; + close($fh); + } else { + unlink($path); + } +} + +# Stores the raw config $raw for $vmid, like a manual edit of the config file. +sub set_config($vmid, $raw) { + $configs->{$vmid} = $raw; + sync_config_file($vmid); +} + +# Returns the last 'meta' property of the top-level section of the raw config $raw, like the config +# class reads the stored identity. +my sub top_level_meta($raw) { + my $meta; + for my $line (split(/\n/, $raw // '')) { + last if $line =~ m/^\[/; + # like pmxcfs, a line with nothing after the colon does not count + $meta = $1 =~ s/^\s+|\s+$//gr if $line =~ m/^meta:(.+)$/; + } + + return $meta; +} + +# Returns the identity stored in the raw config of $vmid, undef if there is none. +sub stored_identity($vmid) { + return PVE::GuestIdentity::parse_identity(top_level_meta($configs->{$vmid})); +} + +# Returns the uid that the $n-th generation of an identity yields. +sub generated_uid($n) { + my $alphabet = 'abcdefghijklmnopqrstuvwxyz234567'; + return ('a' x 17) . join('', map { substr($alphabet, ($n >> (5 * $_)) & 31, 1) } (2, 1, 0)); +} + +# the list of used guest IDs is not part of the fake cluster filesystem +our $guest_id_module = Test::MockModule->new('PVE::GuestID'); +$guest_id_module->mock(register_used_id => sub { return; }); + +our $guest_identity_module = Test::MockModule->new('PVE::GuestIdentity'); +$guest_identity_module->mock( + __random_bytes => sub { + my ($len) = @_; + die "unexpected length $len\n" if $len != 20; + $generated++; + # one byte per uid character, the last three hold 5 bits of the counter each + return ("\0" x 17) . pack('C3', map { ($generated >> $_) & 31 } (10, 5, 0)); + }, + __time_ms => sub { + return $now_ms; + }, +); + +our $cluster_module = Test::MockModule->new('PVE::Cluster'); +$cluster_module->mock( + get_config => sub { + my ($path) = @_; + my $vmid = vmid_of_path($path) // die "get_config (mocked) - unexpected path '$path'\n"; + return $configs->{$vmid}; + }, + cfs_read_file => sub { + my ($path) = @_; + return {} if $path eq 'datacenter.cfg'; + my $vmid = vmid_of_path($path) + // die "cfs_read_file (mocked) - unexpected path '$path'\n"; + return PVE::QemuServer::parse_vm_config($path, $configs->{$vmid}); + }, + cfs_write_file => sub { + my ($path, $conf) = @_; + my $vmid = vmid_of_path($path) + // die "cfs_write_file (mocked) - unexpected path '$path'\n"; + $configs->{$vmid} = PVE::QemuServer::write_vm_config($path, $conf); + sync_config_file($vmid); + push $writes->@*, $vmid; + return; + }, + cfs_update => sub { + return; + }, + check_vmid_unused => sub { + my ($vmid, $noerr) = @_; + return 1 if !defined($configs->{$vmid}) && !defined($other_guests->{$vmid}); + return undef if $noerr; + die "VM $vmid already exists\n"; + }, + cfs_lock_domain => sub { + my ($domain, $timeout, $code, @param) = @_; + die "cfs_lock_domain (mocked) - unexpected domain '$domain'\n" + if $domain ne 'guest-identity'; + if ($lock_domain_error) { + $@ = $lock_domain_error; + return undef; + } + my $res = eval { $code->(@param) }; + return $res; + }, + get_guest_config_properties => sub { + my ($properties) = @_; + die "get_guest_config_properties (mocked) - unexpected properties\n" + if join(',', $properties->@*) ne 'meta'; + + my $res = {}; + for my $vmid (keys $configs->%*) { + my $meta = top_level_meta($configs->{$vmid}); + $res->{$vmid} = { meta => $meta } if defined($meta); + } + $res->{$_} = { meta => $other_guests->{$_} } for keys $other_guests->%*; + return $res; + }, +); + +our $qemu_config_module = Test::MockModule->new('PVE::QemuConfig'); +$qemu_config_module->mock( + config_file => sub { + my ($class, $vmid, $node) = @_; + return "$tmpdir/etc-pve/" . $class->cfs_config_path($vmid, $node); + }, + config_file_lock => sub { + my ($class, $vmid) = @_; + return "$tmpdir/lock/lock-$vmid.conf"; + }, + destroy_config => sub { + my ($class, $vmid) = @_; + die "failed to remove config file: No such file or directory\n" + if !defined($configs->{$vmid}); + delete $configs->{$vmid}; + sync_config_file($vmid); + }, +); + +1; diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl index 20cb3df0..24abf046 100755 --- a/src/test/run_qemu_restore_config_tests.pl +++ b/src/test/run_qemu_restore_config_tests.pl @@ -3,35 +3,31 @@ use strict; use warnings; -use lib qw(..); +use lib qw(.. .); +use Errno qw(EACCES ENOENT); +use Fcntl qw(:flock O_RDONLY O_NONBLOCK); +use File::Path qw(make_path); +use File::Temp qw(tempdir); +use JSON; use Test::MockModule; use Test::More; use File::Basename; +use PVE::API2::Qemu; +use PVE::QemuConfig; use PVE::QemuServer; -use PVE::Tools qw(dir_glob_foreach file_get_contents); +use PVE::QemuServer::Helpers; +use PVE::RPCEnvironment; +use PVE::Tools qw(dir_glob_foreach file_get_contents file_set_contents); + +# also provides the 'datacenter.cfg' that restore_update_config_line reads +use QemuServerTest::FakeCFS; my $INPUT_DIR = './restore-config-input'; my $EXPECTED_DIR = './restore-config-expected'; -# NOTE update when you add/remove tests -plan tests => 5; - -my $pve_cluster_module = Test::MockModule->new("PVE::Cluster"); -$pve_cluster_module->mock( - cfs_read_file => sub { - my ($file) = @_; - - if ($file eq 'datacenter.cfg') { - return {}; - } else { - die "'cfs_read_file' called - missing mock?\n"; - } - }, -); - dir_glob_foreach( './restore-config-input', '[0-9]+.conf', @@ -78,4 +74,481 @@ dir_glob_foreach( }, ); +# Restores through the create API and the restore paths for PBS, backup providers, VMA and legacy +# tar archives, against the fake cluster filesystem. Storage, archive tools and other external +# effects are mocked. The backups have no disks, so that no volumes get allocated. + +my $runtime_dir = tempdir('restore-run-XXXXXX', DIR => '.', CLEANUP => 1); +$PVE::QemuServer::Helpers::var_run_tmpdir = $runtime_dir; + +my $rpcenv = PVE::RPCEnvironment->init('cli'); +$rpcenv->set_user('root@pam'); + +my $ARCHIVES = { + pbs => 'pbs:backup/vm/100/2025-01-01T00:00:00Z', + provider => 'provider:backup/vm-100-2025-01-01', + 'vma file' => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.vma', + 'vma stdin' => '-', + tar => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.tar', +}; + +our $current_backup; # raw config of the backup +# where the restore fails: 'read-config', 'firewall', 'data', 'after-restore' or 'live' +our $fail_at = ''; +our $firewall_removal_errno; # errno with which removing the firewall config fails, if set +our $firewall_existed = 0; # whether VM 100 had a firewall config before the restore +my $calls = {}; # calls with effects on the target VM that tests check + +sub backup_config { + my ($meta) = @_; + + return + "boot: order=scsi0\ncores: 1\nmemory: 512\n" + . (defined($meta) ? "meta: $meta\n" : '') + . "name: backup\n"; +} + +package TestBackupProvider { + + sub new { + my ($class) = @_; + return bless {}, $class; + } + + sub restore_get_mechanism { return ('qemu-img', 'qemu'); } + sub restore_vm_init { return {}; } + + sub archive_get_guest_config { + die "unable to read the backup\n" if $main::fail_at eq 'read-config'; + return $main::current_backup; + } + sub archive_get_firewall_config { return "[OPTIONS]\nenable: 1\n"; } + sub restore_vm_cleanup { return; } +} + +my $replication_state_module = Test::MockModule->new('PVE::ReplicationState'); +$replication_state_module->mock(delete_guest_states => sub { $calls->{delete_guest_states}++; }); + +my $firewall_module = Test::MockModule->new('PVE::Firewall'); +$firewall_module->mock( + remove_vmfw_conf => sub { + my ($vmid) = @_; + $calls->{remove_vmfw_conf}++; + $calls->{'remove_vmfw_conf-reserved'}++ + if defined($QemuServerTest::FakeCFS::configs->{$vmid}); + # like PVE::Firewall, which returns the result of unlink + return 1 if !$firewall_removal_errno; + $! = $firewall_removal_errno; + return 0; + }, +); + +my $tools_module = Test::MockModule->new('PVE::Tools'); +$tools_module->mock( + file_copy => sub { + my ($src, $dst) = @_; + die "file_copy (mocked) - unexpected destination '$dst'\n" if $dst !~ m|/firewall/|; + $calls->{firewall}++; + }, +); + +my $storage_module = Test::MockModule->new('PVE::Storage'); +$storage_module->mock( + abs_filesystem_path => sub { + my ($cfg, $path) = @_; + return $path; + }, + check_volume_access => sub { return; }, + config => sub { return {}; }, + deactivate_volumes => sub { return 1; }, + extract_vzdump_config => sub { return $current_backup; }, + new_backup_provider => sub { return TestBackupProvider->new(); }, + parse_volname => sub { + return ('backup', 'vm/100/2025-01-01T00:00:00Z', 100, undef, undef, undef, 'pbs-vm'); + }, + parse_volume_id => sub { + my ($volid, $noerr) = @_; + return ($1, $2) if $volid =~ m/^([a-z]+):(.+)$/; + die "unable to parse volume ID '$volid'\n" if !$noerr; + return; + }, + path_to_volume_id => sub { return; }, + storage_config => sub { + my ($cfg, $storeid) = @_; + return { type => 'pbs', server => 'localhost', datastore => 'ds' } if $storeid eq 'pbs'; + return { type => 'dir', path => '/mnt/provider' }; + }, + storage_has_feature => sub { + my ($cfg, $storeid, $feature) = @_; + return $storeid eq 'provider' && $feature eq 'backup-provider'; + }, +); + +my $pbs_plugin_module = Test::MockModule->new('PVE::Storage::PBSPlugin'); +$pbs_plugin_module->mock( + pbs_encryption_key_file_name => sub { return '/nonexistent/key'; }, + pbs_get_password => sub { return 'secret'; }, + run_raw_client_cmd => sub { + my ($scfg, $storeid, $cmd, $param) = @_; + my (undef, $file, $target) = $param->@*; + if ($file eq 'index.json') { + my $files = + [{ filename => 'qemu-server.conf.blob' }, { filename => 'fw.conf.blob' }]; + file_set_contents($target, encode_json({ files => $files })); + } elsif ($file eq 'qemu-server.conf') { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents($target, $current_backup); + } elsif ($file eq 'fw.conf') { + file_set_contents($target, "[OPTIONS]\nenable: 1\n"); + } else { + die "run_raw_client_cmd (mocked) - unexpected file '$file'\n"; + } + }, +); + +my $network_module = Test::MockModule->new('PVE::QemuServer::Network'); +$network_module->mock( + create_ifaces_ipams_ips => sub { + die "unable to register IPs\n" if $fail_at eq 'after-restore'; + }, +); + +my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); +$qemu_server_module->mock( + check_running => sub { return 0; }, + pbs_live_restore => sub { + die "live restore failed\n" if $fail_at eq 'live'; + $calls->{live_restore}++; + }, + firewall_config_exists => sub { return $firewall_existed; }, + rescan => sub { return; }, + restore_firewall_config => sub { + die "unable to restore the firewall config\n" if $fail_at eq 'firewall'; + $calls->{firewall}++; + }, + restore_update_config_line => sub { + die "restoring the data failed\n" if $fail_at eq 'data'; + return $qemu_server_module->original('restore_update_config_line')->(@_); + }, + tar_archive_read_firstfile => sub { return 'qemu-server.conf'; }, + # like 'tar' with qmextract and 'vma extract': write the config and report the VMA header + run_command => sub { + my ($cmd, %param) = @_; + + if (!ref($cmd) && $cmd =~ m/^zcat -f\|tar xf /) { + die "unable to read the backup\n" if $fail_at eq 'read-config'; + file_set_contents("$ENV{VZDUMP_TMPDIR}/qemu-server.conf", $current_backup); + return 0; + } + + my $vma = $cmd->[-1]; + die "run_command (mocked) - unexpected command\n" + if ref($vma) ne 'ARRAY' || $vma->[0] ne 'vma'; + my ($mapfifo, $tmpdir) = $vma->@[4, 6]; + + die "unable to read the backup\n" if $fail_at eq 'read-config'; + + make_path($tmpdir); + file_set_contents("$tmpdir/qemu-server.conf", $current_backup); + file_set_contents("$tmpdir/qemu-server.fw", "[OPTIONS]\nenable: 1\n"); + + sysopen(my $reader, $mapfifo, O_RDONLY | O_NONBLOCK) + or die "unable to open fifo - $!\n"; + $param{afterfork}->(); + $param{outfunc}->('CTIME: Wed Jan 1 00:00:00 2025'); + close($reader); + + return 0; + }, +); + +my $task_error; +my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment'); +$rpcenv_module->mock( + check => sub { return 1; }, + fork_worker => sub { + my ($self, $type, $id, $user, $code) = @_; + eval { $code->('UPID:test') }; + $task_error = $@; + return 'UPID:test'; + }, +); + +$QemuServerTest::FakeCFS::cluster_module->mock(check_cfs_quorum => sub { return 1; }); + +# Restores a backup with the raw config $backup to VMID 100 through the create API, from an archive +# of type $type with the additional API parameters $param. Returns the error of the API call or the +# task and the task output. +sub restore_api { + my ($type, $backup, $param) = @_; + + $current_backup = $backup; + $calls = {}; + $task_error = ''; + + my ($err, $output) = ('', ''); + { + open(my $out_fh, '>', \$output); + local *STDERR = $out_fh; + my $oldfh = select($out_fh); + + eval { + PVE::API2::Qemu->create_vm({ + node => 'localhost', + vmid => 100, + archive => $ARCHIVES->{$type}, + $param->%*, + }); + }; + $err = $@ || $task_error; + + select($oldfh); + close($out_fh); + } + + return ($err, $output); +} + +my $TARGET_CONF = "memory: 1024\nmeta: creation-qemu=10.0.0,ctime=1700000000\nname: target\n"; +my $NOT_CLEANED_UP = qr/VM 100 state is NOT cleaned up/; + +for my $type (sort keys $ARCHIVES->%*) { + subtest "$type: failed restore to a free VMID" => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'data'; + my ($err, $output) = restore_api($type, backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + # legacy tar archives have no firewall config + my $restored_firewall = $type eq 'tar' ? undef : 1; + is($calls->{remove_vmfw_conf}, $restored_firewall, 'restored firewall config removed'); + is( + $calls->{'remove_vmfw_conf-reserved'}, + $restored_firewall, + 'while the VMID was still reserved', + ); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); + + $fail_at = ''; + ($err, $output) = restore_api($type, backup_config(), {}); + is($err, '', 'retry needs no force') or diag($output); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'VM restored'); + }; +} + +subtest 'failed restore to a free VMID before reading the backup' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "unable to read the backup\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed'); + ok(!$calls->{remove_vmfw_conf}, 'firewall config that the restore did not write is kept'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); +}; + +subtest 'removing the restored firewall config warns if it replaced one' => sub { + my $replaced = + qr/^WARN: removed the firewall config of VM 100, the failed restore had replaced/m; + for my $existed (0, 1) { + my $what = $existed ? 'firewall config before' : 'no firewall config before'; + QemuServerTest::FakeCFS::reset(); + + local $fail_at = 'data'; + local $firewall_existed = $existed; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "restoring the data failed\n", "$what: restore failed"); + is($calls->{remove_vmfw_conf}, 1, "$what: restored firewall config removed"); + if ($existed) { + like($output, $replaced, "$what: warning about the replaced firewall config"); + } else { + unlike($output, $replaced, "$what: no warning"); + } + } +}; + +subtest 'failed removal of the restored firewall config keeps the VMID' => sub { + QemuServerTest::FakeCFS::reset(); + + local $fail_at = 'data'; + local $firewall_removal_errno = EACCES; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + like($output, qr/^unable to remove the firewall config - /m, 'failed removal is reported'); + ok(defined($QemuServerTest::FakeCFS::configs->{100}), 'reservation is kept'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, $NOT_CLEANED_UP, 'warning about the VM left behind'); + + # the restore can fail before the firewall config was written + QemuServerTest::FakeCFS::reset(); + $fail_at = 'firewall'; + $firewall_removal_errno = ENOENT; + ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "unable to restore the firewall config\n", 'restore failed'); + ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'missing firewall config: removed'); +}; + +subtest 'failed removal of a restored volume keeps the VMID' => sub { + QemuServerTest::FakeCFS::reset(); + + my $allocated = []; + my $run_raw_client_cmd = \&PVE::Storage::PBSPlugin::run_raw_client_cmd; + $pbs_plugin_module->mock( + run_raw_client_cmd => sub { + my ($scfg, $storeid, $cmd, $param) = @_; + my (undef, $file, $target) = $param->@*; + return $run_raw_client_cmd->(@_) if $file ne 'index.json'; + my $files = [ + { filename => 'qemu-server.conf.blob' }, + { filename => 'drive-scsi0.img.fidx', size => 1048576 }, + ]; + file_set_contents($target, encode_json({ files => $files })); + }, + ); + my $storage_config = \&PVE::Storage::storage_config; + my @storage_functions = + qw(resolve_format_hint vdisk_alloc activate_volumes path volume_has_feature vdisk_free); + $storage_module->mock( + storage_config => sub { + my ($cfg, $storeid) = @_; + return { type => 'pbs', server => 'localhost', datastore => 'ds' } + if $storeid eq 'pbs'; + return { type => 'dir', path => '/nonexistent', content => { images => 1 } }; + }, + resolve_format_hint => sub { return 'raw'; }, + vdisk_alloc => sub { + my ($cfg, $storeid, $vmid) = @_; + push $allocated->@*, "$storeid:vm-$vmid-disk-0"; + return "$storeid:vm-$vmid-disk-0"; + }, + activate_volumes => sub { return; }, + path => sub { return '/nonexistent/vm-100-disk-0.raw'; }, + volume_has_feature => sub { return 0; }, + vdisk_free => sub { die "volume is busy\n"; }, + ); + my $run_command = \&PVE::QemuServer::run_command; + $qemu_server_module->mock( + run_command => sub { + my ($cmd) = @_; + return 0 if ref($cmd) && $cmd->[0] eq '/usr/bin/pbs-restore'; + return $run_command->(@_); + }, + ); + + local $fail_at = 'data'; + my $backup = backup_config() + . "scsi0: target:vm-100-disk-0,size=1M\n#qmdump#map:scsi0:drive-scsi0:target:raw:\n"; + my ($err, $output) = restore_api('pbs', $backup, {}); + is($err, "restoring the data failed\n", 'restore failed after allocating the volume'); + is_deeply($allocated, ['target:vm-100-disk-0'], 'volume was allocated'); + like($output, qr/^unable to cleanup 'target:vm-100-disk-0' - volume is busy$/m, 'reported'); + ok(defined($QemuServerTest::FakeCFS::configs->{100}), 'reservation is kept'); + like($output, $NOT_CLEANED_UP, 'warning about the VM left behind'); + + $pbs_plugin_module->mock(run_raw_client_cmd => $run_raw_client_cmd); + $storage_module->mock(storage_config => $storage_config); + $storage_module->unmock($_) for @storage_functions; + $qemu_server_module->mock(run_command => $run_command); +}; + +subtest "'force' does not overwrite a VM created after the checks" => sub { + QemuServerTest::FakeCFS::reset(); + + # the archive is checked after the target VM + my $check_volume_access = \&PVE::Storage::check_volume_access; + $storage_module->mock( + check_volume_access => sub { QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF); }, + ); + + my ($err) = restore_api('pbs', backup_config(), { force => 1 }); + $storage_module->mock(check_volume_access => $check_volume_access); + is($err, "unable to restore VM 100 - VM 100 already exists\n", 'restore refused'); + is( + $QemuServerTest::FakeCFS::configs->{100}, + $TARGET_CONF, + 'VM created in between unchanged', + ); +}; + +subtest 'another restore right after a failed restore to a free VMID' => sub { + QemuServerTest::FakeCFS::reset(); + + # run a second restore with 'force' as soon as the failed one releases the guest lock + my $lock_config_full = PVE::QemuConfig->can('lock_config_full'); + my ($interposed, $lock_free, $second_err) = (0, 0, ''); + $QemuServerTest::FakeCFS::qemu_config_module->mock( + lock_config_full => sub { + my ($class, $vmid, @param) = @_; + my $res = eval { $lock_config_full->($class, $vmid, @param) }; + my $err = $@; + if ($err && !$interposed++) { + open(my $fh, '>>', PVE::QemuConfig->config_file_lock($vmid)) or die "$!\n"; + $lock_free = flock($fh, LOCK_EX | LOCK_NB); + close($fh); + local $fail_at = ''; + ($second_err) = restore_api('pbs', backup_config(), { force => 1 }); + } + die $err if $err; + return $res; + }, + ); + + $fail_at = 'data'; + my ($err) = restore_api('vma file', backup_config(), {}); + $QemuServerTest::FakeCFS::qemu_config_module->unmock('lock_config_full'); + + is($err, "restoring the data failed\n", 'first restore failed'); + ok($lock_free, 'guest lock was free for the second restore'); + is($second_err, '', 'second restore succeeded'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'its config stays'); + is($calls->{firewall}, 1, 'its firewall config was restored'); + ok(!$calls->{remove_vmfw_conf}, 'and stays'); +}; + +for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { + subtest "$type: failed restore over an existing VM" => sub { + QemuServerTest::FakeCFS::reset(); + QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF); + + $fail_at = 'read-config'; + my ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + like($err, qr/unable to read the backup\n$/, 'fails before changing the VM'); + is($QemuServerTest::FakeCFS::configs->{100}, $TARGET_CONF, 'config unchanged'); + is_deeply($calls, {}, 'replication state and firewall config untouched'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM'); + + $fail_at = 'data'; + ($err, $output) = restore_api($type, backup_config(), { force => 1 }); + is($err, "restoring the data failed\n", 'fails while restoring'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^memory: 128$/m, 'temporary config'); + unlike($output, qr/no lock found/, 'temporary config had the create lock'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, qr/error before or during data restore/, 'warning about the changed VM'); + }; +} + +subtest 'failed live restore to a free VMID keeps the restored config' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'live'; + my ($err, $output) = restore_api('pbs', backup_config(), { 'live-restore' => 1 }); + is($err, "live restore failed\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed'); + like($output, $NOT_CLEANED_UP, 'warning about the VM left behind'); +}; + +subtest 'failure after the data was restored keeps the VM' => sub { + QemuServerTest::FakeCFS::reset(); + + $fail_at = 'after-restore'; + my ($err, $output) = restore_api('pbs', backup_config(), { unique => 1 }); + is($err, "unable to register IPs\n", 'restore failed'); + like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept'); + like($output, qr/error after data was restored/, 'warning about the VM left behind'); +}; + +$fail_at = ''; + done_testing(); -- 2.47.3