From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id AC0191FF0DA for ; Thu, 19 Nov 2026 20:22:57 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 26D58219B9; Thu, 19 Nov 2026 20:20:17 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v5 14/41] api: restore: release the VMID if a restore to an unused VMID fails Date: Thu, 19 Nov 2026 20:18:46 +0100 Message-ID: <20261119191914.2336934-15-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119191914.2336934-1-t.lamprecht@proxmox.com> References: <20261119191914.2336934-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795115959947 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.156 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LV3PEW65NRXW47CYR2ZPXRWLRNL2427P X-Message-ID-Hash: LV3PEW65NRXW47CYR2ZPXRWLRNL2427P X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: If a restore to a previously unused VMID failed, its temporary or empty config stayed, even if the restore had removed the volumes that it allocated. The VMID stayed taken and a retry needed 'force'. Containers already remove their config in this case. Remove the VM config if the VMID was unused before the operation and the restore fails before the point from which it keeps the restored volumes. Later failures, for example in the live phase of a live restore, keep the VM as before. Also remove the reservation if the restore could not start at all, for example because it did not get the guest lock in time. If the next-id setting enforces unique guest IDs, the released VMID counts as used. A retry then needs another VMID, as for containers. The cleanup can be incomplete: a volume that the restore allocated cannot be removed, or the previous firewall config cannot be put back. Keep the VM with its 'create' lock in that case, so that what is left behind keeps its VMID, and log what is left. For a legacy tar archive, a record of the allocated volumes that cannot be read counts like a volume that cannot be removed. Containers keep their locked config in this case as well. The lock ensures that the leftover does not get started or restored over before an admin has looked at it and removed the lock. Remove the config before the guest config lock is released. Otherwise, another operation could take the VMID in between and lose its config to this cleanup. Put the firewall config back even before that, for a similar reason: the guest config lock only holds on this node, so another node can take the VMID as soon as the VM config is gone. Only warn that the VM was not cleaned up if the restore changed the target and left it behind. A failure to read the backup, for example, should not produce that warning. Signed-off-by: Thomas Lamprecht Reviewed-by: Dominik Csapak Tested-by: Dominik Csapak --- no changes since v4 changes v3 -> v4: * the firewall part moved to an independent fix before the series, which puts the previous firewall config back after a failed restore, instead of removing the one of the backup with a warning; here, a config that cannot be put back keeps the VM * a VM that stays at a previously unused VMID because its cleanup was incomplete keeps its 'create' lock, like a container, and the task log says what is left; v3 unlocked it * remove the reservation also if the restore could not take its guest lock * tar archives: an allocation record that cannot be read or parsed keeps the VMID * the fake cluster filesystem and the tests of unchanged behavior moved to an independent test commit before the series * commit message and comments: plainer wording, same content changes v2 -> v3: * keep the VM if a restored volume or the restored firewall config cannot be removed * only remove a firewall config that the restore wrote, and warn if that one had replaced a firewall config that existed for the free VMID before * tests: retry without force, and cases for the independent fixes before the series ('force' and a VM created in between, lock of the temporary VMA config) changes v1 -> v2: * remove the firewall config before the VM config, with a test for the order * note the effect with enforced unique guest IDs in the commit message * test helper: use v5.36, mock the registration of used guest IDs src/PVE/API2/Qemu.pm | 45 ++++- src/PVE/QemuServer.pm | 79 ++++++-- src/test/run_qemu_restore_config_tests.pl | 234 +++++++++++++++++++++- 3 files changed, 332 insertions(+), 26 deletions(-) diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index b798a590..16779440 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -1509,12 +1509,15 @@ __PACKAGE__->register_method({ # 'force' only covers the VM that the checks above saw, not one created in the meantime, and # the permission to overwrite a VM does not cover creating it anew if it is gone by now my $allow_existing = $overwrite_only ? 'require' : $force && $vm_exists; - eval { PVE::QemuConfig->create_and_lock_config($vmid, $allow_existing) }; + my $reservation = + eval { PVE::QemuConfig->create_and_lock_config($vmid, $allow_existing) }; die "$emsg $@" if $@; my $restored_data = 0; - # the restore paths record whether the restored volumes stay with the VM + # the restore paths record whether they changed the VM, whether the restored volumes stay + # with it, and whether they could not remove all volumes after a failure my $restore_progress = {}; + my $restore_started; # whether the restore got its guest lock my $restorefn = sub { my $conf = PVE::QemuConfig->load_config($vmid); @@ -1590,15 +1593,36 @@ __PACKAGE__->register_method({ my $restore_locked = sub { my $firewall_conf = PVE::GuestHelpers::get_firewall_config_raw($vmid); + $restore_started = 1; eval { $realcmd->() }; my $err = $@ or return; # the VM that keeps its restored volumes also keeps the firewall config of the backup die $err if $restore_progress->{'volumes-kept'}; + my @left = + $restore_progress->{'volumes-left'} ? ('volumes of the restore') : (); # any other failed restore must not leave that firewall config behind eval { PVE::GuestHelpers::set_firewall_config_raw($vmid, $firewall_conf) }; - warn "unable to reset the firewall config - $@" if $@; + if (my $firewall_err = $@) { + warn "unable to reset the firewall config - $firewall_err"; + push @left, 'the firewall config'; + } + + if (!$reservation->{existed}) { + if (@left) { + # what is left keeps the VMID, so that no other guest gets it + log_warn("keeping VM $vmid locked, the cleanup left behind " + . join(', ', @left)); + } else { + # Release the VMID while the guest lock is still held, so that no other + # operation can take the VMID in between and then lose its config here. + # The firewall config is settled before that, as another node can take + # the VMID as soon as the config is gone. + eval { PVE::QemuConfig->destroy_config($vmid) }; + warn $@ if $@; + } + } die $err; }; @@ -1735,13 +1759,26 @@ __PACKAGE__->register_method({ $code = sub { eval { $restorefn->() }; if (my $err = $@) { + # For a VMID that was free before, the restore already removed the config, or + # kept it locked because its cleanup left something behind. Only unlock a VM + # that existed before or that keeps its restored volumes. + if (!$reservation->{existed} && !$restore_progress->{'volumes-kept'}) { + # the reservation of a restore that did not start is untouched, for example + # if taking its guest lock timed out + if (!$restore_started) { + eval { PVE::QemuConfig->remove_unused_reservation($vmid) }; + warn $@ if $@; + } + die $err; + } + eval { PVE::QemuConfig->remove_lock($vmid, 'create') }; warn $@ if $@; if ($restored_data) { warn "error after data was restored, VM disks should be OK but config may " . "require adaptions. VM $vmid state is NOT cleaned up.\n"; - } else { + } elsif ($restore_progress->{'target-changed'}) { warn "error before or during data restore, some or all disks were not " . "completely restored. VM $vmid state is NOT cleaned up.\n"; } diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index 3104fa73..d08dc33e 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -6422,30 +6422,48 @@ sub tar_archive_read_firstfile { return $firstfile; } +# Returns the number of volumes that could not be removed. qmextract creates the stat file before it +# allocates a volume, so a stat file or a line of it that cannot be read counts as one. sub tar_restore_cleanup { my ($storecfg, $statfile) = @_; print STDERR "starting cleanup\n"; - if (my $fd = IO::File->new($statfile, "r")) { - while (defined(my $line = <$fd>)) { - if ($line =~ m/vzdump:([^\s:]*):(\S+)$/) { - my $volid = $2; - eval { - if (classify_drive_file($volid) eq "absolute") { - unlink($volid) || $!{ENOENT} || die "unlink failed - $!\n"; - } else { - PVE::Storage::vdisk_free($storecfg, $volid); - } - print STDERR "temporary volume '$volid' successfully removed\n"; - }; - print STDERR "unable to cleanup '$volid' - $@" if $@; - } else { - print STDERR "unable to parse line in statfile - $line"; + my $fd = IO::File->new($statfile, "r"); + if (!$fd) { + return 0 if $!{ENOENT}; + print STDERR "unable to read '$statfile' - $!\n"; + return 1; + } + + my $left = 0; + while (defined(my $line = <$fd>)) { + if ($line =~ m/vzdump:([^\s:]*):(\S+)$/) { + my $volid = $2; + eval { + if (classify_drive_file($volid) eq "absolute") { + unlink($volid) || $!{ENOENT} || die "unlink failed - $!\n"; + } else { + PVE::Storage::vdisk_free($storecfg, $volid); + } + print STDERR "temporary volume '$volid' successfully removed\n"; + }; + if (my $err = $@) { + print STDERR "unable to cleanup '$volid' - $err"; + $left++; } + } else { + print STDERR "unable to parse line in statfile - $line"; + $left++; } - $fd->close(); } + if ($fd->error()) { + print STDERR "unable to read '$statfile'\n"; + $left++; + } + $fd->close(); + + return $left; } sub restore_file_archive { @@ -6712,17 +6730,24 @@ my $restore_deactivate_volumes = sub { print STDERR $@ if $@; }; +# Returns the number of volumes that could not be removed. my $restore_destroy_volumes = sub { my ($storecfg, $virtdev_hash) = @_; + my $left = 0; for my $dev (values $virtdev_hash->%*) { my $volid = $dev->{volid} or next; eval { PVE::Storage::vdisk_free($storecfg, $volid); print STDERR "temporary volume '$volid' successfully removed\n"; }; - print STDERR "unable to cleanup '$volid' - $@" if $@; + if (my $err = $@) { + print STDERR "unable to cleanup '$volid' - $err"; + $left++; + } } + + return $left; }; sub restore_merge_config { @@ -6972,6 +6997,7 @@ sub restore_proxmox_backup_archive { $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn]; PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param); + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); if ($has_firewall_config) { @@ -7061,7 +7087,9 @@ sub restore_proxmox_backup_archive { rmtree $tmpdir; if ($err) { - $restore_destroy_volumes->($storecfg, $virtdev_hash); + # what could not be removed keeps the VMID taken + $options->{progress}->{'volumes-left'} = 1 + if $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } $options->{progress}->{'volumes-kept'} = 1; @@ -7151,6 +7179,7 @@ sub restore_external_archive { my $data = $backup_provider->archive_get_guest_config($volname) or die "backup provider failed to extract guest configuration\n"; + $options->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); PVE::Tools::file_set_contents($cfgfn, $data); @@ -7235,7 +7264,9 @@ sub restore_external_archive { rmtree($tmpdir); if ($err) { - $restore_destroy_volumes->($storecfg, $virtdev_hash); + # what could not be removed keeps the VMID taken + $options->{progress}->{'volumes-left'} = 1 + if $restore_destroy_volumes->($storecfg, $virtdev_hash); die $err; } $options->{progress}->{'volumes-kept'} = 1; @@ -7563,6 +7594,7 @@ sub restore_vma_archive { my $print_devmap = sub { my $cfgfn = "$tmpdir/qemu-server.conf"; + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # we can read the config - that is already extracted @@ -7699,7 +7731,9 @@ sub restore_vma_archive { rmtree $tmpdir; if ($err) { - $restore_destroy_volumes->($cfg, $virtdev_hash); + # what could not be removed keeps the VMID taken + $opts->{progress}->{'volumes-left'} = 1 + if $restore_destroy_volumes->($cfg, $virtdev_hash); die $err; } $opts->{progress}->{'volumes-kept'} = 1; @@ -7730,6 +7764,7 @@ sub restore_tar_archive { my $storecfg = PVE::Storage::config(); + $opts->{progress}->{'target-changed'} = 1; PVE::ReplicationState::delete_guest_states($vmid); # avoid zombie disks when restoring over an existing VM -> cleanup first @@ -7809,7 +7844,9 @@ sub restore_tar_archive { $srcfd->close(); }; if (my $err = $@) { - tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat") if !$opts->{info}; + # what could not be removed keeps the VMID taken + $opts->{progress}->{'volumes-left'} = 1 + if !$opts->{info} && tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat"); die $err; } $opts->{progress}->{'volumes-kept'} = 1; diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl index d91b46b5..f5f9e026 100755 --- a/src/test/run_qemu_restore_config_tests.pl +++ b/src/test/run_qemu_restore_config_tests.pl @@ -5,7 +5,7 @@ use warnings; use lib qw(.. .); -use Fcntl qw(O_RDONLY O_NONBLOCK); +use Fcntl qw(:flock O_RDONLY O_NONBLOCK); use File::Path qw(make_path remove_tree); use File::Temp qw(tempdir); use JSON; @@ -98,6 +98,9 @@ my $BACKUP_FIREWALL = "[OPTIONS]\nenable: 1\n"; our $current_backup; # raw config of the backup # where the restore fails: 'read-config', 'command', 'data', 'after-restore' or 'live' our $fail_at = ''; +# the stat file that qmextract leaves: undef for none, 'directory' for one that cannot be read, or +# its content +our $tar_stat_file; my $calls = {}; # calls with effects on the target VM that tests check sub backup_config { @@ -216,6 +219,16 @@ $qemu_server_module->mock( if (!ref($cmd) && $cmd =~ m/^zcat -f\|tar xf /) { die "unable to read the backup\n" if $fail_at eq 'read-config'; file_set_contents("$ENV{VZDUMP_TMPDIR}/qemu-server.conf", $current_backup); + # a failed restore leaves its temporary directory behind + my $statfile = "$ENV{VZDUMP_TMPDIR}/qmrestore.stat"; + remove_tree($statfile); + if (defined($tar_stat_file)) { + if ($tar_stat_file eq 'directory') { + make_path($statfile); + } else { + file_set_contents($statfile, $tar_stat_file); + } + } return 0; } @@ -301,6 +314,7 @@ sub restore_api { my $TARGET_CONF = "memory: 1024\nmeta: creation-qemu=10.0.0,ctime=1700000000\nname: target\n"; my $NOT_CLEANED_UP = qr/VM 100 state is NOT cleaned up/; +my $KEPT_LOCKED = 'WARN: keeping VM 100 locked, the cleanup left behind'; for my $type (sort keys $ARCHIVES->%*) { subtest "$type: restore to a free VMID" => sub { @@ -319,6 +333,34 @@ for my $type (sort keys $ARCHIVES->%*) { }; } +for my $type (sort keys $ARCHIVES->%*) { + subtest "$type: failed restore to a free VMID" => sub { + reset_cluster(); + + local $fail_at = 'data'; + my ($err, $output) = restore_api($type, backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + ok(!defined(stored_config(100)), 'reserved config removed'); + is(PVE::GuestHelpers::get_firewall_config_raw(100), undef, 'no firewall config left'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); + + $fail_at = ''; + ($err, $output) = restore_api($type, backup_config(), {}); + is($err, '', 'retry needs no force') or diag($output); + like(stored_config(100), qr/^name: backup$/m, 'VM restored'); + }; +} + +subtest 'failed restore to a free VMID before reading the backup' => sub { + reset_cluster(); + + local $fail_at = 'read-config'; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "unable to read the backup\n", 'restore failed'); + ok(!defined(stored_config(100)), 'reserved config removed'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind'); +}; + subtest 'vma: an error of the restore setup is reported without the command line' => sub { reset_cluster(); local $fail_at = 'data'; @@ -380,6 +422,128 @@ subtest 'failure once the data is restored keeps the firewall config of the back } }; +subtest 'firewall config that cannot be put back keeps the VMID' => sub { + reset_cluster(); + + # only the removal of the firewall config of the backup fails + my $set_firewall_config_raw = \&PVE::GuestHelpers::set_firewall_config_raw; + local *PVE::GuestHelpers::set_firewall_config_raw = sub { + my ($vmid, $raw) = @_; + die "unable to remove the file - Permission denied\n" if !defined($raw); + return $set_firewall_config_raw->($vmid, $raw); + }; + + local $fail_at = 'data'; + my ($err, $output) = restore_api('pbs', backup_config(), {}); + is($err, "restoring the data failed\n", 'restore failed'); + my $reported = 'unable to reset the firewall config - unable to remove the file'; + like($output, qr/^\Q$reported\E - Permission denied$/m, 'failure is reported'); + like($output, qr/^\Q$KEPT_LOCKED\E the firewall config$/m, 'what is left is reported'); + like(stored_config(100), qr/^lock: create$/m, 'reservation is kept, locked'); + unlike($output, $NOT_CLEANED_UP, 'no other warning about the VM left behind'); +}; + +subtest 'firewall config is put back before the VMID is released' => sub { + reset_cluster(); + + my $reserved; + my $set_firewall_config_raw = \&PVE::GuestHelpers::set_firewall_config_raw; + local *PVE::GuestHelpers::set_firewall_config_raw = sub { + $reserved = defined(stored_config(100)); + return $set_firewall_config_raw->(@_); + }; + + local $fail_at = 'data'; + restore_api('pbs', backup_config(), {}); + ok($reserved, 'VMID was still reserved'); + ok(!defined(stored_config(100)), 'and is released afterwards'); +}; + +subtest 'tar: an allocation record that cannot be read keeps the VMID' => sub { + for my $case ( + ['line that cannot be parsed', "garbage\n", qr/^unable to parse line in statfile/m], + [ + 'stat file that cannot be read', + 'directory', + qr/^unable to read '.*qmrestore\.stat'/m, + ], + ) { + my ($what, $stat_file, $reported) = $case->@*; + reset_cluster(); + + local $fail_at = 'data'; + local $tar_stat_file = $stat_file; + my ($err, $output) = restore_api('tar', backup_config(), {}); + is($err, "restoring the data failed\n", "$what: restore failed"); + like($output, $reported, "$what: reported"); + like(stored_config(100), qr/^lock: create$/m, "$what: reservation is kept, locked"); + like( + $output, + qr/^\Q$KEPT_LOCKED\E volumes of the restore$/m, + "$what: what is left is reported", + ); + } +}; + +subtest 'failed removal of a restored volume keeps the VMID' => sub { + reset_cluster(); + + my $allocated = []; + my $run_raw_client_cmd = \&PVE::Storage::PBSPlugin::run_raw_client_cmd; + local *PVE::Storage::PBSPlugin::run_raw_client_cmd = sub { + my ($scfg, $storeid, $cmd, $param) = @_; + my (undef, $file, $target) = $param->@*; + return $run_raw_client_cmd->(@_) if $file ne 'index.json'; + my $files = [ + { filename => 'qemu-server.conf.blob' }, + { filename => 'drive-scsi0.img.fidx', size => 1048576 }, + ]; + file_set_contents($target, encode_json({ files => $files })); + }; + local *PVE::Storage::storage_config = sub { + my ($cfg, $storeid) = @_; + return { type => 'pbs', server => 'localhost', datastore => 'ds' } if $storeid eq 'pbs'; + return { type => 'dir', path => '/nonexistent', content => { images => 1 } }; + }; + local *PVE::Storage::resolve_format_hint = sub { return 'raw'; }; + local *PVE::Storage::vdisk_alloc = sub { + my ($cfg, $storeid, $vmid) = @_; + push $allocated->@*, "$storeid:vm-$vmid-disk-0"; + return "$storeid:vm-$vmid-disk-0"; + }; + local *PVE::Storage::activate_volumes = sub { return; }; + local *PVE::Storage::path = sub { return '/nonexistent/vm-100-disk-0.raw'; }; + local *PVE::Storage::volume_has_feature = sub { return 0; }; + local *PVE::Storage::vdisk_free = sub { die "volume is busy\n"; }; + my $run_command = \&PVE::QemuServer::run_command; + local *PVE::QemuServer::run_command = sub { + my ($cmd) = @_; + return 0 if ref($cmd) && $cmd->[0] eq '/usr/bin/pbs-restore'; + return $run_command->(@_); + }; + + local $fail_at = 'data'; + my $backup = backup_config() + . "scsi0: target:vm-100-disk-0,size=1M\n#qmdump#map:scsi0:drive-scsi0:target:raw:\n"; + my ($err, $output) = restore_api('pbs', $backup, {}); + is($err, "restoring the data failed\n", 'restore failed after allocating the volume'); + is_deeply($allocated, ['target:vm-100-disk-0'], 'volume was allocated'); + like($output, qr/^unable to cleanup 'target:vm-100-disk-0' - volume is busy$/m, 'reported'); + like(stored_config(100), qr/^lock: create$/m, 'reservation is kept, locked'); + like( + $output, qr/^\Q$KEPT_LOCKED\E volumes of the restore$/m, 'what is left is reported', + ); + + # the leftover is no VM to restore over + $fail_at = ''; + ($err) = restore_api('pbs', backup_config(), { force => 1 }); + is( + $err, + "unable to restore VM 100 - VM is locked (create)\n", + 'retry needs an unlock first', + ); +}; + subtest "'force' does not overwrite a VM created after the checks" => sub { reset_cluster(); @@ -392,6 +556,38 @@ subtest "'force' does not overwrite a VM created after the checks" => sub { is(stored_config(100), $TARGET_CONF, 'VM created in between unchanged'); }; +subtest 'a restore that cannot take its guest lock removes the reservation' => sub { + my $lock_config_full = PVE::QemuConfig->can('lock_config_full'); + my $before_timeout = sub { }; + local *PVE::QemuConfig::lock_config_full = sub { + my ($class, $vmid, $timeout, @param) = @_; + # the restore waits one second for the lock + if ($timeout == 1) { + $before_timeout->(); + die "can't lock file 'lock-$vmid.conf' - got timeout\n"; + } + return $lock_config_full->($class, $vmid, $timeout, @param); + }; + + reset_cluster(); + my ($err, $output) = restore_api('pbs', backup_config(), {}); + like($err, qr/got timeout/, 'free VMID: restore failed'); + ok(!defined(stored_config(100)), 'free VMID: reservation removed'); + + reset_cluster($TARGET_CONF); + ($err, $output) = restore_api('pbs', backup_config(), { force => 1 }); + like($err, qr/got timeout/, 'existing VM: restore failed'); + like(stored_config(100), qr/^name: target$/m, 'existing VM: kept'); + unlike(stored_config(100), qr/^lock:/m, 'existing VM: lock removed'); + + # without the lock of the reservation, the config is not one of this restore anymore + reset_cluster(); + $before_timeout = sub { QemuServerTest::FakeCFS::set_config(100, "memory: 512\n"); }; + ($err, $output) = restore_api('pbs', backup_config(), {}); + like($err, qr/got timeout/, 'config changed meanwhile: restore failed'); + is(stored_config(100), "memory: 512\n", 'config changed meanwhile: kept'); +}; + subtest 'the permission to overwrite a VM does not cover creating it' => sub { # the VM goes away after the permission checks local *PVE::Storage::check_volume_access = @@ -418,6 +614,41 @@ subtest 'the permission to overwrite a VM does not cover creating it' => sub { is_deeply($QemuServerTest::FakeCFS::written, {}, 'with VM.Backup only: nothing written'); }; +subtest 'another restore right after a failed restore to a free VMID' => sub { + reset_cluster(); + + # run a second restore with 'force' as soon as the failed one releases the guest lock + my $lock_config_full = PVE::QemuConfig->can('lock_config_full'); + my ($interposed, $lock_free, $second_err) = (0, 0, ''); + local *PVE::QemuConfig::lock_config_full = sub { + my ($class, $vmid, @param) = @_; + my $res = eval { $lock_config_full->($class, $vmid, @param) }; + my $err = $@; + if ($err && !$interposed++) { + open(my $fh, '>>', PVE::QemuConfig->config_file_lock($vmid)) or die "$!\n"; + $lock_free = flock($fh, LOCK_EX | LOCK_NB); + close($fh); + local $fail_at = ''; + ($second_err) = restore_api('pbs', backup_config(), { force => 1 }); + } + die $err if $err; + return $res; + }; + + local $fail_at = 'data'; + my ($err) = restore_api('vma file', backup_config(), {}); + + is($err, "restoring the data failed\n", 'first restore failed'); + ok($lock_free, 'guest lock was free for the second restore'); + is($second_err, '', 'second restore succeeded'); + like(stored_config(100), qr/^name: backup$/m, 'its config stays'); + is( + PVE::GuestHelpers::get_firewall_config_raw(100), + $BACKUP_FIREWALL, + 'and its firewall config', + ); +}; + for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { subtest "$type: failed restore over an existing VM" => sub { reset_cluster($TARGET_CONF); @@ -427,6 +658,7 @@ for my $type ('pbs', 'provider', 'vma file', 'vma stdin') { like($err, qr/unable to read the backup\n$/, 'fails before changing the VM'); is(stored_config(100), $TARGET_CONF, 'config unchanged'); is_deeply($calls, {}, 'replication state and firewall config untouched'); + unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM'); $fail_at = 'data'; ($err, $output) = restore_api($type, backup_config(), { force => 1 }); -- 2.47.3