all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu-server v2 12/36] api: restore: release the VMID if a restore to an unused VMID fails
Date: Tue, 17 Nov 2026 23:34:22 +0100	[thread overview]
Message-ID: <20261117223945.862914-13-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com>

A failed restore to a previously unused VMID left its temporary or
empty config behind, even when the restore cleaned up the volumes it
had allocated. The VMID stayed taken and a retry needed 'force'.
Containers already remove their config in this case.

Remove the VM config and the restored firewall config if the VMID was
unused before the operation and the restore has not reached the point
from which it keeps the restored volumes. Keep the VM after later
failures, such as a failure in the live phase of a live restore. If the
next-id setting enforces unique guest IDs, the released VMID counts as
used, so a retry needs another VMID, as for containers.

Remove the config before releasing the guest config lock. The temporary
config of a VMA restore has no lock property, so removing it later could
delete the config of another restore that took the VMID in the meantime.
Remove the firewall config before the VM config for a similar reason:
the guest config lock is local to the node, so another node can take the
VMID once the VM config is gone. The lock is held, so the VM cannot run
unprotected in between.

Only warn that the VM was not cleaned up if the restore changed the
target and left it behind. A failure to read the backup should not
produce that warning.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v1 -> v2:
* remove the firewall config before the VM config, with a test for the
  order
* note the effect with enforced unique guest IDs in the commit message
* test helper: use v5.36, mock the registration of used guest IDs

 src/PVE/API2/Qemu.pm                      |  29 +-
 src/PVE/QemuServer.pm                     |   8 +
 src/test/QemuServerTest/FakeCFS.pm        | 185 +++++++++++
 src/test/run_qemu_restore_config_tests.pl | 370 ++++++++++++++++++++--
 4 files changed, 571 insertions(+), 21 deletions(-)
 create mode 100644 src/test/QemuServerTest/FakeCFS.pm

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 89a3ad9d..be0928ea 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -1421,10 +1421,12 @@ __PACKAGE__->register_method({
 
         my $emsg = $is_restore ? "unable to restore VM $vmid -" : "unable to create VM $vmid -";
 
-        eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) };
+        my $reservation = eval { PVE::QemuConfig->create_and_lock_config($vmid, $force) };
         die "$emsg $@" if $@;
 
         my $restored_data = 0;
+        # the restore paths record whether they changed the VM and whether restored volumes stay
+        my $restore_progress = {};
         my $restorefn = sub {
             my $conf = PVE::QemuConfig->load_config($vmid);
 
@@ -1440,6 +1442,7 @@ __PACKAGE__->register_method({
                     bwlimit => $bwlimit,
                     live => $live_restore,
                     override_conf => $param,
+                    progress => $restore_progress,
                 };
                 if (my $volid = $archive->{volid}) {
                     # best effort, real check is after restoring!
@@ -1497,7 +1500,25 @@ __PACKAGE__->register_method({
                     if $unique;
             };
 
-            PVE::QemuConfig->lock_config_full($vmid, 1, $realcmd);
+            my $restore_locked = sub {
+                eval { $realcmd->() };
+                my $err = $@ or return;
+
+                if (!$reservation->{existed} && !$restore_progress->{'volumes-kept'}) {
+                    # Remove the reservation before releasing the guest lock, so another
+                    # operation cannot take ownership first. The firewall config goes first,
+                    # before the VMID is freed. The lock is held, so the VM cannot run unprotected.
+                    eval {
+                        PVE::Firewall::remove_vmfw_conf($vmid);
+                        PVE::QemuConfig->destroy_config($vmid);
+                        $restore_progress->{'reservation-removed'} = 1;
+                    };
+                    warn $@ if $@;
+                }
+                die $err;
+            };
+
+            PVE::QemuConfig->lock_config_full($vmid, 1, $restore_locked);
 
             if ($start_after_create && !$live_restore) {
                 print "Execute autostart\n";
@@ -1630,13 +1651,15 @@ __PACKAGE__->register_method({
             $code = sub {
                 eval { $restorefn->() };
                 if (my $err = $@) {
+                    die $err if $restore_progress->{'reservation-removed'};
+
                     eval { PVE::QemuConfig->remove_lock($vmid, 'create') };
                     warn $@ if $@;
                     if ($restored_data) {
                         warn
                             "error after data was restored, VM disks should be OK but config may "
                             . "require adaptions. VM $vmid state is NOT cleaned up.\n";
-                    } else {
+                    } elsif ($restore_progress->{'target-changed'}) {
                         warn "error before or during data restore, some or all disks were not "
                             . "completely restored. VM $vmid state is NOT cleaned up.\n";
                     }
diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index cf0067c3..d8077ba0 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -6960,6 +6960,7 @@ sub restore_proxmox_backup_archive {
         $param = [$pbs_backup_name, "qemu-server.conf", $cfgfn];
         PVE::Storage::PBSPlugin::run_raw_client_cmd($scfg, $storeid, $cmd, $param);
 
+        $options->{progress}->{'target-changed'} = 1;
         PVE::ReplicationState::delete_guest_states($vmid);
 
         if ($has_firewall_config) {
@@ -7052,6 +7053,7 @@ sub restore_proxmox_backup_archive {
         $restore_destroy_volumes->($storecfg, $virtdev_hash);
         die $err;
     }
+    $options->{progress}->{'volumes-kept'} = 1;
 
     if ($options->{live}) {
         # keep lock during live-restore
@@ -7139,6 +7141,7 @@ sub restore_external_archive {
         my $data = $backup_provider->archive_get_guest_config($volname)
             or die "backup provider failed to extract guest configuration\n";
 
+        $options->{progress}->{'target-changed'} = 1;
         PVE::ReplicationState::delete_guest_states($vmid);
 
         PVE::Tools::file_set_contents($cfgfn, $data);
@@ -7226,6 +7229,7 @@ sub restore_external_archive {
         $restore_destroy_volumes->($storecfg, $virtdev_hash);
         die $err;
     }
+    $options->{progress}->{'volumes-kept'} = 1;
 
     my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf});
     check_restore_permissions($rpcenv, $user, $new_conf);
@@ -7551,6 +7555,7 @@ sub restore_vma_archive {
     my $print_devmap = sub {
         my $cfgfn = "$tmpdir/qemu-server.conf";
 
+        $opts->{progress}->{'target-changed'} = 1;
         PVE::ReplicationState::delete_guest_states($vmid);
 
         # we can read the config - that is already extracted
@@ -7680,6 +7685,7 @@ sub restore_vma_archive {
         $restore_destroy_volumes->($cfg, $virtdev_hash);
         die $err;
     }
+    $opts->{progress}->{'volumes-kept'} = 1;
 
     my $new_conf = restore_merge_config($conffile, $new_conf_raw, $opts->{override_conf});
     check_restore_permissions($rpcenv, $user, $new_conf);
@@ -7707,6 +7713,7 @@ sub restore_tar_archive {
 
     my $storecfg = PVE::Storage::config();
 
+    $opts->{progress}->{'target-changed'} = 1;
     PVE::ReplicationState::delete_guest_states($vmid);
 
     # avoid zombie disks when restoring over an existing VM -> cleanup first
@@ -7789,6 +7796,7 @@ sub restore_tar_archive {
         tar_restore_cleanup($storecfg, "$tmpdir/qmrestore.stat") if !$opts->{info};
         die $err;
     }
+    $opts->{progress}->{'volumes-kept'} = 1;
 
     rmtree $tmpdir;
 
diff --git a/src/test/QemuServerTest/FakeCFS.pm b/src/test/QemuServerTest/FakeCFS.pm
new file mode 100644
index 00000000..8eacc736
--- /dev/null
+++ b/src/test/QemuServerTest/FakeCFS.pm
@@ -0,0 +1,185 @@
+package QemuServerTest::FakeCFS;
+
+# Fake cluster filesystem for VM config files, so that tests can use the real config class methods
+# of PVE::QemuConfig, including the guest identity handling of write_config, without pmxcfs.
+#
+# Configs are kept as raw text, indexed by VMID, and an empty file at the path that config_file()
+# returns exists for each, for checks of the file's existence. Generated guest identities are
+# deterministic: the n-th generated uid is 'a' repeated 17 times followed by the base32 encoding of
+# n, and rtime is $now_ms.
+
+use v5.36;
+
+use File::Basename qw(dirname);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+
+use PVE::GuestHelpers;
+use PVE::GuestID;
+use PVE::GuestIdentity;
+use PVE::QemuConfig;
+use PVE::QemuServer;
+
+our $configs = {}; # VMID => raw config, for VMs on the local node
+our $other_guests = {}; # VMID => raw 'meta' property of other guests, like on other nodes or CTs
+our $now_ms = 1700000000000;
+our $generated = 0;
+our $lock_domain_error;
+our $writes = [];
+
+my $tmpdir = tempdir('fake-cfs-XXXXXX', DIR => '.', CLEANUP => 1);
+make_path("$tmpdir/lock");
+$PVE::GuestHelpers::lockdir = "$tmpdir/lock";
+
+sub reset() {
+    remove_tree("$tmpdir/etc-pve");
+    $configs = {};
+    $other_guests = {};
+    $now_ms = 1700000000000;
+    $generated = 0;
+    $lock_domain_error = undef;
+    $writes = [];
+}
+
+my sub vmid_of_path($path) {
+    return $path =~ m|/qemu-server/(\d+)\.conf$| ? $1 : undef;
+}
+
+# Creates or removes the file at the path of the config of $vmid, matching the stored config.
+my sub sync_config_file($vmid) {
+    my $path = PVE::QemuConfig->config_file($vmid);
+    if (defined($configs->{$vmid})) {
+        make_path(dirname($path));
+        open(my $fh, '>', $path) or die "unable to create '$path' - $!\n";
+        close($fh);
+    } else {
+        unlink($path);
+    }
+}
+
+# Stores the raw config $raw for $vmid, like a manual edit of the config file.
+sub set_config($vmid, $raw) {
+    $configs->{$vmid} = $raw;
+    sync_config_file($vmid);
+}
+
+# Returns the last 'meta' property of the top-level section of the raw config $raw, like the config
+# class reads the stored identity.
+my sub top_level_meta($raw) {
+    my $meta;
+    for my $line (split(/\n/, $raw // '')) {
+        last if $line =~ m/^\[/;
+        $meta = $1 if $line =~ m/^meta:\s*(.*?)\s*$/;
+    }
+
+    return $meta;
+}
+
+# Returns the identity stored in the raw config of $vmid, undef if there is none.
+sub stored_identity($vmid) {
+    return PVE::GuestIdentity::parse_identity(top_level_meta($configs->{$vmid}));
+}
+
+# Returns the uid that the $n-th generation of an identity yields.
+sub generated_uid($n) {
+    my $alphabet = 'abcdefghijklmnopqrstuvwxyz234567';
+    return ('a' x 17) . join('', map { substr($alphabet, ($n >> (5 * $_)) & 31, 1) } (2, 1, 0));
+}
+
+# the list of used guest IDs is not part of the fake cluster filesystem
+our $guest_id_module = Test::MockModule->new('PVE::GuestID');
+$guest_id_module->mock(register_used_id => sub { return; });
+
+our $guest_identity_module = Test::MockModule->new('PVE::GuestIdentity');
+$guest_identity_module->mock(
+    __random_bytes => sub {
+        my ($len) = @_;
+        die "unexpected length $len\n" if $len != 20;
+        $generated++;
+        # one byte per uid character, the last three hold 5 bits of the counter each
+        return ("\0" x 17) . pack('C3', map { ($generated >> $_) & 31 } (10, 5, 0));
+    },
+    __time_ms => sub {
+        return $now_ms;
+    },
+);
+
+our $cluster_module = Test::MockModule->new('PVE::Cluster');
+$cluster_module->mock(
+    get_config => sub {
+        my ($path) = @_;
+        my $vmid = vmid_of_path($path) // die "get_config (mocked) - unexpected path '$path'\n";
+        return $configs->{$vmid};
+    },
+    cfs_read_file => sub {
+        my ($path) = @_;
+        return {} if $path eq 'datacenter.cfg';
+        my $vmid = vmid_of_path($path)
+            // die "cfs_read_file (mocked) - unexpected path '$path'\n";
+        return PVE::QemuServer::parse_vm_config($path, $configs->{$vmid});
+    },
+    cfs_write_file => sub {
+        my ($path, $conf) = @_;
+        my $vmid = vmid_of_path($path)
+            // die "cfs_write_file (mocked) - unexpected path '$path'\n";
+        $configs->{$vmid} = PVE::QemuServer::write_vm_config($path, $conf);
+        sync_config_file($vmid);
+        push $writes->@*, $vmid;
+        return;
+    },
+    cfs_update => sub {
+        return;
+    },
+    check_vmid_unused => sub {
+        my ($vmid, $noerr) = @_;
+        return 1 if !defined($configs->{$vmid}) && !defined($other_guests->{$vmid});
+        return undef if $noerr;
+        die "VM $vmid already exists\n";
+    },
+    cfs_lock_domain => sub {
+        my ($domain, $timeout, $code, @param) = @_;
+        die "cfs_lock_domain (mocked) - unexpected domain '$domain'\n"
+            if $domain ne 'guest-identity';
+        if ($lock_domain_error) {
+            $@ = $lock_domain_error;
+            return undef;
+        }
+        my $res = eval { $code->(@param) };
+        return $res;
+    },
+    get_guest_config_properties => sub {
+        my ($properties) = @_;
+        die "get_guest_config_properties (mocked) - unexpected properties\n"
+            if join(',', $properties->@*) ne 'meta';
+
+        my $res = {};
+        for my $vmid (keys $configs->%*) {
+            my $meta = top_level_meta($configs->{$vmid});
+            $res->{$vmid} = { meta => $meta } if defined($meta);
+        }
+        $res->{$_} = { meta => $other_guests->{$_} } for keys $other_guests->%*;
+        return $res;
+    },
+);
+
+our $qemu_config_module = Test::MockModule->new('PVE::QemuConfig');
+$qemu_config_module->mock(
+    config_file => sub {
+        my ($class, $vmid, $node) = @_;
+        return "$tmpdir/etc-pve/" . $class->cfs_config_path($vmid, $node);
+    },
+    config_file_lock => sub {
+        my ($class, $vmid) = @_;
+        return "$tmpdir/lock/lock-$vmid.conf";
+    },
+    destroy_config => sub {
+        my ($class, $vmid) = @_;
+        die "failed to remove config file: No such file or directory\n"
+            if !defined($configs->{$vmid});
+        delete $configs->{$vmid};
+        sync_config_file($vmid);
+    },
+);
+
+1;
diff --git a/src/test/run_qemu_restore_config_tests.pl b/src/test/run_qemu_restore_config_tests.pl
index 20cb3df0..ab4dd887 100755
--- a/src/test/run_qemu_restore_config_tests.pl
+++ b/src/test/run_qemu_restore_config_tests.pl
@@ -3,35 +3,30 @@
 use strict;
 use warnings;
 
-use lib qw(..);
+use lib qw(.. .);
 
+use Fcntl qw(:flock O_RDONLY O_NONBLOCK);
+use File::Path qw(make_path);
+use File::Temp qw(tempdir);
+use JSON;
 use Test::MockModule;
 use Test::More;
 
 use File::Basename;
 
+use PVE::API2::Qemu;
+use PVE::QemuConfig;
 use PVE::QemuServer;
-use PVE::Tools qw(dir_glob_foreach file_get_contents);
+use PVE::QemuServer::Helpers;
+use PVE::RPCEnvironment;
+use PVE::Tools qw(dir_glob_foreach file_get_contents file_set_contents);
+
+# also provides the 'datacenter.cfg' that restore_update_config_line reads
+use QemuServerTest::FakeCFS;
 
 my $INPUT_DIR = './restore-config-input';
 my $EXPECTED_DIR = './restore-config-expected';
 
-# NOTE update when you add/remove tests
-plan tests => 5;
-
-my $pve_cluster_module = Test::MockModule->new("PVE::Cluster");
-$pve_cluster_module->mock(
-    cfs_read_file => sub {
-        my ($file) = @_;
-
-        if ($file eq 'datacenter.cfg') {
-            return {};
-        } else {
-            die "'cfs_read_file' called - missing mock?\n";
-        }
-    },
-);
-
 dir_glob_foreach(
     './restore-config-input',
     '[0-9]+.conf',
@@ -78,4 +73,343 @@ dir_glob_foreach(
     },
 );
 
+# Restores through the create API and the restore paths for PBS, backup providers, VMA and legacy
+# tar archives, against the fake cluster filesystem. Storage, archive tools and other external
+# effects are mocked. The backups have no disks, so that no volumes get allocated.
+
+my $runtime_dir = tempdir('restore-run-XXXXXX', DIR => '.', CLEANUP => 1);
+$PVE::QemuServer::Helpers::var_run_tmpdir = $runtime_dir;
+
+my $rpcenv = PVE::RPCEnvironment->init('cli');
+$rpcenv->set_user('root@pam');
+
+my $ARCHIVES = {
+    pbs => 'pbs:backup/vm/100/2025-01-01T00:00:00Z',
+    provider => 'provider:backup/vm-100-2025-01-01',
+    'vma file' => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.vma',
+    'vma stdin' => '-',
+    tar => '/var/lib/vz/dump/vzdump-qemu-100-2025_01_01-00_00_00.tar',
+};
+
+our $current_backup; # raw config of the backup
+our $fail_at = ''; # where the restore fails: 'read-config', 'data', 'after-restore' or 'live'
+my $calls = {}; # calls with effects on the target VM that tests check
+
+sub backup_config {
+    my ($meta) = @_;
+
+    return
+        "boot: order=scsi0\ncores: 1\nmemory: 512\n"
+        . (defined($meta) ? "meta: $meta\n" : '')
+        . "name: backup\n";
+}
+
+package TestBackupProvider {
+
+    sub new {
+        my ($class) = @_;
+        return bless {}, $class;
+    }
+
+    sub restore_get_mechanism { return ('qemu-img', 'qemu'); }
+    sub restore_vm_init { return {}; }
+
+    sub archive_get_guest_config {
+        die "unable to read the backup\n" if $main::fail_at eq 'read-config';
+        return $main::current_backup;
+    }
+    sub archive_get_firewall_config { return "[OPTIONS]\nenable: 1\n"; }
+    sub restore_vm_cleanup { return; }
+}
+
+my $replication_state_module = Test::MockModule->new('PVE::ReplicationState');
+$replication_state_module->mock(delete_guest_states => sub { $calls->{delete_guest_states}++; });
+
+my $firewall_module = Test::MockModule->new('PVE::Firewall');
+$firewall_module->mock(
+    remove_vmfw_conf => sub {
+        my ($vmid) = @_;
+        $calls->{remove_vmfw_conf}++;
+        $calls->{'remove_vmfw_conf-reserved'}++
+            if defined($QemuServerTest::FakeCFS::configs->{$vmid});
+    },
+);
+
+my $tools_module = Test::MockModule->new('PVE::Tools');
+$tools_module->mock(
+    file_copy => sub {
+        my ($src, $dst) = @_;
+        die "file_copy (mocked) - unexpected destination '$dst'\n" if $dst !~ m|/firewall/|;
+        $calls->{firewall}++;
+    },
+);
+
+my $storage_module = Test::MockModule->new('PVE::Storage');
+$storage_module->mock(
+    abs_filesystem_path => sub {
+        my ($cfg, $path) = @_;
+        return $path;
+    },
+    check_volume_access => sub { return; },
+    config => sub { return {}; },
+    deactivate_volumes => sub { return 1; },
+    extract_vzdump_config => sub { return $current_backup; },
+    new_backup_provider => sub { return TestBackupProvider->new(); },
+    parse_volname => sub {
+        return ('backup', 'vm/100/2025-01-01T00:00:00Z', 100, undef, undef, undef, 'pbs-vm');
+    },
+    parse_volume_id => sub {
+        my ($volid, $noerr) = @_;
+        return ($1, $2) if $volid =~ m/^([a-z]+):(.+)$/;
+        die "unable to parse volume ID '$volid'\n" if !$noerr;
+        return;
+    },
+    path_to_volume_id => sub { return; },
+    storage_config => sub {
+        my ($cfg, $storeid) = @_;
+        return { type => 'pbs', server => 'localhost', datastore => 'ds' } if $storeid eq 'pbs';
+        return { type => 'dir', path => '/mnt/provider' };
+    },
+    storage_has_feature => sub {
+        my ($cfg, $storeid, $feature) = @_;
+        return $storeid eq 'provider' && $feature eq 'backup-provider';
+    },
+);
+
+my $pbs_plugin_module = Test::MockModule->new('PVE::Storage::PBSPlugin');
+$pbs_plugin_module->mock(
+    pbs_encryption_key_file_name => sub { return '/nonexistent/key'; },
+    pbs_get_password => sub { return 'secret'; },
+    run_raw_client_cmd => sub {
+        my ($scfg, $storeid, $cmd, $param) = @_;
+        my (undef, $file, $target) = $param->@*;
+        if ($file eq 'index.json') {
+            my $files =
+                [{ filename => 'qemu-server.conf.blob' }, { filename => 'fw.conf.blob' }];
+            file_set_contents($target, encode_json({ files => $files }));
+        } elsif ($file eq 'qemu-server.conf') {
+            die "unable to read the backup\n" if $fail_at eq 'read-config';
+            file_set_contents($target, $current_backup);
+        } elsif ($file eq 'fw.conf') {
+            file_set_contents($target, "[OPTIONS]\nenable: 1\n");
+        } else {
+            die "run_raw_client_cmd (mocked) - unexpected file '$file'\n";
+        }
+    },
+);
+
+my $network_module = Test::MockModule->new('PVE::QemuServer::Network');
+$network_module->mock(
+    create_ifaces_ipams_ips => sub {
+        die "unable to register IPs\n" if $fail_at eq 'after-restore';
+    },
+);
+
+my $qemu_server_module = Test::MockModule->new('PVE::QemuServer');
+$qemu_server_module->mock(
+    check_running => sub { return 0; },
+    pbs_live_restore => sub {
+        die "live restore failed\n" if $fail_at eq 'live';
+        $calls->{live_restore}++;
+    },
+    rescan => sub { return; },
+    restore_firewall_config => sub {
+        die "unable to restore the firewall config\n" if $fail_at eq 'firewall';
+        $calls->{firewall}++;
+    },
+    restore_update_config_line => sub {
+        die "restoring the data failed\n" if $fail_at eq 'data';
+        return $qemu_server_module->original('restore_update_config_line')->(@_);
+    },
+    tar_archive_read_firstfile => sub { return 'qemu-server.conf'; },
+    # like 'tar' with qmextract and 'vma extract': write the config and report the VMA header
+    run_command => sub {
+        my ($cmd, %param) = @_;
+
+        if (!ref($cmd) && $cmd =~ m/^zcat -f\|tar xf /) {
+            die "unable to read the backup\n" if $fail_at eq 'read-config';
+            file_set_contents("$ENV{VZDUMP_TMPDIR}/qemu-server.conf", $current_backup);
+            return 0;
+        }
+
+        my $vma = $cmd->[-1];
+        die "run_command (mocked) - unexpected command\n"
+            if ref($vma) ne 'ARRAY' || $vma->[0] ne 'vma';
+        my ($mapfifo, $tmpdir) = $vma->@[4, 6];
+
+        die "unable to read the backup\n" if $fail_at eq 'read-config';
+
+        make_path($tmpdir);
+        file_set_contents("$tmpdir/qemu-server.conf", $current_backup);
+        file_set_contents("$tmpdir/qemu-server.fw", "[OPTIONS]\nenable: 1\n");
+
+        sysopen(my $reader, $mapfifo, O_RDONLY | O_NONBLOCK)
+            or die "unable to open fifo - $!\n";
+        $param{afterfork}->();
+        $param{outfunc}->('CTIME: Wed Jan  1 00:00:00 2025');
+        close($reader);
+
+        return 0;
+    },
+);
+
+my $task_error;
+my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment');
+$rpcenv_module->mock(
+    check => sub { return 1; },
+    fork_worker => sub {
+        my ($self, $type, $id, $user, $code) = @_;
+        eval { $code->('UPID:test') };
+        $task_error = $@;
+        return 'UPID:test';
+    },
+);
+
+$QemuServerTest::FakeCFS::cluster_module->mock(check_cfs_quorum => sub { return 1; });
+
+# Restores a backup with the raw config $backup to VMID 100 through the create API, from an archive
+# of type $type with the additional API parameters $param. Returns the error of the API call or the
+# task and the task output.
+sub restore_api {
+    my ($type, $backup, $param) = @_;
+
+    $current_backup = $backup;
+    $calls = {};
+    $task_error = '';
+
+    my ($err, $output) = ('', '');
+    {
+        open(my $out_fh, '>', \$output);
+        local *STDERR = $out_fh;
+        my $oldfh = select($out_fh);
+
+        eval {
+            PVE::API2::Qemu->create_vm({
+                node => 'localhost',
+                vmid => 100,
+                archive => $ARCHIVES->{$type},
+                $param->%*,
+            });
+        };
+        $err = $@ || $task_error;
+
+        select($oldfh);
+        close($out_fh);
+    }
+
+    return ($err, $output);
+}
+
+my $TARGET_CONF = "memory: 1024\nmeta: creation-qemu=10.0.0,ctime=1700000000\nname: target\n";
+my $NOT_CLEANED_UP = qr/VM 100 state is NOT cleaned up/;
+
+for my $type (sort keys $ARCHIVES->%*) {
+    subtest "$type: failed restore to a free VMID" => sub {
+        QemuServerTest::FakeCFS::reset();
+
+        $fail_at = 'data';
+        my ($err, $output) = restore_api($type, backup_config(), {});
+        is($err, "restoring the data failed\n", 'restore failed');
+        ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed');
+        is($calls->{remove_vmfw_conf}, 1, 'restored firewall config removed');
+        is($calls->{'remove_vmfw_conf-reserved'}, 1, 'while the VMID was still reserved');
+        unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind');
+
+        $fail_at = '';
+        ($err, $output) = restore_api($type, backup_config(), { force => 1 });
+        is($err, '', 'retry succeeds') or diag($output);
+        like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'VM restored');
+    };
+}
+
+subtest 'failed restore to a free VMID before reading the backup' => sub {
+    QemuServerTest::FakeCFS::reset();
+
+    $fail_at = 'read-config';
+    my ($err, $output) = restore_api('pbs', backup_config(), {});
+    is($err, "unable to read the backup\n", 'restore failed');
+    ok(!defined($QemuServerTest::FakeCFS::configs->{100}), 'reserved config removed');
+    unlike($output, $NOT_CLEANED_UP, 'no warning about a VM left behind');
+};
+
+subtest 'another restore right after a failed restore to a free VMID' => sub {
+    QemuServerTest::FakeCFS::reset();
+
+    # run a second restore with 'force' as soon as the failed one releases the guest lock
+    my $lock_config_full = PVE::QemuConfig->can('lock_config_full');
+    my ($interposed, $lock_free, $second_err) = (0, 0, '');
+    $QemuServerTest::FakeCFS::qemu_config_module->mock(
+        lock_config_full => sub {
+            my ($class, $vmid, @param) = @_;
+            my $res = eval { $lock_config_full->($class, $vmid, @param) };
+            my $err = $@;
+            if ($err && !$interposed++) {
+                open(my $fh, '>>', PVE::QemuConfig->config_file_lock($vmid)) or die "$!\n";
+                $lock_free = flock($fh, LOCK_EX | LOCK_NB);
+                close($fh);
+                local $fail_at = '';
+                ($second_err) = restore_api('pbs', backup_config(), { force => 1 });
+            }
+            die $err if $err;
+            return $res;
+        },
+    );
+
+    # the temporary config of a VMA restore has no lock that would stop the second restore
+    $fail_at = 'data';
+    my ($err) = restore_api('vma file', backup_config(), {});
+    $QemuServerTest::FakeCFS::qemu_config_module->unmock('lock_config_full');
+
+    is($err, "restoring the data failed\n", 'first restore failed');
+    ok($lock_free, 'guest lock was free for the second restore');
+    is($second_err, '', 'second restore succeeded');
+    like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'its config stays');
+    is($calls->{firewall}, 1, 'its firewall config was restored');
+    ok(!$calls->{remove_vmfw_conf}, 'and stays');
+};
+
+for my $type ('pbs', 'provider', 'vma file', 'vma stdin') {
+    subtest "$type: failed restore over an existing VM" => sub {
+        QemuServerTest::FakeCFS::reset();
+        QemuServerTest::FakeCFS::set_config(100, $TARGET_CONF);
+
+        $fail_at = 'read-config';
+        my ($err, $output) = restore_api($type, backup_config(), { force => 1 });
+        like($err, qr/unable to read the backup\n$/, 'fails before changing the VM');
+        is($QemuServerTest::FakeCFS::configs->{100}, $TARGET_CONF, 'config unchanged');
+        is_deeply($calls, {}, 'replication state and firewall config untouched');
+        unlike($output, $NOT_CLEANED_UP, 'no warning about a changed VM');
+
+        $fail_at = 'data';
+        ($err, $output) = restore_api($type, backup_config(), { force => 1 });
+        is($err, "restoring the data failed\n", 'fails while restoring');
+        like($QemuServerTest::FakeCFS::configs->{100}, qr/^memory: 128$/m, 'temporary config');
+        unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed');
+        like($output, qr/error before or during data restore/, 'warning about the changed VM');
+    };
+}
+
+subtest 'failed live restore to a free VMID keeps the restored config' => sub {
+    QemuServerTest::FakeCFS::reset();
+
+    $fail_at = 'live';
+    my ($err, $output) = restore_api('pbs', backup_config(), { 'live-restore' => 1 });
+    is($err, "live restore failed\n", 'restore failed');
+    like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept');
+    unlike($QemuServerTest::FakeCFS::configs->{100}, qr/^lock:/m, 'lock removed');
+    like($output, $NOT_CLEANED_UP, 'warning about the VM left behind');
+};
+
+subtest 'failure after the data was restored keeps the VM' => sub {
+    QemuServerTest::FakeCFS::reset();
+
+    $fail_at = 'after-restore';
+    my ($err, $output) = restore_api('pbs', backup_config(), { unique => 1 });
+    is($err, "unable to register IPs\n", 'restore failed');
+    like($QemuServerTest::FakeCFS::configs->{100}, qr/^name: backup$/m, 'restored config kept');
+    like($output, qr/error after data was restored/, 'warning about the VM left behind');
+};
+
+$fail_at = '';
+
 done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-17 22:44 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 22:34 [PATCH cluster/guest-common/qemu-server/container/manager/docs v2 00/36] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 01/36] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 02/36] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 03/36] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 04/36] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 05/36] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 06/36] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH cluster v2 07/36] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 08/36] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 09/36] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 10/36] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-17 22:34 ` [PATCH guest-common v2 11/36] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-17 22:34 ` Thomas Lamprecht [this message]
2026-11-17 22:34 ` [PATCH qemu-server v2 13/36] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 14/36] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 15/36] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 16/36] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 17/36] api: status: return the guest uid Thomas Lamprecht
2026-11-17 22:34 ` [PATCH qemu-server v2 18/36] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 19/36] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 20/36] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 21/36] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 22/36] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 23/36] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 24/36] test: remote migration: check the cleanup of the target after failures Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 25/36] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-17 22:34 ` [PATCH container v2 26/36] api: status: return the guest uid Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 27/36] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 28/36] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 29/36] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 30/36] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 31/36] ui: guest summary: show " Thomas Lamprecht
2026-11-17 22:34 ` [PATCH manager v2 32/36] api: migrate all: refuse targets that would drop " Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 33/36] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 34/36] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 35/36] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-17 22:34 ` [PATCH docs v2 36/36] pct: " Thomas Lamprecht
2026-11-18  5:57 ` superseded: [PATCH cluster/guest-common/qemu-server/container/manager/docs v2 00/36] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117223945.862914-13-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal