* [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers
@ 2026-11-19 14:12 Michael Köppl
2026-11-19 14:12 ` [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles Michael Köppl
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Michael Köppl @ 2026-11-19 14:12 UTC (permalink / raw)
To: pve-devel; +Cc: l.sichert
Destroying a VM through the API with a disk on an LVM storage with
saferemove=1 leaves its VMID locked until the wipe of the disk is done.
The imgdel worker is forked while the qmdestroy worker holds the
config lock and inherits the locked file handle. The worker keeps the
lock held after the qmdestroy worker exited. Re-creating a VM with the
same VMID fails with a lock timeout in the meantime.
This series tries to keep the fix targeted, since closing all inherited
file descriptors in fork_worker() has a lot of regression
potential [0]. An opt-in helper is added in pve-common to close the
lock handles a forked child inherited from its ancestors. This call is
added to vdisk_free before running the cleanup workers of the
respective plugin. At the moment, this affects only the LVM plugin,
which only operates on volumes that are not referenced anymore and
takes the storage lock on its own where needed. Thus, this also
expects external plugins with cleanup workers to not rely on the
cleanup worker holding the caller's lock.
The last patch also closes the inherited lock handles in the cleanup
workers the LVM plugin forks itself for snapshot rollback and deletion
and for failed imports. It is not needed to fix the bug, but these
workers inherit their caller's locks too. For example, restoring a VM
over an existing one with qcow2 snapshots on LVM deletes the snapshots
of disks excluded from the backup under the config lock.
Since the cleanup workers no longer hold their caller's locks, a volume
can now be freed while an earlier volume with the same name is still
being wiped, e.g. when deleting a snapshot right after a rollback.
Renaming the volume to del-<name> would then fail, but only after the
guest was already locked for the operation. 2/4 avoids such name
collisions by falling back to del-<N>-<name>. This also fixes
destroying a VM right after a rollback, which already left the disk
behind before. Lukas S. and I discussed this for a bit off-list were
both wondering if this is a sensible approach and, in general, if these
naming collisions couldn't be avoided entirely by e.g. having
del-<N>-<name>-snapshot and del-<N>-<name> for different actions.
Tested on a single node by destroying a VM with a disk on an LVM
storage with saferemove through the API and re-creating it right after
the destroy task finished, while the wipe was still running. The last
patch was tested by restoring a backup through the API over an existing
VM with a qcow2 snapshot on an LVM storage with saferemove and a disk
excluded from the backup. The config of the VM could be changed while
the snapshot of the excluded disk was still being wiped. Both cleanup
workers finished successfully. The new logic for avoiding name
collisions was tested manually by rolling back a stopped VM to its only
snapshot and then deleting the snapshot, and by rolling back and then
destroying the VM, both while the volume replaced by the rollback was
still being wiped. In both cases, the freed volume was renamed to
del-1-<name> and all cleanup workers finished successfully.
Also passed the full set of E2E tests from proxmox-e2e-tests.
Dependencies: 3/4 and 4/4 call the new helper from pve-common, so
pve-storage needs a bumped dependency on libpve-common-perl.
Changes since v1 [1]:
* Add a patch that avoids name collisions with volumes pending
removal, instead of refusing a rollback while the volume replaced by
a previous rollback is still being wiped. Unlike the rollback check,
this also covers deleting a snapshot right after a rollback, and
deleting a snapshot with RAM again after re-creating it under the
same name, which the vdisk_free patch otherwise makes possible.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=8286
[1] https://lore.proxmox.com/pve-devel/20261105163637.1318072-1-m.koeppl@proxmox.com/
common:
Michael Köppl (1):
tools: lock file: add helper to close inherited lock handles
src/PVE/Tools.pm | 23 +++++++++++++++-----
test/lock_file.pl | 54 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+), 5 deletions(-)
storage:
Michael Köppl (3):
lvm plugin: avoid name collisions with volumes pending removal
fix #8286: vdisk free: do not hold inherited locks in cleanup worker
lvm plugin: do not hold inherited locks in cleanup workers
src/PVE/Storage.pm | 10 +-
src/PVE/Storage/LVMPlugin.pm | 71 +++++++----
src/test/Makefile | 5 +-
src/test/run_test_lvm_cleanup_name.pl | 165 ++++++++++++++++++++++++++
4 files changed, 228 insertions(+), 23 deletions(-)
create mode 100755 src/test/run_test_lvm_cleanup_name.pl
Summary over all repositories:
6 files changed, 300 insertions(+), 28 deletions(-)
--
Generated by murpp 0.12.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles 2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl @ 2026-11-19 14:12 ` Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 2/4] lvm plugin: avoid name collisions with volumes pending removal Michael Köppl ` (2 subsequent siblings) 3 siblings, 0 replies; 5+ messages in thread From: Michael Köppl @ 2026-11-19 14:12 UTC (permalink / raw) To: pve-devel; +Cc: l.sichert A child forked while its parent holds a lock from lock_file_full() inherits the locked file handle. The lock then is held until the child exits too, even after the parent released it. Keep the locked handle instead of a flag in the scalar that $lock_handles weakly references. Add a helper that lets a child close its copies of the handles locked by its ancestors. The ancestor keeps holding the lock, since its own handle stays open. Signed-off-by: Michael Köppl <m.koeppl@proxmox.com> --- src/PVE/Tools.pm | 23 +++++++++++++++----- test/lock_file.pl | 54 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 5 deletions(-) diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm index 9176b358..2f2123c1 100644 --- a/src/PVE/Tools.pm +++ b/src/PVE/Tools.pm @@ -198,7 +198,8 @@ sub run_with_timeout { # Since this is the only way we lock now and we don't have the old # 'lock(); code(); unlock();' pattern anymore we do not actually need to # count how deep we're nesting. Therefore this hash now stores a weak reference -# to a boolean telling us whether we already have a lock. +# to a scalar holding the locked file handle. If the lock has not been acquired +# yet, it holds a false value. my $lock_handles = {}; @@ -243,7 +244,7 @@ sub lock_file_full { if (!$checkptr || !$$checkptr) { # We cannot create a weak reference in a single atomic step, so we first # create a false-value, then create a reference to it, then weaken it, - # and after successfully locking the file we change the boolean value. + # and after successfully locking the file we store the handle in it. # # The reason for this is that if an outer SIGALRM throws an exception # between creating the reference and weakening it, a subsequent call to @@ -253,8 +254,8 @@ sub lock_file_full { # # An externally triggered exception elsewhere in the code will cause the # weak reference to become 'undef', and since the file handle is only - # stored in the local scope in $local_fh, the file will be closed by - # perl's cleanup routines as well. + # stored in the local scope in $local_fh and $check, the file will be + # closed by perl's cleanup routines as well. # # This still assumes that an IO::File handle can properly deal with such # exceptions thrown during its own destruction, but that's up to perls @@ -266,7 +267,7 @@ sub lock_file_full { $@ = "can't lock file '$filename' - $@"; return undef; } - $check = 1; + $check = $local_fh; } $res = eval { &$code(@param); }; return undef if $@; @@ -279,6 +280,18 @@ sub lock_file { return lock_file_full($filename, $timeout, 0, $code, @param); } +# Close this process's copies of lock handles inherited from its ancestors, so +# it does not keep their locks held after they released them. +sub close_inherited_lock_handles { + for my $pid (keys %$lock_handles) { + next if $pid == $$; + for my $checkptr (values $lock_handles->{$pid}->%*) { + close($$checkptr) if $checkptr && $$checkptr; + } + delete $lock_handles->{$pid}; + } +} + # file_ helper for backward compat, prefer PVE::File directly for new usage. # TODO: remove with PVE 10 / Debian Forky releases. sub file_set_contents { diff --git a/test/lock_file.pl b/test/lock_file.pl index 83965c55..ffa2404a 100755 --- a/test/lock_file.pl +++ b/test/lock_file.pl @@ -5,6 +5,7 @@ use strict; use warnings; use Socket; +use Fcntl qw(:flock); use POSIX (); # don't import assert() use PVE::Tools 'lock_file_full'; @@ -91,6 +92,54 @@ sub forktest1($) { close($fmain); } +sub lock_is_free { + open(my $fh, '>>', $name) or die "open(): $!\n"; + return flock($fh, LOCK_EX | LOCK_NB); +} + +sub inherited_lock_test { + my ($close_in_child) = @_; + + socketpair(my $fparent, my $fchild, AF_UNIX, SOCK_STREAM, PF_UNSPEC) + or die "socketpair(): $!\n"; + + my $pid; + lock_file_full( + $name, + 10, + 0, + sub { + $pid = fork() // die "fork(): $!\n"; + if (!$pid) { + close($fparent); + PVE::Tools::close_inherited_lock_handles() if $close_in_child; + print {$fchild} "1\n"; + $fchild->flush(); + <$fchild>; # wait for parent to finish checks + POSIX::_exit(0); + } + close($fchild); + my $line = <$fparent>; + die "child failed to start\n" if !$line || $line ne "1\n"; + die "lock released while the parent still holds it\n" if lock_is_free(); + }, + ); + + my $err = $@; + my $free = !$err && lock_is_free(); + print {$fparent} "2\n"; + $fparent->flush(); + waitpid($pid, 0) if $pid; + close($fparent); + die $err if $err; + + if ($close_in_child) { + die "child kept the inherited lock held\n" if !$free; + } else { + die "child did not inherit the lock\n" if $free; + } +} + eval { # Regular lock: new(); @@ -143,6 +192,11 @@ eval { # Can we get two simultaneous shared locks? (shared=1) forktest1(0); forktest1(1); + + # Lock held by child after parent released it? + inherited_lock_test(0); + # Lock free once child closed inherited copies? + inherited_lock_test(1); }; my $err = $@; system("rm $name*"); -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH storage v2 2/4] lvm plugin: avoid name collisions with volumes pending removal 2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl 2026-11-19 14:12 ` [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles Michael Köppl @ 2026-11-19 14:13 ` Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 3/4] fix #8286: vdisk free: do not hold inherited locks in cleanup worker Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 4/4] lvm plugin: do not hold inherited locks in cleanup workers Michael Köppl 3 siblings, 0 replies; 5+ messages in thread From: Michael Köppl @ 2026-11-19 14:13 UTC (permalink / raw) To: pve-devel; +Cc: l.sichert With saferemove or discard on removal, a freed volume is renamed to del-<name> and wiped by a worker in the background. If a volume with the same name is freed again before that worker has finished, the rename fails because del-<name> still exists (e.g. when destroying a VM right after rolling back a qcow2 snapshot). If del-<name> is taken, rename the volume to the first free del-<N>-<name> instead. The name is chosen with the storage lock held, so it cannot race with other removals. If del-<name> is free, it is used as before. The numbered names still contain the volume name, so they keep its disk number reserved for new allocations, like del-<name>. The cleanup worker and the recovery of a failed rollback now use that name. Signed-off-by: Michael Köppl <m.koeppl@proxmox.com> --- Notes: Changing this might be too much regression potential for the fix in 3/4. This is based on the feedback of the automated LLM review pointing out that snapshot delection could collide with an outstanding rollback cleanup in v1 of this series, leaving behind a locked guest. So the choice is either only applying the pve-common patch and the actual fix for 8268 and dropping the rest, or avoiding the naming overlap as done here. There might be other ways to do this, but I wanted to keep the potential for regressions as small as possible. src/PVE/Storage/LVMPlugin.pm | 61 ++++++---- src/test/Makefile | 5 +- src/test/run_test_lvm_cleanup_name.pl | 165 ++++++++++++++++++++++++++ 3 files changed, 210 insertions(+), 21 deletions(-) create mode 100755 src/test/run_test_lvm_cleanup_name.pl diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index 3cdab9e4..e12145bb 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -465,7 +465,7 @@ my sub lock_storage_with_acquire_retry { } my sub rename_after_failed_cleanup { - my ($class, $scfg, $storeid, $vg, $name) = @_; + my ($class, $scfg, $storeid, $vg, $del_name) = @_; eval { my $failed_name; @@ -482,7 +482,7 @@ my sub rename_after_failed_cleanup { my $existing = $lvs->{$vg} // {}; my $prefix = 'failed-'; - my $suffix = "-del-$name"; + my $suffix = "-$del_name"; my $last_fail = max( -1, @@ -493,17 +493,17 @@ my sub rename_after_failed_cleanup { $failed_name = $prefix . ($last_fail + 1) . $suffix; - my $cmd = ['/sbin/lvrename', $vg, "del-$name", $failed_name]; + my $cmd = ['/sbin/lvrename', $vg, $del_name, $failed_name]; run_command( $cmd, - errmsg => "lvrename '$vg/del-$name' to '$vg/$failed_name' error", + errmsg => "lvrename '$vg/$del_name' to '$vg/$failed_name' error", ); - print "renamed '$vg/del-$name' to '$vg/$failed_name'\n"; + print "renamed '$vg/$del_name' to '$vg/$failed_name'\n"; }, ); }; if (my $rename_err = $@) { - print STDERR "ERROR: unable to rename '$vg/del-$name': $rename_err"; + print STDERR "ERROR: unable to rename '$vg/$del_name': $rename_err"; } } @@ -526,10 +526,23 @@ my sub parse_saferemove_throughput { return $number * $multiplier; } +# Freeing a volume again while an earlier one with the same name is still being cleaned up, e.g. +# after two rollbacks in a row, must not collide with the earlier one's name. +my sub cleanup_name { + my ($existing, $name) = @_; + + return "del-$name" if !$existing->{"del-$name"}; + + my $n = 1; + $n++ while $existing->{"del-$n-$name"}; + return "del-$n-$name"; +} + my sub free_lvm_volumes_locked { - my ($class, $scfg, $storeid, $volnames) = @_; + my ($class, $scfg, $storeid, $volnames, $cleanup_names) = @_; my $vg = $scfg->{vgname}; + my $del_names = {}; my $on_remove_opts = {}; if ($scfg->{'on-volume-remove'}) { @@ -759,7 +772,8 @@ my sub free_lvm_volumes_locked { my $total_cleanup_errors = 0; for my $name (@$volnames) { - my $lvmpath = "/dev/$vg/del-$name"; + my $del_name = $del_names->{$name}; + my $lvmpath = "/dev/$vg/$del_name"; my $discard_action; if ($scfg->{saferemove} && $on_remove_opts->{discard}) { @@ -790,7 +804,7 @@ my sub free_lvm_volumes_locked { }; if (my $activation_err = $@) { print STDERR "ERROR: $activation_err"; - eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $name) }; + eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $del_name) }; $total_cleanup_errors += 1; next; } @@ -802,18 +816,18 @@ my sub free_lvm_volumes_locked { $storeid, $scfg, sub { - my $cmd = ['/sbin/lvremove', '-f', "$vg/del-$name"]; - run_command($cmd, errmsg => "lvremove '$vg/del-$name' error"); + my $cmd = ['/sbin/lvremove', '-f', "$vg/$del_name"]; + run_command($cmd, errmsg => "lvremove '$vg/$del_name' error"); }, ); }; if (my $cleanup_err = $@) { print STDERR "ERROR: cleanup failed for lv $name: $cleanup_err"; - eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $name) }; + eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $del_name) }; $total_cleanup_errors += 1; next; } - print "successfully removed volume $name ($vg/del-$name)\n"; + print "successfully removed volume $name ($vg/$del_name)\n"; } if ($total_cleanup_errors != 0) { my $number_of_vols = scalar @$volnames; @@ -829,11 +843,15 @@ my sub free_lvm_volumes_locked { my $cmd = ['/sbin/lvchange', '-aln', "$vg/$name"]; run_command($cmd, errmsg => "can't deactivate LV '$vg/$name'"); } + my $existing = lvm_list_volumes($vg)->{$vg} // {}; for my $name (@$volnames) { # avoid long running task, so we only rename here - my $cmd = ['/sbin/lvrename', $vg, $name, "del-$name"]; + my $del_name = cleanup_name($existing, $name); + my $cmd = ['/sbin/lvrename', $vg, $name, $del_name]; run_command($cmd, errmsg => "lvrename '$vg/$name' error"); + $del_names->{$name} = $del_name; } + $cleanup_names->%* = $del_names->%* if $cleanup_names; return $cleanup_worker; } else { for my $name (@$volnames) { @@ -1450,10 +1468,10 @@ my sub alloc_snap_image { } my sub free_snap_image_locked { - my ($class, $storeid, $scfg, $volname, $snap) = @_; + my ($class, $storeid, $scfg, $volname, $snap, $cleanup_names) = @_; my $snap_volname = get_snap_name($class, $volname, $snap); - return free_lvm_volumes_locked($class, $scfg, $storeid, [$snap_volname]); + return free_lvm_volumes_locked($class, $scfg, $storeid, [$snap_volname], $cleanup_names); } # Must be called with the storage lock held. @@ -1858,19 +1876,22 @@ my sub volume_snapshot_rollback_locked { die "can't rollback snapshot for '$format' volume\n" if $format ne 'qcow2'; - my $cleanup_worker = - eval { free_snap_image_locked($class, $storeid, $scfg, $volname, 'current'); }; + my $cleanup_names = {}; + my $cleanup_worker = eval { + free_snap_image_locked($class, $storeid, $scfg, $volname, 'current', $cleanup_names); + }; die "error deleting snapshot $snap $@\n" if $@; eval { alloc_snap_image($class, $storeid, $scfg, $volname, $snap) }; if (my $err = $@) { my $original_state = ''; if ($cleanup_worker) { # rename original image back - eval { lvrename($scfg, "del-${volname}", $volname) }; + my $del_name = $cleanup_names->{$volname}; + eval { lvrename($scfg, $del_name, $volname) }; if ($@) { warn $@; # no cleanup worker is started, so the original stays available under that name - $original_state = " (original volume kept as 'del-${volname}')"; + $original_state = " (original volume kept as '$del_name')"; } else { $original_state = ' (original volume restored)'; } diff --git a/src/test/Makefile b/src/test/Makefile index b2676762..10f0d9b5 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -1,6 +1,6 @@ all: test -test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream +test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_cleanup_name test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_zfspoolplugin: run_test_zfspoolplugin.pl ./run_test_zfspoolplugin.pl @@ -16,6 +16,9 @@ test_lvm_create: run_test_lvm_create.pl run_test_lvm_base_conflicts.pl test_lvm_lock_retry: run_test_lvm_lock_retry.pl ./run_test_lvm_lock_retry.pl +test_lvm_cleanup_name: run_test_lvm_cleanup_name.pl + ./run_test_lvm_cleanup_name.pl + test_lvm_base_volume: run_test_lvm_base_volume.pl ./run_test_lvm_base_volume.pl diff --git a/src/test/run_test_lvm_cleanup_name.pl b/src/test/run_test_lvm_cleanup_name.pl new file mode 100755 index 00000000..964de004 --- /dev/null +++ b/src/test/run_test_lvm_cleanup_name.pl @@ -0,0 +1,165 @@ +#!/usr/bin/perl + +# Unit test for freeing a volume while an earlier one with the same name is still pending removal. + +use strict; +use warnings; + +use lib '..'; +use Test::More; +use Test::MockModule; + +use PVE::Storage::LVMPlugin; + +my $CLASS = 'PVE::Storage::LVMPlugin'; +my $vg = 'test-vg'; +my $scfg = { type => 'lvm', vgname => $vg, saferemove => 1 }; +my $volname = 'vm-100-disk-0.qcow2'; +my $snapvolname = 'snap_vm-100-disk-0_snap1.qcow2'; + +my $lvs; # names of the existing LVs +my @renames; +my @activated; +my @removed; +my $workers; + +my $lvm_mock = Test::MockModule->new('PVE::Storage::LVMPlugin'); +$lvm_mock->redefine( + cluster_lock_storage => sub { + my ($class, $storeid, $shared, $timeout, $func, @param) = @_; + return $func->(@param); + }, + lvm_vgs => sub { + return { $vg => {} }; + }, + lvm_list_volumes => sub { + return { $vg => { map { $_ => {} } keys %$lvs } }; + }, + run_command => sub { + my ($cmd) = @_; + if ($cmd->[0] eq '/sbin/lvrename') { + my (undef, undef, $from, $to) = @$cmd; + die "Logical Volume \"$from\" not found in volume group \"$vg\"\n" + if !$lvs->{$from}; + die "Logical Volume \"$to\" already exists in volume group \"$vg\"\n" + if $lvs->{$to}; + push @renames, "$from -> $to"; + delete $lvs->{$from}; + $lvs->{$to} = 1; + } elsif ($cmd->[0] eq '/sbin/lvremove') { + my $name = $cmd->[-1] =~ s|^\Q$vg\E/||r; + die "Failed to find logical volume \"$vg/$name\"\n" if !$lvs->{$name}; + push @removed, $name; + delete $lvs->{$name}; + } elsif ($cmd->[0] eq '/sbin/lvchange' && $cmd->[1] eq '-aly') { + push @activated, $cmd->[2]; + } + return 0; + }, + activate_volume => sub { }, + # no volume has a device mapper link, so the cleanup worker cannot zero out + abs_path => sub { return undef }, + volume_snapshot_info => sub { + return { + snap1 => { file => '/dev/null', volname => $snapvolname, child => 'current' }, + current => { file => '/dev/null', volname => $volname, parent => 'snap1' }, + }; + }, +); + +# makes allocating the new current volume of a rollback fail +my $plugin_mock = Test::MockModule->new('PVE::Storage::Plugin'); +$plugin_mock->redefine(file_size_info => sub { return undef }); + +my $rpcenv_mock = Test::MockModule->new('PVE::RPCEnvironment'); +$rpcenv_mock->redefine( + get => sub { return bless({}, 'PVE::RPCEnvironment') }, + get_user => sub { return 'root@pam' }, + fork_worker => sub { $workers++; return 'UPID:test' }, +); + +my sub setup { + my (@names) = @_; + $lvs = { map { $_ => 1 } @names }; + @renames = (); + @activated = (); + @removed = (); + $workers = 0; +} + +# keep plugin progress and errors out of the test output +my sub quietly { + my ($code) = @_; + local *STDOUT; + local *STDERR; + open(STDOUT, '>', '/dev/null') or die "cannot open /dev/null - $!\n"; + open(STDERR, '>', '/dev/null') or die "cannot open /dev/null - $!\n"; + my $ok = eval { $code->(); 1 }; + my $err = $@; + return ($ok, $err); +} + +# the previous current volume replaced by a rollback is still pending removal +setup($volname, $snapvolname, "del-$volname"); +my ($ok, $err) = + quietly(sub { $CLASS->volume_snapshot_delete($scfg, 'mystore', $volname, 'snap1', 0) }); +ok($ok, 'deleting the snapshot right after a rollback succeeds') or diag($err); +is_deeply( + \@renames, + ["$volname -> del-1-$volname", "$snapvolname -> $volname"], + 'frees the current volume under a numbered name', +); +ok($lvs->{"del-$volname"}, 'volume replaced by the rollback is left alone'); +is($workers, 1, 'started a cleanup worker'); + +setup($volname, $snapvolname, "del-$volname"); +($ok, $err) = + quietly(sub { $CLASS->volume_snapshot_rollback($scfg, 'mystore', $volname, 'snap1') }); +ok(!$ok, 'second rollback fails when the new volume cannot be allocated'); +like($err, qr/original volume restored/, 'reports the original volume as restored'); +is_deeply( + \@renames, + ["$volname -> del-1-$volname", "del-1-$volname -> $volname"], + 'restores the volume it renamed, not the one of the earlier rollback', +); +ok($lvs->{"del-$volname"}, 'volume replaced by the earlier rollback is left alone'); +is($workers, 0, 'started no cleanup worker'); + +my $rawvolname = 'vm-100-disk-1'; + +# only a numbered volume is still pending removal, so del-<name> is free +setup($rawvolname, "del-1-$rawvolname"); +my $discard_scfg = { type => 'lvm', vgname => $vg, 'on-volume-remove' => 'discard=1' }; +my $worker = $CLASS->free_image('mystore', $discard_scfg, $rawvolname, 0, 'raw'); +is_deeply(\@renames, ["$rawvolname -> del-$rawvolname"], 'renamed to del-<name> if that is free'); +($ok, $err) = quietly($worker); +ok($ok, 'cleanup worker succeeds') or diag($err); +is_deeply(\@activated, ["/dev/$vg/del-$rawvolname"], 'cleanup worker activates its own volume'); +is_deeply(\@removed, ["del-$rawvolname"], 'cleanup worker removes its own volume'); +ok($lvs->{"del-1-$rawvolname"}, 'volume pending removal is left alone'); + +# zeroing out fails without a device mapper link, so the cleanup worker renames its volume +setup($rawvolname, "del-$rawvolname", "del-1-$rawvolname"); +$worker = $CLASS->free_image('mystore', $scfg, $rawvolname, 0, 'raw'); +is_deeply( + \@renames, + ["$rawvolname -> del-2-$rawvolname"], + 'renamed to the first free numbered name', +); +@renames = (); +($ok, $err) = quietly($worker); +ok(!$ok, 'cleanup worker fails'); +like($err, qr/cleanup failed for 1 out of 1 volumes/, 'cleanup worker reports the failed cleanup'); +is_deeply(\@activated, ["/dev/$vg/del-2-$rawvolname"], 'cleanup worker activates its own volume'); +is_deeply( + \@renames, + ["del-2-$rawvolname -> failed-0-del-2-$rawvolname"], + 'failed cleanup renames its own volume', +); +is_deeply(\@removed, [], 'failed cleanup removes no volume'); +ok( + $lvs->{"del-$rawvolname"} && $lvs->{"del-1-$rawvolname"}, + 'volumes pending removal are left alone', +); + +done_testing(); -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH storage v2 3/4] fix #8286: vdisk free: do not hold inherited locks in cleanup worker 2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl 2026-11-19 14:12 ` [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 2/4] lvm plugin: avoid name collisions with volumes pending removal Michael Köppl @ 2026-11-19 14:13 ` Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 4/4] lvm plugin: do not hold inherited locks in cleanup workers Michael Köppl 3 siblings, 0 replies; 5+ messages in thread From: Michael Köppl @ 2026-11-19 14:13 UTC (permalink / raw) To: pve-devel; +Cc: l.sichert vdisk_free() forks the cleanup worker of a volume (e.g. for wiping LVs on LVM storages with saferemove), while the caller often still holds the guest config lock. The worker inherits the locked handle and keeps the lock held until the cleanup is done. For example, after destroying a VM through the API, re-creating a VM with the same VMID fails with a lock timeout until the wipe has finished. The cleanup workers in the LVM plugin only operate on volumes that are not referenced anymore. The plugin's implementation takes the lock on its own where needed, so close the inherited lock handles when the worker starts. This applies to the cleanup worker of any plugin, which must not rely on the cleanup worker holding the caller's lock. Signed-off-by: Michael Köppl <m.koeppl@proxmox.com> --- Notes: I'm not quite sure about this one. I understand that it breaks this behavior for existing external plugins, but I'm not sure it's something they should rely on in the first place. An alternative to this would be to put the cleanup call into the LVM plugin's cleanup worker specifically, which would also require every plugin to implement this in its cleanup worker. Could perhaps also be a sensible approach for now to have the call in the LVM plugin specifically and move it to vdisk_free for PVE 10. I mostly decided on this version for now because of Fabian's comment on the Bugzilla ticket. src/PVE/Storage.pm | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index faf325fb..854f8e8d 100755 --- a/src/PVE/Storage.pm +++ b/src/PVE/Storage.pm @@ -1303,7 +1303,15 @@ sub vdisk_free { my $rpcenv = PVE::RPCEnvironment::get(); my $authuser = $rpcenv->get_user(); - $rpcenv->fork_worker('imgdel', undef, $authuser, $cleanup_worker); + $rpcenv->fork_worker( + 'imgdel', + undef, + $authuser, + sub { + PVE::Tools::close_inherited_lock_handles(); + $cleanup_worker->(@_); + }, + ); } sub vdisk_list { -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH storage v2 4/4] lvm plugin: do not hold inherited locks in cleanup workers 2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl ` (2 preceding siblings ...) 2026-11-19 14:13 ` [PATCH storage v2 3/4] fix #8286: vdisk free: do not hold inherited locks in cleanup worker Michael Köppl @ 2026-11-19 14:13 ` Michael Köppl 3 siblings, 0 replies; 5+ messages in thread From: Michael Köppl @ 2026-11-19 14:13 UTC (permalink / raw) To: pve-devel; +Cc: l.sichert The LVM plugin forks its own cleanup workers when rolling back or deleting snapshots of qcow2 volumes and after failed imports. Like the worker forked by vdisk_free(), they inherit the locks of their caller and keep them held until the cleanup is done. This can happen, e.g., when restoring a backup over an existing VM with a qcow2 snapshot on LVM and a disk excluded from the backup, which deletes that disk's snapshot while holding the config lock. After the restore has finished, changing or starting the VM then fails with a lock timeout until the cleanup is done. Close the inherited lock handles at the start of these workers too. The cleanup worker of a rollback or snapshot deletion no longer keeps the guest migration lock, so another rollback, snapshot deletion, or migration can run while the previous volume is still being wiped. This is safe, since the volumes pending removal no longer have colliding names. Signed-off-by: Michael Köppl <m.koeppl@proxmox.com> --- src/PVE/Storage/LVMPlugin.pm | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index e12145bb..365dc21d 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -52,7 +52,15 @@ my sub fork_cleanup_worker { return if !$cleanup_worker; my $rpcenv = PVE::RPCEnvironment::get(); my $authuser = $rpcenv->get_user(); - $rpcenv->fork_worker('imgdel', undef, $authuser, $cleanup_worker); + $rpcenv->fork_worker( + 'imgdel', + undef, + $authuser, + sub { + PVE::Tools::close_inherited_lock_handles(); + $cleanup_worker->(@_); + }, + ); } sub lvm_pv_info { -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-19 14:13 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl 2026-11-19 14:12 ` [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 2/4] lvm plugin: avoid name collisions with volumes pending removal Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 3/4] fix #8286: vdisk free: do not hold inherited locks in cleanup worker Michael Köppl 2026-11-19 14:13 ` [PATCH storage v2 4/4] lvm plugin: do not hold inherited locks in cleanup workers Michael Köppl
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox