all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "Michael Köppl" <m.koeppl@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: l.sichert@proxmox.com
Subject: [PATCH storage v2 2/4] lvm plugin: avoid name collisions with volumes pending removal
Date: Thu, 19 Nov 2026 15:13:00 +0100	[thread overview]
Message-ID: <20261119141302.600137-3-m.koeppl@proxmox.com> (raw)
In-Reply-To: <20261119141302.600137-1-m.koeppl@proxmox.com>

With saferemove or discard on removal, a freed volume is renamed to
del-<name> and wiped by a worker in the background. If a volume with
the same name is freed again before that worker has finished, the
rename fails because del-<name> still exists (e.g. when destroying a VM
right after rolling back a qcow2 snapshot).

If del-<name> is taken, rename the volume to the first free
del-<N>-<name> instead. The name is chosen with the storage lock held,
so it cannot race with other removals. If del-<name> is free, it is used
as before. The numbered names still contain the volume name, so they
keep its disk number reserved for new allocations, like del-<name>.

The cleanup worker and the recovery of a failed rollback now use that
name.

Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---

Notes:
    Changing this might be too much regression potential for the fix in 3/4.
    This is based on the feedback of the automated LLM review pointing out
    that snapshot delection could collide with an outstanding rollback
    cleanup in v1 of this series, leaving behind a locked guest. So the
    choice is either only applying the pve-common patch and the actual fix
    for 8268 and dropping the rest, or avoiding the naming overlap as done
    here. There might be other ways to do this, but I wanted to keep the
    potential for regressions as small as possible.

 src/PVE/Storage/LVMPlugin.pm          |  61 ++++++----
 src/test/Makefile                     |   5 +-
 src/test/run_test_lvm_cleanup_name.pl | 165 ++++++++++++++++++++++++++
 3 files changed, 210 insertions(+), 21 deletions(-)
 create mode 100755 src/test/run_test_lvm_cleanup_name.pl

diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 3cdab9e4..e12145bb 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -465,7 +465,7 @@ my sub lock_storage_with_acquire_retry {
 }
 
 my sub rename_after_failed_cleanup {
-    my ($class, $scfg, $storeid, $vg, $name) = @_;
+    my ($class, $scfg, $storeid, $vg, $del_name) = @_;
 
     eval {
         my $failed_name;
@@ -482,7 +482,7 @@ my sub rename_after_failed_cleanup {
                 my $existing = $lvs->{$vg} // {};
 
                 my $prefix = 'failed-';
-                my $suffix = "-del-$name";
+                my $suffix = "-$del_name";
 
                 my $last_fail = max(
                     -1,
@@ -493,17 +493,17 @@ my sub rename_after_failed_cleanup {
 
                 $failed_name = $prefix . ($last_fail + 1) . $suffix;
 
-                my $cmd = ['/sbin/lvrename', $vg, "del-$name", $failed_name];
+                my $cmd = ['/sbin/lvrename', $vg, $del_name, $failed_name];
                 run_command(
                     $cmd,
-                    errmsg => "lvrename '$vg/del-$name' to '$vg/$failed_name' error",
+                    errmsg => "lvrename '$vg/$del_name' to '$vg/$failed_name' error",
                 );
-                print "renamed '$vg/del-$name' to '$vg/$failed_name'\n";
+                print "renamed '$vg/$del_name' to '$vg/$failed_name'\n";
             },
         );
     };
     if (my $rename_err = $@) {
-        print STDERR "ERROR: unable to rename '$vg/del-$name': $rename_err";
+        print STDERR "ERROR: unable to rename '$vg/$del_name': $rename_err";
     }
 }
 
@@ -526,10 +526,23 @@ my sub parse_saferemove_throughput {
     return $number * $multiplier;
 }
 
+# Freeing a volume again while an earlier one with the same name is still being cleaned up, e.g.
+# after two rollbacks in a row, must not collide with the earlier one's name.
+my sub cleanup_name {
+    my ($existing, $name) = @_;
+
+    return "del-$name" if !$existing->{"del-$name"};
+
+    my $n = 1;
+    $n++ while $existing->{"del-$n-$name"};
+    return "del-$n-$name";
+}
+
 my sub free_lvm_volumes_locked {
-    my ($class, $scfg, $storeid, $volnames) = @_;
+    my ($class, $scfg, $storeid, $volnames, $cleanup_names) = @_;
 
     my $vg = $scfg->{vgname};
+    my $del_names = {};
 
     my $on_remove_opts = {};
     if ($scfg->{'on-volume-remove'}) {
@@ -759,7 +772,8 @@ my sub free_lvm_volumes_locked {
 
         my $total_cleanup_errors = 0;
         for my $name (@$volnames) {
-            my $lvmpath = "/dev/$vg/del-$name";
+            my $del_name = $del_names->{$name};
+            my $lvmpath = "/dev/$vg/$del_name";
 
             my $discard_action;
             if ($scfg->{saferemove} && $on_remove_opts->{discard}) {
@@ -790,7 +804,7 @@ my sub free_lvm_volumes_locked {
             };
             if (my $activation_err = $@) {
                 print STDERR "ERROR: $activation_err";
-                eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $name) };
+                eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $del_name) };
                 $total_cleanup_errors += 1;
                 next;
             }
@@ -802,18 +816,18 @@ my sub free_lvm_volumes_locked {
                     $storeid,
                     $scfg,
                     sub {
-                        my $cmd = ['/sbin/lvremove', '-f', "$vg/del-$name"];
-                        run_command($cmd, errmsg => "lvremove '$vg/del-$name' error");
+                        my $cmd = ['/sbin/lvremove', '-f', "$vg/$del_name"];
+                        run_command($cmd, errmsg => "lvremove '$vg/$del_name' error");
                     },
                 );
             };
             if (my $cleanup_err = $@) {
                 print STDERR "ERROR: cleanup failed for lv $name: $cleanup_err";
-                eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $name) };
+                eval { rename_after_failed_cleanup($class, $scfg, $storeid, $vg, $del_name) };
                 $total_cleanup_errors += 1;
                 next;
             }
-            print "successfully removed volume $name ($vg/del-$name)\n";
+            print "successfully removed volume $name ($vg/$del_name)\n";
         }
         if ($total_cleanup_errors != 0) {
             my $number_of_vols = scalar @$volnames;
@@ -829,11 +843,15 @@ my sub free_lvm_volumes_locked {
             my $cmd = ['/sbin/lvchange', '-aln', "$vg/$name"];
             run_command($cmd, errmsg => "can't deactivate LV '$vg/$name'");
         }
+        my $existing = lvm_list_volumes($vg)->{$vg} // {};
         for my $name (@$volnames) {
             # avoid long running task, so we only rename here
-            my $cmd = ['/sbin/lvrename', $vg, $name, "del-$name"];
+            my $del_name = cleanup_name($existing, $name);
+            my $cmd = ['/sbin/lvrename', $vg, $name, $del_name];
             run_command($cmd, errmsg => "lvrename '$vg/$name' error");
+            $del_names->{$name} = $del_name;
         }
+        $cleanup_names->%* = $del_names->%* if $cleanup_names;
         return $cleanup_worker;
     } else {
         for my $name (@$volnames) {
@@ -1450,10 +1468,10 @@ my sub alloc_snap_image {
 }
 
 my sub free_snap_image_locked {
-    my ($class, $storeid, $scfg, $volname, $snap) = @_;
+    my ($class, $storeid, $scfg, $volname, $snap, $cleanup_names) = @_;
 
     my $snap_volname = get_snap_name($class, $volname, $snap);
-    return free_lvm_volumes_locked($class, $scfg, $storeid, [$snap_volname]);
+    return free_lvm_volumes_locked($class, $scfg, $storeid, [$snap_volname], $cleanup_names);
 }
 
 # Must be called with the storage lock held.
@@ -1858,19 +1876,22 @@ my sub volume_snapshot_rollback_locked {
 
     die "can't rollback snapshot for '$format' volume\n" if $format ne 'qcow2';
 
-    my $cleanup_worker =
-        eval { free_snap_image_locked($class, $storeid, $scfg, $volname, 'current'); };
+    my $cleanup_names = {};
+    my $cleanup_worker = eval {
+        free_snap_image_locked($class, $storeid, $scfg, $volname, 'current', $cleanup_names);
+    };
     die "error deleting snapshot $snap $@\n" if $@;
 
     eval { alloc_snap_image($class, $storeid, $scfg, $volname, $snap) };
     if (my $err = $@) {
         my $original_state = '';
         if ($cleanup_worker) { # rename original image back
-            eval { lvrename($scfg, "del-${volname}", $volname) };
+            my $del_name = $cleanup_names->{$volname};
+            eval { lvrename($scfg, $del_name, $volname) };
             if ($@) {
                 warn $@;
                 # no cleanup worker is started, so the original stays available under that name
-                $original_state = " (original volume kept as 'del-${volname}')";
+                $original_state = " (original volume kept as '$del_name')";
             } else {
                 $original_state = ' (original volume restored)';
             }
diff --git a/src/test/Makefile b/src/test/Makefile
index b2676762..10f0d9b5 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
 all: test
 
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_cleanup_name test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream
 
 test_zfspoolplugin: run_test_zfspoolplugin.pl
 	./run_test_zfspoolplugin.pl
@@ -16,6 +16,9 @@ test_lvm_create: run_test_lvm_create.pl run_test_lvm_base_conflicts.pl
 test_lvm_lock_retry: run_test_lvm_lock_retry.pl
 	./run_test_lvm_lock_retry.pl
 
+test_lvm_cleanup_name: run_test_lvm_cleanup_name.pl
+	./run_test_lvm_cleanup_name.pl
+
 test_lvm_base_volume: run_test_lvm_base_volume.pl
 	./run_test_lvm_base_volume.pl
 
diff --git a/src/test/run_test_lvm_cleanup_name.pl b/src/test/run_test_lvm_cleanup_name.pl
new file mode 100755
index 00000000..964de004
--- /dev/null
+++ b/src/test/run_test_lvm_cleanup_name.pl
@@ -0,0 +1,165 @@
+#!/usr/bin/perl
+
+# Unit test for freeing a volume while an earlier one with the same name is still pending removal.
+
+use strict;
+use warnings;
+
+use lib '..';
+use Test::More;
+use Test::MockModule;
+
+use PVE::Storage::LVMPlugin;
+
+my $CLASS = 'PVE::Storage::LVMPlugin';
+my $vg = 'test-vg';
+my $scfg = { type => 'lvm', vgname => $vg, saferemove => 1 };
+my $volname = 'vm-100-disk-0.qcow2';
+my $snapvolname = 'snap_vm-100-disk-0_snap1.qcow2';
+
+my $lvs; # names of the existing LVs
+my @renames;
+my @activated;
+my @removed;
+my $workers;
+
+my $lvm_mock = Test::MockModule->new('PVE::Storage::LVMPlugin');
+$lvm_mock->redefine(
+    cluster_lock_storage => sub {
+        my ($class, $storeid, $shared, $timeout, $func, @param) = @_;
+        return $func->(@param);
+    },
+    lvm_vgs => sub {
+        return { $vg => {} };
+    },
+    lvm_list_volumes => sub {
+        return { $vg => { map { $_ => {} } keys %$lvs } };
+    },
+    run_command => sub {
+        my ($cmd) = @_;
+        if ($cmd->[0] eq '/sbin/lvrename') {
+            my (undef, undef, $from, $to) = @$cmd;
+            die "Logical Volume \"$from\" not found in volume group \"$vg\"\n"
+                if !$lvs->{$from};
+            die "Logical Volume \"$to\" already exists in volume group \"$vg\"\n"
+                if $lvs->{$to};
+            push @renames, "$from -> $to";
+            delete $lvs->{$from};
+            $lvs->{$to} = 1;
+        } elsif ($cmd->[0] eq '/sbin/lvremove') {
+            my $name = $cmd->[-1] =~ s|^\Q$vg\E/||r;
+            die "Failed to find logical volume \"$vg/$name\"\n" if !$lvs->{$name};
+            push @removed, $name;
+            delete $lvs->{$name};
+        } elsif ($cmd->[0] eq '/sbin/lvchange' && $cmd->[1] eq '-aly') {
+            push @activated, $cmd->[2];
+        }
+        return 0;
+    },
+    activate_volume => sub { },
+    # no volume has a device mapper link, so the cleanup worker cannot zero out
+    abs_path => sub { return undef },
+    volume_snapshot_info => sub {
+        return {
+            snap1 => { file => '/dev/null', volname => $snapvolname, child => 'current' },
+            current => { file => '/dev/null', volname => $volname, parent => 'snap1' },
+        };
+    },
+);
+
+# makes allocating the new current volume of a rollback fail
+my $plugin_mock = Test::MockModule->new('PVE::Storage::Plugin');
+$plugin_mock->redefine(file_size_info => sub { return undef });
+
+my $rpcenv_mock = Test::MockModule->new('PVE::RPCEnvironment');
+$rpcenv_mock->redefine(
+    get => sub { return bless({}, 'PVE::RPCEnvironment') },
+    get_user => sub { return 'root@pam' },
+    fork_worker => sub { $workers++; return 'UPID:test' },
+);
+
+my sub setup {
+    my (@names) = @_;
+    $lvs = { map { $_ => 1 } @names };
+    @renames = ();
+    @activated = ();
+    @removed = ();
+    $workers = 0;
+}
+
+# keep plugin progress and errors out of the test output
+my sub quietly {
+    my ($code) = @_;
+    local *STDOUT;
+    local *STDERR;
+    open(STDOUT, '>', '/dev/null') or die "cannot open /dev/null - $!\n";
+    open(STDERR, '>', '/dev/null') or die "cannot open /dev/null - $!\n";
+    my $ok = eval { $code->(); 1 };
+    my $err = $@;
+    return ($ok, $err);
+}
+
+# the previous current volume replaced by a rollback is still pending removal
+setup($volname, $snapvolname, "del-$volname");
+my ($ok, $err) =
+    quietly(sub { $CLASS->volume_snapshot_delete($scfg, 'mystore', $volname, 'snap1', 0) });
+ok($ok, 'deleting the snapshot right after a rollback succeeds') or diag($err);
+is_deeply(
+    \@renames,
+    ["$volname -> del-1-$volname", "$snapvolname -> $volname"],
+    'frees the current volume under a numbered name',
+);
+ok($lvs->{"del-$volname"}, 'volume replaced by the rollback is left alone');
+is($workers, 1, 'started a cleanup worker');
+
+setup($volname, $snapvolname, "del-$volname");
+($ok, $err) =
+    quietly(sub { $CLASS->volume_snapshot_rollback($scfg, 'mystore', $volname, 'snap1') });
+ok(!$ok, 'second rollback fails when the new volume cannot be allocated');
+like($err, qr/original volume restored/, 'reports the original volume as restored');
+is_deeply(
+    \@renames,
+    ["$volname -> del-1-$volname", "del-1-$volname -> $volname"],
+    'restores the volume it renamed, not the one of the earlier rollback',
+);
+ok($lvs->{"del-$volname"}, 'volume replaced by the earlier rollback is left alone');
+is($workers, 0, 'started no cleanup worker');
+
+my $rawvolname = 'vm-100-disk-1';
+
+# only a numbered volume is still pending removal, so del-<name> is free
+setup($rawvolname, "del-1-$rawvolname");
+my $discard_scfg = { type => 'lvm', vgname => $vg, 'on-volume-remove' => 'discard=1' };
+my $worker = $CLASS->free_image('mystore', $discard_scfg, $rawvolname, 0, 'raw');
+is_deeply(\@renames, ["$rawvolname -> del-$rawvolname"], 'renamed to del-<name> if that is free');
+($ok, $err) = quietly($worker);
+ok($ok, 'cleanup worker succeeds') or diag($err);
+is_deeply(\@activated, ["/dev/$vg/del-$rawvolname"], 'cleanup worker activates its own volume');
+is_deeply(\@removed, ["del-$rawvolname"], 'cleanup worker removes its own volume');
+ok($lvs->{"del-1-$rawvolname"}, 'volume pending removal is left alone');
+
+# zeroing out fails without a device mapper link, so the cleanup worker renames its volume
+setup($rawvolname, "del-$rawvolname", "del-1-$rawvolname");
+$worker = $CLASS->free_image('mystore', $scfg, $rawvolname, 0, 'raw');
+is_deeply(
+    \@renames,
+    ["$rawvolname -> del-2-$rawvolname"],
+    'renamed to the first free numbered name',
+);
+@renames = ();
+($ok, $err) = quietly($worker);
+ok(!$ok, 'cleanup worker fails');
+like($err, qr/cleanup failed for 1 out of 1 volumes/, 'cleanup worker reports the failed cleanup');
+is_deeply(\@activated, ["/dev/$vg/del-2-$rawvolname"], 'cleanup worker activates its own volume');
+is_deeply(
+    \@renames,
+    ["del-2-$rawvolname -> failed-0-del-2-$rawvolname"],
+    'failed cleanup renames its own volume',
+);
+is_deeply(\@removed, [], 'failed cleanup removes no volume');
+ok(
+    $lvs->{"del-$rawvolname"} && $lvs->{"del-1-$rawvolname"},
+    'volumes pending removal are left alone',
+);
+
+done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-19 14:13 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 14:12 [PATCH common/storage v2 0/4] fix #8286: do not hold inherited locks in volume cleanup workers Michael Köppl
2026-11-19 14:12 ` [PATCH common v2 1/4] tools: lock file: add helper to close inherited lock handles Michael Köppl
2026-11-19 14:13 ` Michael Köppl [this message]
2026-11-19 14:13 ` [PATCH storage v2 3/4] fix #8286: vdisk free: do not hold inherited locks in cleanup worker Michael Köppl
2026-11-19 14:13 ` [PATCH storage v2 4/4] lvm plugin: do not hold inherited locks in cleanup workers Michael Köppl

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119141302.600137-3-m.koeppl@proxmox.com \
    --to=m.koeppl@proxmox.com \
    --cc=l.sichert@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal