all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v8 storage 25/43] disks: allow wiping a SAN LUN that no storage uses
Date: Thu, 19 Nov 2026 20:46:30 +0100	[thread overview]
Message-ID: <20261119194655.3163258-26-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261119194655.3163258-1-t.lamprecht@proxmox.com>

A recycled LUN can hold old data, for example the partition table of
a VMFS datastore, which the volume group creation refuses. The disk
wipe of a node wipes any block device that is not mounted and has no
holder. So it refuses a multipath map whose partitions are mapped,
but wipes a LUN that is only in use on other nodes.

For a LUN of the SAN inventory, run the wipe under the cluster-wide
lock of the LUN and only if no storage references it. As for the
volume group creation, the wipe also refuses a LUN or partition that
a process of the node has open, and a LUN in use through another of
its paths. It removes the partition mappings of a multipath map and
writes through an exclusively opened descriptor. LVM keeps listing a
wiped PV in its hint file until two more LVM commands have run, and
the LUN list takes the volume groups from that file. So the wipe
removes the file afterwards and lets LVM write it again.

A LUN can also be given by its by-id or mapper name. Other devices,
including the kernel partitions of a LUN, are wiped as before, and
only by the names that were accepted so far. A by-id, mapper or dm-N
name could also select a logical volume or a partition that kpartx
maps for a multipath map, which the mount and holder checks do not
protect. The wipe of the map covers such a partition.

A device name can pass to another device while the wipe looks up the
LUN, waits for the lock or runs, on nodes where maps come and go
quickly. So, as for the volume group creation, the lookup checks the
requested name again after its scan, and the wipe holds the LUN open
by its by-id name. Before each command for a partition and before the
exclusive open, the names of the LUN must still lead to the held
device, and each partition must still belong to the LUN. A command
that takes a name still cannot be made atomic with its check.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 src/PVE/API2/Disks.pm          |  40 ++++-
 src/PVE/Diskmanage.pm          |  47 ++++--
 src/PVE/Storage/LVMPlugin.pm   |   7 +
 src/PVE/Storage/SAN.pm         |  54 ++++++-
 src/test/Makefile              |   3 +-
 src/test/disk_api_test.pm      |  46 ++++++
 src/test/run_san_nvme_tests.pl |  16 ++
 src/test/run_san_wipe_tests.pl | 271 +++++++++++++++++++++++++++++++++
 8 files changed, 461 insertions(+), 23 deletions(-)
 create mode 100755 src/test/run_san_wipe_tests.pl

diff --git a/src/PVE/API2/Disks.pm b/src/PVE/API2/Disks.pm
index 980ba7ff..d157c430 100644
--- a/src/PVE/API2/Disks.pm
+++ b/src/PVE/API2/Disks.pm
@@ -9,6 +9,8 @@ use HTTP::Status qw(:constants);
 use PVE::Diskmanage;
 use PVE::JSONSchema qw(get_standard_option);
 use PVE::SafeSyslog;
+use PVE::Storage::LVMPlugin;
+use PVE::Storage::SAN;
 use PVE::Tools qw(run_command);
 
 use PVE::API2::Disks::Directory;
@@ -295,8 +297,9 @@ __PACKAGE__->register_method({
             node => get_standard_option('pve-node'),
             disk => {
                 type => 'string',
-                description => "Block device name",
-                pattern => '^/dev/[a-zA-Z0-9\/]+$',
+                description =>
+                    "Block device name. A SAN LUN can also be given by a by-id or mapper name.",
+                pattern => '^/dev/[a-zA-Z0-9._:+/-]+$',
             },
         },
     },
@@ -306,18 +309,39 @@ __PACKAGE__->register_method({
 
         my $disk = PVE::Diskmanage::verify_blockdev_path($param->{disk});
 
-        my $mounted = PVE::Diskmanage::is_mounted($disk);
-        die "disk/partition '${mounted}' is mounted\n" if $mounted;
+        my $lun = PVE::Storage::SAN::lun_for_device($param->{disk});
+        if (!$lun) {
+            # A by-id, mapper or dm-N name could also select a logical volume or a partition of a
+            # multipath map, which the checks for mounts and holders do not protect.
+            if ($param->{disk} !~ m{^/dev/[a-zA-Z0-9/]+$}) {
+                die "cannot wipe '$param->{disk}' - a name of this form is only accepted for a"
+                    . " SAN LUN\n";
+            }
 
-        my $held = PVE::Diskmanage::has_holder($disk);
-        die "disk/partition '${held}' has a holder\n" if $held;
+            my $mounted = PVE::Diskmanage::is_mounted($disk);
+            die "disk/partition '${mounted}' is mounted\n" if $mounted;
+
+            my $held = PVE::Diskmanage::has_holder($disk);
+            die "disk/partition '${held}' has a holder\n" if $held;
+        }
 
         my $rpcenv = PVE::RPCEnvironment::get();
         my $authuser = $rpcenv->get_user();
 
         my $worker = sub {
-            PVE::Diskmanage::wipe_blockdev($disk);
-            PVE::Diskmanage::udevadm_trigger($disk);
+            if ($lun) {
+                PVE::Storage::SAN::with_lun_lock(
+                    $lun,
+                    sub { PVE::Storage::SAN::wipe_lun($lun) },
+                );
+                PVE::Diskmanage::udevadm_trigger($disk, PVE::Storage::SAN::lun_paths($lun)->@*);
+                eval { run_command(['udevadm', 'settle', '--timeout=30'], timeout => 35) };
+                warn $@ if $@;
+                PVE::Storage::LVMPlugin::refresh_cached_pv_volume_groups();
+            } else {
+                PVE::Diskmanage::wipe_blockdev($disk);
+                PVE::Diskmanage::udevadm_trigger($disk);
+            }
         };
 
         my $basename = basename($disk); # avoid '/' in the ID
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 29669a28..6e643ef0 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -1213,8 +1213,10 @@ sub device_opener {
 # Wipes all labels and the first 200 MiB of a disk/partition (or the whole if it is smaller).
 # If called with a partition, also sets the partition type to 0x83 'Linux filesystem'.
 # Expected to be called with a result of verify_blockdev_path().
+# With $writepath, writes through that path instead, for example the /proc path of a descriptor of
+# the caller, and leaves child partitions and the partition type alone.
 sub wipe_blockdev {
-    my ($devpath) = @_;
+    my ($devpath, $writepath) = @_;
 
     my $devname = basename($devpath);
     my $dev_size = PVE::Tools::file_get_contents("/sys/class/block/$devname/size");
@@ -1226,30 +1228,49 @@ sub wipe_blockdev {
     my $count = ($size < 200) ? $size : 200;
 
     my $to_wipe = [];
-    dir_glob_foreach(
-        "/sys/class/block/${devname}",
-        "${devname}.+",
-        sub {
-            my ($part) = @_;
-            push $to_wipe->@*, "/dev/${part}" if -b "/dev/${part}";
-        },
-    );
+    if (!defined($writepath)) {
+        dir_glob_foreach(
+            "/sys/class/block/${devname}",
+            "${devname}.+",
+            sub {
+                my ($part) = @_;
+                push $to_wipe->@*, "/dev/${part}" if -b "/dev/${part}";
+            },
+        );
+    }
 
     if (scalar($to_wipe->@*) > 0) {
         print "found child partitions to wipe: " . join(', ', $to_wipe->@*) . "\n";
     }
-    push $to_wipe->@*, $devpath; # put actual device last
+    push $to_wipe->@*, $writepath // $devpath; # put actual device last
 
     print "wiping block device ${devpath}\n";
 
-    run_command(['wipefs', '--all', $to_wipe->@*], errmsg => "error wiping '${devpath}'");
+    # With a write path, the caller holds the device open exclusively, which already checked for
+    # other exclusive users. Unless forced, wipefs would try the same and fail on that descriptor.
+    run_command(
+        ['wipefs', '--all', defined($writepath) ? ('--force') : (), $to_wipe->@*],
+        errmsg => "error wiping '${devpath}'",
+        defined($writepath) ? (timeout => 10) : (),
+    );
 
+    my $target = $writepath // $devpath;
     run_command(
-        ['dd', 'if=/dev/zero', "of=${devpath}", 'bs=1M', 'conv=fdatasync', "count=${count}"],
+        [
+            'dd',
+            'if=/dev/zero',
+            "of=${target}",
+            'bs=1M',
+            'conv=fdatasync',
+            defined($writepath)
+            ? ('iflag=count_bytes', 'count=' . ($count * 1024 * 1024))
+            : "count=${count}",
+        ],
         errmsg => "error wiping '${devpath}'",
+        defined($writepath) ? (timeout => 60) : (),
     );
 
-    if (is_partition($devpath)) {
+    if (!defined($writepath) && is_partition($devpath)) {
         eval { change_parttype($devpath, '8300'); };
         warn $@ if $@;
     }
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index ca82ba98..21558661 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -162,6 +162,13 @@ sub cached_pv_volume_groups {
     return $res;
 }
 
+# LVM keeps listing a PV that was wiped by other means in its hint file until two more LVM commands
+# have run. It writes a new file when there is none.
+sub refresh_cached_pv_volume_groups {
+    unlink '/run/lvm/hints';
+    lvm_vgs();
+}
+
 sub lun_volume_groups {
     my ($devices) = @_;
     my $output = '';
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 8d559d4f..f811fee0 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -3,11 +3,12 @@ package PVE::Storage::SAN;
 use v5.36;
 
 use Cwd qw(abs_path);
+use Fcntl qw(O_RDWR O_EXCL);
 use File::Basename qw(basename);
 
 use PVE::Cluster;
 use PVE::INotify;
-use PVE::Tools qw(file_read_firstline dir_glob_foreach);
+use PVE::Tools qw(file_read_firstline dir_glob_foreach run_command);
 
 use PVE::Diskmanage;
 use PVE::ISCSI::Topology;
@@ -334,6 +335,9 @@ sub assert_same_device($lun, $fh, @names) {
     }
 }
 
+# The SCSI paths of the LUN. Without a map, udev keeps the signatures it has seen for each of them,
+# so a wipe needs a udev trigger for every path. The members of a map count whatever their identity,
+# which without multipathd comes from udev and can differ from the one of the map.
 sub lun_paths($lun) {
     my $paths = PVE::Multipath::Identity::collect(all => 1)->{paths};
     my %names = map { $_ => 1 } grep { $paths->{$_}->{wwid} eq $lun->{wwid} } keys %$paths;
@@ -385,4 +389,52 @@ sub assert_lun_paths_unused($lun) {
     }
 }
 
+sub wipe_lun($lun) {
+    assert_lun_unreferenced(PVE::Storage::config(), $lun);
+    assert_lun_paths_unused($lun);
+    my $device = PVE::Diskmanage::verify_blockdev_path($lun->{'by-id'});
+    my $parts = lun_partitions($lun);
+    my %parts = map { basename($_) => 1 } @$parts;
+    dir_glob_foreach(
+        '/sys/class/block/' . basename($device) . '/holders',
+        '[^.].*',
+        sub($name, @) {
+            die "cannot wipe '$device' - another block device uses it\n" if !$parts{$name};
+        },
+    );
+    for my $path ($device, @$parts) {
+        die "cannot wipe '$device' - '$path' is mounted\n" if PVE::Diskmanage::is_mounted($path);
+        next if $path eq $device;
+        die "cannot wipe '$device' - another block device uses '$path'\n"
+            if PVE::Diskmanage::has_holder($path);
+    }
+    # a guest with the LUN as disk has it open, without being a holder or a storage reference
+    my $opener = PVE::Diskmanage::device_opener($device, @$parts);
+    die "cannot wipe '$device' - process $opener has it open\n" if defined($opener);
+    sysopen(my $fh, $lun->{'by-id'}, O_RDWR) or die "cannot open '$device' - $!\n";
+    # the name of a partition can pass to another device as well, since it got listed
+    my $check_part = sub($part) {
+        assert_same_device($lun, $fh, $device);
+        die "cannot wipe '$device' - '$part' is no longer one of its partitions\n"
+            if !grep { $_ eq $part } lun_partitions($lun)->@*;
+    };
+    for my $part (@$parts) {
+        $check_part->($part);
+        run_command(['wipefs', '--all', $part], timeout => 10);
+        if ($part =~ m{^/dev/dm-}) {
+            $check_part->($part);
+            run_command(['dmsetup', 'remove', '--retry', $part], timeout => 10);
+        }
+    }
+    assert_same_device($lun, $fh, $device);
+    # claim what was checked: while the first descriptor is open, its device number cannot pass to
+    # another device, so an equal number means the same device
+    sysopen(my $claim, $lun->{'by-id'}, O_RDWR | O_EXCL) or die "cannot claim '$device' - $!\n";
+    die "cannot wipe '$device' - the device behind '$lun->{'by-id'}' changed\n"
+        if (stat($claim))[6] != (stat($fh))[6];
+    close($fh);
+    my $path = '/proc/' . $$ . '/fd/' . fileno($claim);
+    PVE::Diskmanage::wipe_blockdev($device, $path);
+}
+
 1;
diff --git a/src/test/Makefile b/src/test/Makefile
index 0d866d91..279ba1ee 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -76,8 +76,9 @@ test_san_taint: run_san_taint_tests.pl
 test_san_lun_scan: run_san_lun_scan_tests.pl
 	./run_san_lun_scan_tests.pl
 
-test_san_api: run_san_api_tests.pl run_san_nvme_tests.pl
+test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
 	./run_san_api_tests.pl
+	./run_san_wipe_tests.pl
 	./run_san_nvme_tests.pl
 
 test_disk_api: run_disk_api_tests.pl
diff --git a/src/test/disk_api_test.pm b/src/test/disk_api_test.pm
index 6fa2a2e3..1d2f8d33 100644
--- a/src/test/disk_api_test.pm
+++ b/src/test/disk_api_test.pm
@@ -265,4 +265,50 @@ subtest 'shared volume groups cannot be destroyed' => sub {
     is_deeply(\@writes, ['data'], 'node-local deletion still works');
 };
 
+subtest 'wipe takes names beyond the old pattern for a LUN only' => sub {
+    my $san = Test::MockModule->new('PVE::Storage::SAN');
+    my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+    $rpc->mock(get => sub { bless {}, 'MultipathTestRPC' });
+    my ($lun, @wiped);
+    $san->mock(
+        lun_for_device => sub { $lun },
+        with_lun_lock => sub { push @wiped, $_[0]->{wwid} },
+        lun_paths => sub { [] },
+    );
+    $diskmanage_module->mock(
+        verify_blockdev_path => sub { $_[0] },
+        is_mounted => sub { },
+        has_holder => sub { },
+        wipe_blockdev => sub { push @wiped, $_[0] },
+        udevadm_trigger => sub { },
+    );
+    my $api = Test::MockModule->new('PVE::API2::Disks');
+    $api->mock(run_command => sub { });
+    my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+    $lvm->mock(refresh_cached_pv_volume_groups => sub { });
+    my $wipe = sub($name) {
+        return
+            eval { PVE::API2::Disks->wipe_disk({ node => 'n1', disk => $name }); 1 } ? '' : $@;
+    };
+
+    my @other = qw(
+        /dev/dm-7 /dev/mapper/3600a-part1 /dev/disk/by-id/dm-uuid-part1-mpath-3600a
+        /dev/pve/vm-100-disk-0
+    );
+    for my $name (@other) {
+        like($wipe->($name), qr/only accepted for a SAN LUN/, "$name is refused");
+    }
+    is($wipe->('/dev/sdb1'), '', 'a partition by its kernel name is wiped');
+    $lun = { wwid => '3600a' };
+    is(
+        $wipe->('/dev/disk/by-id/dm-uuid-mpath-3600a'),
+        '',
+        'a LUN by the name of its map is wiped',
+    );
+    is_deeply(\@wiped, ['/dev/sdb1', '3600a'], 'nothing else got wiped');
+
+    my @mocked = qw(verify_blockdev_path is_mounted has_holder wipe_blockdev udevadm_trigger);
+    $diskmanage_module->unmock($_) for @mocked;
+};
+
 done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 7b963211..59a009c9 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -31,6 +31,7 @@ my $mp = Test::MockModule->new('PVE::Multipath');
 my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
 my $storage = Test::MockModule->new('PVE::Storage');
 my $cluster = Test::MockModule->new('PVE::Cluster');
+my $tools = Test::MockModule->new('PVE::Tools');
 my $cfg = { ids => {} };
 my @order;
 $storage->mock(config => sub { $cfg });
@@ -45,6 +46,7 @@ $mp->mock(device_map => sub { undef });
 $san->mock(abs_path => sub { $device });
 $san->mock(scan_luns => sub { [$lun] });
 $san->mock(lun_partitions => sub { [] });
+$san->mock(dir_glob_foreach => sub { });
 $san->mock(assert_same_device => sub { ok(defined(fileno($_[1])), 'held descriptor checked') });
 $lvm->mock(lun_volume_groups => sub { push @order, 'check'; [] });
 $cluster->mock(
@@ -79,6 +81,20 @@ for my $base (qw(fabric nvme-fabric)) {
     like($@, qr/base volume/, "native base reference on storage '$base' refuses");
 }
 $cfg->{ids} = {};
+@order = @opens = ();
+$tools->mock(file_get_contents => sub { 1024 });
+$disk->mock(
+    run_command => sub {
+        my ($cmd, %opts) = @_;
+        push @order, $cmd->[0];
+        my ($path) = map { /^of=(.*)$/ ? $1 : m{^(/proc/\d+/fd/\d+)$} ? $1 : () } @$cmd;
+        ok(defined($path) && -e $path, 'native wipe writes through the held descriptor');
+    },
+);
+PVE::Storage::SAN::with_lun_lock($lun, sub { PVE::Storage::SAN::wipe_lun($lun) });
+is_deeply(\@order, [qw(lock check wipefs dd unlock)], 'native wipe sequencing');
+is_deeply(\@opens, [$link, $link], 'native wipe opens its by-id name');
+
 @order = @opens = ();
 $lvm->mock(file_read_firstline => sub { $_[0] =~ m{/size$} ? 1024 : undef });
 $lvm->mock(lvm_pv_info => sub { undef });
diff --git a/src/test/run_san_wipe_tests.pl b/src/test/run_san_wipe_tests.pl
new file mode 100755
index 00000000..5828e20a
--- /dev/null
+++ b/src/test/run_san_wipe_tests.pl
@@ -0,0 +1,271 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+our (@open_modes, $claim_other);
+
+BEGIN {
+    *CORE::GLOBAL::sysopen = sub (*$$;$) {
+        push @open_modes, $_[2];
+        # lets a name mean another device by the time it is opened exclusively
+        my $exclusive = $_[2] & Fcntl::O_EXCL();
+        my $name = $exclusive && defined($claim_other) ? $claim_other : $_[1];
+        return @_ == 4
+            ? CORE::sysopen($_[0], $name, $_[2], $_[3])
+            : CORE::sysopen($_[0], $name, $_[2]);
+    };
+}
+
+use lib qw(..);
+use Cwd qw(abs_path);
+use Fcntl qw(O_RDWR O_EXCL);
+use File::Temp qw(tempfile);
+use Test::MockModule;
+use Test::More;
+use PVE::Storage::SAN;
+
+{
+    my ($fh, $name) = tempfile(UNLINK => 1);
+    my ($tmp, $read_name) = tempfile(UNLINK => 1);
+    open(my $read_fh, '<', $read_name) or die "cannot open '$read_name' - $!\n";
+    close($tmp);
+    my $pid = fork() // die "fork failed - $!\n";
+    if (!$pid) {
+        sleep(60);
+        exit(0);
+    }
+    like(PVE::Diskmanage::device_opener($name), qr/^$pid \(/, 'a process with the file open');
+    my $all = PVE::Diskmanage::device_openers([$name, $read_name, "$name-none"]);
+    is_deeply(
+        [sort keys %$all],
+        [sort map { abs_path($_) } $name, $read_name],
+        'each open file has a process, by its resolved path',
+    );
+    my $writers = PVE::Diskmanage::device_openers([$name, $read_name], writable => 1);
+    is_deeply([keys %$writers], [abs_path($name)], 'a file open for reading can be left out');
+    kill('TERM', $pid);
+    waitpid($pid, 0);
+    is(PVE::Diskmanage::device_opener($name), undef, 'the current process does not count');
+}
+
+my ($file, $name) = tempfile('san-wipe-XXXX', DIR => '.', UNLINK => 1);
+close($file);
+my $lun = { devpath => $name, 'by-id' => $name, wwid => '3600a' };
+my $san = Test::MockModule->new('PVE::Storage::SAN');
+my $disk = Test::MockModule->new('PVE::Diskmanage');
+my $tools = Test::MockModule->new('PVE::Tools');
+my (@commands, @parts, @parttypes);
+my $sectors = 1024 * 1024;
+my $run = sub {
+    my ($cmd, %opts) = @_;
+    push @commands, [$cmd, \%opts];
+    if ($cmd->[0] eq 'dd' || ($cmd->[0] eq 'wipefs' && grep { $_ eq '--force' } @$cmd)) {
+        my ($path) = map { /^of=(.*)$/ ? $1 : m{^(/proc/\d+/fd/\d+)$} ? $1 : () } @$cmd;
+        ok(defined($path) && -e $path, 'whole-device command uses a live descriptor');
+        is((stat($path))[1], (stat($name))[1], 'descriptor still names the opened device');
+        ok($open_modes[-1] & O_EXCL, 'exclusive descriptor held for whole-device commands');
+    }
+};
+$san->mock(run_command => $run);
+my $referenced = 0;
+my $storage = Test::MockModule->new('PVE::Storage');
+$storage->mock(config => sub { { ids => {} } });
+$san->mock(assert_lun_unreferenced => sub { die "referenced\n" if $referenced });
+my $path_used = 0;
+$san->mock(assert_lun_paths_unused => sub { die "path in use\n" if $path_used });
+$san->mock(lun_partitions => sub { [@parts] });
+$san->mock(dir_glob_foreach => sub { });
+my $canonical = $name;
+$disk->mock(verify_blockdev_path => sub { $canonical });
+$disk->mock(is_mounted => sub { 0 });
+$disk->mock(has_holder => sub { 0 });
+my $opener;
+$disk->mock(device_opener => sub { $opener });
+$disk->mock(run_command => $run);
+$disk->mock(dir_glob_foreach => sub { die "partition enumeration after exclusive open\n" });
+$disk->mock(is_partition => sub { die "partition check after exclusive open\n" });
+$tools->mock(file_get_contents => sub { $sectors });
+
+$referenced = 1;
+@commands = @open_modes = ();
+eval { PVE::Storage::SAN::wipe_lun($lun) };
+like($@, qr/referenced/, 'a referenced LUN is refused');
+is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written');
+$referenced = 0;
+$path_used = 1;
+eval { PVE::Storage::SAN::wipe_lun($lun) };
+like($@, qr/path in use/, 'a LUN in use through another path is refused');
+is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written');
+$path_used = 0;
+$opener = '4711 (kvm)';
+eval { PVE::Storage::SAN::wipe_lun($lun) };
+like($@, qr/process 4711 \(kvm\) has it open/, 'a LUN that a process has open is refused');
+is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written');
+$opener = undef;
+
+for my $count (0, 2, 10) {
+    @parts = map { "/dev/dm-$_" } 1 .. $count;
+    @commands = @open_modes = ();
+    PVE::Storage::SAN::wipe_lun($lun);
+    is_deeply(\@open_modes, [O_RDWR, O_RDWR | O_EXCL], 'one exclusive whole-device open');
+    my $fdpath = $commands[-2]->[0]->[-1];
+    is_deeply(
+        $commands[-2]->[0],
+        ['wipefs', '--all', '--force', $fdpath],
+        'force avoids a second exclusive claim',
+    );
+    is($commands[-1]->[1]->{timeout}, 60, 'dd has 60 seconds for fdatasync');
+    ok(grep({ $_ eq 'count=209715200' } $commands[-1]->[0]->@*), 'wipe capped at 200 MiB');
+
+    for my $i (0 .. $#parts) {
+        is_deeply(
+            $commands[2 * $i]->[0],
+            ['wipefs', '--all', $parts[$i]],
+            'partition signatures first',
+        );
+        is_deeply(
+            $commands[2 * $i + 1]->[0],
+            ['dmsetup', 'remove', '--retry', $parts[$i]],
+            'remove partition map with retry',
+        );
+    }
+    is($_->[1]->{timeout}, 10, 'partition command has a fixed timeout')
+        for @commands[0 .. $#commands - 2];
+    ok(!-e $fdpath, 'descriptor released on return');
+}
+
+@parts = ();
+@commands = ();
+$sectors = 1;
+PVE::Storage::SAN::wipe_lun($lun);
+ok(grep({ $_ eq 'count=512' } $commands[-1]->[0]->@*), 'small device uses a byte count');
+
+@parts = ('/dev/dm-1');
+@commands = ();
+eval { PVE::Storage::SAN::wipe_lun({ %$lun, devpath => '/dev/null' }) };
+like($@, qr{the device behind '/dev/null' changed}, 'a path that means another device refuses');
+is_deeply(\@commands, [], 'before a partition or the device is wiped');
+my $listed = 0;
+$san->mock(lun_partitions => sub { $listed++ ? [] : ['/dev/dm-1'] });
+eval { PVE::Storage::SAN::wipe_lun($lun) };
+like($@, qr{'/dev/dm-1' is no longer one of its partitions}, 'a partition name that passed on');
+is_deeply(\@commands, [], 'is not wiped');
+$san->mock(lun_partitions => sub { [@parts] });
+@parts = ();
+my $err = do {
+    local $claim_other = '/dev/zero';
+    eval { PVE::Storage::SAN::wipe_lun($lun) };
+    $@;
+};
+like(
+    $err,
+    qr{the device behind '\Q$name\E' changed},
+    'a by-id name that passed on before the claim',
+);
+is_deeply(\@commands, [], 'is not wiped either');
+$canonical = '/dev/null';
+eval { PVE::Storage::SAN::wipe_lun($lun) };
+like($@, qr{the device behind '/dev/null' changed}, 'a resolved name that no longer means the LUN');
+is_deeply(\@commands, [], 'refuses before a write as well');
+$canonical = $name;
+
+$disk->mock(dir_glob_foreach => sub { });
+$disk->mock(is_partition => sub { 1 });
+$disk->mock(change_parttype => sub { push @parttypes, [@_] });
+$disk->mock(run_command => sub { my ($cmd, %opts) = @_; push @commands, [$cmd, \%opts] });
+@commands = ();
+$sectors = 204800;
+PVE::Diskmanage::wipe_blockdev('/dev/sda1');
+is_deeply(
+    [map { $_->[0] } @commands],
+    [
+        ['wipefs', '--all', '/dev/sda1'],
+        ['dd', 'if=/dev/zero', 'of=/dev/sda1', 'bs=1M', 'conv=fdatasync', 'count=100'],
+    ],
+    'local disk command lines unchanged',
+);
+ok(!grep({ exists($_->[1]->{timeout}) } @commands), 'local disk timeout behavior unchanged');
+is_deeply(\@parttypes, [['/dev/sda1', '8300']], 'local partition type still updated');
+
+my $identity = Test::MockModule->new('PVE::Multipath::Identity');
+$identity->mock(
+    collect => sub {
+        {
+            paths => {
+                sdb => { wwid => '3600a' },
+                sdc => { wwid => '3600b' },
+                sdd => { wwid => '3600a' },
+            },
+        };
+    },
+);
+is_deeply(
+    PVE::Storage::SAN::lun_paths($lun),
+    ['/dev/sdb', '/dev/sdd'],
+    'all paths of the LUN are refreshed after a wipe',
+);
+
+# the other paths of a LUN: sdd beside sdb without a map, both below the map dm-3
+$san->unmock('assert_lun_paths_unused');
+my ($mounted, %holders, $writers, $asked) = (undef, ());
+$disk->mock(is_mounted => sub { $_[0] eq '/dev/sdd' ? $mounted : undef });
+$san->mock(
+    dir_glob_foreach => sub {
+        my ($dir, $regex, $code) = @_;
+        $code->($_) for ($holders{$dir} // [])->@*;
+    },
+);
+$disk->mock(
+    device_openers => sub {
+        my ($devices, %param) = @_;
+        $asked = [$devices, $param{writable}];
+        return $writers // {};
+    },
+);
+$san->mock(abs_path => sub { $_[0] });
+my $path_lun = { %$lun, devpath => '/dev/sdb' };
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+is($@, '', 'unused paths pass');
+is_deeply($asked, [['/dev/sdd'], 1], 'writers on the other paths, multipathd only reads them');
+$mounted = '/dev/sdd1';
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+like($@, qr{the LUN is mounted through '/dev/sdd1'}, 'a mounted path or partition refuses');
+$mounted = undef;
+$holders{'/sys/class/block/sdd/holders'} = ['dm-3'];
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+like($@, qr{another block device uses the LUN through '/dev/sdd'}, 'a held path refuses');
+$holders{'/sys/class/block/sdd/holders'} = [];
+$holders{'/sys/class/block/sdd'} = ['sdd1'];
+$holders{'/sys/class/block/sdd1/holders'} = ['dm-7'];
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+like($@, qr{uses the LUN through '/dev/sdd1'}, 'so does a held partition of a path');
+$holders{'/sys/class/block/sdd1/holders'} = [];
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+is_deeply($asked->[0], ['/dev/sdd', '/dev/sdd1'], 'writers on the partitions of a path count');
+$holders{'/sys/class/block/sdb/holders'} = ['dm-3'];
+eval { PVE::Storage::SAN::assert_lun_paths_unused({ %$lun, devpath => '/dev/dm-3' }) };
+is($@, '', 'the map of the LUN holds its paths');
+%holders = ();
+$writers = { '/dev/sdd' => { pid => 4711, command => 'kvm' } };
+eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) };
+like($@, qr{process 4711 \(kvm\) has the LUN open through '/dev/sdd'},
+    'a writer on a path refuses');
+
+# without multipathd, udev can know a member of the map by another identity than the map
+my $map_lun = { %$lun, devpath => '/dev/dm-3' };
+$holders{'/sys/class/block/dm-3/slaves'} = ['sdc'];
+is_deeply(
+    PVE::Storage::SAN::lun_paths($map_lun),
+    ['/dev/sdb', '/dev/sdc', '/dev/sdd'],
+    'the members of the map are paths of the LUN, whatever their identity',
+);
+$writers = { '/dev/sdc' => { pid => 4712, command => 'kvm' } };
+eval { PVE::Storage::SAN::assert_lun_paths_unused($map_lun) };
+like(
+    $@,
+    qr{process 4712 \(kvm\) has the LUN open through '/dev/sdc'},
+    'a writer on such a member refuses',
+);
+
+done_testing();




  parent reply	other threads:[~2026-11-19 19:54 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 19:46 [PATCH v8 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 cluster 01/43] cfs: observe the multipath configuration files Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 02/43] diskmanage: collect disk transport type from lsblk Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 03/43] diskmanage: qualify NVMe over Fabrics transport Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 04/43] diskmanage: optionally include iSCSI session devices in disk enumeration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 05/43] diskmanage: collect native NVMe multipath path state Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 06/43] iscsi: factor out session device map from device list Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 07/43] multipath: add a library around multipathd Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 08/43] diskmanage: add helper to list multipath devices Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 09/43] multipath: add a cluster-wide configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 10/43] iscsi: add an option to manage storage LUNs with multipath Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 11/43] lvm: activate a missing base volume before using the volume group Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 12/43] multipath: apply the configuration on every node Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 13/43] multipath: publish per-node status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 14/43] api: add endpoints for the cluster-wide multipath configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 15/43] api: multipath: add the cluster-wide health status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 16/43] diskmanage: look up the requested disks by name Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 17/43] diskmanage: read the udev data of all disks with one call Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 18/43] api: scan: add a SAN LUN inventory Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 19/43] pvesm: expose the " Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 20/43] api: add node-level iSCSI status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 21/43] api: scan: name the portal of an iSCSI path Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 22/43] disks: lvm: drop unused imports Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 23/43] disks: lvm: allow volume groups on SAN LUNs and multipath maps Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 24/43] disks: lvm: refuse to destroy the volume group of a shared storage Thomas Lamprecht
2026-11-19 19:46 ` Thomas Lamprecht [this message]
2026-11-19 19:46 ` [PATCH v8 storage 26/43] api: scan: report the process that has a SAN LUN open Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 docs 27/43] storage: document the cluster-wide multipath management Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 docs 28/43] storage: multipath: describe the use shown for an open LUN Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 29/43] pvestatd: apply the multipath configuration and report the node status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 30/43] report: add the managed multipath configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 31/43] api: cluster: add the multipath endpoints Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 32/43] api: nodes: expose the iSCSI status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 33/43] ui: dc: add a SAN view Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 34/43] ui: storage: iscsi: add the multipath option Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 35/43] ui: lvm: enable shared when a base storage is selected Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 36/43] ui: node: add SAN LUNs panel Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 37/43] ui: node: san: wipe a LUN that no storage uses Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 38/43] ui: node: san: show iSCSI targets and sessions Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 39/43] ui: dc: san: add a guided setup for shared LVM on iSCSI Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 40/43] ui: dc: san: add a guided setup for shared LVM on Fibre Channel Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 41/43] ui: storage: point the iSCSI and LVM add dialogs to the SAN view Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 42/43] ui: san: show the guest or process that has a LUN open Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 43/43] ui: node: san: add a column for the target or subsystem of a LUN Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119194655.3163258-26-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal