From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 44CD11FF0DA for ; Thu, 19 Nov 2026 20:54:10 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 85DF921F3A; Thu, 19 Nov 2026 20:48:45 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v8 storage 25/43] disks: allow wiping a SAN LUN that no storage uses Date: Thu, 19 Nov 2026 20:46:30 +0100 Message-ID: <20261119194655.3163258-26-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119194655.3163258-1-t.lamprecht@proxmox.com> References: <20261119194655.3163258-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795117622731 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.150 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KAJL6K7JKEUH5ZN3V4TLB2NOKFOM734Z X-Message-ID-Hash: KAJL6K7JKEUH5ZN3V4TLB2NOKFOM734Z X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A recycled LUN can hold old data, for example the partition table of a VMFS datastore, which the volume group creation refuses. The disk wipe of a node wipes any block device that is not mounted and has no holder. So it refuses a multipath map whose partitions are mapped, but wipes a LUN that is only in use on other nodes. For a LUN of the SAN inventory, run the wipe under the cluster-wide lock of the LUN and only if no storage references it. As for the volume group creation, the wipe also refuses a LUN or partition that a process of the node has open, and a LUN in use through another of its paths. It removes the partition mappings of a multipath map and writes through an exclusively opened descriptor. LVM keeps listing a wiped PV in its hint file until two more LVM commands have run, and the LUN list takes the volume groups from that file. So the wipe removes the file afterwards and lets LVM write it again. A LUN can also be given by its by-id or mapper name. Other devices, including the kernel partitions of a LUN, are wiped as before, and only by the names that were accepted so far. A by-id, mapper or dm-N name could also select a logical volume or a partition that kpartx maps for a multipath map, which the mount and holder checks do not protect. The wipe of the map covers such a partition. A device name can pass to another device while the wipe looks up the LUN, waits for the lock or runs, on nodes where maps come and go quickly. So, as for the volume group creation, the lookup checks the requested name again after its scan, and the wipe holds the LUN open by its by-id name. Before each command for a partition and before the exclusive open, the names of the LUN must still lead to the held device, and each partition must still belong to the LUN. A command that takes a name still cannot be made atomic with its check. Signed-off-by: Thomas Lamprecht --- src/PVE/API2/Disks.pm | 40 ++++- src/PVE/Diskmanage.pm | 47 ++++-- src/PVE/Storage/LVMPlugin.pm | 7 + src/PVE/Storage/SAN.pm | 54 ++++++- src/test/Makefile | 3 +- src/test/disk_api_test.pm | 46 ++++++ src/test/run_san_nvme_tests.pl | 16 ++ src/test/run_san_wipe_tests.pl | 271 +++++++++++++++++++++++++++++++++ 8 files changed, 461 insertions(+), 23 deletions(-) create mode 100755 src/test/run_san_wipe_tests.pl diff --git a/src/PVE/API2/Disks.pm b/src/PVE/API2/Disks.pm index 980ba7ff..d157c430 100644 --- a/src/PVE/API2/Disks.pm +++ b/src/PVE/API2/Disks.pm @@ -9,6 +9,8 @@ use HTTP::Status qw(:constants); use PVE::Diskmanage; use PVE::JSONSchema qw(get_standard_option); use PVE::SafeSyslog; +use PVE::Storage::LVMPlugin; +use PVE::Storage::SAN; use PVE::Tools qw(run_command); use PVE::API2::Disks::Directory; @@ -295,8 +297,9 @@ __PACKAGE__->register_method({ node => get_standard_option('pve-node'), disk => { type => 'string', - description => "Block device name", - pattern => '^/dev/[a-zA-Z0-9\/]+$', + description => + "Block device name. A SAN LUN can also be given by a by-id or mapper name.", + pattern => '^/dev/[a-zA-Z0-9._:+/-]+$', }, }, }, @@ -306,18 +309,39 @@ __PACKAGE__->register_method({ my $disk = PVE::Diskmanage::verify_blockdev_path($param->{disk}); - my $mounted = PVE::Diskmanage::is_mounted($disk); - die "disk/partition '${mounted}' is mounted\n" if $mounted; + my $lun = PVE::Storage::SAN::lun_for_device($param->{disk}); + if (!$lun) { + # A by-id, mapper or dm-N name could also select a logical volume or a partition of a + # multipath map, which the checks for mounts and holders do not protect. + if ($param->{disk} !~ m{^/dev/[a-zA-Z0-9/]+$}) { + die "cannot wipe '$param->{disk}' - a name of this form is only accepted for a" + . " SAN LUN\n"; + } - my $held = PVE::Diskmanage::has_holder($disk); - die "disk/partition '${held}' has a holder\n" if $held; + my $mounted = PVE::Diskmanage::is_mounted($disk); + die "disk/partition '${mounted}' is mounted\n" if $mounted; + + my $held = PVE::Diskmanage::has_holder($disk); + die "disk/partition '${held}' has a holder\n" if $held; + } my $rpcenv = PVE::RPCEnvironment::get(); my $authuser = $rpcenv->get_user(); my $worker = sub { - PVE::Diskmanage::wipe_blockdev($disk); - PVE::Diskmanage::udevadm_trigger($disk); + if ($lun) { + PVE::Storage::SAN::with_lun_lock( + $lun, + sub { PVE::Storage::SAN::wipe_lun($lun) }, + ); + PVE::Diskmanage::udevadm_trigger($disk, PVE::Storage::SAN::lun_paths($lun)->@*); + eval { run_command(['udevadm', 'settle', '--timeout=30'], timeout => 35) }; + warn $@ if $@; + PVE::Storage::LVMPlugin::refresh_cached_pv_volume_groups(); + } else { + PVE::Diskmanage::wipe_blockdev($disk); + PVE::Diskmanage::udevadm_trigger($disk); + } }; my $basename = basename($disk); # avoid '/' in the ID diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm index 29669a28..6e643ef0 100644 --- a/src/PVE/Diskmanage.pm +++ b/src/PVE/Diskmanage.pm @@ -1213,8 +1213,10 @@ sub device_opener { # Wipes all labels and the first 200 MiB of a disk/partition (or the whole if it is smaller). # If called with a partition, also sets the partition type to 0x83 'Linux filesystem'. # Expected to be called with a result of verify_blockdev_path(). +# With $writepath, writes through that path instead, for example the /proc path of a descriptor of +# the caller, and leaves child partitions and the partition type alone. sub wipe_blockdev { - my ($devpath) = @_; + my ($devpath, $writepath) = @_; my $devname = basename($devpath); my $dev_size = PVE::Tools::file_get_contents("/sys/class/block/$devname/size"); @@ -1226,30 +1228,49 @@ sub wipe_blockdev { my $count = ($size < 200) ? $size : 200; my $to_wipe = []; - dir_glob_foreach( - "/sys/class/block/${devname}", - "${devname}.+", - sub { - my ($part) = @_; - push $to_wipe->@*, "/dev/${part}" if -b "/dev/${part}"; - }, - ); + if (!defined($writepath)) { + dir_glob_foreach( + "/sys/class/block/${devname}", + "${devname}.+", + sub { + my ($part) = @_; + push $to_wipe->@*, "/dev/${part}" if -b "/dev/${part}"; + }, + ); + } if (scalar($to_wipe->@*) > 0) { print "found child partitions to wipe: " . join(', ', $to_wipe->@*) . "\n"; } - push $to_wipe->@*, $devpath; # put actual device last + push $to_wipe->@*, $writepath // $devpath; # put actual device last print "wiping block device ${devpath}\n"; - run_command(['wipefs', '--all', $to_wipe->@*], errmsg => "error wiping '${devpath}'"); + # With a write path, the caller holds the device open exclusively, which already checked for + # other exclusive users. Unless forced, wipefs would try the same and fail on that descriptor. + run_command( + ['wipefs', '--all', defined($writepath) ? ('--force') : (), $to_wipe->@*], + errmsg => "error wiping '${devpath}'", + defined($writepath) ? (timeout => 10) : (), + ); + my $target = $writepath // $devpath; run_command( - ['dd', 'if=/dev/zero', "of=${devpath}", 'bs=1M', 'conv=fdatasync', "count=${count}"], + [ + 'dd', + 'if=/dev/zero', + "of=${target}", + 'bs=1M', + 'conv=fdatasync', + defined($writepath) + ? ('iflag=count_bytes', 'count=' . ($count * 1024 * 1024)) + : "count=${count}", + ], errmsg => "error wiping '${devpath}'", + defined($writepath) ? (timeout => 60) : (), ); - if (is_partition($devpath)) { + if (!defined($writepath) && is_partition($devpath)) { eval { change_parttype($devpath, '8300'); }; warn $@ if $@; } diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index ca82ba98..21558661 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -162,6 +162,13 @@ sub cached_pv_volume_groups { return $res; } +# LVM keeps listing a PV that was wiped by other means in its hint file until two more LVM commands +# have run. It writes a new file when there is none. +sub refresh_cached_pv_volume_groups { + unlink '/run/lvm/hints'; + lvm_vgs(); +} + sub lun_volume_groups { my ($devices) = @_; my $output = ''; diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm index 8d559d4f..f811fee0 100644 --- a/src/PVE/Storage/SAN.pm +++ b/src/PVE/Storage/SAN.pm @@ -3,11 +3,12 @@ package PVE::Storage::SAN; use v5.36; use Cwd qw(abs_path); +use Fcntl qw(O_RDWR O_EXCL); use File::Basename qw(basename); use PVE::Cluster; use PVE::INotify; -use PVE::Tools qw(file_read_firstline dir_glob_foreach); +use PVE::Tools qw(file_read_firstline dir_glob_foreach run_command); use PVE::Diskmanage; use PVE::ISCSI::Topology; @@ -334,6 +335,9 @@ sub assert_same_device($lun, $fh, @names) { } } +# The SCSI paths of the LUN. Without a map, udev keeps the signatures it has seen for each of them, +# so a wipe needs a udev trigger for every path. The members of a map count whatever their identity, +# which without multipathd comes from udev and can differ from the one of the map. sub lun_paths($lun) { my $paths = PVE::Multipath::Identity::collect(all => 1)->{paths}; my %names = map { $_ => 1 } grep { $paths->{$_}->{wwid} eq $lun->{wwid} } keys %$paths; @@ -385,4 +389,52 @@ sub assert_lun_paths_unused($lun) { } } +sub wipe_lun($lun) { + assert_lun_unreferenced(PVE::Storage::config(), $lun); + assert_lun_paths_unused($lun); + my $device = PVE::Diskmanage::verify_blockdev_path($lun->{'by-id'}); + my $parts = lun_partitions($lun); + my %parts = map { basename($_) => 1 } @$parts; + dir_glob_foreach( + '/sys/class/block/' . basename($device) . '/holders', + '[^.].*', + sub($name, @) { + die "cannot wipe '$device' - another block device uses it\n" if !$parts{$name}; + }, + ); + for my $path ($device, @$parts) { + die "cannot wipe '$device' - '$path' is mounted\n" if PVE::Diskmanage::is_mounted($path); + next if $path eq $device; + die "cannot wipe '$device' - another block device uses '$path'\n" + if PVE::Diskmanage::has_holder($path); + } + # a guest with the LUN as disk has it open, without being a holder or a storage reference + my $opener = PVE::Diskmanage::device_opener($device, @$parts); + die "cannot wipe '$device' - process $opener has it open\n" if defined($opener); + sysopen(my $fh, $lun->{'by-id'}, O_RDWR) or die "cannot open '$device' - $!\n"; + # the name of a partition can pass to another device as well, since it got listed + my $check_part = sub($part) { + assert_same_device($lun, $fh, $device); + die "cannot wipe '$device' - '$part' is no longer one of its partitions\n" + if !grep { $_ eq $part } lun_partitions($lun)->@*; + }; + for my $part (@$parts) { + $check_part->($part); + run_command(['wipefs', '--all', $part], timeout => 10); + if ($part =~ m{^/dev/dm-}) { + $check_part->($part); + run_command(['dmsetup', 'remove', '--retry', $part], timeout => 10); + } + } + assert_same_device($lun, $fh, $device); + # claim what was checked: while the first descriptor is open, its device number cannot pass to + # another device, so an equal number means the same device + sysopen(my $claim, $lun->{'by-id'}, O_RDWR | O_EXCL) or die "cannot claim '$device' - $!\n"; + die "cannot wipe '$device' - the device behind '$lun->{'by-id'}' changed\n" + if (stat($claim))[6] != (stat($fh))[6]; + close($fh); + my $path = '/proc/' . $$ . '/fd/' . fileno($claim); + PVE::Diskmanage::wipe_blockdev($device, $path); +} + 1; diff --git a/src/test/Makefile b/src/test/Makefile index 0d866d91..279ba1ee 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -76,8 +76,9 @@ test_san_taint: run_san_taint_tests.pl test_san_lun_scan: run_san_lun_scan_tests.pl ./run_san_lun_scan_tests.pl -test_san_api: run_san_api_tests.pl run_san_nvme_tests.pl +test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl ./run_san_api_tests.pl + ./run_san_wipe_tests.pl ./run_san_nvme_tests.pl test_disk_api: run_disk_api_tests.pl diff --git a/src/test/disk_api_test.pm b/src/test/disk_api_test.pm index 6fa2a2e3..1d2f8d33 100644 --- a/src/test/disk_api_test.pm +++ b/src/test/disk_api_test.pm @@ -265,4 +265,50 @@ subtest 'shared volume groups cannot be destroyed' => sub { is_deeply(\@writes, ['data'], 'node-local deletion still works'); }; +subtest 'wipe takes names beyond the old pattern for a LUN only' => sub { + my $san = Test::MockModule->new('PVE::Storage::SAN'); + my $rpc = Test::MockModule->new('PVE::RPCEnvironment'); + $rpc->mock(get => sub { bless {}, 'MultipathTestRPC' }); + my ($lun, @wiped); + $san->mock( + lun_for_device => sub { $lun }, + with_lun_lock => sub { push @wiped, $_[0]->{wwid} }, + lun_paths => sub { [] }, + ); + $diskmanage_module->mock( + verify_blockdev_path => sub { $_[0] }, + is_mounted => sub { }, + has_holder => sub { }, + wipe_blockdev => sub { push @wiped, $_[0] }, + udevadm_trigger => sub { }, + ); + my $api = Test::MockModule->new('PVE::API2::Disks'); + $api->mock(run_command => sub { }); + my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); + $lvm->mock(refresh_cached_pv_volume_groups => sub { }); + my $wipe = sub($name) { + return + eval { PVE::API2::Disks->wipe_disk({ node => 'n1', disk => $name }); 1 } ? '' : $@; + }; + + my @other = qw( + /dev/dm-7 /dev/mapper/3600a-part1 /dev/disk/by-id/dm-uuid-part1-mpath-3600a + /dev/pve/vm-100-disk-0 + ); + for my $name (@other) { + like($wipe->($name), qr/only accepted for a SAN LUN/, "$name is refused"); + } + is($wipe->('/dev/sdb1'), '', 'a partition by its kernel name is wiped'); + $lun = { wwid => '3600a' }; + is( + $wipe->('/dev/disk/by-id/dm-uuid-mpath-3600a'), + '', + 'a LUN by the name of its map is wiped', + ); + is_deeply(\@wiped, ['/dev/sdb1', '3600a'], 'nothing else got wiped'); + + my @mocked = qw(verify_blockdev_path is_mounted has_holder wipe_blockdev udevadm_trigger); + $diskmanage_module->unmock($_) for @mocked; +}; + done_testing(); diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl index 7b963211..59a009c9 100755 --- a/src/test/run_san_nvme_tests.pl +++ b/src/test/run_san_nvme_tests.pl @@ -31,6 +31,7 @@ my $mp = Test::MockModule->new('PVE::Multipath'); my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); my $storage = Test::MockModule->new('PVE::Storage'); my $cluster = Test::MockModule->new('PVE::Cluster'); +my $tools = Test::MockModule->new('PVE::Tools'); my $cfg = { ids => {} }; my @order; $storage->mock(config => sub { $cfg }); @@ -45,6 +46,7 @@ $mp->mock(device_map => sub { undef }); $san->mock(abs_path => sub { $device }); $san->mock(scan_luns => sub { [$lun] }); $san->mock(lun_partitions => sub { [] }); +$san->mock(dir_glob_foreach => sub { }); $san->mock(assert_same_device => sub { ok(defined(fileno($_[1])), 'held descriptor checked') }); $lvm->mock(lun_volume_groups => sub { push @order, 'check'; [] }); $cluster->mock( @@ -79,6 +81,20 @@ for my $base (qw(fabric nvme-fabric)) { like($@, qr/base volume/, "native base reference on storage '$base' refuses"); } $cfg->{ids} = {}; +@order = @opens = (); +$tools->mock(file_get_contents => sub { 1024 }); +$disk->mock( + run_command => sub { + my ($cmd, %opts) = @_; + push @order, $cmd->[0]; + my ($path) = map { /^of=(.*)$/ ? $1 : m{^(/proc/\d+/fd/\d+)$} ? $1 : () } @$cmd; + ok(defined($path) && -e $path, 'native wipe writes through the held descriptor'); + }, +); +PVE::Storage::SAN::with_lun_lock($lun, sub { PVE::Storage::SAN::wipe_lun($lun) }); +is_deeply(\@order, [qw(lock check wipefs dd unlock)], 'native wipe sequencing'); +is_deeply(\@opens, [$link, $link], 'native wipe opens its by-id name'); + @order = @opens = (); $lvm->mock(file_read_firstline => sub { $_[0] =~ m{/size$} ? 1024 : undef }); $lvm->mock(lvm_pv_info => sub { undef }); diff --git a/src/test/run_san_wipe_tests.pl b/src/test/run_san_wipe_tests.pl new file mode 100755 index 00000000..5828e20a --- /dev/null +++ b/src/test/run_san_wipe_tests.pl @@ -0,0 +1,271 @@ +#!/usr/bin/perl + +use strict; +use warnings; + +our (@open_modes, $claim_other); + +BEGIN { + *CORE::GLOBAL::sysopen = sub (*$$;$) { + push @open_modes, $_[2]; + # lets a name mean another device by the time it is opened exclusively + my $exclusive = $_[2] & Fcntl::O_EXCL(); + my $name = $exclusive && defined($claim_other) ? $claim_other : $_[1]; + return @_ == 4 + ? CORE::sysopen($_[0], $name, $_[2], $_[3]) + : CORE::sysopen($_[0], $name, $_[2]); + }; +} + +use lib qw(..); +use Cwd qw(abs_path); +use Fcntl qw(O_RDWR O_EXCL); +use File::Temp qw(tempfile); +use Test::MockModule; +use Test::More; +use PVE::Storage::SAN; + +{ + my ($fh, $name) = tempfile(UNLINK => 1); + my ($tmp, $read_name) = tempfile(UNLINK => 1); + open(my $read_fh, '<', $read_name) or die "cannot open '$read_name' - $!\n"; + close($tmp); + my $pid = fork() // die "fork failed - $!\n"; + if (!$pid) { + sleep(60); + exit(0); + } + like(PVE::Diskmanage::device_opener($name), qr/^$pid \(/, 'a process with the file open'); + my $all = PVE::Diskmanage::device_openers([$name, $read_name, "$name-none"]); + is_deeply( + [sort keys %$all], + [sort map { abs_path($_) } $name, $read_name], + 'each open file has a process, by its resolved path', + ); + my $writers = PVE::Diskmanage::device_openers([$name, $read_name], writable => 1); + is_deeply([keys %$writers], [abs_path($name)], 'a file open for reading can be left out'); + kill('TERM', $pid); + waitpid($pid, 0); + is(PVE::Diskmanage::device_opener($name), undef, 'the current process does not count'); +} + +my ($file, $name) = tempfile('san-wipe-XXXX', DIR => '.', UNLINK => 1); +close($file); +my $lun = { devpath => $name, 'by-id' => $name, wwid => '3600a' }; +my $san = Test::MockModule->new('PVE::Storage::SAN'); +my $disk = Test::MockModule->new('PVE::Diskmanage'); +my $tools = Test::MockModule->new('PVE::Tools'); +my (@commands, @parts, @parttypes); +my $sectors = 1024 * 1024; +my $run = sub { + my ($cmd, %opts) = @_; + push @commands, [$cmd, \%opts]; + if ($cmd->[0] eq 'dd' || ($cmd->[0] eq 'wipefs' && grep { $_ eq '--force' } @$cmd)) { + my ($path) = map { /^of=(.*)$/ ? $1 : m{^(/proc/\d+/fd/\d+)$} ? $1 : () } @$cmd; + ok(defined($path) && -e $path, 'whole-device command uses a live descriptor'); + is((stat($path))[1], (stat($name))[1], 'descriptor still names the opened device'); + ok($open_modes[-1] & O_EXCL, 'exclusive descriptor held for whole-device commands'); + } +}; +$san->mock(run_command => $run); +my $referenced = 0; +my $storage = Test::MockModule->new('PVE::Storage'); +$storage->mock(config => sub { { ids => {} } }); +$san->mock(assert_lun_unreferenced => sub { die "referenced\n" if $referenced }); +my $path_used = 0; +$san->mock(assert_lun_paths_unused => sub { die "path in use\n" if $path_used }); +$san->mock(lun_partitions => sub { [@parts] }); +$san->mock(dir_glob_foreach => sub { }); +my $canonical = $name; +$disk->mock(verify_blockdev_path => sub { $canonical }); +$disk->mock(is_mounted => sub { 0 }); +$disk->mock(has_holder => sub { 0 }); +my $opener; +$disk->mock(device_opener => sub { $opener }); +$disk->mock(run_command => $run); +$disk->mock(dir_glob_foreach => sub { die "partition enumeration after exclusive open\n" }); +$disk->mock(is_partition => sub { die "partition check after exclusive open\n" }); +$tools->mock(file_get_contents => sub { $sectors }); + +$referenced = 1; +@commands = @open_modes = (); +eval { PVE::Storage::SAN::wipe_lun($lun) }; +like($@, qr/referenced/, 'a referenced LUN is refused'); +is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written'); +$referenced = 0; +$path_used = 1; +eval { PVE::Storage::SAN::wipe_lun($lun) }; +like($@, qr/path in use/, 'a LUN in use through another path is refused'); +is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written'); +$path_used = 0; +$opener = '4711 (kvm)'; +eval { PVE::Storage::SAN::wipe_lun($lun) }; +like($@, qr/process 4711 \(kvm\) has it open/, 'a LUN that a process has open is refused'); +is_deeply([\@commands, \@open_modes], [[], []], 'before the device is opened or written'); +$opener = undef; + +for my $count (0, 2, 10) { + @parts = map { "/dev/dm-$_" } 1 .. $count; + @commands = @open_modes = (); + PVE::Storage::SAN::wipe_lun($lun); + is_deeply(\@open_modes, [O_RDWR, O_RDWR | O_EXCL], 'one exclusive whole-device open'); + my $fdpath = $commands[-2]->[0]->[-1]; + is_deeply( + $commands[-2]->[0], + ['wipefs', '--all', '--force', $fdpath], + 'force avoids a second exclusive claim', + ); + is($commands[-1]->[1]->{timeout}, 60, 'dd has 60 seconds for fdatasync'); + ok(grep({ $_ eq 'count=209715200' } $commands[-1]->[0]->@*), 'wipe capped at 200 MiB'); + + for my $i (0 .. $#parts) { + is_deeply( + $commands[2 * $i]->[0], + ['wipefs', '--all', $parts[$i]], + 'partition signatures first', + ); + is_deeply( + $commands[2 * $i + 1]->[0], + ['dmsetup', 'remove', '--retry', $parts[$i]], + 'remove partition map with retry', + ); + } + is($_->[1]->{timeout}, 10, 'partition command has a fixed timeout') + for @commands[0 .. $#commands - 2]; + ok(!-e $fdpath, 'descriptor released on return'); +} + +@parts = (); +@commands = (); +$sectors = 1; +PVE::Storage::SAN::wipe_lun($lun); +ok(grep({ $_ eq 'count=512' } $commands[-1]->[0]->@*), 'small device uses a byte count'); + +@parts = ('/dev/dm-1'); +@commands = (); +eval { PVE::Storage::SAN::wipe_lun({ %$lun, devpath => '/dev/null' }) }; +like($@, qr{the device behind '/dev/null' changed}, 'a path that means another device refuses'); +is_deeply(\@commands, [], 'before a partition or the device is wiped'); +my $listed = 0; +$san->mock(lun_partitions => sub { $listed++ ? [] : ['/dev/dm-1'] }); +eval { PVE::Storage::SAN::wipe_lun($lun) }; +like($@, qr{'/dev/dm-1' is no longer one of its partitions}, 'a partition name that passed on'); +is_deeply(\@commands, [], 'is not wiped'); +$san->mock(lun_partitions => sub { [@parts] }); +@parts = (); +my $err = do { + local $claim_other = '/dev/zero'; + eval { PVE::Storage::SAN::wipe_lun($lun) }; + $@; +}; +like( + $err, + qr{the device behind '\Q$name\E' changed}, + 'a by-id name that passed on before the claim', +); +is_deeply(\@commands, [], 'is not wiped either'); +$canonical = '/dev/null'; +eval { PVE::Storage::SAN::wipe_lun($lun) }; +like($@, qr{the device behind '/dev/null' changed}, 'a resolved name that no longer means the LUN'); +is_deeply(\@commands, [], 'refuses before a write as well'); +$canonical = $name; + +$disk->mock(dir_glob_foreach => sub { }); +$disk->mock(is_partition => sub { 1 }); +$disk->mock(change_parttype => sub { push @parttypes, [@_] }); +$disk->mock(run_command => sub { my ($cmd, %opts) = @_; push @commands, [$cmd, \%opts] }); +@commands = (); +$sectors = 204800; +PVE::Diskmanage::wipe_blockdev('/dev/sda1'); +is_deeply( + [map { $_->[0] } @commands], + [ + ['wipefs', '--all', '/dev/sda1'], + ['dd', 'if=/dev/zero', 'of=/dev/sda1', 'bs=1M', 'conv=fdatasync', 'count=100'], + ], + 'local disk command lines unchanged', +); +ok(!grep({ exists($_->[1]->{timeout}) } @commands), 'local disk timeout behavior unchanged'); +is_deeply(\@parttypes, [['/dev/sda1', '8300']], 'local partition type still updated'); + +my $identity = Test::MockModule->new('PVE::Multipath::Identity'); +$identity->mock( + collect => sub { + { + paths => { + sdb => { wwid => '3600a' }, + sdc => { wwid => '3600b' }, + sdd => { wwid => '3600a' }, + }, + }; + }, +); +is_deeply( + PVE::Storage::SAN::lun_paths($lun), + ['/dev/sdb', '/dev/sdd'], + 'all paths of the LUN are refreshed after a wipe', +); + +# the other paths of a LUN: sdd beside sdb without a map, both below the map dm-3 +$san->unmock('assert_lun_paths_unused'); +my ($mounted, %holders, $writers, $asked) = (undef, ()); +$disk->mock(is_mounted => sub { $_[0] eq '/dev/sdd' ? $mounted : undef }); +$san->mock( + dir_glob_foreach => sub { + my ($dir, $regex, $code) = @_; + $code->($_) for ($holders{$dir} // [])->@*; + }, +); +$disk->mock( + device_openers => sub { + my ($devices, %param) = @_; + $asked = [$devices, $param{writable}]; + return $writers // {}; + }, +); +$san->mock(abs_path => sub { $_[0] }); +my $path_lun = { %$lun, devpath => '/dev/sdb' }; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +is($@, '', 'unused paths pass'); +is_deeply($asked, [['/dev/sdd'], 1], 'writers on the other paths, multipathd only reads them'); +$mounted = '/dev/sdd1'; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +like($@, qr{the LUN is mounted through '/dev/sdd1'}, 'a mounted path or partition refuses'); +$mounted = undef; +$holders{'/sys/class/block/sdd/holders'} = ['dm-3']; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +like($@, qr{another block device uses the LUN through '/dev/sdd'}, 'a held path refuses'); +$holders{'/sys/class/block/sdd/holders'} = []; +$holders{'/sys/class/block/sdd'} = ['sdd1']; +$holders{'/sys/class/block/sdd1/holders'} = ['dm-7']; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +like($@, qr{uses the LUN through '/dev/sdd1'}, 'so does a held partition of a path'); +$holders{'/sys/class/block/sdd1/holders'} = []; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +is_deeply($asked->[0], ['/dev/sdd', '/dev/sdd1'], 'writers on the partitions of a path count'); +$holders{'/sys/class/block/sdb/holders'} = ['dm-3']; +eval { PVE::Storage::SAN::assert_lun_paths_unused({ %$lun, devpath => '/dev/dm-3' }) }; +is($@, '', 'the map of the LUN holds its paths'); +%holders = (); +$writers = { '/dev/sdd' => { pid => 4711, command => 'kvm' } }; +eval { PVE::Storage::SAN::assert_lun_paths_unused($path_lun) }; +like($@, qr{process 4711 \(kvm\) has the LUN open through '/dev/sdd'}, + 'a writer on a path refuses'); + +# without multipathd, udev can know a member of the map by another identity than the map +my $map_lun = { %$lun, devpath => '/dev/dm-3' }; +$holders{'/sys/class/block/dm-3/slaves'} = ['sdc']; +is_deeply( + PVE::Storage::SAN::lun_paths($map_lun), + ['/dev/sdb', '/dev/sdc', '/dev/sdd'], + 'the members of the map are paths of the LUN, whatever their identity', +); +$writers = { '/dev/sdc' => { pid => 4712, command => 'kvm' } }; +eval { PVE::Storage::SAN::assert_lun_paths_unused($map_lun) }; +like( + $@, + qr{process 4712 \(kvm\) has the LUN open through '/dev/sdc'}, + 'a writer on such a member refuses', +); + +done_testing();