From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 3B42B1FF0AF for ; Fri, 06 Nov 2026 01:58:26 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D468421DE8; Fri, 06 Nov 2026 01:51:20 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v3 storage 37/76] api: scan: san-luns: report the multipath state of each LUN Date: Fri, 6 Nov 2026 00:31:12 +0100 Message-ID: <20261105233336.2178064-38-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261105233336.2178064-1-t.lamprecht@proxmox.com> References: <20261105233336.2178064-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793926179885 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.300 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YEQKKUSDJSRMWWMKGZXXGCV4Q7ZY2WIB X-Message-ID-Hash: YEQKKUSDJSRMWWMKGZXXGCV4Q7ZY2WIB X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Clients need a LUN's WWID and local management state to offer multipath configuration actions. For paths without maps, use the WWID from multipathd so it matches storage-derived identities. On nodes without multipath-tools, take NAA identities from the udev property that multipathd reads, checked against the VPD page of the device, so plain iSCSI LUNs keep their identity there. Distinguish maps managed by a storage or explicit entry, maps retained while in use, unmanaged maps, pending maps, and LUNs not allow-listed. Include the associated storage, iSCSI target and LUN number, and expected path count where known. Keep unresolved SCSI identities distinct from devices without identity evidence. Changed or disagreeing paths and an unavailable daemon must not be mistaken for a LUN the node cannot see. Signed-off-by: Thomas Lamprecht --- src/PVE/API2/Disks.pm | 2 +- src/PVE/API2/Storage/Scan.pm | 55 +++++- src/PVE/CLI/pvesm.pm | 4 +- src/PVE/Multipath.pm | 6 + src/PVE/Multipath/Identity.pm | 38 +++- src/PVE/Storage/SANScan.pm | 106 +++++++++-- src/test/Makefile | 5 +- src/test/run_lun_identity_tests.pl | 45 +++++ src/test/run_san_api_tests.pl | 47 +---- src/test/run_san_lun_scan_tests.pl | 295 +++++++++++++++++++++++++++++ 10 files changed, 529 insertions(+), 74 deletions(-) create mode 100755 src/test/run_san_lun_scan_tests.pl diff --git a/src/PVE/API2/Disks.pm b/src/PVE/API2/Disks.pm index dd3524b5..58a89c67 100644 --- a/src/PVE/API2/Disks.pm +++ b/src/PVE/API2/Disks.pm @@ -101,7 +101,7 @@ __PACKAGE__->register_method({ 'include-remote' => { description => "Also include devices attached through a remote storage" . " transport (for example FC, iSCSI, or NVMe over TCP), which are" - . " excluded by default.", + . " excluded by default. See scan/san-luns for paths and multipath state.", type => 'boolean', optional => 1, default => 0, diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm index 73632a1f..93c78ac2 100644 --- a/src/PVE/API2/Storage/Scan.pm +++ b/src/PVE/API2/Storage/Scan.pm @@ -380,10 +380,9 @@ __PACKAGE__->register_method({ name => 'sanlunscan', path => 'san-luns', method => 'GET', - description => "List block devices (disks and multipath devices) together with their" - . " usage, for example as candidates for a SAN backed LVM volume group. For multipath" - . " mapped devices and NVMe namespaces handled by native NVMe multipath, the state" - . " of the member paths is included.", + description => "List the block devices of the node with their SAN details: paths, multipath" + . " state, target, and usage. Local disks are included; the web UI filters them by" + . " transport.", protected => 1, proxyto => "node", permissions => { @@ -438,6 +437,13 @@ __PACKAGE__->register_method({ type => 'integer', optional => 1, }, + 'paths-expected' => { + description => "The number of paths the LUN should have according to the" + . " storage configuration, one per portal for iSCSI; absent when it is" + . " not known.", + type => 'integer', + optional => 1, + }, 'path-faults' => { description => "The number of path failures of a multipath mapped" . " device since its creation, as reported by multipathd.", @@ -514,6 +520,47 @@ __PACKAGE__->register_method({ type => 'string', optional => 1, }, + wwid => { + description => "The WWID of the LUN as used by dm-multipath, for SCSI" + . " devices with a designator, and for multipath mapped devices.", + type => 'string', + optional => 1, + }, + 'wwid-unknown' => { + description => "The SCSI path has no trusted LUN identity because identity" + . " discovery failed, disagreed, or the path changed during the scan.", + type => 'boolean', + optional => 1, + }, + multipath => { + description => "Multipath management on this node: 'storage' for a map" + . " allow-listed by a multipath-enabled storage; 'manual' for a map" + . " allow-listed by an explicit cluster-wide entry; 'kept' for a map no" + . " longer configured but still allow-listed while in use; 'unmanaged'" + . " for a map assembled outside the managed configuration; 'pending' for" + . " a path of an allow-listed LUN whose map is not yet assembled; or" + . " 'none' for a path of a LUN that is not allow-listed.", + type => 'string', + enum => ['none', 'pending', 'storage', 'manual', 'kept', 'unmanaged'], + optional => 1, + }, + 'multipath-storage' => { + description => + "The multipath-enabled storage the LUN belongs to, whether or" + . " not it also has an explicit multipath entry.", + type => 'string', + optional => 1, + }, + 'iscsi-target' => { + description => "The IQN of the iSCSI target the LUN is attached through.", + type => 'string', + optional => 1, + }, + 'iscsi-lun' => { + description => "The number of the LUN on its iSCSI target.", + type => 'integer', + optional => 1, + }, detail => { description => "Details about other usage, for example the file system type.", diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm index 67419da2..dfe6c25b 100755 --- a/src/PVE/CLI/pvesm.pm +++ b/src/PVE/CLI/pvesm.pm @@ -743,8 +743,8 @@ our $cmddef = { { node => $nodename }, sub { my ($data, $schema, $options) = @_; - my $columns = - [qw(devpath size transport paths-active usage vgname used-by-storage)]; + my $columns = [qw(devpath size transport paths-active paths-expected multipath + usage vgname used-by-storage wwid)]; PVE::CLIFormatter::print_api_result($data, $schema, $columns, $options); }, $PVE::RESTHandler::standard_output_options, diff --git a/src/PVE/Multipath.pm b/src/PVE/Multipath.pm index acca5a11..81f6d867 100644 --- a/src/PVE/Multipath.pm +++ b/src/PVE/Multipath.pm @@ -36,6 +36,12 @@ sub assert_supported() { return 1; } +# Neither tool is installed, so there is no daemon to ask for identities. A partial installation +# does not count, as a daemon with its own identity settings may still run. +sub is_absent() { + return !-e $MULTIPATH && !-e $MULTIPATHD; +} + sub is_running() { return 0 if !is_supported(); diff --git a/src/PVE/Multipath/Identity.pm b/src/PVE/Multipath/Identity.pm index 3281c6d4..5fa6542d 100644 --- a/src/PVE/Multipath/Identity.pm +++ b/src/PVE/Multipath/Identity.pm @@ -1,8 +1,9 @@ package PVE::Multipath::Identity; -# Use multipathd's configured identity, not a guessed sysfs identifier. Maps pin their paths; -# unmapped names can be reused, so daemon entries must match dev_t, H:C:T:L and target, plus -# the cached VPD NAA designator when applicable. Only iSCSI has grouping evidence before resolution. +# Use multipathd's configured identity, not a guessed sysfs identifier; on nodes without +# multipath-tools, NAA identities from the udev property it reads. Maps pin their paths; unmapped +# names can be reused, so daemon entries must match dev_t, H:C:T:L and target, plus the cached VPD +# NAA designator when applicable. Only iSCSI has grouping evidence before resolution. use v5.36; @@ -14,6 +15,7 @@ use PVE::Multipath::Config; use PVE::Tools qw(dir_glob_foreach dir_glob_regex file_get_contents file_read_firstline); our $SYSFS = '/sys'; +our $UDEV_DATA = '/run/udev/data'; my sub sysfs_line($path) { return file_read_firstline("$SYSFS/$path"); @@ -189,6 +191,28 @@ my sub scsi_disks($sessions, $targets, $all) { return $res; } +# Without multipath-tools there is no daemon to ask. Take the udev property that multipathd reads +# the WWID of a SCSI path from by default, in the form of its path list. Unlike a daemon entry, a +# udev record does not name the address of its device and can outlive it, so only an NAA identity +# counts, which line_identifies() checks against the VPD page of the current device. Paths with +# other identities count as not listed. +my sub udev_path_identities($disks) { + my $res = {}; + for my $name (sort keys %$disks) { + my $disk = $disks->{$name}; + my $data = eval { file_get_contents("$UDEV_DATA/b$disk->{devt}") } // next; + my ($wwid) = $data =~ m/^E:ID_SERIAL=(3(?:[0-9a-fA-F]{16}){1,2})$/m or next; + $res->{$name} = { + devt => $disk->{devt}, + hctl => $disk->{hctl}, + init => 'ok', + target => $disk->{target} // '[undef]', + wwid => $wwid, + }; + } + return $res; +} + sub collect(%param) { my $sessions = iscsi_sessions(); my $disks = scsi_disks($sessions, $param{targets}, $param{all}); @@ -196,7 +220,13 @@ sub collect(%param) { my ($daemon, $error); if (grep { !defined($_->{map}) } values %$disks) { $daemon = eval { PVE::Multipath::query_path_identities() }; - chomp($error = $@) if $@; + if (my $err = $@) { + if (PVE::Multipath::is_absent()) { + $daemon = udev_path_identities($disks); + } else { + chomp($error = $err); + } + } } my $naa_ok = sub($disk, $hex) { my $page = eval { file_get_contents("$SYSFS/$disk->{dir}/vpd_pg83") }; diff --git a/src/PVE/Storage/SANScan.pm b/src/PVE/Storage/SANScan.pm index ee311cb9..415efdba 100644 --- a/src/PVE/Storage/SANScan.pm +++ b/src/PVE/Storage/SANScan.pm @@ -4,11 +4,16 @@ use v5.36; use PVE::Diskmanage; use PVE::Multipath; +use PVE::Multipath::ClusterConfig; +use PVE::Multipath::Config; +use PVE::Multipath::Identity; use PVE::Storage; use PVE::Storage::ISCSIPlugin; use PVE::Storage::LVMPlugin; my sub collect_inventory($cfg, $node) { + # Collect identities before disks so path_wwid() rejects names reused during enumeration. + my $ids = PVE::Multipath::Identity::collect(all => 1); my $disks = PVE::Diskmanage::get_disks(undef, 1, 0); my $multipath = PVE::Diskmanage::get_multipath_disks(); @@ -61,31 +66,74 @@ my sub collect_inventory($cfg, $node) { } } - # map kernel device name to the storage using its iSCSI target + # map kernel device name to its iSCSI session and to the storage using that target + my $device_map = PVE::Storage::ISCSIPlugin::iscsi_session_device_map(); my $iscsi_used_by = {}; - if (scalar(keys %$iscsi_target_used_by)) { - my $device_map = PVE::Storage::ISCSIPlugin::iscsi_session_device_map(); - for my $bdev (sort keys %$device_map) { - my $sid = $iscsi_target_used_by->{ $device_map->{$bdev}->{target} } // next; - $iscsi_used_by->{$bdev} = $sid; + for my $bdev (sort keys %$device_map) { + my $sid = $iscsi_target_used_by->{ $device_map->{$bdev}->{target} } // next; + $iscsi_used_by->{$bdev} = $sid; + } + + # A pre-existing allow-list entry can be managed for a storage without being owned by PVE. + my $mp_current = { map { $_ => 1 } PVE::Multipath::list_etc_multipath_wwids()->@* }; + my $mp_cfg = eval { PVE::Multipath::ClusterConfig::read_config() }; + warn $@ if $@; + my $mp_explicit = { map { $_ => 1 } + $mp_cfg ? PVE::Multipath::Config::wwids_for_node($mp_cfg, $node)->@* : () }; + my $mp_owned = eval { PVE::Multipath::managed_wwid_record() } // {}; + warn $@ if $@; + my ($mp_storage, $mp_expected) = ({}, {}); + my $derived = PVE::Storage::derived_multipath_wwids($cfg, $ids); + + for my $sid (sort keys %$derived) { + my $luns = $derived->{$sid}->{wwids}; + for my $wwid (keys %$luns) { + $mp_storage->{$wwid} //= $sid; + my $expected = $luns->{$wwid}->{'paths-expected'}; + $mp_expected->{$wwid} = $expected + if defined($expected) && $expected > ($mp_expected->{$wwid} // 0); } } return { + ids => $ids, disks => $disks, multipath => $multipath, mp_status => $mp_status, nvme_status => $nvme_status, + device_map => $device_map, vg_used_by => $vg_used_by, iscsi_target_used_by => $iscsi_target_used_by, + mp_current => $mp_current, + mp_explicit => $mp_explicit, + mp_owned => $mp_owned, + mp_storage => $mp_storage, + mp_expected => $mp_expected, find_vg => $find_vg, iscsi_used_by => $iscsi_used_by, }; } my sub build_lun_rows($cfg, $inventory) { - my ($disks, $multipath, $mp_status, $nvme_status, $vg_used_by, $find_vg, $iscsi_used_by) = - $inventory->@{ qw(disks multipath mp_status nvme_status vg_used_by find_vg - iscsi_used_by) }; + my ( + $ids, + $disks, + $multipath, + $mp_status, + $nvme_status, + $device_map, + $vg_used_by, + $mp_current, + $mp_explicit, + $mp_owned, + $mp_storage, + $mp_expected, + $find_vg, + $iscsi_used_by, + ) + = $inventory->@{ + qw(ids disks multipath mp_status nvme_status device_map + vg_used_by mp_current mp_explicit mp_owned mp_storage mp_expected find_vg iscsi_used_by) + }; my $records = []; my $add_entry = sub($disk) { @@ -126,14 +174,38 @@ my sub build_lun_rows($cfg, $inventory) { $entry->{usage} = 'other'; $entry->{detail} = $disk->{used}; } - if (!defined($entry->{'used-by-storage'})) { - my @members = - map { $_ =~ s|^/dev/||r } ($disk->{slaves} // [$disk->{devpath}])->@*; - for my $member (@members) { - if (defined(my $sid = $iscsi_used_by->{$member})) { - $entry->{'used-by-storage'} = $sid; - last; - } + my @members = map { $_ =~ s|^/dev/||r } ($disk->{slaves} // [$disk->{devpath}])->@*; + for my $member (@members) { + my $session_device = $device_map->{$member} // next; + $entry->{'iscsi-target'} //= $session_device->{target}; + $entry->{'iscsi-lun'} //= $session_device->{lun}; + $entry->{'used-by-storage'} //= $iscsi_used_by->{$member} + if defined($iscsi_used_by->{$member}); + } + my $wwid; + if (defined($disk->{slaves})) { + $wwid = $disk->{wwn}; + } elsif ($disk->{devpath} =~ m|^/dev/(sd[a-z]+)$|) { + my $name = $1; + $wwid = PVE::Multipath::Identity::path_wwid($ids, $name); + $entry->{'wwid-unknown'} = 1 + if !defined($wwid) && $ids->{paths}->{$name}->{unresolved}; + } + if (defined($wwid)) { + $entry->{wwid} = $wwid; + $entry->{'multipath-storage'} = $mp_storage->{$wwid} if $mp_storage->{$wwid}; + $entry->{'paths-expected'} = $mp_expected->{$wwid} + if defined($mp_expected->{$wwid}); + if (!defined($disk->{slaves})) { + $entry->{multipath} = $mp_current->{$wwid} ? 'pending' : 'none'; + } elsif ($mp_explicit->{$wwid}) { + $entry->{multipath} = 'manual'; + } elsif ($mp_storage->{$wwid}) { + $entry->{multipath} = 'storage'; + } elsif ($mp_owned->{$wwid}) { + $entry->{multipath} = 'kept'; + } else { + $entry->{multipath} = 'unmanaged'; } } push $records->@*, { row => $entry, disk => $disk }; diff --git a/src/test/Makefile b/src/test/Makefile index 9ecd59d4..91c685e7 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -1,6 +1,6 @@ all: test -test: test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_lvm_base_activation test_lvm_status test_status_command test_disklist test_disk_api test_bwlimit test_plugin test_ovf test_volume_access test_multipath test_iscsi test_lun_identity test_multipath_api test_san_api test_san_libraries test_san_taint +test: test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_lvm_base_activation test_lvm_status test_status_command test_disklist test_disk_api test_bwlimit test_plugin test_ovf test_volume_access test_multipath test_iscsi test_lun_identity test_multipath_api test_san_lun_scan test_san_api test_san_libraries test_san_taint test_zfspoolplugin: run_test_zfspoolplugin.pl ./run_test_zfspoolplugin.pl @@ -70,6 +70,9 @@ test_lun_identity: run_lun_identity_tests.pl test_multipath_api: run_multipath_api_tests.pl ./run_multipath_api_tests.pl +test_san_lun_scan: run_san_lun_scan_tests.pl + ./run_san_lun_scan_tests.pl + test_san_api: run_san_api_tests.pl ./run_san_api_tests.pl diff --git a/src/test/run_lun_identity_tests.pl b/src/test/run_lun_identity_tests.pl index 9f89c923..c5363e46 100755 --- a/src/test/run_lun_identity_tests.pl +++ b/src/test/run_lun_identity_tests.pl @@ -348,6 +348,9 @@ put('bus/scsi/devices/7:0:0:9/type', "1\n"); my ($queries, $reply) = (0); my $multipath = Test::MockModule->new('PVE::Multipath'); $multipath->mock( + # a failed query means a broken daemon, not a node without multipath-tools, whatever the build + # host has installed + is_absent => sub { 0 }, query_path_identities => sub { $queries++; die $reply if !ref($reply); @@ -400,6 +403,48 @@ is_deeply( { sdb => '3600a09803832355367242f747a716b7a' }, 'collect: and the LUNs with a map still resolve, through the holder link of their paths', ); +{ + # a node without multipath-tools: the udev property that multipathd would read + local $PVE::Multipath::Identity::UDEV_DATA = "$sys/udev"; + $multipath->mock(is_absent => sub { 1 }); + $reply = "multipath-tools is not installed\n"; + put( + 'udev/b8:32', + "S:disk/by-id/scsi-3600a09803832355367242f747a716b7a\n" + . "E:ID_SERIAL=3600a09803832355367242f747a716b7a\n", + ); + $ids = PVE::Multipath::Identity::collect(targets => { $T => 1 }); + is($ids->{error}, undef, 'collect: no multipath-tools is not a failed query'); + is_deeply( + { map { $_->{lun} => $_->{wwid} } PVE::Multipath::Identity::target_luns($ids, $T)->@* }, + { 0 => '3600a09803832355367242f747a716b7a', 1 => '3600a09803832355367242f747a716b7a' }, + 'collect: without multipath-tools, a path udev and the VPD page of the device name alike', + ); + put('udev/b8:32', "E:ID_SERIAL=3600a09803832355367242f747a716c30\n"); + $ids = PVE::Multipath::Identity::collect(targets => { $T => 1 }); + is($ids->{paths}->{sdc}->{unresolved}, 1, 'collect: but not one that the VPD page contradicts'); + unlink("$sys/udev/b8:32") or die "$!\n"; + $ids = PVE::Multipath::Identity::collect(targets => { $T => 1 }); + ok( + !$ids->{paths}->{sdc}->{unresolved}, + 'collect: a path without udev identity is not unresolved', + ); + is(PVE::Multipath::Identity::path_wwid($ids, 'sdc'), undef, 'collect: and has no identity'); + # a serial without an NAA designator that the VPD page could confirm, maybe of the device that + # had the number before + put('udev/b8:32', "E:ID_SERIAL=20011223344556677\n"); + $ids = PVE::Multipath::Identity::collect(targets => { $T => 1 }); + ok( + !$ids->{paths}->{sdc}->{unresolved}, + 'collect: a path with another kind of serial is not listed', + ); + is( + PVE::Multipath::Identity::path_wwid($ids, 'sdc'), + undef, + 'collect: and has no identity either', + ); + $multipath->mock(is_absent => sub { 0 }); +} # a later device under the name of a path: another LUN, the same dev_t $reply = { diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl index 249185eb..c6a3bb54 100755 --- a/src/test/run_san_api_tests.pl +++ b/src/test/run_san_api_tests.pl @@ -165,7 +165,8 @@ subtest 'SAN CLI columns leave machine output complete' => sub { is($_[1], $schema, 'complete schema forwarded'); is_deeply( $_[2], - [qw(devpath size transport paths-active usage vgname used-by-storage)], + [qw(devpath size transport paths-active paths-expected multipath + usage vgname used-by-storage wwid)], 'concise text columns', ); is($_[3]->{'output-format'}, 'json', 'format option unchanged'); @@ -176,48 +177,4 @@ subtest 'SAN CLI columns leave machine output complete' => sub { is($schema->{items}->{properties}->{size}->{renderer}, 'bytes', 'size renderer'); }; -subtest 'SAN LUN path counts' => sub { - my $disk = Test::MockModule->new('PVE::Diskmanage'); - my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); - my $storage = Test::MockModule->new('PVE::Storage'); - my $disks = - { map { ("/dev/$_" => { devpath => "/dev/$_", size => 1024, multipath => 'm0' }) } - qw(sdb sdc) }; - $disk->mock( - get_disks => sub { $disks }, - get_multipath_disks => sub { - return { - '/dev/mapper/m0' => { - devpath => '/dev/mapper/m0', - size => 1024, - slaves => ['/dev/sdb', '/dev/sdc'], - paths => 2, - }, - }; - }, - ); - $lvm->mock(lvm_vgs => sub { {} }); - $storage->mock(config => sub { { ids => {} } }); - # native NVMe multipath: two controllers, one of them usable - $disks->{'/dev/nvme0n1'} = { devpath => '/dev/nvme0n1', size => 1024 }; - my $paths = [{ device => '/dev/nvme0' }, { device => '/dev/nvme1' }]; - $disk->mock( - get_nvme_path_status => sub { { nvme0n1 => { active => 1, paths => $paths } } }); - my $multipath = Test::MockModule->new('PVE::Multipath'); - $multipath->mock(path_status_by_wwid => sub { {} }); - my $declared = - PVE::API2::Storage::Scan->map_method_by_name('sanlunscan')->{returns}->{items} - ->{properties}; - my $rows = PVE::API2::Storage::Scan->sanlunscan({ node => 'test' }); - my $by_dev = { map { ($_->{devpath} => $_) } @$rows }; - is($by_dev->{'/dev/mapper/m0'}->{'paths-total'}, 2, 'map path count'); - # native NVMe multipath - is($by_dev->{'/dev/nvme0n1'}->{'paths-total'}, 2, 'namespace path count'); - is($by_dev->{'/dev/nvme0n1'}->{'paths-active'}, 1, 'namespace usable paths'); - for my $row (@$rows) { - my @undeclared = grep { !$declared->{$_} } sort keys %$row; - is_deeply(\@undeclared, [], "$row->{devpath}: every field declared"); - } -}; - done_testing(); diff --git a/src/test/run_san_lun_scan_tests.pl b/src/test/run_san_lun_scan_tests.pl new file mode 100755 index 00000000..c9f130ff --- /dev/null +++ b/src/test/run_san_lun_scan_tests.pl @@ -0,0 +1,295 @@ +#!/usr/bin/perl + +use strict; +use warnings; + +use lib '..'; +use File::Path; +use File::Temp; +use Test::MockModule; +use Test::More; + +use PVE::API2::Storage::Scan; +use PVE::Multipath::Identity; + +my $T = 'iqn.1992-08.com.netapp:sn.1:vs.5'; + +# the devices, with the PV each carries: { name => { disk => get_disks entry, pv => uuid } }, and +# the maps: { wwid => { slaves => [names], pv => uuid } } +my ($devices, $maps, $resolvable, $namespace_ids) = ({}, {}, 1, {}); + +my sub sd { + my ($name, $lun, $pv, %extra) = @_; + return ( + $name => { + disk => { + devpath => "/dev/$name", + size => 1024, + transport => 'iscsi', + wwn => $extra{wwn} // 'unknown', + defined($pv) ? (used => 'LVM') : (), + defined($extra{map}) ? (multipath => $extra{map}) : (), + }, + lun => $lun, + pv => $pv, + map => $extra{map}, + daemon => $extra{daemon}, + }, + ); +} + +# what the scan collects in which order +my @order; +my $diskmanage = Test::MockModule->new('PVE::Diskmanage'); +$diskmanage->mock( + get_disks => sub { + push @order, 'disks'; + return { map { ("/dev/$_" => $devices->{$_}->{disk}) } keys %$devices }; + }, + get_multipath_disks => sub { + return { + map { + my $wwid = $_; + ( + "/dev/mapper/$wwid" => { + devpath => "/dev/mapper/$wwid", + size => 1024, + wwn => $wwid, + slaves => [map { "/dev/$_" } $maps->{$wwid}->{slaves}->@*], + paths => scalar($maps->{$wwid}->{slaves}->@*), + used => 'LVM', + }, + ) + } keys %$maps + }; + }, + get_nvme_path_status => sub { {} }, +); +my $multipath = Test::MockModule->new('PVE::Multipath'); +$multipath->mock( + path_status_by_wwid => sub { {} }, + list_etc_multipath_wwids => sub { [] }, + managed_wwid_record => sub { {} }, +); +my $cluster_config = Test::MockModule->new('PVE::Multipath::ClusterConfig'); +$cluster_config->mock(read_config => sub { { ids => {} } }); +my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); +$lvm->mock(lvm_vgs => sub { {} }); +my $storage = Test::MockModule->new('PVE::Storage'); +# names whose device changed since the collection, and what the storages derive from +my ($reused, $collected, $derived_with) = ({}); +$storage->mock( + config => sub { { ids => {} } }, + derived_multipath_wwids => sub { $derived_with = $_[1]; return {} }, +); +my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin'); +$iscsi->mock( + iscsi_session_device_map => sub { + return { + map { $_ => { target => $T, channel => 0, id => 0, lun => $devices->{$_}->{lun} } } + grep { defined($devices->{$_}->{lun}) } keys %$devices + }; + }, +); +my $identity = Test::MockModule->new('PVE::Multipath::Identity'); +$identity->mock( + path_wwid => sub { + my ($ids, $name) = @_; + if ($reused->{$name}) { + $ids->{paths}->{$name}->{unresolved} = 1; + return undef; + } + my ($lun) = grep { + grep { $_ eq $name } $_->{paths}->@* + } values $ids->{luns}->%*; + return $lun ? $lun->{wwid} : undef; + }, + collect => sub { + push @order, 'identities'; + my $disks = {}; + my $daemon = {}; + for my $name (keys %$devices) { + my $dev = $devices->{$name}; + next if $name !~ m/^sd/; + my $hctl = "7:0:0:" . ($dev->{lun} // 99); + $disks->{$name} = { + defined($dev->{lun}) ? (target => $T) : (), + lun => $dev->{lun} // 99, + devt => "8:$name", + hctl => $hctl, + defined($dev->{map}) ? (map => $dev->{map}) : (), + }; + $daemon->{$name} = { + devt => "8:$name", + hctl => $hctl, + init => 'ok', + target => $T, + wwid => $dev->{daemon}, + } + if defined($dev->{daemon}); + } + my $luns = + PVE::Multipath::Identity::resolve($disks, $resolvable ? $daemon : undef, sub { 1 }); + my $paths = {}; + for my $lun (values %$luns) { + $paths->{$_} = { unresolved => $lun->{unresolved} } for $lun->{paths}->@*; + } + $collected = { luns => $luns, paths => $paths }; + return $collected; + }, +); + +# the schema must declare every field of a row, as the CLI and the web UI rely on it +my $declared = + PVE::API2::Storage::Scan->map_method_by_name('sanlunscan')->{returns}->{items}->{properties}; +my sub scan { + my $res = PVE::API2::Storage::Scan->sanlunscan({ node => 'lab1' }); + for my $row (@$res) { + my @undeclared = grep { !$declared->{$_} } sort keys %$row; + fail("$row->{devpath} has undeclared fields: @undeclared") if @undeclared; + } + return { map { $_->{devpath} =~ s|^/dev/(mapper/)?||r => $_ } @$res }; +} +# a map of LUN 0 over sdb and sdc, and sdd, a path of it without a map yet +($devices, $maps) = ( + { + sd('sdb', 0, undef, map => '3W0'), + sd('sdc', 0, undef, map => '3W0'), + sd('sdd', 0, 'PV0', daemon => '3W0'), + }, + { '3W0' => { slaves => ['sdb', 'sdc'], pv => 'PV0' } }, +); +is(scan()->{'3W0'}->{'paths-total'}, 2, 'map path count of the disk scan'); +my $res = scan(); +is($res->{sdd}->{wwid}, '3W0', 'wwid: a path without a map has the identity of its LUN'); +ok($derived_with && $derived_with == $collected, 'the storages derive from the same collection'); +is_deeply([@order[0, 1]], ['identities', 'disks'], 'the identities are collected before the disks'); +$reused->{sdd} = 1; +ok(!defined(scan()->{sdd}->{wwid}), 'wwid: none for a name that denotes another device by now'); +is(scan()->{sdd}->{'wwid-unknown'}, 1, 'changed path exports identity uncertainty'); +$reused = {}; +$resolvable = 0; +$res = scan(); +is($res->{sdd}->{wwid}, '3W0', 'wwid: also without the daemon, through the map of its LUN'); +ok(!$res->{sdd}->{'wwid-unknown'}, 'trusted map identity is not uncertain'); +ok(!$res->{'3W0'}->{'wwid-unknown'}, 'map row retains trusted identity'); + +# LUN 5 without any map +($devices, $maps) = ( + { sd('sdl', 5, 'PV0', daemon => '3W5'), sd('sdb', 0, undef, map => '3W0') }, + { '3W0' => { slaves => ['sdb'], pv => 'PV0' } }, +); +$res = scan(); +ok( + !defined($res->{sdl}->{wwid}), + 'wwid: none for a LUN without a map while the daemon cannot be asked', +); +ok(!defined($res->{sdl}->{multipath}), 'and no multipath state either, as it is not known'); +is($res->{sdl}->{'wwid-unknown'}, 1, 'daemon unavailable exports uncertainty'); +$resolvable = 1; + +# two paths of LUN 1 without a map +($devices, $maps) = + ({ sd('sde', 1, 'PV1', daemon => '3W1'), sd('sdf', 1, 'PV1', daemon => '3W1') }, {}); +$devices->{sdf}->{daemon} = '3OTHER'; +$res = scan(); +is($res->{$_}->{'wwid-unknown'}, 1, "disagreeing LUN path $_ is uncertain") for qw(sde sdf); +$devices->{sdf}->{daemon} = '3W1'; + +# local disks without any identity +($devices, $maps) = ({ sd('sdj', undef, 'PV5'), sd('sdk', undef, 'PV5') }, {}); +ok(!scan()->{sdj}->{'wwid-unknown'}, 'unlisted path is not marked uncertain'); +$devices->{sdj}->{daemon} = ''; +is(scan()->{sdj}->{'wwid-unknown'}, 1, 'listed path without identity is uncertain'); + +# the real collection and derivation on a fake sysfs: a failing daemon query runs only once, and +# both the scan and the storage derivation do without its data +{ + $identity->unmock_all(); + $storage->unmock('derived_multipath_wwids'); + my $sys = File::Temp::tempdir(CLEANUP => 1); + $PVE::Multipath::Identity::SYSFS = $sys; + my sub put { + my ($path, $content) = @_; + File::Path::make_path("$sys/$path" =~ s|/[^/]+$||r); + open(my $fh, '>', "$sys/$path") or die "$!\n"; + print $fh $content; + close($fh); + } + # the LUN below the target of its session, as the iSCSI transport class places it + my $lun = "$sys/devices/platform/host7/session1/target7:0:0/7:0:0:6"; + File::Path::make_path($lun, "$sys/bus/scsi/devices"); + put('class/iscsi_session/session1/targetname', "$T\n"); + symlink("$sys/devices/platform/host7/session1", "$sys/class/iscsi_session/session1/device") + or die "$!\n"; + symlink($lun, "$sys/bus/scsi/devices/7:0:0:6") or die "$!\n"; + put('bus/scsi/devices/7:0:0:6/type', "0\n"); + put('bus/scsi/devices/7:0:0:6/block/sdq/dev', "65:0\n"); + File::Path::make_path("$sys/bus/scsi/devices/7:0:0:6/block/sdq/holders"); + my $queries = 0; + $multipath->mock( + is_absent => sub { 0 }, + is_supported => sub { 1 }, + query_path_identities => + sub { $queries++; die "multipathd returned an invalid path list\n" }, + ); + $storage->mock( + config => sub { { + ids => { + san => + { type => 'iscsi', target => $T, portal => '192.0.2.10', multipath => 1 }, + }, + } }, + ); + $iscsi->mock(iscsi_portals => sub { ['192.0.2.10:3260'] }); + ($devices, $maps) = ({ sd('sdq', 6, 'PVQ') }, {}); + $res = scan(); + is($queries, 1, 'shared: the daemon is asked once for the scan and the storages'); + ok(!defined($res->{sdq}->{wwid}), 'shared: and the path has no WWID without it'); + is($res->{sdq}->{'wwid-unknown'}, 1, 'real collection exports uncertainty to scan'); + my $ids = PVE::Multipath::Identity::collect(targets => { $T => 1 }); + is_deeply( + [ + map { [$_->@{qw(lun paths unresolved)}] } + PVE::Multipath::Identity::target_luns($ids, $T)->@* + ], + [[6, ['sdq'], 1]], + 'shared: the path counts as an unresolved LUN of the target of its session', + ); +} + +subtest 'direct image consumers win target attribution' => sub { + ($devices, $maps) = ({ sd('sdb', 0, undef, daemon => '3W0') }, {}); + for my $type ('iscsi', 'iscsidirect', 'zfs', 'vendor') { + $storage->mock( + config => sub { + return { + ids => { + a => { type => 'iscsi', target => $T, content => { none => 1 } }, + z => { + type => $type, + target => $T, + content => { images => 1 }, + disable => 1, + nodes => { elsewhere => 1 }, + }, + zz => { type => $type, target => $T, content => { images => 1 } }, + }, + }; + }, + ); + is(scan()->{sdb}->{'used-by-storage'}, 'z', "$type image consumer, deterministic tie"); + } + $devices = { nvme0n1 => { disk => { devpath => '/dev/nvme0n1', size => 1024 } } }; + $diskmanage->mock( + get_nvme_path_status => sub { + return { nvme0n1 => { active => 1, paths => [{ device => '/dev/nvme0' }] } }; + }, + ); + my $row = scan()->{nvme0n1}; + is($row->{'paths-active'}, 1, 'active path count'); + is($row->{'paths-total'}, 1, 'total path count'); + is(ref($row->{'path-list'}), 'ARRAY', 'path list'); +}; + +done_testing(); -- 2.47.3