From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 263851FF0B2 for ; Sun, 22 Nov 2026 02:28:42 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 40F11218A8; Sun, 22 Nov 2026 02:27:15 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Date: Sun, 22 Nov 2026 02:25:44 +0100 Message-ID: <20261122012605.1848824-8-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261122012605.1848824-1-t.lamprecht@proxmox.com> References: <20261122012605.1848824-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795310771871 X-SPAM-LEVEL: Spam detection results: 0 AWL -1.004 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LWENGBGTV43YCXSG45ZVZKAALJMZKCFZ X-Message-ID-Hash: LWENGBGTV43YCXSG45ZVZKAALJMZKCFZ X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The host NQN for the access rules of an array and the controllers of a node are only visible with nvme-cli on a shell. Add the API module for /nodes/{node}/san/nvmeof with the host identity and the fabrics controllers with their subsystem, portal, state and storage. The namespaces of all subsystems are counted in one pass for that list, as a node can have a thousand controllers, for example with a storage plugin that uses a subsystem per guest disk. Removing a portal or the storage does not disconnect, so a controller can be removed here, unless a namespace that is mounted, has holders, is a swap area, is open in a process or is referenced by a storage would lose its last usable path. A controller that is no usable path itself, such as one that keeps reconnecting to a deleted subsystem, only needs the local checks, as the check of the storage references reads the namespace. A controller with namespaces outside of native NVMe multipath is refused, as they lack the head devices that these checks go by. The caller can pass the subsystem and portal it listed, to refuse a controller number that the kernel reused meanwhile. Reading needs Sys.Audit on the node or Datastore.Allocate on /storage, the removal Sys.Modify on '/'. Signed-off-by: Thomas Lamprecht --- src/PVE/API2/Makefile | 1 + src/PVE/API2/NVMeOF.pm | 199 +++++++++++++++++++++++++++++ src/PVE/API2/SAN/Node.pm | 8 +- src/PVE/Diskmanage.pm | 18 +-- src/PVE/NVMeOF.pm | 127 +++++++++++++++++-- src/test/Makefile | 3 +- src/test/run_nvmeof_api_tests.pl | 207 +++++++++++++++++++++++++++++++ src/test/run_nvmeof_tests.pl | 7 ++ src/test/run_san_api_tests.pl | 2 +- 9 files changed, 553 insertions(+), 19 deletions(-) create mode 100644 src/PVE/API2/NVMeOF.pm create mode 100755 src/test/run_nvmeof_api_tests.pl diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile index 70ff1ec4..07773db1 100644 --- a/src/PVE/API2/Makefile +++ b/src/PVE/API2/Makefile @@ -4,6 +4,7 @@ install: install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm + install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm make -C Storage install make -C Disks install diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm new file mode 100644 index 00000000..a2c657b7 --- /dev/null +++ b/src/PVE/API2/NVMeOF.pm @@ -0,0 +1,199 @@ +package PVE::API2::NVMeOF; + +use v5.36; + +use PVE::JSONSchema qw(get_standard_option); +use PVE::RESTHandler; +use PVE::RPCEnvironment; + +use PVE::NVMeOF; +use PVE::Storage; + +use base qw(PVE::RESTHandler); + +my $read_permissions = { + check => [ + 'or', + ['perm', '/nodes/{node}', ['Sys.Audit']], + ['perm', '/storage', ['Datastore.Allocate']], + ], +}; +my $controller_schema = { + description => 'The fabrics controller name, for example nvme3.', + type => 'string', + pattern => 'nvme[0-9]+', +}; + +__PACKAGE__->register_method({ + name => 'index', + path => '', + method => 'GET', + proxyto => 'node', + permissions => { user => 'all' }, + description => 'Node NVMe over Fabrics index.', + parameters => + { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } }, + returns => { + type => 'array', + items => { type => 'object', properties => { name => { type => 'string' } } }, + links => [{ rel => 'child', href => '{name}' }], + }, + code => sub { return [{ name => 'status' }, { name => 'controller' }] }, +}); + +__PACKAGE__->register_method({ + name => 'status', + path => 'status', + method => 'GET', + protected => 1, + proxyto => 'node', + permissions => $read_permissions, + description => 'Report the NVMe host identity and nvme-cli availability.', + parameters => + { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } }, + returns => { + type => 'object', + additionalProperties => 0, + properties => { + supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' }, + 'host-nqn' => { + type => 'string', + optional => 1, + description => 'The host NQN for target access control.', + }, + 'host-id' => + { type => 'string', optional => 1, description => 'The persistent host UUID.' }, + error => { + type => 'string', + optional => 1, + description => 'Why the host identity is unavailable.', + }, + }, + }, + code => sub { + my $res = { supported => PVE::NVMeOF::is_supported() }; + my $host = eval { PVE::NVMeOF::host_identity() }; + if (my $error = $@) { + chomp $error; + $res->{error} = $error; + } else { + $res->{'host-nqn'} = $host->{nqn}; + $res->{'host-id'} = $host->{id}; + } + return $res; + }, +}); + +__PACKAGE__->register_method({ + name => 'controllers', + path => 'controller', + method => 'GET', + protected => 1, + proxyto => 'node', + permissions => $read_permissions, + description => 'List connected NVMe over Fabrics controllers and their states.', + parameters => + { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } }, + returns => { + type => 'array', + items => { + type => 'object', + additionalProperties => 0, + properties => { + controller => $controller_schema, + subsysnqn => { type => 'string', description => 'The subsystem NQN.' }, + transport => { type => 'string', description => 'The controller transport.' }, + portal => { + type => 'string', + description => 'The controller address and port, if applicable.', + }, + state => { + type => 'string', + description => 'The controller state, such as live or connecting.', + }, + namespaces => { + type => 'integer', + description => 'The number of namespace devices in the subsystem.', + }, + 'used-by-storage' => { + type => 'string', + optional => 1, + description => 'An NVMe over Fabrics storage using the subsystem.', + }, + }, + }, + }, + code => sub { + my $cfg = PVE::Storage::config(); + my %owners; + for my $sid (sort keys $cfg->{ids}->%*) { + my $scfg = $cfg->{ids}->{$sid}; + $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof'; + } + my $controllers = PVE::NVMeOF::controllers(); + my $counts = PVE::NVMeOF::namespace_counts(); + my @rows; + for my $name (sort keys %$controllers) { + my $ctrl = $controllers->{$name}; + my $nqn = $ctrl->{subsystem}; + push @rows, + { + controller => $name, + subsysnqn => $nqn, + transport => $ctrl->{transport}, + portal => PVE::NVMeOF::format_path($ctrl), + state => $ctrl->{state}, + namespaces => $counts->{$nqn} // 0, + defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (), + }; + } + return \@rows; + }, +}); + +__PACKAGE__->register_method({ + name => 'disconnect', + path => 'controller/{controller}', + method => 'DELETE', + protected => 1, + proxyto => 'node', + permissions => { check => ['perm', '/', ['Sys.Modify']] }, + description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no' + . ' other usable path. A storage still listing the portal reconnects it on activation.', + parameters => { + additionalProperties => 0, + properties => { + node => get_standard_option('pve-node'), + controller => $controller_schema, + subsysnqn => { + type => 'string', + maxLength => 223, + optional => 1, + description => 'Refuse unless the controller still has this subsystem NQN.', + }, + portal => { + type => 'string', + maxLength => 128, + optional => 1, + description => 'Refuse unless the controller still has this portal.', + }, + }, + }, + returns => { type => 'string', description => 'The task identifier.' }, + code => sub($param) { + my $rpcenv = PVE::RPCEnvironment::get(); + return $rpcenv->fork_worker( + 'nvmedisconnect', + $param->{controller}, + $rpcenv->get_user(), + sub { + PVE::NVMeOF::disconnect_guarded( + $param->{controller}, + { subsysnqn => $param->{subsysnqn}, portal => $param->{portal} }, + ); + }, + ); + }, +}); + +1; diff --git a/src/PVE/API2/SAN/Node.pm b/src/PVE/API2/SAN/Node.pm index 4b96caf5..260c95f7 100644 --- a/src/PVE/API2/SAN/Node.pm +++ b/src/PVE/API2/SAN/Node.pm @@ -10,6 +10,7 @@ use PVE::Storage; use PVE::Storage::SAN; use PVE::API2::ISCSI; +use PVE::API2::NVMeOF; use base qw(PVE::RESTHandler); @@ -18,6 +19,11 @@ __PACKAGE__->register_method({ path => 'iscsi', }); +__PACKAGE__->register_method({ + subclass => "PVE::API2::NVMeOF", + path => 'nvmeof', +}); + __PACKAGE__->register_method({ name => 'index', path => '', @@ -42,7 +48,7 @@ __PACKAGE__->register_method({ links => [{ rel => 'child', href => "{name}" }], }, code => sub { - return [map { +{ name => $_ } } qw(iscsi luns)]; + return [map { +{ name => $_ } } qw(iscsi luns nvmeof)]; }, }); diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm index 21857dc6..523d9c05 100644 --- a/src/PVE/Diskmanage.pm +++ b/src/PVE/Diskmanage.pm @@ -958,6 +958,15 @@ sub get_multipath_disks { return $disklist; } +# 'change' is transient; paths in 'inaccessible' or 'persistent-loss' cannot serve IO even on a live +# controller +sub nvme_path_is_usable { + my ($path) = @_; + my $ana = $path->{'ana-state'}; + return $path->{state} eq 'live' + && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized'); +} + # Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local # single-controller devices lack these children and are omitted. sub get_nvme_path_status { @@ -993,15 +1002,8 @@ sub get_nvme_path_status { my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state"); $entry->{'ana-state'} = $ana if defined($ana); - # 'change' is transient; paths in 'inaccessible' or 'persistent-loss' - # cannot serve IO even on a live controller - my $usable = $state eq 'live' - && (!defined($ana) - || $ana eq 'optimized' - || $ana eq 'non-optimized'); - my $status = $res->{$head} //= { active => 0, paths => [] }; - $status->{active}++ if $usable; + $status->{active}++ if nvme_path_is_usable($entry); push @{ $status->{paths} }, $entry; }, ); diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm index 15612b04..7961a49e 100644 --- a/src/PVE/NVMeOF.pm +++ b/src/PVE/NVMeOF.pm @@ -9,6 +9,7 @@ use JSON qw(decode_json); use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton); use Time::HiRes (); +use PVE::Cluster; use PVE::JSONSchema; use PVE::Network; use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command); @@ -21,6 +22,7 @@ our $HOSTNQN_FILE = '/etc/nvme/hostnqn'; our $HOSTID_FILE = '/etc/nvme/hostid'; our $RUN_DIR = '/run/pve/nvmeof'; our $LOCK_WAIT = 10; +our $SWAPS = '/proc/swaps'; our $NQN_PATTERN = 'nqn\.[^\s,=]+'; our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery'; @@ -162,26 +164,42 @@ sub path_controller($controllers, $path) { return undef; } -sub namespaces($subsystem) { - my $res = {}; +# Calls $code with the subsystem NQN and the device name of each namespace of this node, or only +# of those of the subsystem $only. +my sub foreach_namespace($code, $only = undef) { my $dir = "$SYSFS/class/nvme-subsystem"; dir_glob_foreach( $dir, 'nvme-subsys[0-9]+', sub($subsys, @) { my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return; - return if $nqn ne $subsystem; + return if defined($only) && $nqn ne $only; dir_glob_foreach( "$dir/$subsys", 'nvme[0-9]+n[0-9]+', - sub($dev, @) { - my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return; - my $size = file_read_firstline("$SYSFS/block/$dev/size") // return; - $res->{$dev} = { wwid => $wwid, size => int($size) * 512 }; - }, + sub($dev, @) { $code->($nqn, $dev) }, ); }, ); +} + +sub namespaces($subsystem) { + my $res = {}; + foreach_namespace( + sub($nqn, $dev) { + my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return; + my $size = file_read_firstline("$SYSFS/block/$dev/size") // return; + $res->{$dev} = { wwid => $wwid, size => int($size) * 512 }; + }, + $subsystem, + ); + return $res; +} + +# The number of namespaces of each subsystem of this node, from one pass over the subsystems. +sub namespace_counts() { + my $res = {}; + foreach_namespace(sub($nqn, @) { $res->{$nqn}++ }); return $res; } @@ -331,4 +349,97 @@ sub discover($portal) { die "discovery at portal '$portal' failed - $error"; } +# A swap area on a namespace has no holder, mount or file descriptor of a process. +my sub is_swap($device) { + my %rdevs = map { + my $rdev = (stat($_))[6]; + defined($rdev) ? ($rdev => 1) : () + } ($device, glob("${device}p[0-9]*")); + for my $line (split(/\n/, PVE::Tools::file_get_contents($SWAPS))) { + my ($swap) = split(/\s+/, $line); + next if !defined($swap) || $swap !~ m!^/!; + return 1 if $swap =~ m!^\Q$device\E(?:p[0-9]+)?$!; + my $rdev = (stat($swap))[6]; + return 1 if defined($rdev) && $rdevs{$rdev}; + } + return 0; +} + +# The expected subsystem and portal refuse a controller that replaced the one the caller listed. +sub disconnect_guarded($controller, $expected = {}) { + my ($name) = $controller =~ /\A(nvme[0-9]+)\z/; + die "invalid controller name '$controller'\n" if !defined($name); + return with_lock(sub { + require PVE::Storage; + require PVE::Storage::SAN; + my $ctrl = controllers()->{$name} + // die "'$name' is not an NVMe over Fabrics controller\n"; + if ( + (defined($expected->{subsysnqn}) && $expected->{subsysnqn} ne $ctrl->{subsystem}) + || (defined($expected->{portal}) && $expected->{portal} ne format_path($ctrl)) + ) { + die + "controller '$name' is no longer connected to the listed subsystem and portal\n"; + } + # Without native multipath a namespace is a device of its controller and has no head + # device, which the checks below go by. + my $local = 0; + dir_glob_foreach( + "$SYSFS/class/nvme-fabrics/ctl/$name", + "${name}n[0-9]+", + sub { $local = 1 }, + ); + die "cannot disconnect '$name' - its namespaces do not use native NVMe multipath\n" + if $local; + my $namespaces = namespaces($ctrl->{subsystem}); + my $status = PVE::Diskmanage::get_nvme_path_status(); + my %heads = map { $_ => 1 } keys %$namespaces; + for my $head (keys %$status) { + $heads{$head} = 1 + if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*; + } + PVE::Cluster::cfs_update(1); + my $cfg = PVE::Storage::config(); + for my $head (sort keys %heads) { + my $paths = $status->{$head}->{paths} // []; + my @others = grep { $_->{device} ne "/dev/$name" } @$paths; + next if grep { PVE::Diskmanage::nvme_path_is_usable($_) } @others; + my $ns = $namespaces->{$head}; + if (!defined($ns)) { + die "cannot disconnect '$name' - namespace '$head' has no device on this node" + . " to check for use\n"; + } + my $device = "/dev/$head"; + if ( + PVE::Diskmanage::is_mounted($device) + || PVE::Diskmanage::has_holder($device) + || is_swap($device) + ) { + die "cannot disconnect '$name' - namespace '$head' is in use and has no other" + . " usable path\n"; + } + my $opener = PVE::Diskmanage::device_opener($device, glob("${device}p[0-9]*")); + if (defined($opener)) { + die "cannot disconnect '$name' - process $opener has namespace '$head' open and" + . " it has no other usable path\n"; + } + # The check of the storage references reads the namespace. Removing a path that is not + # usable either costs no usable path, and a reconnecting one cannot be read. + next if !grep { + $_->{device} eq "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_) + } @$paths; + eval { + PVE::Storage::SAN::assert_lun_unreferenced( + $cfg, + { devpath => $device, wwid => $ns->{wwid} }, + ); + }; + die "cannot disconnect '$name' - $@" if $@; + } + die "nvme-cli is not installed\n" if !is_supported(); + run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30); + return; + }); +} + 1; diff --git a/src/test/Makefile b/src/test/Makefile index ceabd991..d323e8a1 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -117,6 +117,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl test_disk_api: run_disk_api_tests.pl ./run_disk_api_tests.pl -test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl +test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl ./run_nvmeof_tests.pl + ./run_nvmeof_api_tests.pl perl -T ./run_nvmeof_taint_tests.pl diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl new file mode 100755 index 00000000..f86291d0 --- /dev/null +++ b/src/test/run_nvmeof_api_tests.pl @@ -0,0 +1,207 @@ +#!/usr/bin/perl + +use v5.36; +use lib qw(..); +use File::Path qw(make_path remove_tree); +use File::Temp qw(tempdir); +use Test::MockModule; +use Test::More; + +use PVE::API2::NVMeOF; +use PVE::Storage::SAN; + +my $sys = tempdir('nvmeof-api-test-XXXX', DIR => '.', CLEANUP => 1); +local $PVE::NVMeOF::SYSFS = $sys; +local $PVE::NVMeOF::SWAPS = "$sys/swaps"; +PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n"); + +subtest 'last-path disconnect is checked inside serialization' => sub { + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + my $disk = Test::MockModule->new('PVE::Diskmanage'); + my $storage = Test::MockModule->new('PVE::Storage'); + my $san = Test::MockModule->new('PVE::Storage::SAN'); + my $cluster = Test::MockModule->new('PVE::Cluster'); + our $locked; + my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0); + my $own = { device => '/dev/nvme1', state => 'live' }; + my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' }; + $nvme->mock(with_lock => + sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() }); + $nvme->mock( + controllers => sub { + ok($locked, 'controller snapshot is locked'); + { nvme1 => { subsystem => 'nqn.test' } }; + }, + ); + $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } }); + $nvme->mock(is_supported => sub { 1 }); + $disk->mock( + get_nvme_path_status => sub { { nvme0n1 => { paths => [$own, $other] } } }, + ); + $disk->mock(is_mounted => sub { $mounted }); + $disk->mock(has_holder => sub { 0 }); + my $opener; + $disk->mock(device_opener => sub { $opener }); + $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') }); + $storage->mock(config => sub { { ids => {} } }); + $san->mock( + assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" }); + $nvme->mock( + run_command => sub($cmd, %opts) { + ok($locked, 'deletion keeps the lock'); + is($opts{timeout}, 30, 'deletion is bounded'); + is_deeply( + $cmd, + ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'], + 'only the selected controller is removed', + ); + $commands++; + }, + ); + for my $ana ('optimized', 'non-optimized') { + $other->{'ana-state'} = $ana; + PVE::NVMeOF::disconnect_guarded('nvme1'); + } + is($commands, 2, 'another usable path permits removal'); + is($checks, 0, 'no usage probe when another path is usable'); + for my $ana ('inaccessible', 'persistent-loss', 'change') { + $other->{'ana-state'} = $ana; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace"); + } + $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' }; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path'); + is($commands, 2, 'refusals never delete'); + $own->{state} = 'connecting'; + PVE::NVMeOF::disconnect_guarded('nvme1'); + is($commands, 3, 'a controller that is no usable path either needs no storage check'); + $opener = '4711 (kvm)'; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like( + $@, + qr/process 4711 \(kvm\) has namespace/, + 'but a process that has the namespace open refuses', + ); + $opener = undef; + $own->{state} = 'live'; + $san->mock(assert_lun_unreferenced => sub { $checks++ }); + $mounted = 1; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/is in use/, 'local mount refuses'); + $mounted = 0; + PVE::Tools::file_set_contents( + $PVE::NVMeOF::SWAPS, + "Filename Type Size Used Priority\n/dev/nvme0n1p2 partition 1024 0 -2\n", + ); + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/is in use/, 'a swap area on the namespace refuses'); + PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n"); + eval { PVE::NVMeOF::disconnect_guarded('nvme1', { subsysnqn => 'nqn.other' }) }; + like($@, qr/no longer connected to the listed subsystem/, 'a replaced controller refuses'); + $opener = '4711 (kvm)'; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/process 4711 \(kvm\) has namespace 'nvme0n1' open/, + 'an open namespace refuses'); + $opener = undef; + PVE::NVMeOF::disconnect_guarded('nvme1'); + is($commands, 4, 'unused namespace can lose its final path'); + make_path( + "$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1", + "$sys/class/nvme-fabrics/ctl/nvme1/nvme0c1n1", + ); + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/do not use native NVMe multipath/, 'a namespace without head device refuses'); + is($commands, 4, 'and nothing is disconnected'); + remove_tree("$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1"); + PVE::NVMeOF::disconnect_guarded('nvme1'); + is($commands, 5, 'the path device of native multipath does not'); + $headless = 1; + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/has no device/, 'uncheckable namespace refuses'); + $busy = 1; + $nvme->mock(controllers => sub { die "must not count paths\n" }); + eval { PVE::NVMeOF::disconnect_guarded('nvme1') }; + like($@, qr/is busy/, 'ongoing deletion blocks another snapshot'); + eval { PVE::NVMeOF::disconnect_guarded('../nvme1') }; + like($@, qr/invalid controller/, 'invalid controller refused before locking'); +}; + +subtest 'node API contract' => sub { + my $api = 'PVE::API2::NVMeOF'; + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + my $storage = Test::MockModule->new('PVE::Storage'); + $storage->mock( + config => + sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } }, + ); + $nvme->mock(is_supported => sub { 0 }); + $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } }); + my $status = $api->status({ node => 'n1' }); + is($status->{supported}, 0, 'availability means nvme-cli, not identity or version'); + is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently'); + $nvme->mock(is_supported => sub { 1 }); + $nvme->mock(host_identity => sub { die "missing host identity\n" }); + $status = $api->status({ node => 'n1' }); + is_deeply( + $status, + { supported => 1, error => 'missing host identity' }, + 'identity error does not hide installed tools', + ); + $nvme->mock( + controllers => sub { { + nvme1 => { + subsystem => 'nqn.test', + transport => 'tcp', + traddr => '192.0.2.1', + trsvcid => 4420, + state => 'connecting', + }, + } }, + ); + $nvme->mock( + namespaces => sub { die "must not walk the subsystems for each controller\n" }, + namespace_counts => sub { { 'nqn.test' => 2, 'nqn.other' => 5 } }, + ); + my $rows = $api->controllers({ node => 'n1' }); + is($rows->[0]->{namespaces}, 2, 'namespaces counted for the subsystem of the controller'); + is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed'); + is($rows->[0]->{state}, 'connecting', 'kernel state remains visible'); + + for my $pair (['status', $status], ['controllers', $rows]) { + my $method = $api->map_method_by_name($pair->[0]); + eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) }; + is($@, '', "$pair->[0] schema"); + ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied'); + } + is( + $api->map_method_by_name('disconnect')->{path}, + 'controller/{controller}', + 'REST-shaped deletion', + ); + eval { + PVE::JSONSchema::validate( + { node => 'n1', controller => 'nvme1', subsysnqn => 'testsubsys' }, + $api->map_method_by_name('disconnect')->{parameters}, + ); + }; + is($@, '', 'the listed subsystem name is taken as is, also when it is no NQN'); + my $rpc = Test::MockModule->new('PVE::RPCEnvironment'); + { + + package NVMeTestRPC; + sub get_user { 'root@pam' } + sub fork_worker { $_[4]->(); return 'UPID:test' } + } + $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' }); + my $removed; + $nvme->mock(disconnect_guarded => sub { $removed = $_[0] }); + is( + $api->disconnect({ node => 'n1', controller => 'nvme1' }), + 'UPID:test', + 'API returns task identifier', + ); + is($removed, 'nvme1', 'task calls guarded disconnect'); +}; + +done_testing(); diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl index ddd905ee..ac029207 100755 --- a/src/test/run_nvmeof_tests.pl +++ b/src/test/run_nvmeof_tests.pl @@ -212,6 +212,13 @@ subtest 'sysfs topology' => sub { 'namespace facts', ); is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded'); + make_path(map { "$sys/class/nvme-subsystem/nvme-subsys1/nvme1n$_" } 1 .. 3); + file_set_contents("$sys/class/nvme-subsystem/nvme-subsys1/subsysnqn", "nqn.other\n"); + is_deeply( + PVE::NVMeOF::namespace_counts(), + { 'nqn.test' => 1, 'nqn.other' => 3 }, + 'namespaces counted per subsystem', + ); }; subtest 'serialized commands inherit the lock' => sub { diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl index 73938a7e..2ae7b2b4 100755 --- a/src/test/run_san_api_tests.pl +++ b/src/test/run_san_api_tests.pl @@ -109,7 +109,7 @@ is_deeply( ); is_deeply( PVE::API2::SAN::Node->index({ node => 'test' }), - [{ name => 'iscsi' }, { name => 'luns' }], + [{ name => 'iscsi' }, { name => 'luns' }, { name => 'nvmeof' }], 'node SAN index', ); for my $case (