From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal
Date: Sun, 22 Nov 2026 02:25:44 +0100 [thread overview]
Message-ID: <20261122012605.1848824-8-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261122012605.1848824-1-t.lamprecht@proxmox.com>
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/san/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage. The
namespaces of all subsystems are counted in one pass for that list, as
a node can have a thousand controllers, for example with a storage
plugin that uses a subsystem per guest disk.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace that is mounted, has holders,
is a swap area, is open in a process or is referenced by a storage
would lose its last usable path. A controller that is no usable path
itself, such as one that keeps reconnecting to a deleted subsystem,
only needs the local checks, as the check of the storage references
reads the namespace. A controller with namespaces outside of native
NVMe multipath is refused, as they lack the head devices that these
checks go by. The caller can pass the subsystem and portal it listed,
to refuse a controller number that the kernel reused meanwhile.
Reading needs Sys.Audit on the node or Datastore.Allocate on /storage,
the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 199 +++++++++++++++++++++++++++++
src/PVE/API2/SAN/Node.pm | 8 +-
src/PVE/Diskmanage.pm | 18 +--
src/PVE/NVMeOF.pm | 127 +++++++++++++++++--
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 207 +++++++++++++++++++++++++++++++
src/test/run_nvmeof_tests.pl | 7 ++
src/test/run_san_api_tests.pl | 2 +-
9 files changed, 553 insertions(+), 19 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index 70ff1ec4..07773db1 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..a2c657b7
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,199 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme[0-9]+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my %owners;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my $counts = PVE::NVMeOF::namespace_counts();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts->{$nqn} // 0,
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ controller => $controller_schema,
+ subsysnqn => {
+ type => 'string',
+ maxLength => 223,
+ optional => 1,
+ description => 'Refuse unless the controller still has this subsystem NQN.',
+ },
+ portal => {
+ type => 'string',
+ maxLength => 128,
+ optional => 1,
+ description => 'Refuse unless the controller still has this portal.',
+ },
+ },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub($param) {
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded(
+ $param->{controller},
+ { subsysnqn => $param->{subsysnqn}, portal => $param->{portal} },
+ );
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/API2/SAN/Node.pm b/src/PVE/API2/SAN/Node.pm
index 4b96caf5..260c95f7 100644
--- a/src/PVE/API2/SAN/Node.pm
+++ b/src/PVE/API2/SAN/Node.pm
@@ -10,6 +10,7 @@ use PVE::Storage;
use PVE::Storage::SAN;
use PVE::API2::ISCSI;
+use PVE::API2::NVMeOF;
use base qw(PVE::RESTHandler);
@@ -18,6 +19,11 @@ __PACKAGE__->register_method({
path => 'iscsi',
});
+__PACKAGE__->register_method({
+ subclass => "PVE::API2::NVMeOF",
+ path => 'nvmeof',
+});
+
__PACKAGE__->register_method({
name => 'index',
path => '',
@@ -42,7 +48,7 @@ __PACKAGE__->register_method({
links => [{ rel => 'child', href => "{name}" }],
},
code => sub {
- return [map { +{ name => $_ } } qw(iscsi luns)];
+ return [map { +{ name => $_ } } qw(iscsi luns nvmeof)];
},
});
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 21857dc6..523d9c05 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -958,6 +958,15 @@ sub get_multipath_disks {
return $disklist;
}
+# 'change' is transient; paths in 'inaccessible' or 'persistent-loss' cannot serve IO even on a live
+# controller
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -993,15 +1002,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index 15612b04..7961a49e 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -9,6 +9,7 @@ use JSON qw(decode_json);
use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
use Time::HiRes ();
+use PVE::Cluster;
use PVE::JSONSchema;
use PVE::Network;
use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
@@ -21,6 +22,7 @@ our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
our $HOSTID_FILE = '/etc/nvme/hostid';
our $RUN_DIR = '/run/pve/nvmeof';
our $LOCK_WAIT = 10;
+our $SWAPS = '/proc/swaps';
our $NQN_PATTERN = 'nqn\.[^\s,=]+';
our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery';
@@ -162,26 +164,42 @@ sub path_controller($controllers, $path) {
return undef;
}
-sub namespaces($subsystem) {
- my $res = {};
+# Calls $code with the subsystem NQN and the device name of each namespace of this node, or only
+# of those of the subsystem $only.
+my sub foreach_namespace($code, $only = undef) {
my $dir = "$SYSFS/class/nvme-subsystem";
dir_glob_foreach(
$dir,
'nvme-subsys[0-9]+',
sub($subsys, @) {
my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
- return if $nqn ne $subsystem;
+ return if defined($only) && $nqn ne $only;
dir_glob_foreach(
"$dir/$subsys",
'nvme[0-9]+n[0-9]+',
- sub($dev, @) {
- my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
- my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
- $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
- },
+ sub($dev, @) { $code->($nqn, $dev) },
);
},
);
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ foreach_namespace(
+ sub($nqn, $dev) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+ },
+ $subsystem,
+ );
+ return $res;
+}
+
+# The number of namespaces of each subsystem of this node, from one pass over the subsystems.
+sub namespace_counts() {
+ my $res = {};
+ foreach_namespace(sub($nqn, @) { $res->{$nqn}++ });
return $res;
}
@@ -331,4 +349,97 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+# A swap area on a namespace has no holder, mount or file descriptor of a process.
+my sub is_swap($device) {
+ my %rdevs = map {
+ my $rdev = (stat($_))[6];
+ defined($rdev) ? ($rdev => 1) : ()
+ } ($device, glob("${device}p[0-9]*"));
+ for my $line (split(/\n/, PVE::Tools::file_get_contents($SWAPS))) {
+ my ($swap) = split(/\s+/, $line);
+ next if !defined($swap) || $swap !~ m!^/!;
+ return 1 if $swap =~ m!^\Q$device\E(?:p[0-9]+)?$!;
+ my $rdev = (stat($swap))[6];
+ return 1 if defined($rdev) && $rdevs{$rdev};
+ }
+ return 0;
+}
+
+# The expected subsystem and portal refuse a controller that replaced the one the caller listed.
+sub disconnect_guarded($controller, $expected = {}) {
+ my ($name) = $controller =~ /\A(nvme[0-9]+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage;
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ if (
+ (defined($expected->{subsysnqn}) && $expected->{subsysnqn} ne $ctrl->{subsystem})
+ || (defined($expected->{portal}) && $expected->{portal} ne format_path($ctrl))
+ ) {
+ die
+ "controller '$name' is no longer connected to the listed subsystem and portal\n";
+ }
+ # Without native multipath a namespace is a device of its controller and has no head
+ # device, which the checks below go by.
+ my $local = 0;
+ dir_glob_foreach(
+ "$SYSFS/class/nvme-fabrics/ctl/$name",
+ "${name}n[0-9]+",
+ sub { $local = 1 },
+ );
+ die "cannot disconnect '$name' - its namespaces do not use native NVMe multipath\n"
+ if $local;
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ my @others = grep { $_->{device} ne "/dev/$name" } @$paths;
+ next if grep { PVE::Diskmanage::nvme_path_is_usable($_) } @others;
+ my $ns = $namespaces->{$head};
+ if (!defined($ns)) {
+ die "cannot disconnect '$name' - namespace '$head' has no device on this node"
+ . " to check for use\n";
+ }
+ my $device = "/dev/$head";
+ if (
+ PVE::Diskmanage::is_mounted($device)
+ || PVE::Diskmanage::has_holder($device)
+ || is_swap($device)
+ ) {
+ die "cannot disconnect '$name' - namespace '$head' is in use and has no other"
+ . " usable path\n";
+ }
+ my $opener = PVE::Diskmanage::device_opener($device, glob("${device}p[0-9]*"));
+ if (defined($opener)) {
+ die "cannot disconnect '$name' - process $opener has namespace '$head' open and"
+ . " it has no other usable path\n";
+ }
+ # The check of the storage references reads the namespace. Removing a path that is not
+ # usable either costs no usable path, and a reconnecting one cannot be read.
+ next if !grep {
+ $_->{device} eq "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_)
+ } @$paths;
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index ceabd991..d323e8a1 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -117,6 +117,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..f86291d0
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,207 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::API2::NVMeOF;
+use PVE::Storage::SAN;
+
+my $sys = tempdir('nvmeof-api-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::SYSFS = $sys;
+local $PVE::NVMeOF::SWAPS = "$sys/swaps";
+PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n");
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $own = { device => '/dev/nvme1', state => 'live' };
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status => sub { { nvme0n1 => { paths => [$own, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ my $opener;
+ $disk->mock(device_opener => sub { $opener });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $own->{state} = 'connecting';
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'a controller that is no usable path either needs no storage check');
+ $opener = '4711 (kvm)';
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like(
+ $@,
+ qr/process 4711 \(kvm\) has namespace/,
+ 'but a process that has the namespace open refuses',
+ );
+ $opener = undef;
+ $own->{state} = 'live';
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ PVE::Tools::file_set_contents(
+ $PVE::NVMeOF::SWAPS,
+ "Filename Type Size Used Priority\n/dev/nvme0n1p2 partition 1024 0 -2\n",
+ );
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'a swap area on the namespace refuses');
+ PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n");
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1', { subsysnqn => 'nqn.other' }) };
+ like($@, qr/no longer connected to the listed subsystem/, 'a replaced controller refuses');
+ $opener = '4711 (kvm)';
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/process 4711 \(kvm\) has namespace 'nvme0n1' open/,
+ 'an open namespace refuses');
+ $opener = undef;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 4, 'unused namespace can lose its final path');
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1",
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme0c1n1",
+ );
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/do not use native NVMe multipath/, 'a namespace without head device refuses');
+ is($commands, 4, 'and nothing is disconnected');
+ remove_tree("$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1");
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 5, 'the path device of native multipath does not');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(
+ namespaces => sub { die "must not walk the subsystems for each controller\n" },
+ namespace_counts => sub { { 'nqn.test' => 2, 'nqn.other' => 5 } },
+ );
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{namespaces}, 2, 'namespaces counted for the subsystem of the controller');
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ eval {
+ PVE::JSONSchema::validate(
+ { node => 'n1', controller => 'nvme1', subsysnqn => 'testsubsys' },
+ $api->map_method_by_name('disconnect')->{parameters},
+ );
+ };
+ is($@, '', 'the listed subsystem name is taken as is, also when it is no NQN');
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index ddd905ee..ac029207 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -212,6 +212,13 @@ subtest 'sysfs topology' => sub {
'namespace facts',
);
is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+ make_path(map { "$sys/class/nvme-subsystem/nvme-subsys1/nvme1n$_" } 1 .. 3);
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys1/subsysnqn", "nqn.other\n");
+ is_deeply(
+ PVE::NVMeOF::namespace_counts(),
+ { 'nqn.test' => 1, 'nqn.other' => 3 },
+ 'namespaces counted per subsystem',
+ );
};
subtest 'serialized commands inherit the lock' => sub {
diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl
index 73938a7e..2ae7b2b4 100755
--- a/src/test/run_san_api_tests.pl
+++ b/src/test/run_san_api_tests.pl
@@ -109,7 +109,7 @@ is_deeply(
);
is_deeply(
PVE::API2::SAN::Node->index({ node => 'test' }),
- [{ name => 'iscsi' }, { name => 'luns' }],
+ [{ name => 'iscsi' }, { name => 'luns' }, { name => 'nvmeof' }],
'node SAN index',
);
for my $case (
next prev parent reply other threads:[~2026-11-22 1:28 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22 1:25 ` Thomas Lamprecht [this message]
2026-11-22 1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122012605.1848824-8-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox