From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal
Date: Fri, 13 Nov 2026 04:04:35 +0100 [thread overview]
Message-ID: <20261113030541.1591528-8-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261113030541.1591528-1-t.lamprecht@proxmox.com>
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace in use would lose its last
usable path. Reading needs Sys.Audit on the node or
Datastore.Allocate on /storage, the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 182 +++++++++++++++++++++++++++++++
src/PVE/Diskmanage.pm | 16 +--
src/PVE/NVMeOF.pm | 41 +++++++
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 151 +++++++++++++++++++++++++
6 files changed, 385 insertions(+), 9 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index 9ab978b0..1be9ab04 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..57e70df4
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,182 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme\d+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my (%owners, %counts);
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ $counts{$nqn} //= scalar(keys PVE::NVMeOF::namespaces($nqn)->%*);
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts{$nqn},
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties =>
+ { node => get_standard_option('pve-node'), controller => $controller_schema },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub {
+ my ($param) = @_;
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded($param->{controller});
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 7bee3b08..3141ca7a 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -898,6 +898,13 @@ sub get_multipath_disks {
return $disklist;
}
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -933,15 +940,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index d1dfe15b..c81cb72f 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -308,4 +308,45 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+sub disconnect_guarded($controller) {
+ my ($name) = $controller =~ /\A(nvme\d+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ next
+ if grep { $_->{device} ne "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_) }
+ @$paths;
+ my $ns = $namespaces->{$head} // die
+ "cannot disconnect '$name' - namespace '$head' has no device on this node to check for use\n";
+ my $device = "/dev/$head";
+ die
+ "cannot disconnect '$name' - namespace '$head' is in use and has no other usable path\n"
+ if PVE::Diskmanage::is_mounted($device) || PVE::Diskmanage::has_holder($device);
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index 124612da..73183dc1 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -77,6 +77,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..3849dba5
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,151 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Test::More;
+use Test::MockModule;
+use PVE::API2::NVMeOF;
+use PVE::Storage::SAN;
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status =>
+ sub { {
+ nvme0n1 => { paths => [{ device => '/dev/nvme1', state => 'live' }, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'unused namespace can lose its final path');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(namespaces => sub { { nvme0n1 => {} } });
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
next prev parent reply other threads:[~2026-11-13 3:33 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13 3:04 ` Thomas Lamprecht [this message]
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113030541.1591528-8-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.