From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal
Date: Fri, 13 Nov 2026 04:04:35 +0100 [thread overview]
Message-ID: <20261113030541.1591528-8-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261113030541.1591528-1-t.lamprecht@proxmox.com>
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace in use would lose its last
usable path. Reading needs Sys.Audit on the node or
Datastore.Allocate on /storage, the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 182 +++++++++++++++++++++++++++++++
src/PVE/Diskmanage.pm | 16 +--
src/PVE/NVMeOF.pm | 41 +++++++
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 151 +++++++++++++++++++++++++
6 files changed, 385 insertions(+), 9 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index 9ab978b0..1be9ab04 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..57e70df4
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,182 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme\d+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my (%owners, %counts);
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ $counts{$nqn} //= scalar(keys PVE::NVMeOF::namespaces($nqn)->%*);
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts{$nqn},
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties =>
+ { node => get_standard_option('pve-node'), controller => $controller_schema },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub {
+ my ($param) = @_;
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded($param->{controller});
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 7bee3b08..3141ca7a 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -898,6 +898,13 @@ sub get_multipath_disks {
return $disklist;
}
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -933,15 +940,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index d1dfe15b..c81cb72f 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -308,4 +308,45 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+sub disconnect_guarded($controller) {
+ my ($name) = $controller =~ /\A(nvme\d+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ next
+ if grep { $_->{device} ne "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_) }
+ @$paths;
+ my $ns = $namespaces->{$head} // die
+ "cannot disconnect '$name' - namespace '$head' has no device on this node to check for use\n";
+ my $device = "/dev/$head";
+ die
+ "cannot disconnect '$name' - namespace '$head' is in use and has no other usable path\n"
+ if PVE::Diskmanage::is_mounted($device) || PVE::Diskmanage::has_holder($device);
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index 124612da..73183dc1 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -77,6 +77,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..3849dba5
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,151 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Test::More;
+use Test::MockModule;
+use PVE::API2::NVMeOF;
+use PVE::Storage::SAN;
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status =>
+ sub { {
+ nvme0n1 => { paths => [{ device => '/dev/nvme1', state => 'live' }, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'unused namespace can lose its final path');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(namespaces => sub { { nvme0n1 => {} } });
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
next prev parent reply other threads:[~2026-11-13 3:33 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13 3:04 ` Thomas Lamprecht [this message]
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113030541.1591528-8-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox