* [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
` (15 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The disk list queries the SMART health of every disk. For an NVMe
namespace attached over TCP or RDMA, the query goes to the target and
can stall the listing while a path recovers.
Skip the query for these transports. It depends on the transport of
the namespace only, so namespaces connected by hand are covered too.
Their health shows as unknown, local NVMe drives are queried as
before.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Diskmanage.pm | 4 +++-
src/test/disklist_test.pm | 14 ++++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 557a8a4a..8fda7afc 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -694,7 +694,9 @@ sub get_disks {
}
my ($health, $wearout) = ('UNKNOWN', 'N/A');
- if (!$nosmart) {
+ # SMART queries over fabrics can stall while a path recovers.
+ my $remote_nvme = defined($nvme_transport) && $nvme_transport =~ /^(?:tcp|rdma)$/;
+ if (!$nosmart && !$remote_nvme) {
eval {
my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
$health = $smartdata->{health} if $smartdata->{health};
diff --git a/src/test/disklist_test.pm b/src/test/disklist_test.pm
index f79c59d3..74ca3b2d 100644
--- a/src/test/disklist_test.pm
+++ b/src/test/disklist_test.pm
@@ -284,6 +284,20 @@ sub test_disk_list {
ok($san_disks->{sda}, 'SAN inventory opts into iSCSI');
}
+ if ($testcasedir eq 'nvme_smart') {
+ my $smart_calls;
+ $diskmanage_module->mock(get_smart_data => sub { $smart_calls++; return {} });
+ for my $transport (qw(tcp rdma pcie)) {
+ $smart_calls = 0;
+ $diskmanage_module->mock(get_nvme_transport => sub { $transport });
+ PVE::Diskmanage::get_disks();
+ $testcount++;
+ is($smart_calls, $transport eq 'pcie' ? 1 : 0, "SMART queries for $transport");
+ }
+ $diskmanage_module->unmock('get_smart_data');
+ $diskmanage_module->unmock('get_nvme_transport');
+ }
+
if (-f "disk_tests/$testcasedir/multipathd_status_expected.json") {
my $status = PVE::Multipath::path_status_by_wwid();
my $expected_status =
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
` (14 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
NVMe/TCP namespaces need a manual nvme-cli setup on every node so
far. Add a storage type that connects one subsystem through each
configured portal and offers its namespaces as raw volumes. Native
NVMe multipath joins the paths and the kernel reconnects lost ones.
A second storage for a subsystem is refused: the LUN list, the
controller list and the node status attribute a subsystem to one
storage. Resolving the path of a volume checks an existing device
against the subsystem: a volume name can be the by-id name of any NVMe
device, and not every user of a path activates the volume first.
The plugin isolates its properties, so names like 'portals' belong to
this storage type alone, and a third-party plugin that declares one,
like 'subsysnqn', is not left out. As the first in-tree plugin that
isolates, it makes the storage API check the parameters for adding and
updating a storage of any type against the options of that type alone.
A third-party plugin then has to list the sensitive properties it uses
in options(), as the API change log says.
Local disk setup refuses namespaces attached over a fabric, as for
iSCSI LUNs. Wiping a namespace or creating a volume group on it is
refused while a storage serves images from its subsystem. nvme-cli
is only recommended, as no other storage type needs it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
ApiChangeLog | 7 +-
debian/control | 1 +
src/PVE/Diskmanage.pm | 2 +
src/PVE/Makefile | 1 +
src/PVE/NVMeOF.pm | 244 +++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 223 +++++++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 13 ++
src/test/Makefile | 6 +-
src/test/run_nvmeof_taint_tests.pl | 60 ++++++
src/test/run_nvmeof_tests.pl | 302 +++++++++++++++++++++++++++++
src/test/run_san_nvme_tests.pl | 16 ++
14 files changed, 875 insertions(+), 5 deletions(-)
create mode 100644 src/PVE/NVMeOF.pm
create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
create mode 100755 src/test/run_nvmeof_taint_tests.pl
create mode 100755 src/test/run_nvmeof_tests.pl
diff --git a/ApiChangeLog b/ApiChangeLog
index 10f39508..d7310fe4 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -101,9 +101,10 @@ Future changes should be documented in here.
declares a property of that name is then not loaded until it renames it, so prefer names that
are specific to the plugin, for example prefixed with its type.
- As soon as one such plugin is installed, the API checks the parameters for adding and updating a
- storage against the options of its type alone, for all storage types. So every plugin has to
- list the sensitive properties that it uses in `options()`.
+ Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for
+ adding and updating a storage against the options of its type alone, for all storage types,
+ also for plugins of older API versions. So every plugin has to list the sensitive properties
+ that it uses in `options()`.
Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of
version 11 also applies to the plugin's own properties of these names.
diff --git a/debian/control b/debian/control
index d10dfb23..8b9b6455 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
${misc:Depends},
${perl:Depends},
Recommends: multipath-tools,
+ nvme-cli,
pve-esxi-import-tools (>= 0.6.0),
skopeo,
zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 8fda7afc..1ad91e06 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -108,6 +108,8 @@ sub disk_is_used {
my $disklist = get_disks($dev, 1, 1);
die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+ die "'$disk' is not a valid local disk\n"
+ if ($disklist->{$dev}->{transport} // '') =~ /^nvme-(?:tcp|rdma|fc)$/;
return 1 if $disklist->{$dev}->{used};
return 0;
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..ebaedf9b
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,244 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+use PVE::Diskmanage;
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+
+sub canonical_address($addr) {
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ return inet_ntop(AF_INET, $packed);
+ }
+ if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+ return inet_ntop(AF_INET, substr($packed, 12))
+ if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+ return inet_ntop(AF_INET6, $packed);
+ }
+ return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+ my ($addr, $port);
+ if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } else {
+ return undef;
+ }
+ return undef if defined($port) && ($port < 1 || $port > 65535);
+ $addr = canonical_address($addr);
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0;
+ return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast;
+ } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+ my $prefix = unpack('n', $packed);
+ return undef
+ if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+ } else {
+ return undef;
+ }
+ return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+ 'pve-storage-nvme-portal',
+ sub($portal, $noerr = undef) {
+ return $portal if parse_portal($portal);
+ return undef if $noerr;
+ die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+ },
+);
+
+sub format_path($path) {
+ my $addr = $path->{traddr} =~ /:/ ? "[$path->{traddr}]" : $path->{traddr};
+ return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+ return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+ PVE::Tools::split_list($scfg->{portals})];
+}
+
+sub host_identity() {
+ my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+ my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+ /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+ if (!defined($nqn) || length($nqn) > 223) {
+ die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the"
+ . " host identity\n";
+ }
+ die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+ if !defined($id) || $id =~ /^[0-]+$/;
+ return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+ if !-e "$SYSFS/module/nvme_tcp";
+ my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+ die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+ dir_glob_foreach(
+ $dir,
+ 'nvme[0-9]+',
+ sub($name, @) {
+ my $address = file_read_firstline("$dir/$name/address") // return;
+ my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+ return if !defined($address{traddr});
+ $res->{$name} = {
+ transport => file_read_firstline("$dir/$name/transport") // '',
+ subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+ state => file_read_firstline("$dir/$name/state") // 'unknown',
+ traddr => canonical_address($address{traddr}),
+ trsvcid => $address{trsvcid},
+ };
+ },
+ );
+ return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+ my $controllers = controllers();
+ return {
+ map { $_ => $controllers->{$_} } grep {
+ $controllers->{$_}->{subsystem} eq $subsystem
+ && $controllers->{$_}->{transport} eq 'tcp'
+ } keys %$controllers
+ };
+}
+
+sub path_controller($controllers, $path) {
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) {
+ return $name;
+ }
+ }
+ return undef;
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-subsystem";
+ dir_glob_foreach(
+ $dir,
+ 'nvme-subsys[0-9]+',
+ sub($subsys, @) {
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+ return if $nqn ne $subsystem;
+ dir_glob_foreach(
+ "$dir/$subsys",
+ 'nvme[0-9]+n[0-9]+',
+ sub($dev, @) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+ },
+ );
+ },
+ );
+ return $res;
+}
+
+sub with_lock($code) {
+ my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+ 35,
+ sub {
+ make_path($RUN_DIR);
+ open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+ flock($lock, LOCK_EX | LOCK_NB)
+ or die "NVMe over Fabrics is busy - another operation is still running\n";
+ # A command still in the kernel must retain serialization after its caller times out.
+ my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+ fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+ or die "cannot inherit NVMe lock - $!\n";
+ return $code->();
+ },
+ );
+ die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+ return $res;
+}
+
+my sub retry_file($scfg, $path) {
+ my ($key) =
+ sha1_hex(join(
+ "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~
+ /\A([0-9a-f]{40})\z/;
+ return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+ my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+ my $age = time() - $mtime;
+ return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+ make_path("$RUN_DIR/retry");
+ file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub connect_path($scfg, $path) {
+ my ($nqn) = $scfg->{subsysnqn} =~ /\A($NQN_PATTERN)\z/;
+ if (!defined($nqn) || length($nqn) > 223 || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery') {
+ die "invalid subsystem NQN\n";
+ }
+ my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/;
+ die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+ return with_lock(sub {
+ return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ assert_support();
+ my $host = host_identity();
+ run_command(
+ [
+ $NVME,
+ 'connect',
+ '--transport=tcp',
+ '--nqn',
+ $nqn,
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $path->{trsvcid},
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--ctrl-loss-tmo=-1',
+ '--fast_io_fail_tmo',
+ $fail,
+ ],
+ timeout => 30,
+ );
+ # nvme-cli can exit successfully without having connected
+ die "no controller appeared\n"
+ if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ unlink(retry_file($scfg, $path));
+ return;
+ });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 1e9c670a..91283ff2 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin;
use PVE::Storage::NFSPlugin;
use PVE::Storage::CIFSPlugin;
use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
use PVE::Storage::RBDPlugin;
use PVE::Storage::CephFSPlugin;
use PVE::Storage::ISCSIDirectPlugin;
@@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register();
PVE::Storage::NFSPlugin->register();
PVE::Storage::CIFSPlugin->register();
PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
PVE::Storage::RBDPlugin->register();
PVE::Storage::CephFSPlugin->register();
PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
NFSPlugin.pm \
CIFSPlugin.pm \
ISCSIPlugin.pm \
+ NVMeOFPlugin.pm \
CephFSPlugin.pm \
RBDPlugin.pm \
ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..cb514c1d
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,223 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use v5.36;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Network;
+
+use PVE::Diskmanage;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+ return {
+ content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }],
+ select_existing => 1,
+ 'isolate-properties' => 1,
+ 'advanced-properties' => {
+ 'fast-io-fail-tmo' => 1,
+ },
+ 'hidden-properties' => {
+ bwlimit => 1,
+ },
+ 'sensitive-properties' => {},
+ };
+}
+
+sub properties {
+ return {
+ subsysnqn => {
+ title => "Subsystem",
+ description => 'The NQN of the NVMe subsystem.',
+ type => 'string',
+ pattern => $PVE::NVMeOF::NQN_PATTERN,
+ maxLength => 223,
+ },
+ portals => {
+ title => "Portals",
+ description =>
+ 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+ . ' with an optional port (default 4420).',
+ type => 'string',
+ format => 'pve-storage-nvme-portal-list',
+ },
+ 'fast-io-fail-tmo' => {
+ title => "Fail I/O After (s)",
+ description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+ . ' this timeout; I/O can remain queued while paths reconnect.',
+ type => 'integer',
+ minimum => -1,
+ default => 120,
+ },
+ };
+}
+
+sub options {
+ return {
+ subsysnqn => { fixed => 1 },
+ portals => {},
+ 'fast-io-fail-tmo' => { optional => 1 },
+ nodes => { optional => 1 },
+ disable => { optional => 1 },
+ content => { optional => 1 },
+ bwlimit => { optional => 1 },
+ };
+}
+
+my sub assert_valid_config($scfg) {
+ die "the discovery subsystem cannot be used as storage\n"
+ if $scfg->{subsysnqn} eq 'nqn.2014-08.org.nvmexpress.discovery';
+ my %seen;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ my $portal = PVE::NVMeOF::format_path($path);
+ die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+ }
+}
+
+sub on_add_hook($class, $storeid, $scfg, %param) {
+ assert_valid_config($scfg);
+ # the LUN and controller lists and the node status attribute a subsystem to its one storage
+ my $ids = PVE::Storage::config()->{ids};
+ for my $sid (sort keys %$ids) {
+ next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof';
+ die "storage '$sid' already uses this subsystem\n"
+ if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn};
+ }
+ return;
+}
+
+sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) {
+ my $merged = { %$scfg, %$update };
+ delete $merged->{$_} for ($delete // [])->@*;
+ assert_valid_config($merged);
+ return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname($class, $volname) {
+ if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) {
+ return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw');
+ }
+ die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a
+# namespace of another subsystem.
+sub assert_namespace($scfg, $volname, $real) {
+ my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!;
+ die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+ my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$dev/device/subsysnqn") // '';
+ die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+ if $nqn ne $scfg->{subsysnqn};
+}
+
+sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) {
+ die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+ my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+ my $path = "/dev/disk/by-id/$name";
+ # not every user of a path activates the volume first, and the base volume of an LVM storage
+ # has no device before its storage connects
+ my $real = Cwd::realpath($path);
+ assert_namespace($scfg, $volname, $real) if defined($real) && -e $real;
+ return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) {
+ my $res = $class->list_images($storeid, $scfg, $vmid);
+ $_->{content} = 'images' for @$res;
+ return $res;
+}
+
+sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) {
+ return [] if defined($vmid) && !$vollist;
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ my $res = [];
+ for my $dev (sort keys %$namespaces) {
+ my $ns = $namespaces->{$dev};
+ my $volid = "$storeid:nvme-$ns->{wwid}";
+ next if $vollist && !grep { $_ eq $volid } @$vollist;
+ push @$res,
+ {
+ volid => $volid,
+ format => 'raw',
+ size => $ns->{size},
+ vmid => 0,
+ };
+ }
+ return $res;
+}
+
+sub status($class, $storeid, $scfg, $cache = undef, @) {
+ my $active = grep { $_->{state} eq 'live' }
+ values PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ return (0, 0, 0, $active ? 1 : 0);
+}
+
+sub activate_storage($class, $storeid, $scfg, $cache = undef, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ eval {
+ die "no answer from portal\n"
+ if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::connect_path($scfg, $path);
+ };
+ if (my $err = $@) {
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ warn "storage '$storeid': connecting to '"
+ . PVE::NVMeOF::format_path($path)
+ . "' failed - $err";
+ }
+ }
+ return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection($class, $storeid, $scfg, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ return 1 if grep { $_->{state} eq 'live' } values %$controllers;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ }
+ return 0;
+}
+
+sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) {
+ my $path = $class->filesystem_path($scfg, $volname, $snapname);
+ # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device
+ my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10);
+ while (!-b $path && Time::HiRes::time() < $deadline) {
+ last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ Time::HiRes::usleep(250_000);
+ }
+ my $real = Cwd::realpath($path);
+ die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+ assert_namespace($scfg, $volname, $real);
+}
+
+sub volume_has_feature(
+ $class, $scfg, $feature, $storeid, $volname,
+ $snapname = undef,
+ $running = undef, @,
+) {
+ return 1 if $feature eq 'copy' && !defined($snapname);
+ return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 724c2d35..6e305580 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
);
our @SHARED_STORAGE = (
- 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+ 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
);
our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index b025c03f..86469f5c 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -298,6 +298,11 @@ sub lun_partitions($lun) {
}
sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+ my $name = basename($lun->{devpath});
+ my $nqn =
+ $name =~ /^nvme\d+n\d+$/
+ ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+ : undef;
my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
[$lun->{devpath}, lun_partitions($lun)->@*],
)->@*;
@@ -320,6 +325,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
) {
die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
}
+ if (
+ $type eq 'nvmeof'
+ && defined($nqn)
+ && $scfg->{subsysnqn} eq $nqn
+ && PVE::Storage::san_storage_serves_images($scfg)
+ ) {
+ die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+ }
next if $type ne 'lvm' || !$scfg->{base};
my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base});
my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/;
diff --git a/src/test/Makefile b/src/test/Makefile
index 279ba1ee..c88666f5 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
all: test
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api test_nvmeof
test_zfspoolplugin: run_test_zfspoolplugin.pl
./run_test_zfspoolplugin.pl
@@ -83,3 +83,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+ ./run_nvmeof_tests.pl
+ perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..62405291
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,60 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::More;
+use Test::MockModule;
+use PVE::NVMeOF;
+use PVE::Tools qw(file_set_contents file_get_contents);
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+ subsystem_controllers => sub {
+ return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+ },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+ run_command => sub($cmd, %opts) {
+ $commands++;
+ is(
+ scalar(grep { tainted($_) } @$cmd),
+ 0,
+ 'all connect operands are validated and untainted',
+ );
+ PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+ PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+ 'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+ eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+ like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..b75f7a5a
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,302 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Fcntl qw(:flock);
+use File::Path qw(make_path);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Tools qw(file_set_contents);
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ portals => '192.0.2.1,[2001:db8::2]:4421',
+ content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+ is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+ is_deeply(
+ [sort keys $class->properties()->%*],
+ [qw(fast-io-fail-tmo portals subsysnqn)],
+ 'only three plugin-specific properties',
+ );
+ my sub nvmeof_schema($schema) {
+ my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*;
+ my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types;
+ return $nvmeof->{properties} // {};
+ }
+ my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema());
+ is_deeply(
+ [sort grep { !$create->{$_}->{optional} } keys $create->%*],
+ [qw(portals subsysnqn)],
+ 'adding a storage requires the subsystem and the portals',
+ );
+ my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema());
+ ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed');
+ my $parsed = PVE::Storage::Plugin->parse_config(
+ 'storage.cfg',
+ "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n",
+ );
+ is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+ is_deeply(
+ $parsed->{ids}->{fabric}->{content},
+ { none => 1 },
+ 'connection-only content parses',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ { traddr => '192.0.2.1', trsvcid => 4420 },
+ 'default I/O port',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+ { traddr => '2001:db8::1', trsvcid => 4421 },
+ 'canonical IPv6',
+ );
+
+ for my $bad (
+ 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+ ) {
+ ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+ }
+ eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); };
+ like($@, qr/listed more than once/, 'duplicate portals refused');
+ eval {
+ $class->on_add_hook(
+ 'fabric',
+ { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+ );
+ };
+ like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $ids = { fabric => $cfg, local => { type => 'dir' } };
+ $storage->mock(config => sub { { ids => $ids } });
+ is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way');
+ eval { $class->on_add_hook('second', {%$cfg}) };
+ like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem');
+ is(
+ scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+ '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+ 'stable volume path',
+ );
+ {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $nqn = 'nqn.other';
+ $disk->redefine(read_subsysnqn => sub { $nqn });
+ my $check = sub {
+ eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) };
+ $@;
+ };
+ like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem');
+ like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type');
+ like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition');
+ $nqn = 'nqn.test';
+ is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem');
+ my $cwd = Test::MockModule->new('Cwd');
+ $cwd->redefine(realpath => sub { $dir });
+ eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') };
+ like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked');
+ }
+
+ for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+ eval { $class->parse_volname($bad) };
+ like($@, qr/unable to parse/, "invalid volume $bad");
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } });
+ is(
+ $class->list_volumes('fabric', $cfg)->[0]->{volid},
+ 'fabric:nvme-eui.abc',
+ 'content none still offers bases',
+ );
+ is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+ $nvme->mock(subsystem_controllers => sub { {} });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+ like($@, qr/not available/,
+ 'missing namespace fails without waiting on absent controllers');
+ my ($now, $queries) = (0, 0);
+ my $clock = Test::MockModule->new('Time::HiRes');
+ $clock->redefine(time => sub { $now += 5 }, usleep => sub { });
+ $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) };
+ is($queries, 1, 'a missing namespace of a connected subsystem is waited for');
+ eval {
+ $class->activate_volume(
+ 'fabric',
+ $cfg,
+ 'nvme-nonexistent-test',
+ undef,
+ { status_only => 1 },
+ );
+ };
+ like($@, qr/not available/, 'in a status pass it fails as well');
+ is($queries, 1, 'but without a wait');
+ is(
+ ($class->parse_volname('nvme-eui.0123456789abcdef'))[0],
+ 'vm-vol',
+ 'namespaces are VM disks',
+ );
+};
+
+subtest 'sysfs topology' => sub {
+ my $sys = $PVE::NVMeOF::SYSFS;
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1",
+ "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+ "$sys/block/nvme0n1",
+ );
+ my %attrs = (
+ address => 'traddr=192.0.2.1,trsvcid=4420',
+ transport => 'tcp',
+ subsysnqn => 'nqn.test',
+ state => 'connecting',
+ );
+ file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+ file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+ file_set_contents("$sys/block/nvme0n1/size", "8\n");
+ is(
+ PVE::NVMeOF::controllers()->{nvme1}->{state},
+ 'connecting',
+ 'reconnecting controller is still present',
+ );
+ is(
+ PVE::NVMeOF::path_controller(
+ PVE::NVMeOF::subsystem_controllers('nqn.test'),
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ ),
+ 'nvme1',
+ 'portal controller found',
+ );
+ is_deeply(
+ PVE::NVMeOF::namespaces('nqn.test'),
+ { nvme0n1 => { wwid => 'eui.abc', size => 4096 } },
+ 'namespace facts',
+ );
+ is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+ my $value = PVE::NVMeOF::with_lock(sub {
+ my $out = '';
+ PVE::Tools::run_command(
+ [
+ '/usr/bin/perl',
+ '-MFcntl=:flock',
+ '-e',
+ 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*"); die "not inherited" if !@held; open(my $fh, ">>", $ARGV[0]) or die $!; die "lock missing" if flock($fh, LOCK_EX|LOCK_NB); print "locked\\n";',
+ "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+ ],
+ timeout => 5,
+ outfunc => sub { $out .= $_[0] },
+ );
+ return $out;
+ });
+ is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+ open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+ flock($lock, LOCK_EX) or die $!;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'busy operation never enters callback');
+ close($lock);
+
+ my $tools = Test::MockModule->new('PVE::Tools');
+ my $timed_out = 1;
+ $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+ my $retained;
+ eval {
+ PVE::NVMeOF::with_lock(sub {
+ my ($path) =
+ grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+ my ($fd) = $path =~ m{/(\d+)$};
+ open($retained, '>&', $fd) or die $!;
+ return 1;
+ });
+ };
+ like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+ $timed_out = 0;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+ close($retained);
+ is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my @commands;
+ my $controllers = {};
+ $nvme->mock(with_lock => sub { $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity =>
+ sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+ $nvme->mock(subsystem_controllers => sub { $controllers });
+ my $appear = 1;
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ is($opts{timeout}, 30, 'bounded connect');
+ push @commands, $cmd;
+ my %arg =
+ map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+ my $controller =
+ { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' };
+ $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear;
+ },
+ );
+ $net->mock(tcp_ping => sub { 1 });
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 2, 'one connect per missing portal');
+ ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+ is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+ @commands = ();
+ $controllers = {
+ nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+ nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+ };
+ $class->activate_storage('fabric', $cfg);
+ is_deeply(\@commands, [],
+ 'existing controllers are never reconnected, regardless of state');
+ $class->deactivate_storage('fabric', $cfg);
+ is_deeply(\@commands, [], 'deactivation does not remove controllers');
+ $controllers = {};
+ my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+ PVE::NVMeOF::note_connect_failure($cfg, $path);
+ ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 1, 'only the other portal connects');
+ ($controllers, $appear) = ({}, 0);
+ my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+ my $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ $class->activate_storage('fabric', $cfg);
+ }
+ like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+ ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+ $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+ eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+ like($@, qr/not a valid local disk/,
+ "$transport refuses without storage configuration");
+ }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 51236f87..c74b638a 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
$disk->mock(verify_blockdev_path => sub { $device });
$disk->mock(is_partition => sub { 0 });
$disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
$disk->mock(is_iscsi_device => sub { 0 });
$disk->mock(is_mounted => sub { 0 });
$disk->mock(has_holder => sub { 0 });
@@ -78,6 +79,21 @@ $disk->mock(is_partition => sub { 0 });
$cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' };
eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
like($@, qr/base volume/, 'native base reference refuses');
+$cfg->{ids} =
+ { fabric =>
+ { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+ $@,
+ qr/serves images from this subsystem/,
+ 'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks');
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
$cfg->{ids} = {};
@order = @opens = ();
$tools->mock(file_get_contents => sub { 1024 });
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
` (13 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
An LVM storage only accepts an iSCSI LUN as base volume. Accept a
namespace of an NVMe over Fabrics storage too. Adding such an LVM
storage connects the subsystem, waits for the namespace and creates
the volume group on it, and activating it on another node connects
that node first.
The creation treats the namespace like any SAN LUN: it takes the
cluster-wide lock of the LUN and refuses a namespace that another
storage references.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 3 +--
src/test/run_san_nvme_tests.pl | 17 +++++++++++++++++
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index b9f7f34d..cccf3bb8 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -1194,9 +1194,8 @@ sub on_add_hook {
my $basecfg = PVE::Storage::storage_config($cfg, $baseid, 1);
die "base storage ID '$baseid' does not exist\n" if !$basecfg;
- # we only support iscsi for now
die "unsupported base type '$basecfg->{type}'"
- if $basecfg->{type} ne 'iscsi';
+ if $basecfg->{type} ne 'iscsi' && $basecfg->{type} ne 'nvmeof';
# Managed multipath requires the coalesced map or a partition of it, never a transient
# single-path PV.
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index c74b638a..7df849e3 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -133,4 +133,21 @@ is_deeply(
);
is_deeply(\@opens, [$link], 'VG creation holds the native by-id device');
+$cfg->{ids} = { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } };
+my @activation;
+$storage->mock(path => sub { $link });
+$storage->mock(activate_storage => sub { push @activation, 'storage' });
+$storage->mock(activate_volumes => sub { push @activation, 'volume' });
+$lvm->mock(
+ lvm_create_volume_group => sub {
+ is($_[0], $link, 'add hook passes native base');
+ is($_[3]->{exempt}, 'data', 'add hook exempts new storage');
+ },
+);
+PVE::Storage::LVMPlugin->on_add_hook(
+ 'data',
+ { base => 'fabric:nvme-eui.0123456789abcdef', vgname => 'data', shared => 1 },
+);
+is_deeply(\@activation, ['storage', 'volume'], 'LVM activates the NVMe base before creation');
+
done_testing();
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (2 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
` (12 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Setting up an NVMe over Fabrics storage needs the NQN of the
subsystem and its portals. Add /nodes/{node}/scan/nvmeof, which asks
the discovery service at a portal for the NVMe/TCP subsystems it
offers to the node and lists them with their portals.
Without a port, it tries the discovery port 8009 and then the I/O
port 4420, where Linux targets answer discovery too. The scan needs
Datastore.Allocate on /storage, like the iSCSI scan.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Storage/Scan.pm | 49 ++++++++++++++++++++++
src/PVE/NVMeOF.pm | 65 ++++++++++++++++++++++++++++++
src/test/run_nvmeof_taint_tests.pl | 5 +++
src/test/run_nvmeof_tests.pl | 54 +++++++++++++++++++++++++
4 files changed, 173 insertions(+)
diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm
index cf19f835..84848847 100644
--- a/src/PVE/API2/Storage/Scan.pm
+++ b/src/PVE/API2/Storage/Scan.pm
@@ -6,6 +6,7 @@ use warnings;
# NOTE: This API endpoints are mounted by pve-manager's API2::Node module and pvesm CLI
use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
use PVE::RESTHandler;
use PVE::RPCEnvironment;
use PVE::SafeSyslog;
@@ -50,6 +51,7 @@ __PACKAGE__->register_method({
{ method => 'iscsi' },
{ method => 'lvm' },
{ method => 'nfs' },
+ { method => 'nvmeof' },
{ method => 'pbs' },
{ method => 'san-luns' },
{ method => 'zfs' },
@@ -302,6 +304,53 @@ __PACKAGE__->register_method({
},
});
+__PACKAGE__->register_method({
+ name => 'nvmeofscan',
+ path => 'nvmeof',
+ method => 'GET',
+ description =>
+ 'List the NVMe over TCP subsystems offered to this node by a discovery controller.',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/storage', ['Datastore.Allocate']] },
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ portal => {
+ description => 'IPv4 or bracketed IPv6 discovery address with an optional port.'
+ . ' Without a port, try 8009 then 4420.',
+ type => 'string',
+ format => 'pve-storage-nvme-portal',
+ },
+ },
+ },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ portals => {
+ type => 'string',
+ description => 'The discovered TCP portals, comma separated.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my ($param) = @_;
+ my $res = PVE::NVMeOF::discover($param->{portal});
+ return [
+ map { {
+ subsysnqn => $_,
+ portals => join(',', map { PVE::NVMeOF::format_path($_) } $res->{$_}->@*),
+ } } sort keys %$res
+ ];
+ },
+});
+
__PACKAGE__->register_method({
name => 'lvmscan',
path => 'lvm',
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index ebaedf9b..f9a293d9 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -5,9 +5,11 @@ use v5.36;
use Digest::SHA qw(sha1_hex);
use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
use File::Path qw(make_path);
+use JSON qw(decode_json);
use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
use PVE::JSONSchema;
+use PVE::Network;
use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
use PVE::Diskmanage;
@@ -241,4 +243,67 @@ sub connect_path($scfg, $path) {
});
}
+sub parse_discovery_log($raw) {
+ my $log = decode_json($raw);
+ die "invalid NVMe discovery log\n"
+ if ref($log) ne 'HASH' || ref($log->{records}) ne 'ARRAY';
+ my (%seen, %res);
+ for my $record ($log->{records}->@*) {
+ next
+ if ($record->{subtype} // '') ne 'nvme subsystem' || ($record->{trtype} // '') ne 'tcp';
+ my $nqn = $record->{subnqn} // next;
+ if (
+ $nqn !~ /^$NQN_PATTERN$/
+ || length($nqn) > 223
+ || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery'
+ ) {
+ next;
+ }
+ my $addr = $record->{traddr} // '';
+ my $port = $record->{trsvcid} // '';
+ my $path = parse_portal(($addr =~ /:/ ? "[$addr]" : $addr) . ":$port") // next;
+ next if $seen{$nqn}->{ format_path($path) }++;
+ push $res{$nqn}->@*, $path;
+ }
+ return \%res;
+}
+
+sub discover($portal) {
+ my $path = parse_portal($portal) // die "invalid portal '$portal'\n";
+ my @ports = $portal =~ /:[0-9]+$/ ? ($path->{trsvcid}) : (8009, 4420);
+ my $error;
+ for my $port (@ports) {
+ my $result = eval {
+ die "no answer from portal\n" if !PVE::Network::tcp_ping($path->{traddr}, $port, 2);
+ with_lock(sub {
+ assert_support();
+ my $host = host_identity();
+ my $out = '';
+ run_command(
+ [
+ $NVME,
+ 'discover',
+ '--transport=tcp',
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $port,
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--output-format=json',
+ ],
+ timeout => 30,
+ outfunc => sub($line) { $out .= "$line\n" },
+ );
+ return parse_discovery_log($out);
+ });
+ };
+ $error = $@;
+ return $result if !$error;
+ }
+ die "discovery at portal '$portal' failed - $error";
+}
+
1;
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
index 62405291..c6546c30 100755
--- a/src/test/run_nvmeof_taint_tests.pl
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -38,6 +38,7 @@ $nvme->mock(
'all connect operands are validated and untainted',
);
PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ $opts{outfunc}->('{"records":[]}') if $opts{outfunc};
},
);
my $path = PVE::NVMeOF::parse_portal($portal);
@@ -53,6 +54,10 @@ for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.disc
eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
}
+my $net = Test::MockModule->new('PVE::Network');
+$net->mock(tcp_ping => sub { 1 });
+eval { PVE::NVMeOF::discover($portal) };
+is($@, '', 'discovery can execute with file-backed host identity in taint mode');
file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
eval { PVE::NVMeOF::host_identity() };
like($@, qr/nonzero UUID/, 'zero host ID refused');
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index b75f7a5a..ffbaa45f 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -5,6 +5,7 @@ use lib qw(..);
use Fcntl qw(:flock);
use File::Path qw(make_path);
use File::Temp qw(tempdir);
+use JSON qw(encode_json);
use Test::MockModule;
use Test::More;
@@ -12,6 +13,7 @@ use PVE::Tools qw(file_set_contents);
use PVE::NVMeOF;
use PVE::Storage;
+use PVE::API2::Storage::Scan;
my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
local $PVE::NVMeOF::RUN_DIR = "$dir/run";
@@ -299,4 +301,56 @@ subtest 'local disk setup rejects fabric namespaces by transport' => sub {
}
};
+subtest 'TCP discovery, fallback and API result' => sub {
+ my $record = {
+ subtype => 'nvme subsystem',
+ trtype => 'tcp',
+ subnqn => 'nqn.test',
+ traddr => '192.0.2.1',
+ trsvcid => '4420',
+ };
+ my $raw = encode_json({
+ records =>
+ [$record, $record, { %$record, trtype => 'rdma' }, { %$record, trsvcid => 70000 }],
+ });
+ is_deeply(
+ PVE::NVMeOF::parse_discovery_log($raw),
+ { 'nqn.test' => [{ traddr => '192.0.2.1', trsvcid => 4420 }] },
+ 'TCP only, unique valid portals',
+ );
+ for my $bad ('garbage', '{}', '[]') {
+ eval { PVE::NVMeOF::parse_discovery_log($bad) };
+ ok($@, 'malformed discovery is not an empty success');
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my (@ports, @commands);
+ our $locked;
+ $nvme->mock(with_lock => sub { local $locked = 1; $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'id' } });
+ $net->mock(tcp_ping => sub { push @ports, $_[1]; $_[1] != 8009 });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'discovery holds the NVMe lock');
+ is($opts{timeout}, 30, 'discovery command timeout');
+ push @commands, $cmd;
+ $opts{outfunc}->($raw);
+ },
+ );
+ my $rows = PVE::API2::Storage::Scan->nvmeofscan({ node => 'n1', portal => '192.0.2.1' });
+ is_deeply(\@ports, [8009, 4420], 'unspecified port tries discovery then I/O port');
+ is_deeply($rows, [{ subsysnqn => 'nqn.test', portals => '192.0.2.1:4420' }], 'API shape');
+ eval {
+ PVE::JSONSchema::validate(
+ $rows,
+ PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns},
+ );
+ };
+ is($@, '', 'API schema accepts result');
+ @ports = ();
+ PVE::NVMeOF::discover('[2001:db8::1]:4421');
+ is_deeply(\@ports, [4421], 'explicit port does not fall back');
+};
+
done_testing();
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (3 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
` (11 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Add 'pvesm scan nvmeof <portal>', which prints the subsystems a
portal offers to the node with their portals, like the iSCSI scan
prints targets. That gives the values for a new storage on the
command line, with the usual output formats for scripts.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/CLI/pvesm.pm | 13 +++++++++++++
src/test/run_nvmeof_tests.pl | 12 ++++++++++++
2 files changed, 25 insertions(+)
diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm
index 8c3a2eda..b60b733b 100755
--- a/src/PVE/CLI/pvesm.pm
+++ b/src/PVE/CLI/pvesm.pm
@@ -736,6 +736,19 @@ our $cmddef = {
$print_api_result,
$PVE::RESTHandler::standard_output_options,
],
+ nvmeof => [
+ "PVE::API2::Storage::Scan",
+ 'nvmeofscan',
+ ['portal'],
+ { node => $nodename },
+ sub {
+ my ($data, $schema, $options) = @_;
+ PVE::CLIFormatter::print_api_result(
+ $data, $schema, [qw(subsysnqn portals)], $options,
+ );
+ },
+ $PVE::RESTHandler::standard_output_options,
+ ],
'san-luns' => [
"PVE::API2::Storage::Scan",
'sanlunscan',
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index ffbaa45f..90b34857 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -14,6 +14,7 @@ use PVE::Tools qw(file_set_contents);
use PVE::NVMeOF;
use PVE::Storage;
use PVE::API2::Storage::Scan;
+use PVE::CLI::pvesm;
my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
local $PVE::NVMeOF::RUN_DIR = "$dir/run";
@@ -351,6 +352,17 @@ subtest 'TCP discovery, fallback and API result' => sub {
@ports = ();
PVE::NVMeOF::discover('[2001:db8::1]:4421');
is_deeply(\@ports, [4421], 'explicit port does not fall back');
+ my $formatter = Test::MockModule->new('PVE::CLIFormatter');
+ my $schema = PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns};
+ $formatter->mock(
+ print_api_result => sub {
+ is($_[0], $rows, 'CLI forwards complete discovery rows');
+ is($_[1], $schema, 'CLI forwards discovery schema');
+ is($_[3]->{'output-format'}, 'json', 'CLI supports structured output');
+ },
+ );
+ $PVE::CLI::pvesm::cmddef->{scan}->{nvmeof}->[4]
+ ->($rows, $schema, { 'output-format' => 'json' });
};
done_testing();
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (4 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
` (10 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The SAN LUN inventory lists NVMe namespaces with their subsystem, but
without relation to an NVMe over Fabrics storage. Expect one path per
portal of the storage that connects the subsystem of a namespace, so
that a namespace with fewer paths shows as degraded.
A namespace counts as used by the LVM storage that has it as base
volume, else by the storage of its subsystem if that one serves the
namespaces as images.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Storage/Scan.pm | 9 +++++----
src/PVE/Storage/SAN.pm | 16 +++++++++++++---
src/test/run_san_lun_scan_tests.pl | 28 ++++++++++++++++++++++++++++
3 files changed, 46 insertions(+), 7 deletions(-)
diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm
index 84848847..7d3fd242 100644
--- a/src/PVE/API2/Storage/Scan.pm
+++ b/src/PVE/API2/Storage/Scan.pm
@@ -499,8 +499,9 @@ __PACKAGE__->register_method({
'paths-expected' => {
description => "The number of paths the LUN should have on this node: the"
. " highest of the count of its multipath entry and, for each managed"
- . " iSCSI storage of the LUN, the number of its portals. Absent if no"
- . " count is known.",
+ . " iSCSI storage of the LUN, the number of its portals. For a namespace"
+ . " of an NVMe over Fabrics storage, the number of its portals. Absent if"
+ . " no count is known.",
type => 'integer',
optional => 1,
},
@@ -579,8 +580,8 @@ __PACKAGE__->register_method({
},
'used-by-storage' => {
description => "The storage that uses the LUN: as its base volume or"
- . " through the volume group on it, or as an iSCSI storage that serves the"
- . " LUNs of its target as images.",
+ . " through the volume group on it, or as an iSCSI or NVMe over Fabrics"
+ . " storage that serves the LUNs of its target or subsystem as images.",
type => 'string',
optional => 1,
},
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 86469f5c..abf992d9 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -16,6 +16,7 @@ use PVE::Multipath;
use PVE::Multipath::Config;
use PVE::Multipath::Generator;
use PVE::Multipath::Identity;
+use PVE::NVMeOF;
use PVE::Storage;
use PVE::Storage::LVMPlugin;
@@ -42,7 +43,8 @@ sub scan_luns($cfg) {
PVE::Multipath::Config::wwids_for_node($mpcfg, PVE::INotify::nodename())->@*;
my %listed = map { $_ => 1 } PVE::Multipath::list_etc_multipath_wwids()->@*;
my $derived = PVE::Multipath::Generator::derived_wwids($cfg, $ids);
- my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %targets, %paths, %unmapped);
+ my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %nvme_owner, %targets, %paths);
+ my %unmapped;
for my $sid (sort keys %$derived) {
for my $wwid (keys $derived->{$sid}->{wwids}->%*) {
@@ -61,7 +63,8 @@ sub scan_luns($cfg) {
$target_owner{ $scfg->{target} } //= $sid
if defined($scfg->{target}) && PVE::Storage::san_storage_serves_images($scfg);
$base_owner{$1} //= $sid
- if ($scfg->{base} // '') =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi)-(.+)$/;
+ if ($scfg->{base} // '') =~ m/(?:^|[.:])(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/;
+ $nvme_owner{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
}
my %rows;
@@ -106,6 +109,10 @@ sub scan_luns($cfg) {
$row->{'nvme-subsystem'} = $nqn;
my $nsid = file_read_firstline("/sys/block/$name/nsid");
$row->{'nvme-nsid'} = int($nsid) if defined($nsid) && $nsid =~ /^\d+$/;
+ if (my $sid = $nvme_owner{$nqn}) {
+ $row->{'paths-expected'} =
+ scalar(PVE::NVMeOF::storage_paths($cfg->{ids}->{$sid})->@*);
+ }
}
}
for my $key (qw(transport vendor model serial)) {
@@ -205,9 +212,12 @@ sub scan_luns($cfg) {
$row->{'paths-expected'},
);
}
+ my $nvme = $nvme_owner{ $row->{'nvme-subsystem'} // '' };
+ undef $nvme if $nvme && !PVE::Storage::san_storage_serves_images($cfg->{ids}->{$nvme});
my ($target) =
map { $target_owner{$_} // () } sort keys(($targets{ $row->{wwid} } // {})->%*);
- my $owner = $base_owner{ $row->{wwid} } // $vg_owner{ $row->{vgname} // '' } // $target;
+ my $owner = $base_owner{ $row->{wwid} } // $vg_owner{ $row->{vgname} // '' } // $target
+ // $nvme;
$row->{'used-by-storage'} = $owner if defined($owner);
}
return [map { $rows{$_} } sort keys %rows];
diff --git a/src/test/run_san_lun_scan_tests.pl b/src/test/run_san_lun_scan_tests.pl
index a98e2563..06db1202 100755
--- a/src/test/run_san_lun_scan_tests.pl
+++ b/src/test/run_san_lun_scan_tests.pl
@@ -304,6 +304,12 @@ is($rows->[0]->{devpath}, '/dev/sdc', 'the row names the path that its by-id lin
$disks = { nvme0n1 => { devpath => '/dev/nvme0n1', size => 100, transport => 'nvme-fc' } };
$scan->mock(file_read_firstline => sub { $_[0] =~ m{/nsid$} ? 1 : 'eui.0123456789abcdef' });
$disk->mock(read_subsysnqn => sub { 'nqn.test' });
+$cfg->{ids}->{fabric} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ portals => '192.0.2.1,192.0.2.2',
+ content => { images => 1 },
+};
$scan->unmock('stable_path');
$scan->mock(abs_path => sub { '/dev/nvme0n1' });
$rows = PVE::Storage::SAN::scan_luns($cfg);
@@ -321,6 +327,28 @@ eval { PVE::JSONSchema::validate($rows, $schema) };
is($@, '', 'native row matches the API schema');
is($rows->[0]->{'nvme-subsystem'}, 'nqn.test', 'subsystem NQN exposed');
is($rows->[0]->{'nvme-nsid'}, 1, 'namespace ID exposed');
+is($rows->[0]->{'used-by-storage'}, 'fabric', 'a storage serving the namespaces uses them');
+is($rows->[0]->{'paths-expected'}, 2, 'one expected path per configured portal');
+
+for my $count (1, 3) {
+ $config->mock(
+ read_config => sub { {
+ ids => { 'eui.0123456789abcdef' => { type => 'wwid', 'paths-expected' => $count } },
+ } },
+ );
+ my $entry = PVE::Storage::SAN::scan_luns($cfg);
+ is($entry->[0]->{'paths-expected'}, 2, "a multipath entry does not change it ($count)");
+}
+$config->mock(read_config => sub { { ids => { '3600map' => { type => 'wwid' } } } });
+{
+ local $cfg->{ids}->{fabric}->{content} = { none => 1 };
+ my $plain = PVE::Storage::SAN::scan_luns($cfg);
+ is($plain->[0]->{'used-by-storage'}, undef, 'a connection storage does not');
+ is($plain->[0]->{'paths-expected'}, 2, 'but still tells the expected paths');
+}
+$cfg->{ids}->{data}->{base} = 'fabric:nvme-eui.0123456789abcdef';
+$rows = PVE::Storage::SAN::scan_luns($cfg);
+is($rows->[0]->{'used-by-storage'}, 'data', 'LVM base consumer takes precedence');
$disk->mock(is_partition => sub { 1 });
ok(!PVE::Storage::SAN::is_san_device('/dev/sdb1'), 'a partition is never a LUN');
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (5 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
` (9 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage. The
namespaces of all subsystems are counted in one pass for that list, as
a node can have a thousand controllers, for example with a storage
plugin that uses a subsystem per guest disk.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace that is mounted, has holders,
is open in a process or is referenced by a storage would lose its
last usable path. A controller with namespaces outside of native
NVMe multipath is refused, as they lack the head devices that these
checks go by. Reading needs Sys.Audit on the node or
Datastore.Allocate on /storage, the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 182 +++++++++++++++++++++++++++++++
src/PVE/Diskmanage.pm | 16 +--
src/PVE/NVMeOF.pm | 71 ++++++++++++
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 178 ++++++++++++++++++++++++++++++
src/test/run_nvmeof_tests.pl | 7 ++
7 files changed, 449 insertions(+), 9 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index c3066fa0..e5411bdb 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..71b4423d
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,182 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme[0-9]+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my %owners;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my $counts = PVE::NVMeOF::namespace_counts();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts->{$nqn} // 0,
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties =>
+ { node => get_standard_option('pve-node'), controller => $controller_schema },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub($param) {
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded($param->{controller});
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 1ad91e06..05bed420 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -940,6 +940,13 @@ sub get_multipath_disks {
return $disklist;
}
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -975,15 +982,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index f9a293d9..bce63768 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -165,6 +165,21 @@ sub namespaces($subsystem) {
return $res;
}
+# The number of namespaces of each subsystem of this node, from one pass over the subsystems.
+sub namespace_counts() {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-subsystem";
+ dir_glob_foreach(
+ $dir,
+ 'nvme-subsys[0-9]+',
+ sub($subsys, @) {
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+ dir_glob_foreach("$dir/$subsys", 'nvme[0-9]+n[0-9]+', sub { $res->{$nqn}++ });
+ },
+ );
+ return $res;
+}
+
sub with_lock($code) {
my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
35,
@@ -306,4 +321,60 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+sub disconnect_guarded($controller) {
+ my ($name) = $controller =~ /\A(nvme[0-9]+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ # Without native multipath a namespace is a device of its controller and has no head
+ # device, which the checks below go by.
+ my $local = 0;
+ dir_glob_foreach(
+ "$SYSFS/class/nvme-fabrics/ctl/$name",
+ "${name}n[0-9]+",
+ sub { $local = 1 },
+ );
+ die "cannot disconnect '$name' - its namespaces do not use native NVMe multipath\n"
+ if $local;
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ my @others = grep { $_->{device} ne "/dev/$name" } @$paths;
+ next if grep { PVE::Diskmanage::nvme_path_is_usable($_) } @others;
+ my $ns = $namespaces->{$head} // die
+ "cannot disconnect '$name' - namespace '$head' has no device on this node to check for use\n";
+ my $device = "/dev/$head";
+ if (PVE::Diskmanage::is_mounted($device) || PVE::Diskmanage::has_holder($device)) {
+ die "cannot disconnect '$name' - namespace '$head' is in use and has no other"
+ . " usable path\n";
+ }
+ my $opener = PVE::Diskmanage::device_opener($device, glob("${device}p[0-9]*"));
+ if (defined($opener)) {
+ die "cannot disconnect '$name' - process $opener has namespace '$head' open and"
+ . " it has no other usable path\n";
+ }
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index c88666f5..ac4265b4 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -84,6 +84,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..8c881ee6
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,178 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Storage::SAN;
+use PVE::API2::NVMeOF;
+
+my $sys = tempdir('nvmeof-api-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::SYSFS = $sys;
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status =>
+ sub { {
+ nvme0n1 => { paths => [{ device => '/dev/nvme1', state => 'live' }, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ my $opener;
+ $disk->mock(device_opener => sub { $opener });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ $opener = '4711 (kvm)';
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/process 4711 \(kvm\) has namespace 'nvme0n1' open/,
+ 'an open namespace refuses');
+ $opener = undef;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'unused namespace can lose its final path');
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1",
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme0c1n1",
+ );
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/do not use native NVMe multipath/, 'a namespace without head device refuses');
+ is($commands, 3, 'and nothing is disconnected');
+ remove_tree("$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1");
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 4, 'the path device of native multipath does not');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(
+ namespaces => sub { die "must not walk the subsystems for each controller\n" },
+ namespace_counts => sub { { 'nqn.test' => 2, 'nqn.other' => 5 } },
+ );
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{namespaces}, 2, 'namespaces counted for the subsystem of the controller');
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index 90b34857..aca76d7a 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -188,6 +188,13 @@ subtest 'sysfs topology' => sub {
'namespace facts',
);
is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+ make_path(map { "$sys/class/nvme-subsystem/nvme-subsys1/nvme1n$_" } 1 .. 3);
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys1/subsysnqn", "nqn.other\n");
+ is_deeply(
+ PVE::NVMeOF::namespace_counts(),
+ { 'nqn.test' => 1, 'nqn.other' => 3 },
+ 'namespaces counted per subsystem',
+ );
};
subtest 'serialized commands inherit the lock' => sub {
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (6 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
` (8 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The status report of a node has its iSCSI initiator name, so that
the cluster-wide status shows the identities an admin needs for the
access rules of an array in one place. Add the NVMe host NQN, which
an NVMe over Fabrics target needs for the same purpose. A node
without a valid NVMe host identity reports none.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Multipath.pm | 2 +-
src/PVE/Multipath/Status.pm | 5 ++++-
src/test/run_multipath_api_tests.pl | 12 +++++++++++-
3 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index 8e60f70a..e69919c2 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -94,7 +94,7 @@ my $node_schema = {
. ' base volume of a storage or to hold its volume group.',
},
map { $_ => { type => 'string', optional => 1 } }
- qw(error apply-error iscsi-initiator-name),
+ qw(error apply-error iscsi-initiator-name nvme-host-nqn),
},
};
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index 8e4b4eb7..ae386164 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -52,6 +52,9 @@ sub update_node_status() {
my $initiator = eval { PVE::Storage::ISCSIPlugin::read_initiator_info() };
$report->{'iscsi-initiator-name'} = $initiator->{name}
if $initiator && defined($initiator->{name});
+ require PVE::NVMeOF;
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ $report->{'nvme-host-nqn'} = $host->{nqn} if $host;
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running or does not respond\n" if !PVE::Multipath::is_running();
@@ -171,7 +174,7 @@ sub cluster_status() {
$entry->{time} = $report->{time} if defined($report->{time});
my $offline = exists($members->{$node}) && !$members->{$node}->{online};
next if $offline || !defined($report->{time}) || time() - $report->{time} > 180;
- my @keys = qw(error apply-error iscsi-initiator-name other-maps);
+ my @keys = qw(error apply-error iscsi-initiator-name nvme-host-nqn other-maps);
$entry->{$_} = $report->{$_} for grep { defined($report->{$_}) } @keys;
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
$unhinted{$node} = 1 if $report->{'vgs-unknown'};
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 6955076b..078ba66d 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -15,6 +15,7 @@ use Test::More;
use Test::MockModule;
use PVE::API2::Multipath;
use PVE::Multipath::Generator;
+use PVE::NVMeOF;
use PVE::Storage;
subtest 'explicit entries are ordinary configuration writes' => sub {
@@ -205,9 +206,12 @@ subtest 'an unconfigured node does not write drop-ins or reload' => sub {
}
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
$iscsi->mock(read_initiator_info => sub { {} });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $report;
$status->mock(publish => sub { $report = $_[0] });
+
for my $ids ({}, { plain => { type => 'iscsi', target => 'iqn.test' } }) {
$storage->mock(config => sub { { ids => $ids } });
PVE::Multipath::Status::update_node_status();
@@ -383,11 +387,15 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.2025-01.example:node', id => 'unused' } });
$mp->mock(is_supported => sub { 0 });
PVE::Multipath::Status::update_node_status();
is($report->{error}, 'multipath-tools is not installed', 'missing tools reason only');
+ is($report->{'nvme-host-nqn'}, 'nqn.2025-01.example:node', 'host NQN reported in any case');
is_deeply(\@order, [qw(apply publish)], 'single call owns ordering');
is($report->{'apply-error'}, 'reload failed', 'apply failure is separate');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
$mp->mock(is_supported => sub { 1 });
$mp->mock(is_running => sub { 0 });
PVE::Multipath::Status::update_node_status();
@@ -396,6 +404,7 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
'multipathd is not running or does not respond',
'stopped daemon reason only',
);
+ ok(!exists($report->{'nvme-host-nqn'}), 'no host NQN without a valid host identity');
$mp->mock(is_running => sub { 1 });
$mp->mock(get_maps => sub { die "bad JSON\n" });
PVE::Multipath::Status::update_node_status();
@@ -450,7 +459,7 @@ subtest 'API returns final node and LUN states' => sub {
},
],
}),
- n2 => encode_json({ time => $NOW, luns => [] }),
+ n2 => encode_json({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_json({ time => $NOW - 181, luns => [] }),
n4 => encode_json({ time => $NOW, luns => [] }),
n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
@@ -481,6 +490,7 @@ subtest 'API returns final node and LUN states' => sub {
'final node states',
);
is($res->{nodes}->{n3}->{time}, $NOW - 181, 'old report time retained');
+ is($res->{nodes}->{n2}->{'nvme-host-nqn'}, 'nqn.test:n2', 'host NQN passed on');
my $lun = $res->{luns}->[0];
is($lun->{state}, 'missing', 'worst known state');
is($lun->{nodes}->{n1}->{state}, 'optimal', 'observed healthy map');
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (7 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
` (7 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the LUNs that multipathd handles on
each node. The namespaces of an NVMe over Fabrics storage use the
native multipath of the kernel and were missing from it, so a lost
path or a node without access to a namespace only showed in the LUN
list of that node.
Let each node report the namespaces of the subsystems such a storage
names, with the path counts its LUN list already shows, and expect
them on the nodes of the storage like the LUNs of an iSCSI storage.
They do not depend on multipathd, so a node without it reports them
as well and only the state of its maps stays unknown.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Multipath.pm | 11 +++-
src/PVE/Multipath/Status.pm | 58 +++++++++++++++---
src/test/run_multipath_api_tests.pl | 95 ++++++++++++++++++++++++++++-
3 files changed, 152 insertions(+), 12 deletions(-)
diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index e69919c2..51ef7281 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -130,14 +130,21 @@ __PACKAGE__->register_method({
wwid => { type => 'string', description => 'The LUN WWID.' },
explicit =>
{ type => 'boolean', description => 'An explicit entry exists.' },
+ native => {
+ type => 'boolean',
+ optional => 1,
+ description => 'An NVMe namespace, which uses the native'
+ . ' multipath of the kernel.',
+ },
name => { type => 'string', optional => 1 },
'used-by-storage' => {
type => 'string',
optional => 1,
description =>
'The storage that uses the LUN: as its base volume or through'
- . ' the volume group on it, or as an iSCSI storage that serves'
- . ' the LUNs of its target as images.',
+ . ' the volume group on it, or as an iSCSI or NVMe over Fabrics'
+ . ' storage that serves the LUNs of its target or subsystem as'
+ . ' images.',
},
'multipath-storage' => {
type => 'string',
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index ae386164..d26eb7f8 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -7,6 +7,7 @@ use JSON qw(decode_json);
use PVE::Cluster;
use PVE::INotify;
+use PVE::Diskmanage;
use PVE::Multipath;
use PVE::Multipath::Config;
use PVE::Multipath::Generator;
@@ -24,6 +25,39 @@ sub worst_state(@states) {
return $worst;
}
+# Namespaces use the native multipath of the kernel, so their state does not depend on multipathd.
+my sub namespace_luns($pv_vgs) {
+ require PVE::Storage;
+ my $cfg = PVE::Storage::config();
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %luns;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ next if $scfg->{type} ne 'nvmeof';
+ my $expected = eval { scalar(PVE::NVMeOF::storage_paths($scfg)->@*) };
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ for my $name (sort keys %$namespaces) {
+ my $wwid = $namespaces->{$name}->{wwid};
+ next if $luns{$wwid};
+ my $paths = $status->{$name} // { active => 0, paths => [] };
+ my $total = scalar($paths->{paths}->@*);
+ my $lun = $luns{$wwid} = {
+ wwid => $wwid,
+ native => 1,
+ size => $namespaces->{$name}->{size},
+ storages => [$sid],
+ 'paths-total' => $total,
+ 'paths-active' => $paths->{active},
+ state => PVE::Multipath::derive_health($paths->{active}, $total, $expected),
+ };
+ $lun->{'paths-expected'} = $expected if defined($expected);
+ my $vg = ($pv_vgs // {})->{"/dev/$name"};
+ $lun->{vgname} = $vg if defined($vg);
+ }
+ }
+ return [map { $luns{$_} } sort keys %luns];
+}
+
sub publish($report) {
my $node = PVE::INotify::nodename();
my $now = time();
@@ -55,11 +89,11 @@ sub update_node_status() {
require PVE::NVMeOF;
my $host = eval { PVE::NVMeOF::host_identity() };
$report->{'nvme-host-nqn'} = $host->{nqn} if $host;
+ require PVE::Storage::LVMPlugin;
+ my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running or does not respond\n" if !PVE::Multipath::is_running();
- require PVE::Storage::LVMPlugin;
- my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
# A node can have many maps that something else set up, for example a storage plugin with
# one LUN per guest disk. Only the maps that the node manages, that are the base volume of a
# storage or hold its volume group get a row, the others are counted.
@@ -119,8 +153,11 @@ sub update_node_status() {
if (my $error = $@) {
chomp $error;
$report->{error} = $error;
- delete $report->{luns};
+ $report->{luns} = [];
}
+ my $namespaces = eval { namespace_luns($pv_vgs) };
+ warn "multipath: cannot query NVMe namespaces - $@" if $@;
+ push $report->{luns}->@*, @$namespaces if $namespaces;
eval { publish($report) };
warn "multipath: cannot publish node status - $@" if $@;
}
@@ -133,7 +170,7 @@ sub cluster_status() {
my $all_nodes = PVE::Cluster::get_nodelist();
my $members = PVE::Cluster::get_members();
my $raw = PVE::Cluster::get_node_kv($KEY);
- my (%nodes, %luns, %expected, %observed, %vg_owner);
+ my (%nodes, %luns, %expected, %observed, %vg_owner, %lists);
# the nodes that list a map because the configuration names it for them
my (%managed, %unhinted);
my $storage_nodes = sub($scfg) {
@@ -158,12 +195,13 @@ sub cluster_status() {
my $scfg = $storage->{ids}->{$sid};
$vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
next if $scfg->{type} ne 'lvm';
- my ($wwid) = ($scfg->{base} // '') =~ /(?:^|\.)dm-uuid-(?:part\d+-)?mpath-(\S+)$/;
+ my ($base, $volname) = PVE::Storage::parse_volume_id($scfg->{base} // '', 1);
+ my ($kind, $wwid) = ($volname // '') =~ /(?:^|\.)(dm-uuid-(?:part\d+-)?mpath|nvme)-(\S+)$/;
next if !defined($wwid);
$luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $luns{$wwid}->{native} = 1 if $kind eq 'nvme';
$luns{$wwid}->{'used-by-storage'} //= $sid;
$managed{$wwid}->{$_} = 1 for @$all_nodes;
- my ($base) = $scfg->{base} =~ /^([^:]+):/;
$luns{$wwid}->{'multipath-storage'} //= $base if $requests_maps->($base);
$expected{$wwid}->{$_} = 1 for $storage_nodes->($scfg)->@*;
}
@@ -178,11 +216,13 @@ sub cluster_status() {
$entry->{$_} = $report->{$_} for grep { defined($report->{$_}) } @keys;
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
$unhinted{$node} = 1 if $report->{'vgs-unknown'};
- next if $entry->{state} ne 'ok';
+ # an unavailable node lists its namespaces only, or nothing
+ $lists{$node} = 1 if $report->{luns};
for my $lun (($report->{luns} // [])->@*) {
my $wwid = $lun->{wwid} // next;
my $row = $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $row->{native} = 1 if $lun->{native};
$row->{$_} //= $lun->{$_} for grep { defined($lun->{$_}) } qw(name size);
my $owner = defined($lun->{vgname}) ? $vg_owner{ $lun->{vgname} } : undef;
$row->{'used-by-storage'} //= $owner if defined($owner);
@@ -216,10 +256,10 @@ sub cluster_status() {
if ($observed{$wwid}->{$node}) {
$row->{nodes}->{$node} = $observed{$wwid}->{$node};
} elsif ($expected{$wwid}->{$node}) {
- my $listed = $nodes{$node}->{state} eq 'ok';
+ my $listed = $nodes{$node}->{state} eq 'ok' || ($row->{native} && $lists{$node});
# without the hint file of LVM a node leaves a map out that it only knows through
# the volume group of a storage
- $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node};
+ $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node} && !$row->{native};
$row->{nodes}->{$node} = { state => $listed ? 'missing' : 'unknown' };
}
}
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 078ba66d..fd63503a 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -383,7 +383,12 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $mp = Test::MockModule->new('PVE::Multipath');
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $disks = Test::MockModule->new('PVE::Diskmanage');
my ($report, @order);
+ my $ids = {};
+ $storage->mock(config => sub { { ids => $ids } });
+ $disks->mock(get_nvme_path_status => sub { { nvme1n1 => { active => 1, paths => [{}] } } });
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
@@ -421,6 +426,42 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
is($report->{luns}->[0]->{'paths-expected'}, 3, 'expected count reported');
is($report->{luns}->[0]->{state}, 'degraded', 'fewer paths than expected');
ok(!exists($report->{supported}), 'no separate capability state');
+
+ $ids->{nvme} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test:sub',
+ portals => '192.0.2.1,192.0.2.2',
+ };
+ $nvme->mock(
+ namespaces => sub($nqn) {
+ return $nqn eq 'nqn.test:sub'
+ ? { nvme1n1 => { wwid => 'uuid.1', size => 4096 } }
+ : {};
+ },
+ );
+ my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+ $lvm->mock(cached_pv_volume_groups => sub { { '/dev/nvme1n1' => 'vgns' } });
+ my $namespace = {
+ wwid => 'uuid.1',
+ native => 1,
+ size => 4096,
+ storages => ['nvme'],
+ vgname => 'vgns',
+ 'paths-total' => 1,
+ 'paths-active' => 1,
+ 'paths-expected' => 2,
+ state => 'degraded',
+ };
+ PVE::Multipath::Status::update_node_status();
+ is_deeply($report->{luns}->[1], $namespace, 'namespace with fewer paths than portals');
+ $mp->mock(is_running => sub { 0 });
+ PVE::Multipath::Status::update_node_status();
+ is(
+ $report->{error},
+ 'multipathd is not running or does not respond',
+ 'stopped daemon still reported',
+ );
+ is_deeply($report->{luns}, [$namespace], 'namespaces do not need multipathd');
};
subtest 'API returns final node and LUN states' => sub {
@@ -457,12 +498,31 @@ subtest 'API returns final node and LUN states' => sub {
'paths-active' => 2,
storages => ['scoped'],
},
+ {
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'optimal',
+ 'paths-total' => 2,
+ 'paths-active' => 2,
+ storages => ['nvme'],
+ },
],
}),
n2 => encode_json({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_json({ time => $NOW - 181, luns => [] }),
n4 => encode_json({ time => $NOW, luns => [] }),
- n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
+ n5 => encode_json({
+ time => $NOW,
+ error => 'multipath-tools is not installed',
+ luns => [{
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'degraded',
+ 'paths-total' => 2,
+ 'paths-active' => 1,
+ storages => ['nvme'],
+ }],
+ }),
);
$cluster->mock(get_node_kv => sub { \%raw });
my $entries = { '3600a' => { type => 'wwid' } };
@@ -480,6 +540,8 @@ subtest 'API returns final node and LUN states' => sub {
{ type => 'iscsi', 'managed-multipath' => 1, nodes => { departed => 1 } },
lvmgone => { type => 'lvm', base => 'gone:0.0.0.dm-uuid-mpath-3600e' },
scoped => { type => 'iscsi', 'managed-multipath' => 1, nodes => { n2 => 1 } },
+ nvme => { type => 'nvmeof', subsysnqn => 'nqn.test:sub' },
+ lvmns => { type => 'lvm', base => 'nvme:nvme-uuid.2', nodes => { n5 => 1 } },
},
} },
);
@@ -541,6 +603,37 @@ subtest 'API returns final node and LUN states' => sub {
'an entry for other nodes does not change that',
);
}
+ my $namespace = $lun{'uuid.1'};
+ ok($namespace->{native}, 'namespace marked as native');
+ ok(!exists($namespace->{'multipath-storage'}), 'a namespace has no requested map');
+ is_deeply(
+ { map { $_ => $namespace->{nodes}->{$_}->{state} } keys $namespace->{nodes}->%* },
+ {
+ n1 => 'optimal',
+ n2 => 'missing',
+ n3 => 'unknown',
+ n4 => 'unknown',
+ n5 => 'degraded',
+ },
+ 'namespace state of a node without multipathd counts',
+ );
+ is($lun->{nodes}->{n5}->{state}, 'unknown', 'no map state from a node without multipathd');
+ is_deeply(
+ $lun{'uuid.2'},
+ {
+ wwid => 'uuid.2',
+ explicit => 0,
+ native => 1,
+ 'used-by-storage' => 'lvmns',
+ state => 'missing',
+ nodes => { n5 => { state => 'missing' } },
+ },
+ 'base namespace that a node without multipathd does not report',
+ );
+ $raw{n5} = encode_json({ time => $NOW, error => 'too many LUNs to report' });
+ $res = PVE::API2::Multipath->status({});
+ %lun = map { $_->{wwid} => $_ } $res->{luns}->@*;
+ is($lun{'uuid.2'}->{nodes}->{n5}->{state}, 'unknown', 'node without a list');
eval {
PVE::JSONSchema::validate(
$res,
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (8 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
` (6 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of the
storage type.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
pve-storage-multipath.adoc | 4 +-
pve-storage-nvmeof.adoc | 167 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 20 +++--
3 files changed, 183 insertions(+), 8 deletions(-)
create mode 100644 pve-storage-nvmeof.adoc
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index d74fa23..caeae9b 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node.
{pve} configures maps cluster-wide and shows their state under *Datacenter ->
Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its
LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe
-namespaces use kernel-native multipath instead; the procedures below cover SCSI
-LUNs.
+namespaces use kernel-native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs.
Prerequisites
~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..aa464b6
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,167 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+ package recommends it, but installations without recommended packages need it
+ added separately. Its installation creates the node's host NQN in
+ `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+ image share them. Give each a new identity before it connects, with
+ `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+ *Datacenter -> Storage -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+ addresses that every node can reach; host names are not accepted.
+. Use a storage network that only trusted hosts can reach. The connections use
+ neither authentication nor TLS.
+. Keep native NVMe multipath enabled, which is the kernel default
+ (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach
+ the array and select *Guided Setup -> LVM on NVMe over Fabrics*.
+. Enter a discovery portal, select the subsystem, check the portals and name
+ the connection storage. This storage connects to the subsystem with
+ `content none`, while the LVM storage manages guest disks. Continuing creates
+ or reuses the connection storage before you select a namespace.
+. Select the namespace, choose the storage ID and volume group, then confirm.
+ Leave *Nodes* empty for all nodes.
+. Check the state of the namespace on each node of the storage, which the
+ setup shows once the storage is added, as described for LUNs in
+ xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+ check its access to the namespace on the array before placing guests on the
+ storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace reads *OK (1 path)*.
+Portals can also be changed later in the edit dialog of the connection storage;
+each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection
+storage. Enable *Use namespaces directly* there to use whole namespaces as guest
+disks, or add an *LVM* storage on a namespace later, with the connection storage
+as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <connection> --subsysnqn <nqn> --portals <portals> --content none
+# pvesm list <connection>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<connection>:nvme-<wwid>`. Adding
+the LVM storage creates the volume group if it does not exist yet. Creating it
+is refused if the namespace holds data that {pve} recognizes, such as
+partitions, a file system or a volume group, or if another storage references
+it. To reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later, and only one storage
+can use a subsystem.
+
+portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace have been lost for
+that long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+ subsysnqn nqn.2010-06.com.example:array1
+ portals 192.0.2.10,192.0.2.11
+ content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types |Image formats |Shared |Snapshots |Clones
+|images none |raw |yes |no |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace there to inspect its paths.
+Its *State* is *Degraded* while a path is unusable or fewer paths exist than
+the storage has portals, and *Failed* without a usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal:
+*Connected*, or *Reconnecting* while the kernel retries a lost path.
+
+An enabled storage connects the portals a node has no controller for. A portal
+that failed is retried after a minute. Removing a portal, or disabling or
+removing the storage, does not disconnect anything. To remove such a path,
+select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and
+press *Disconnect*, or run
+`pvesh delete /nodes/<node>/nvmeof/controller/<controller>`. This is refused if
+a namespace that is mounted, in use by another device, open in a process such
+as a running guest, or referenced by a storage would lose its last usable path.
+The process check does not find one that opened the namespace through another
+device node. A controller with namespaces that do not use native NVMe multipath
+is refused as well. A storage that still lists the portal connects to it again.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS.
+* NVMe/FC namespaces need no storage of this type, and the guided setups and
+ the cluster-wide status do not cover them. *Node -> SAN* lists them. Create
+ their volume group under *Node -> Disks -> LVM*; with *Add Storage* enabled,
+ this adds a shared LVM storage.
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index f7de231..5cf0df2 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
|LVM-thin |lvmthin |block |no |yes |yes
|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes
|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes
+|NVMe-oF |nvmeof |block |yes^3^|yes^4^ |yes
|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes
|Ceph/RBD |rbd |block |yes |yes |yes
|ZFS over iSCSI |zfs |block |yes |yes |yes
@@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
hours. If your setup is affected, create and remove snapshots while the VM is
shut down, expecting a long task duration.
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster.
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
These snapshots use separate volumes for the snapshot data and layer them. For
@@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have
totally different content.
Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use *Guided Setup* under
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use *Guided Setup* under
*Datacenter -> Storage -> SAN* instead. It creates the storages that are needed
-and enables multipath for the LUN.
+and enables multipath for a LUN.
Storage Pools
@@ -506,6 +510,8 @@ See Also
* link:/wiki/Storage:_Multipath[Storage: Multipath]
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
* link:/wiki/Storage:_LVM[Storage: LVM]
* link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -550,6 +556,8 @@ include::pve-storage-iscsidirect.adoc[]
include::pve-storage-multipath.adoc[]
+include::pve-storage-nvmeof.adoc[]
+
include::pve-storage-rbd.adoc[]
include::pve-storage-cephfs.adoc[]
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (9 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
` (5 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Make the node status, the controller list and the disconnect call of
pve-storage reachable at /nodes/{node}/nvmeof for the web UI.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/Nodes.pm | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 49039bc30..0cfb7b79c 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -54,6 +54,7 @@ use PVE::API2::Hardware;
use PVE::API2::ISCSI;
use PVE::API2::LXC::Status;
use PVE::API2::LXC;
+use PVE::API2::NVMeOF;
use PVE::API2::Network;
use PVE::API2::Network::SDN::Nodes::Status;
use PVE::API2::NodeConfig;
@@ -167,6 +168,11 @@ __PACKAGE__->register_method({
path => 'iscsi',
});
+__PACKAGE__->register_method({
+ subclass => "PVE::API2::NVMeOF",
+ path => 'nvmeof',
+});
+
__PACKAGE__->register_method({
subclass => "PVE::API2::Capabilities",
path => 'capabilities',
@@ -259,6 +265,7 @@ __PACKAGE__->register_method({
{ name => 'migrateall' },
{ name => 'netstat' },
{ name => 'network' },
+ { name => 'nvmeof' },
{ name => 'qemu' },
{ name => 'query-oci-repo-tags' },
{ name => 'query-url-metadata' },
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (10 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
` (4 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Show NVMe over Fabrics storages with their subsystem in the storage
list, and add a dialog for them. The dialog lists the subsystems that a
discovery portal reports and fills in the portals of the selected one,
which can be changed. As in the iSCSI dialog, the discovery can run on
another node, since the node serving the web UI may not be allowed on
the array. The node of the discovery is then preselected as the node of
the storage.
The subsystem selector builds on the iSCSI target selector, which lets a
subclass set its store fields and sort key.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/Utils.js | 6 ++
www/manager6/dc/StorageView.js | 5 +-
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/NVMeOFEdit.js | 137 +++++++++++++++++++++++++++++
5 files changed, 151 insertions(+), 3 deletions(-)
create mode 100644 www/manager6/storage/NVMeOFEdit.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 1644e1d41..f8a1a2698 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -378,6 +378,7 @@ JSSRC= \
storage/LvmThinEdit.js \
storage/BTRFSEdit.js \
storage/NFSEdit.js \
+ storage/NVMeOFEdit.js \
storage/PBSEdit.js \
storage/RBDEdit.js \
storage/StatusView.js \
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index 64269755d..ef8218918 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -886,6 +886,12 @@ Ext.define('PVE.Utils', {
faIcon: 'building',
backups: false,
},
+ nvmeof: {
+ name: 'NVMe over Fabrics',
+ ipanel: 'NVMeOFInputPanel',
+ faIcon: 'building',
+ backups: false,
+ },
cephfs: {
name: 'CephFS',
ipanel: 'CephFSInputPanel',
diff --git a/www/manager6/dc/StorageView.js b/www/manager6/dc/StorageView.js
index f0f298102..4d7e93dfd 100644
--- a/www/manager6/dc/StorageView.js
+++ b/www/manager6/dc/StorageView.js
@@ -247,7 +247,9 @@ Ext.define(
sortable: true,
dataIndex: 'path',
renderer: function (value, metaData, record) {
- return Ext.htmlEncode(record.data.target || value);
+ return Ext.htmlEncode(
+ record.data.target || record.data.subsysnqn || value,
+ );
},
},
{
@@ -298,6 +300,7 @@ Ext.define(
'server',
'portal',
'target',
+ 'subsysnqn',
'export',
'storage',
{ name: 'shared', type: 'boolean' },
diff --git a/www/manager6/storage/IScsiEdit.js b/www/manager6/storage/IScsiEdit.js
index c20aa2896..14f5d83f3 100644
--- a/www/manager6/storage/IScsiEdit.js
+++ b/www/manager6/storage/IScsiEdit.js
@@ -5,6 +5,7 @@ Ext.define('PVE.storage.IScsiScan', {
queryParam: 'portal',
valueField: 'target',
displayField: 'target',
+ scanFields: ['target', 'portal'],
matchFieldWidth: false,
allowBlank: false,
@@ -53,13 +54,13 @@ Ext.define('PVE.storage.IScsiScan', {
}
let store = Ext.create('Ext.data.Store', {
- fields: ['target', 'portal'],
+ fields: me.scanFields,
proxy: {
type: 'proxmox',
url: `${me.apiBaseUrl}${me.nodename}${me.apiSuffix}`,
},
});
- store.sort('target', 'ASC');
+ store.sort(me.valueField, 'ASC');
Ext.apply(me, {
store: store,
diff --git a/www/manager6/storage/NVMeOFEdit.js b/www/manager6/storage/NVMeOFEdit.js
new file mode 100644
index 000000000..5d816cbbe
--- /dev/null
+++ b/www/manager6/storage/NVMeOFEdit.js
@@ -0,0 +1,137 @@
+Ext.define('PVE.storage.NVMeOFScan', {
+ extend: 'PVE.storage.IScsiScan',
+ alias: 'widget.pveNVMeOFScan',
+
+ valueField: 'subsysnqn',
+ displayField: 'subsysnqn',
+ scanFields: ['subsysnqn', 'portals'],
+
+ listConfig: {
+ width: 650,
+ columns: [
+ {
+ dataIndex: 'subsysnqn',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+ emptyText: PVE.Utils.renderNotFound(gettext('Subsystem')),
+ },
+
+ config: {
+ apiSuffix: '/scan/nvmeof',
+ },
+});
+
+Ext.define('PVE.storage.NVMeOFInputPanel', {
+ extend: 'PVE.panel.StorageBase',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ onlineHelp: 'storage_nvmeof',
+
+ namespaceContent: 'images',
+
+ onGetValues: function (values) {
+ let me = this;
+
+ values.content = values.namespaces ? me.namespaceContent : 'none';
+ delete values.namespaces;
+
+ return me.callParent([values]);
+ },
+
+ // a storage can be restricted to some volume types, keep its content types then
+ setValues: function (values) {
+ let me = this;
+
+ values.namespaces = PVE.Utils.contentAllows(values.content, 'vm-vol');
+ if (values.namespaces) {
+ me.namespaceContent = Ext.Array.from(values.content).join(',');
+ }
+ me.callParent([values]);
+ },
+
+ column1: [
+ {
+ xtype: 'textfield',
+ name: 'discovery-portal',
+ fieldLabel: gettext('Discovery Portal'),
+ allowBlank: false,
+ submitValue: false,
+ cbind: {
+ hidden: '{!isCreate}',
+ disabled: '{!isCreate}',
+ },
+ listeners: {
+ change: {
+ fn: function (field, value) {
+ let scan = this.up('inputpanel').lookup('subsystemScan');
+ scan.setDisabled(!value);
+ scan.setPortal(value);
+ scan.setValue('');
+ },
+ buffer: 500,
+ },
+ },
+ },
+ {
+ xtype: 'checkbox',
+ name: 'namespaces',
+ boxLabel: gettext('Use namespaces directly'),
+ },
+ ],
+
+ // NQNs and portal lists are longer than a column is wide
+ columnB: [
+ {
+ cbind: {
+ xtype: (get) => (get('isCreate') ? 'pveNVMeOFScan' : 'displayfield'),
+ disabled: '{isCreate}',
+ },
+ name: 'subsysnqn',
+ value: '',
+ fieldLabel: gettext('Subsystem'),
+ allowBlank: false,
+ reference: 'subsystemScan',
+ listeners: {
+ change: function (field, value) {
+ let rec = field
+ .getStore?.()
+ .findRecord('subsysnqn', value, 0, false, true, true);
+ if (rec) {
+ field
+ .up('inputpanel')
+ .down('field[name=portals]')
+ .setValue(rec.get('portals'));
+ }
+ },
+ nodechanged: function (value) {
+ this.up('inputpanel').lookup('storageNodeRestriction').setValue(value);
+ },
+ },
+ },
+ {
+ xtype: 'textfield',
+ name: 'portals',
+ fieldLabel: gettext('Portals'),
+ allowBlank: false,
+ },
+ ],
+
+ advancedColumn1: [
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 'fast-io-fail-tmo',
+ fieldLabel: gettext('Fail I/O After (s)'),
+ minValue: -1,
+ emptyText: '120',
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext('Applies to newly connected paths. -1 disables the timeout.'),
+ },
+ },
+ ],
+});
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (11 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
` (3 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
A namespace of an NVMe over Fabrics storage can hold the volume group
of an LVM storage like an iSCSI LUN, so list such storages in the base
storage selector as well and name the type of each.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/storage/LVMEdit.js | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/www/manager6/storage/LVMEdit.js b/www/manager6/storage/LVMEdit.js
index d3f068166..a8c9bba00 100644
--- a/www/manager6/storage/LVMEdit.js
+++ b/www/manager6/storage/LVMEdit.js
@@ -70,10 +70,13 @@ Ext.define('PVE.storage.BaseStorageSelector', {
let store = Ext.create('Ext.data.Store', {
autoLoad: {
addRecords: true,
- params: {
- type: 'iscsi',
- },
},
+ filters: [
+ {
+ filterFn: (rec) =>
+ !rec.get('storage') || ['iscsi', 'nvmeof'].includes(rec.get('type')),
+ },
+ ],
fields: [
'storage',
'type',
@@ -82,7 +85,8 @@ Ext.define('PVE.storage.BaseStorageSelector', {
name: 'text',
convert: function (value, record) {
if (record.data.storage) {
- return record.data.storage + ' (iSCSI)';
+ let type = PVE.Utils.format_storage_type(record.data.type);
+ return `${record.data.storage} (${type})`;
} else {
return me.existingGroupsText;
}
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (12 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
` (2 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Add an NVMe over Fabrics entry to the guided setup of the SAN view, with
the same steps as for iSCSI. It connects to a subsystem found at a
discovery portal and shows the host NQNs of all nodes for the access
rules of the array.
The target step is shared with iSCSI, as only the storage type, its scan
and the node identities differ. The add dialog of NVMe over Fabrics
storages names the SAN view, as those of iSCSI and LVM do.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 2 +-
www/manager6/SANUtils.js | 1 +
www/manager6/dc/SAN.js | 2 +-
www/manager6/storage/NVMeOFEdit.js | 3 +
www/manager6/storage/wizard/LunSelect.js | 34 ++--
www/manager6/storage/wizard/NodeIdentities.js | 45 +++---
.../storage/wizard/{IScsi.js => Target.js} | 153 ++++++++++++------
www/manager6/storage/wizard/Wizard.js | 43 ++---
www/manager6/window/SANLunState.js | 23 ++-
9 files changed, 203 insertions(+), 103 deletions(-)
rename www/manager6/storage/wizard/{IScsi.js => Target.js} (61%)
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index f8a1a2698..1fbce8c4d 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -390,7 +390,7 @@ JSSRC= \
storage/wizard/Wizard.js \
storage/wizard/LunSelect.js \
storage/wizard/NodeIdentities.js \
- storage/wizard/IScsi.js \
+ storage/wizard/Target.js \
storage/wizard/FibreChannel.js \
Workspace.js \
# end of JSSRC list
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index c2df2d1ad..1b2b25c12 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -27,6 +27,7 @@ Ext.define('pve-san-lun', {
'iscsi-target',
'iscsi-lun',
'nvme-subsystem',
+ 'nvme-nsid',
// one column shows the target of a LUN and the subsystem of a namespace
{
name: 'target',
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index e9a5e5ccc..fcddc3e2e 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -126,7 +126,7 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Guided Setup'),
disabled: !canSetup,
- menu: [setupItem('iscsi'), setupItem('fc', !canModify)],
+ menu: [setupItem('iscsi'), setupItem('fc', !canModify), setupItem('nvmeof')],
},
];
me.items = [
diff --git a/www/manager6/storage/NVMeOFEdit.js b/www/manager6/storage/NVMeOFEdit.js
index 5d816cbbe..306cbea3d 100644
--- a/www/manager6/storage/NVMeOFEdit.js
+++ b/www/manager6/storage/NVMeOFEdit.js
@@ -29,6 +29,9 @@ Ext.define('PVE.storage.NVMeOFInputPanel', {
onlineHelp: 'storage_nvmeof',
+ // TRANSLATORS: "Guided Setup" is a button of the SAN view, {0} the menu path of that view.
+ sanNote: gettext('To set up shared LVM on a namespace, use Guided Setup in {0}.'),
+
namespaceContent: 'images',
onGetValues: function (values) {
diff --git a/www/manager6/storage/wizard/LunSelect.js b/www/manager6/storage/wizard/LunSelect.js
index 834a3412c..9da0c195b 100644
--- a/www/manager6/storage/wizard/LunSelect.js
+++ b/www/manager6/storage/wizard/LunSelect.js
@@ -30,10 +30,11 @@ Ext.define('PVE.storage.wizard.LunSelect', {
// The scan names one target per LUN, also for a LUN reached through several. The content
// listing has the LUNs of the connection, and a LUN that shows up later is listed once the
// scan names the target of the connection for it.
+ let targetKey = wizard.wizardType === 'nvmeof' ? 'nvme-subsystem' : 'iscsi-target';
let filter = {
id: 'connection',
filterFn: (rec) =>
- rec.get('iscsi-target') === connection.target ||
+ rec.get(targetKey) === connection.target ||
(me.connectionWwids ?? []).includes(rec.get('wwid')),
};
me.store.addFilter(filter);
@@ -45,7 +46,7 @@ Ext.define('PVE.storage.wizard.LunSelect', {
success: function (response) {
if (!me.destroyed) {
me.connectionWwids = response.result.data
- .map((entry) => entry.volid.match(/\.dm-uuid-mpath-(.+)$/)?.[1])
+ .map((entry) => entry.volid.match(/(?:\.dm-uuid-mpath-|:nvme-)(.+)$/)?.[1])
.filter(Boolean);
me.store.addFilter(filter);
}
@@ -71,7 +72,9 @@ Ext.define('PVE.storage.wizard.LunSelect', {
sortAfterUpdate: true,
});
let inUse = (row) => row['used-by-storage'] || row['used-by-process'];
- let noLunsText = gettext('No LUNs found. Check connections and array access.');
+ let noLunsText = me.nvme
+ ? gettext('No namespaces found. Check connections and array access.')
+ : gettext('No LUNs found. Check connections and array access.');
let updateSelection = function () {
let row = me.getSelection();
me.down('#selection').setValue(row && !inUse(row) ? row.wwid : '');
@@ -88,9 +91,13 @@ Ext.define('PVE.storage.wizard.LunSelect', {
xtype: 'component',
cls: 'pmx-hint',
margin: '0 0 10 0',
- html: gettext(
- 'A LUN that is in use cannot be selected. A LUN with data that is no longer needed has to be wiped under Node > SAN first.',
- ),
+ html: me.nvme
+ ? gettext(
+ 'A namespace that is in use cannot be selected. A namespace with data that is no longer needed has to be wiped under Node > SAN first.',
+ )
+ : gettext(
+ 'A LUN that is in use cannot be selected. A LUN with data that is no longer needed has to be wiped under Node > SAN first.',
+ ),
},
...(me.rescanNote
? [
@@ -123,6 +130,7 @@ Ext.define('PVE.storage.wizard.LunSelect', {
renderer: Ext.htmlEncode,
},
{ text: 'LUN', dataIndex: 'iscsi-lun', width: 60 },
+ { text: 'NSID', dataIndex: 'nvme-nsid', width: 60 },
{
text: gettext('Model'),
dataIndex: 'model',
@@ -140,6 +148,8 @@ Ext.define('PVE.storage.wizard.LunSelect', {
text: gettext('Multipath'),
dataIndex: 'multipath',
width: 170,
+ // native NVMe multipath needs no setup per namespace
+ hidden: me.nvme,
renderer: PVE.SANUtils.renderSanLunMultipath,
},
{
@@ -186,7 +196,11 @@ Ext.define('PVE.storage.wizard.LunSelect', {
let grid = me.down('grid');
if (grid?.rendered) {
grid.setEmptyText(
- query ? gettext('No LUNs match the search.') : noLunsText,
+ query
+ ? me.nvme
+ ? gettext('No namespaces match the search.')
+ : gettext('No LUNs match the search.')
+ : noLunsText,
);
}
},
@@ -202,8 +216,10 @@ Ext.define('PVE.storage.wizard.LunSelect', {
let search = me.down('pveRecordSearchField');
me.on('activate', function () {
search.setTargetStore(me.store);
- if (me.up('window').wizardType === 'fc') {
- me.down('gridcolumn[dataIndex=iscsi-lun]').hide();
+ let type = me.up('window').wizardType;
+ me.down('gridcolumn[dataIndex=iscsi-lun]').setHidden(type !== 'iscsi');
+ me.down('gridcolumn[dataIndex=nvme-nsid]').setHidden(type !== 'nvmeof');
+ if (type === 'fc') {
me.store.addFilter({
id: 'transport',
filterFn: (rec) => ['fc', 'fcoe'].includes(rec.get('transport')),
diff --git a/www/manager6/storage/wizard/NodeIdentities.js b/www/manager6/storage/wizard/NodeIdentities.js
index a0ef62fee..ed0ad8357 100644
--- a/www/manager6/storage/wizard/NodeIdentities.js
+++ b/www/manager6/storage/wizard/NodeIdentities.js
@@ -1,7 +1,7 @@
Ext.define('pve-san-node-identity', {
extend: 'Ext.data.Model',
idProperty: 'node',
- fields: ['node', 'state', 'iscsi-initiator-name'],
+ fields: ['node', 'state', 'iscsi-initiator-name', 'nvme-host-nqn'],
});
Ext.define('PVE.storage.wizard.NodeIdentities', {
@@ -10,21 +10,8 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
height: 220,
disableSelection: true,
viewConfig: { enableTextSelection: true },
- columns: [
- { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
- {
- text: gettext('State'),
- dataIndex: 'state',
- width: 175,
- renderer: PVE.SANUtils.renderState,
- },
- {
- text: gettext('Initiator Name'),
- dataIndex: 'iscsi-initiator-name',
- flex: 1,
- renderer: Ext.htmlEncode,
- },
- ],
+
+ identity: 'iscsi-initiator-name',
// Lists only the given nodes, or all nodes for none, which Copy then follows.
setNodes: function (nodes) {
@@ -39,6 +26,24 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
initComponent: function () {
let me = this;
+ let nvme = me.identity === 'nvme-host-nqn';
+ me.columns = [
+ { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
+ {
+ text: gettext('State'),
+ dataIndex: 'state',
+ width: 175,
+ renderer: PVE.SANUtils.renderState,
+ // the state of multipathd, which NVMe does not use
+ hidden: nvme,
+ },
+ {
+ text: nvme ? gettext('Host NQN') : gettext('Initiator Name'),
+ dataIndex: me.identity,
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ];
me.rstore = Ext.create('Proxmox.data.UpdateStore', {
storeid: 'san-wizard-identities',
model: 'pve-san-node-identity',
@@ -67,17 +72,19 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
me.tbar = [
{
xtype: 'tbtext',
- text: gettext('Allow the initiator names of the nodes that will use the target.'),
+ text: nvme
+ ? gettext('Allow the host NQNs of the nodes that will use the target.')
+ : gettext('Allow the initiator names of the nodes that will use the target.'),
},
'->',
{
- text: gettext('Copy Initiator Names'),
+ text: nvme ? gettext('Copy Host NQNs') : gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
handler: () =>
PVE.Utils.copyToClipboard(
me.store
.getRange()
- .map((rec) => rec.get('iscsi-initiator-name'))
+ .map((rec) => rec.get(me.identity))
.filter(Boolean)
.join('\n'),
),
diff --git a/www/manager6/storage/wizard/IScsi.js b/www/manager6/storage/wizard/Target.js
similarity index 61%
rename from www/manager6/storage/wizard/IScsi.js
rename to www/manager6/storage/wizard/Target.js
index d23cf3bb2..3fd0912fd 100644
--- a/www/manager6/storage/wizard/IScsi.js
+++ b/www/manager6/storage/wizard/Target.js
@@ -1,13 +1,14 @@
-Ext.define('PVE.storage.wizard.IScsiTarget', {
+Ext.define('PVE.storage.wizard.Target', {
extend: 'Proxmox.panel.InputPanel',
- xtype: 'pveStorageWizardIScsiTarget',
- onlineHelp: 'storage_open_iscsi',
+ xtype: 'pveStorageWizardTarget',
+
+ storageType: 'iscsi',
updateConnectionField: function () {
let me = this;
let target = me.down('field[name=target]').getValue();
let storage = me.storages?.find(
- (entry) => entry.type === 'iscsi' && entry.target === target,
+ (entry) => entry.type === me.storageType && entry[me.targetKey] === target,
);
let field = me.down('field[name=connection]');
if (storage) {
@@ -20,16 +21,20 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
onSubmit: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
let wizard = me.up('window');
let values = me.getValues();
- let params = {
- type: 'iscsi',
- storage: values.connection,
- portal: values.portal,
- target: values.target,
- content: 'none',
- 'managed-multipath': 1,
- };
+ let params = { type: me.storageType, storage: values.connection, content: 'none' };
+ if (nvme) {
+ params.subsysnqn = values.target;
+ params.portals = values.portals;
+ } else {
+ Object.assign(params, {
+ portal: values.portal,
+ target: values.target,
+ 'managed-multipath': 1,
+ });
+ }
let nodes = []
.concat(values.nodes || [])
.filter(Boolean)
@@ -61,29 +66,41 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
}
me.storages = response.result.data;
me.updateConnectionField();
- let users = me.storages.filter((entry) => entry.target === values.target);
+ let users = me.storages.filter((entry) => entry[me.targetKey] === values.target);
if (users.length) {
let base = users[0];
let conflict;
if (users.length !== 1) {
conflict = Ext.String.format(
- gettext(
- 'Storages {0} use this target. The setup can only reuse a target that one storage uses.',
- ),
+ nvme
+ ? gettext(
+ 'Storages {0} use this subsystem. The setup can only reuse a subsystem that one storage uses.',
+ )
+ : gettext(
+ 'Storages {0} use this target. The setup can only reuse a target that one storage uses.',
+ ),
users.map((entry) => entry.storage).join(', '),
);
- } else if (base.type !== 'iscsi') {
+ } else if (base.type !== me.storageType) {
conflict = Ext.String.format(
- gettext(
- 'Storage "{0}" uses this target. The setup can only reuse an iSCSI storage.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" uses this subsystem. The setup can only reuse an NVMe over Fabrics storage.',
+ )
+ : gettext(
+ 'Storage "{0}" uses this target. The setup can only reuse an iSCSI storage.',
+ ),
base.storage,
);
} else if (`${base.content ?? ''}` !== 'none') {
conflict = Ext.String.format(
- gettext(
- 'Storage "{0}" uses the LUNs of this target directly. If no guest uses them, clear Use LUNs directly in its edit dialog, otherwise use another target.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" uses the namespaces of this subsystem directly. If no guest uses them, clear Use namespaces directly in its edit dialog, otherwise use another subsystem.',
+ )
+ : gettext(
+ 'Storage "{0}" uses the LUNs of this target directly. If no guest uses them, clear Use LUNs directly in its edit dialog, otherwise use another target.',
+ ),
base.storage,
);
}
@@ -95,9 +112,13 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
wizard.showError(
Ext.htmlEncode(
Ext.String.format(
- gettext(
- 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" connects to this subsystem, but it is disabled. Enable that storage first.',
+ )
+ : gettext(
+ 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
+ ),
base.storage,
),
),
@@ -133,10 +154,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
}
let connection = {
storage: base.storage,
- target: base.target,
+ target: values.target,
nodes: selectedNodes.join(','),
};
- if (!base['managed-multipath']) {
+ if (!nvme && !base['managed-multipath']) {
wizard.showError(
Ext.htmlEncode(
Ext.String.format(
@@ -159,7 +180,7 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
waitMsgTarget: wizard,
failure: fail,
success: () =>
- connected({ storage: params.storage, target: params.target, nodes }),
+ connected({ storage: params.storage, target: values.target, nodes }),
});
},
});
@@ -167,6 +188,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
initComponent: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
+ me.targetKey = nvme ? 'subsysnqn' : 'target';
+ me.onlineHelp = nvme ? 'storage_nvmeof' : 'storage_open_iscsi';
+ let targetLabel = nvme ? gettext('Subsystem') : gettext('Target');
let common = PVE.panel.StorageBase.commonFields(true);
// wide enough for the longest label of the step
let labelWidth = 150;
@@ -174,7 +199,7 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
{
xtype: 'textfield',
name: 'portal',
- fieldLabel: gettext('Portal'),
+ fieldLabel: nvme ? gettext('Discovery Portal') : gettext('Portal'),
labelWidth,
allowBlank: false,
listeners: {
@@ -190,33 +215,54 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
},
},
{
- xtype: 'pveIScsiScan',
+ xtype: nvme ? 'pveNVMeOFScan' : 'pveIScsiScan',
name: 'target',
- fieldLabel: gettext('Target'),
+ fieldLabel: targetLabel,
labelWidth,
disabled: true,
showNodeSelector: false,
nodename: Proxmox.NodeName,
- listeners: { change: () => me.updateConnectionField() },
+ listeners: {
+ change: function (field, value) {
+ me.updateConnectionField();
+ let rec = nvme
+ ? field.getStore().findRecord('subsysnqn', value, 0, false, true, true)
+ : undefined;
+ if (rec) {
+ me.down('field[name=portals]').setValue(rec.get('portals'));
+ }
+ },
+ },
listConfig: {
width: 650,
- emptyText: PVE.Utils.renderNotFound(gettext('iSCSI Target')),
+ emptyText: PVE.Utils.renderNotFound(targetLabel),
columns: [
{
- text: gettext('Target'),
- dataIndex: 'target',
+ text: targetLabel,
+ dataIndex: me.targetKey,
flex: 1,
renderer: Ext.htmlEncode,
},
{
text: gettext('Portals'),
- dataIndex: 'portal',
+ dataIndex: nvme ? 'portals' : 'portal',
width: 220,
renderer: Ext.htmlEncode,
},
],
},
},
+ ...(nvme
+ ? [
+ {
+ xtype: 'textfield',
+ name: 'portals',
+ fieldLabel: gettext('Portals'),
+ labelWidth,
+ allowBlank: false,
+ },
+ ]
+ : []),
{
...common.id,
name: 'connection',
@@ -227,12 +273,19 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
xtype: 'component',
margin: '0 0 10 0',
html: `<span class="pmx-hint">${Ext.htmlEncode(gettext('Note:'))}</span> ${Ext.htmlEncode(
- gettext(
- 'Next creates or reuses the connection storage before you select a LUN.',
- ),
+ nvme
+ ? gettext(
+ 'Next creates or reuses the connection storage before you select a namespace.',
+ )
+ : gettext(
+ 'Next creates or reuses the connection storage before you select a LUN.',
+ ),
)}`,
},
- { xtype: 'pveStorageWizardNodeIdentities' },
+ {
+ xtype: 'pveStorageWizardNodeIdentities',
+ identity: nvme ? 'nvme-host-nqn' : 'iscsi-initiator-name',
+ },
];
me.advancedItems = [
{
@@ -267,8 +320,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
},
});
-PVE.storage.wizard.createIScsiLVM = function (wizard, values, lun) {
+PVE.storage.wizard.createLVMOnBase = function (wizard, values, lun) {
let connection = wizard.connection;
+ let suffix =
+ wizard.wizardType === 'nvmeof' ? `:nvme-${lun.wwid}` : `.dm-uuid-mpath-${lun.wwid}`;
let fail = (response) => wizard.showError(response.htmlStatus);
Proxmox.Utils.API2Request({
url: `/nodes/${Proxmox.NodeName}/storage/${encodeURIComponent(connection.storage)}/content`,
@@ -280,14 +335,16 @@ PVE.storage.wizard.createIScsiLVM = function (wizard, values, lun) {
if (wizard.destroyed) {
return;
}
- let volume = response.result.data.find((entry) =>
- entry.volid.endsWith(`.dm-uuid-mpath-${lun.wwid}`),
- );
+ let volume = response.result.data.find((entry) => entry.volid.endsWith(suffix));
if (!volume) {
wizard.showError(
- gettext(
- 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
- ),
+ wizard.wizardType === 'nvmeof'
+ ? gettext(
+ 'The selected namespace is not available yet. Check Node > SAN, then retry.',
+ )
+ : gettext(
+ 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
+ ),
);
return;
}
diff --git a/www/manager6/storage/wizard/Wizard.js b/www/manager6/storage/wizard/Wizard.js
index bfcc5a6cb..485641bac 100644
--- a/www/manager6/storage/wizard/Wizard.js
+++ b/www/manager6/storage/wizard/Wizard.js
@@ -138,6 +138,7 @@ Ext.define('PVE.storage.wizard.SAN', {
storage: values.storage,
nodes: params.nodes,
fc: me.wizardType === 'fc',
+ nvme: me.wizardType === 'nvmeof',
autoShow: true,
});
},
@@ -146,6 +147,9 @@ Ext.define('PVE.storage.wizard.SAN', {
multipathSummary: function (lun) {
let me = this;
+ if (me.wizardType === 'nvmeof') {
+ return PVE.SANUtils.multipathText('native');
+ }
// an explicit entry takes precedence, also for a LUN of the connection storage
if (lun.multipath === 'manual') {
return PVE.SANUtils.multipathText('manual');
@@ -191,23 +195,24 @@ Ext.define('PVE.storage.wizard.SAN', {
if (me.wizardType === 'fc') {
PVE.storage.wizard.createFC(me, values, lun);
} else {
- PVE.storage.wizard.createIScsiLVM(me, values, lun);
+ PVE.storage.wizard.createLVMOnBase(me, values, lun);
}
},
initComponent: function () {
let me = this;
- if (!['iscsi', 'fc'].includes(me.wizardType)) {
+ if (!['iscsi', 'fc', 'nvmeof'].includes(me.wizardType)) {
throw 'no SAN storage type specified';
}
- let iscsi = me.wizardType === 'iscsi';
+ let fc = me.wizardType === 'fc';
me.subject = PVE.storage.wizard.subject(me.wizardType);
me.title = Proxmox.Utils.dialog_title(me.subject, true, true);
+ let onlineHelp = me.wizardType === 'nvmeof' ? 'storage_nvmeof' : 'storage_multipath';
let common = PVE.panel.StorageBase.commonFields(true);
let settings = {
xtype: 'inputpanel',
title: gettext('Volume Group'),
- onlineHelp: 'storage_multipath',
+ onlineHelp,
items: [
{ ...common.id, fieldLabel: gettext('Storage ID') },
{
@@ -220,35 +225,37 @@ Ext.define('PVE.storage.wizard.SAN', {
PVE.panel.StorageBase.volumeChainField(true, 'lvm'),
],
};
- if (!iscsi) {
+ if (fc) {
settings.advancedItems = [
{ ...common.nodes, validator: PVE.storage.wizard.validateNodes },
];
}
me.items = [
- ...(iscsi
- ? [
+ ...(fc
+ ? []
+ : [
{
- xtype: 'pveStorageWizardIScsiTarget',
+ xtype: 'pveStorageWizardTarget',
title: gettext('Target'),
itemId: 'target',
+ storageType: me.wizardType,
},
- ]
- : []),
+ ]),
{
xtype: 'pveStorageWizardLunSelect',
- title: 'LUN',
+ title: me.wizardType === 'nvmeof' ? gettext('Namespace') : 'LUN',
itemId: 'luns',
- onlineHelp: 'storage_multipath',
+ nvme: me.wizardType === 'nvmeof',
+ onlineHelp,
// the nodes only see a newly mapped Fibre Channel LUN after a rescan
- rescanNote: iscsi
- ? undefined
- : Ext.String.format(
+ rescanNote: fc
+ ? Ext.String.format(
gettext(
'After mapping a new LUN on the array, rescan the Fibre Channel hosts of each node (<a href="{0}" target="_blank" rel="noopener noreferrer">documentation</a>).',
),
Ext.htmlEncode(Proxmox.Utils.get_help_link('storage_multipath')),
- ),
+ )
+ : undefined,
},
settings,
{
@@ -283,14 +290,14 @@ Ext.define('PVE.storage.wizard.SAN', {
me.callParent();
let tabs = me.down('#wizcontent');
tabs.on('tabchange', function (tp, tab) {
- if (iscsi && tab === me.down('#luns')) {
+ if (!fc && tab === me.down('#luns')) {
me.down('#back').disable();
}
me.down('#submit').setText(
tab === me.down('#target') ? gettext('Next') : gettext('Finish'),
);
});
- if (iscsi) {
+ if (!fc) {
me.down('#submit').setText(gettext('Next'));
}
me.on('destroy', () => me.reloadCallback?.());
diff --git a/www/manager6/window/SANLunState.js b/www/manager6/window/SANLunState.js
index 6215f8e57..6cdcdce2d 100644
--- a/www/manager6/window/SANLunState.js
+++ b/www/manager6/window/SANLunState.js
@@ -19,7 +19,10 @@ Ext.define('PVE.window.SANLunState', {
if (!me.wwid || !me.storage) {
throw 'no LUN or storage specified';
}
- me.title = Ext.String.format(gettext('Storage {0}: LUN State'), me.storage);
+ me.title = Ext.String.format(
+ me.nvme ? gettext('Storage {0}: Namespace State') : gettext('Storage {0}: LUN State'),
+ me.storage,
+ );
// The other nodes set up a new LUN with their next status updates.
let pendingUntil = Date.now() + 60000;
let rstore = Ext.create('Proxmox.data.UpdateStore', {
@@ -55,13 +58,17 @@ Ext.define('PVE.window.SANLunState', {
margin: '0 0 10 0',
html: Ext.htmlEncode(
Ext.String.format(
- me.fc
+ me.nvme
? gettext(
- 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array and rescan its Fibre Channel hosts.',
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the namespace on the array.',
)
- : gettext(
- 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array.',
- ),
+ : me.fc
+ ? gettext(
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array and rescan its Fibre Channel hosts.',
+ )
+ : gettext(
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array.',
+ ),
me.storage,
),
),
@@ -71,7 +78,9 @@ Ext.define('PVE.window.SANLunState', {
flex: 1,
store,
disableSelection: true,
- emptyText: gettext('No node reported this LUN yet.'),
+ emptyText: me.nvme
+ ? gettext('No node reported this namespace yet.')
+ : gettext('No node reported this LUN yet.'),
columns: [
{
text: gettext('Node'),
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (13 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
Add a view below Node > SAN with the controllers of a node, one per
connected portal, their state and the storage they belong to, and the
host NQN of the node. A controller can be disconnected from there, for
example after its portal was removed from the storage. The node
refuses that for the last usable path of a namespace in use.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/SANUtils.js | 5 ++
www/manager6/Utils.js | 1 +
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 115 +++++++++++++++++++++++++
5 files changed, 130 insertions(+), 1 deletion(-)
create mode 100644 www/manager6/node/NVMeOFControllers.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 1fbce8c4d..5877c660d 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -252,6 +252,7 @@ JSSRC= \
node/ISCSITargets.js \
node/LVM.js \
node/LVMThin.js \
+ node/NVMeOFControllers.js \
node/SANLuns.js \
node/StatusView.js \
node/Subscription.js \
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index 1b2b25c12..2a4877df5 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -135,6 +135,11 @@ Ext.define('PVE.SANUtils', {
'logged-in': ['check-circle good', gettext('Connected')],
recovering: ['exclamation-circle warning', gettext('Reconnecting')],
'not-connected': ['times-circle faded', gettext('Not connected')],
+ live: ['check-circle good', gettext('Connected')],
+ connecting: ['exclamation-circle warning', gettext('Reconnecting')],
+ resetting: ['exclamation-circle warning', gettext('Reconnecting')],
+ deleting: ['times-circle faded', gettext('Disconnecting')],
+ 'deleting (no IO)': ['times-circle faded', gettext('Disconnecting')],
}[value];
let [icon, label] = display ?? ['question-circle faded', value];
return `<i class="fa fa-fw fa-${icon}"></i> ${Ext.htmlEncode(label)}`;
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index ef8218918..5610f0020 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -2268,6 +2268,7 @@ Ext.define('PVE.Utils', {
lvmthinremove: ['Thinpool', gettext('Remove')],
migrateall: ['', gettext('Bulk migrate VMs and Containers')],
move_volume: ['CT', gettext('Move Volume')],
+ nvmedisconnect: ['NVMe-oF', gettext('Disconnect')],
'pbs-download': ['VM/CT', gettext('File Restore Download')],
pull_file: ['CT', gettext('Pull file')],
push_file: ['CT', gettext('Push file')],
diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
index 476463a65..f6c2506b2 100644
--- a/www/manager6/node/Config.js
+++ b/www/manager6/node/Config.js
@@ -385,7 +385,7 @@ Ext.define('PVE.node.Config', {
);
}
- // the san-luns and iscsi reads accept either privilege
+ // the san-luns, iscsi and nvmeof reads accept either privilege
if (caps.nodes['Sys.Audit'] || caps.storage['Datastore.Allocate']) {
me.items.push(
{
@@ -401,6 +401,13 @@ Ext.define('PVE.node.Config', {
iconCls: 'fa fa-bullseye',
groups: ['san'],
},
+ {
+ xtype: 'pveNVMeOFControllerList',
+ title: 'NVMe over Fabrics',
+ itemId: 'san-nvmeof',
+ iconCls: 'fa fa-plug',
+ groups: ['san'],
+ },
);
}
diff --git a/www/manager6/node/NVMeOFControllers.js b/www/manager6/node/NVMeOFControllers.js
new file mode 100644
index 000000000..93955442d
--- /dev/null
+++ b/www/manager6/node/NVMeOFControllers.js
@@ -0,0 +1,115 @@
+Ext.define('PVE.node.NVMeOFControllerList', {
+ extend: 'Ext.grid.Panel',
+ xtype: 'pveNVMeOFControllerList',
+ onlineHelp: 'storage_nvmeof',
+ emptyText: gettext('No NVMe over Fabrics controllers found'),
+ viewConfig: { enableTextSelection: true },
+ store: {
+ fields: ['controller', 'subsysnqn', 'portal', 'state', 'namespaces', 'used-by-storage'],
+ sorters: ['subsysnqn', 'portal'],
+ },
+ columns: [
+ { text: gettext('Controller'), dataIndex: 'controller', width: 100 },
+ {
+ text: gettext('Subsystem'),
+ dataIndex: 'subsysnqn',
+ flex: 5,
+ renderer: Ext.htmlEncode,
+ },
+ { text: gettext('Portal'), dataIndex: 'portal', width: 180, renderer: Ext.htmlEncode },
+ {
+ text: gettext('State'),
+ dataIndex: 'state',
+ width: 150,
+ renderer: PVE.SANUtils.renderState,
+ },
+ { text: gettext('Namespaces'), dataIndex: 'namespaces', width: 110 },
+ {
+ text: gettext('Storage'),
+ dataIndex: 'used-by-storage',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+
+ reload: function () {
+ let me = this;
+ let load = (path, success) =>
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/nvmeof/${path}`,
+ method: 'GET',
+ waitMsgTarget: me.getView(),
+ failure: (response) =>
+ Proxmox.Utils.setErrorMask(me.getView(), response.htmlStatus),
+ success: (response) => !me.destroyed && success(response.result.data),
+ });
+ load('status', function (status) {
+ me.down('#hostnqn').setValue(status['host-nqn'] || '');
+ let error = status.supported
+ ? status.error
+ : gettext('nvme-cli is not installed on this node.');
+ if (error) {
+ Proxmox.Utils.setErrorMask(me.getView(), Ext.htmlEncode(error));
+ return;
+ }
+ load('controller', function (controllers) {
+ Proxmox.Utils.setErrorMask(me.getView(), false);
+ me.getStore().setData(controllers);
+ });
+ });
+ },
+
+ listeners: {
+ activate: function () {
+ this.reload();
+ },
+ },
+
+ initComponent: function () {
+ let me = this;
+ me.nodename = me.pveSelNode.data.node;
+ me.tbar = [
+ { text: gettext('Reload'), iconCls: 'fa fa-refresh', handler: () => me.reload() },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Disconnect'),
+ iconCls: 'fa fa-unlink',
+ disabled: true,
+ dangerous: true,
+ enableFn: () => !!Ext.state.Manager.get('GuiCap').dc['Sys.Modify'],
+ confirmMsg: (rec) =>
+ Ext.String.format(
+ gettext(
+ 'Disconnect controller {0}? A storage that still lists its portal connects to it again.',
+ ),
+ Ext.htmlEncode(rec.get('controller')),
+ ),
+ handler: (button, event, rec) =>
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/nvmeof/controller/${rec.get('controller')}`,
+ method: 'DELETE',
+ waitMsgTarget: me,
+ failure: (response) => Ext.Msg.alert(gettext('Error'), response.htmlStatus),
+ success: (response) =>
+ Ext.create('Proxmox.window.TaskProgress', {
+ upid: response.result.data,
+ autoShow: true,
+ taskDone: () => !me.destroyed && me.reload(),
+ }),
+ }),
+ },
+ '->',
+ {
+ xtype: 'displayfield',
+ itemId: 'hostnqn',
+ fieldLabel: gettext('Host NQN'),
+ },
+ {
+ iconCls: 'fa fa-clipboard',
+ tooltip: gettext('Copy to Clipboard'),
+ handler: () => PVE.Utils.copyToClipboard(me.down('#hostnqn').getValue()),
+ },
+ ];
+ me.callParent();
+ },
+});
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (14 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The access rules of an array need the host NQN of every node, like the
initiator names for iSCSI. Show them next to those in the node list of
the SAN view, with a button to copy them all.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 29 +++++++++++++++++++++--------
1 file changed, 21 insertions(+), 8 deletions(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index fcddc3e2e..475ea4e33 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -10,6 +10,7 @@ Ext.define('pve-san-status', {
'time',
'iscsi-initiator-name',
'other-maps',
+ 'nvme-host-nqn',
'wwid',
'name',
'explicit',
@@ -104,6 +105,14 @@ Ext.define('PVE.dc.SANView', {
sortAfterUpdate: true,
});
let reload = () => me.rstore.load();
+ let copy = (field) => () =>
+ PVE.Utils.copyToClipboard(
+ nodeStore
+ .getRange()
+ .map((rec) => rec.get(field))
+ .filter(Boolean)
+ .join('\n'),
+ );
let edit = (rec) =>
Ext.create('PVE.dc.MultipathWwidEdit', {
wwid: rec?.get('wwid'),
@@ -140,14 +149,12 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
- handler: () =>
- PVE.Utils.copyToClipboard(
- nodeStore
- .getRange()
- .map((rec) => rec.get('iscsi-initiator-name'))
- .filter(Boolean)
- .join('\n'),
- ),
+ handler: copy('iscsi-initiator-name'),
+ },
+ {
+ text: gettext('Copy Host NQNs'),
+ iconCls: 'fa fa-clipboard',
+ handler: copy('nvme-host-nqn'),
},
],
columns: [
@@ -179,6 +186,12 @@ Ext.define('PVE.dc.SANView', {
flex: 2,
renderer: Ext.htmlEncode,
},
+ {
+ text: gettext('NVMe Host NQN'),
+ dataIndex: 'nvme-host-nqn',
+ flex: 4,
+ renderer: Ext.htmlEncode,
+ },
{
text: gettext('Other Maps'),
tooltip: gettext(
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (15 preceding siblings ...)
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-18 20:42 ` Thomas Lamprecht
16 siblings, 0 replies; 18+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:42 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the namespaces of NVMe over Fabrics
storages as well. They use the native multipath of the kernel, with
neither an entry nor a storage that requests a map, so the Multipath
column would call them unmanaged. Show "Native" for them, as the LUN
list of a node does.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index 475ea4e33..30bb65e8e 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -15,6 +15,7 @@ Ext.define('pve-san-status', {
'name',
'explicit',
'multipath-storage',
+ 'native',
'nodes',
'size',
'used-by-storage',
@@ -371,7 +372,14 @@ Ext.define('PVE.dc.SANView', {
width: 170,
renderer: (value, md, rec) => {
let storage = rec.get('multipath-storage');
- let kind = value ? 'manual' : storage ? 'storage' : 'unmanaged';
+ // a namespace uses the native multipath of the kernel
+ let kind = rec.get('native')
+ ? 'native'
+ : value
+ ? 'manual'
+ : storage
+ ? 'storage'
+ : 'unmanaged';
return Ext.htmlEncode(PVE.SANUtils.multipathText(kind, storage));
},
},
^ permalink raw reply related [flat|nested] 18+ messages in thread