* [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage
@ 2026-11-13 3:04 Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
` (17 more replies)
0 siblings, 18 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
A storage type for NVMe subsystems reached over TCP, on top of v4 of the
SAN series [0]: namespaces as base volumes of shared LVM, or directly
as disk images, with the kernel's native multipath.
This is a deliberately small first version: TCP only, without
authentication or TLS, so access rests on the host NQN list of the
target and a separate storage network. It relies on the kernel to
reconnect lost paths and on the SAN series for the LUN inventory, the
guided setup and the checks before a namespace is written. The plugin
connects what is missing and removes nothing, also not when a portal or
the storage is removed: disconnecting a controller is an explicit
action per node.
Two prerequisites from the chapter: nodes cloned from one image need
distinct host NQNs before they connect. And as with every new storage
type, all nodes need the upgrade before the first such storage is
added, as a node that does not know the type drops its definitions when
it writes storage.cfg. I ran into that in the lab.
What it adds:
- storage type 'nvmeof' with a subsystem NQN and a list of TCP
portals. Each node connects the portals it has no controller for and
leaves existing controllers as they are. New controllers get
ctrl_loss_tmo=-1, so the kernel keeps reconnecting, and fail I/O on
a lost path after 120 seconds unless configured otherwise. Volumes
are named nvme-<wwid>.
- LVM accepts such a volume as base, with the reference check and the
per-LUN lock of the SAN series.
- discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'.
- /nodes/{node}/nvmeof: host identity, controller list, and removal
of one controller. That is refused if a namespace that is mounted,
has a holder or is referenced by a storage would lose its last
usable path. A storage that still lists the portal connects again.
- the node report and with it the cluster-wide status of the SAN
series include the namespaces of nvmeof storages, with their state
on each node of the storage. They do not need multipathd.
- web UI: the Add Storage menu of the SAN panel gets a third entry for
shared LVM on a namespace, which ends with the state of the namespace
on every node like the other two; Datacenter > Storage > Add gets the
dialog of the storage type, with subsystem discovery and the note
pointing to the SAN panel; Node > SAN lists the controllers; the SAN
panel shows the host NQN of each node.
- nvme-cli becomes a Recommends of libpve-storage-perl.
Namespaces already show up in the LUN list of Node > SAN through the
SAN series, with their paths.
Patches:
- storage 01-09: SMART skip for fabrics namespaces, the plugin,
LVM base, discovery, namespace attribution in the LUN inventory, the
node API, the host NQN and the namespaces in the node report
- docs 10: the storage chapter
- manager 11-17: API mount, storage type and its dialog, LVM base
selector, guided setup, controller list, host NQN column, native
multipath in the LUN list of the SAN panel
I tested this on a six-node nested cluster against a Linux nvmet target
and a NetApp AFF A20, each with two portals: the guided setup with a new
and with a reused connection storage and with a node that has no access
at first, one path down on both targets, a NetApp path cut and a stopped
multipathd as the cluster-wide status shows them, the plain dialogs on
the nvmet target, a guest disk on a NetApp namespace through snapshot,
rollback, live migration and a path cut under I/O, wiping a namespace
and its refusals on the NetApp, and connecting after a node reboot on
both.
[0]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/
pve-storage:
Thomas Lamprecht (9):
diskmanage: skip SMART queries for NVMe fabric namespaces
nvmeof: add a storage for NVMe over Fabrics subsystems
lvm: allow a volume group on an NVMe over Fabrics namespace
api: scan: add NVMe over Fabrics discovery
pvesm: add a scan command for NVMe over Fabrics
api: scan: san-luns: attribute NVMe namespaces to storage
api: add node-level NVMe over Fabrics status and controller removal
multipath: report the NVMe host NQN of each node
multipath: report NVMe over Fabrics namespaces in the cluster status
debian/control | 1 +
src/PVE/API2/Makefile | 1 +
src/PVE/API2/Multipath.pm | 8 +-
src/PVE/API2/NVMeOF.pm | 182 ++++++++++++++
src/PVE/API2/Storage/Scan.pm | 59 +++++
src/PVE/CLI/pvesm.pm | 13 +
src/PVE/Diskmanage.pm | 30 ++-
src/PVE/Makefile | 1 +
src/PVE/Multipath/Status.pm | 64 ++++-
src/PVE/NVMeOF.pm | 352 ++++++++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/LVMPlugin.pm | 3 +-
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 204 ++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 38 ++-
src/test/Makefile | 7 +-
src/test/disklist_test.pm | 14 ++
src/test/run_multipath_api_tests.pl | 103 +++++++-
src/test/run_nvmeof_api_tests.pl | 151 ++++++++++++
src/test/run_nvmeof_taint_tests.pl | 65 +++++
src/test/run_nvmeof_tests.pl | 309 ++++++++++++++++++++++++
src/test/run_san_lun_scan_tests.pl | 33 ++-
src/test/run_san_nvme_tests.pl | 30 +++
24 files changed, 1642 insertions(+), 31 deletions(-)
pve-docs:
Thomas Lamprecht (1):
storage: document NVMe over Fabrics storages
pve-storage-multipath.adoc | 4 +-
pve-storage-nvmeof.adoc | 161 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 20 +++--
3 files changed, 177 insertions(+), 8 deletions(-)
pve-manager:
Thomas Lamprecht (7):
api: nodes: mount the NVMe over Fabrics endpoints
ui: storage: add NVMe over Fabrics storage
ui: lvm: offer NVMe over Fabrics storages as base storage
ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
ui: node: san: list NVMe over Fabrics controllers
ui: dc: san: show the NVMe host NQN of each node
ui: dc: san: show that a namespace uses native multipath
PVE/API2/Nodes.pm | 7 +
www/manager6/Makefile | 4 +-
www/manager6/SANUtils.js | 7 +
www/manager6/Utils.js | 7 +
www/manager6/dc/SAN.js | 41 ++++--
www/manager6/dc/StorageView.js | 5 +-
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 115 ++++++++++++++++
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/LVMEdit.js | 12 +-
www/manager6/storage/NVMeOFEdit.js | 130 ++++++++++++++++++
www/manager6/storage/wizard/LunSelect.js | 29 +++-
www/manager6/storage/wizard/NodeIdentities.js | 45 +++---
.../storage/wizard/{IScsi.js => Target.js} | 129 +++++++++++------
www/manager6/storage/wizard/Wizard.js | 43 +++---
www/manager6/window/SANLunState.js | 23 +++-
16 files changed, 499 insertions(+), 112 deletions(-)
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
` (16 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The disk list queries the SMART health of every disk. For an NVMe
namespace attached over TCP or RDMA, the query goes to the target and
can stall the listing while a path recovers.
Skip the query for these transports. It depends on the transport of
the namespace only, so namespaces connected by hand are covered too.
Their health shows as unknown, local NVMe drives are queried as
before.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Diskmanage.pm | 4 +++-
src/test/disklist_test.pm | 14 ++++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index d6340d32..1fec9ec5 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -644,7 +644,9 @@ sub get_disks {
}
my ($health, $wearout) = ('UNKNOWN', 'N/A');
- if (!$nosmart) {
+ # SMART queries over fabrics can stall while a path recovers.
+ my $remote_nvme = defined($nvme_transport) && $nvme_transport =~ /^(?:tcp|rdma)$/;
+ if (!$nosmart && !$remote_nvme) {
eval {
my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
$health = $smartdata->{health} if $smartdata->{health};
diff --git a/src/test/disklist_test.pm b/src/test/disklist_test.pm
index 2f22fae8..c32ee09a 100644
--- a/src/test/disklist_test.pm
+++ b/src/test/disklist_test.pm
@@ -266,6 +266,20 @@ sub test_disk_list {
ok($san_disks->{sda}, 'SAN inventory opts into iSCSI');
}
+ if ($testcasedir eq 'nvme_smart') {
+ my $smart_calls;
+ $diskmanage_module->mock(get_smart_data => sub { $smart_calls++; return {} });
+ for my $transport (qw(tcp rdma pcie)) {
+ $smart_calls = 0;
+ $diskmanage_module->mock(get_nvme_transport => sub { $transport });
+ PVE::Diskmanage::get_disks();
+ $testcount++;
+ is($smart_calls, $transport eq 'pcie' ? 1 : 0, "SMART queries for $transport");
+ }
+ $diskmanage_module->unmock('get_smart_data');
+ $diskmanage_module->unmock('get_nvme_transport');
+ }
+
if (-f "disk_tests/$testcasedir/multipathd_status_expected.json") {
my $status = PVE::Multipath::path_status_by_wwid();
my $expected_status =
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
` (15 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
NVMe/TCP namespaces need a manual nvme-cli setup on every node so
far. Add a storage type that connects one subsystem through each
configured portal and offers its namespaces as raw volumes. Native
NVMe multipath joins the paths and the kernel reconnects lost ones.
Local disk setup refuses namespaces attached over a fabric, as for
iSCSI LUNs. Wiping a namespace or creating a volume group on it is
refused while a storage serves images from its subsystem. nvme-cli
is only recommended, as no other storage type needs it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
debian/control | 1 +
src/PVE/Diskmanage.pm | 10 ++
src/PVE/Makefile | 1 +
src/PVE/NVMeOF.pm | 249 +++++++++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 204 +++++++++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 13 ++
src/test/Makefile | 6 +-
src/test/run_nvmeof_taint_tests.pl | 60 +++++++
src/test/run_nvmeof_tests.pl | 243 ++++++++++++++++++++++++++++
src/test/run_san_nvme_tests.pl | 13 ++
13 files changed, 803 insertions(+), 2 deletions(-)
create mode 100644 src/PVE/NVMeOF.pm
create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
create mode 100755 src/test/run_nvmeof_taint_tests.pl
create mode 100755 src/test/run_nvmeof_tests.pl
diff --git a/debian/control b/debian/control
index 14d8a648..99b41f92 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
${misc:Depends},
${perl:Depends},
Recommends: multipath-tools,
+ nvme-cli,
pve-esxi-import-tools (>= 0.6.0),
skopeo,
zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 1fec9ec5..7bee3b08 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -108,6 +108,8 @@ sub disk_is_used {
my $disklist = get_disks($dev, 1, 1);
die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+ die "'$disk' is not a valid local disk\n"
+ if ($disklist->{$dev}->{transport} // '') =~ /^nvme-(?:tcp|rdma|fc)$/;
return 1 if $disklist->{$dev}->{used};
return 0;
@@ -503,6 +505,14 @@ sub dir_is_empty {
return 1;
}
+# Some controllers pad the NQN or derive it from padded serial/model fields.
+sub read_subsysnqn {
+ my ($path) = @_;
+ my $nqn = file_read_firstline($path) // return undef;
+ $nqn =~ s/\s+$//;
+ return $nqn;
+}
+
# Returns the NVMe transport (pcie, fc, tcp, rdma, loop) of a namespace, either directly from
# its controller, or from any controller of its subsystem for native NVMe multipath setups.
sub get_nvme_transport {
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..7a3140cb
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,249 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+
+use PVE::Diskmanage;
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+
+sub canonical_address($addr) {
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ return inet_ntop(AF_INET, $packed);
+ }
+ if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+ return inet_ntop(AF_INET, substr($packed, 12))
+ if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+ return inet_ntop(AF_INET6, $packed);
+ }
+ return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+ my ($addr, $port);
+ if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::(\d+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } elsif ($portal =~ m/^(\d+\.\d+\.\d+\.\d+)(?::(\d+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } else {
+ return undef;
+ }
+ return undef if defined($port) && ($port < 1 || $port > 65535);
+ $addr = canonical_address($addr);
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ return undef
+ if $addr eq '0.0.0.0'
+ || $addr eq '255.255.255.255'
+ || (unpack('C', $packed) & 0xf0) == 0xe0;
+ } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+ my $prefix = unpack('n', $packed);
+ return undef
+ if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+ } else {
+ return undef;
+ }
+ return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+ 'pve-storage-nvme-portal',
+ sub($portal, $noerr = undef) {
+ return $portal if parse_portal($portal);
+ return undef if $noerr;
+ die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+ },
+);
+
+sub format_path($path) {
+ my $addr = $path->{traddr} =~ /:/ ? "[$path->{traddr}]" : $path->{traddr};
+ return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+ return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+ PVE::Tools::split_list($scfg->{'nvme-portals'})];
+}
+
+sub host_identity() {
+ my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+ my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+ /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+ die
+ "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the host identity\n"
+ if !defined($nqn) || length($nqn) > 223;
+ die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+ if !defined($id) || $id =~ /^[0-]+$/;
+ return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+ if !-e "$SYSFS/module/nvme_tcp";
+ my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+ die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+ dir_glob_foreach(
+ $dir,
+ 'nvme\d+',
+ sub($name, @) {
+ my $address = file_read_firstline("$dir/$name/address") // return;
+ my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+ return if !defined($address{traddr});
+ $res->{$name} = {
+ transport => file_read_firstline("$dir/$name/transport") // '',
+ subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+ state => file_read_firstline("$dir/$name/state") // 'unknown',
+ traddr => canonical_address($address{traddr}),
+ trsvcid => $address{trsvcid},
+ };
+ },
+ );
+ return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+ my $controllers = controllers();
+ return {
+ map { $_ => $controllers->{$_} } grep {
+ $controllers->{$_}->{subsystem} eq $subsystem
+ && $controllers->{$_}->{transport} eq 'tcp'
+ } keys %$controllers
+ };
+}
+
+sub path_controller($controllers, $path) {
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ return $name
+ if $ctrl->{traddr} eq $path->{traddr}
+ && ($ctrl->{trsvcid} // '') eq $path->{trsvcid};
+ }
+ return undef;
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-subsystem";
+ dir_glob_foreach(
+ $dir,
+ 'nvme-subsys\d+',
+ sub($subsys, @) {
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+ return if $nqn ne $subsystem;
+ dir_glob_foreach(
+ "$dir/$subsys",
+ 'nvme\d+n\d+',
+ sub($dev, @) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = {
+ nsid => int(file_read_firstline("$SYSFS/block/$dev/nsid") // 0),
+ wwid => $wwid,
+ size => int($size) * 512,
+ };
+ },
+ );
+ },
+ );
+ return $res;
+}
+
+sub with_lock($code) {
+ my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+ 35,
+ sub {
+ make_path($RUN_DIR);
+ open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+ flock($lock, LOCK_EX | LOCK_NB)
+ or die "NVMe over Fabrics is busy - another operation is still running\n";
+ # A command still in the kernel must retain serialization after its caller times out.
+ my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+ fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+ or die "cannot inherit NVMe lock - $!\n";
+ return $code->();
+ },
+ );
+ die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+ return $res;
+}
+
+my sub retry_file($scfg, $path) {
+ my ($key) = sha1_hex(join("\0",
+ $scfg->{subsysnqn},
+ format_path($path),
+ $scfg->{'nvme-fast-io-fail-tmo'} // 120)) =~ /\A([0-9a-f]{40})\z/;
+ return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+ my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+ my $age = time() - $mtime;
+ return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+ make_path("$RUN_DIR/retry");
+ file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub connect_path($scfg, $path) {
+ my ($nqn) = $scfg->{subsysnqn} =~ /\A($NQN_PATTERN)\z/;
+ die "invalid subsystem NQN\n"
+ if !defined($nqn)
+ || length($nqn) > 223
+ || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery';
+ my ($fail) = ($scfg->{'nvme-fast-io-fail-tmo'} // 120) =~ /\A(-?\d+)\z/;
+ die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+ return with_lock(sub {
+ return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ assert_support();
+ my $host = host_identity();
+ run_command(
+ [
+ $NVME,
+ 'connect',
+ '--transport=tcp',
+ '--nqn',
+ $nqn,
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $path->{trsvcid},
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--ctrl-loss-tmo=-1',
+ '--fast_io_fail_tmo',
+ $fail,
+ ],
+ timeout => 30,
+ );
+ # nvme-cli can exit successfully without having connected
+ die "no controller appeared\n"
+ if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ unlink(retry_file($scfg, $path));
+ return;
+ });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index be6e9743..287d64f0 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -32,6 +32,7 @@ use PVE::Storage::LvmThinPlugin;
use PVE::Storage::NFSPlugin;
use PVE::Storage::CIFSPlugin;
use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
use PVE::Storage::RBDPlugin;
use PVE::Storage::CephFSPlugin;
use PVE::Storage::ISCSIDirectPlugin;
@@ -57,6 +58,7 @@ PVE::Storage::LvmThinPlugin->register();
PVE::Storage::NFSPlugin->register();
PVE::Storage::CIFSPlugin->register();
PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
PVE::Storage::RBDPlugin->register();
PVE::Storage::CephFSPlugin->register();
PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
NFSPlugin.pm \
CIFSPlugin.pm \
ISCSIPlugin.pm \
+ NVMeOFPlugin.pm \
CephFSPlugin.pm \
RBDPlugin.pm \
ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..4dc9803a
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,204 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use strict;
+use warnings;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Diskmanage;
+use PVE::Network;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+ return {
+ content => [{ images => 1, none => 1 }, { images => 1 }],
+ select_existing => 1,
+ 'sensitive-properties' => {},
+ };
+}
+
+sub properties {
+ return {
+ subsysnqn => {
+ description => 'The NQN of the NVMe subsystem.',
+ type => 'string',
+ pattern => $PVE::NVMeOF::NQN_PATTERN,
+ maxLength => 223,
+ },
+ 'nvme-portals' => {
+ description =>
+ 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+ . ' with an optional port (default 4420).',
+ type => 'string',
+ format => 'pve-storage-nvme-portal-list',
+ },
+ 'nvme-fast-io-fail-tmo' => {
+ description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+ . ' this timeout; I/O can remain queued while paths reconnect.',
+ type => 'integer',
+ minimum => -1,
+ default => 120,
+ },
+ };
+}
+
+sub options {
+ return {
+ subsysnqn => { fixed => 1 },
+ 'nvme-portals' => {},
+ 'nvme-fast-io-fail-tmo' => { optional => 1 },
+ nodes => { optional => 1 },
+ disable => { optional => 1 },
+ content => { optional => 1 },
+ bwlimit => { optional => 1 },
+ };
+}
+
+my sub assert_valid_config {
+ my ($scfg) = @_;
+ die "the discovery subsystem cannot be used as storage\n"
+ if $scfg->{subsysnqn} eq 'nqn.2014-08.org.nvmexpress.discovery';
+ my %seen;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ my $portal = PVE::NVMeOF::format_path($path);
+ die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+ }
+}
+
+sub on_add_hook {
+ my ($class, $storeid, $scfg, %param) = @_;
+ assert_valid_config($scfg);
+ return;
+}
+
+sub on_update_hook_full {
+ my ($class, $storeid, $scfg, $update, $delete, $sensitive) = @_;
+ my $merged = { %$scfg, %$update };
+ delete $merged->{$_} for ($delete // [])->@*;
+ assert_valid_config($merged);
+ return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname {
+ my ($class, $volname) = @_;
+ if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part\d+$/) {
+ return ('images', $1, undef, undef, undef, undef, 'raw');
+ }
+ die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+sub filesystem_path {
+ my ($class, $scfg, $volname, $snapname) = @_;
+ die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+ my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+ my $path = "/dev/disk/by-id/$name";
+ return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes {
+ my ($class, $storeid, $scfg, $vmid, $content_types) = @_;
+ my $res = $class->list_images($storeid, $scfg, $vmid);
+ $_->{content} = 'images' for @$res;
+ return $res;
+}
+
+sub list_images {
+ my ($class, $storeid, $scfg, $vmid, $vollist, $cache) = @_;
+ return [] if defined($vmid) && !$vollist;
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ my $res = [];
+ for my $dev (sort keys %$namespaces) {
+ my $ns = $namespaces->{$dev};
+ my $volid = "$storeid:nvme-$ns->{wwid}";
+ next if $vollist && !grep { $_ eq $volid } @$vollist;
+ push @$res,
+ {
+ volid => $volid,
+ format => 'raw',
+ size => $ns->{size},
+ vmid => 0,
+ nsid => $ns->{nsid},
+ };
+ }
+ return $res;
+}
+
+sub status {
+ my ($class, $storeid, $scfg, $cache) = @_;
+ my $active = grep { $_->{state} eq 'live' }
+ values PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ return (0, 0, 0, $active ? 1 : 0);
+}
+
+sub activate_storage {
+ my ($class, $storeid, $scfg, $cache) = @_;
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ eval {
+ die "no answer from portal\n"
+ if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::connect_path($scfg, $path);
+ };
+ if (my $err = $@) {
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ warn "storage '$storeid': connecting to '"
+ . PVE::NVMeOF::format_path($path)
+ . "' failed - $err";
+ }
+ }
+ return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection {
+ my ($class, $storeid, $scfg) = @_;
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ return 1 if grep { $_->{state} eq 'live' } values %$controllers;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ }
+ return 0;
+}
+
+sub activate_volume {
+ my ($class, $storeid, $scfg, $volname, $snapname, $cache) = @_;
+ my $path = $class->filesystem_path($scfg, $volname, $snapname);
+ my $deadline = Time::HiRes::time() + 10;
+ while (!-b $path && Time::HiRes::time() < $deadline) {
+ last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ Time::HiRes::usleep(250_000);
+ }
+ my $real = Cwd::realpath($path);
+ die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+ my ($dev) = $real =~ m!^/dev/(nvme\d+n\d+)$!;
+ die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+ my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$dev/device/subsysnqn") // '';
+ die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+ if $nqn ne $scfg->{subsysnqn};
+}
+
+sub volume_has_feature {
+ my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running) = @_;
+ return 1 if $feature eq 'copy' && !defined($snapname);
+ return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index e4ad7e6f..783d4145 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
);
our @SHARED_STORAGE = (
- 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+ 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
);
our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index bba3c128..f3682106 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -232,6 +232,11 @@ sub lun_partitions($lun) {
}
sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+ my $name = basename($lun->{devpath});
+ my $nqn =
+ $name =~ /^nvme\d+n\d+$/
+ ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+ : undef;
my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
[$lun->{devpath}, lun_partitions($lun)->@*],
)->@*;
@@ -253,6 +258,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
) {
die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
}
+ if (
+ $type eq 'nvmeof'
+ && defined($nqn)
+ && $scfg->{subsysnqn} eq $nqn
+ && PVE::Storage::san_storage_serves_images($scfg)
+ ) {
+ die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+ }
next if $type ne 'lvm' || !$scfg->{base};
my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base});
my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-mpath|scsi|nvme)-(.+)$/;
diff --git a/src/test/Makefile b/src/test/Makefile
index c9bcea8f..124612da 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
all: test
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api test_nvmeof
test_zfspoolplugin: run_test_zfspoolplugin.pl
./run_test_zfspoolplugin.pl
@@ -76,3 +76,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+ ./run_nvmeof_tests.pl
+ perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..3c19f8cc
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,60 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::More;
+use Test::MockModule;
+use PVE::NVMeOF;
+use PVE::Tools qw(file_set_contents file_get_contents);
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+ subsystem_controllers => sub {
+ return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+ },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+ run_command => sub($cmd, %opts) {
+ $commands++;
+ is(
+ scalar(grep { tainted($_) } @$cmd),
+ 0,
+ 'all connect operands are validated and untainted',
+ );
+ PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'nvme-fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+ PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+ 'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+ eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+ like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..5ff96833
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,243 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use File::Path qw(make_path);
+use Fcntl qw(:flock);
+use Test::More;
+use Test::MockModule;
+use PVE::NVMeOF;
+use PVE::Storage;
+use PVE::Tools qw(file_set_contents);
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ 'nvme-portals' => '192.0.2.1,[2001:db8::2]:4421',
+ content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+ is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+ is_deeply(
+ [sort keys $class->properties()->%*],
+ [qw(nvme-fast-io-fail-tmo nvme-portals subsysnqn)],
+ 'only three plugin-specific properties',
+ );
+ my $parsed = PVE::Storage::Plugin->parse_config(
+ 'storage.cfg',
+ "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tnvme-portals 192.0.2.1\n\tcontent none\n",
+ );
+ is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+ is_deeply(
+ $parsed->{ids}->{fabric}->{content},
+ { none => 1 },
+ 'connection-only content parses',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ { traddr => '192.0.2.1', trsvcid => 4420 },
+ 'default I/O port',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+ { traddr => '2001:db8::1', trsvcid => 4421 },
+ 'canonical IPv6',
+ );
+ for my $bad (
+ 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+ ) {
+ ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+ }
+ eval {
+ $class->on_add_hook('fabric', { %$cfg, 'nvme-portals' => '192.0.2.1,192.0.2.1:4420' });
+ };
+ like($@, qr/listed more than once/, 'duplicate portals refused');
+ eval {
+ $class->on_add_hook(
+ 'fabric',
+ { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+ );
+ };
+ like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+ is(
+ scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+ '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+ 'stable volume path',
+ );
+ for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+ eval { $class->parse_volname($bad) };
+ like($@, qr/unable to parse/, "invalid volume $bad");
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(
+ namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096, nsid => 1 } } });
+ is(
+ $class->list_volumes('fabric', $cfg)->[0]->{volid},
+ 'fabric:nvme-eui.abc',
+ 'content none still offers bases',
+ );
+ is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+ $nvme->mock(subsystem_controllers => sub { {} });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+ like($@, qr/not available/,
+ 'missing namespace fails without waiting on absent controllers');
+};
+
+subtest 'sysfs topology' => sub {
+ my $sys = $PVE::NVMeOF::SYSFS;
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1",
+ "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+ "$sys/block/nvme0n1",
+ );
+ my %attrs = (
+ address => 'traddr=192.0.2.1,trsvcid=4420',
+ transport => 'tcp',
+ subsysnqn => 'nqn.test',
+ state => 'connecting',
+ );
+ file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+ file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+ file_set_contents("$sys/block/nvme0n1/size", "8\n");
+ file_set_contents("$sys/block/nvme0n1/nsid", "1\n");
+ is(
+ PVE::NVMeOF::controllers()->{nvme1}->{state},
+ 'connecting',
+ 'reconnecting controller is still present',
+ );
+ is(
+ PVE::NVMeOF::path_controller(
+ PVE::NVMeOF::subsystem_controllers('nqn.test'),
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ ),
+ 'nvme1',
+ 'portal controller found',
+ );
+ is_deeply(
+ PVE::NVMeOF::namespaces('nqn.test'),
+ { nvme0n1 => { wwid => 'eui.abc', size => 4096, nsid => 1 } },
+ 'namespace facts',
+ );
+ is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+ my $value = PVE::NVMeOF::with_lock(sub {
+ my $out = '';
+ PVE::Tools::run_command(
+ [
+ '/usr/bin/perl',
+ '-MFcntl=:flock',
+ '-e',
+ 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*"); die "not inherited" if !@held; open(my $fh, ">>", $ARGV[0]) or die $!; die "lock missing" if flock($fh, LOCK_EX|LOCK_NB); print "locked\\n";',
+ "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+ ],
+ timeout => 5,
+ outfunc => sub { $out .= $_[0] },
+ );
+ return $out;
+ });
+ is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+ open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+ flock($lock, LOCK_EX) or die $!;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'busy operation never enters callback');
+ close($lock);
+
+ my $tools = Test::MockModule->new('PVE::Tools');
+ my $timed_out = 1;
+ $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+ my $retained;
+ eval {
+ PVE::NVMeOF::with_lock(sub {
+ my ($path) =
+ grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+ my ($fd) = $path =~ m{/(\d+)$};
+ open($retained, '>&', $fd) or die $!;
+ return 1;
+ });
+ };
+ like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+ $timed_out = 0;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+ close($retained);
+ is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my @commands;
+ my $controllers = {};
+ $nvme->mock(with_lock => sub { $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity =>
+ sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+ $nvme->mock(subsystem_controllers => sub { $controllers });
+ my $appear = 1;
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ is($opts{timeout}, 30, 'bounded connect');
+ push @commands, $cmd;
+ my %arg =
+ map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+ $controllers->{ 'nvme' . scalar(@commands) } =
+ { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' }
+ if $appear;
+ },
+ );
+ $net->mock(tcp_ping => sub { 1 });
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 2, 'one connect per missing portal');
+ ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+ is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+ @commands = ();
+ $controllers = {
+ nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+ nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+ };
+ $class->activate_storage('fabric', $cfg);
+ is_deeply(\@commands, [],
+ 'existing controllers are never reconnected, regardless of state');
+ $class->deactivate_storage('fabric', $cfg);
+ is_deeply(\@commands, [], 'deactivation does not remove controllers');
+ $controllers = {};
+ my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+ PVE::NVMeOF::note_connect_failure($cfg, $path);
+ ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 1, 'only the other portal connects');
+ ($controllers, $appear) = ({}, 0);
+ my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+ my $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ $class->activate_storage('fabric', $cfg);
+ }
+ like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+ ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+ $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+ eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+ like($@, qr/not a valid local disk/,
+ "$transport refuses without storage configuration");
+ }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index c2f3c65e..b1a39629 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
$disk->mock(verify_blockdev_path => sub { $device });
$disk->mock(is_partition => sub { 0 });
$disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
$disk->mock(is_iscsi_device => sub { 0 });
$disk->mock(is_mounted => sub { 0 });
$disk->mock(has_holder => sub { 0 });
@@ -65,6 +66,18 @@ $disk->mock(is_partition => sub { 0 });
$cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' };
eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
like($@, qr/base volume/, 'native base reference refuses');
+$cfg->{ids} =
+ { fabric =>
+ { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+ $@,
+ qr/serves images from this subsystem/,
+ 'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
$cfg->{ids} = {};
@order = @opens = ();
$tools->mock(file_get_contents => sub { 1024 });
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
` (14 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
An LVM storage only accepts an iSCSI LUN as base volume. Accept a
namespace of an NVMe over Fabrics storage too. Adding such an LVM
storage connects the subsystem, waits for the namespace and creates
the volume group on it, and activating it on another node connects
that node first.
The creation treats the namespace like any SAN LUN: it takes the
cluster-wide lock of the LUN and refuses a namespace that another
storage references.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 3 +--
src/test/run_san_nvme_tests.pl | 17 +++++++++++++++++
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index 4e885429..e7fb83d1 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -1158,9 +1158,8 @@ sub on_add_hook {
my $basecfg = PVE::Storage::storage_config($cfg, $baseid, 1);
die "base storage ID '$baseid' does not exist\n" if !$basecfg;
- # we only support iscsi for now
die "unsupported base type '$basecfg->{type}'"
- if $basecfg->{type} ne 'iscsi';
+ if $basecfg->{type} ne 'iscsi' && $basecfg->{type} ne 'nvmeof';
# Managed multipath requires the coalesced map, never a transient single-path PV.
die "base volume '$volname' is no multipath volume of storage '$baseid'\n"
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index b1a39629..43b1219e 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -117,4 +117,21 @@ is_deeply(
);
is_deeply(\@opens, [$link], 'VG creation holds the native by-id device');
+$cfg->{ids} = { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } };
+my @activation;
+$storage->mock(path => sub { $link });
+$storage->mock(activate_storage => sub { push @activation, 'storage' });
+$storage->mock(activate_volumes => sub { push @activation, 'volume' });
+$lvm->mock(
+ lvm_create_volume_group => sub {
+ is($_[0], $link, 'add hook passes native base');
+ is($_[3]->{exempt}, 'data', 'add hook exempts new storage');
+ },
+);
+PVE::Storage::LVMPlugin->on_add_hook(
+ 'data',
+ { base => 'fabric:nvme-eui.0123456789abcdef', vgname => 'data', shared => 1 },
+);
+is_deeply(\@activation, ['storage', 'volume'], 'LVM activates the NVMe base before creation');
+
done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (2 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
` (13 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Setting up an NVMe over Fabrics storage needs the NQN of the
subsystem and its portals. Add /nodes/{node}/scan/nvmeof, which asks
the discovery service at a portal for the NVMe/TCP subsystems it
offers to the node and lists them with their portals.
Without a port, it tries the discovery port 8009 and then the I/O
port 4420, where Linux targets answer discovery too. The scan needs
Datastore.Allocate on /storage, like the iSCSI scan.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Storage/Scan.pm | 49 +++++++++++++++++++++++
src/PVE/NVMeOF.pm | 62 ++++++++++++++++++++++++++++++
src/test/run_nvmeof_taint_tests.pl | 5 +++
src/test/run_nvmeof_tests.pl | 54 ++++++++++++++++++++++++++
4 files changed, 170 insertions(+)
diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm
index 96e1bd34..a9d05f79 100644
--- a/src/PVE/API2/Storage/Scan.pm
+++ b/src/PVE/API2/Storage/Scan.pm
@@ -6,6 +6,7 @@ use warnings;
# NOTE: This API endpoints are mounted by pve-manager's API2::Node module and pvesm CLI
use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
use PVE::RESTHandler;
use PVE::SafeSyslog;
@@ -49,6 +50,7 @@ __PACKAGE__->register_method({
{ method => 'iscsi' },
{ method => 'lvm' },
{ method => 'nfs' },
+ { method => 'nvmeof' },
{ method => 'pbs' },
{ method => 'san-luns' },
{ method => 'zfs' },
@@ -299,6 +301,53 @@ __PACKAGE__->register_method({
},
});
+__PACKAGE__->register_method({
+ name => 'nvmeofscan',
+ path => 'nvmeof',
+ method => 'GET',
+ description =>
+ 'List the NVMe over TCP subsystems offered to this node by a discovery controller.',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/storage', ['Datastore.Allocate']] },
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ portal => {
+ description => 'IPv4 or bracketed IPv6 discovery address with an optional port.'
+ . ' Without a port, try 8009 then 4420.',
+ type => 'string',
+ format => 'pve-storage-nvme-portal',
+ },
+ },
+ },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ portals => {
+ type => 'string',
+ description => 'The discovered TCP portals, comma separated.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my ($param) = @_;
+ my $res = PVE::NVMeOF::discover($param->{portal});
+ return [
+ map { {
+ subsysnqn => $_,
+ portals => join(',', map { PVE::NVMeOF::format_path($_) } $res->{$_}->@*),
+ } } sort keys %$res
+ ];
+ },
+});
+
__PACKAGE__->register_method({
name => 'lvmscan',
path => 'lvm',
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index 7a3140cb..d1dfe15b 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -5,10 +5,12 @@ use v5.36;
use Digest::SHA qw(sha1_hex);
use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
use File::Path qw(make_path);
+use JSON qw(decode_json);
use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
use PVE::Diskmanage;
use PVE::JSONSchema;
+use PVE::Network;
use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
our $NVME = '/usr/sbin/nvme';
@@ -246,4 +248,64 @@ sub connect_path($scfg, $path) {
});
}
+sub parse_discovery_log($raw) {
+ my $log = decode_json($raw);
+ die "invalid NVMe discovery log\n"
+ if ref($log) ne 'HASH' || ref($log->{records}) ne 'ARRAY';
+ my (%seen, %res);
+ for my $record ($log->{records}->@*) {
+ next
+ if ($record->{subtype} // '') ne 'nvme subsystem' || ($record->{trtype} // '') ne 'tcp';
+ my $nqn = $record->{subnqn} // next;
+ next
+ if $nqn !~ /^$NQN_PATTERN$/
+ || length($nqn) > 223
+ || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery';
+ my $addr = $record->{traddr} // '';
+ my $port = $record->{trsvcid} // '';
+ my $path = parse_portal(($addr =~ /:/ ? "[$addr]" : $addr) . ":$port") // next;
+ next if $seen{$nqn}->{ format_path($path) }++;
+ push $res{$nqn}->@*, $path;
+ }
+ return \%res;
+}
+
+sub discover($portal) {
+ my $path = parse_portal($portal) // die "invalid portal '$portal'\n";
+ my @ports = $portal =~ /:\d+$/ ? ($path->{trsvcid}) : (8009, 4420);
+ my $error;
+ for my $port (@ports) {
+ my $result = eval {
+ die "no answer from portal\n" if !PVE::Network::tcp_ping($path->{traddr}, $port, 2);
+ with_lock(sub {
+ assert_support();
+ my $host = host_identity();
+ my $out = '';
+ run_command(
+ [
+ $NVME,
+ 'discover',
+ '--transport=tcp',
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $port,
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--output-format=json',
+ ],
+ timeout => 30,
+ outfunc => sub($line) { $out .= "$line\n" },
+ );
+ return parse_discovery_log($out);
+ });
+ };
+ $error = $@;
+ return $result if !$error;
+ }
+ die "discovery at portal '$portal' failed - $error";
+}
+
1;
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
index 3c19f8cc..d136c4f1 100755
--- a/src/test/run_nvmeof_taint_tests.pl
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -38,6 +38,7 @@ $nvme->mock(
'all connect operands are validated and untainted',
);
PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ $opts{outfunc}->('{"records":[]}') if $opts{outfunc};
},
);
my $path = PVE::NVMeOF::parse_portal($portal);
@@ -53,6 +54,10 @@ for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.disc
eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
}
+my $net = Test::MockModule->new('PVE::Network');
+$net->mock(tcp_ping => sub { 1 });
+eval { PVE::NVMeOF::discover($portal) };
+is($@, '', 'discovery can execute with file-backed host identity in taint mode');
file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
eval { PVE::NVMeOF::host_identity() };
like($@, qr/nonzero UUID/, 'zero host ID refused');
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index 5ff96833..ead8d03c 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -9,6 +9,8 @@ use Test::More;
use Test::MockModule;
use PVE::NVMeOF;
use PVE::Storage;
+use PVE::API2::Storage::Scan;
+use JSON qw(encode_json);
use PVE::Tools qw(file_set_contents);
my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
@@ -240,4 +242,56 @@ subtest 'local disk setup rejects fabric namespaces by transport' => sub {
}
};
+subtest 'TCP discovery, fallback and API result' => sub {
+ my $record = {
+ subtype => 'nvme subsystem',
+ trtype => 'tcp',
+ subnqn => 'nqn.test',
+ traddr => '192.0.2.1',
+ trsvcid => '4420',
+ };
+ my $raw = encode_json({
+ records =>
+ [$record, $record, { %$record, trtype => 'rdma' }, { %$record, trsvcid => 70000 }],
+ });
+ is_deeply(
+ PVE::NVMeOF::parse_discovery_log($raw),
+ { 'nqn.test' => [{ traddr => '192.0.2.1', trsvcid => 4420 }] },
+ 'TCP only, unique valid portals',
+ );
+ for my $bad ('garbage', '{}', '[]') {
+ eval { PVE::NVMeOF::parse_discovery_log($bad) };
+ ok($@, 'malformed discovery is not an empty success');
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my (@ports, @commands);
+ our $locked;
+ $nvme->mock(with_lock => sub { local $locked = 1; $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'id' } });
+ $net->mock(tcp_ping => sub { push @ports, $_[1]; $_[1] != 8009 });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'discovery holds the NVMe lock');
+ is($opts{timeout}, 30, 'discovery command timeout');
+ push @commands, $cmd;
+ $opts{outfunc}->($raw);
+ },
+ );
+ my $rows = PVE::API2::Storage::Scan->nvmeofscan({ node => 'n1', portal => '192.0.2.1' });
+ is_deeply(\@ports, [8009, 4420], 'unspecified port tries discovery then I/O port');
+ is_deeply($rows, [{ subsysnqn => 'nqn.test', portals => '192.0.2.1:4420' }], 'API shape');
+ eval {
+ PVE::JSONSchema::validate(
+ $rows,
+ PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns},
+ );
+ };
+ is($@, '', 'API schema accepts result');
+ @ports = ();
+ PVE::NVMeOF::discover('[2001:db8::1]:4421');
+ is_deeply(\@ports, [4421], 'explicit port does not fall back');
+};
+
done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (3 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
` (12 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Add 'pvesm scan nvmeof <portal>', which prints the subsystems a
portal offers to the node with their portals, like the iSCSI scan
prints targets. That gives the values for a new storage on the
command line, with the usual output formats for scripts.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/CLI/pvesm.pm | 13 +++++++++++++
src/test/run_nvmeof_tests.pl | 12 ++++++++++++
2 files changed, 25 insertions(+)
diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm
index 347fbffa..aa3c587e 100755
--- a/src/PVE/CLI/pvesm.pm
+++ b/src/PVE/CLI/pvesm.pm
@@ -736,6 +736,19 @@ our $cmddef = {
$print_api_result,
$PVE::RESTHandler::standard_output_options,
],
+ nvmeof => [
+ "PVE::API2::Storage::Scan",
+ 'nvmeofscan',
+ ['portal'],
+ { node => $nodename },
+ sub {
+ my ($data, $schema, $options) = @_;
+ PVE::CLIFormatter::print_api_result(
+ $data, $schema, [qw(subsysnqn portals)], $options,
+ );
+ },
+ $PVE::RESTHandler::standard_output_options,
+ ],
'san-luns' => [
"PVE::API2::Storage::Scan",
'sanlunscan',
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index ead8d03c..ef15d17a 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -292,6 +292,18 @@ subtest 'TCP discovery, fallback and API result' => sub {
@ports = ();
PVE::NVMeOF::discover('[2001:db8::1]:4421');
is_deeply(\@ports, [4421], 'explicit port does not fall back');
+ require PVE::CLI::pvesm;
+ my $formatter = Test::MockModule->new('PVE::CLIFormatter');
+ my $schema = PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns};
+ $formatter->mock(
+ print_api_result => sub {
+ is($_[0], $rows, 'CLI forwards complete discovery rows');
+ is($_[1], $schema, 'CLI forwards discovery schema');
+ is($_[3]->{'output-format'}, 'json', 'CLI supports structured output');
+ },
+ );
+ $PVE::CLI::pvesm::cmddef->{scan}->{nvmeof}->[4]
+ ->($rows, $schema, { 'output-format' => 'json' });
};
done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (4 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
` (11 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The SAN LUN inventory lists NVMe namespaces without relation to an
NVMe over Fabrics storage. Report the subsystem NQN and the namespace
ID of a namespace, and expect one path per portal of the storage that
connects its subsystem, so that a namespace with fewer paths shows as
degraded.
A namespace counts as used by the LVM storage that has it as base
volume, else by the storage of its subsystem if that one serves the
namespaces as images.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Storage/Scan.pm | 10 +++++++++
src/PVE/Storage/SAN.pm | 25 ++++++++++++++++++----
src/test/run_san_lun_scan_tests.pl | 33 +++++++++++++++++++++++++++++-
3 files changed, 63 insertions(+), 5 deletions(-)
diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm
index a9d05f79..35c1c71f 100644
--- a/src/PVE/API2/Storage/Scan.pm
+++ b/src/PVE/API2/Storage/Scan.pm
@@ -596,6 +596,16 @@ __PACKAGE__->register_method({
type => 'string',
optional => 1,
},
+ 'nvme-subsystem' => {
+ description => 'The NVMe subsystem NQN.',
+ type => 'string',
+ optional => 1,
+ },
+ 'nvme-nsid' => {
+ description => 'The namespace ID within the NVMe subsystem.',
+ type => 'integer',
+ optional => 1,
+ },
'iscsi-target' => {
description => "The IQN of the iSCSI target the LUN is attached through.",
type => 'string',
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index f3682106..b281177b 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -15,6 +15,7 @@ use PVE::Multipath;
use PVE::Multipath::Generator;
use PVE::Multipath::Config;
use PVE::Multipath::Identity;
+use PVE::NVMeOF;
use PVE::Storage;
use PVE::Storage::LVMPlugin;
use PVE::Tools qw(file_read_firstline dir_glob_foreach run_command);
@@ -34,7 +35,7 @@ sub scan_luns($cfg) {
PVE::Multipath::Config::wwids_for_node($mpcfg, PVE::INotify::nodename())->@*;
my %listed = map { $_ => 1 } PVE::Multipath::list_etc_multipath_wwids()->@*;
my $derived = PVE::Multipath::Generator::derived_wwids($cfg, $ids);
- my (%storage, %expected, %vg_owner, %target_owner, %base_owner);
+ my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %nvme_owner);
for my $sid (sort keys %$derived) {
for my $wwid (keys $derived->{$sid}->{wwids}->%*) {
@@ -52,7 +53,9 @@ sub scan_luns($cfg) {
$vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
$target_owner{ $scfg->{target} } //= $sid
if defined($scfg->{target}) && PVE::Storage::san_storage_serves_images($scfg);
- $base_owner{$1} //= $sid if ($scfg->{base} // '') =~ m/(?:^|\.)dm-uuid-mpath-(.+)$/;
+ $base_owner{$1} //= $sid
+ if ($scfg->{base} // '') =~ m/(?:^|[.:])(?:dm-uuid-mpath|nvme)-(.+)$/;
+ $nvme_owner{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
}
my %rows;
@@ -96,7 +99,19 @@ sub scan_luns($cfg) {
: $storage{$wwid} ? 'storage'
: 'unmanaged';
}
- $row->{multipath} = 'native' if $name =~ /^nvme\d+n\d+$/;
+ if ($name =~ /^nvme\d+n\d+$/) {
+ $row->{multipath} = 'native';
+ my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn");
+ if (defined($nqn)) {
+ $row->{'nvme-subsystem'} = $nqn;
+ my $nsid = file_read_firstline("/sys/block/$name/nsid");
+ $row->{'nvme-nsid'} = int($nsid) if defined($nsid) && $nsid =~ /^\d+$/;
+ if (my $sid = $nvme_owner{$nqn}) {
+ $row->{'paths-expected'} =
+ scalar(PVE::NVMeOF::storage_paths($cfg->{ids}->{$sid})->@*);
+ }
+ }
+ }
for my $key (qw(transport vendor model serial)) {
my $value = $disk->{$key};
$row->{$key} //= $value if defined($value) && $value ne 'unknown';
@@ -150,8 +165,10 @@ sub scan_luns($cfg) {
$row->{'paths-expected'},
);
}
+ my $nvme = $nvme_owner{ $row->{'nvme-subsystem'} // '' };
+ undef $nvme if $nvme && !PVE::Storage::san_storage_serves_images($cfg->{ids}->{$nvme});
my $owner = $base_owner{ $row->{wwid} } // $vg_owner{ $row->{vgname} // '' }
- // $target_owner{ $row->{'iscsi-target'} // '' };
+ // $target_owner{ $row->{'iscsi-target'} // '' } // $nvme;
$row->{'used-by-storage'} = $owner if defined($owner);
}
return [map { $rows{$_} } sort keys %rows];
diff --git a/src/test/run_san_lun_scan_tests.pl b/src/test/run_san_lun_scan_tests.pl
index 94df0d5c..3578a3af 100755
--- a/src/test/run_san_lun_scan_tests.pl
+++ b/src/test/run_san_lun_scan_tests.pl
@@ -165,7 +165,14 @@ is($rows->[0]->{'by-id'}, '/dev/disk/by-id/scsi-3600plain', 'by-id link of the s
is($rows->[0]->{devpath}, '/dev/sdc', 'the row names the path that its by-id link resolves to');
$disks = { nvme0n1 => { devpath => '/dev/nvme0n1', size => 100, transport => 'nvme-fc' } };
-$scan->mock(file_read_firstline => sub { 'eui.0123456789abcdef' });
+$scan->mock(file_read_firstline => sub { $_[0] =~ m{/nsid$} ? 1 : 'eui.0123456789abcdef' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
+$cfg->{ids}->{fabric} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ 'nvme-portals' => '192.0.2.1,192.0.2.2',
+ content => { images => 1 },
+};
$scan->unmock('stable_path');
$scan->mock(abs_path => sub { '/dev/nvme0n1' });
$rows = PVE::Storage::SAN::scan_luns($cfg);
@@ -173,6 +180,30 @@ is($rows->[0]->{multipath}, 'native', 'NVMe uses kernel-native multipath');
is($rows->[0]->{'by-id'}, '/dev/disk/by-id/nvme-eui.0123456789abcdef', 'NVMe stable name');
eval { PVE::JSONSchema::validate($rows, $schema) };
is($@, '', 'native row matches the API schema');
+is($rows->[0]->{'nvme-subsystem'}, 'nqn.test', 'subsystem NQN exposed');
+is($rows->[0]->{'nvme-nsid'}, 1, 'namespace ID exposed');
+is($rows->[0]->{'used-by-storage'}, 'fabric', 'a storage serving the namespaces uses them');
+is($rows->[0]->{'paths-expected'}, 2, 'one expected path per configured portal');
+
+for my $count (1, 3) {
+ $config->mock(
+ read_config => sub { {
+ ids => { 'eui.0123456789abcdef' => { type => 'wwid', 'paths-expected' => $count } },
+ } },
+ );
+ my $entry = PVE::Storage::SAN::scan_luns($cfg);
+ is($entry->[0]->{'paths-expected'}, 2, "a multipath entry does not change it ($count)");
+}
+$config->mock(read_config => sub { { ids => { '3600map' => { type => 'wwid' } } } });
+{
+ local $cfg->{ids}->{fabric}->{content} = { none => 1 };
+ my $plain = PVE::Storage::SAN::scan_luns($cfg);
+ is($plain->[0]->{'used-by-storage'}, undef, 'a connection storage does not');
+ is($plain->[0]->{'paths-expected'}, 2, 'but still tells the expected paths');
+}
+$cfg->{ids}->{data}->{base} = 'fabric:nvme-eui.0123456789abcdef';
+$rows = PVE::Storage::SAN::scan_luns($cfg);
+is($rows->[0]->{'used-by-storage'}, 'data', 'LVM base consumer takes precedence');
$disk->mock(is_partition => sub { 1 });
ok(!PVE::Storage::SAN::is_san_device('/dev/sdb1'), 'a partition is never a LUN');
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (5 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
` (10 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace in use would lose its last
usable path. Reading needs Sys.Audit on the node or
Datastore.Allocate on /storage, the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 182 +++++++++++++++++++++++++++++++
src/PVE/Diskmanage.pm | 16 +--
src/PVE/NVMeOF.pm | 41 +++++++
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 151 +++++++++++++++++++++++++
6 files changed, 385 insertions(+), 9 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index 9ab978b0..1be9ab04 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..57e70df4
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,182 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme\d+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my (%owners, %counts);
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ $counts{$nqn} //= scalar(keys PVE::NVMeOF::namespaces($nqn)->%*);
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts{$nqn},
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties =>
+ { node => get_standard_option('pve-node'), controller => $controller_schema },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub {
+ my ($param) = @_;
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded($param->{controller});
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 7bee3b08..3141ca7a 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -898,6 +898,13 @@ sub get_multipath_disks {
return $disklist;
}
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -933,15 +940,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index d1dfe15b..c81cb72f 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -308,4 +308,45 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+sub disconnect_guarded($controller) {
+ my ($name) = $controller =~ /\A(nvme\d+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ next
+ if grep { $_->{device} ne "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_) }
+ @$paths;
+ my $ns = $namespaces->{$head} // die
+ "cannot disconnect '$name' - namespace '$head' has no device on this node to check for use\n";
+ my $device = "/dev/$head";
+ die
+ "cannot disconnect '$name' - namespace '$head' is in use and has no other usable path\n"
+ if PVE::Diskmanage::is_mounted($device) || PVE::Diskmanage::has_holder($device);
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index 124612da..73183dc1 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -77,6 +77,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..3849dba5
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,151 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Test::More;
+use Test::MockModule;
+use PVE::API2::NVMeOF;
+use PVE::Storage::SAN;
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status =>
+ sub { {
+ nvme0n1 => { paths => [{ device => '/dev/nvme1', state => 'live' }, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'unused namespace can lose its final path');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(namespaces => sub { { nvme0n1 => {} } });
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 08/17] multipath: report the NVMe host NQN of each node
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (6 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
` (9 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The status report of a node has its iSCSI initiator name, so that
the cluster-wide status shows the identities an admin needs for the
access rules of an array in one place. Add the NVMe host NQN, which
an NVMe over Fabrics target needs for the same purpose. A node
without a valid NVMe host identity reports none.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Multipath.pm | 2 +-
src/PVE/Multipath/Status.pm | 6 +++++-
src/test/run_multipath_api_tests.pl | 12 +++++++++++-
3 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index d5d2df9d..1d7f5707 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -86,7 +86,7 @@ my $node_schema = {
state => { type => 'string', enum => ['ok', 'unavailable', 'no-report'] },
time => { type => 'integer', optional => 1 },
map { $_ => { type => 'string', optional => 1 } }
- qw(error apply-error iscsi-initiator-name),
+ qw(error apply-error iscsi-initiator-name nvme-host-nqn),
},
};
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index 58d11f5d..9dc8e2cf 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -48,6 +48,9 @@ sub update_node_status() {
my $initiator = eval { PVE::Storage::ISCSIPlugin::read_initiator_info() };
$report->{'iscsi-initiator-name'} = $initiator->{name}
if $initiator && defined($initiator->{name});
+ require PVE::NVMeOF;
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ $report->{'nvme-host-nqn'} = $host->{nqn} if $host;
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running\n" if !PVE::Multipath::is_running();
@@ -134,7 +137,8 @@ sub cluster_status() {
|| !defined($report->{time})
|| time() - $report->{time} > 180;
$entry->{$_} = $report->{$_}
- for grep { defined($report->{$_}) } qw(error apply-error iscsi-initiator-name);
+ for grep { defined($report->{$_}) }
+ qw(error apply-error iscsi-initiator-name nvme-host-nqn);
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
next if $entry->{state} ne 'ok';
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 77ee3b9f..9bf44733 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -14,6 +14,7 @@ use Test::More;
use Test::MockModule;
use PVE::API2::Multipath;
use PVE::Multipath::Generator;
+use PVE::NVMeOF;
use PVE::Storage;
subtest 'explicit entries are ordinary configuration writes' => sub {
@@ -156,9 +157,12 @@ subtest 'an unconfigured node does not write drop-ins or reload' => sub {
}
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
$iscsi->mock(read_initiator_info => sub { {} });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $report;
$status->mock(publish => sub { $report = $_[0] });
+
for my $ids ({}, { plain => { type => 'iscsi', target => 'iqn.test' } }) {
$storage->mock(config => sub { { ids => $ids } });
PVE::Multipath::Status::update_node_status();
@@ -215,15 +219,20 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.2025-01.example:node', id => 'unused' } });
$mp->mock(is_supported => sub { 0 });
PVE::Multipath::Status::update_node_status();
is($report->{error}, 'multipath-tools is not installed', 'missing tools reason only');
+ is($report->{'nvme-host-nqn'}, 'nqn.2025-01.example:node', 'host NQN reported in any case');
is_deeply(\@order, [qw(apply publish)], 'single call owns ordering');
is($report->{'apply-error'}, 'reload failed', 'apply failure is separate');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
$mp->mock(is_supported => sub { 1 });
$mp->mock(is_running => sub { 0 });
PVE::Multipath::Status::update_node_status();
is($report->{error}, 'multipathd is not running', 'stopped daemon reason only');
+ ok(!exists($report->{'nvme-host-nqn'}), 'no host NQN without a valid host identity');
$mp->mock(is_running => sub { 1 });
$mp->mock(get_maps => sub { die "bad JSON\n" });
PVE::Multipath::Status::update_node_status();
@@ -278,7 +287,7 @@ subtest 'API returns final node and LUN states' => sub {
},
],
}),
- n2 => encode_json({ time => $NOW, luns => [] }),
+ n2 => encode_json({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_json({ time => $NOW - 181, luns => [] }),
n4 => encode_json({ time => $NOW, luns => [] }),
n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
@@ -307,6 +316,7 @@ subtest 'API returns final node and LUN states' => sub {
'final node states',
);
is($res->{nodes}->{n3}->{time}, $NOW - 181, 'old report time retained');
+ is($res->{nodes}->{n2}->{'nvme-host-nqn'}, 'nqn.test:n2', 'host NQN passed on');
my $lun = $res->{luns}->[0];
is($lun->{state}, 'missing', 'worst known state');
is($lun->{nodes}->{n1}->{state}, 'optimal', 'observed healthy map');
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (7 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
` (8 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the LUNs that multipathd handles on
each node. The namespaces of an NVMe over Fabrics storage use the
native multipath of the kernel and were missing from it, so a lost
path or a node without access to a namespace only showed in the LUN
list of that node.
Let each node report the namespaces of the subsystems such a storage
names, with the path counts its LUN list already shows, and expect
them on the nodes of the storage like the LUNs of an iSCSI storage.
They do not depend on multipathd, so a node without it reports them
as well and only the state of its maps stays unknown.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Multipath.pm | 6 ++
src/PVE/Multipath/Status.pm | 58 +++++++++++++++---
src/test/run_multipath_api_tests.pl | 91 ++++++++++++++++++++++++++++-
3 files changed, 145 insertions(+), 10 deletions(-)
diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index 1d7f5707..82d14c9f 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -121,6 +121,12 @@ __PACKAGE__->register_method({
wwid => { type => 'string', description => 'The LUN WWID.' },
explicit =>
{ type => 'boolean', description => 'An explicit entry exists.' },
+ native => {
+ type => 'boolean',
+ optional => 1,
+ description => 'An NVMe namespace, which uses the native'
+ . ' multipath of the kernel.',
+ },
name => { type => 'string', optional => 1 },
'used-by-storage' => { type => 'string', optional => 1 },
'multipath-storage' => {
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index 9dc8e2cf..092d26a6 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -3,6 +3,7 @@ package PVE::Multipath::Status;
use v5.36;
use JSON qw(decode_json);
use PVE::Cluster;
+use PVE::Diskmanage;
use PVE::INotify;
use PVE::Multipath;
use PVE::Multipath::Config;
@@ -21,6 +22,39 @@ sub worst_state(@states) {
return $worst;
}
+# Namespaces use the native multipath of the kernel, so their state does not depend on multipathd.
+my sub namespace_luns($pv_vgs) {
+ require PVE::Storage;
+ my $cfg = PVE::Storage::config();
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %luns;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ next if $scfg->{type} ne 'nvmeof';
+ my $expected = eval { scalar(PVE::NVMeOF::storage_paths($scfg)->@*) };
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ for my $name (sort keys %$namespaces) {
+ my $wwid = $namespaces->{$name}->{wwid};
+ next if $luns{$wwid};
+ my $paths = $status->{$name} // { active => 0, paths => [] };
+ my $total = scalar($paths->{paths}->@*);
+ my $lun = $luns{$wwid} = {
+ wwid => $wwid,
+ native => 1,
+ size => $namespaces->{$name}->{size},
+ storage => $sid,
+ 'paths-total' => $total,
+ 'paths-active' => $paths->{active},
+ state => PVE::Multipath::derive_health($paths->{active}, $total, $expected),
+ };
+ $lun->{'paths-expected'} = $expected if defined($expected);
+ my $vg = $pv_vgs->{"/dev/$name"};
+ $lun->{vgname} = $vg if defined($vg);
+ }
+ }
+ return [map { $luns{$_} } sort keys %luns];
+}
+
sub publish($report) {
my $node = PVE::INotify::nodename();
my $now = time();
@@ -51,11 +85,11 @@ sub update_node_status() {
require PVE::NVMeOF;
my $host = eval { PVE::NVMeOF::host_identity() };
$report->{'nvme-host-nqn'} = $host->{nqn} if $host;
+ require PVE::Storage::LVMPlugin;
+ my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running\n" if !PVE::Multipath::is_running();
- require PVE::Storage::LVMPlugin;
- my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
for my $map (PVE::Multipath::get_maps()->@*) {
my $wwid = $map->{wwid} // next;
my $info = $apply->{wwids}->{$wwid} // {};
@@ -82,8 +116,11 @@ sub update_node_status() {
if (my $error = $@) {
chomp $error;
$report->{error} = $error;
- delete $report->{luns};
+ $report->{luns} = [];
}
+ my $namespaces = eval { namespace_luns($pv_vgs) };
+ warn "multipath: cannot query NVMe namespaces - $@" if $@;
+ push $report->{luns}->@*, @$namespaces if $namespaces;
eval { publish($report) };
warn "multipath: cannot publish node status - $@" if $@;
}
@@ -96,7 +133,7 @@ sub cluster_status() {
my $all_nodes = PVE::Cluster::get_nodelist();
my $members = PVE::Cluster::get_members();
my $raw = PVE::Cluster::get_node_kv($KEY);
- my (%nodes, %luns, %expected, %observed, %vg_owner);
+ my (%nodes, %luns, %expected, %observed, %vg_owner, %lists);
my $storage_nodes = sub($scfg) {
return $scfg->{nodes} && $scfg->{nodes}->%* ? [keys $scfg->{nodes}->%*] : $all_nodes;
};
@@ -119,11 +156,12 @@ sub cluster_status() {
my $scfg = $storage->{ids}->{$sid};
$vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
next if $scfg->{type} ne 'lvm';
- my ($wwid) = ($scfg->{base} // '') =~ /(?:^|\.)dm-uuid-mpath-(\S+)$/;
+ my ($base, $volname) = PVE::Storage::parse_volume_id($scfg->{base} // '', 1);
+ my ($kind, $wwid) = ($volname // '') =~ /(?:^|\.)(dm-uuid-mpath|nvme)-(\S+)$/;
next if !defined($wwid);
$luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $luns{$wwid}->{native} = 1 if $kind eq 'nvme';
$luns{$wwid}->{'used-by-storage'} //= $sid;
- my ($base) = $scfg->{base} =~ /^([^:]+):/;
$luns{$wwid}->{'multipath-storage'} //= $base if $requests_maps->($base);
$expected{$wwid}->{$_} = 1 for $storage_nodes->($scfg)->@*;
}
@@ -140,11 +178,13 @@ sub cluster_status() {
for grep { defined($report->{$_}) }
qw(error apply-error iscsi-initiator-name nvme-host-nqn);
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
- next if $entry->{state} ne 'ok';
+ # an unavailable node lists its namespaces only, or nothing
+ $lists{$node} = 1 if $report->{luns};
for my $lun (($report->{luns} // [])->@*) {
my $wwid = $lun->{wwid} // next;
my $row = $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $row->{native} = 1 if $lun->{native};
$row->{$_} //= $lun->{$_} for grep { defined($lun->{$_}) } qw(name size);
my $owner = defined($lun->{vgname}) ? $vg_owner{ $lun->{vgname} } : undef;
$row->{'used-by-storage'} //= $owner if defined($owner);
@@ -176,8 +216,8 @@ sub cluster_status() {
if ($observed{$wwid}->{$node}) {
$row->{nodes}->{$node} = $observed{$wwid}->{$node};
} elsif ($expected{$wwid}->{$node}) {
- $row->{nodes}->{$node} =
- { state => $nodes{$node}->{state} eq 'ok' ? 'missing' : 'unknown' };
+ my $listed = $nodes{$node}->{state} eq 'ok' || ($row->{native} && $lists{$node});
+ $row->{nodes}->{$node} = { state => $listed ? 'missing' : 'unknown' };
}
}
$row->{state} = worst_state(map { $_->{state} } values $row->{nodes}->%*);
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 9bf44733..bd62f756 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -215,7 +215,12 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $mp = Test::MockModule->new('PVE::Multipath');
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $disks = Test::MockModule->new('PVE::Diskmanage');
my ($report, @order);
+ my $ids = {};
+ $storage->mock(config => sub { { ids => $ids } });
+ $disks->mock(get_nvme_path_status => sub { { nvme1n1 => { active => 1, paths => [{}] } } });
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
@@ -249,6 +254,38 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
is($report->{luns}->[0]->{'paths-expected'}, 3, 'expected count reported');
is($report->{luns}->[0]->{state}, 'degraded', 'fewer paths than expected');
ok(!exists($report->{supported}), 'no separate capability state');
+
+ $ids->{nvme} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test:sub',
+ 'nvme-portals' => '192.0.2.1,192.0.2.2',
+ };
+ $nvme->mock(
+ namespaces => sub($nqn) {
+ return $nqn eq 'nqn.test:sub'
+ ? { nvme1n1 => { wwid => 'uuid.1', size => 4096 } }
+ : {};
+ },
+ );
+ my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+ $lvm->mock(cached_pv_volume_groups => sub { { '/dev/nvme1n1' => 'vgns' } });
+ my $namespace = {
+ wwid => 'uuid.1',
+ native => 1,
+ size => 4096,
+ storage => 'nvme',
+ vgname => 'vgns',
+ 'paths-total' => 1,
+ 'paths-active' => 1,
+ 'paths-expected' => 2,
+ state => 'degraded',
+ };
+ PVE::Multipath::Status::update_node_status();
+ is_deeply($report->{luns}->[1], $namespace, 'namespace with fewer paths than portals');
+ $mp->mock(is_running => sub { 0 });
+ PVE::Multipath::Status::update_node_status();
+ is($report->{error}, 'multipathd is not running', 'stopped daemon still reported');
+ is_deeply($report->{luns}, [$namespace], 'namespaces do not need multipathd');
};
subtest 'API returns final node and LUN states' => sub {
@@ -285,12 +322,31 @@ subtest 'API returns final node and LUN states' => sub {
'paths-active' => 2,
storage => 'scoped',
},
+ {
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'optimal',
+ 'paths-total' => 2,
+ 'paths-active' => 2,
+ storage => 'nvme',
+ },
],
}),
n2 => encode_json({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_json({ time => $NOW - 181, luns => [] }),
n4 => encode_json({ time => $NOW, luns => [] }),
- n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
+ n5 => encode_json({
+ time => $NOW,
+ error => 'multipath-tools is not installed',
+ luns => [{
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'degraded',
+ 'paths-total' => 2,
+ 'paths-active' => 1,
+ storage => 'nvme',
+ }],
+ }),
);
$cluster->mock(get_node_kv => sub { \%raw });
$config->mock(read_config => sub { { ids => { '3600a' => { type => 'wwid' } } } });
@@ -306,6 +362,8 @@ subtest 'API returns final node and LUN states' => sub {
gone => { type => 'iscsi', multipath => 1, nodes => { departed => 1 } },
lvmgone => { type => 'lvm', base => 'gone:0.0.0.dm-uuid-mpath-3600e' },
scoped => { type => 'iscsi', multipath => 1, nodes => { n2 => 1 } },
+ nvme => { type => 'nvmeof', subsysnqn => 'nqn.test:sub' },
+ lvmns => { type => 'lvm', base => 'nvme:nvme-uuid.2', nodes => { n5 => 1 } },
},
} },
);
@@ -345,6 +403,37 @@ subtest 'API returns final node and LUN states' => sub {
'the nodes of a shared LVM expect the map',
);
is($lun{'3600d'}->{'used-by-storage'}, 'shared', 'shared LVM attributed by volume group');
+ my $namespace = $lun{'uuid.1'};
+ ok($namespace->{native}, 'namespace marked as native');
+ ok(!exists($namespace->{'multipath-storage'}), 'a namespace has no requested map');
+ is_deeply(
+ { map { $_ => $namespace->{nodes}->{$_}->{state} } keys $namespace->{nodes}->%* },
+ {
+ n1 => 'optimal',
+ n2 => 'missing',
+ n3 => 'unknown',
+ n4 => 'unknown',
+ n5 => 'degraded',
+ },
+ 'namespace state of a node without multipathd counts',
+ );
+ is($lun->{nodes}->{n5}->{state}, 'unknown', 'no map state from a node without multipathd');
+ is_deeply(
+ $lun{'uuid.2'},
+ {
+ wwid => 'uuid.2',
+ explicit => 0,
+ native => 1,
+ 'used-by-storage' => 'lvmns',
+ state => 'missing',
+ nodes => { n5 => { state => 'missing' } },
+ },
+ 'base namespace that a node without multipathd does not report',
+ );
+ $raw{n5} = encode_json({ time => $NOW, error => 'too many LUNs to report' });
+ $res = PVE::API2::Multipath->status({});
+ %lun = map { $_->{wwid} => $_ } $res->{luns}->@*;
+ is($lun{'uuid.2'}->{nodes}->{n5}->{state}, 'unknown', 'node without a list');
eval {
PVE::JSONSchema::validate(
$res,
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH docs 10/17] storage: document NVMe over Fabrics storages
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (8 preceding siblings ...)
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
` (7 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of this
first version.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
pve-storage-multipath.adoc | 4 +-
pve-storage-nvmeof.adoc | 161 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 20 +++--
3 files changed, 177 insertions(+), 8 deletions(-)
create mode 100644 pve-storage-nvmeof.adoc
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index d4503a7..c7b2143 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node.
{pve} configures maps cluster-wide and shows their state under *Datacenter ->
Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its
LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe
-namespaces use kernel-native multipath instead; the procedures below cover SCSI
-LUNs.
+namespaces use kernel-native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs.
Prerequisites
~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..d2faf0f
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,161 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+ package recommends it, but installations without recommended packages need it
+ added separately. Its installation creates the node's host NQN in
+ `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+ image share them. Give each a new identity before it connects, with
+ `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+ *Datacenter -> Storage -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+ addresses that every node can reach; host names are not accepted.
+. Keep native NVMe multipath enabled, which is the kernel default
+ (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach
+ the array and select *Add Storage -> LVM on NVMe over Fabrics*.
+. Enter a discovery portal, select the subsystem, check the portals and name
+ the connection storage. This storage connects to the subsystem with
+ `content none`, while the LVM storage manages guest disks. Continuing creates
+ or reuses the connection storage before you select a namespace.
+. Select the namespace, choose the storage ID and volume group, then confirm.
+ Leave *Nodes* empty for all nodes.
+. Check the state of the namespace on each node of the storage, which the
+ setup shows once the storage is added, as described for LUNs in
+ xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+ check its access to the namespace on the array before placing guests on the
+ storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace reads *OK (1 path)*.
+Portals can also be changed later in the edit dialog of the connection storage;
+each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection
+storage. Enable *Use namespaces directly* there to use whole namespaces as guest
+disks, or add an *LVM* storage on a namespace later, with the connection storage
+as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <connection> --subsysnqn <nqn> --nvme-portals <portals> --content none
+# pvesm list <connection>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<connection>:nvme-<wwid>`. Adding
+the LVM storage creates the volume group if it does not exist yet. Creating it
+is refused if the namespace holds data or another storage references it. To
+reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later.
+
+nvme-portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+nvme-fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace were lost for that
+long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+ subsysnqn nqn.2010-06.com.example:array1
+ nvme-portals 192.0.2.10,192.0.2.11
+ content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types |Image formats |Shared |Snapshots |Clones
+|images none |raw |yes |no |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace there to inspect its paths.
+Its *State* is *Degraded* while a path is unusable or fewer paths exist than
+the storage has portals, and *Failed* without a usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal:
+*Connected*, or *Reconnecting* while the kernel retries a lost path.
+
+An enabled storage connects the portals a node has no controller for. A portal
+that failed is retried after a minute. Removing a portal, or disabling or
+removing the storage, does not disconnect anything. To remove such a path,
+select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and
+press *Disconnect*, or run
+`pvesh delete /nodes/<node>/nvmeof/controller/<controller>`. This is refused if
+a namespace that is mounted, in use by another device, or referenced by a
+storage would lose its last usable path. A storage that still lists the portal
+connects to it again.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS. Use a
+ storage network that only trusted hosts can reach.
+* NVMe/FC namespaces need no storage of this type. Set them up with
+ *Datacenter -> Storage -> SAN -> Add Storage -> LVM on Fibre Channel*, see
+ xref:storage_multipath_setup[Multipath].
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 5668a41..442ac2d 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
|LVM-thin |lvmthin |block |no |yes |yes
|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes
|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes
+|NVMe-oF |nvmeof |block |yes^3^|yes^4^ |yes
|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes
|Ceph/RBD |rbd |block |yes |yes |yes
|ZFS over iSCSI |zfs |block |yes |yes |yes
@@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
hours. If your setup is affected, create and remove snapshots while the VM is
shut down, expecting a long task duration.
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster.
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
These snapshots use separate volumes for the snapshot data and layer them. For
@@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have
totally different content.
Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use the guided setup under
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use the guided setup under
*Datacenter -> Storage -> SAN -> Add Storage* instead. It creates the storages
-that are needed and enables multipath for the LUN.
+that are needed and enables multipath for a LUN.
Storage Pools
@@ -427,6 +431,8 @@ See Also
* link:/wiki/Storage:_Multipath[Storage: Multipath]
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
* link:/wiki/Storage:_LVM[Storage: LVM]
* link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -471,6 +477,8 @@ include::pve-storage-iscsidirect.adoc[]
include::pve-storage-multipath.adoc[]
+include::pve-storage-nvmeof.adoc[]
+
include::pve-storage-rbd.adoc[]
include::pve-storage-cephfs.adoc[]
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (9 preceding siblings ...)
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
` (6 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Make the node status, the controller list and the disconnect call of
pve-storage reachable at /nodes/{node}/nvmeof for the web UI.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/API2/Nodes.pm | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index 2a7ab4133..3495c3e17 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -53,6 +53,7 @@ use PVE::API2::Hardware;
use PVE::API2::ISCSI;
use PVE::API2::LXC::Status;
use PVE::API2::LXC;
+use PVE::API2::NVMeOF;
use PVE::API2::Network;
use PVE::API2::Network::SDN::Nodes::Status;
use PVE::API2::NodeConfig;
@@ -166,6 +167,11 @@ __PACKAGE__->register_method({
path => 'iscsi',
});
+__PACKAGE__->register_method({
+ subclass => "PVE::API2::NVMeOF",
+ path => 'nvmeof',
+});
+
__PACKAGE__->register_method({
subclass => "PVE::API2::Capabilities",
path => 'capabilities',
@@ -252,6 +258,7 @@ __PACKAGE__->register_method({
{ name => 'migrateall' },
{ name => 'netstat' },
{ name => 'network' },
+ { name => 'nvmeof' },
{ name => 'qemu' },
{ name => 'query-oci-repo-tags' },
{ name => 'query-url-metadata' },
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (10 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
` (5 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Show storages of the new type with their subsystem in the storage list
and add their dialog. Adding one lists the subsystems a discovery
portal reports and fills in the portals of the selected one, which can
be changed. Portals, nodes, content and the fast I/O fail timeout can
be edited later.
The subsystem selector builds on the iSCSI target selector, whose
store fields and sort key a subclass can now set.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/Utils.js | 6 ++
www/manager6/dc/StorageView.js | 5 +-
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/NVMeOFEdit.js | 127 +++++++++++++++++++++++++++++
5 files changed, 141 insertions(+), 3 deletions(-)
create mode 100644 www/manager6/storage/NVMeOFEdit.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index ebdc3f802..fbd866b4b 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -377,6 +377,7 @@ JSSRC= \
storage/LvmThinEdit.js \
storage/BTRFSEdit.js \
storage/NFSEdit.js \
+ storage/NVMeOFEdit.js \
storage/PBSEdit.js \
storage/RBDEdit.js \
storage/StatusView.js \
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index 865ba44ff..4a97358ac 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -869,6 +869,12 @@ Ext.define('PVE.Utils', {
faIcon: 'building',
backups: false,
},
+ nvmeof: {
+ name: 'NVMe over Fabrics',
+ ipanel: 'NVMeOFInputPanel',
+ faIcon: 'building',
+ backups: false,
+ },
cephfs: {
name: 'CephFS',
ipanel: 'CephFSInputPanel',
diff --git a/www/manager6/dc/StorageView.js b/www/manager6/dc/StorageView.js
index 54fbde711..4fc49f5cd 100644
--- a/www/manager6/dc/StorageView.js
+++ b/www/manager6/dc/StorageView.js
@@ -128,7 +128,9 @@ Ext.define(
sortable: true,
dataIndex: 'path',
renderer: function (value, metaData, record) {
- return Ext.htmlEncode(record.data.target || value);
+ return Ext.htmlEncode(
+ record.data.target || record.data.subsysnqn || value,
+ );
},
},
{
@@ -171,6 +173,7 @@ Ext.define(
'server',
'portal',
'target',
+ 'subsysnqn',
'export',
'storage',
{ name: 'shared', type: 'boolean' },
diff --git a/www/manager6/storage/IScsiEdit.js b/www/manager6/storage/IScsiEdit.js
index f11995585..83d31017f 100644
--- a/www/manager6/storage/IScsiEdit.js
+++ b/www/manager6/storage/IScsiEdit.js
@@ -5,6 +5,7 @@ Ext.define('PVE.storage.IScsiScan', {
queryParam: 'portal',
valueField: 'target',
displayField: 'target',
+ scanFields: ['target', 'portal'],
matchFieldWidth: false,
allowBlank: false,
@@ -53,13 +54,13 @@ Ext.define('PVE.storage.IScsiScan', {
}
let store = Ext.create('Ext.data.Store', {
- fields: ['target', 'portal'],
+ fields: me.scanFields,
proxy: {
type: 'proxmox',
url: `${me.apiBaseUrl}${me.nodename}${me.apiSuffix}`,
},
});
- store.sort('target', 'ASC');
+ store.sort(me.valueField, 'ASC');
Ext.apply(me, {
store: store,
diff --git a/www/manager6/storage/NVMeOFEdit.js b/www/manager6/storage/NVMeOFEdit.js
new file mode 100644
index 000000000..92057dc4f
--- /dev/null
+++ b/www/manager6/storage/NVMeOFEdit.js
@@ -0,0 +1,127 @@
+Ext.define('PVE.storage.NVMeOFScan', {
+ extend: 'PVE.storage.IScsiScan',
+ alias: 'widget.pveNVMeOFScan',
+
+ valueField: 'subsysnqn',
+ displayField: 'subsysnqn',
+ scanFields: ['subsysnqn', 'portals'],
+ showNodeSelector: false,
+
+ listConfig: {
+ width: 650,
+ columns: [
+ {
+ dataIndex: 'subsysnqn',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+ emptyText: PVE.Utils.renderNotFound(gettext('Subsystem')),
+ },
+
+ config: {
+ apiSuffix: '/scan/nvmeof',
+ },
+});
+
+Ext.define('PVE.storage.NVMeOFInputPanel', {
+ extend: 'PVE.panel.StorageBase',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ onlineHelp: 'storage_nvmeof',
+
+ onGetValues: function (values) {
+ let me = this;
+
+ values.content = values.namespaces ? 'images' : 'none';
+ delete values.namespaces;
+
+ return me.callParent([values]);
+ },
+
+ setValues: function (values) {
+ values.namespaces = values.content.indexOf('images') !== -1;
+ this.callParent([values]);
+ },
+
+ column1: [
+ {
+ xtype: 'textfield',
+ name: 'discovery-portal',
+ fieldLabel: gettext('Discovery Portal'),
+ allowBlank: false,
+ submitValue: false,
+ cbind: {
+ hidden: '{!isCreate}',
+ disabled: '{!isCreate}',
+ },
+ listeners: {
+ change: {
+ fn: function (field, value) {
+ let scan = this.up('inputpanel').lookup('subsystemScan');
+ scan.setDisabled(!value);
+ scan.setPortal(value);
+ scan.setValue('');
+ },
+ buffer: 500,
+ },
+ },
+ },
+ {
+ xtype: 'checkbox',
+ name: 'namespaces',
+ boxLabel: gettext('Use namespaces directly'),
+ },
+ ],
+
+ // NQNs and portal lists are longer than a column is wide
+ columnB: [
+ {
+ cbind: {
+ xtype: (get) => (get('isCreate') ? 'pveNVMeOFScan' : 'displayfield'),
+ disabled: '{isCreate}',
+ },
+ name: 'subsysnqn',
+ value: '',
+ fieldLabel: gettext('Subsystem'),
+ allowBlank: false,
+ reference: 'subsystemScan',
+ listeners: {
+ change: function (field, value) {
+ let rec = field
+ .getStore?.()
+ .findRecord('subsysnqn', value, 0, false, true, true);
+ if (rec) {
+ field
+ .up('inputpanel')
+ .down('field[name=nvme-portals]')
+ .setValue(rec.get('portals'));
+ }
+ },
+ },
+ },
+ {
+ xtype: 'textfield',
+ name: 'nvme-portals',
+ fieldLabel: gettext('Portals'),
+ allowBlank: false,
+ },
+ ],
+
+ advancedColumn1: [
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 'nvme-fast-io-fail-tmo',
+ fieldLabel: gettext('Fail I/O After (s)'),
+ minValue: -1,
+ emptyText: '120',
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext('Applies to newly connected paths. -1 disables the timeout.'),
+ },
+ },
+ ],
+});
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (11 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
` (4 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
A namespace of an NVMe over Fabrics storage can hold the volume group
of an LVM storage like an iSCSI LUN, so list such storages in the base
storage selector as well and name the type of each.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/storage/LVMEdit.js | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/www/manager6/storage/LVMEdit.js b/www/manager6/storage/LVMEdit.js
index 9c4df2609..7ac366f35 100644
--- a/www/manager6/storage/LVMEdit.js
+++ b/www/manager6/storage/LVMEdit.js
@@ -70,10 +70,13 @@ Ext.define('PVE.storage.BaseStorageSelector', {
let store = Ext.create('Ext.data.Store', {
autoLoad: {
addRecords: true,
- params: {
- type: 'iscsi',
- },
},
+ filters: [
+ {
+ filterFn: (rec) =>
+ !rec.get('storage') || ['iscsi', 'nvmeof'].includes(rec.get('type')),
+ },
+ ],
fields: [
'storage',
'type',
@@ -82,7 +85,8 @@ Ext.define('PVE.storage.BaseStorageSelector', {
name: 'text',
convert: function (value, record) {
if (record.data.storage) {
- return record.data.storage + ' (iSCSI)';
+ let type = PVE.Utils.format_storage_type(record.data.type);
+ return `${record.data.storage} (${type})`;
} else {
return me.existingGroupsText;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (12 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
` (3 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Add an NVMe over Fabrics entry to the guided setup of the SAN view,
with the same steps as for iSCSI: connect to the subsystem found at a
discovery portal, with the host NQNs of all nodes at hand for the
access rules of the array, select a namespace, name the volume group
and storage, and confirm.
The target step is shared with iSCSI, as only the storage type, its
scan and the node identities differ. The add dialog of NVMe over
Fabrics storages names the SAN view, as those of iSCSI and LVM do.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 2 +-
www/manager6/SANUtils.js | 2 +
www/manager6/dc/SAN.js | 2 +-
www/manager6/storage/NVMeOFEdit.js | 3 +
www/manager6/storage/wizard/LunSelect.js | 29 +++-
www/manager6/storage/wizard/NodeIdentities.js | 45 +++---
.../storage/wizard/{IScsi.js => Target.js} | 129 ++++++++++++------
www/manager6/storage/wizard/Wizard.js | 43 +++---
www/manager6/window/SANLunState.js | 23 +++-
9 files changed, 183 insertions(+), 95 deletions(-)
rename www/manager6/storage/wizard/{IScsi.js => Target.js} (64%)
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index fbd866b4b..ab4b85d0c 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -389,7 +389,7 @@ JSSRC= \
storage/wizard/Wizard.js \
storage/wizard/LunSelect.js \
storage/wizard/NodeIdentities.js \
- storage/wizard/IScsi.js \
+ storage/wizard/Target.js \
storage/wizard/FibreChannel.js \
Workspace.js \
# end of JSSRC list
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index c08ac2a1a..d61c74df4 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -23,6 +23,8 @@ Ext.define('pve-san-lun', {
'multipath-storage',
'iscsi-target',
'iscsi-lun',
+ 'nvme-subsystem',
+ 'nvme-nsid',
],
});
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index 6b32926c7..21f79272a 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -120,7 +120,7 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Add Storage'),
disabled: !allCaps.storage['Datastore.Allocate'],
- menu: [setupItem('iscsi'), setupItem('fc', !canModify)],
+ menu: [setupItem('iscsi'), setupItem('fc', !canModify), setupItem('nvmeof')],
},
];
me.items = [
diff --git a/www/manager6/storage/NVMeOFEdit.js b/www/manager6/storage/NVMeOFEdit.js
index 92057dc4f..b23b828bc 100644
--- a/www/manager6/storage/NVMeOFEdit.js
+++ b/www/manager6/storage/NVMeOFEdit.js
@@ -30,6 +30,9 @@ Ext.define('PVE.storage.NVMeOFInputPanel', {
onlineHelp: 'storage_nvmeof',
+ // TRANSLATORS: "Add Storage" is a button of the SAN view, {0} the menu path of that view.
+ sanNote: gettext('To set up shared LVM on a namespace, use Add Storage in {0}.'),
+
onGetValues: function (values) {
let me = this;
diff --git a/www/manager6/storage/wizard/LunSelect.js b/www/manager6/storage/wizard/LunSelect.js
index a34b95d29..18572015c 100644
--- a/www/manager6/storage/wizard/LunSelect.js
+++ b/www/manager6/storage/wizard/LunSelect.js
@@ -27,7 +27,11 @@ Ext.define('PVE.storage.wizard.LunSelect', {
return;
}
me.store.setFilters([
- { property: 'iscsi-target', value: wizard.connection.target, exactMatch: true },
+ {
+ property: wizard.wizardType === 'nvmeof' ? 'nvme-subsystem' : 'iscsi-target',
+ value: wizard.connection.target,
+ exactMatch: true,
+ },
]);
Proxmox.Utils.API2Request({
url: `/nodes/${Proxmox.NodeName}/storage/${encodeURIComponent(wizard.connection.storage)}/content`,
@@ -67,9 +71,13 @@ Ext.define('PVE.storage.wizard.LunSelect', {
xtype: 'component',
cls: 'pmx-hint',
margin: '0 0 10 0',
- html: gettext(
- 'A LUN used by a storage cannot be selected. A LUN with other data has to be wiped under Node > SAN first.',
- ),
+ html: me.nvme
+ ? gettext(
+ 'A namespace used by a storage cannot be selected. A namespace with other data has to be wiped under Node > SAN first.',
+ )
+ : gettext(
+ 'A LUN used by a storage cannot be selected. A LUN with other data has to be wiped under Node > SAN first.',
+ ),
},
...(me.rescanNote
? [
@@ -84,7 +92,9 @@ Ext.define('PVE.storage.wizard.LunSelect', {
xtype: 'grid',
flex: 1,
store: me.store,
- emptyText: gettext('No LUNs found. Check connections and array access.'),
+ emptyText: me.nvme
+ ? gettext('No namespaces found. Check connections and array access.')
+ : gettext('No LUNs found. Check connections and array access.'),
viewConfig: {
enableTextSelection: true,
getRowClass: (rec) => (rec.get('used-by-storage') ? 'x-item-disabled' : ''),
@@ -102,6 +112,7 @@ Ext.define('PVE.storage.wizard.LunSelect', {
renderer: (value, md, rec) => Ext.htmlEncode(value || rec.get('wwid')),
},
{ text: 'LUN', dataIndex: 'iscsi-lun', width: 60 },
+ { text: 'NSID', dataIndex: 'nvme-nsid', width: 60 },
{
text: gettext('Model'),
dataIndex: 'model',
@@ -122,6 +133,8 @@ Ext.define('PVE.storage.wizard.LunSelect', {
text: gettext('Multipath'),
dataIndex: 'multipath',
width: 170,
+ // native NVMe multipath needs no setup per namespace
+ hidden: me.nvme,
renderer: PVE.SANUtils.renderSanLunMultipath,
},
{
@@ -162,8 +175,10 @@ Ext.define('PVE.storage.wizard.LunSelect', {
Proxmox.Utils.monStoreErrors(me.down('grid'), me.rstore);
me.mon(me.rstore, 'load', updateSelection);
me.on('activate', function () {
- if (me.up('window').wizardType === 'fc') {
- me.down('gridcolumn[dataIndex=iscsi-lun]').hide();
+ let type = me.up('window').wizardType;
+ me.down('gridcolumn[dataIndex=iscsi-lun]').setHidden(type !== 'iscsi');
+ me.down('gridcolumn[dataIndex=nvme-nsid]').setHidden(type !== 'nvmeof');
+ if (type === 'fc') {
me.store.setFilters([
{
filterFn: (rec) => ['fc', 'fcoe', 'nvme-fc'].includes(rec.get('transport')),
diff --git a/www/manager6/storage/wizard/NodeIdentities.js b/www/manager6/storage/wizard/NodeIdentities.js
index 9c7ab8cce..69c7fbca2 100644
--- a/www/manager6/storage/wizard/NodeIdentities.js
+++ b/www/manager6/storage/wizard/NodeIdentities.js
@@ -1,7 +1,7 @@
Ext.define('pve-san-node-identity', {
extend: 'Ext.data.Model',
idProperty: 'node',
- fields: ['node', 'state', 'iscsi-initiator-name'],
+ fields: ['node', 'state', 'iscsi-initiator-name', 'nvme-host-nqn'],
});
Ext.define('PVE.storage.wizard.NodeIdentities', {
@@ -10,24 +10,29 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
height: 220,
disableSelection: true,
viewConfig: { enableTextSelection: true },
- columns: [
- { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
- {
- text: gettext('State'),
- dataIndex: 'state',
- width: 175,
- renderer: PVE.SANUtils.renderState,
- },
- {
- text: gettext('Initiator Name'),
- dataIndex: 'iscsi-initiator-name',
- flex: 1,
- renderer: Ext.htmlEncode,
- },
- ],
+
+ identity: 'iscsi-initiator-name',
initComponent: function () {
let me = this;
+ let nvme = me.identity === 'nvme-host-nqn';
+ me.columns = [
+ { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
+ {
+ text: gettext('State'),
+ dataIndex: 'state',
+ width: 175,
+ renderer: PVE.SANUtils.renderState,
+ // the state of multipathd, which NVMe does not use
+ hidden: nvme,
+ },
+ {
+ text: nvme ? gettext('Host NQN') : gettext('Initiator Name'),
+ dataIndex: me.identity,
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ];
me.rstore = Ext.create('Proxmox.data.UpdateStore', {
storeid: 'san-wizard-identities',
model: 'pve-san-node-identity',
@@ -56,17 +61,19 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
me.tbar = [
{
xtype: 'tbtext',
- text: gettext('Allow these initiator names on the target.'),
+ text: nvme
+ ? gettext('Allow these host NQNs on the target.')
+ : gettext('Allow these initiator names on the target.'),
},
'->',
{
- text: gettext('Copy Initiator Names'),
+ text: nvme ? gettext('Copy Host NQNs') : gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
handler: () =>
PVE.Utils.copyToClipboard(
me.store
.getRange()
- .map((rec) => rec.get('iscsi-initiator-name'))
+ .map((rec) => rec.get(me.identity))
.filter(Boolean)
.join('\n'),
),
diff --git a/www/manager6/storage/wizard/IScsi.js b/www/manager6/storage/wizard/Target.js
similarity index 64%
rename from www/manager6/storage/wizard/IScsi.js
rename to www/manager6/storage/wizard/Target.js
index 515d1206f..e505ddca0 100644
--- a/www/manager6/storage/wizard/IScsi.js
+++ b/www/manager6/storage/wizard/Target.js
@@ -1,13 +1,14 @@
-Ext.define('PVE.storage.wizard.IScsiTarget', {
+Ext.define('PVE.storage.wizard.Target', {
extend: 'Proxmox.panel.InputPanel',
- xtype: 'pveStorageWizardIScsiTarget',
- onlineHelp: 'storage_open_iscsi',
+ xtype: 'pveStorageWizardTarget',
+
+ storageType: 'iscsi',
updateConnectionField: function () {
let me = this;
let target = me.down('field[name=target]').getValue();
let storage = me.storages?.find(
- (entry) => entry.type === 'iscsi' && entry.target === target,
+ (entry) => entry.type === me.storageType && entry[me.targetKey] === target,
);
let field = me.down('field[name=connection]');
if (storage) {
@@ -20,16 +21,16 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
onSubmit: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
let wizard = me.up('window');
let values = me.getValues();
- let params = {
- type: 'iscsi',
- storage: values.connection,
- portal: values.portal,
- target: values.target,
- content: 'none',
- multipath: 1,
- };
+ let params = { type: me.storageType, storage: values.connection, content: 'none' };
+ if (nvme) {
+ params.subsysnqn = values.target;
+ params['nvme-portals'] = values['nvme-portals'];
+ } else {
+ Object.assign(params, { portal: values.portal, target: values.target, multipath: 1 });
+ }
let nodes = []
.concat(values.nodes || [])
.filter(Boolean)
@@ -59,18 +60,22 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
}
me.storages = response.result.data;
me.updateConnectionField();
- let users = me.storages.filter((entry) => entry.target === values.target);
+ let users = me.storages.filter((entry) => entry[me.targetKey] === values.target);
if (users.length) {
let base = users[0];
if (
users.length !== 1 ||
- base.type !== 'iscsi' ||
+ base.type !== me.storageType ||
`${base.content ?? ''}` !== 'none'
) {
wizard.showError(
- gettext(
- 'This target is already used by another storage. Check Datacenter > Storage.',
- ),
+ nvme
+ ? gettext(
+ 'This subsystem is already used by another storage. Check Datacenter > Storage.',
+ )
+ : gettext(
+ 'This target is already used by another storage. Check Datacenter > Storage.',
+ ),
);
return;
}
@@ -78,9 +83,13 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
wizard.showError(
Ext.htmlEncode(
Ext.String.format(
- gettext(
- 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" connects to this subsystem, but it is disabled. Enable that storage first.',
+ )
+ : gettext(
+ 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
+ ),
base.storage,
),
),
@@ -116,10 +125,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
}
let connection = {
storage: base.storage,
- target: base.target,
+ target: values.target,
nodes: selectedNodes.join(','),
};
- if (!base.multipath) {
+ if (!nvme && !base.multipath) {
wizard.showError(
Ext.htmlEncode(
Ext.String.format(
@@ -142,7 +151,7 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
waitMsgTarget: wizard,
failure: fail,
success: () =>
- connected({ storage: params.storage, target: params.target, nodes }),
+ connected({ storage: params.storage, target: values.target, nodes }),
});
},
});
@@ -150,6 +159,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
initComponent: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
+ me.targetKey = nvme ? 'subsysnqn' : 'target';
+ me.onlineHelp = nvme ? 'storage_nvmeof' : 'storage_open_iscsi';
+ let targetLabel = nvme ? gettext('Subsystem') : gettext('Target');
let common = PVE.panel.StorageBase.commonFields(true);
// wide enough for the longest label of the step
let labelWidth = 150;
@@ -157,7 +170,7 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
{
xtype: 'textfield',
name: 'portal',
- fieldLabel: 'Portal',
+ fieldLabel: nvme ? gettext('Discovery Portal') : 'Portal',
labelWidth,
allowBlank: false,
listeners: {
@@ -173,33 +186,54 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
},
},
{
- xtype: 'pveIScsiScan',
+ xtype: nvme ? 'pveNVMeOFScan' : 'pveIScsiScan',
name: 'target',
- fieldLabel: gettext('Target'),
+ fieldLabel: targetLabel,
labelWidth,
disabled: true,
showNodeSelector: false,
nodename: Proxmox.NodeName,
- listeners: { change: () => me.updateConnectionField() },
+ listeners: {
+ change: function (field, value) {
+ me.updateConnectionField();
+ let rec = nvme
+ ? field.getStore().findRecord('subsysnqn', value, 0, false, true, true)
+ : undefined;
+ if (rec) {
+ me.down('field[name=nvme-portals]').setValue(rec.get('portals'));
+ }
+ },
+ },
listConfig: {
width: 650,
- emptyText: PVE.Utils.renderNotFound(gettext('iSCSI Target')),
+ emptyText: PVE.Utils.renderNotFound(targetLabel),
columns: [
{
- text: gettext('Target'),
- dataIndex: 'target',
+ text: targetLabel,
+ dataIndex: me.targetKey,
flex: 1,
renderer: Ext.htmlEncode,
},
{
text: gettext('Portals'),
- dataIndex: 'portal',
+ dataIndex: nvme ? 'portals' : 'portal',
width: 220,
renderer: Ext.htmlEncode,
},
],
},
},
+ ...(nvme
+ ? [
+ {
+ xtype: 'textfield',
+ name: 'nvme-portals',
+ fieldLabel: gettext('Portals'),
+ labelWidth,
+ allowBlank: false,
+ },
+ ]
+ : []),
{
...common.id,
name: 'connection',
@@ -210,12 +244,19 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
xtype: 'component',
margin: '0 0 10 0',
html: `<span class="pmx-hint">${Ext.htmlEncode(gettext('Note:'))}</span> ${Ext.htmlEncode(
- gettext(
- 'Next creates or reuses the connection storage before you select a LUN.',
- ),
+ nvme
+ ? gettext(
+ 'Next creates or reuses the connection storage before you select a namespace.',
+ )
+ : gettext(
+ 'Next creates or reuses the connection storage before you select a LUN.',
+ ),
)}`,
},
- { xtype: 'pveStorageWizardNodeIdentities' },
+ {
+ xtype: 'pveStorageWizardNodeIdentities',
+ identity: nvme ? 'nvme-host-nqn' : 'iscsi-initiator-name',
+ },
];
me.advancedItems = [
{ ...common.nodes, labelWidth, validator: PVE.storage.wizard.validateNodes },
@@ -242,8 +283,10 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
},
});
-PVE.storage.wizard.createIScsiLVM = function (wizard, values, lun) {
+PVE.storage.wizard.createLVMOnBase = function (wizard, values, lun) {
let connection = wizard.connection;
+ let suffix =
+ wizard.wizardType === 'nvmeof' ? `:nvme-${lun.wwid}` : `.dm-uuid-mpath-${lun.wwid}`;
let fail = (response) => wizard.showError(response.htmlStatus);
Proxmox.Utils.API2Request({
url: `/nodes/${Proxmox.NodeName}/storage/${encodeURIComponent(connection.storage)}/content`,
@@ -255,14 +298,16 @@ PVE.storage.wizard.createIScsiLVM = function (wizard, values, lun) {
if (wizard.destroyed) {
return;
}
- let volume = response.result.data.find((entry) =>
- entry.volid.endsWith(`.dm-uuid-mpath-${lun.wwid}`),
- );
+ let volume = response.result.data.find((entry) => entry.volid.endsWith(suffix));
if (!volume) {
wizard.showError(
- gettext(
- 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
- ),
+ wizard.wizardType === 'nvmeof'
+ ? gettext(
+ 'The selected namespace is not available yet. Check Node > SAN, then retry.',
+ )
+ : gettext(
+ 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
+ ),
);
return;
}
diff --git a/www/manager6/storage/wizard/Wizard.js b/www/manager6/storage/wizard/Wizard.js
index 9d4e6a256..2739cdeb3 100644
--- a/www/manager6/storage/wizard/Wizard.js
+++ b/www/manager6/storage/wizard/Wizard.js
@@ -92,6 +92,7 @@ Ext.define('PVE.storage.wizard.SAN', {
wwid,
storage: values.storage,
fc: me.wizardType === 'fc',
+ nvme: me.wizardType === 'nvmeof',
autoShow: true,
});
},
@@ -100,6 +101,9 @@ Ext.define('PVE.storage.wizard.SAN', {
multipathSummary: function (lun) {
let me = this;
+ if (me.wizardType === 'nvmeof') {
+ return PVE.SANUtils.multipathText('native');
+ }
// an explicit entry configures the map, also of a LUN of the connection storage
if (lun.multipath === 'manual') {
return PVE.SANUtils.multipathText('manual');
@@ -145,24 +149,25 @@ Ext.define('PVE.storage.wizard.SAN', {
if (me.wizardType === 'fc') {
PVE.storage.wizard.createFC(me, values, lun);
} else {
- PVE.storage.wizard.createIScsiLVM(me, values, lun);
+ PVE.storage.wizard.createLVMOnBase(me, values, lun);
}
},
initComponent: function () {
let me = this;
- if (!['iscsi', 'fc'].includes(me.wizardType)) {
+ if (!['iscsi', 'fc', 'nvmeof'].includes(me.wizardType)) {
throw 'no SAN storage type specified';
}
- let iscsi = me.wizardType === 'iscsi';
+ let fc = me.wizardType === 'fc';
me.subject = PVE.storage.wizard.subject(me.wizardType);
me.title = Proxmox.Utils.dialog_title(me.subject, true, true);
+ let onlineHelp = me.wizardType === 'nvmeof' ? 'storage_nvmeof' : 'storage_multipath';
let common = PVE.panel.StorageBase.commonFields(true);
let snapshots = PVE.panel.StorageBase.volumeChainFields(true, 'lvm');
let settings = {
xtype: 'inputpanel',
title: gettext('Volume Group'),
- onlineHelp: 'storage_multipath',
+ onlineHelp,
items: [
{ ...common.id, fieldLabel: gettext('Storage ID') },
{
@@ -176,35 +181,37 @@ Ext.define('PVE.storage.wizard.SAN', {
snapshots.hint,
],
};
- if (!iscsi) {
+ if (fc) {
settings.advancedItems = [
{ ...common.nodes, validator: PVE.storage.wizard.validateNodes },
];
}
me.items = [
- ...(iscsi
- ? [
+ ...(fc
+ ? []
+ : [
{
- xtype: 'pveStorageWizardIScsiTarget',
+ xtype: 'pveStorageWizardTarget',
title: gettext('Target'),
itemId: 'target',
+ storageType: me.wizardType,
},
- ]
- : []),
+ ]),
{
xtype: 'pveStorageWizardLunSelect',
- title: 'LUN',
+ title: me.wizardType === 'nvmeof' ? gettext('Namespace') : 'LUN',
itemId: 'luns',
- onlineHelp: 'storage_multipath',
+ nvme: me.wizardType === 'nvmeof',
+ onlineHelp,
// the nodes only see a newly mapped Fibre Channel LUN after a rescan
- rescanNote: iscsi
- ? undefined
- : Ext.String.format(
+ rescanNote: fc
+ ? Ext.String.format(
gettext(
'After mapping a new LUN on the array, rescan the Fibre Channel hosts of each node (<a href="{0}" target="_blank" rel="noopener noreferrer">documentation</a>).',
),
Ext.htmlEncode(Proxmox.Utils.get_help_link('storage_multipath')),
- ),
+ )
+ : undefined,
},
settings,
{
@@ -239,14 +246,14 @@ Ext.define('PVE.storage.wizard.SAN', {
me.callParent();
let tabs = me.down('#wizcontent');
tabs.on('tabchange', function (tp, tab) {
- if (iscsi && tab === me.down('#luns')) {
+ if (!fc && tab === me.down('#luns')) {
me.down('#back').disable();
}
me.down('#submit').setText(
tab === me.down('#target') ? gettext('Next') : gettext('Finish'),
);
});
- if (iscsi) {
+ if (!fc) {
me.down('#submit').setText(gettext('Next'));
}
me.on('destroy', () => me.reloadCallback?.());
diff --git a/www/manager6/window/SANLunState.js b/www/manager6/window/SANLunState.js
index a0a6d514d..1de84ea2b 100644
--- a/www/manager6/window/SANLunState.js
+++ b/www/manager6/window/SANLunState.js
@@ -19,7 +19,10 @@ Ext.define('PVE.window.SANLunState', {
if (!me.wwid || !me.storage) {
throw 'no LUN or storage specified';
}
- me.title = Ext.String.format(gettext('Storage {0}: LUN State'), me.storage);
+ me.title = Ext.String.format(
+ me.nvme ? gettext('Storage {0}: Namespace State') : gettext('Storage {0}: LUN State'),
+ me.storage,
+ );
// The other nodes set up a new LUN with their next status updates.
let pendingUntil = Date.now() + 60000;
let rstore = Ext.create('Proxmox.data.UpdateStore', {
@@ -51,13 +54,17 @@ Ext.define('PVE.window.SANLunState', {
margin: '0 0 10 0',
html: Ext.htmlEncode(
Ext.String.format(
- me.fc
+ me.nvme
? gettext(
- 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array and rescan its Fibre Channel hosts.',
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the namespace on the array.',
)
- : gettext(
- 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array.',
- ),
+ : me.fc
+ ? gettext(
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array and rescan its Fibre Channel hosts.',
+ )
+ : gettext(
+ 'Storage "{0}" was added. Guests on this storage cannot run on a node that stays on "Missing": check its access to the LUN on the array.',
+ ),
me.storage,
),
),
@@ -67,7 +74,9 @@ Ext.define('PVE.window.SANLunState', {
flex: 1,
store,
disableSelection: true,
- emptyText: gettext('No node reported this LUN yet.'),
+ emptyText: me.nvme
+ ? gettext('No node reported this namespace yet.')
+ : gettext('No node reported this LUN yet.'),
columns: [
{
text: gettext('Node'),
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (13 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
` (2 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
Add a view below Node > SAN with the controllers of a node, one per
connected portal, their state and the storage they belong to, and the
host NQN of the node. A controller can be disconnected from there, for
example after its portal was removed from the storage. The node
refuses that for the last usable path of a namespace in use.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/SANUtils.js | 5 ++
www/manager6/Utils.js | 1 +
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 115 +++++++++++++++++++++++++
5 files changed, 130 insertions(+), 1 deletion(-)
create mode 100644 www/manager6/node/NVMeOFControllers.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index ab4b85d0c..83fa225b7 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -252,6 +252,7 @@ JSSRC= \
node/ISCSITargets.js \
node/LVM.js \
node/LVMThin.js \
+ node/NVMeOFControllers.js \
node/SANLuns.js \
node/StatusView.js \
node/Subscription.js \
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index d61c74df4..8c3ddb8a2 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -78,6 +78,11 @@ Ext.define('PVE.SANUtils', {
'logged-in': ['check-circle good', gettext('Connected')],
recovering: ['exclamation-circle warning', gettext('Reconnecting')],
'not-connected': ['times-circle faded', gettext('Not connected')],
+ live: ['check-circle good', gettext('Connected')],
+ connecting: ['exclamation-circle warning', gettext('Reconnecting')],
+ resetting: ['exclamation-circle warning', gettext('Reconnecting')],
+ deleting: ['times-circle faded', gettext('Disconnecting')],
+ 'deleting (no IO)': ['times-circle faded', gettext('Disconnecting')],
}[value];
let [icon, label] = display ?? ['question-circle faded', value];
return `<i class="fa fa-fw fa-${icon}"></i> ${Ext.htmlEncode(label)}`;
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index 4a97358ac..55d5806d5 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -2251,6 +2251,7 @@ Ext.define('PVE.Utils', {
lvmthinremove: ['Thinpool', gettext('Remove')],
migrateall: ['', gettext('Bulk migrate VMs and Containers')],
move_volume: ['CT', gettext('Move Volume')],
+ nvmedisconnect: ['NVMe-oF', gettext('Disconnect')],
'pbs-download': ['VM/CT', gettext('File Restore Download')],
pull_file: ['CT', gettext('Pull file')],
push_file: ['CT', gettext('Push file')],
diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
index d4a292d68..f3db4acba 100644
--- a/www/manager6/node/Config.js
+++ b/www/manager6/node/Config.js
@@ -360,7 +360,7 @@ Ext.define('PVE.node.Config', {
);
}
- // the san-luns and iscsi reads accept either privilege
+ // the san-luns, iscsi and nvmeof reads accept either privilege
if (caps.nodes['Sys.Audit'] || caps.storage['Datastore.Allocate']) {
me.items.push(
{
@@ -376,6 +376,13 @@ Ext.define('PVE.node.Config', {
iconCls: 'fa fa-bullseye',
groups: ['san'],
},
+ {
+ xtype: 'pveNVMeOFControllerList',
+ title: 'NVMe over Fabrics',
+ itemId: 'san-nvmeof',
+ iconCls: 'fa fa-plug',
+ groups: ['san'],
+ },
);
}
diff --git a/www/manager6/node/NVMeOFControllers.js b/www/manager6/node/NVMeOFControllers.js
new file mode 100644
index 000000000..93955442d
--- /dev/null
+++ b/www/manager6/node/NVMeOFControllers.js
@@ -0,0 +1,115 @@
+Ext.define('PVE.node.NVMeOFControllerList', {
+ extend: 'Ext.grid.Panel',
+ xtype: 'pveNVMeOFControllerList',
+ onlineHelp: 'storage_nvmeof',
+ emptyText: gettext('No NVMe over Fabrics controllers found'),
+ viewConfig: { enableTextSelection: true },
+ store: {
+ fields: ['controller', 'subsysnqn', 'portal', 'state', 'namespaces', 'used-by-storage'],
+ sorters: ['subsysnqn', 'portal'],
+ },
+ columns: [
+ { text: gettext('Controller'), dataIndex: 'controller', width: 100 },
+ {
+ text: gettext('Subsystem'),
+ dataIndex: 'subsysnqn',
+ flex: 5,
+ renderer: Ext.htmlEncode,
+ },
+ { text: gettext('Portal'), dataIndex: 'portal', width: 180, renderer: Ext.htmlEncode },
+ {
+ text: gettext('State'),
+ dataIndex: 'state',
+ width: 150,
+ renderer: PVE.SANUtils.renderState,
+ },
+ { text: gettext('Namespaces'), dataIndex: 'namespaces', width: 110 },
+ {
+ text: gettext('Storage'),
+ dataIndex: 'used-by-storage',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+
+ reload: function () {
+ let me = this;
+ let load = (path, success) =>
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/nvmeof/${path}`,
+ method: 'GET',
+ waitMsgTarget: me.getView(),
+ failure: (response) =>
+ Proxmox.Utils.setErrorMask(me.getView(), response.htmlStatus),
+ success: (response) => !me.destroyed && success(response.result.data),
+ });
+ load('status', function (status) {
+ me.down('#hostnqn').setValue(status['host-nqn'] || '');
+ let error = status.supported
+ ? status.error
+ : gettext('nvme-cli is not installed on this node.');
+ if (error) {
+ Proxmox.Utils.setErrorMask(me.getView(), Ext.htmlEncode(error));
+ return;
+ }
+ load('controller', function (controllers) {
+ Proxmox.Utils.setErrorMask(me.getView(), false);
+ me.getStore().setData(controllers);
+ });
+ });
+ },
+
+ listeners: {
+ activate: function () {
+ this.reload();
+ },
+ },
+
+ initComponent: function () {
+ let me = this;
+ me.nodename = me.pveSelNode.data.node;
+ me.tbar = [
+ { text: gettext('Reload'), iconCls: 'fa fa-refresh', handler: () => me.reload() },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Disconnect'),
+ iconCls: 'fa fa-unlink',
+ disabled: true,
+ dangerous: true,
+ enableFn: () => !!Ext.state.Manager.get('GuiCap').dc['Sys.Modify'],
+ confirmMsg: (rec) =>
+ Ext.String.format(
+ gettext(
+ 'Disconnect controller {0}? A storage that still lists its portal connects to it again.',
+ ),
+ Ext.htmlEncode(rec.get('controller')),
+ ),
+ handler: (button, event, rec) =>
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/nvmeof/controller/${rec.get('controller')}`,
+ method: 'DELETE',
+ waitMsgTarget: me,
+ failure: (response) => Ext.Msg.alert(gettext('Error'), response.htmlStatus),
+ success: (response) =>
+ Ext.create('Proxmox.window.TaskProgress', {
+ upid: response.result.data,
+ autoShow: true,
+ taskDone: () => !me.destroyed && me.reload(),
+ }),
+ }),
+ },
+ '->',
+ {
+ xtype: 'displayfield',
+ itemId: 'hostnqn',
+ fieldLabel: gettext('Host NQN'),
+ },
+ {
+ iconCls: 'fa fa-clipboard',
+ tooltip: gettext('Copy to Clipboard'),
+ handler: () => PVE.Utils.copyToClipboard(me.down('#hostnqn').getValue()),
+ },
+ ];
+ me.callParent();
+ },
+});
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (14 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The access rules of an array need the host NQN of every node, like the
initiator names for iSCSI. Show them next to those in the node list of
the SAN view, with a button to copy them all.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 29 +++++++++++++++++++++--------
1 file changed, 21 insertions(+), 8 deletions(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index 21f79272a..98aad36ab 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -9,6 +9,7 @@ Ext.define('pve-san-status', {
'apply-error',
'time',
'iscsi-initiator-name',
+ 'nvme-host-nqn',
'wwid',
'name',
'explicit',
@@ -98,6 +99,14 @@ Ext.define('PVE.dc.SANView', {
sortAfterUpdate: true,
});
let reload = () => me.rstore.load();
+ let copy = (field) => () =>
+ PVE.Utils.copyToClipboard(
+ nodeStore
+ .getRange()
+ .map((rec) => rec.get(field))
+ .filter(Boolean)
+ .join('\n'),
+ );
let edit = (rec) =>
Ext.create('PVE.dc.MultipathWwidEdit', {
wwid: rec?.get('wwid'),
@@ -134,14 +143,12 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
- handler: () =>
- PVE.Utils.copyToClipboard(
- nodeStore
- .getRange()
- .map((rec) => rec.get('iscsi-initiator-name'))
- .filter(Boolean)
- .join('\n'),
- ),
+ handler: copy('iscsi-initiator-name'),
+ },
+ {
+ text: gettext('Copy Host NQNs'),
+ iconCls: 'fa fa-clipboard',
+ handler: copy('nvme-host-nqn'),
},
],
columns: [
@@ -173,6 +180,12 @@ Ext.define('PVE.dc.SANView', {
flex: 2,
renderer: Ext.htmlEncode,
},
+ {
+ text: gettext('NVMe Host NQN'),
+ dataIndex: 'nvme-host-nqn',
+ flex: 4,
+ renderer: Ext.htmlEncode,
+ },
{
text: gettext('Last Report'),
dataIndex: 'time',
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (15 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-13 3:04 ` Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13 3:04 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the namespaces of NVMe over Fabrics
storages as well. They use the native multipath of the kernel, with
neither an entry nor a storage that requests a map, so the Multipath
column would call them unmanaged. Show "Native" for them, as the LUN
list of a node does.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index 98aad36ab..083bc8c21 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -14,6 +14,7 @@ Ext.define('pve-san-status', {
'name',
'explicit',
'multipath-storage',
+ 'native',
'nodes',
'size',
'used-by-storage',
@@ -312,7 +313,14 @@ Ext.define('PVE.dc.SANView', {
width: 170,
renderer: (value, md, rec) => {
let storage = rec.get('multipath-storage');
- let kind = value ? 'manual' : storage ? 'storage' : 'unmanaged';
+ // a namespace uses the native multipath of the kernel
+ let kind = rec.get('native')
+ ? 'native'
+ : value
+ ? 'manual'
+ : storage
+ ? 'storage'
+ : 'unmanaged';
return Ext.htmlEncode(PVE.SANUtils.multipathText(kind, storage));
},
},
^ permalink raw reply related [flat|nested] 19+ messages in thread
* superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
` (16 preceding siblings ...)
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
@ 2026-11-18 20:44 ` Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-18 20:44 UTC (permalink / raw)
To: pve-devel
superseded-by: https://lore.proxmox.com/pve-devel/20261118204302.3100813-1-t.lamprecht@proxmox.com/
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2026-11-18 20:46 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.