* [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage
@ 2026-11-22 1:25 Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
` (17 more replies)
0 siblings, 18 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
A storage type for NVMe subsystems over TCP on top of the SAN series
[0] (libpve-storage-perl 9.1.22, pve-manager 9.2.36, pve-docs 9.2.20):
namespaces as base volumes for shared LVM or directly as disk images,
using the kernel's native multipath.
This is v3. Besides review fixes for v2 [2], it follows the SAN rework
on master: the guided setup lives under Datacenter > Storage > Add and
asks for one ID, the node API moved under /nodes/{node}/san, and
pvestatd's path notifications cover namespaces too.
This is a deliberately small first version: TCP only, without
authentication or TLS, relying on the array's host NQN allow-list and
a separate storage network. The kernel handles path failover and
reconnects, while the plugin only connects missing portals and never
disconnects on portal or storage removal; disconnecting a controller
is an explicit per-node action. Two prerequisites from the docs: cloned
nodes need distinct host NQNs before connecting, and all nodes must be
upgraded before adding the first nvmeof storage (older nodes drop
unknown storage types when writing storage.cfg).
What it adds:
- Storage type 'nvmeof' (isolated properties, one storage per
subsystem) connecting configured TCP portals with ctrl_loss_tmo=-1
and a configurable fast-io-fail-tmo (default 120s), exposing
namespaces as nvme-<wwid> volumes.
- LVM support for nvmeof namespaces as base volumes, reusing the SAN
reference checks and per-LUN locking.
- Discovery via /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof', and
node management via /nodes/{node}/san/nvmeof (host identity,
controller list, and guarded per-controller disconnect refusing the
last usable path of any mounted, held, swap, open, or storage-used
namespace; as in the SAN series, the open-process check misses opens
through another device node).
- Cluster status and path-loss notifications covering nvmeof
namespaces. Status reporting works without multipathd; notifications
still require multipathd to be queryable.
- Web UI: guided 'Shared LVM on NVMe over Fabrics' setup under
Datacenter > Storage > Add (shared with iSCSI), nvmeof storage
dialog with discovery, Node > SAN controller list, and host NQNs in
Datacenter > SAN.
- pvereport host NQN and subsystem output, plus nvme-cli as a
Recommends of libpve-storage-perl.
Changes since v2 [2]:
- Follow the SAN rework: single-ID wizard deriving '<ID>-base', node
selection in the host NQN list, portal suggestions without default
port 4420, early-close cleanup, and /nodes/{node}/san/nvmeof.
- Notify on lost and restored namespace paths in pvestatd.
- Tighten activation and locking: fail only if no controller is live
(reporting per-portal errors) and warn otherwise; cap extra portal
connects at 15s while holding storage.cfg's lock; wait up to 10s for
the node fabrics lock; clear portal cooldowns on storage creation.
- Enforce WWID-based volume names (model/serial by-id links can alias
within a subsystem) and refuse local disk setup on fabric namespace
partitions.
- Harden Disconnect: also refuse the last usable path of swap
namespaces, and pass expected subsystem and portal from the UI to
catch reused controller numbers.
- Skip the NVMe path walk on nodes without nvmeof storages, and add
NVMe-oF info to pvereport.
- Validation and UI fixes: require at least one portal, trim and clear
wizard portals on discovery portal change, ignore empty or stale
scans, skip SMART on NVMe/FC too, leave Add Multipath Entry blank
for namespaces, and parse LVM base volume IDs first (fixing IDs like
'nvme-a').
- Docs: explain upgrading all nodes first, document local-only checks
for unusable controllers, and list nvmeof in the intro and LVM
chapters.
The ZFS over NVMe/TCP plugin on the list [3] brings its own initiator.
Because 'nvmeof' isolates its properties, names do not clash; sharing
connect helpers can follow once both are in.
Packaging: pve-manager needs versioned Depends on libpve-storage-perl
and pve-docs (plus Build-Depends on pve-doc-generator);
libpve-storage-perl should break pve-manager without patch 18.
Testing builds on the v1 [1] and v2 [2] runs on a six-node nested
cluster (dual-portal Linux nvmet and NetApp AFF A20). For v3 on master,
I re-ran both guided setups from Add (new nvmet at 1/1, reused NetApp
at 2/2), cut one NetApp path for a minute and both for 40s under guest
I/O (1/2 and Failed 0/2, storages inactive, I/O requeued cleanly, full
recovery), live-migrated a guest with disks on both storages, and
disconnected controllers via the API on every node. After the final
review fixes, I also verified model/serial by-id refusals, Disconnect
refusals for a mismatched subsystem or in-use LVM namespace, and UI
controller disconnect.
The 15s activation cap, swap and partition refusals, non-NQN subsystem
names, and skipped path walk are covered by unit tests only. Earlier on
v3, storage status reconnected a controller once another process
released the fabrics lock after five seconds.
[0]: https://lore.proxmox.com/pve-devel/20261119194655.3163258-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/
[2]: https://lore.proxmox.com/pve-devel/20261118204302.3100813-1-t.lamprecht@proxmox.com/
[3]: https://lore.proxmox.com/pve-devel/20261005002609.571-1-joaquinvarela@neatech.ar/
pve-storage:
Thomas Lamprecht (9):
diskmanage: skip SMART queries for NVMe fabric namespaces
nvmeof: add a storage for NVMe over Fabrics subsystems
lvm: allow a volume group on an NVMe over Fabrics namespace
api: scan: add NVMe over Fabrics discovery
pvesm: add a scan command for NVMe over Fabrics
api: san luns: attribute NVMe namespaces to storage
api: add node-level NVMe over Fabrics status and controller removal
multipath: report the NVMe host NQN of each node
multipath: report NVMe over Fabrics namespaces in the cluster status
ApiChangeLog | 9 +-
debian/control | 1 +
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 199 +++++++++++
src/PVE/API2/SAN/Cluster.pm | 22 +-
src/PVE/API2/SAN/Node.pm | 17 +-
src/PVE/API2/Storage/Scan.pm | 49 +++
src/PVE/CLI/pvesm.pm | 13 +
src/PVE/Diskmanage.pm | 29 +-
src/PVE/Makefile | 1 +
src/PVE/Multipath/Status.pm | 72 +++-
src/PVE/NVMeOF.pm | 445 +++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/LVMPlugin.pm | 3 +-
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 266 ++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 32 +-
src/test/Makefile | 6 +
src/test/disklist_test.pm | 14 +
src/test/run_multipath_api_tests.pl | 132 ++++++-
src/test/run_nvmeof_api_tests.pl | 207 +++++++++++
src/test/run_nvmeof_taint_tests.pl | 67 ++++
src/test/run_nvmeof_tests.pl | 533 ++++++++++++++++++++++++++++
src/test/run_san_api_tests.pl | 4 +-
src/test/run_san_lun_scan_tests.pl | 41 +++
src/test/run_san_nvme_tests.pl | 33 ++
27 files changed, 2160 insertions(+), 41 deletions(-)
pve-docs:
Thomas Lamprecht (1):
storage: document NVMe over Fabrics storages
pve-gui.adoc | 12 ++-
pve-intro.adoc | 3 +-
pve-storage-lvm.adoc | 10 +-
pve-storage-multipath.adoc | 3 +-
pve-storage-nvmeof.adoc | 189 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 24 +++--
6 files changed, 222 insertions(+), 19 deletions(-)
pve-manager:
Thomas Lamprecht (8):
report: add the NVMe over Fabrics host NQN and subsystems
ui: storage: add NVMe over Fabrics storage
ui: lvm: offer NVMe over Fabrics storages as base storage
ui: storage: add a guided setup for shared LVM on NVMe over Fabrics
ui: node san: list NVMe over Fabrics controllers
ui: san: show the NVMe host NQN of each node
ui: san: show that a namespace uses native multipath
pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps
PVE/MultipathNotification.pm | 5 +-
PVE/Report.pm | 6 +
PVE/Service/pvestatd.pm | 2 +-
test/MultipathNotification_test.pl | 3 +
www/manager6/Makefile | 4 +-
www/manager6/SANUtils.js | 7 +-
www/manager6/Utils.js | 7 +
www/manager6/dc/SAN.js | 49 +++-
www/manager6/dc/StorageView.js | 23 +-
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 152 ++++++++++++
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/LVMEdit.js | 12 +-
www/manager6/storage/NVMeOFEdit.js | 150 +++++++++++
www/manager6/storage/wizard/LunSelect.js | 35 ++-
www/manager6/storage/wizard/NodeIdentities.js | 57 +++--
.../storage/wizard/{IScsi.js => Target.js} | 232 +++++++++++++-----
www/manager6/storage/wizard/Wizard.js | 60 +++--
www/manager6/window/SANLunState.js | 38 ++-
19 files changed, 699 insertions(+), 157 deletions(-)
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
` (16 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The disk list queries the SMART health of every disk. For an NVMe
namespace attached over TCP, RDMA or Fibre Channel, the query goes to
the target and can stall the listing while a path recovers. Skip it for
these transports, also for namespaces connected by hand. Local NVMe
drives are queried as before.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Diskmanage.pm | 4 +++-
src/test/disklist_test.pm | 14 ++++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 77f1a0ec..be3fb8fc 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -709,7 +709,9 @@ sub get_disks {
}
my ($health, $wearout) = ('UNKNOWN', 'N/A');
- if (!$nosmart) {
+ # SMART queries over fabrics can stall while a path recovers.
+ my $remote_nvme = ($nvme_transport // '') =~ /^(?:tcp|rdma|fc)$/;
+ if (!$nosmart && !$remote_nvme) {
eval {
my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
$health = $smartdata->{health} if $smartdata->{health};
diff --git a/src/test/disklist_test.pm b/src/test/disklist_test.pm
index f79c59d3..38823f8b 100644
--- a/src/test/disklist_test.pm
+++ b/src/test/disklist_test.pm
@@ -284,6 +284,20 @@ sub test_disk_list {
ok($san_disks->{sda}, 'SAN inventory opts into iSCSI');
}
+ if ($testcasedir eq 'nvme_smart') {
+ my $smart_calls;
+ $diskmanage_module->mock(get_smart_data => sub { $smart_calls++; return {} });
+ for my $transport (qw(tcp rdma fc pcie)) {
+ $smart_calls = 0;
+ $diskmanage_module->mock(get_nvme_transport => sub { $transport });
+ PVE::Diskmanage::get_disks();
+ $testcount++;
+ is($smart_calls, $transport eq 'pcie' ? 1 : 0, "SMART queries for $transport");
+ }
+ $diskmanage_module->unmock('get_smart_data');
+ $diskmanage_module->unmock('get_nvme_transport');
+ }
+
if (-f "disk_tests/$testcasedir/multipathd_status_expected.json") {
my $status = PVE::Multipath::path_status_by_wwid();
my $expected_status =
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
` (15 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
NVMe/TCP namespaces need a manual nvme-cli setup on every node so far.
Add a storage type that connects one subsystem through each configured
portal and offers its namespaces as raw volumes. Native NVMe multipath
joins the paths and the kernel reconnects lost ones. Only one storage
may use a subsystem, as the LUN list, the controller list and the node
status attribute a subsystem to one storage.
The activation fails if no usable path exists after connecting the
missing portals, so that adding a storage tells why. As adding a
storage holds the lock of the configuration, it tries no further portal
after 15 seconds once a controller is live, and leaves the rest to the
next activation. A portal that failed is retried after a minute, and
adding a storage forgets those failures. Disk setup refuses namespaces
attached over a fabric, as for iSCSI LUNs, and nvme-cli is only
recommended, as no other storage type needs it.
The plugin isolates its properties, so names like 'portals' belong to
this storage type alone, and a third-party plugin that declares one of
them is still loaded. As the first in-tree plugin that isolates, it
makes the storage API check the parameters for adding and updating a
storage of any type against the options of that type alone. A
third-party plugin then has to list the sensitive properties it uses in
options(), as the API change log says.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
ApiChangeLog | 9 +-
debian/control | 1 +
src/PVE/Diskmanage.pm | 7 +-
src/PVE/Makefile | 1 +
src/PVE/NVMeOF.pm | 271 ++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 266 +++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 13 +
src/test/Makefile | 5 +
src/test/run_nvmeof_taint_tests.pl | 62 ++++
src/test/run_nvmeof_tests.pl | 446 +++++++++++++++++++++++++++++
src/test/run_san_nvme_tests.pl | 16 ++
14 files changed, 1096 insertions(+), 6 deletions(-)
create mode 100644 src/PVE/NVMeOF.pm
create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
create mode 100755 src/test/run_nvmeof_taint_tests.pl
create mode 100755 src/test/run_nvmeof_tests.pl
diff --git a/ApiChangeLog b/ApiChangeLog
index a51578a1..1816e18a 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -101,10 +101,11 @@ Future changes should be documented in here.
declares a property of that name is then not loaded until it renames it, so prefer names that
are specific to the plugin, for example prefixed with its type.
- As soon as one such plugin is installed, the API checks the parameters for adding and updating a
- storage against the options of its type alone, for all storage types. So every plugin has to
- list the sensitive properties that it uses in `options()`.
- Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of
+ Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for
+ adding and updating a storage against the options of its type alone, for all storage types,
+ also for plugins of older API versions. So every plugin has to list the sensitive properties
+ that it uses in `options()`.
+ Set 'sensitive-properties' in an isolating plugin, even if empty. Without it, the fallback list of
version 11 also applies to the plugin's own properties of these names.
* Pass the guest identity to backup providers
diff --git a/debian/control b/debian/control
index fdcf8117..b5c2f529 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
${misc:Depends},
${perl:Depends},
Recommends: multipath-tools,
+ nvme-cli,
pve-esxi-import-tools (>= 0.6.0),
skopeo,
zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index be3fb8fc..21857dc6 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -107,7 +107,12 @@ sub disk_is_used {
my $disklist = get_disks($dev, 1, 1);
- die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+ my $entry = $disklist->{$dev};
+ die "'$disk' is not a valid local disk\n" if !defined($entry);
+ # a partition has the transport of its disk
+ my ($parent) = ($entry->{parent} // '') =~ m!^/dev/(.+)$!;
+ my $transport = (defined($parent) ? $disklist->{$parent} : $entry)->{transport} // '';
+ die "'$disk' is not a valid local disk\n" if $transport =~ /^nvme-(?:tcp|rdma|fc)$/;
return 1 if $disklist->{$dev}->{used};
return 0;
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..45b3acfc
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,271 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+use Time::HiRes ();
+
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+use PVE::Diskmanage;
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $LOCK_WAIT = 10;
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery';
+
+sub canonical_address($addr) {
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ return inet_ntop(AF_INET, $packed);
+ }
+ if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+ return inet_ntop(AF_INET, substr($packed, 12))
+ if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+ return inet_ntop(AF_INET6, $packed);
+ }
+ return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+ my ($addr, $port);
+ if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } else {
+ return undef;
+ }
+ return undef if defined($port) && ($port < 1 || $port > 65535);
+ $addr = canonical_address($addr);
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0;
+ return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast;
+ } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+ my $prefix = unpack('n', $packed);
+ return undef
+ if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+ } else {
+ return undef;
+ }
+ return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+ 'pve-storage-nvme-portal',
+ sub($portal, $noerr = undef) {
+ return $portal if parse_portal($portal);
+ return undef if $noerr;
+ die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+ },
+);
+
+sub format_path($path) {
+ # an IPv6 address, not the address of a Fibre Channel port, which has a colon too
+ my $ipv6 = $path->{traddr} =~ /:/ && $path->{traddr} =~ /^[0-9a-fA-F:.]+$/;
+ my $addr = $ipv6 ? "[$path->{traddr}]" : $path->{traddr};
+ return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+ return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+ PVE::Tools::split_list($scfg->{portals})];
+}
+
+sub host_identity() {
+ my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+ my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+ /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+ if (!defined($nqn) || length($nqn) > 223) {
+ die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the"
+ . " host identity\n";
+ }
+ die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+ if !defined($id) || $id =~ /^[0-]+$/;
+ return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+ if !-e "$SYSFS/module/nvme_tcp";
+ my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+ die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+ dir_glob_foreach(
+ $dir,
+ 'nvme[0-9]+',
+ sub($name, @) {
+ my $address = file_read_firstline("$dir/$name/address") // return;
+ my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+ return if !defined($address{traddr});
+ $res->{$name} = {
+ transport => file_read_firstline("$dir/$name/transport") // '',
+ subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+ state => file_read_firstline("$dir/$name/state") // 'unknown',
+ traddr => canonical_address($address{traddr}),
+ trsvcid => $address{trsvcid},
+ };
+ },
+ );
+ return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+ my $controllers = controllers();
+ return {
+ map { $_ => $controllers->{$_} } grep {
+ $controllers->{$_}->{subsystem} eq $subsystem
+ && $controllers->{$_}->{transport} eq 'tcp'
+ } keys %$controllers
+ };
+}
+
+# The NQN of a subsystem, untainted, or undef if it is invalid or the one of the discovery service.
+sub subsystem_nqn($nqn) {
+ my ($res) = ($nqn // '') =~ /\A($NQN_PATTERN)\z/;
+ return undef if !defined($res) || length($res) > 223 || $res eq $DISCOVERY_NQN;
+ return $res;
+}
+
+sub namespace_wwid($dev) {
+ return file_read_firstline("$SYSFS/block/$dev/wwid");
+}
+
+sub has_live_controller($controllers) {
+ return scalar(grep { $_->{state} eq 'live' } values %$controllers);
+}
+
+sub path_controller($controllers, $path) {
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) {
+ return $name;
+ }
+ }
+ return undef;
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-subsystem";
+ dir_glob_foreach(
+ $dir,
+ 'nvme-subsys[0-9]+',
+ sub($subsys, @) {
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+ return if $nqn ne $subsystem;
+ dir_glob_foreach(
+ "$dir/$subsys",
+ 'nvme[0-9]+n[0-9]+',
+ sub($dev, @) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+ },
+ );
+ },
+ );
+ return $res;
+}
+
+sub with_lock($code) {
+ my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+ $LOCK_WAIT + 35,
+ sub {
+ make_path($RUN_DIR);
+ open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+ # a busy lock fails a connect and arms the cooldown of its portal, so wait a while
+ my $deadline = Time::HiRes::time() + $LOCK_WAIT;
+ while (!flock($lock, LOCK_EX | LOCK_NB)) {
+ die "NVMe over Fabrics is busy - another operation is still running\n"
+ if Time::HiRes::time() >= $deadline;
+ Time::HiRes::usleep(100_000);
+ }
+ # Commands inherit the lock, so that it stays held until they exit, even if the caller
+ # timed out while one was stuck in the kernel.
+ my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+ fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+ or die "cannot inherit NVMe lock - $!\n";
+ return $code->();
+ },
+ );
+ die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+ return $res;
+}
+
+my sub retry_file($scfg, $path) {
+ my ($key) =
+ sha1_hex(join(
+ "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~
+ /\A([0-9a-f]{40})\z/;
+ return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+ my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+ my $age = time() - $mtime;
+ return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+ make_path("$RUN_DIR/retry");
+ file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub clear_connect_failures($scfg) {
+ unlink(retry_file($scfg, $_)) for storage_paths($scfg)->@*;
+}
+
+sub connect_path($scfg, $path) {
+ my $nqn = subsystem_nqn($scfg->{subsysnqn}) // die "invalid subsystem NQN\n";
+ my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/;
+ die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+ return with_lock(sub {
+ return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ assert_support();
+ my $host = host_identity();
+ run_command(
+ [
+ $NVME,
+ 'connect',
+ '--transport=tcp',
+ '--nqn',
+ $nqn,
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $path->{trsvcid},
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--ctrl-loss-tmo=-1',
+ '--fast_io_fail_tmo',
+ $fail,
+ ],
+ timeout => 30,
+ );
+ # nvme-cli can exit successfully without having connected
+ die "no controller appeared\n"
+ if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ unlink(retry_file($scfg, $path));
+ return;
+ });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index c640cd8e..c77a6755 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin;
use PVE::Storage::NFSPlugin;
use PVE::Storage::CIFSPlugin;
use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
use PVE::Storage::RBDPlugin;
use PVE::Storage::CephFSPlugin;
use PVE::Storage::ISCSIDirectPlugin;
@@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register();
PVE::Storage::NFSPlugin->register();
PVE::Storage::CIFSPlugin->register();
PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
PVE::Storage::RBDPlugin->register();
PVE::Storage::CephFSPlugin->register();
PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
NFSPlugin.pm \
CIFSPlugin.pm \
ISCSIPlugin.pm \
+ NVMeOFPlugin.pm \
CephFSPlugin.pm \
RBDPlugin.pm \
ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..76f73af0
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,266 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use v5.36;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Network;
+
+use PVE::Diskmanage;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+# Adding a storage activates it under the lock of the storage configuration, so once a controller
+# is live, no further portal is tried after this many seconds. Later activations connect the rest.
+our $ACTIVATION_BUDGET = 15;
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+ return {
+ content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }],
+ select_existing => 1,
+ 'isolate-properties' => 1,
+ 'advanced-properties' => {
+ 'fast-io-fail-tmo' => 1,
+ },
+ 'hidden-properties' => {
+ bwlimit => 1,
+ },
+ 'sensitive-properties' => {},
+ };
+}
+
+sub properties {
+ return {
+ subsysnqn => {
+ title => "Subsystem",
+ description => 'The NQN of the NVMe subsystem.',
+ type => 'string',
+ pattern => $PVE::NVMeOF::NQN_PATTERN,
+ maxLength => 223,
+ },
+ portals => {
+ title => "Portals",
+ description =>
+ 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+ . ' with an optional port (default 4420).',
+ type => 'string',
+ format => 'pve-storage-nvme-portal-list',
+ },
+ 'fast-io-fail-tmo' => {
+ title => "Fail I/O After (s)",
+ description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+ . ' this timeout; I/O can remain queued while paths reconnect.',
+ type => 'integer',
+ minimum => -1,
+ default => 120,
+ },
+ };
+}
+
+sub options {
+ return {
+ subsysnqn => { fixed => 1 },
+ portals => {},
+ 'fast-io-fail-tmo' => { optional => 1 },
+ nodes => { optional => 1 },
+ disable => { optional => 1 },
+ content => { optional => 1 },
+ bwlimit => { optional => 1 },
+ };
+}
+
+my sub assert_valid_config($scfg) {
+ die "the discovery subsystem cannot be used as storage\n"
+ if $scfg->{subsysnqn} eq $PVE::NVMeOF::DISCOVERY_NQN;
+ my $paths = PVE::NVMeOF::storage_paths($scfg);
+ # the list format also accepts a list of separators only
+ die "no portal given\n" if !@$paths;
+ my %seen;
+ for my $path (@$paths) {
+ my $portal = PVE::NVMeOF::format_path($path);
+ die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+ }
+}
+
+sub on_add_hook($class, $storeid, $scfg, %param) {
+ assert_valid_config($scfg);
+ # the LUN and controller lists and the node status attribute a subsystem to its one storage
+ my $ids = PVE::Storage::config()->{ids};
+ for my $sid (sort keys %$ids) {
+ next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof';
+ die "storage '$sid' already uses this subsystem\n"
+ if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn};
+ }
+ # adding the storage again, for example once the array allows the node, tries all portals
+ PVE::NVMeOF::clear_connect_failures($scfg);
+ return;
+}
+
+sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) {
+ my $merged = { %$scfg, %$update };
+ delete $merged->{$_} for ($delete // [])->@*;
+ assert_valid_config($merged);
+ return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname($class, $volname) {
+ if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) {
+ return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw');
+ }
+ die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a
+# namespace of another subsystem.
+sub assert_namespace($scfg, $volname, $real) {
+ my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!;
+ die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$PVE::NVMeOF::SYSFS/block/$dev/device/subsysnqn")
+ // '';
+ die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+ if $nqn ne $scfg->{subsysnqn};
+ # another by-id name of the subsystem, like model and serial, can name another namespace
+ my $wwid = PVE::NVMeOF::namespace_wwid($dev) // '';
+ die "volume '$volname' is not named after the WWID of its namespace\n"
+ if $volname ne "nvme-$wwid";
+}
+
+sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) {
+ die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+ my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+ my $path = "/dev/disk/by-id/$name";
+ # not every user of a path activates the volume first, and the base volume of an LVM storage
+ # has no device before its storage connects
+ my $real = Cwd::realpath($path);
+ assert_namespace($scfg, $volname, $real) if defined($real) && -e $real;
+ return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) {
+ my $res = $class->list_images($storeid, $scfg, $vmid);
+ $_->{content} = 'images' for @$res;
+ return $res;
+}
+
+sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) {
+ return [] if defined($vmid) && !$vollist;
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ my $res = [];
+ for my $dev (sort keys %$namespaces) {
+ my $ns = $namespaces->{$dev};
+ my $volid = "$storeid:nvme-$ns->{wwid}";
+ next if $vollist && !grep { $_ eq $volid } @$vollist;
+ push @$res,
+ {
+ volid => $volid,
+ format => 'raw',
+ size => $ns->{size},
+ vmid => 0,
+ };
+ }
+ return $res;
+}
+
+sub status($class, $storeid, $scfg, $cache = undef, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ return (0, 0, 0, PVE::NVMeOF::has_live_controller($controllers) ? 1 : 0);
+}
+
+sub activate_storage($class, $storeid, $scfg, $cache = undef, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ my @missing = grep { !defined(PVE::NVMeOF::path_controller($controllers, $_)) }
+ PVE::NVMeOF::storage_paths($scfg)->@*;
+ return if !@missing;
+ # Without these no path connects, which is no reason to wait before the next attempt. A storage
+ # with a usable path still works, with fewer paths.
+ eval {
+ PVE::NVMeOF::assert_support();
+ PVE::NVMeOF::host_identity();
+ };
+ if (my $err = $@) {
+ die $err if !PVE::NVMeOF::has_live_controller($controllers);
+ warn "storage '$storeid': cannot connect to the other portals - $err";
+ return;
+ }
+ my @errors;
+ my $live = PVE::NVMeOF::has_live_controller($controllers);
+ my $deadline = Time::HiRes::time() + $ACTIVATION_BUDGET;
+ for my $path (@missing) {
+ my $portal = PVE::NVMeOF::format_path($path);
+ if ($live && Time::HiRes::time() >= $deadline) {
+ push @errors, "$portal: left for the next activation";
+ next;
+ }
+ if (PVE::NVMeOF::in_cooldown($scfg, $path)) {
+ push @errors, "$portal: failed less than a minute ago";
+ next;
+ }
+ eval {
+ die "no answer from portal\n"
+ if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::connect_path($scfg, $path);
+ };
+ if (my $err = $@) {
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ warn "storage '$storeid': connecting to '$portal' failed - $err";
+ chomp $err;
+ push @errors, "$portal: $err";
+ } else {
+ $live = 1;
+ }
+ }
+ # some paths are enough, but without a usable one the namespaces cannot be used
+ $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ die "cannot connect to any portal - " . join('; ', @errors) . "\n"
+ if !PVE::NVMeOF::has_live_controller($controllers);
+ return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection($class, $storeid, $scfg, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ return 1 if PVE::NVMeOF::has_live_controller($controllers);
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ }
+ return 0;
+}
+
+sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) {
+ my $path = $class->filesystem_path($scfg, $volname, $snapname);
+ # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device
+ my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10);
+ while (!-b $path && Time::HiRes::time() < $deadline) {
+ last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ Time::HiRes::usleep(250_000);
+ }
+ my $real = Cwd::realpath($path);
+ die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+ assert_namespace($scfg, $volname, $real);
+}
+
+sub volume_has_feature(
+ $class, $scfg, $feature, $storeid, $volname,
+ $snapname = undef,
+ $running = undef, @,
+) {
+ return 1 if $feature eq 'copy' && !defined($snapname);
+ return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index a97f361c..d61e78b6 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
);
our @SHARED_STORAGE = (
- 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+ 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
);
our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 0a9e441c..53c24c9e 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -302,6 +302,11 @@ sub lun_partitions($lun) {
}
sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+ my $name = basename($lun->{devpath});
+ my $nqn =
+ $name =~ /^nvme\d+n\d+$/
+ ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+ : undef;
my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
[$lun->{devpath}, lun_partitions($lun)->@*],
)->@*;
@@ -324,6 +329,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
) {
die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
}
+ if (
+ $type eq 'nvmeof'
+ && defined($nqn)
+ && $scfg->{subsysnqn} eq $nqn
+ && PVE::Storage::san_storage_serves_images($scfg)
+ ) {
+ die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+ }
next if $type ne 'lvm' || !$scfg->{base};
my $base = PVE::Multipath::parse_base_volume($scfg->{base});
my $matches = $base
diff --git a/src/test/Makefile b/src/test/Makefile
index 3b902c43..ceabd991 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,6 +17,7 @@ test: \
test_lvmplugin \
test_multipath \
test_multipath_api \
+ test_nvmeof \
test_ovf \
test_plugin \
test_remote_command \
@@ -115,3 +116,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+ ./run_nvmeof_tests.pl
+ perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..6fd37410
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,62 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Tools qw(file_get_contents file_set_contents);
+
+use PVE::NVMeOF;
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+ subsystem_controllers => sub {
+ return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+ },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+ run_command => sub($cmd, %opts) {
+ $commands++;
+ is(
+ scalar(grep { tainted($_) } @$cmd),
+ 0,
+ 'all connect operands are validated and untainted',
+ );
+ PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+ PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+ 'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+ eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+ like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..cdf228c0
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,446 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Fcntl qw(:flock);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use POSIX ();
+use Test::MockModule;
+use Test::More;
+use Time::HiRes ();
+
+use PVE::Tools qw(file_set_contents);
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ portals => '192.0.2.1,[2001:db8::2]:4421',
+ content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+ is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+ is_deeply(
+ [sort keys $class->properties()->%*],
+ [qw(fast-io-fail-tmo portals subsysnqn)],
+ 'only three plugin-specific properties',
+ );
+ my sub nvmeof_schema($schema) {
+ my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*;
+ my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types;
+ return $nvmeof->{properties} // {};
+ }
+ my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema());
+ is_deeply(
+ [sort grep { !$create->{$_}->{optional} } keys $create->%*],
+ [qw(portals subsysnqn)],
+ 'adding a storage requires the subsystem and the portals',
+ );
+ my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema());
+ ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed');
+ my $parsed = PVE::Storage::Plugin->parse_config(
+ 'storage.cfg',
+ "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n",
+ );
+ is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+ is_deeply(
+ $parsed->{ids}->{fabric}->{content},
+ { none => 1 },
+ 'connection-only content parses',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ { traddr => '192.0.2.1', trsvcid => 4420 },
+ 'default I/O port',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+ { traddr => '2001:db8::1', trsvcid => 4421 },
+ 'canonical IPv6',
+ );
+ is(
+ PVE::NVMeOF::format_path({ traddr => '2001:db8::1', trsvcid => 4421 }),
+ '[2001:db8::1]:4421',
+ 'an IPv6 portal in brackets',
+ );
+ is(
+ PVE::NVMeOF::format_path({ traddr => 'nn-0x2000:pn-0x2001' }),
+ 'nn-0x2000:pn-0x2001',
+ 'the address of a Fibre Channel port as it is',
+ );
+
+ for my $bad (
+ 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+ ) {
+ ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+ }
+ eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); };
+ like($@, qr/listed more than once/, 'duplicate portals refused');
+ eval { $class->on_add_hook('fabric', { %$cfg, portals => ',,' }); };
+ like($@, qr/no portal given/, 'a list without a portal refused');
+ eval {
+ $class->on_add_hook(
+ 'fabric',
+ { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+ );
+ };
+ like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $ids = { fabric => $cfg, local => { type => 'dir' } };
+ $storage->mock(config => sub { { ids => $ids } });
+ my $first = PVE::NVMeOF::storage_paths($cfg)->[0];
+ PVE::NVMeOF::note_connect_failure($cfg, $first);
+ is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way');
+ ok(!PVE::NVMeOF::in_cooldown($cfg, $first), 'adding it tries all portals again');
+ eval { $class->on_add_hook('second', {%$cfg}) };
+ like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem');
+ is(
+ scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+ '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+ 'stable volume path',
+ );
+ {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $nqn = 'nqn.other';
+ $disk->redefine(read_subsysnqn => sub { $nqn });
+ my $sysfs = Test::MockModule->new('PVE::NVMeOF');
+ $sysfs->redefine(namespace_wwid => sub { 'x' });
+ my $check = sub {
+ eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) };
+ $@;
+ };
+ like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem');
+ like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type');
+ like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition');
+ $nqn = 'nqn.test';
+ is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem');
+ like(
+ $check->('nvme-Vendor_Model_123_1', '/dev/nvme0n1'),
+ qr/not named after the WWID/,
+ 'another by-id name of a namespace of the subsystem',
+ );
+ my $cwd = Test::MockModule->new('Cwd');
+ $cwd->redefine(realpath => sub { $dir });
+ eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') };
+ like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked');
+ }
+
+ for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+ eval { $class->parse_volname($bad) };
+ like($@, qr/unable to parse/, "invalid volume $bad");
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } });
+ is(
+ $class->list_volumes('fabric', $cfg)->[0]->{volid},
+ 'fabric:nvme-eui.abc',
+ 'content none still offers bases',
+ );
+ is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+ $nvme->mock(subsystem_controllers => sub { {} });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+ like($@, qr/not available/,
+ 'missing namespace fails without waiting on absent controllers');
+ my ($now, $queries) = (0, 0);
+ my $clock = Test::MockModule->new('Time::HiRes');
+ $clock->redefine(time => sub { $now += 5 }, usleep => sub { });
+ $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) };
+ is($queries, 1, 'a missing namespace of a connected subsystem is waited for');
+ eval {
+ $class->activate_volume(
+ 'fabric',
+ $cfg,
+ 'nvme-nonexistent-test',
+ undef,
+ { status_only => 1 },
+ );
+ };
+ like($@, qr/not available/, 'in a status pass it fails as well');
+ is($queries, 1, 'but without a wait');
+ is(
+ ($class->parse_volname('nvme-eui.0123456789abcdef'))[0],
+ 'vm-vol',
+ 'namespaces are VM disks',
+ );
+};
+
+subtest 'sysfs topology' => sub {
+ my $sys = $PVE::NVMeOF::SYSFS;
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1",
+ "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+ "$sys/block/nvme0n1",
+ );
+ my %attrs = (
+ address => 'traddr=192.0.2.1,trsvcid=4420',
+ transport => 'tcp',
+ subsysnqn => 'nqn.test',
+ state => 'connecting',
+ );
+ file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+ file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+ file_set_contents("$sys/block/nvme0n1/size", "8\n");
+ is(
+ PVE::NVMeOF::controllers()->{nvme1}->{state},
+ 'connecting',
+ 'reconnecting controller is still present',
+ );
+ is(
+ PVE::NVMeOF::path_controller(
+ PVE::NVMeOF::subsystem_controllers('nqn.test'),
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ ),
+ 'nvme1',
+ 'portal controller found',
+ );
+ is_deeply(
+ PVE::NVMeOF::namespaces('nqn.test'),
+ { nvme0n1 => { wwid => 'eui.abc', size => 4096 } },
+ 'namespace facts',
+ );
+ is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+ local $PVE::NVMeOF::LOCK_WAIT = 0;
+ my $script = join(
+ ' ',
+ 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*");',
+ 'die "not inherited" if !@held;',
+ 'open(my $fh, ">>", $ARGV[0]) or die $!;',
+ 'die "lock missing" if flock($fh, LOCK_EX|LOCK_NB);',
+ 'print "locked\\n";',
+ );
+ my $value = PVE::NVMeOF::with_lock(sub {
+ my $out = '';
+ PVE::Tools::run_command(
+ [
+ '/usr/bin/perl',
+ '-MFcntl=:flock',
+ '-e',
+ $script,
+ "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+ ],
+ timeout => 5,
+ outfunc => sub { $out .= $_[0] },
+ );
+ return $out;
+ });
+ is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+ open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+ flock($lock, LOCK_EX) or die $!;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'busy operation never enters callback');
+ close($lock);
+
+ my $tools = Test::MockModule->new('PVE::Tools');
+ my $timed_out = 1;
+ $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+ my $retained;
+ eval {
+ PVE::NVMeOF::with_lock(sub {
+ my ($path) =
+ grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+ my ($fd) = $path =~ m{/(\d+)$};
+ open($retained, '>&', $fd) or die $!;
+ return 1;
+ });
+ };
+ like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+ $timed_out = 0;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+ close($retained);
+ is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'a short hold of the lock does not fail a connect' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my $scfg = { %$cfg, subsysnqn => 'nqn.wait', portals => '192.0.2.3' };
+ my $ctl = "$PVE::NVMeOF::SYSFS/class/nvme-fabrics/ctl/nvme9";
+ $net->mock(tcp_ping => sub { 1 });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity =>
+ sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+ $nvme->mock(
+ run_command => sub(@) {
+ my %attrs = (
+ address => 'traddr=192.0.2.3,trsvcid=4420',
+ transport => 'tcp',
+ subsysnqn => 'nqn.wait',
+ state => 'live',
+ );
+ make_path($ctl);
+ file_set_contents("$ctl/$_", "$attrs{$_}\n") for keys %attrs;
+ },
+ );
+ pipe(my $held, my $signal) or die "pipe failed - $!\n";
+ my $holder = fork() // die "fork failed - $!\n";
+ if (!$holder) {
+ open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or POSIX::_exit(1);
+ flock($lock, LOCK_EX) or POSIX::_exit(1);
+ syswrite($signal, '1');
+ Time::HiRes::sleep(0.5);
+ POSIX::_exit(0);
+ }
+ close($signal);
+ sysread($held, my $state, 1);
+ is($state, '1', 'another process holds the lock');
+ my $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ $class->activate_storage('wait', $scfg);
+ }
+ waitpid($holder, 0);
+ is($warning, '', 'the connect waits for the lock');
+ my $path = PVE::NVMeOF::storage_paths($scfg)->[0];
+ ok(!PVE::NVMeOF::in_cooldown($scfg, $path), 'and arms no cooldown');
+ is(
+ PVE::NVMeOF::path_controller(PVE::NVMeOF::subsystem_controllers('nqn.wait'), $path),
+ 'nvme9',
+ 'the portal is connected',
+ );
+ remove_tree($ctl);
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my @commands;
+ my $controllers = {};
+ $nvme->mock(with_lock => sub { $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity =>
+ sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+ $nvme->mock(subsystem_controllers => sub { $controllers });
+ my $appear = 1;
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ is($opts{timeout}, 30, 'bounded connect');
+ push @commands, $cmd;
+ my %arg =
+ map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+ my $controller =
+ { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' };
+ $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear;
+ },
+ );
+ $net->mock(tcp_ping => sub { 1 });
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 2, 'one connect per missing portal');
+ ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+ is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+ {
+ local $PVE::Storage::NVMeOFPlugin::ACTIVATION_BUDGET = 0;
+ @commands = ();
+ $controllers = {};
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 1, 'with a live controller, a spent time budget leaves the rest');
+ ($controllers, $appear) = ({}, 0);
+ @commands = ();
+ {
+ local $SIG{__WARN__} = sub { };
+ eval { $class->activate_storage('fabric', $cfg) };
+ }
+ is(scalar(@commands), 2, 'without one, every portal is tried');
+ unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*"));
+ $appear = 1;
+ }
+ @commands = ();
+ $controllers = {
+ nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+ nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+ };
+ $class->activate_storage('fabric', $cfg);
+ is_deeply(\@commands, [],
+ 'existing controllers are never reconnected, regardless of state');
+ $class->deactivate_storage('fabric', $cfg);
+ is_deeply(\@commands, [], 'deactivation does not remove controllers');
+ $controllers = {};
+ my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+ PVE::NVMeOF::note_connect_failure($cfg, $path);
+ ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 1, 'only the other portal connects');
+ ($controllers, $appear) = ({}, 0);
+ my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+ my $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ eval { $class->activate_storage('fabric', $cfg) };
+ }
+ like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+ ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+ like(
+ $@,
+ qr/^cannot connect to any portal - .*failed less than a minute ago; .*no controller appeared/,
+ 'without any controller the activation fails with the reason of each portal',
+ );
+ unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*"));
+ $nvme->mock(assert_support => sub { die "nvme-cli is not installed\n" });
+ @commands = ();
+ eval { $class->activate_storage('fabric', $cfg) };
+ is($@, "nvme-cli is not installed\n", 'a missing prerequisite fails the activation');
+ is_deeply(\@commands, [], 'before any connect');
+ ok(!PVE::NVMeOF::in_cooldown($cfg, $path), 'and without a cooldown');
+ $controllers = { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'live' } };
+ $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ is(
+ eval { $class->activate_storage('fabric', $cfg); 1 },
+ 1,
+ 'but not with a usable path',
+ );
+ }
+ like($warning, qr/cannot connect to the other portals - nvme-cli/, 'which it warns about');
+ $nvme->mock(assert_support => sub { });
+ $controllers =
+ { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' } };
+ {
+ local $SIG{__WARN__} = sub { };
+ eval { $class->activate_storage('fabric', $cfg) };
+ }
+ like(
+ $@,
+ qr/^cannot connect to any portal/,
+ 'a reconnecting path is not usable when the missing one fails',
+ );
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+ $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+ eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+ like($@, qr/not a valid local disk/,
+ "$transport refuses without storage configuration");
+ $disk->mock(
+ get_disks => sub {
+ {
+ nvme0n1 => { transport => $transport },
+ nvme0n1p1 => { parent => '/dev/nvme0n1', type => 'partition' },
+ };
+ },
+ );
+ eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1p1') };
+ like($@, qr/not a valid local disk/, "$transport refuses a partition of a namespace");
+ }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 3245b774..2a9a2121 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
$disk->mock(verify_blockdev_path => sub { $device });
$disk->mock(is_partition => sub { 0 });
$disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
$disk->mock(is_iscsi_device => sub { 0 });
$disk->mock(is_mounted => sub { 0 });
$disk->mock(has_holder => sub { 0 });
@@ -86,6 +87,21 @@ for my $base (qw(fabric nvme-fabric)) {
eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
like($@, qr/base volume/, "native base reference on storage '$base' refuses");
}
+$cfg->{ids} =
+ { fabric =>
+ { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+ $@,
+ qr/serves images from this subsystem/,
+ 'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks');
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
$cfg->{ids} = {};
@order = @opens = ();
$tools->mock(file_get_contents => sub { 1024 });
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
` (14 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
An LVM storage only accepts an iSCSI LUN as base volume. Accept a
namespace of an NVMe over Fabrics storage too. Adding such an LVM
storage connects the subsystem, waits for the namespace and creates
the volume group on it, and activating it on another node connects
that node first.
The creation treats the namespace like any SAN LUN: it takes the
cluster-wide lock of the LUN and refuses a namespace that another
storage references.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Storage/LVMPlugin.pm | 3 +--
src/test/run_san_nvme_tests.pl | 17 +++++++++++++++++
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index b759554c..224f1c68 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -1355,9 +1355,8 @@ sub on_add_hook {
my $basecfg = PVE::Storage::storage_config($cfg, $baseid, 1);
die "base storage ID '$baseid' does not exist\n" if !$basecfg;
- # we only support iscsi for now
die "unsupported base type '$basecfg->{type}'"
- if $basecfg->{type} ne 'iscsi';
+ if $basecfg->{type} ne 'iscsi' && $basecfg->{type} ne 'nvmeof';
# Managed multipath requires the coalesced map or a partition of it, never a transient
# single-path PV.
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 2a9a2121..15ede8be 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -142,4 +142,21 @@ is_deeply(
);
is_deeply(\@opens, [$link], 'VG creation holds the native by-id device');
+$cfg->{ids} = { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } };
+my @activation;
+$storage->mock(path => sub { $link });
+$storage->mock(activate_storage => sub { push @activation, 'storage' });
+$storage->mock(activate_volumes => sub { push @activation, 'volume' });
+$lvm->mock(
+ lvm_create_volume_group => sub {
+ is($_[0], $link, 'add hook passes native base');
+ is($_[3]->{exempt}, 'data', 'add hook exempts new storage');
+ },
+);
+PVE::Storage::LVMPlugin->on_add_hook(
+ 'data',
+ { base => 'fabric:nvme-eui.0123456789abcdef', vgname => 'data', shared => 1 },
+);
+is_deeply(\@activation, ['storage', 'volume'], 'LVM activates the NVMe base before creation');
+
done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (2 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
` (13 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Setting up an NVMe over Fabrics storage needs the NQN of the
subsystem and its portals. Add /nodes/{node}/scan/nvmeof, which asks
the discovery service at a portal for the NVMe/TCP subsystems it
offers to the node and lists them with their portals.
Without a port, it tries the discovery port 8009 and then the I/O
port 4420, where Linux targets answer discovery too. The scan needs
Datastore.Allocate on /storage, like the iSCSI scan.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Storage/Scan.pm | 49 +++++++++++++++++++++
src/PVE/NVMeOF.pm | 63 +++++++++++++++++++++++++++
src/test/run_nvmeof_taint_tests.pl | 5 +++
src/test/run_nvmeof_tests.pl | 68 ++++++++++++++++++++++++++++++
src/test/run_san_api_tests.pl | 2 +-
5 files changed, 186 insertions(+), 1 deletion(-)
diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm
index f37e7d4e..c6263994 100644
--- a/src/PVE/API2/Storage/Scan.pm
+++ b/src/PVE/API2/Storage/Scan.pm
@@ -6,6 +6,7 @@ use warnings;
# NOTE: This API endpoints are mounted by pve-manager's API2::Node module and pvesm CLI
use PVE::JSONSchema qw(get_standard_option);
+use PVE::NVMeOF;
use PVE::RESTHandler;
use PVE::SafeSyslog;
@@ -49,6 +50,7 @@ __PACKAGE__->register_method({
{ method => 'lvm' },
{ method => 'lvmthin' },
{ method => 'nfs' },
+ { method => 'nvmeof' },
{ method => 'pbs' },
{ method => 'zfs' },
];
@@ -300,6 +302,53 @@ __PACKAGE__->register_method({
},
});
+__PACKAGE__->register_method({
+ name => 'nvmeofscan',
+ path => 'nvmeof',
+ method => 'GET',
+ description =>
+ 'List the NVMe over TCP subsystems offered to this node by a discovery controller.',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/storage', ['Datastore.Allocate']] },
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ portal => {
+ description => 'IPv4 or bracketed IPv6 discovery address with an optional port.'
+ . ' Without a port, try 8009 then 4420.',
+ type => 'string',
+ format => 'pve-storage-nvme-portal',
+ },
+ },
+ },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ portals => {
+ type => 'string',
+ description => 'The discovered TCP portals, comma separated.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my ($param) = @_;
+ my $res = PVE::NVMeOF::discover($param->{portal});
+ return [
+ map { {
+ subsysnqn => $_,
+ portals => join(',', map { PVE::NVMeOF::format_path($_) } $res->{$_}->@*),
+ } } sort keys %$res
+ ];
+ },
+});
+
__PACKAGE__->register_method({
name => 'lvmscan',
path => 'lvm',
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index 45b3acfc..15612b04 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -5,10 +5,12 @@ use v5.36;
use Digest::SHA qw(sha1_hex);
use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
use File::Path qw(make_path);
+use JSON qw(decode_json);
use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
use Time::HiRes ();
use PVE::JSONSchema;
+use PVE::Network;
use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
use PVE::Diskmanage;
@@ -268,4 +270,65 @@ sub connect_path($scfg, $path) {
});
}
+sub parse_discovery_log($raw) {
+ my $log = decode_json($raw);
+ die "invalid NVMe discovery log\n"
+ if ref($log) ne 'HASH' || ref($log->{records}) ne 'ARRAY';
+ my (%seen, %res);
+ for my $record ($log->{records}->@*) {
+ next
+ if ($record->{subtype} // '') ne 'nvme subsystem' || ($record->{trtype} // '') ne 'tcp';
+ my $nqn = subsystem_nqn($record->{subnqn}) // next;
+ my $addr = $record->{traddr} // '';
+ my $port = $record->{trsvcid} // '';
+ my $path = parse_portal(($addr =~ /:/ ? "[$addr]" : $addr) . ":$port") // next;
+ next if $seen{$nqn}->{ format_path($path) }++;
+ push $res{$nqn}->@*, $path;
+ }
+ return \%res;
+}
+
+sub discover($portal) {
+ my $path = parse_portal($portal) // die "invalid portal '$portal'\n";
+ my @ports = $portal =~ /:[0-9]+$/ ? ($path->{trsvcid}) : (8009, 4420);
+ # a missing prerequisite fails on every port, so report it as it is
+ assert_support();
+ my $host = host_identity();
+ my $error;
+ for my $port (@ports) {
+ # the reason of a port that answered tells more than a silent one
+ if (!PVE::Network::tcp_ping($path->{traddr}, $port, 2)) {
+ $error //= "no answer from portal\n";
+ next;
+ }
+ my $result = eval {
+ with_lock(sub {
+ my $out = '';
+ run_command(
+ [
+ $NVME,
+ 'discover',
+ '--transport=tcp',
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $port,
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--output-format=json',
+ ],
+ timeout => 30,
+ outfunc => sub($line) { $out .= "$line\n" },
+ );
+ return parse_discovery_log($out);
+ });
+ };
+ return $result if !$@;
+ $error = $@;
+ }
+ die "discovery at portal '$portal' failed - $error";
+}
+
1;
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
index 6fd37410..ef332633 100755
--- a/src/test/run_nvmeof_taint_tests.pl
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -40,6 +40,7 @@ $nvme->mock(
'all connect operands are validated and untainted',
);
PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ $opts{outfunc}->('{"records":[]}') if $opts{outfunc};
},
);
my $path = PVE::NVMeOF::parse_portal($portal);
@@ -55,6 +56,10 @@ for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.disc
eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
}
+my $net = Test::MockModule->new('PVE::Network');
+$net->mock(tcp_ping => sub { 1 });
+eval { PVE::NVMeOF::discover($portal) };
+is($@, '', 'discovery can execute with file-backed host identity in taint mode');
file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
eval { PVE::NVMeOF::host_identity() };
like($@, qr/nonzero UUID/, 'zero host ID refused');
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index cdf228c0..e33aed75 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -5,6 +5,7 @@ use lib qw(..);
use Fcntl qw(:flock);
use File::Path qw(make_path remove_tree);
use File::Temp qw(tempdir);
+use JSON qw(encode_json);
use POSIX ();
use Test::MockModule;
use Test::More;
@@ -12,6 +13,7 @@ use Time::HiRes ();
use PVE::Tools qw(file_set_contents);
+use PVE::API2::Storage::Scan;
use PVE::NVMeOF;
use PVE::Storage;
@@ -443,4 +445,70 @@ subtest 'local disk setup rejects fabric namespaces by transport' => sub {
}
};
+subtest 'TCP discovery, fallback and API result' => sub {
+ my $record = {
+ subtype => 'nvme subsystem',
+ trtype => 'tcp',
+ subnqn => 'nqn.test',
+ traddr => '192.0.2.1',
+ trsvcid => '4420',
+ };
+ my $raw = encode_json({
+ records =>
+ [$record, $record, { %$record, trtype => 'rdma' }, { %$record, trsvcid => 70000 }],
+ });
+ is_deeply(
+ PVE::NVMeOF::parse_discovery_log($raw),
+ { 'nqn.test' => [{ traddr => '192.0.2.1', trsvcid => 4420 }] },
+ 'TCP only, unique valid portals',
+ );
+ for my $bad ('garbage', '{}', '[]') {
+ eval { PVE::NVMeOF::parse_discovery_log($bad) };
+ ok($@, 'malformed discovery is not an empty success');
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my (@ports, @commands);
+ our $locked;
+ $nvme->mock(with_lock => sub { local $locked = 1; $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'id' } });
+ $net->mock(tcp_ping => sub { push @ports, $_[1]; $_[1] != 8009 });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'discovery holds the NVMe lock');
+ is($opts{timeout}, 30, 'discovery command timeout');
+ push @commands, $cmd;
+ $opts{outfunc}->($raw);
+ },
+ );
+ my $rows = PVE::API2::Storage::Scan->nvmeofscan({ node => 'n1', portal => '192.0.2.1' });
+ is_deeply(\@ports, [8009, 4420], 'unspecified port tries discovery then I/O port');
+ is_deeply($rows, [{ subsysnqn => 'nqn.test', portals => '192.0.2.1:4420' }], 'API shape');
+ eval {
+ PVE::JSONSchema::validate(
+ $rows,
+ PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns},
+ );
+ };
+ is($@, '', 'API schema accepts result');
+ @ports = ();
+ PVE::NVMeOF::discover('[2001:db8::1]:4421');
+ is_deeply(\@ports, [4421], 'explicit port does not fall back');
+ # the discovery port answers but refuses, the I/O port does not answer
+ $net->mock(tcp_ping => sub { $_[1] == 8009 });
+ $nvme->mock(run_command => sub { die "host not allowed\n" });
+ eval { PVE::NVMeOF::discover('192.0.2.1') };
+ like($@, qr/failed - host not allowed$/, 'the error of the answering port is kept');
+ $net->mock(tcp_ping => sub { $_[1] == 4420 });
+ eval { PVE::NVMeOF::discover('192.0.2.1') };
+ like($@, qr/failed - host not allowed$/, 'also if the other port answers');
+ $net->mock(tcp_ping => sub { 0 });
+ eval { PVE::NVMeOF::discover('192.0.2.1') };
+ like($@, qr/failed - no answer from portal$/, 'without any answer');
+ $nvme->mock(assert_support => sub { die "nvme-cli is not installed\n" });
+ eval { PVE::NVMeOF::discover('192.0.2.1') };
+ is($@, "nvme-cli is not installed\n", 'a missing prerequisite is reported as it is');
+};
+
done_testing();
diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl
index faa29fc3..73938a7e 100755
--- a/src/test/run_san_api_tests.pl
+++ b/src/test/run_san_api_tests.pl
@@ -104,7 +104,7 @@ is_deeply(
);
is_deeply(
[map { $_->{method} } PVE::API2::Storage::Scan->index({ node => 'test' })->@*],
- [qw(cifs iscsi lvm lvmthin nfs pbs zfs)],
+ [qw(cifs iscsi lvm lvmthin nfs nvmeof pbs zfs)],
'scan index lists the discovery scans',
);
is_deeply(
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (3 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
` (12 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Add 'pvesm scan nvmeof <portal>', which prints the subsystems a
portal offers to the node with their portals, like the iSCSI scan
prints targets. That gives the values for a new storage on the
command line, with the usual output formats for scripts.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/CLI/pvesm.pm | 13 +++++++++++++
src/test/run_nvmeof_tests.pl | 12 ++++++++++++
2 files changed, 25 insertions(+)
diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm
index 070c1596..05da60d8 100755
--- a/src/PVE/CLI/pvesm.pm
+++ b/src/PVE/CLI/pvesm.pm
@@ -737,6 +737,19 @@ our $cmddef = {
$print_api_result,
$PVE::RESTHandler::standard_output_options,
],
+ nvmeof => [
+ "PVE::API2::Storage::Scan",
+ 'nvmeofscan',
+ ['portal'],
+ { node => $nodename },
+ sub {
+ my ($data, $schema, $options) = @_;
+ PVE::CLIFormatter::print_api_result(
+ $data, $schema, [qw(subsysnqn portals)], $options,
+ );
+ },
+ $PVE::RESTHandler::standard_output_options,
+ ],
'san-luns' => [
"PVE::API2::SAN::Node",
'luns',
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index e33aed75..ddd905ee 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -14,6 +14,7 @@ use Time::HiRes ();
use PVE::Tools qw(file_set_contents);
use PVE::API2::Storage::Scan;
+use PVE::CLI::pvesm;
use PVE::NVMeOF;
use PVE::Storage;
@@ -509,6 +510,17 @@ subtest 'TCP discovery, fallback and API result' => sub {
$nvme->mock(assert_support => sub { die "nvme-cli is not installed\n" });
eval { PVE::NVMeOF::discover('192.0.2.1') };
is($@, "nvme-cli is not installed\n", 'a missing prerequisite is reported as it is');
+ my $formatter = Test::MockModule->new('PVE::CLIFormatter');
+ my $schema = PVE::API2::Storage::Scan->map_method_by_name('nvmeofscan')->{returns};
+ $formatter->mock(
+ print_api_result => sub {
+ is($_[0], $rows, 'CLI forwards complete discovery rows');
+ is($_[1], $schema, 'CLI forwards discovery schema');
+ is($_[3]->{'output-format'}, 'json', 'CLI supports structured output');
+ },
+ );
+ $PVE::CLI::pvesm::cmddef->{scan}->{nvmeof}->[4]
+ ->($rows, $schema, { 'output-format' => 'json' });
};
done_testing();
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (4 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
` (11 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The SAN LUN inventory lists NVMe namespaces with their subsystem, but
without relation to an NVMe over Fabrics storage. Expect one path per
portal of the storage that connects the subsystem of a namespace, so
that a namespace with fewer paths shows as degraded.
A namespace counts as used by the LVM storage that has it as base
volume, else by the storage of its subsystem if that one serves the
namespaces as images.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/SAN/Node.pm | 9 ++++---
src/PVE/Storage/SAN.pm | 19 ++++++++++++--
src/test/run_san_lun_scan_tests.pl | 41 ++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/src/PVE/API2/SAN/Node.pm b/src/PVE/API2/SAN/Node.pm
index 08a7473b..4b96caf5 100644
--- a/src/PVE/API2/SAN/Node.pm
+++ b/src/PVE/API2/SAN/Node.pm
@@ -118,8 +118,9 @@ __PACKAGE__->register_method({
'paths-expected' => {
description => "The number of paths the LUN should have on this node: the"
. " highest of the count of its multipath entry and, for each managed"
- . " iSCSI storage of the LUN, the number of its portals. Absent if no"
- . " count is known.",
+ . " iSCSI storage of the LUN, the number of its portals. For a namespace"
+ . " of an NVMe over Fabrics storage, the number of its portals. Absent if"
+ . " no count is known.",
type => 'integer',
optional => 1,
},
@@ -198,8 +199,8 @@ __PACKAGE__->register_method({
},
'used-by-storage' => {
description => "The storage that uses the LUN: as its base volume or"
- . " through the volume group on it, or as an iSCSI storage that serves the"
- . " LUNs of its target as images.",
+ . " through the volume group on it, or as an iSCSI or NVMe over Fabrics"
+ . " storage that serves the LUNs of its target or subsystem as images.",
type => 'string',
optional => 1,
},
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 53c24c9e..f0ca5aef 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -16,6 +16,7 @@ use PVE::Multipath;
use PVE::Multipath::Config;
use PVE::Multipath::Generator;
use PVE::Multipath::Identity;
+use PVE::NVMeOF;
use PVE::Storage;
use PVE::Storage::LVMPlugin;
@@ -42,7 +43,9 @@ sub scan_luns($cfg) {
PVE::Multipath::Config::wwids_for_node($mpcfg, PVE::INotify::nodename())->@*;
my %listed = map { $_ => 1 } PVE::Multipath::list_etc_multipath_wwids()->@*;
my $derived = PVE::Multipath::Generator::derived_wwids($cfg, $ids);
- my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %targets, %paths, %unmapped);
+ my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %nvme_owner, %nvme_paths);
+ my (%targets, %paths);
+ my %unmapped;
my $wanted =
PVE::Multipath::Generator::wanted_wwids($mpcfg, PVE::INotify::nodename(), $derived);
@@ -58,6 +61,11 @@ sub scan_luns($cfg) {
if defined($scfg->{target}) && PVE::Storage::san_storage_serves_images($scfg);
my $base = PVE::Multipath::parse_base_volume($scfg->{base});
$base_owner{ $base->{wwid} } //= $sid if $base;
+ next if $scfg->{type} ne 'nvmeof';
+ $nvme_owner{ $scfg->{subsysnqn} } //= $sid;
+ # only a storage in use on this node expects its paths here
+ my $here = !$scfg->{nodes} || $scfg->{nodes}->{ PVE::INotify::nodename() };
+ $nvme_paths{ $scfg->{subsysnqn} } //= $sid if $here && !$scfg->{disable};
}
my %rows;
@@ -102,6 +110,10 @@ sub scan_luns($cfg) {
$row->{'nvme-subsystem'} = $nqn;
my $nsid = file_read_firstline("/sys/block/$name/nsid");
$row->{'nvme-nsid'} = int($nsid) if defined($nsid) && $nsid =~ /^\d+$/;
+ if (my $sid = $nvme_paths{$nqn}) {
+ $row->{'paths-expected'} =
+ scalar(PVE::NVMeOF::storage_paths($cfg->{ids}->{$sid})->@*);
+ }
}
}
for my $key (qw(transport vendor model serial)) {
@@ -208,9 +220,12 @@ sub scan_luns($cfg) {
$row->{'paths-expected'},
);
}
+ my $nvme = $nvme_owner{ $row->{'nvme-subsystem'} // '' };
+ undef $nvme if $nvme && !PVE::Storage::san_storage_serves_images($cfg->{ids}->{$nvme});
my ($target) =
map { $target_owner{$_} // () } sort keys(($targets{ $row->{wwid} } // {})->%*);
- my $owner = $base_owner{ $row->{wwid} } // $vg_owner{ $row->{vgname} // '' } // $target;
+ my $owner = $base_owner{ $row->{wwid} } // $vg_owner{ $row->{vgname} // '' } // $target
+ // $nvme;
$row->{'used-by-storage'} = $owner if defined($owner);
}
return [map { $rows{$_} } sort keys %rows];
diff --git a/src/test/run_san_lun_scan_tests.pl b/src/test/run_san_lun_scan_tests.pl
index 7119c88a..7fd9563b 100755
--- a/src/test/run_san_lun_scan_tests.pl
+++ b/src/test/run_san_lun_scan_tests.pl
@@ -329,6 +329,12 @@ is($rows->[0]->{devpath}, '/dev/sdc', 'the row names the path that its by-id lin
$disks = { nvme0n1 => { devpath => '/dev/nvme0n1', size => 100, transport => 'nvme-fc' } };
$scan->mock(file_read_firstline => sub { $_[0] =~ m{/nsid$} ? 1 : 'eui.0123456789abcdef' });
$disk->mock(read_subsysnqn => sub { 'nqn.test' });
+$cfg->{ids}->{fabric} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ portals => '192.0.2.1,192.0.2.2',
+ content => { images => 1 },
+};
$scan->unmock('stable_path');
$scan->mock(abs_path => sub { '/dev/nvme0n1' });
$rows = PVE::Storage::SAN::scan_luns($cfg);
@@ -346,6 +352,41 @@ eval { PVE::JSONSchema::validate($rows, $schema) };
is($@, '', 'native row matches the API schema');
is($rows->[0]->{'nvme-subsystem'}, 'nqn.test', 'subsystem NQN exposed');
is($rows->[0]->{'nvme-nsid'}, 1, 'namespace ID exposed');
+is($rows->[0]->{'used-by-storage'}, 'fabric', 'a storage serving the namespaces uses them');
+is($rows->[0]->{'paths-expected'}, 2, 'one expected path per configured portal');
+
+for my $count (1, 3) {
+ $config->mock(
+ read_config => sub { {
+ ids => { 'eui.0123456789abcdef' => { type => 'wwid', 'paths-expected' => $count } },
+ } },
+ );
+ my $entry = PVE::Storage::SAN::scan_luns($cfg);
+ is($entry->[0]->{'paths-expected'}, 2, "a multipath entry does not change it ($count)");
+}
+$config->mock(read_config => sub { { ids => { '3600map' => { type => 'wwid' } } } });
+{
+ local $cfg->{ids}->{fabric}->{content} = { none => 1 };
+ my $plain = PVE::Storage::SAN::scan_luns($cfg);
+ is($plain->[0]->{'used-by-storage'}, undef, 'a connection storage does not');
+ is($plain->[0]->{'paths-expected'}, 2, 'but still tells the expected paths');
+}
+{
+ local $cfg->{ids}->{fabric}->{disable} = 1;
+ my $disabled = PVE::Storage::SAN::scan_luns($cfg);
+ is($disabled->[0]->{'paths-expected'}, undef, 'a disabled storage expects no paths');
+ is($disabled->[0]->{'used-by-storage'}, 'fabric', 'but still uses the namespaces');
+}
+{
+ local $cfg->{ids}->{fabric}->{nodes} = { other => 1 };
+ my $elsewhere = PVE::Storage::SAN::scan_luns($cfg);
+ is($elsewhere->[0]->{'paths-expected'}, undef, 'nor a storage of other nodes');
+}
+for my $base (qw(fabric nvme-fabric)) {
+ $cfg->{ids}->{data}->{base} = "$base:nvme-eui.0123456789abcdef";
+ $rows = PVE::Storage::SAN::scan_luns($cfg);
+ is($rows->[0]->{'used-by-storage'}, 'data', "LVM base consumer takes precedence ($base)");
+}
$disk->mock(is_partition => sub { 1 });
ok(!PVE::Storage::SAN::is_san_device('/dev/sdb1'), 'a partition is never a LUN');
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (5 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
` (10 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The host NQN for the access rules of an array and the controllers of
a node are only visible with nvme-cli on a shell. Add the API module
for /nodes/{node}/san/nvmeof with the host identity and the fabrics
controllers with their subsystem, portal, state and storage. The
namespaces of all subsystems are counted in one pass for that list, as
a node can have a thousand controllers, for example with a storage
plugin that uses a subsystem per guest disk.
Removing a portal or the storage does not disconnect, so a controller
can be removed here, unless a namespace that is mounted, has holders,
is a swap area, is open in a process or is referenced by a storage
would lose its last usable path. A controller that is no usable path
itself, such as one that keeps reconnecting to a deleted subsystem,
only needs the local checks, as the check of the storage references
reads the namespace. A controller with namespaces outside of native
NVMe multipath is refused, as they lack the head devices that these
checks go by. The caller can pass the subsystem and portal it listed,
to refuse a controller number that the kernel reused meanwhile.
Reading needs Sys.Audit on the node or Datastore.Allocate on /storage,
the removal Sys.Modify on '/'.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 199 +++++++++++++++++++++++++++++
src/PVE/API2/SAN/Node.pm | 8 +-
src/PVE/Diskmanage.pm | 18 +--
src/PVE/NVMeOF.pm | 127 +++++++++++++++++--
src/test/Makefile | 3 +-
src/test/run_nvmeof_api_tests.pl | 207 +++++++++++++++++++++++++++++++
src/test/run_nvmeof_tests.pl | 7 ++
src/test/run_san_api_tests.pl | 2 +-
9 files changed, 553 insertions(+), 19 deletions(-)
create mode 100644 src/PVE/API2/NVMeOF.pm
create mode 100755 src/test/run_nvmeof_api_tests.pl
diff --git a/src/PVE/API2/Makefile b/src/PVE/API2/Makefile
index 70ff1ec4..07773db1 100644
--- a/src/PVE/API2/Makefile
+++ b/src/PVE/API2/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Disks.pm ${DESTDIR}${PERLDIR}/PVE/API2/Disks.pm
install -D -m 0644 ISCSI.pm ${DESTDIR}${PERLDIR}/PVE/API2/ISCSI.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/API2/NVMeOF.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/API2/Multipath.pm
make -C Storage install
make -C Disks install
diff --git a/src/PVE/API2/NVMeOF.pm b/src/PVE/API2/NVMeOF.pm
new file mode 100644
index 00000000..a2c657b7
--- /dev/null
+++ b/src/PVE/API2/NVMeOF.pm
@@ -0,0 +1,199 @@
+package PVE::API2::NVMeOF;
+
+use v5.36;
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::RPCEnvironment;
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+use base qw(PVE::RESTHandler);
+
+my $read_permissions = {
+ check => [
+ 'or',
+ ['perm', '/nodes/{node}', ['Sys.Audit']],
+ ['perm', '/storage', ['Datastore.Allocate']],
+ ],
+};
+my $controller_schema = {
+ description => 'The fabrics controller name, for example nvme3.',
+ type => 'string',
+ pattern => 'nvme[0-9]+',
+};
+
+__PACKAGE__->register_method({
+ name => 'index',
+ path => '',
+ method => 'GET',
+ proxyto => 'node',
+ permissions => { user => 'all' },
+ description => 'Node NVMe over Fabrics index.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => { type => 'object', properties => { name => { type => 'string' } } },
+ links => [{ rel => 'child', href => '{name}' }],
+ },
+ code => sub { return [{ name => 'status' }, { name => 'controller' }] },
+});
+
+__PACKAGE__->register_method({
+ name => 'status',
+ path => 'status',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'Report the NVMe host identity and nvme-cli availability.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ supported => { type => 'boolean', description => 'Whether nvme-cli is installed.' },
+ 'host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The host NQN for target access control.',
+ },
+ 'host-id' =>
+ { type => 'string', optional => 1, description => 'The persistent host UUID.' },
+ error => {
+ type => 'string',
+ optional => 1,
+ description => 'Why the host identity is unavailable.',
+ },
+ },
+ },
+ code => sub {
+ my $res = { supported => PVE::NVMeOF::is_supported() };
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ if (my $error = $@) {
+ chomp $error;
+ $res->{error} = $error;
+ } else {
+ $res->{'host-nqn'} = $host->{nqn};
+ $res->{'host-id'} = $host->{id};
+ }
+ return $res;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'controllers',
+ path => 'controller',
+ method => 'GET',
+ protected => 1,
+ proxyto => 'node',
+ permissions => $read_permissions,
+ description => 'List connected NVMe over Fabrics controllers and their states.',
+ parameters =>
+ { additionalProperties => 0, properties => { node => get_standard_option('pve-node') } },
+ returns => {
+ type => 'array',
+ items => {
+ type => 'object',
+ additionalProperties => 0,
+ properties => {
+ controller => $controller_schema,
+ subsysnqn => { type => 'string', description => 'The subsystem NQN.' },
+ transport => { type => 'string', description => 'The controller transport.' },
+ portal => {
+ type => 'string',
+ description => 'The controller address and port, if applicable.',
+ },
+ state => {
+ type => 'string',
+ description => 'The controller state, such as live or connecting.',
+ },
+ namespaces => {
+ type => 'integer',
+ description => 'The number of namespace devices in the subsystem.',
+ },
+ 'used-by-storage' => {
+ type => 'string',
+ optional => 1,
+ description => 'An NVMe over Fabrics storage using the subsystem.',
+ },
+ },
+ },
+ },
+ code => sub {
+ my $cfg = PVE::Storage::config();
+ my %owners;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ $owners{ $scfg->{subsysnqn} } //= $sid if $scfg->{type} eq 'nvmeof';
+ }
+ my $controllers = PVE::NVMeOF::controllers();
+ my $counts = PVE::NVMeOF::namespace_counts();
+ my @rows;
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ my $nqn = $ctrl->{subsystem};
+ push @rows,
+ {
+ controller => $name,
+ subsysnqn => $nqn,
+ transport => $ctrl->{transport},
+ portal => PVE::NVMeOF::format_path($ctrl),
+ state => $ctrl->{state},
+ namespaces => $counts->{$nqn} // 0,
+ defined($owners{$nqn}) ? ('used-by-storage' => $owners{$nqn}) : (),
+ };
+ }
+ return \@rows;
+ },
+});
+
+__PACKAGE__->register_method({
+ name => 'disconnect',
+ path => 'controller/{controller}',
+ method => 'DELETE',
+ protected => 1,
+ proxyto => 'node',
+ permissions => { check => ['perm', '/', ['Sys.Modify']] },
+ description => 'Disconnect one fabrics controller. Refuse when a namespace in use has no'
+ . ' other usable path. A storage still listing the portal reconnects it on activation.',
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ controller => $controller_schema,
+ subsysnqn => {
+ type => 'string',
+ maxLength => 223,
+ optional => 1,
+ description => 'Refuse unless the controller still has this subsystem NQN.',
+ },
+ portal => {
+ type => 'string',
+ maxLength => 128,
+ optional => 1,
+ description => 'Refuse unless the controller still has this portal.',
+ },
+ },
+ },
+ returns => { type => 'string', description => 'The task identifier.' },
+ code => sub($param) {
+ my $rpcenv = PVE::RPCEnvironment::get();
+ return $rpcenv->fork_worker(
+ 'nvmedisconnect',
+ $param->{controller},
+ $rpcenv->get_user(),
+ sub {
+ PVE::NVMeOF::disconnect_guarded(
+ $param->{controller},
+ { subsysnqn => $param->{subsysnqn}, portal => $param->{portal} },
+ );
+ },
+ );
+ },
+});
+
+1;
diff --git a/src/PVE/API2/SAN/Node.pm b/src/PVE/API2/SAN/Node.pm
index 4b96caf5..260c95f7 100644
--- a/src/PVE/API2/SAN/Node.pm
+++ b/src/PVE/API2/SAN/Node.pm
@@ -10,6 +10,7 @@ use PVE::Storage;
use PVE::Storage::SAN;
use PVE::API2::ISCSI;
+use PVE::API2::NVMeOF;
use base qw(PVE::RESTHandler);
@@ -18,6 +19,11 @@ __PACKAGE__->register_method({
path => 'iscsi',
});
+__PACKAGE__->register_method({
+ subclass => "PVE::API2::NVMeOF",
+ path => 'nvmeof',
+});
+
__PACKAGE__->register_method({
name => 'index',
path => '',
@@ -42,7 +48,7 @@ __PACKAGE__->register_method({
links => [{ rel => 'child', href => "{name}" }],
},
code => sub {
- return [map { +{ name => $_ } } qw(iscsi luns)];
+ return [map { +{ name => $_ } } qw(iscsi luns nvmeof)];
},
});
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 21857dc6..523d9c05 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -958,6 +958,15 @@ sub get_multipath_disks {
return $disklist;
}
+# 'change' is transient; paths in 'inaccessible' or 'persistent-loss' cannot serve IO even on a live
+# controller
+sub nvme_path_is_usable {
+ my ($path) = @_;
+ my $ana = $path->{'ana-state'};
+ return $path->{state} eq 'live'
+ && (!defined($ana) || $ana eq 'optimized' || $ana eq 'non-optimized');
+}
+
# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
# single-controller devices lack these children and are omitted.
sub get_nvme_path_status {
@@ -993,15 +1002,8 @@ sub get_nvme_path_status {
my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
$entry->{'ana-state'} = $ana if defined($ana);
- # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
- # cannot serve IO even on a live controller
- my $usable = $state eq 'live'
- && (!defined($ana)
- || $ana eq 'optimized'
- || $ana eq 'non-optimized');
-
my $status = $res->{$head} //= { active => 0, paths => [] };
- $status->{active}++ if $usable;
+ $status->{active}++ if nvme_path_is_usable($entry);
push @{ $status->{paths} }, $entry;
},
);
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
index 15612b04..7961a49e 100644
--- a/src/PVE/NVMeOF.pm
+++ b/src/PVE/NVMeOF.pm
@@ -9,6 +9,7 @@ use JSON qw(decode_json);
use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
use Time::HiRes ();
+use PVE::Cluster;
use PVE::JSONSchema;
use PVE::Network;
use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
@@ -21,6 +22,7 @@ our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
our $HOSTID_FILE = '/etc/nvme/hostid';
our $RUN_DIR = '/run/pve/nvmeof';
our $LOCK_WAIT = 10;
+our $SWAPS = '/proc/swaps';
our $NQN_PATTERN = 'nqn\.[^\s,=]+';
our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery';
@@ -162,26 +164,42 @@ sub path_controller($controllers, $path) {
return undef;
}
-sub namespaces($subsystem) {
- my $res = {};
+# Calls $code with the subsystem NQN and the device name of each namespace of this node, or only
+# of those of the subsystem $only.
+my sub foreach_namespace($code, $only = undef) {
my $dir = "$SYSFS/class/nvme-subsystem";
dir_glob_foreach(
$dir,
'nvme-subsys[0-9]+',
sub($subsys, @) {
my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
- return if $nqn ne $subsystem;
+ return if defined($only) && $nqn ne $only;
dir_glob_foreach(
"$dir/$subsys",
'nvme[0-9]+n[0-9]+',
- sub($dev, @) {
- my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
- my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
- $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
- },
+ sub($dev, @) { $code->($nqn, $dev) },
);
},
);
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ foreach_namespace(
+ sub($nqn, $dev) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+ },
+ $subsystem,
+ );
+ return $res;
+}
+
+# The number of namespaces of each subsystem of this node, from one pass over the subsystems.
+sub namespace_counts() {
+ my $res = {};
+ foreach_namespace(sub($nqn, @) { $res->{$nqn}++ });
return $res;
}
@@ -331,4 +349,97 @@ sub discover($portal) {
die "discovery at portal '$portal' failed - $error";
}
+# A swap area on a namespace has no holder, mount or file descriptor of a process.
+my sub is_swap($device) {
+ my %rdevs = map {
+ my $rdev = (stat($_))[6];
+ defined($rdev) ? ($rdev => 1) : ()
+ } ($device, glob("${device}p[0-9]*"));
+ for my $line (split(/\n/, PVE::Tools::file_get_contents($SWAPS))) {
+ my ($swap) = split(/\s+/, $line);
+ next if !defined($swap) || $swap !~ m!^/!;
+ return 1 if $swap =~ m!^\Q$device\E(?:p[0-9]+)?$!;
+ my $rdev = (stat($swap))[6];
+ return 1 if defined($rdev) && $rdevs{$rdev};
+ }
+ return 0;
+}
+
+# The expected subsystem and portal refuse a controller that replaced the one the caller listed.
+sub disconnect_guarded($controller, $expected = {}) {
+ my ($name) = $controller =~ /\A(nvme[0-9]+)\z/;
+ die "invalid controller name '$controller'\n" if !defined($name);
+ return with_lock(sub {
+ require PVE::Storage;
+ require PVE::Storage::SAN;
+ my $ctrl = controllers()->{$name}
+ // die "'$name' is not an NVMe over Fabrics controller\n";
+ if (
+ (defined($expected->{subsysnqn}) && $expected->{subsysnqn} ne $ctrl->{subsystem})
+ || (defined($expected->{portal}) && $expected->{portal} ne format_path($ctrl))
+ ) {
+ die
+ "controller '$name' is no longer connected to the listed subsystem and portal\n";
+ }
+ # Without native multipath a namespace is a device of its controller and has no head
+ # device, which the checks below go by.
+ my $local = 0;
+ dir_glob_foreach(
+ "$SYSFS/class/nvme-fabrics/ctl/$name",
+ "${name}n[0-9]+",
+ sub { $local = 1 },
+ );
+ die "cannot disconnect '$name' - its namespaces do not use native NVMe multipath\n"
+ if $local;
+ my $namespaces = namespaces($ctrl->{subsystem});
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %heads = map { $_ => 1 } keys %$namespaces;
+ for my $head (keys %$status) {
+ $heads{$head} = 1
+ if grep { $_->{device} eq "/dev/$name" } $status->{$head}->{paths}->@*;
+ }
+ PVE::Cluster::cfs_update(1);
+ my $cfg = PVE::Storage::config();
+ for my $head (sort keys %heads) {
+ my $paths = $status->{$head}->{paths} // [];
+ my @others = grep { $_->{device} ne "/dev/$name" } @$paths;
+ next if grep { PVE::Diskmanage::nvme_path_is_usable($_) } @others;
+ my $ns = $namespaces->{$head};
+ if (!defined($ns)) {
+ die "cannot disconnect '$name' - namespace '$head' has no device on this node"
+ . " to check for use\n";
+ }
+ my $device = "/dev/$head";
+ if (
+ PVE::Diskmanage::is_mounted($device)
+ || PVE::Diskmanage::has_holder($device)
+ || is_swap($device)
+ ) {
+ die "cannot disconnect '$name' - namespace '$head' is in use and has no other"
+ . " usable path\n";
+ }
+ my $opener = PVE::Diskmanage::device_opener($device, glob("${device}p[0-9]*"));
+ if (defined($opener)) {
+ die "cannot disconnect '$name' - process $opener has namespace '$head' open and"
+ . " it has no other usable path\n";
+ }
+ # The check of the storage references reads the namespace. Removing a path that is not
+ # usable either costs no usable path, and a reconnecting one cannot be read.
+ next if !grep {
+ $_->{device} eq "/dev/$name" && PVE::Diskmanage::nvme_path_is_usable($_)
+ } @$paths;
+ eval {
+ PVE::Storage::SAN::assert_lun_unreferenced(
+ $cfg,
+ { devpath => $device, wwid => $ns->{wwid} },
+ );
+ };
+ die "cannot disconnect '$name' - $@" if $@;
+ }
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command([$NVME, 'disconnect', '--device', "/dev/$name"], timeout => 30);
+ return;
+ });
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index ceabd991..d323e8a1 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -117,6 +117,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
-test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_api_tests.pl run_nvmeof_taint_tests.pl
./run_nvmeof_tests.pl
+ ./run_nvmeof_api_tests.pl
perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_api_tests.pl b/src/test/run_nvmeof_api_tests.pl
new file mode 100755
index 00000000..f86291d0
--- /dev/null
+++ b/src/test/run_nvmeof_api_tests.pl
@@ -0,0 +1,207 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::API2::NVMeOF;
+use PVE::Storage::SAN;
+
+my $sys = tempdir('nvmeof-api-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::SYSFS = $sys;
+local $PVE::NVMeOF::SWAPS = "$sys/swaps";
+PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n");
+
+subtest 'last-path disconnect is checked inside serialization' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ my $cluster = Test::MockModule->new('PVE::Cluster');
+ our $locked;
+ my ($commands, $checks, $mounted, $busy, $headless) = (0, 0, 0, 0, 0);
+ my $own = { device => '/dev/nvme1', state => 'live' };
+ my $other = { device => '/dev/nvme2', state => 'live', 'ana-state' => 'optimized' };
+ $nvme->mock(with_lock =>
+ sub { die "NVMe over Fabrics is busy\n" if $busy; local $locked = 1; $_[0]->() });
+ $nvme->mock(
+ controllers => sub {
+ ok($locked, 'controller snapshot is locked');
+ { nvme1 => { subsystem => 'nqn.test' } };
+ },
+ );
+ $nvme->mock(namespaces => sub { $headless ? {} : { nvme0n1 => { wwid => 'eui.abc' } } });
+ $nvme->mock(is_supported => sub { 1 });
+ $disk->mock(
+ get_nvme_path_status => sub { { nvme0n1 => { paths => [$own, $other] } } },
+ );
+ $disk->mock(is_mounted => sub { $mounted });
+ $disk->mock(has_holder => sub { 0 });
+ my $opener;
+ $disk->mock(device_opener => sub { $opener });
+ $cluster->mock(cfs_update => sub { ok($locked, 'configuration refreshed under lock') });
+ $storage->mock(config => sub { { ids => {} } });
+ $san->mock(
+ assert_lun_unreferenced => sub { $checks++; die "storage uses this namespace\n" });
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ ok($locked, 'deletion keeps the lock');
+ is($opts{timeout}, 30, 'deletion is bounded');
+ is_deeply(
+ $cmd,
+ ['/usr/sbin/nvme', 'disconnect', '--device', '/dev/nvme1'],
+ 'only the selected controller is removed',
+ );
+ $commands++;
+ },
+ );
+ for my $ana ('optimized', 'non-optimized') {
+ $other->{'ana-state'} = $ana;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ }
+ is($commands, 2, 'another usable path permits removal');
+ is($checks, 0, 'no usage probe when another path is usable');
+ for my $ana ('inaccessible', 'persistent-loss', 'change') {
+ $other->{'ana-state'} = $ana;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, "$ana does not protect an in-use namespace");
+ }
+ $other = { device => '/dev/nvme2', state => 'connecting', 'ana-state' => 'optimized' };
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/storage uses this namespace/, 'reconnecting is not a usable path');
+ is($commands, 2, 'refusals never delete');
+ $own->{state} = 'connecting';
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 3, 'a controller that is no usable path either needs no storage check');
+ $opener = '4711 (kvm)';
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like(
+ $@,
+ qr/process 4711 \(kvm\) has namespace/,
+ 'but a process that has the namespace open refuses',
+ );
+ $opener = undef;
+ $own->{state} = 'live';
+ $san->mock(assert_lun_unreferenced => sub { $checks++ });
+ $mounted = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'local mount refuses');
+ $mounted = 0;
+ PVE::Tools::file_set_contents(
+ $PVE::NVMeOF::SWAPS,
+ "Filename Type Size Used Priority\n/dev/nvme0n1p2 partition 1024 0 -2\n",
+ );
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is in use/, 'a swap area on the namespace refuses');
+ PVE::Tools::file_set_contents($PVE::NVMeOF::SWAPS, "Filename Type Size Used Priority\n");
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1', { subsysnqn => 'nqn.other' }) };
+ like($@, qr/no longer connected to the listed subsystem/, 'a replaced controller refuses');
+ $opener = '4711 (kvm)';
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/process 4711 \(kvm\) has namespace 'nvme0n1' open/,
+ 'an open namespace refuses');
+ $opener = undef;
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 4, 'unused namespace can lose its final path');
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1",
+ "$sys/class/nvme-fabrics/ctl/nvme1/nvme0c1n1",
+ );
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/do not use native NVMe multipath/, 'a namespace without head device refuses');
+ is($commands, 4, 'and nothing is disconnected');
+ remove_tree("$sys/class/nvme-fabrics/ctl/nvme1/nvme1n1");
+ PVE::NVMeOF::disconnect_guarded('nvme1');
+ is($commands, 5, 'the path device of native multipath does not');
+ $headless = 1;
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/has no device/, 'uncheckable namespace refuses');
+ $busy = 1;
+ $nvme->mock(controllers => sub { die "must not count paths\n" });
+ eval { PVE::NVMeOF::disconnect_guarded('nvme1') };
+ like($@, qr/is busy/, 'ongoing deletion blocks another snapshot');
+ eval { PVE::NVMeOF::disconnect_guarded('../nvme1') };
+ like($@, qr/invalid controller/, 'invalid controller refused before locking');
+};
+
+subtest 'node API contract' => sub {
+ my $api = 'PVE::API2::NVMeOF';
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ $storage->mock(
+ config =>
+ sub { { ids => { fabric => { type => 'nvmeof', subsysnqn => 'nqn.test' } } } },
+ );
+ $nvme->mock(is_supported => sub { 0 });
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.host', id => 'uuid' } });
+ my $status = $api->status({ node => 'n1' });
+ is($status->{supported}, 0, 'availability means nvme-cli, not identity or version');
+ is($status->{'host-nqn'}, 'nqn.host', 'host identity shown independently');
+ $nvme->mock(is_supported => sub { 1 });
+ $nvme->mock(host_identity => sub { die "missing host identity\n" });
+ $status = $api->status({ node => 'n1' });
+ is_deeply(
+ $status,
+ { supported => 1, error => 'missing host identity' },
+ 'identity error does not hide installed tools',
+ );
+ $nvme->mock(
+ controllers => sub { {
+ nvme1 => {
+ subsystem => 'nqn.test',
+ transport => 'tcp',
+ traddr => '192.0.2.1',
+ trsvcid => 4420,
+ state => 'connecting',
+ },
+ } },
+ );
+ $nvme->mock(
+ namespaces => sub { die "must not walk the subsystems for each controller\n" },
+ namespace_counts => sub { { 'nqn.test' => 2, 'nqn.other' => 5 } },
+ );
+ my $rows = $api->controllers({ node => 'n1' });
+ is($rows->[0]->{namespaces}, 2, 'namespaces counted for the subsystem of the controller');
+ is($rows->[0]->{'used-by-storage'}, 'fabric', 'controller attributed');
+ is($rows->[0]->{state}, 'connecting', 'kernel state remains visible');
+
+ for my $pair (['status', $status], ['controllers', $rows]) {
+ my $method = $api->map_method_by_name($pair->[0]);
+ eval { PVE::JSONSchema::validate($pair->[1], $method->{returns}) };
+ is($@, '', "$pair->[0] schema");
+ ok($method->{protected} && $method->{proxyto} eq 'node', 'protected and node-proxied');
+ }
+ is(
+ $api->map_method_by_name('disconnect')->{path},
+ 'controller/{controller}',
+ 'REST-shaped deletion',
+ );
+ eval {
+ PVE::JSONSchema::validate(
+ { node => 'n1', controller => 'nvme1', subsysnqn => 'testsubsys' },
+ $api->map_method_by_name('disconnect')->{parameters},
+ );
+ };
+ is($@, '', 'the listed subsystem name is taken as is, also when it is no NQN');
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ {
+
+ package NVMeTestRPC;
+ sub get_user { 'root@pam' }
+ sub fork_worker { $_[4]->(); return 'UPID:test' }
+ }
+ $rpc->mock(get => sub { bless {}, 'NVMeTestRPC' });
+ my $removed;
+ $nvme->mock(disconnect_guarded => sub { $removed = $_[0] });
+ is(
+ $api->disconnect({ node => 'n1', controller => 'nvme1' }),
+ 'UPID:test',
+ 'API returns task identifier',
+ );
+ is($removed, 'nvme1', 'task calls guarded disconnect');
+};
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
index ddd905ee..ac029207 100755
--- a/src/test/run_nvmeof_tests.pl
+++ b/src/test/run_nvmeof_tests.pl
@@ -212,6 +212,13 @@ subtest 'sysfs topology' => sub {
'namespace facts',
);
is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+ make_path(map { "$sys/class/nvme-subsystem/nvme-subsys1/nvme1n$_" } 1 .. 3);
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys1/subsysnqn", "nqn.other\n");
+ is_deeply(
+ PVE::NVMeOF::namespace_counts(),
+ { 'nqn.test' => 1, 'nqn.other' => 3 },
+ 'namespaces counted per subsystem',
+ );
};
subtest 'serialized commands inherit the lock' => sub {
diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl
index 73938a7e..2ae7b2b4 100755
--- a/src/test/run_san_api_tests.pl
+++ b/src/test/run_san_api_tests.pl
@@ -109,7 +109,7 @@ is_deeply(
);
is_deeply(
PVE::API2::SAN::Node->index({ node => 'test' }),
- [{ name => 'iscsi' }, { name => 'luns' }],
+ [{ name => 'iscsi' }, { name => 'luns' }, { name => 'nvmeof' }],
'node SAN index',
);
for my $case (
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (6 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
` (9 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The status report of a node has its iSCSI initiator name, so that
the cluster-wide status shows the identities an admin needs for the
access rules of an array in one place. Add the NVMe host NQN, which
an NVMe over Fabrics target needs for the same purpose. A node
without a valid NVMe host identity reports none.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/SAN/Cluster.pm | 5 +++++
src/PVE/Multipath/Status.pm | 5 ++++-
src/test/run_multipath_api_tests.pl | 12 +++++++++++-
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/src/PVE/API2/SAN/Cluster.pm b/src/PVE/API2/SAN/Cluster.pm
index 54cb0dbc..ac372d65 100644
--- a/src/PVE/API2/SAN/Cluster.pm
+++ b/src/PVE/API2/SAN/Cluster.pm
@@ -124,6 +124,11 @@ my $node_schema = {
optional => 1,
description => 'The iSCSI initiator name of the node.',
},
+ 'nvme-host-nqn' => {
+ type => 'string',
+ optional => 1,
+ description => 'The NVMe host NQN of the node.',
+ },
},
};
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index e7ffb41c..1b85e108 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -65,6 +65,9 @@ sub update_node_status() {
my $initiator = eval { PVE::Storage::ISCSIPlugin::read_initiator_info() };
$report->{'iscsi-initiator-name'} = $initiator->{name}
if $initiator && defined($initiator->{name});
+ require PVE::NVMeOF;
+ my $host = eval { PVE::NVMeOF::host_identity() };
+ $report->{'nvme-host-nqn'} = $host->{nqn} if $host;
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running or does not respond\n" if !PVE::Multipath::is_running();
@@ -199,7 +202,7 @@ sub cluster_status() {
$entry->{time} = $report->{time} if defined($report->{time});
my $offline = exists($members->{$node}) && !$members->{$node}->{online};
next if $offline || !defined($report->{time}) || time() - $report->{time} > 180;
- my @keys = qw(error apply-error iscsi-initiator-name other-maps);
+ my @keys = qw(error apply-error iscsi-initiator-name nvme-host-nqn other-maps);
$entry->{$_} = $report->{$_} for grep { defined($report->{$_}) } @keys;
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
$unhinted{$node} = 1 if $report->{'vgs-unknown'};
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 60683ad8..a91332df 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -15,6 +15,7 @@ use Test::MockModule;
use PVE::API2::Multipath;
use PVE::API2::SAN::Cluster;
use PVE::Multipath::Generator;
+use PVE::NVMeOF;
use PVE::Storage;
subtest 'add, update and remove an explicit entry' => sub {
@@ -457,9 +458,12 @@ subtest 'an unconfigured node does not write drop-ins or reload' => sub {
}
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
$iscsi->mock(read_initiator_info => sub { {} });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $report;
$status->mock(publish => sub { $report = $_[0] });
+
for my $ids ({}, { plain => { type => 'iscsi', target => 'iqn.test' } }) {
$storage->mock(config => sub { { ids => $ids } });
PVE::Multipath::Status::update_node_status();
@@ -686,15 +690,19 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(host_identity => sub { { nqn => 'nqn.2025-01.example:node', id => 'unused' } });
$mp->mock(is_supported => sub { 0 });
PVE::Multipath::Status::update_node_status();
is($report->{error}, 'multipath-tools is not installed', 'missing tools reason only');
+ is($report->{'nvme-host-nqn'}, 'nqn.2025-01.example:node', 'host NQN reported in any case');
is_deeply(
\@order,
[qw(apply publish)],
'the configuration is applied before the status is published',
);
is($report->{'apply-error'}, 'reload failed', 'apply failure is separate');
+ $nvme->mock(host_identity => sub { die "no host identity\n" });
$mp->mock(is_supported => sub { 1 });
$mp->mock(is_running => sub { 0 });
PVE::Multipath::Status::update_node_status();
@@ -703,6 +711,7 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
'multipathd is not running or does not respond',
'stopped daemon reason only',
);
+ ok(!exists($report->{'nvme-host-nqn'}), 'no host NQN without a valid host identity');
$mp->mock(is_running => sub { 1 });
$mp->mock(get_maps => sub { die "bad JSON\n" });
PVE::Multipath::Status::update_node_status();
@@ -757,7 +766,7 @@ subtest 'API returns final node and LUN states' => sub {
},
],
}),
- n2 => encode_report({ time => $NOW, luns => [] }),
+ n2 => encode_report({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_report({ time => $NOW - 181, luns => [] }),
n4 => encode_report({ time => $NOW, luns => [] }),
n5 => encode_report({ time => $NOW, error => 'multipath-tools is not installed' }),
@@ -788,6 +797,7 @@ subtest 'API returns final node and LUN states' => sub {
'final node states',
);
is($res->{nodes}->{n3}->{time}, $NOW - 181, 'old report time retained');
+ is($res->{nodes}->{n2}->{'nvme-host-nqn'}, 'nqn.test:n2', 'host NQN passed on');
my $lun = $res->{luns}->[0];
is($lun->{state}, 'missing', 'worst known state');
is($lun->{nodes}->{n1}->{state}, 'optimal', 'observed healthy map');
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (7 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
` (8 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the LUNs that multipathd handles on
each node. The namespaces of an NVMe over Fabrics storage use the
native multipath of the kernel and were missing from it, so a lost
path or a node without access to a namespace only showed in the LUN
list of that node.
Let each node report the namespaces of the subsystems such a storage
names, with the path counts its LUN list already shows, and expect
them on the nodes of the storage like the LUNs of an iSCSI storage.
They do not depend on multipathd, so a node without it reports them
as well and only the state of its maps stays unknown. A node that
cannot list its namespaces marks its report, so that they show as
unknown there instead of missing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/SAN/Cluster.pm | 17 +++-
src/PVE/Multipath/Status.pm | 67 ++++++++++++++--
src/test/run_multipath_api_tests.pl | 120 +++++++++++++++++++++++++++-
3 files changed, 191 insertions(+), 13 deletions(-)
diff --git a/src/PVE/API2/SAN/Cluster.pm b/src/PVE/API2/SAN/Cluster.pm
index ac372d65..4d7f0011 100644
--- a/src/PVE/API2/SAN/Cluster.pm
+++ b/src/PVE/API2/SAN/Cluster.pm
@@ -63,7 +63,8 @@ my $lun_node_schema = {
minimum => 0,
optional => 1,
description => 'The number of active paths of the map whose checker state is ready,'
- . ' ghost, or not reported.',
+ . ' ghost, or not reported. For an NVMe namespace, the number of its paths over a'
+ . ' live controller whose ANA state is optimized, non-optimized, or not reported.',
},
'paths-total' => {
type => 'integer',
@@ -77,7 +78,8 @@ my $lun_node_schema = {
optional => 1,
description => 'The number of paths the LUN should have on the node: the highest of'
. ' the count of its multipath entry and, for each managed iSCSI storage of the'
- . ' LUN, the number of its portals.',
+ . ' LUN, the number of its portals. For an NVMe namespace, the number of portals'
+ . ' of its NVMe over Fabrics storage.',
},
},
};
@@ -165,6 +167,12 @@ __PACKAGE__->register_method({
wwid => { type => 'string', description => 'The LUN WWID.' },
explicit =>
{ type => 'boolean', description => 'An explicit entry exists.' },
+ native => {
+ type => 'boolean',
+ optional => 1,
+ description => 'An NVMe namespace, which uses the native'
+ . ' multipath of the kernel.',
+ },
name => {
type => 'string',
optional => 1,
@@ -177,8 +185,9 @@ __PACKAGE__->register_method({
optional => 1,
description =>
'The storage that uses the LUN: as its base volume or through'
- . ' the volume group on it, or as an iSCSI storage that serves'
- . ' the LUNs of its target as images.',
+ . ' the volume group on it, or as an iSCSI or NVMe over Fabrics'
+ . ' storage that serves the LUNs of its target or subsystem as'
+ . ' images.',
},
'multipath-storage' => {
type => 'string',
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index 1b85e108..baae32a6 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -9,6 +9,7 @@ use MIME::Base64 qw(decode_base64 encode_base64);
use PVE::Cluster;
use PVE::INotify;
+use PVE::Diskmanage;
use PVE::Multipath;
use PVE::Multipath::Config;
use PVE::Multipath::Generator;
@@ -37,6 +38,47 @@ sub worst_state(@states) {
return $worst;
}
+# Namespaces use the native multipath of the kernel, so their state does not depend on multipathd.
+my sub namespace_luns($pv_vgs) {
+ require PVE::Storage;
+ my $cfg = PVE::Storage::config();
+ my $node = PVE::INotify::nodename();
+ my @storages = grep {
+ my $scfg = $cfg->{ids}->{$_};
+ $scfg->{type} eq 'nvmeof'
+ && !$scfg->{disable}
+ && (!$scfg->{nodes} || $scfg->{nodes}->{$node})
+ } sort keys $cfg->{ids}->%*;
+ # reading the paths walks all NVMe namespaces of the node, also those of other plugins
+ return [] if !@storages;
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %luns;
+ for my $sid (@storages) {
+ my $scfg = $cfg->{ids}->{$sid};
+ my $expected = eval { scalar(PVE::NVMeOF::storage_paths($scfg)->@*) };
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ for my $name (sort keys %$namespaces) {
+ my $wwid = $namespaces->{$name}->{wwid};
+ next if $luns{$wwid};
+ my $paths = $status->{$name} // { active => 0, paths => [] };
+ my $total = scalar($paths->{paths}->@*);
+ my $lun = $luns{$wwid} = {
+ wwid => $wwid,
+ native => 1,
+ size => $namespaces->{$name}->{size},
+ storages => [$sid],
+ 'paths-total' => $total,
+ 'paths-active' => $paths->{active},
+ state => PVE::Multipath::derive_health($paths->{active}, $total, $expected),
+ };
+ $lun->{'paths-expected'} = $expected if defined($expected);
+ my $vg = ($pv_vgs // {})->{"/dev/$name"};
+ $lun->{vgname} = $vg if defined($vg);
+ }
+ }
+ return [map { $luns{$_} } sort keys %luns];
+}
+
sub publish($report) {
my $node = PVE::INotify::nodename();
my $now = time();
@@ -68,11 +110,11 @@ sub update_node_status() {
require PVE::NVMeOF;
my $host = eval { PVE::NVMeOF::host_identity() };
$report->{'nvme-host-nqn'} = $host->{nqn} if $host;
+ require PVE::Storage::LVMPlugin;
+ my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running or does not respond\n" if !PVE::Multipath::is_running();
- require PVE::Storage::LVMPlugin;
- my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
# A node can have many maps that something else set up, for example a storage plugin with
# one LUN per guest disk. Only the maps that the node manages, that are the base volume of a
# storage or hold its volume group get a row, the others are counted.
@@ -134,8 +176,15 @@ sub update_node_status() {
if (my $error = $@) {
chomp $error;
$report->{error} = $error;
- delete $report->{luns};
+ $report->{luns} = [];
}
+ my $namespaces = eval { namespace_luns($pv_vgs) };
+ if (my $error = $@) {
+ warn "multipath: cannot query NVMe namespaces - $error";
+ # a consumer must not take the namespaces as gone
+ $report->{'namespaces-unknown'} = 1;
+ }
+ push $report->{luns}->@*, @$namespaces if $namespaces;
eval { publish($report) };
warn "multipath: cannot publish node status - $@" if $@;
return $report;
@@ -152,7 +201,7 @@ sub cluster_status() {
my %reports = map { $_ => decode_report($raw->{$_}) // {} } @$all_nodes;
my %reported = map { $_->{wwid} => 1 }
grep { defined($_->{wwid}) } map { ($_->{luns} // [])->@* } values %reports;
- my (%nodes, %luns, %expected, %observed, %vg_owner, %disabled_owner);
+ my (%nodes, %luns, %expected, %observed, %vg_owner, %lists, %disabled_owner);
# the nodes that list a map because the configuration names it for them
my (%managed, %unhinted);
my $storage_nodes = sub($scfg) {
@@ -178,7 +227,6 @@ sub cluster_status() {
$vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
next if $scfg->{type} ne 'lvm';
my $base = PVE::Multipath::parse_base_volume($scfg->{base}) // next;
- next if $base->{kind} eq 'nvme'; # a namespace, which has no map
# A base volume by its SCSI name is a map only if its iSCSI storage requests maps. The name
# can also be another by-id link of the LUN, so only a WWID that a node reports counts.
my $map = $requests_maps->($base->{storage}) && $reported{ $base->{wwid} };
@@ -190,6 +238,7 @@ sub cluster_status() {
next;
}
$luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $luns{$wwid}->{native} = 1 if $base->{kind} eq 'nvme';
$luns{$wwid}->{'used-by-storage'} //= $sid;
$managed{$wwid}->{$_} = 1 for @$all_nodes;
$luns{$wwid}->{'multipath-storage'} //= $base->{storage}
@@ -206,11 +255,13 @@ sub cluster_status() {
$entry->{$_} = $report->{$_} for grep { defined($report->{$_}) } @keys;
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
$unhinted{$node} = 1 if $report->{'vgs-unknown'};
- next if $entry->{state} ne 'ok';
+ # the node lists its namespaces, also if it is unavailable for its maps
+ $lists{$node} = 1 if $report->{luns} && !$report->{'namespaces-unknown'};
for my $lun (($report->{luns} // [])->@*) {
my $wwid = $lun->{wwid} // next;
my $row = $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $row->{native} = 1 if $lun->{native};
$row->{$_} //= $lun->{$_} for grep { defined($lun->{$_}) } qw(name size);
my $owner = defined($lun->{vgname}) ? $vg_owner{ $lun->{vgname} } : undef;
$row->{'used-by-storage'} //= $owner if defined($owner);
@@ -244,10 +295,10 @@ sub cluster_status() {
if ($observed{$wwid}->{$node}) {
$row->{nodes}->{$node} = $observed{$wwid}->{$node};
} elsif ($expected{$wwid}->{$node}) {
- my $listed = $nodes{$node}->{state} eq 'ok';
+ my $listed = $row->{native} ? $lists{$node} : $nodes{$node}->{state} eq 'ok';
# without the hint file of LVM a node leaves a map out that it only knows through
# the volume group of a storage
- $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node};
+ $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node} && !$row->{native};
$row->{nodes}->{$node} = { state => $listed ? 'missing' : 'unknown' };
}
}
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index a91332df..85ddce7b 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -686,7 +686,12 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $mp = Test::MockModule->new('PVE::Multipath');
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $disks = Test::MockModule->new('PVE::Diskmanage');
my ($report, @order);
+ my $ids = {};
+ $storage->mock(config => sub { { ids => $ids } });
+ $disks->mock(get_nvme_path_status => sub { { nvme1n1 => { active => 1, paths => [{}] } } });
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
@@ -728,6 +733,58 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
is($report->{luns}->[0]->{'paths-expected'}, 3, 'expected count reported');
is($report->{luns}->[0]->{state}, 'degraded', 'fewer paths than expected');
ok(!exists($report->{supported}), 'the report has no separate "supported" field');
+
+ $ids->{nvme} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test:sub',
+ portals => '192.0.2.1,192.0.2.2',
+ };
+ $nvme->mock(
+ namespaces => sub($nqn) {
+ return $nqn eq 'nqn.test:sub'
+ ? { nvme1n1 => { wwid => 'uuid.1', size => 4096 } }
+ : {};
+ },
+ );
+ my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+ $lvm->mock(cached_pv_volume_groups => sub { { '/dev/nvme1n1' => 'vgns' } });
+ my $namespace = {
+ wwid => 'uuid.1',
+ native => 1,
+ size => 4096,
+ storages => ['nvme'],
+ vgname => 'vgns',
+ 'paths-total' => 1,
+ 'paths-active' => 1,
+ 'paths-expected' => 2,
+ state => 'degraded',
+ };
+ PVE::Multipath::Status::update_node_status();
+ is_deeply($report->{luns}->[1], $namespace, 'namespace with fewer paths than portals');
+ $mp->mock(is_running => sub { 0 });
+ PVE::Multipath::Status::update_node_status();
+ is(
+ $report->{error},
+ 'multipathd is not running or does not respond',
+ 'stopped daemon still reported',
+ );
+ is_deeply($report->{luns}, [$namespace], 'namespaces do not need multipathd');
+ ok(!$report->{'namespaces-unknown'}, 'which the node could list');
+ {
+ local $ids->{nvme}->{disable} = 1;
+ $disks->mock(get_nvme_path_status => sub { die "walked the NVMe paths\n" });
+ PVE::Multipath::Status::update_node_status();
+ is_deeply($report->{luns}, [], 'a disabled storage reports no namespaces');
+ ok(!$report->{'namespaces-unknown'}, 'and reads no NVMe paths for them');
+ $disks->mock(
+ get_nvme_path_status => sub { { nvme1n1 => { active => 1, paths => [{}] } } });
+ }
+ $nvme->mock(namespaces => sub { die "sysfs gone\n" });
+ {
+ local $SIG{__WARN__} = sub { };
+ PVE::Multipath::Status::update_node_status();
+ }
+ ok($report->{'namespaces-unknown'}, 'a failed namespace query marks the report');
};
subtest 'API returns final node and LUN states' => sub {
@@ -764,12 +821,31 @@ subtest 'API returns final node and LUN states' => sub {
'paths-active' => 2,
storages => ['scoped'],
},
+ {
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'optimal',
+ 'paths-total' => 2,
+ 'paths-active' => 2,
+ storages => ['nvme'],
+ },
],
}),
n2 => encode_report({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_report({ time => $NOW - 181, luns => [] }),
n4 => encode_report({ time => $NOW, luns => [] }),
- n5 => encode_report({ time => $NOW, error => 'multipath-tools is not installed' }),
+ n5 => encode_report({
+ time => $NOW,
+ error => 'multipath-tools is not installed',
+ luns => [{
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'degraded',
+ 'paths-total' => 2,
+ 'paths-active' => 1,
+ storages => ['nvme'],
+ }],
+ }),
);
$cluster->mock(get_node_kv => sub { \%raw });
my $entries = { '3600a' => { type => 'wwid' } };
@@ -787,6 +863,8 @@ subtest 'API returns final node and LUN states' => sub {
{ type => 'iscsi', 'managed-multipath' => 1, nodes => { departed => 1 } },
lvmgone => { type => 'lvm', base => 'gone:0.0.0.dm-uuid-mpath-3600e' },
scoped => { type => 'iscsi', 'managed-multipath' => 1, nodes => { n2 => 1 } },
+ nvme => { type => 'nvmeof', subsysnqn => 'nqn.test:sub' },
+ lvmns => { type => 'lvm', base => 'nvme:nvme-uuid.2', nodes => { n5 => 1 } },
},
} },
);
@@ -848,6 +926,46 @@ subtest 'API returns final node and LUN states' => sub {
'an entry for other nodes does not change that',
);
}
+ my $namespace = $lun{'uuid.1'};
+ ok($namespace->{native}, 'namespace marked as native');
+ ok(!exists($namespace->{'multipath-storage'}), 'a namespace has no requested map');
+ is_deeply(
+ { map { $_ => $namespace->{nodes}->{$_}->{state} } keys $namespace->{nodes}->%* },
+ {
+ n1 => 'optimal',
+ n2 => 'missing',
+ n3 => 'unknown',
+ n4 => 'unknown',
+ n5 => 'degraded',
+ },
+ 'namespace state of a node without multipathd counts',
+ );
+ is($lun->{nodes}->{n5}->{state}, 'unknown', 'no map state from a node without multipathd');
+ is_deeply(
+ $lun{'uuid.2'},
+ {
+ wwid => 'uuid.2',
+ explicit => 0,
+ native => 1,
+ 'used-by-storage' => 'lvmns',
+ state => 'missing',
+ nodes => { n5 => { state => 'missing' } },
+ },
+ 'base namespace that a node without multipathd does not report',
+ );
+ $raw{n5} = encode_report({ time => $NOW, error => 'too many LUNs to report' });
+ $res = PVE::API2::SAN::Cluster->status({});
+ %lun = map { $_->{wwid} => $_ } $res->{luns}->@*;
+ is($lun{'uuid.2'}->{nodes}->{n5}->{state}, 'unknown', 'node without a list');
+ {
+ local $raw{n2} = encode_report({ time => $NOW, luns => [], 'namespaces-unknown' => 1 });
+ my %unlisted = map { $_->{wwid} => $_ } PVE::API2::SAN::Cluster->status({})->{luns}->@*;
+ is(
+ $unlisted{'uuid.1'}->{nodes}->{n2}->{state},
+ 'unknown',
+ 'a node that could not query its namespaces does not miss them',
+ );
+ }
eval {
PVE::JSONSchema::validate(
$res,
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (8 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
` (7 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of the
storage type. The introduction lists it among the network storages, and
the LVM chapter names it next to iSCSI as base storage.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
pve-gui.adoc | 12 ++-
pve-intro.adoc | 3 +-
pve-storage-lvm.adoc | 10 +-
pve-storage-multipath.adoc | 3 +-
pve-storage-nvmeof.adoc | 189 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 24 +++--
6 files changed, 222 insertions(+), 19 deletions(-)
create mode 100644 pve-storage-nvmeof.adoc
diff --git a/pve-gui.adoc b/pve-gui.adoc
index 9a1f70b..a514eca 100644
--- a/pve-gui.adoc
+++ b/pve-gui.adoc
@@ -233,10 +233,11 @@ On the datacenter level, you can access cluster-wide settings and information.
* *Options:* view and manage cluster-wide default settings.
* *Storage:* provides an interface for managing cluster storage. The *Add* menu
- offers guided setups for shared LVM on iSCSI and Fibre Channel LUNs.
+ offers guided setups for shared LVM on iSCSI and Fibre Channel LUNs and on
+ NVMe over Fabrics namespaces.
-* *SAN:* shows the state of the SAN LUNs on all nodes and manages the multipath
- configuration.
+* *SAN:* shows the state of the SAN LUNs and NVMe namespaces on all nodes and
+ manages the multipath configuration.
* *Backup:* schedule backup jobs. This operates cluster wide, so it doesn't
matter where the VMs/containers are on your cluster when scheduling.
@@ -288,8 +289,9 @@ The top header has useful buttons such as 'Reboot', 'Shutdown', 'Shell',
* *Disks:* get an overview of the attached disks, and manage how they are used.
-* *SAN:* get an overview of the SAN LUNs of the node, with their paths and
- usage. Below it, *iSCSI* shows the iSCSI targets and sessions.
+* *SAN:* get an overview of the SAN LUNs and NVMe namespaces of the node, with
+ their paths and usage. Below it, *iSCSI* shows the iSCSI targets and sessions,
+ and *NVMe over Fabrics* the connected controllers.
* *Ceph:* is only used if you have installed a Ceph server on your
host. In this case, you can manage your Ceph cluster and see the status
diff --git a/pve-intro.adoc b/pve-intro.adoc
index c2a0b6a..baec59f 100644
--- a/pve-intro.adoc
+++ b/pve-intro.adoc
@@ -103,9 +103,10 @@ the cluster have direct access to VM disk images.
The following network storage types are supported:
-* LVM group (network backing with iSCSI targets)
+* LVM group (network backing with iSCSI targets or NVMe over Fabrics)
* iSCSI target
* iSCSI LUNs (direct use)
+* NVMe over Fabrics subsystem
* NFS share
* CIFS share
* Ceph RBD
diff --git a/pve-storage-lvm.adoc b/pve-storage-lvm.adoc
index 92019fc..8b9564c 100644
--- a/pve-storage-lvm.adoc
+++ b/pve-storage-lvm.adoc
@@ -34,7 +34,8 @@ LVM volume group name. This must point to an existing volume group.
Base volume. This volume is automatically activated before accessing
the storage. This is mostly useful when the LVM volume group resides
-on a remote iSCSI server.
+on a remote iSCSI server or on an xref:storage_nvmeof[NVMe over Fabrics]
+namespace.
`saferemove`::
@@ -143,9 +144,10 @@ A benefit of LVM is that it can be used with shared storage.
For example, an iSCSI LUN. The backend implements proper cluster-wide locking if
the storage is marked as shared in the configuration.
-TIP: *Shared LVM on iSCSI* or *Shared LVM on Fibre Channel* in *Datacenter ->
-Storage -> Add* in the web UI creates a shared LVM storage on a SAN LUN, see
-xref:storage_multipath_setup[Multipath].
+TIP: *Shared LVM on iSCSI*, *Shared LVM on Fibre Channel* or *Shared LVM on NVMe
+over Fabrics* in *Datacenter -> Storage -> Add* in the web UI creates a shared
+LVM storage on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace].
NOTE: If you extend a shared volume group onto another device on one node, the
other nodes may only see the added physical volume after LVM has scanned their
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index 3e07eba..45ab3a8 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -15,7 +15,8 @@ Fibre Channel.
{pve} sets up the maps on all nodes and shows their state under *Datacenter ->
SAN*. An iSCSI storage with *Multipath* enabled requests the maps of its LUNs, a
Fibre Channel LUN needs a multipath entry. NVMe namespaces use the multipath
-support of the kernel instead and are not covered here.
+support of the kernel instead, see xref:storage_nvmeof[NVMe over Fabrics], and
+are not covered here.
Prerequisites
~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..2a5aaa6
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,189 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage, also
+ nodes that will not use it. A node without support for this storage type
+ removes such storages from the cluster-wide storage configuration when it
+ writes that configuration, for example when a storage is added or edited on
+ that node.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+ package recommends it, but installations without recommended packages need it
+ added separately. Its installation creates the node's host NQN in
+ `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+ image share them. Give each a new identity before it connects, with
+ `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+ *Datacenter -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+ addresses that every node can reach; host names are not accepted.
+. Use a storage network that only trusted hosts can reach. The connections use
+ neither authentication nor TLS.
+. Keep native NVMe multipath enabled, which is the kernel default
+ (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. On a node that can reach the array, select *Shared LVM on NVMe over Fabrics*
+ under *Datacenter -> Storage -> Add*. The *Add Shared LVM* menu of
+ *Datacenter -> SAN* has the same setup.
+. Enter the *ID* of the new storage, a discovery portal and the subsystem, and
+ check the portals. If no NVMe over Fabrics storage connects the nodes to that
+ subsystem yet, the setup adds one named after the ID with a `-base` suffix, or
+ as set under *Advanced*. This storage connects to the subsystem with `content
+ none`, while the LVM storage manages guest disks. Clear the nodes in the list
+ of host NQNs that should not use the storages. Continuing creates or reuses
+ the NVMe over Fabrics storage before you select a namespace.
+. Select the namespace after checking that no host uses it and that no data on
+ it must be kept. The volume group is named after the ID unless set under
+ *Advanced*.
+. Check the state of the namespace on each node of the storage, which the
+ setup shows once the storage is added, as described for LUNs in
+ xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+ check its access to the namespace on the array before placing guests on the
+ storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace shows `1/1`, which has no path
+redundancy.
+Portals can also be changed later in the edit dialog of the NVMe over Fabrics
+storage; each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the NVMe over
+Fabrics storage. Without a discovery portal, enter the subsystem NQN and its
+portals as they are. Enable *Use namespaces directly* there to use whole
+namespaces as guest disks, or add an *LVM* storage on a namespace later, with
+the NVMe over Fabrics storage as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <nvmeof-storage> --subsysnqn <nqn> --portals <portals> \
+ --content none
+# pvesm list <nvmeof-storage>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<nvmeof-storage>:nvme-<wwid>`.
+Adding the LVM storage creates the volume group if it does not exist yet.
+Creating it is refused if the namespace holds data that {pve} recognizes, such
+as partitions, a file system or a volume group, or if another storage references
+it. To reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later, and only one storage
+can use a subsystem.
+
+portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace have been lost for
+that long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+ subsysnqn nqn.2010-06.com.example:array1
+ portals 192.0.2.10,192.0.2.11
+ content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types |Image formats |Shared |Snapshots |Clones
+|images none |raw |yes |no |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace under *Node -> SAN* to inspect its paths.
+Its *State* counts the usable paths out of the portals of the storage, for
+example `1/2` while one of two paths is unusable, and shows *Failed* without a
+usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal,
+with their *State*: *Connected*, or *Reconnecting* while the kernel retries a
+lost path. A node sends a xref:notification_events[multipath notification] when
+a namespace loses or regains paths, as for a multipath map. A node that cannot
+query `multipathd` sends no such notifications, also not for its namespaces,
+although *Datacenter -> SAN* still shows their state.
+
+Each node of an enabled storage connects to the portals that it is not yet
+connected to. A portal that failed is retried after a minute. Removing a portal,
+or disabling or removing the storage, does not disconnect anything. To remove
+such a path, select its controller under *Node -> SAN -> NVMe over Fabrics* on
+each node and press *Disconnect*, or run
+`pvesh delete /nodes/<node>/san/nvmeof/controller/<controller>`.
+This is refused if a namespace that is mounted, in use by another device or as
+swap, open in a process such as a running guest, or referenced by a storage
+would lose its last usable path. For a controller that is no usable path itself,
+such as one that keeps reconnecting to a deleted subsystem, only the local uses
+count, not a storage reference. The process check does not find a process that
+opened the namespace through another device node, for example one in a
+container. A controller with namespaces that do not use native NVMe multipath is
+refused as well. A storage that still lists the portal connects to it again.
+Namespaces of a subsystem that no storage uses then show fewer paths, but none
+as missing, as nothing tells how many paths they should have.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS.
+* NVMe/FC namespaces need no storage of this type, and the guided setups and
+ the cluster-wide status do not cover them. *Node -> SAN* lists them. Create
+ their volume group under *Node -> Disks -> LVM*; with *Add Storage* enabled,
+ this adds a shared LVM storage.
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 8f1b24e..0bea1d2 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
|LVM-thin |lvmthin |block |no |yes |yes
|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes
|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes
+|NVMe-oF |nvmeof |block |yes^3^|yes^4^ |yes
|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes
|Ceph/RBD |rbd |block |yes |yes |yes
|ZFS over iSCSI |zfs |block |yes |yes |yes
@@ -96,11 +97,13 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
hours. If your setup is affected, create and remove snapshots while the VM is
shut down, expecting a long task duration.
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster. FC and SAS LUNs need a
-xref:storage_multipath_entries[multipath entry].
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+FC and SAS LUNs need a xref:storage_multipath_entries[multipath entry].
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
These snapshots use separate volumes for the snapshot data and layer them. For
@@ -154,9 +157,10 @@ is available on all nodes, but it is physically different and can have
totally different content.
Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use *Shared LVM on iSCSI* or
-*Shared LVM on Fibre Channel* there instead. It creates the storages that are
-needed and enables multipath for the LUN.
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use *Shared LVM on iSCSI*, *Shared
+LVM on Fibre Channel* or *Shared LVM on NVMe over Fabrics* there instead. It
+creates the storages that are needed and enables multipath for a LUN.
Storage Pools
@@ -507,6 +511,8 @@ See Also
* link:/wiki/Storage:_Multipath[Storage: Multipath]
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
* link:/wiki/Storage:_LVM[Storage: LVM]
* link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -551,6 +557,8 @@ include::pve-storage-iscsidirect.adoc[]
include::pve-storage-multipath.adoc[]
+include::pve-storage-nvmeof.adoc[]
+
include::pve-storage-rbd.adoc[]
include::pve-storage-cephfs.adoc[]
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (9 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
` (6 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The report shows the iSCSI sessions and the multipath topology of a
node, but nothing about NVMe over Fabrics. Add the host NQN and host
ID, which the array has to allow and which cloned nodes can share, and
the subsystems with the state of each controller.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/Report.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/PVE/Report.pm b/PVE/Report.pm
index 9023f968a..2d88f2106 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -255,6 +255,12 @@ my $init_report_cmds = sub {
;
}
+ if (cmd_exists('nvme')) {
+ push @{ $report_def->{'block devices'}->{cmds} }, 'cat /etc/nvme/hostnqn',
+ # nodes cloned from one image can share it, which the array cannot tell apart
+ 'cat /etc/nvme/hostid', 'nvme list-subsys',;
+ }
+
return $report_def;
};
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (10 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
` (5 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Show NVMe over Fabrics storages with their subsystem in the storage
list, and add a dialog for them. The dialog lists the subsystems that a
discovery portal reports and fills in the portals of the selected one,
which can be changed. Without a discovery portal, the subsystem and its
portals can be entered as they are. As in the iSCSI dialog, the
discovery can run on another node, since the node serving the web UI
may not be allowed on the array. The node of the discovery is then
preselected as the node of the storage.
The subsystem selector builds on the iSCSI target selector, which now
lets a subclass set its store fields and sort key.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/Utils.js | 6 ++
www/manager6/dc/StorageView.js | 14 ++-
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/NVMeOFEdit.js | 150 +++++++++++++++++++++++++++++
5 files changed, 172 insertions(+), 4 deletions(-)
create mode 100644 www/manager6/storage/NVMeOFEdit.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 1ee82e9a9..dddf17590 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -380,6 +380,7 @@ JSSRC= \
storage/LvmThinEdit.js \
storage/BTRFSEdit.js \
storage/NFSEdit.js \
+ storage/NVMeOFEdit.js \
storage/PBSEdit.js \
storage/RBDEdit.js \
storage/StatusView.js \
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index faf621bc5..d508d0469 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -886,6 +886,12 @@ Ext.define('PVE.Utils', {
faIcon: 'building',
backups: false,
},
+ nvmeof: {
+ name: 'NVMe over Fabrics',
+ ipanel: 'NVMeOFInputPanel',
+ faIcon: 'building',
+ backups: false,
+ },
cephfs: {
name: 'CephFS',
ipanel: 'CephFSInputPanel',
diff --git a/www/manager6/dc/StorageView.js b/www/manager6/dc/StorageView.js
index baad3990f..6afea33ce 100644
--- a/www/manager6/dc/StorageView.js
+++ b/www/manager6/dc/StorageView.js
@@ -68,8 +68,9 @@ Ext.define(
});
},
- // What a storage is on: the volume group or pool for the types without path or target.
- storageLocation: function ({ type, vgname, thinpool, pool, target, path }) {
+ // What a storage is on: the volume group, pool or subsystem for the types without path or
+ // target.
+ storageLocation: function ({ type, vgname, thinpool, pool, subsysnqn, target, path }) {
switch (type) {
case 'lvm':
return vgname;
@@ -77,6 +78,8 @@ Ext.define(
return vgname && thinpool ? `${vgname}/${thinpool}` : vgname;
case 'zfspool':
return pool;
+ case 'nvmeof':
+ return subsysnqn;
default:
return target || path;
}
@@ -208,6 +211,13 @@ Ext.define(
),
'lun target',
],
+ nvmeof: [
+ 'san',
+ gettext(
+ 'Connect to an NVMe/TCP subsystem, to use its namespaces directly or as base of an LVM storage.',
+ ),
+ 'nvme tcp subsystem namespace',
+ ],
zfs: [
'san',
gettext(
diff --git a/www/manager6/storage/IScsiEdit.js b/www/manager6/storage/IScsiEdit.js
index c26c83467..192b974ea 100644
--- a/www/manager6/storage/IScsiEdit.js
+++ b/www/manager6/storage/IScsiEdit.js
@@ -5,6 +5,7 @@ Ext.define('PVE.storage.IScsiScan', {
queryParam: 'portal',
valueField: 'target',
displayField: 'target',
+ scanFields: ['target', 'portal'],
matchFieldWidth: false,
allowBlank: false,
@@ -56,13 +57,13 @@ Ext.define('PVE.storage.IScsiScan', {
}
let store = Ext.create('Ext.data.Store', {
- fields: ['target', 'portal'],
+ fields: me.scanFields,
proxy: {
type: 'proxmox',
url: `${me.apiBaseUrl}${me.nodename}${me.apiSuffix}`,
},
});
- store.sort('target', 'ASC');
+ store.sort(me.valueField, 'ASC');
Ext.apply(me, {
store: store,
diff --git a/www/manager6/storage/NVMeOFEdit.js b/www/manager6/storage/NVMeOFEdit.js
new file mode 100644
index 000000000..2ff1eb662
--- /dev/null
+++ b/www/manager6/storage/NVMeOFEdit.js
@@ -0,0 +1,150 @@
+Ext.define('PVE.storage.NVMeOFScan', {
+ extend: 'PVE.storage.IScsiScan',
+ alias: 'widget.pveNVMeOFScan',
+
+ valueField: 'subsysnqn',
+ displayField: 'subsysnqn',
+ scanFields: ['subsysnqn', 'portals'],
+
+ listConfig: {
+ width: 650,
+ columns: [
+ {
+ dataIndex: 'subsysnqn',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+ emptyText: PVE.Utils.renderNotFound(gettext('Subsystem')),
+ },
+
+ config: {
+ apiSuffix: '/scan/nvmeof',
+ },
+});
+
+Ext.define('PVE.storage.NVMeOFInputPanel', {
+ extend: 'PVE.panel.StorageBase',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ onlineHelp: 'storage_nvmeof',
+
+ namespaceContent: 'images',
+
+ onGetValues: function (values) {
+ let me = this;
+
+ values.content = values.namespaces ? me.namespaceContent : 'none';
+ delete values.namespaces;
+
+ return me.callParent([values]);
+ },
+
+ // a storage can be restricted to some volume types, keep its content types then
+ setValues: function (values) {
+ let me = this;
+
+ values.namespaces = PVE.Utils.contentAllows(values.content, 'vm-vol');
+ if (values.namespaces) {
+ me.namespaceContent = Ext.Array.from(values.content).join(',');
+ }
+ me.callParent([values]);
+ },
+
+ column1: [
+ {
+ xtype: 'textfield',
+ name: 'discovery-portal',
+ fieldLabel: gettext('Discovery Portal'),
+ // without discovery, the subsystem and portals can be entered as they are
+ allowBlank: true,
+ submitValue: false,
+ cbind: {
+ hidden: '{!isCreate}',
+ disabled: '{!isCreate}',
+ },
+ listeners: {
+ change: {
+ fn: function (field, value) {
+ let scan = this.up('inputpanel').lookup('subsystemScan');
+ scan.setHideTrigger(!value);
+ scan.setPortal(value);
+ scan.setValue('');
+ if (!value) {
+ scan.getStore().removeAll();
+ }
+ },
+ buffer: 500,
+ },
+ },
+ },
+ {
+ xtype: 'checkbox',
+ name: 'namespaces',
+ boxLabel: gettext('Use namespaces directly'),
+ },
+ ],
+
+ // NQNs and portal lists are longer than a column is wide
+ columnB: [
+ {
+ cbind: {
+ xtype: (get) => (get('isCreate') ? 'pveNVMeOFScan' : 'displayfield'),
+ hideTrigger: '{isCreate}',
+ },
+ // an NQN can also be typed, if the array offers no discovery
+ editable: true,
+ notFoundIsValid: true,
+ // the list needs a discovery portal
+ expand: function () {
+ if (this.portal) {
+ this.self.prototype.expand.call(this);
+ }
+ },
+ name: 'subsysnqn',
+ value: '',
+ fieldLabel: gettext('Subsystem'),
+ allowBlank: false,
+ reference: 'subsystemScan',
+ listeners: {
+ change: function (field, value) {
+ let rec = field
+ .getStore?.()
+ .findRecord('subsysnqn', value, 0, false, true, true);
+ if (rec) {
+ field
+ .up('inputpanel')
+ .down('field[name=portals]')
+ .setValue(rec.get('portals'));
+ }
+ },
+ nodechanged: function (value) {
+ this.up('inputpanel').lookup('storageNodeRestriction').setValue(value);
+ },
+ },
+ },
+ {
+ xtype: 'textfield',
+ name: 'portals',
+ fieldLabel: gettext('Portals'),
+ allowBlank: false,
+ },
+ ],
+
+ advancedColumn1: [
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 'fast-io-fail-tmo',
+ fieldLabel: gettext('Fail I/O After (s)'),
+ minValue: -1,
+ emptyText: '120',
+ cbind: {
+ deleteEmpty: '{!isCreate}',
+ },
+ autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext('Applies to newly connected paths. -1 disables the timeout.'),
+ },
+ },
+ ],
+});
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (11 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
` (4 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
A namespace of an NVMe over Fabrics storage can hold the volume group
of an LVM storage like an iSCSI LUN, so list such storages in the base
storage selector as well and name the type of each.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/StorageView.js | 4 ++--
www/manager6/storage/LVMEdit.js | 12 ++++++++----
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/www/manager6/dc/StorageView.js b/www/manager6/dc/StorageView.js
index 6afea33ce..5f4f519be 100644
--- a/www/manager6/dc/StorageView.js
+++ b/www/manager6/dc/StorageView.js
@@ -250,9 +250,9 @@ Ext.define(
lvm: [
'local',
gettext(
- 'Use an existing volume group, or create one on a LUN of an iSCSI storage.',
+ 'Use an existing volume group, or create one on an iSCSI LUN or NVMe/TCP namespace.',
),
- 'san lun shared',
+ 'san lun shared nvme namespace',
],
lvmthin: ['local', gettext('Use an existing LVM thin pool.'), ''],
zfspool: ['local', gettext('Use an existing ZFS pool or dataset.'), 'zpool'],
diff --git a/www/manager6/storage/LVMEdit.js b/www/manager6/storage/LVMEdit.js
index 688130b19..146dac12c 100644
--- a/www/manager6/storage/LVMEdit.js
+++ b/www/manager6/storage/LVMEdit.js
@@ -70,10 +70,13 @@ Ext.define('PVE.storage.BaseStorageSelector', {
let store = Ext.create('Ext.data.Store', {
autoLoad: {
addRecords: true,
- params: {
- type: 'iscsi',
- },
},
+ filters: [
+ {
+ filterFn: (rec) =>
+ !rec.get('storage') || ['iscsi', 'nvmeof'].includes(rec.get('type')),
+ },
+ ],
fields: [
'storage',
'type',
@@ -82,7 +85,8 @@ Ext.define('PVE.storage.BaseStorageSelector', {
name: 'text',
convert: function (value, record) {
if (record.data.storage) {
- return record.data.storage + ' (iSCSI)';
+ let type = PVE.Utils.format_storage_type(record.data.type);
+ return `${record.data.storage} (${type})`;
} else {
return me.existingGroupsText;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (12 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
` (3 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Add a guided setup for shared LVM on a namespace of an NVMe over Fabrics
subsystem, with the same steps as for iSCSI. Datacenter > Storage > Add
and the SAN view list it next to the other setups. It connects to a
subsystem found at a discovery portal and shows the host NQNs of the
nodes for the access rules of the array.
The target step is shared with iSCSI, as only the storage type, its
scan and the node identities differ. A reused NVMe over Fabrics storage
keeps its I/O portals, which the step shows read-only. The portals that
the step suggests leave out the default I/O port of such a storage, so
that the discovery also tries the discovery port.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 2 +-
www/manager6/SANUtils.js | 2 +-
www/manager6/dc/SAN.js | 4 +-
www/manager6/dc/StorageView.js | 5 +
www/manager6/storage/wizard/LunSelect.js | 35 ++-
www/manager6/storage/wizard/NodeIdentities.js | 57 +++--
.../storage/wizard/{IScsi.js => Target.js} | 232 +++++++++++++-----
www/manager6/storage/wizard/Wizard.js | 60 +++--
www/manager6/window/SANLunState.js | 30 ++-
9 files changed, 298 insertions(+), 129 deletions(-)
rename www/manager6/storage/wizard/{IScsi.js => Target.js} (59%)
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index dddf17590..f2fc80279 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -393,7 +393,7 @@ JSSRC= \
storage/wizard/Wizard.js \
storage/wizard/LunSelect.js \
storage/wizard/NodeIdentities.js \
- storage/wizard/IScsi.js \
+ storage/wizard/Target.js \
storage/wizard/FibreChannel.js \
Workspace.js \
# end of JSSRC list
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index ce63fbeda..d28285d22 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -237,7 +237,7 @@ Ext.define('PVE.SANUtils', {
},
openSetup: function (type, reloadCallback) {
- let wizard = type === 'fc' ? 'FibreChannel' : 'IScsi';
+ let wizard = { fc: 'FibreChannel', nvmeof: 'NVMeOF' }[type] ?? 'IScsi';
Ext.create(`PVE.storage.wizard.${wizard}`, { autoShow: true, reloadCallback });
},
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index eea8aea65..2f624cbd2 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -112,7 +112,7 @@ Ext.define('PVE.dc.SANView', {
autoShow: true,
});
let setupItem = (type) => ({
- text: type === 'fc' ? gettext('Fibre Channel') : 'iSCSI',
+ text: type === 'fc' ? gettext('Fibre Channel') : PVE.Utils.format_storage_type(type),
iconCls: `fa fa-fw fa-${type === 'fc' ? 'plug' : 'building'}`,
disabled: !PVE.SANUtils.canSetup(type),
handler: () => PVE.SANUtils.openSetup(type, reload),
@@ -121,7 +121,7 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Add Shared LVM'),
disabled: !PVE.SANUtils.canSetup('iscsi'),
- menu: [setupItem('iscsi'), setupItem('fc')],
+ menu: [setupItem('iscsi'), setupItem('fc'), setupItem('nvmeof')],
},
{
text: gettext('Multipath Overrides'),
diff --git a/www/manager6/dc/StorageView.js b/www/manager6/dc/StorageView.js
index 5f4f519be..4182b4c12 100644
--- a/www/manager6/dc/StorageView.js
+++ b/www/manager6/dc/StorageView.js
@@ -201,6 +201,11 @@ Ext.define(
gettext('Set up shared LVM on a Fibre Channel LUN with multipath.'),
'san lun multipath lvm fc guided',
),
+ setup(
+ 'nvmeof',
+ gettext('Set up shared LVM on a namespace of an NVMe/TCP subsystem.'),
+ 'san nvme namespace lvm tcp guided',
+ ),
];
let types = {
diff --git a/www/manager6/storage/wizard/LunSelect.js b/www/manager6/storage/wizard/LunSelect.js
index 36a9a3804..7fd814bf0 100644
--- a/www/manager6/storage/wizard/LunSelect.js
+++ b/www/manager6/storage/wizard/LunSelect.js
@@ -30,10 +30,11 @@ Ext.define('PVE.storage.wizard.LunSelect', {
// The scan names one target per LUN, also for a LUN reached through several. The content
// listing has the LUNs of the connection, and a LUN that shows up later is listed once the
// scan names the target of the connection for it.
+ let targetKey = wizard.wizardType === 'nvmeof' ? 'nvme-subsystem' : 'iscsi-target';
let filter = {
id: 'connection',
filterFn: (rec) =>
- rec.get('iscsi-target') === connection.target ||
+ rec.get(targetKey) === connection.target ||
(me.connectionWwids ?? []).includes(rec.get('wwid')),
};
me.store.addFilter(filter);
@@ -45,7 +46,7 @@ Ext.define('PVE.storage.wizard.LunSelect', {
success: function (response) {
if (!me.destroyed) {
me.connectionWwids = response.result.data
- .map((entry) => entry.volid.match(/\.dm-uuid-mpath-(.+)$/)?.[1])
+ .map((entry) => entry.volid.match(/(?:\.dm-uuid-mpath-|:nvme-)(.+)$/)?.[1])
.filter(Boolean);
me.store.addFilter(filter);
}
@@ -74,7 +75,9 @@ Ext.define('PVE.storage.wizard.LunSelect', {
// the setup refuses a LUN with data, so do not offer one
let hasData = (row) => row.usage === 'lvm' || row.usage === 'other';
let blocked = (row) => inUse(row) || hasData(row);
- let noLunsText = gettext('No LUNs found. Check connections and array access.');
+ let noLunsText = me.nvme
+ ? gettext('No namespaces found. Check connections and array access.')
+ : gettext('No LUNs found. Check connections and array access.');
let updateSelection = function () {
let row = me.getSelection();
me.down('#selection').setValue(row && !blocked(row) ? row.wwid : '');
@@ -91,9 +94,13 @@ Ext.define('PVE.storage.wizard.LunSelect', {
xtype: 'component',
cls: 'pmx-hint',
margin: '0 0 10 0',
- html: gettext(
- 'LUNs with known data or users cannot be selected. To keep the volume group on a LUN, add it under Datacenter > Storage > Add > LVM instead. To discard the data of a LUN, stop its users on all hosts and wipe it under Node > SAN first.',
- ),
+ html: me.nvme
+ ? gettext(
+ 'Namespaces with known data or users cannot be selected. To keep the volume group on a namespace, add it under Datacenter > Storage > Add > LVM instead. To discard the data of a namespace, stop its users on all hosts and wipe it under Node > SAN first.',
+ )
+ : gettext(
+ 'LUNs with known data or users cannot be selected. To keep the volume group on a LUN, add it under Datacenter > Storage > Add > LVM instead. To discard the data of a LUN, stop its users on all hosts and wipe it under Node > SAN first.',
+ ),
},
...(me.rescanNote
? [
@@ -126,6 +133,7 @@ Ext.define('PVE.storage.wizard.LunSelect', {
renderer: Ext.htmlEncode,
},
{ text: 'LUN', dataIndex: 'iscsi-lun', width: 60 },
+ { text: 'NSID', dataIndex: 'nvme-nsid', width: 60 },
{
text: gettext('Model'),
dataIndex: 'model',
@@ -144,6 +152,8 @@ Ext.define('PVE.storage.wizard.LunSelect', {
dataIndex: 'multipath',
width: 170,
renderer: PVE.SANUtils.renderSanLunMultipath,
+ // native NVMe multipath needs no setup per namespace
+ hidden: me.nvme,
},
{
text: gettext('Used By'),
@@ -189,9 +199,10 @@ Ext.define('PVE.storage.wizard.LunSelect', {
searchchange: function (field, query) {
let grid = me.down('grid');
if (grid?.rendered) {
- grid.setEmptyText(
- query ? gettext('No LUNs match the search.') : noLunsText,
- );
+ let noMatch = me.nvme
+ ? gettext('No namespaces match the search.')
+ : gettext('No LUNs match the search.');
+ grid.setEmptyText(query ? noMatch : noLunsText);
}
},
},
@@ -206,8 +217,10 @@ Ext.define('PVE.storage.wizard.LunSelect', {
let search = me.down('pveRecordSearchField');
me.on('activate', function () {
search.setTargetStore(me.store);
- if (me.up('window').wizardType === 'fc') {
- me.down('gridcolumn[dataIndex=iscsi-lun]').hide();
+ let type = me.up('window').wizardType;
+ me.down('gridcolumn[dataIndex=iscsi-lun]').setHidden(type !== 'iscsi');
+ me.down('gridcolumn[dataIndex=nvme-nsid]').setHidden(type !== 'nvmeof');
+ if (type === 'fc') {
me.store.addFilter({
id: 'transport',
filterFn: (rec) => ['fc', 'fcoe'].includes(rec.get('transport')),
diff --git a/www/manager6/storage/wizard/NodeIdentities.js b/www/manager6/storage/wizard/NodeIdentities.js
index 0a54f7ce1..36d37e164 100644
--- a/www/manager6/storage/wizard/NodeIdentities.js
+++ b/www/manager6/storage/wizard/NodeIdentities.js
@@ -1,32 +1,21 @@
Ext.define('pve-san-node-identity', {
extend: 'Ext.data.Model',
idProperty: 'node',
- fields: ['node', 'state', 'iscsi-initiator-name'],
+ fields: ['node', 'state', 'iscsi-initiator-name', 'nvme-host-nqn'],
});
-// The nodes with their initiator names, to allow them on the array, and to select the nodes that
-// use the target. The node serving the web UI sets up the volume group, so it stays selected.
+// The nodes with their iSCSI initiator names or NVMe host NQNs, to allow them on the array, and to
+// select the nodes that use the target. The node serving the web UI sets up the volume group, so it
+// stays selected.
Ext.define('PVE.storage.wizard.NodeIdentities', {
extend: 'Ext.grid.Panel',
xtype: 'pveStorageWizardNodeIdentities',
height: 220,
selModel: { selType: 'checkboxmodel', mode: 'SIMPLE' },
viewConfig: { enableTextSelection: true },
- columns: [
- { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
- {
- text: gettext('Multipath Setup'),
- dataIndex: 'state',
- width: 175,
- renderer: PVE.SANUtils.renderState,
- },
- {
- text: gettext('Initiator Name'),
- dataIndex: 'iscsi-initiator-name',
- flex: 1,
- renderer: Ext.htmlEncode,
- },
- ],
+
+ // the identity of a node that the array allows
+ identity: 'iscsi-initiator-name',
getSelectedNodes: function () {
return this.getSelection()
@@ -62,6 +51,24 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
initComponent: function () {
let me = this;
+ let nvme = me.identity === 'nvme-host-nqn';
+ me.columns = [
+ { text: gettext('Node'), dataIndex: 'node', width: 100, renderer: Ext.htmlEncode },
+ {
+ text: gettext('Multipath Setup'),
+ dataIndex: 'state',
+ width: 175,
+ renderer: PVE.SANUtils.renderState,
+ // the state of multipathd, which native NVMe multipath does not use
+ hidden: nvme,
+ },
+ {
+ text: nvme ? gettext('Host NQN') : gettext('Initiator Name'),
+ dataIndex: me.identity,
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ];
me.rstore = Ext.create('Proxmox.data.UpdateStore', {
storeid: 'san-wizard-identities',
model: 'pve-san-node-identity',
@@ -90,19 +97,23 @@ Ext.define('PVE.storage.wizard.NodeIdentities', {
me.tbar = [
{
xtype: 'tbtext',
- text: gettext(
- 'Select the nodes that will use the target and allow their initiator names on the array.',
- ),
+ text: nvme
+ ? gettext(
+ 'Select the nodes that will use the subsystem and allow their host NQNs on the array.',
+ )
+ : gettext(
+ 'Select the nodes that will use the target and allow their initiator names on the array.',
+ ),
},
'->',
{
- text: gettext('Copy Initiator Names'),
+ text: nvme ? gettext('Copy Host NQNs') : gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
handler: () =>
PVE.Utils.copyToClipboard(
me
.getSelection()
- .map((rec) => rec.get('iscsi-initiator-name'))
+ .map((rec) => rec.get(me.identity))
.filter(Boolean)
.join('\n'),
),
diff --git a/www/manager6/storage/wizard/IScsi.js b/www/manager6/storage/wizard/Target.js
similarity index 59%
rename from www/manager6/storage/wizard/IScsi.js
rename to www/manager6/storage/wizard/Target.js
index 9ef89caef..37e2111c3 100644
--- a/www/manager6/storage/wizard/IScsi.js
+++ b/www/manager6/storage/wizard/Target.js
@@ -1,16 +1,21 @@
-Ext.define('PVE.storage.wizard.IScsiTarget', {
+// The step of a guided setup that connects to the target of an iSCSI or the subsystem of an NVMe
+// over Fabrics array, through a storage of that type.
+Ext.define('PVE.storage.wizard.Target', {
extend: 'Proxmox.panel.InputPanel',
- xtype: 'pveStorageWizardIScsiTarget',
- onlineHelp: 'storage_open_iscsi',
+ xtype: 'pveStorageWizardTarget',
- // The iSCSI storage the target already has, which the setup uses instead of adding one.
+ storageType: 'iscsi',
+
+ // The storage the target already has, which the setup uses instead of adding one.
existingConnection: function () {
let me = this;
let target = me.down('field[name=target]').getValue();
- return me.storages?.find((entry) => entry.type === 'iscsi' && entry.target === target);
+ return me.storages?.find(
+ (entry) => entry.type === me.storageType && entry[me.targetKey] === target,
+ );
},
- // The ID of the iSCSI storage the setup adds, named after the storage on the LUN by default.
+ // The ID of the storage the setup adds, named after the storage on the LUN by default.
newConnectionId: function () {
let me = this;
let id = me.down('field[name=storage]').getValue();
@@ -21,8 +26,9 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
// step and Next both use it, so that the note does not promise what Next then refuses.
reuseConflict: function (selectedNodes) {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
let target = me.down('field[name=target]').getValue();
- let users = (me.storages ?? []).filter((entry) => entry.target === target);
+ let users = (me.storages ?? []).filter((entry) => entry[me.targetKey] === target);
if (!users.length) {
return undefined;
}
@@ -30,30 +36,46 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
let format = (text, ...args) => Ext.String.format(text, ...args);
if (users.length !== 1) {
return format(
- gettext(
- 'Storages {0} use this target. The setup can only reuse a target that one storage uses.',
- ),
+ nvme
+ ? gettext(
+ 'Storages {0} use this subsystem. The setup can only reuse a subsystem that one storage uses.',
+ )
+ : gettext(
+ 'Storages {0} use this target. The setup can only reuse a target that one storage uses.',
+ ),
users.map((entry) => entry.storage).join(', '),
);
- } else if (base.type !== 'iscsi') {
+ } else if (base.type !== me.storageType) {
return format(
- gettext(
- 'Storage "{0}" uses this target. The setup can only reuse an iSCSI storage.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" uses this subsystem. The setup can only reuse an NVMe over Fabrics storage.',
+ )
+ : gettext(
+ 'Storage "{0}" uses this target. The setup can only reuse an iSCSI storage.',
+ ),
base.storage,
);
} else if (`${base.content ?? ''}` !== 'none') {
return format(
- gettext(
- 'Storage "{0}" uses the LUNs of this target directly. If no guest uses them, clear Use LUNs directly in its edit dialog, otherwise use another target.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" uses the namespaces of this subsystem directly. If no guest uses them, clear Use namespaces directly in its edit dialog, otherwise use another subsystem.',
+ )
+ : gettext(
+ 'Storage "{0}" uses the LUNs of this target directly. If no guest uses them, clear Use LUNs directly in its edit dialog, otherwise use another target.',
+ ),
base.storage,
);
} else if (base.disable) {
return format(
- gettext(
- 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
- ),
+ nvme
+ ? gettext(
+ 'Storage "{0}" connects to this subsystem, but it is disabled. Enable that storage first.',
+ )
+ : gettext(
+ 'Storage "{0}" connects to this target, but it is disabled. Enable that storage first.',
+ ),
base.storage,
);
}
@@ -61,17 +83,17 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
if (baseNodes.length && !baseNodes.includes(Proxmox.NodeName)) {
return format(
gettext(
- 'iSCSI storage "{0}" is not available on node "{1}". Open the web UI on one of its nodes.',
+ 'Storage "{0}" is not available on node "{1}". Open the web UI on one of its nodes.',
),
base.storage,
Proxmox.NodeName,
);
} else if (baseNodes.length && selectedNodes.some((node) => !baseNodes.includes(node))) {
return format(
- gettext('iSCSI storage "{0}" is not available on all selected nodes.'),
+ gettext('Storage "{0}" is not available on all selected nodes.'),
base.storage,
);
- } else if (!base['managed-multipath']) {
+ } else if (!nvme && !base['managed-multipath']) {
return format(
gettext(
'Enable Multipath in the edit dialog of storage "{0}" before using it here.',
@@ -94,6 +116,14 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
field.setReadOnly(!!existing);
let id = me.down('field[name=storage]').getValue();
field.setEmptyText(id ? `${id}-base` : '');
+ // a reused storage keeps its own portals
+ let portals = me.down('field[name=portals]');
+ if (portals) {
+ if (existing) {
+ portals.setValue(existing.portals);
+ }
+ portals.setReadOnly(!!existing);
+ }
// the nodes of a storage that the setup uses are the default selection
let identities = me.down('pveStorageWizardNodeIdentities');
@@ -112,19 +142,32 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
note.setHidden(!conflict);
},
- // The portals of the enabled iSCSI storages, to discover the target of another LUN through one
- // of them.
+ // The portals of the enabled storages of the type, to discover the target of another LUN
+ // through one of them. An NVMe over Fabrics storage lists the I/O portals of its subsystem,
+ // whose default port would skip the discovery port 8009, so the suggestion leaves it out.
updatePortalSuggestions: function () {
let me = this;
let users = new Map();
- for (const { type, portal, storage, disable } of me.storages ?? []) {
- if (['iscsi', 'iscsidirect'].includes(type) && portal && !disable) {
- users.set(portal, [...(users.get(portal) ?? []), storage]);
+ for (const { type, portal, portals, storage, disable } of me.storages ?? []) {
+ let list = [];
+ if (disable) {
+ continue;
+ } else if (me.storageType === 'nvmeof' && type === 'nvmeof') {
+ // as the storage splits its list
+ list = `${portals ?? ''}`
+ .split(/[\s,;]+/)
+ .filter(Boolean)
+ .map((entry) => entry.replace(/:4420$/, ''));
+ } else if (me.storageType === 'iscsi' && ['iscsi', 'iscsidirect'].includes(type)) {
+ list = portal ? [portal] : [];
+ }
+ for (const entry of new Set(list)) {
+ users.set(entry, [...(users.get(entry) ?? []), storage]);
}
}
let suggestions = [...users.keys()]
.sort()
- .map((portal) => ({ portal, storages: users.get(portal).sort().join(', ') }));
+ .map((entry) => ({ portal: entry, storages: users.get(entry).sort().join(', ') }));
let field = me.down('field[name=portal]');
field.getStore().setData(suggestions);
field.setHideTrigger(!suggestions.length);
@@ -132,13 +175,16 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
onSubmit: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
let wizard = me.up('window');
let values = me.getValues();
// a disabled target field counts as valid, while the portal waits for its discovery
let target = me.down('field[name=target]');
if (!values.target || target.discoveredPortal !== values.portal) {
wizard.showError(
- gettext('Select a target once the targets of this portal are listed.'),
+ nvme
+ ? gettext('Select a subsystem once the subsystems of this portal are listed.')
+ : gettext('Select a target once the targets of this portal are listed.'),
);
return;
}
@@ -180,9 +226,13 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
// the step showed which storage it reuses, if any, which must still hold
if (me.existingConnection()?.storage !== shown) {
wizard.showError(
- gettext(
- 'The iSCSI storage of this target changed in the meantime. Check this step, then retry.',
- ),
+ nvme
+ ? gettext(
+ 'The NVMe over Fabrics storage of this subsystem changed in the meantime. Check this step, then retry.',
+ )
+ : gettext(
+ 'The iSCSI storage of this target changed in the meantime. Check this step, then retry.',
+ ),
);
return;
}
@@ -210,25 +260,35 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
if (restricted.length) {
nodes = (selectedNodes.length ? selectedNodes : restricted).join(',');
}
- connected({ storage: base.storage, target: base.target, nodes });
+ connected({ storage: base.storage, target: base[me.targetKey], nodes });
return;
}
let params = {
- type: 'iscsi',
+ type: me.storageType,
storage: me.newConnectionId(),
- portal: values.portal,
- target: values.target,
content: 'none',
- 'managed-multipath': 1,
+ ...(nvme
+ ? // the storage configuration does not keep outer spaces, so neither may the
+ // copy that the cleanup compares
+ { subsysnqn: values.target, portals: values.portals.trim() }
+ : {
+ portal: values.portal,
+ target: values.target,
+ 'managed-multipath': 1,
+ }),
};
if (nodes) {
params.nodes = nodes;
}
if (params.storage === values.storage) {
wizard.showError(
- gettext(
- 'The storage for guest disks and the iSCSI storage need different IDs. Change the iSCSI Storage ID under Advanced.',
- ),
+ nvme
+ ? gettext(
+ 'The storage for guest disks and the NVMe over Fabrics storage need different IDs. Change the NVMe over Fabrics Storage ID under Advanced.',
+ )
+ : gettext(
+ 'The storage for guest disks and the iSCSI storage need different IDs. Change the iSCSI Storage ID under Advanced.',
+ ),
);
return;
}
@@ -236,9 +296,13 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
wizard.showError(
Ext.htmlEncode(
Ext.String.format(
- gettext(
- 'The ID "{0}" for the iSCSI storage is already in use. Set another one under Advanced.',
- ),
+ nvme
+ ? gettext(
+ 'The ID "{0}" for the NVMe over Fabrics storage is already in use. Set another one under Advanced.',
+ )
+ : gettext(
+ 'The ID "{0}" for the iSCSI storage is already in use. Set another one under Advanced.',
+ ),
params.storage,
),
),
@@ -258,9 +322,13 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
initComponent: function () {
let me = this;
+ let nvme = me.storageType === 'nvmeof';
+ me.targetKey = nvme ? 'subsysnqn' : 'target';
+ me.onlineHelp = nvme ? 'storage_nvmeof' : 'storage_open_iscsi';
+ let targetLabel = nvme ? gettext('Subsystem') : gettext('Target');
let common = PVE.panel.StorageBase.commonFields(true);
// wide enough for the longest label of the step
- let labelWidth = 150;
+ let labelWidth = nvme ? 200 : 150;
let update = () => me.updateConnectionField();
let discovery = new Ext.util.DelayedTask(function () {
let portal = me.down('field[name=portal]').getValue();
@@ -285,7 +353,7 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
{
xtype: 'combobox',
name: 'portal',
- fieldLabel: gettext('Portal'),
+ fieldLabel: nvme ? gettext('Discovery Portal') : gettext('Portal'),
labelWidth,
allowBlank: false,
queryMode: 'local',
@@ -312,51 +380,83 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
let target = me.down('field[name=target]');
target.setValue('');
target.disable();
+ // and the I/O portals of its subsystem
+ me.down('field[name=portals]')?.setValue('');
discovery.delay(500);
},
},
},
{
- xtype: 'pveIScsiScan',
+ xtype: nvme ? 'pveNVMeOFScan' : 'pveIScsiScan',
name: 'target',
- fieldLabel: gettext('Target'),
+ fieldLabel: targetLabel,
labelWidth,
disabled: true,
showNodeSelector: false,
nodename: Proxmox.NodeName,
- listeners: { change: update },
+ listeners: {
+ change: function (field, value) {
+ // the I/O portals that the discovery reported for the subsystem
+ let rec =
+ nvme &&
+ field.getStore().findRecord('subsysnqn', value, 0, false, true, true);
+ if (rec) {
+ me.down('field[name=portals]').setValue(rec.get('portals'));
+ }
+ update();
+ },
+ },
listConfig: {
width: 650,
- emptyText: PVE.Utils.renderNotFound(gettext('iSCSI Target')),
+ emptyText: PVE.Utils.renderNotFound(
+ nvme ? gettext('Subsystem') : gettext('iSCSI Target'),
+ ),
columns: [
{
- text: gettext('Target'),
- dataIndex: 'target',
+ text: targetLabel,
+ dataIndex: me.targetKey,
flex: 1,
renderer: Ext.htmlEncode,
},
{
text: gettext('Portals'),
- dataIndex: 'portal',
+ dataIndex: nvme ? 'portals' : 'portal',
width: 220,
renderer: Ext.htmlEncode,
},
],
},
},
+ ...(nvme
+ ? [
+ {
+ xtype: 'textfield',
+ name: 'portals',
+ fieldLabel: gettext('Portals'),
+ labelWidth,
+ allowBlank: false,
+ },
+ ]
+ : []),
{
xtype: 'component',
itemId: 'connectionNote',
margin: '0 0 10 0',
hidden: true,
},
- { xtype: 'pveStorageWizardNodeIdentities', listeners: { nodeschange: update } },
+ {
+ xtype: 'pveStorageWizardNodeIdentities',
+ identity: nvme ? 'nvme-host-nqn' : 'iscsi-initiator-name',
+ listeners: { nodeschange: update },
+ },
];
me.advancedItems = [
{
...common.id,
name: 'connection',
- fieldLabel: gettext('iSCSI Storage ID'),
+ fieldLabel: nvme
+ ? gettext('NVMe over Fabrics Storage ID')
+ : gettext('iSCSI Storage ID'),
labelWidth,
allowBlank: true,
},
@@ -386,12 +486,15 @@ Ext.define('PVE.storage.wizard.IScsiTarget', {
},
});
+// Both add the LVM storage on a volume of the connection storage, which only differs in its name.
Ext.define('PVE.storage.wizard.IScsi', {
extend: 'PVE.storage.wizard.SAN',
wizardType: 'iscsi',
createStorage: function (values, lun) {
let me = this;
+ let nvme = me.wizardType === 'nvmeof';
+ let suffix = nvme ? `:nvme-${lun.wwid}` : `.dm-uuid-mpath-${lun.wwid}`;
let fail = (response) => me.showError(response.htmlStatus);
Proxmox.Utils.API2Request({
url: `/nodes/${Proxmox.NodeName}/storage/${encodeURIComponent(me.connection.storage)}/content`,
@@ -403,14 +506,16 @@ Ext.define('PVE.storage.wizard.IScsi', {
if (me.destroyed) {
return;
}
- let volume = response.result.data.find((entry) =>
- entry.volid.endsWith(`.dm-uuid-mpath-${lun.wwid}`),
- );
+ let volume = response.result.data.find((entry) => entry.volid.endsWith(suffix));
if (!volume) {
me.showError(
- gettext(
- 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
- ),
+ nvme
+ ? gettext(
+ 'The selected namespace is not available yet. Check Node > SAN, then retry.',
+ )
+ : gettext(
+ 'The selected LUN has no multipath volume yet. Check Node > SAN, then retry.',
+ ),
);
return;
}
@@ -419,3 +524,8 @@ Ext.define('PVE.storage.wizard.IScsi', {
});
},
});
+
+Ext.define('PVE.storage.wizard.NVMeOF', {
+ extend: 'PVE.storage.wizard.IScsi',
+ wizardType: 'nvmeof',
+});
diff --git a/www/manager6/storage/wizard/Wizard.js b/www/manager6/storage/wizard/Wizard.js
index 97b1405f5..d86d1cddd 100644
--- a/www/manager6/storage/wizard/Wizard.js
+++ b/www/manager6/storage/wizard/Wizard.js
@@ -10,8 +10,8 @@ Ext.define('PVE.storage.wizard.SAN', {
type === 'fc' ? gettext('Fibre Channel') : PVE.Utils.format_storage_type(type),
),
- // Removes the iSCSI storage that a closed setup added if it still has the settings of the
- // setup and no storage uses it, and calls done in any case.
+ // Removes the iSCSI or NVMe over Fabrics storage that a closed setup added if it still has
+ // the settings of the setup and no storage uses it, and calls done in any case.
removeAddedConnection: function (connection, done) {
let id = connection.storage;
// the storage list adds these to the configuration of a storage
@@ -29,7 +29,7 @@ Ext.define('PVE.storage.wizard.SAN', {
gettext('Error'),
Ext.htmlEncode(
Ext.String.format(
- gettext('Could not remove iSCSI storage "{0}" of the closed setup'),
+ gettext('Could not remove storage "{0}" of the closed setup'),
id,
),
) + `: ${response.htmlStatus}`,
@@ -45,7 +45,7 @@ Ext.define('PVE.storage.wizard.SAN', {
let own = storages.find((entry) => entry.storage === id);
let created = connection.created;
let unchanged =
- own?.type === 'iscsi' &&
+ own?.type === created.type &&
keys(own).join() === keys(created).join() &&
keys(own).every((key) => text(key, own[key]) === text(key, created[key]));
let used = storages.some((entry) => `${entry.base ?? ''}`.startsWith(`${id}:`));
@@ -169,7 +169,7 @@ Ext.define('PVE.storage.wizard.SAN', {
};
},
- // Removes an iSCSI storage that the setup added for a storage it did not add, then reloads.
+ // Removes a connection storage that the setup added for a storage it did not add, then reloads.
cleanUp: function () {
let me = this;
let done = () => me.reloadCallback?.();
@@ -225,6 +225,7 @@ Ext.define('PVE.storage.wizard.SAN', {
storage: values.storage,
nodes: params.nodes,
fc: me.wizardType === 'fc',
+ nvme: me.wizardType === 'nvmeof',
autoShow: true,
});
};
@@ -233,6 +234,9 @@ Ext.define('PVE.storage.wizard.SAN', {
multipathSummary: function (lun) {
let me = this;
+ if (me.wizardType === 'nvmeof') {
+ return PVE.SANUtils.multipathText('native');
+ }
// an explicit entry takes precedence, also for a LUN of the iSCSI storage
if (lun.multipath === 'manual') {
return PVE.SANUtils.multipathText('manual');
@@ -257,7 +261,9 @@ Ext.define('PVE.storage.wizard.SAN', {
...(connection
? [
[
- gettext('iSCSI Storage'),
+ me.wizardType === 'nvmeof'
+ ? gettext('NVMe over Fabrics Storage')
+ : gettext('iSCSI Storage'),
connection.created
? Ext.String.format(
gettext('{0}, added by this setup'),
@@ -302,10 +308,15 @@ Ext.define('PVE.storage.wizard.SAN', {
me.down('#action').setHtml(
Ext.htmlEncode(
Ext.String.format(
- gettext(
- // TRANSLATORS: {0} is the label of the Finish button.
- '{0} creates the volume group "{1}" on this LUN and adds it as shared LVM storage "{2}".',
- ),
+ me.wizardType === 'nvmeof'
+ ? gettext(
+ // TRANSLATORS: {0} is the label of the Finish button.
+ '{0} creates the volume group "{1}" on this namespace and adds it as shared LVM storage "{2}".',
+ )
+ : gettext(
+ // TRANSLATORS: {0} is the label of the Finish button.
+ '{0} creates the volume group "{1}" on this LUN and adds it as shared LVM storage "{2}".',
+ ),
gettext('Finish'),
values.vgname,
values.storage,
@@ -350,10 +361,13 @@ Ext.define('PVE.storage.wizard.SAN', {
initComponent: function () {
let me = this;
- if (!['iscsi', 'fc'].includes(me.wizardType)) {
+ if (!['iscsi', 'fc', 'nvmeof'].includes(me.wizardType)) {
throw 'no SAN storage type specified';
}
- let iscsi = me.wizardType === 'iscsi';
+ // the iSCSI and NVMe over Fabrics setups connect to the array with a storage first
+ let target = me.wizardType !== 'fc';
+ let nvme = me.wizardType === 'nvmeof';
+ let onlineHelp = nvme ? 'storage_nvmeof' : 'storage_multipath';
me.subject = PVE.storage.wizard.SAN.subject(me.wizardType);
me.title = Proxmox.Utils.dialog_title(me.subject, true, true);
let common = PVE.panel.StorageBase.commonFields(true);
@@ -369,9 +383,9 @@ Ext.define('PVE.storage.wizard.SAN', {
let settings = {
xtype: 'inputpanel',
title: gettext('Storage'),
- onlineHelp: 'storage_multipath',
+ onlineHelp,
items: [
- ...(iscsi
+ ...(target
? []
: [
{
@@ -389,17 +403,18 @@ Ext.define('PVE.storage.wizard.SAN', {
],
advancedItems: [
vgname,
- ...(iscsi
+ ...(target
? []
: [{ ...common.nodes, validator: PVE.storage.wizard.SAN.validateNodes }]),
],
itemId: 'settings',
};
me.items = [
- ...(iscsi
+ ...(target
? [
{
- xtype: 'pveStorageWizardIScsiTarget',
+ xtype: 'pveStorageWizardTarget',
+ storageType: me.wizardType,
title: gettext('Target'),
itemId: 'target',
},
@@ -407,11 +422,12 @@ Ext.define('PVE.storage.wizard.SAN', {
: []),
{
xtype: 'pveStorageWizardLunSelect',
- title: 'LUN',
+ title: nvme ? gettext('Namespace') : 'LUN',
itemId: 'luns',
- onlineHelp: 'storage_multipath',
+ nvme,
+ onlineHelp,
// the nodes only see a newly mapped Fibre Channel LUN after a rescan
- rescanNote: iscsi
+ rescanNote: target
? undefined
: Ext.String.format(
gettext(
@@ -460,14 +476,14 @@ Ext.define('PVE.storage.wizard.SAN', {
me.down('#settings').on('activate', followId);
let tabs = me.down('#wizcontent');
tabs.on('tabchange', function (tp, tab) {
- if (iscsi && tab === me.down('#luns')) {
+ if (target && tab === me.down('#luns')) {
me.down('#back').disable();
}
me.down('#submit').setText(
tab === me.down('#target') ? gettext('Next') : gettext('Finish'),
);
});
- if (iscsi) {
+ if (target) {
me.down('#submit').setText(gettext('Next'));
}
me.on('destroy', () => !me.writing && me.cleanUp());
diff --git a/www/manager6/window/SANLunState.js b/www/manager6/window/SANLunState.js
index d257e99ba..5dfce247a 100644
--- a/www/manager6/window/SANLunState.js
+++ b/www/manager6/window/SANLunState.js
@@ -20,8 +20,11 @@ Ext.define('PVE.window.SANLunState', {
if (!me.wwid) {
throw 'no LUN specified';
}
+ let title = me.nvme
+ ? gettext('Storage {0}: Namespace State')
+ : gettext('Storage {0}: LUN State');
me.title = me.storage
- ? Ext.String.format(gettext('Storage {0}: LUN State'), me.storage)
+ ? Ext.String.format(title, me.storage)
: Ext.String.format(
gettext('LUN {0}: Paths by Node'),
Ext.htmlEncode(me.name || me.wwid),
@@ -56,13 +59,17 @@ Ext.define('PVE.window.SANLunState', {
sortAfterUpdate: true,
});
let added = Ext.String.format(gettext('Storage "{0}" was added.'), me.storage);
- let missingAdvice = me.fc
+ let missingAdvice = me.nvme
? gettext(
- 'The LUN is missing on {0}, so guests on this storage cannot run there. Check the LUN mapping and zoning on the array, then rescan the Fibre Channel hosts on {0}.',
+ 'The namespace is missing on {0}, so guests on this storage cannot run there. Check the host access to the subsystem on the array.',
)
- : gettext(
- 'The LUN is missing on {0}, so guests on this storage cannot run there. Check the LUN mapping and initiator access on the array.',
- );
+ : me.fc
+ ? gettext(
+ 'The LUN is missing on {0}, so guests on this storage cannot run there. Check the LUN mapping and zoning on the array, then rescan the Fibre Channel hosts on {0}.',
+ )
+ : gettext(
+ 'The LUN is missing on {0}, so guests on this storage cannot run there. Check the LUN mapping and initiator access on the array.',
+ );
let updateAdvice = function () {
if (me.destroyed || !me.storage) {
return;
@@ -79,7 +86,12 @@ Ext.define('PVE.window.SANLunState', {
let text = [
added,
waiting &&
- Ext.String.format(gettext('Waiting for a LUN report from: {0}.'), waiting),
+ Ext.String.format(
+ me.nvme
+ ? gettext('Waiting for a namespace report from: {0}.')
+ : gettext('Waiting for a LUN report from: {0}.'),
+ waiting,
+ ),
missing && Ext.String.format(missingAdvice, missing),
];
me.down('#advice').setHtml(Ext.htmlEncode(text.filter(Boolean).join(' ')));
@@ -97,7 +109,9 @@ Ext.define('PVE.window.SANLunState', {
flex: 1,
store,
disableSelection: true,
- emptyText: gettext('No node reported this LUN yet.'),
+ emptyText: me.nvme
+ ? gettext('No node reported this namespace yet.')
+ : gettext('No node reported this LUN yet.'),
columns: [
{
text: gettext('Node'),
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (13 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
` (2 subsequent siblings)
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
Add a view below Node > SAN with the controllers of a node, one per
connected portal, with their transport, state and storage, and the host
NQN of the node. A controller can be disconnected there, for example
after its portal was removed from the storage, which the node refuses
for the last usable path of a namespace in use. The controllers come
from sysfs, so the view lists them also without nvme-cli, which only
Disconnect needs.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/Makefile | 1 +
www/manager6/SANUtils.js | 5 +
www/manager6/Utils.js | 1 +
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 152 +++++++++++++++++++++++++
5 files changed, 167 insertions(+), 1 deletion(-)
create mode 100644 www/manager6/node/NVMeOFControllers.js
diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index f2fc80279..f3591e1c8 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -253,6 +253,7 @@ JSSRC= \
node/ISCSITargets.js \
node/LVM.js \
node/LVMThin.js \
+ node/NVMeOFControllers.js \
node/SANLuns.js \
node/StatusView.js \
node/Subscription.js \
diff --git a/www/manager6/SANUtils.js b/www/manager6/SANUtils.js
index d28285d22..abff7f458 100644
--- a/www/manager6/SANUtils.js
+++ b/www/manager6/SANUtils.js
@@ -132,6 +132,11 @@ Ext.define('PVE.SANUtils', {
'logged-in': ['check-circle good', gettext('Connected')],
recovering: ['exclamation-circle warning', gettext('Reconnecting')],
'not-connected': ['times-circle faded', gettext('Not connected')],
+ live: ['check-circle good', gettext('Connected')],
+ connecting: ['exclamation-circle warning', gettext('Reconnecting')],
+ resetting: ['exclamation-circle warning', gettext('Reconnecting')],
+ deleting: ['times-circle faded', gettext('Disconnecting')],
+ 'deleting (no IO)': ['times-circle faded', gettext('Disconnecting')],
}[state];
let [icon, label] = display ?? ['question-circle faded', state];
return `<i class="fa fa-fw fa-${icon}"></i> ${Ext.htmlEncode(text ?? label)}`;
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index d508d0469..fc3667f51 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -2276,6 +2276,7 @@ Ext.define('PVE.Utils', {
lvmthinremove: ['Thinpool', gettext('Remove')],
migrateall: ['', gettext('Bulk migrate VMs and Containers')],
move_volume: ['CT', gettext('Move Volume')],
+ nvmedisconnect: ['NVMe-oF', gettext('Disconnect')],
'pbs-download': ['VM/CT', gettext('File Restore Download')],
pull_file: ['CT', gettext('Pull file')],
push_file: ['CT', gettext('Push file')],
diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
index c85183766..830e8542a 100644
--- a/www/manager6/node/Config.js
+++ b/www/manager6/node/Config.js
@@ -398,7 +398,7 @@ Ext.define('PVE.node.Config', {
);
}
- // the LUN and iSCSI reads of the node accept either privilege
+ // the LUN, iSCSI and NVMe over Fabrics reads of the node accept either privilege
if (caps.nodes['Sys.Audit'] || caps.storage['Datastore.Allocate']) {
me.items.push(
{
@@ -414,6 +414,13 @@ Ext.define('PVE.node.Config', {
iconCls: 'fa fa-bullseye',
groups: ['san'],
},
+ {
+ xtype: 'pveNVMeOFControllerList',
+ title: 'NVMe over Fabrics',
+ itemId: 'san-nvmeof',
+ iconCls: 'fa fa-plug',
+ groups: ['san'],
+ },
);
}
diff --git a/www/manager6/node/NVMeOFControllers.js b/www/manager6/node/NVMeOFControllers.js
new file mode 100644
index 000000000..cad9b4cbc
--- /dev/null
+++ b/www/manager6/node/NVMeOFControllers.js
@@ -0,0 +1,152 @@
+Ext.define('PVE.node.NVMeOFControllerList', {
+ extend: 'Ext.grid.Panel',
+ xtype: 'pveNVMeOFControllerList',
+ onlineHelp: 'storage_nvmeof',
+ emptyText: gettext('No NVMe over Fabrics controllers found'),
+ viewConfig: { enableTextSelection: true },
+ store: {
+ fields: [
+ 'controller',
+ 'transport',
+ 'subsysnqn',
+ 'portal',
+ 'state',
+ 'namespaces',
+ 'used-by-storage',
+ ],
+ sorters: ['subsysnqn', 'portal'],
+ },
+ columns: [
+ { text: gettext('Controller'), dataIndex: 'controller', width: 100 },
+ {
+ text: gettext('Subsystem'),
+ dataIndex: 'subsysnqn',
+ flex: 5,
+ renderer: Ext.htmlEncode,
+ },
+ {
+ text: gettext('Transport'),
+ dataIndex: 'transport',
+ width: 90,
+ // the transport of the controller, named as for the paths of a namespace
+ renderer: (value) => PVE.SANUtils.renderSanTransport(value && `nvme-${value}`),
+ },
+ { text: gettext('Portal'), dataIndex: 'portal', width: 180, renderer: Ext.htmlEncode },
+ {
+ text: gettext('State'),
+ dataIndex: 'state',
+ width: 150,
+ renderer: PVE.SANUtils.renderState,
+ },
+ { text: gettext('Namespaces'), dataIndex: 'namespaces', width: 110 },
+ {
+ text: gettext('Storage'),
+ dataIndex: 'used-by-storage',
+ flex: 1,
+ renderer: Ext.htmlEncode,
+ },
+ ],
+
+ reload: function () {
+ let me = this;
+ let load = (path, success) =>
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/san/nvmeof/${path}`,
+ method: 'GET',
+ waitMsgTarget: me.getView(),
+ failure: (response) =>
+ Proxmox.Utils.setErrorMask(me.getView(), response.htmlStatus),
+ success: (response) => !me.destroyed && success(response.result.data),
+ });
+ load('status', function (status) {
+ me.down('#hostnqn').setValue(status['host-nqn'] || '');
+ me.supported = !!status.supported;
+ let error = status.supported
+ ? status.error
+ : gettext('nvme-cli is not installed on this node.');
+ // the controllers come from sysfs, which needs neither nvme-cli nor a host identity
+ let warning = me.down('#warning');
+ warning.setHtml(
+ error
+ ? `<i class="fa fa-fw fa-exclamation-triangle warning"></i> ${Ext.htmlEncode(error)}`
+ : '',
+ );
+ warning.setHidden(!error);
+ load('controller', function (controllers) {
+ Proxmox.Utils.setErrorMask(me.getView(), false);
+ me.getStore().setData(controllers);
+ });
+ });
+ },
+
+ listeners: {
+ activate: function () {
+ this.reload();
+ },
+ },
+
+ initComponent: function () {
+ let me = this;
+ me.nodename = me.pveSelNode.data.node;
+ me.tbar = [
+ { text: gettext('Reload'), iconCls: 'fa fa-refresh', handler: () => me.reload() },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Disconnect'),
+ iconCls: 'fa fa-unlink',
+ disabled: true,
+ dangerous: true,
+ // the disconnect uses nvme-cli
+ enableFn: () => !!Ext.state.Manager.get('GuiCap').dc['Sys.Modify'] && me.supported,
+ confirmMsg: (rec) =>
+ Ext.String.format(
+ gettext(
+ 'Disconnect controller {0} from portal {1}? A storage that still lists that portal connects to it again.',
+ ),
+ Ext.htmlEncode(rec.get('controller')),
+ Ext.htmlEncode(rec.get('portal')),
+ ),
+ handler: (button, event, rec) =>
+ Proxmox.Utils.API2Request({
+ // refused if the kernel reused the controller number meanwhile
+ url:
+ `/nodes/${me.nodename}/san/nvmeof/controller/${rec.get('controller')}?` +
+ Ext.Object.toQueryString({
+ subsysnqn: rec.get('subsysnqn'),
+ portal: rec.get('portal'),
+ }),
+ method: 'DELETE',
+ waitMsgTarget: me,
+ failure: (response) => Ext.Msg.alert(gettext('Error'), response.htmlStatus),
+ success: (response) =>
+ Ext.create('Proxmox.window.TaskProgress', {
+ upid: response.result.data,
+ autoShow: true,
+ taskDone: () => !me.destroyed && me.reload(),
+ }),
+ }),
+ },
+ '->',
+ {
+ xtype: 'displayfield',
+ itemId: 'hostnqn',
+ fieldLabel: gettext('Host NQN'),
+ },
+ {
+ iconCls: 'fa fa-clipboard',
+ tooltip: gettext('Copy to Clipboard'),
+ handler: () => PVE.Utils.copyToClipboard(me.down('#hostnqn').getValue()),
+ },
+ ];
+ me.dockedItems = [
+ {
+ xtype: 'component',
+ itemId: 'warning',
+ dock: 'top',
+ padding: 5,
+ hidden: true,
+ },
+ ];
+ me.callParent();
+ },
+});
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (14 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The access rules of an array need the host NQN of every node, like the
initiator names for iSCSI. Show them next to those in the node list of
the SAN view, with a button to copy them all.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 29 +++++++++++++++++++++--------
1 file changed, 21 insertions(+), 8 deletions(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index 2f624cbd2..e30b5ef48 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -10,6 +10,7 @@ Ext.define('pve-san-status', {
'time',
'iscsi-initiator-name',
'other-maps',
+ 'nvme-host-nqn',
'wwid',
'name',
'explicit',
@@ -98,6 +99,14 @@ Ext.define('PVE.dc.SANView', {
sortAfterUpdate: true,
});
let reload = () => me.rstore.load();
+ let copy = (field) => () =>
+ PVE.Utils.copyToClipboard(
+ nodeStore
+ .getRange()
+ .map((rec) => rec.get(field))
+ .filter(Boolean)
+ .join('\n'),
+ );
let edit = ({ wwid, isCreate }) =>
Ext.create('PVE.dc.MultipathWwidEdit', {
wwid,
@@ -144,14 +153,12 @@ Ext.define('PVE.dc.SANView', {
{
text: gettext('Copy Initiator Names'),
iconCls: 'fa fa-clipboard',
- handler: () =>
- PVE.Utils.copyToClipboard(
- nodeStore
- .getRange()
- .map((rec) => rec.get('iscsi-initiator-name'))
- .filter(Boolean)
- .join('\n'),
- ),
+ handler: copy('iscsi-initiator-name'),
+ },
+ {
+ text: gettext('Copy Host NQNs'),
+ iconCls: 'fa fa-clipboard',
+ handler: copy('nvme-host-nqn'),
},
],
columns: [
@@ -183,6 +190,12 @@ Ext.define('PVE.dc.SANView', {
flex: 2,
renderer: Ext.htmlEncode,
},
+ {
+ text: gettext('NVMe Host NQN'),
+ dataIndex: 'nvme-host-nqn',
+ flex: 4,
+ renderer: Ext.htmlEncode,
+ },
{
text: gettext('Other Maps'),
tooltip: gettext(
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (15 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The cluster-wide status lists the namespaces of NVMe over Fabrics
storages too. They use the native multipath of the kernel, with
neither an entry nor a storage that requests a map, so the Multipath
column would call them unmanaged. Show "Native" for them, as the LUN
list of a node does, and name them as namespaces in Paths by Node.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
www/manager6/dc/SAN.js | 16 ++++++++++++++--
www/manager6/window/SANLunState.js | 8 ++++----
2 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/www/manager6/dc/SAN.js b/www/manager6/dc/SAN.js
index e30b5ef48..fdf2935bf 100644
--- a/www/manager6/dc/SAN.js
+++ b/www/manager6/dc/SAN.js
@@ -15,6 +15,7 @@ Ext.define('pve-san-status', {
'name',
'explicit',
'multipath-storage',
+ 'native',
'nodes',
'size',
'used-by-storage',
@@ -118,6 +119,7 @@ Ext.define('PVE.dc.SANView', {
Ext.create('PVE.window.SANLunState', {
wwid: rec.get('wwid'),
name: rec.get('name'),
+ nvme: !!rec.get('native'),
autoShow: true,
});
let setupItem = (type) => ({
@@ -258,7 +260,11 @@ Ext.define('PVE.dc.SANView', {
disabled: !canModify,
handler: function (button) {
let lun = button.up('grid').getSelection()[0];
- let wwid = lun?.get('explicit') ? undefined : lun?.get('wwid');
+ // a namespace uses native multipath, which takes no entry
+ let wwid =
+ lun?.get('explicit') || lun?.get('native')
+ ? undefined
+ : lun?.get('wwid');
edit({ wwid, isCreate: true });
},
},
@@ -381,7 +387,13 @@ Ext.define('PVE.dc.SANView', {
width: 170,
renderer: (value, md, rec) => {
let storage = rec.get('multipath-storage');
- let kind = value ? 'manual' : storage ? 'storage' : 'unmanaged';
+ let kind = rec.get('native')
+ ? 'native'
+ : value
+ ? 'manual'
+ : storage
+ ? 'storage'
+ : 'unmanaged';
return Ext.htmlEncode(PVE.SANUtils.multipathText(kind, storage));
},
},
diff --git a/www/manager6/window/SANLunState.js b/www/manager6/window/SANLunState.js
index 5dfce247a..f84940fd8 100644
--- a/www/manager6/window/SANLunState.js
+++ b/www/manager6/window/SANLunState.js
@@ -23,12 +23,12 @@ Ext.define('PVE.window.SANLunState', {
let title = me.nvme
? gettext('Storage {0}: Namespace State')
: gettext('Storage {0}: LUN State');
+ let pathsTitle = me.nvme
+ ? gettext('Namespace {0}: Paths by Node')
+ : gettext('LUN {0}: Paths by Node');
me.title = me.storage
? Ext.String.format(title, me.storage)
- : Ext.String.format(
- gettext('LUN {0}: Paths by Node'),
- Ext.htmlEncode(me.name || me.wwid),
- );
+ : Ext.String.format(pathsTitle, Ext.htmlEncode(me.name || me.wwid));
// The other nodes set up the LUN of a new storage with their next status updates.
let pendingUntil = me.storage ? Date.now() + 60000 : 0;
let rstore = Ext.create('Proxmox.data.UpdateStore', {
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
` (16 preceding siblings ...)
2026-11-22 1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
@ 2026-11-22 1:25 ` Thomas Lamprecht
17 siblings, 0 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-22 1:25 UTC (permalink / raw)
To: pve-devel
The report of a node lists the namespaces of its NVMe over Fabrics
storages next to the multipath maps, so the notifications about lost
and restored paths cover them too. Name the storages of a namespace in
them as for a map, also the one that has it as base volume.
A report whose namespaces could not be listed counts as incomplete, so
that they are not taken as gone. A node that cannot query multipathd
sends no notifications, also not for its namespaces, which its report
still lists.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
PVE/MultipathNotification.pm | 5 +++--
PVE/Service/pvestatd.pm | 2 +-
test/MultipathNotification_test.pl | 3 +++
3 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/PVE/MultipathNotification.pm b/PVE/MultipathNotification.pm
index 99c79405b..d25851d0c 100644
--- a/PVE/MultipathNotification.pm
+++ b/PVE/MultipathNotification.pm
@@ -97,8 +97,9 @@ sub flush($self, $notify) {
sub update($self, $report, $notify) {
my $usable = ref($report) eq 'HASH' && !$report->{error} && !$report->{'scope-unknown'};
my $luns = $usable ? $report->{luns} : undef;
- # without the hints of LVM, maps known only by their volume group can be left out
- my $complete = $usable && !$report->{'vgs-unknown'};
+ # without the hints of LVM, maps known only by their volume group can be left out, and a failed
+ # query leaves out the NVMe namespaces
+ my $complete = $usable && !$report->{'vgs-unknown'} && !$report->{'namespaces-unknown'};
my $previous = $self->{previous};
my $current = $self->{previous} = {};
if (!defined($luns)) {
diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 68d7f2e08..5cdce325a 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -814,7 +814,7 @@ my sub notify_multipath {
my $scfg = PVE::Storage::storage_check_node($storage, $sid, $nodename, 1) // next;
push $by_vg{ $scfg->{vgname} }->@*, $sid if defined($scfg->{vgname});
my $base = PVE::Multipath::parse_base_volume($scfg->{base});
- push $by_wwid{ $base->{wwid} }->@*, $sid if $base && $base->{kind} ne 'nvme';
+ push $by_wwid{ $base->{wwid} }->@*, $sid if $base;
}
my $data = PVE::Notify::common_template_data();
diff --git a/test/MultipathNotification_test.pl b/test/MultipathNotification_test.pl
index f78a31e6e..d61ce6ea1 100755
--- a/test/MultipathNotification_test.pl
+++ b/test/MultipathNotification_test.pl
@@ -93,6 +93,9 @@ pass_with(
$tracker = PVE::MultipathNotification->new($file);
pass_with({}, [], 'notified LUNs left the report across a restart');
pass_with({}, [], 'but an incomplete report does not count', 0);
+report_with(
+ { luns => [], 'namespaces-unknown' => 1 }, [], 'nor one without the NVMe namespaces',
+);
pass_with(
{},
[['info', 'c:gone', 'd:gone']],
^ permalink raw reply related [flat|nested] 19+ messages in thread
end of thread, other threads:[~2026-11-22 1:29 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-22 1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox