From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v6 storage 05/42] diskmanage: collect native NVMe multipath path state
Date: Wed, 18 Nov 2026 07:52:22 +0100 [thread overview]
Message-ID: <20261118065302.3033937-6-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118065302.3033937-1-t.lamprecht@proxmox.com>
From: Dietmar Maurer <dietmar@proxmox.com>
NVMe over Fabrics usually uses native multipath, whose paths
multipathd does not know. Collect the controller state and address
and the ANA state of each path from sysfs, grouped by namespace, so
the SAN LUN inventory can report path health for both multipath
stacks. Devices with a single local controller have no per-path
entries and are left out.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
[TL: drop redundant comments; call the wrapped test function with
arrow syntax]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Diskmanage.pm | 60 +++++++++++++++++++
.../nvme-subsystem/nvme-subsys0/nvme3/address | 1 +
.../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 +
.../nvme-subsystem/nvme-subsys0/nvme3/state | 1 +
.../nvme-subsystem/nvme-subsys0/nvme4/address | 1 +
.../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 +
.../nvme-subsystem/nvme-subsys0/nvme4/state | 1 +
.../nvme-subsystem/nvme-subsys1/nvme0/address | 1 +
.../nvme-subsystem/nvme-subsys1/nvme0/state | 1 +
.../multipath/nvme_path_status_expected.json | 19 ++++++
src/test/disklist_test.pm | 16 +++++
11 files changed, 103 insertions(+)
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/address
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/nvme2c3n1/ana_state
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/state
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/address
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/nvme2c4n1/ana_state
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/state
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/address
create mode 100644 src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/state
create mode 100644 src/test/disk_tests/multipath/nvme_path_status_expected.json
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index c342abcd..516c108d 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -818,6 +818,66 @@ sub get_disks {
return $disklist;
}
+# Native multipath creates per-path namespace children nvmeXcYnZ below controllers. Local
+# single-controller devices lack these children and are omitted.
+sub get_nvme_path_status {
+ my $res = {};
+
+ dir_glob_foreach(
+ '/sys/class/nvme-subsystem',
+ 'nvme-subsys\d+',
+ sub {
+ my ($subsys) = @_;
+ my $subsysdir = "/sys/class/nvme-subsystem/$subsys";
+
+ dir_glob_foreach(
+ $subsysdir,
+ 'nvme\d+',
+ sub {
+ my ($ctrl) = @_;
+ my $ctrldir = "$subsysdir/$ctrl";
+ my $state = file_read_firstline("$ctrldir/state") // return;
+ my $address = file_read_firstline("$ctrldir/address");
+
+ dir_glob_foreach(
+ $ctrldir,
+ 'nvme\d+c\d+n\d+',
+ sub {
+ my ($pathdev) = @_;
+ return if $pathdev !~ m/^nvme(\d+)c\d+n(\d+)$/;
+ my $head = "nvme$1n$2";
+
+ my $entry = { device => "/dev/$ctrl", state => $state };
+ $entry->{address} = $address
+ if defined($address) && $address ne '';
+ my $ana = file_read_firstline("$ctrldir/$pathdev/ana_state");
+ $entry->{'ana-state'} = $ana if defined($ana);
+
+ # 'change' is transient; paths in 'inaccessible' or 'persistent-loss'
+ # cannot serve IO even on a live controller
+ my $usable = $state eq 'live'
+ && (!defined($ana)
+ || $ana eq 'optimized'
+ || $ana eq 'non-optimized');
+
+ my $status = $res->{$head} //= { active => 0, paths => [] };
+ $status->{active}++ if $usable;
+ push @{ $status->{paths} }, $entry;
+ },
+ );
+ },
+ );
+ },
+ );
+
+ for my $status (values %$res) {
+ @{ $status->{paths} } =
+ sort { $a->{device} cmp $b->{device} } @{ $status->{paths} };
+ }
+
+ return $res;
+}
+
sub get_partnum {
my ($part_path) = @_;
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/address b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/address
new file mode 100644
index 00000000..803fab3e
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/address
@@ -0,0 +1 @@
+traddr=192.168.3.209,trsvcid=4420,src_addr=192.168.2.171
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/nvme2c3n1/ana_state b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/nvme2c3n1/ana_state
new file mode 100644
index 00000000..e48c58f5
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/nvme2c3n1/ana_state
@@ -0,0 +1 @@
+optimized
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/state b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/state
new file mode 100644
index 00000000..e23fe645
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme3/state
@@ -0,0 +1 @@
+live
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/address b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/address
new file mode 100644
index 00000000..36618744
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/address
@@ -0,0 +1 @@
+traddr=192.168.3.109,trsvcid=4420,src_addr=192.168.2.171
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/nvme2c4n1/ana_state b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/nvme2c4n1/ana_state
new file mode 100644
index 00000000..436c48ee
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/nvme2c4n1/ana_state
@@ -0,0 +1 @@
+inaccessible
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/state b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/state
new file mode 100644
index 00000000..e23fe645
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys0/nvme4/state
@@ -0,0 +1 @@
+live
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/address b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/address
new file mode 100644
index 00000000..f99fcb14
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/address
@@ -0,0 +1 @@
+0000:02:00.0
diff --git a/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/state b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/state
new file mode 100644
index 00000000..e23fe645
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme-subsystem/nvme-subsys1/nvme0/state
@@ -0,0 +1 @@
+live
diff --git a/src/test/disk_tests/multipath/nvme_path_status_expected.json b/src/test/disk_tests/multipath/nvme_path_status_expected.json
new file mode 100644
index 00000000..62611b3f
--- /dev/null
+++ b/src/test/disk_tests/multipath/nvme_path_status_expected.json
@@ -0,0 +1,19 @@
+{
+ "nvme2n1": {
+ "active": 1,
+ "paths": [
+ {
+ "device": "/dev/nvme3",
+ "state": "live",
+ "address": "traddr=192.168.3.209,trsvcid=4420,src_addr=192.168.2.171",
+ "ana-state": "optimized"
+ },
+ {
+ "device": "/dev/nvme4",
+ "state": "live",
+ "address": "traddr=192.168.3.109,trsvcid=4420,src_addr=192.168.2.171",
+ "ana-state": "inaccessible"
+ }
+ ]
+ }
+}
diff --git a/src/test/disklist_test.pm b/src/test/disklist_test.pm
index 4c85d9e9..a7c00d2d 100644
--- a/src/test/disklist_test.pm
+++ b/src/test/disklist_test.pm
@@ -103,6 +103,7 @@ sub mocked_file_read_firstline {
my $originalsub = $diskmanage_module->original('file_read_firstline');
$path =~ s|^/sys/block|disk_tests/$testcasedir|;
+ $path =~ s|^/sys/class/nvme-subsystem|disk_tests/$testcasedir/nvme-subsystem|;
return $originalsub->($path);
}
@@ -112,6 +113,12 @@ sub mocked_dir_glob_foreach {
my $lines = [];
+ # nvme subsystem trees are provided as directories, glob them like production code
+ if ($dir =~ s|^/sys/class/nvme-subsystem|disk_tests/$testcasedir/nvme-subsystem|) {
+ my $originalsub = $diskmanage_module->original('dir_glob_foreach');
+ return $originalsub->($dir, $regex, $sub);
+ }
+
# read lines in from file
if ($dir =~ m{^/sys/block$}) {
@$lines = split(/\n/, read_test_file('disklist'));
@@ -230,6 +237,15 @@ sub test_disk_list {
ok($san_disks->{sda}, 'SAN inventory opts into iSCSI');
}
+ if (-f "disk_tests/$testcasedir/nvme_path_status_expected.json") {
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my $expected_status = decode_json(read_test_file('nvme_path_status_expected.json'));
+
+ print Dumper($status) if $print;
+ $testcount++;
+ is_deeply($status, $expected_status, 'nvme path status should be the same');
+ }
+
done_testing($testcount);
};
}
next prev parent reply other threads:[~2026-11-18 6:53 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 6:52 [PATCH v6 cluster,storage,docs,manager 00/42] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 cluster 01/42] cfs: observe the multipath configuration files Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 02/42] diskmanage: collect disk transport type from lsblk Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 03/42] diskmanage: qualify NVMe over fabrics transport Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 04/42] diskmanage: optionally include iSCSI session devices in disk enumeration Thomas Lamprecht
2026-11-18 6:52 ` Thomas Lamprecht [this message]
2026-11-18 6:52 ` [PATCH v6 storage 06/42] iscsi: factor out session device map from device list Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 07/42] multipath: add a library around multipathd Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 08/42] diskmanage: add helper to list multipath devices Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 09/42] multipath: add a cluster-wide configuration Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 10/42] iscsi: add an option to manage storage LUNs with multipath Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 11/42] lvm: activate a missing base volume before using the volume group Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 12/42] multipath: apply the configuration on every node Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 13/42] multipath: publish per-node status Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 14/42] api: add endpoints for the cluster-wide multipath configuration Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 15/42] api: multipath: add the cluster-wide health status Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 16/42] diskmanage: look up the requested disks by name Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 17/42] diskmanage: read the udev data of all disks with one call Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 18/42] api: scan: add a SAN LUN inventory Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 19/42] pvesm: expose the " Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 20/42] api: add node-level iSCSI status Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 21/42] disks: lvm: drop unused imports Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 22/42] disks: lvm: allow volume groups on SAN LUNs and multipath maps Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 23/42] disks: lvm: refuse to destroy the volume group of a shared storage Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 24/42] disks: allow wiping a SAN LUN that no storage uses Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 storage 25/42] api: scan: report the process that has a SAN LUN open Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 docs 26/42] storage: document the cluster-wide multipath management Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 docs 27/42] storage: multipath: describe the use shown for an open LUN Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 28/42] pvestatd: apply the multipath configuration and report the node status Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 29/42] report: add the managed multipath configuration Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 30/42] api: cluster: add the multipath endpoints Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 31/42] api: nodes: expose the iSCSI status Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 32/42] ui: dc: add a SAN view Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 33/42] ui: storage: iscsi: add the multipath option Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 34/42] ui: lvm: enable shared when a base storage is selected Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 35/42] ui: node: add SAN LUNs panel Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 36/42] ui: node: san: wipe a LUN that no storage uses Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 37/42] ui: node: san: show iSCSI targets and sessions Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 38/42] ui: dc: san: add a guided setup for shared LVM on iSCSI Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 39/42] ui: dc: san: add a guided setup for shared LVM on Fibre Channel Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 40/42] ui: storage: point the iSCSI and LVM add dialogs to the SAN view Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 41/42] ui: san: show the guest or process that has a LUN open Thomas Lamprecht
2026-11-18 6:52 ` [PATCH v6 manager 42/42] ui: node: san: add a column for the iSCSI target of a LUN Thomas Lamprecht
2026-11-18 20:28 ` superseded: [PATCH v6 cluster,storage,docs,manager 00/42] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118065302.3033937-6-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox