From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status
Date: Wed, 18 Nov 2026 21:42:28 +0100 [thread overview]
Message-ID: <20261118204302.3100813-10-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com>
The cluster-wide status lists the LUNs that multipathd handles on
each node. The namespaces of an NVMe over Fabrics storage use the
native multipath of the kernel and were missing from it, so a lost
path or a node without access to a namespace only showed in the LUN
list of that node.
Let each node report the namespaces of the subsystems such a storage
names, with the path counts its LUN list already shows, and expect
them on the nodes of the storage like the LUNs of an iSCSI storage.
They do not depend on multipathd, so a node without it reports them
as well and only the state of its maps stays unknown.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Multipath.pm | 11 +++-
src/PVE/Multipath/Status.pm | 58 +++++++++++++++---
src/test/run_multipath_api_tests.pl | 95 ++++++++++++++++++++++++++++-
3 files changed, 152 insertions(+), 12 deletions(-)
diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index e69919c2..51ef7281 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -130,14 +130,21 @@ __PACKAGE__->register_method({
wwid => { type => 'string', description => 'The LUN WWID.' },
explicit =>
{ type => 'boolean', description => 'An explicit entry exists.' },
+ native => {
+ type => 'boolean',
+ optional => 1,
+ description => 'An NVMe namespace, which uses the native'
+ . ' multipath of the kernel.',
+ },
name => { type => 'string', optional => 1 },
'used-by-storage' => {
type => 'string',
optional => 1,
description =>
'The storage that uses the LUN: as its base volume or through'
- . ' the volume group on it, or as an iSCSI storage that serves'
- . ' the LUNs of its target as images.',
+ . ' the volume group on it, or as an iSCSI or NVMe over Fabrics'
+ . ' storage that serves the LUNs of its target or subsystem as'
+ . ' images.',
},
'multipath-storage' => {
type => 'string',
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index ae386164..d26eb7f8 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -7,6 +7,7 @@ use JSON qw(decode_json);
use PVE::Cluster;
use PVE::INotify;
+use PVE::Diskmanage;
use PVE::Multipath;
use PVE::Multipath::Config;
use PVE::Multipath::Generator;
@@ -24,6 +25,39 @@ sub worst_state(@states) {
return $worst;
}
+# Namespaces use the native multipath of the kernel, so their state does not depend on multipathd.
+my sub namespace_luns($pv_vgs) {
+ require PVE::Storage;
+ my $cfg = PVE::Storage::config();
+ my $status = PVE::Diskmanage::get_nvme_path_status();
+ my %luns;
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ my $scfg = $cfg->{ids}->{$sid};
+ next if $scfg->{type} ne 'nvmeof';
+ my $expected = eval { scalar(PVE::NVMeOF::storage_paths($scfg)->@*) };
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ for my $name (sort keys %$namespaces) {
+ my $wwid = $namespaces->{$name}->{wwid};
+ next if $luns{$wwid};
+ my $paths = $status->{$name} // { active => 0, paths => [] };
+ my $total = scalar($paths->{paths}->@*);
+ my $lun = $luns{$wwid} = {
+ wwid => $wwid,
+ native => 1,
+ size => $namespaces->{$name}->{size},
+ storages => [$sid],
+ 'paths-total' => $total,
+ 'paths-active' => $paths->{active},
+ state => PVE::Multipath::derive_health($paths->{active}, $total, $expected),
+ };
+ $lun->{'paths-expected'} = $expected if defined($expected);
+ my $vg = ($pv_vgs // {})->{"/dev/$name"};
+ $lun->{vgname} = $vg if defined($vg);
+ }
+ }
+ return [map { $luns{$_} } sort keys %luns];
+}
+
sub publish($report) {
my $node = PVE::INotify::nodename();
my $now = time();
@@ -55,11 +89,11 @@ sub update_node_status() {
require PVE::NVMeOF;
my $host = eval { PVE::NVMeOF::host_identity() };
$report->{'nvme-host-nqn'} = $host->{nqn} if $host;
+ require PVE::Storage::LVMPlugin;
+ my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
eval {
die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported();
die "multipathd is not running or does not respond\n" if !PVE::Multipath::is_running();
- require PVE::Storage::LVMPlugin;
- my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups();
# A node can have many maps that something else set up, for example a storage plugin with
# one LUN per guest disk. Only the maps that the node manages, that are the base volume of a
# storage or hold its volume group get a row, the others are counted.
@@ -119,8 +153,11 @@ sub update_node_status() {
if (my $error = $@) {
chomp $error;
$report->{error} = $error;
- delete $report->{luns};
+ $report->{luns} = [];
}
+ my $namespaces = eval { namespace_luns($pv_vgs) };
+ warn "multipath: cannot query NVMe namespaces - $@" if $@;
+ push $report->{luns}->@*, @$namespaces if $namespaces;
eval { publish($report) };
warn "multipath: cannot publish node status - $@" if $@;
}
@@ -133,7 +170,7 @@ sub cluster_status() {
my $all_nodes = PVE::Cluster::get_nodelist();
my $members = PVE::Cluster::get_members();
my $raw = PVE::Cluster::get_node_kv($KEY);
- my (%nodes, %luns, %expected, %observed, %vg_owner);
+ my (%nodes, %luns, %expected, %observed, %vg_owner, %lists);
# the nodes that list a map because the configuration names it for them
my (%managed, %unhinted);
my $storage_nodes = sub($scfg) {
@@ -158,12 +195,13 @@ sub cluster_status() {
my $scfg = $storage->{ids}->{$sid};
$vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
next if $scfg->{type} ne 'lvm';
- my ($wwid) = ($scfg->{base} // '') =~ /(?:^|\.)dm-uuid-(?:part\d+-)?mpath-(\S+)$/;
+ my ($base, $volname) = PVE::Storage::parse_volume_id($scfg->{base} // '', 1);
+ my ($kind, $wwid) = ($volname // '') =~ /(?:^|\.)(dm-uuid-(?:part\d+-)?mpath|nvme)-(\S+)$/;
next if !defined($wwid);
$luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $luns{$wwid}->{native} = 1 if $kind eq 'nvme';
$luns{$wwid}->{'used-by-storage'} //= $sid;
$managed{$wwid}->{$_} = 1 for @$all_nodes;
- my ($base) = $scfg->{base} =~ /^([^:]+):/;
$luns{$wwid}->{'multipath-storage'} //= $base if $requests_maps->($base);
$expected{$wwid}->{$_} = 1 for $storage_nodes->($scfg)->@*;
}
@@ -178,11 +216,13 @@ sub cluster_status() {
$entry->{$_} = $report->{$_} for grep { defined($report->{$_}) } @keys;
$entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
$unhinted{$node} = 1 if $report->{'vgs-unknown'};
- next if $entry->{state} ne 'ok';
+ # an unavailable node lists its namespaces only, or nothing
+ $lists{$node} = 1 if $report->{luns};
for my $lun (($report->{luns} // [])->@*) {
my $wwid = $lun->{wwid} // next;
my $row = $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+ $row->{native} = 1 if $lun->{native};
$row->{$_} //= $lun->{$_} for grep { defined($lun->{$_}) } qw(name size);
my $owner = defined($lun->{vgname}) ? $vg_owner{ $lun->{vgname} } : undef;
$row->{'used-by-storage'} //= $owner if defined($owner);
@@ -216,10 +256,10 @@ sub cluster_status() {
if ($observed{$wwid}->{$node}) {
$row->{nodes}->{$node} = $observed{$wwid}->{$node};
} elsif ($expected{$wwid}->{$node}) {
- my $listed = $nodes{$node}->{state} eq 'ok';
+ my $listed = $nodes{$node}->{state} eq 'ok' || ($row->{native} && $lists{$node});
# without the hint file of LVM a node leaves a map out that it only knows through
# the volume group of a storage
- $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node};
+ $listed = 0 if $unhinted{$node} && !$managed{$wwid}->{$node} && !$row->{native};
$row->{nodes}->{$node} = { state => $listed ? 'missing' : 'unknown' };
}
}
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index 078ba66d..fd63503a 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -383,7 +383,12 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
my $status = Test::MockModule->new('PVE::Multipath::Status');
my $mp = Test::MockModule->new('PVE::Multipath');
my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $disks = Test::MockModule->new('PVE::Diskmanage');
my ($report, @order);
+ my $ids = {};
+ $storage->mock(config => sub { { ids => $ids } });
+ $disks->mock(get_nvme_path_status => sub { { nvme1n1 => { active => 1, paths => [{}] } } });
$gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } });
$status->mock(publish => sub { push @order, 'publish'; $report = $_[0] });
$iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } });
@@ -421,6 +426,42 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
is($report->{luns}->[0]->{'paths-expected'}, 3, 'expected count reported');
is($report->{luns}->[0]->{state}, 'degraded', 'fewer paths than expected');
ok(!exists($report->{supported}), 'no separate capability state');
+
+ $ids->{nvme} = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test:sub',
+ portals => '192.0.2.1,192.0.2.2',
+ };
+ $nvme->mock(
+ namespaces => sub($nqn) {
+ return $nqn eq 'nqn.test:sub'
+ ? { nvme1n1 => { wwid => 'uuid.1', size => 4096 } }
+ : {};
+ },
+ );
+ my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+ $lvm->mock(cached_pv_volume_groups => sub { { '/dev/nvme1n1' => 'vgns' } });
+ my $namespace = {
+ wwid => 'uuid.1',
+ native => 1,
+ size => 4096,
+ storages => ['nvme'],
+ vgname => 'vgns',
+ 'paths-total' => 1,
+ 'paths-active' => 1,
+ 'paths-expected' => 2,
+ state => 'degraded',
+ };
+ PVE::Multipath::Status::update_node_status();
+ is_deeply($report->{luns}->[1], $namespace, 'namespace with fewer paths than portals');
+ $mp->mock(is_running => sub { 0 });
+ PVE::Multipath::Status::update_node_status();
+ is(
+ $report->{error},
+ 'multipathd is not running or does not respond',
+ 'stopped daemon still reported',
+ );
+ is_deeply($report->{luns}, [$namespace], 'namespaces do not need multipathd');
};
subtest 'API returns final node and LUN states' => sub {
@@ -457,12 +498,31 @@ subtest 'API returns final node and LUN states' => sub {
'paths-active' => 2,
storages => ['scoped'],
},
+ {
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'optimal',
+ 'paths-total' => 2,
+ 'paths-active' => 2,
+ storages => ['nvme'],
+ },
],
}),
n2 => encode_json({ time => $NOW, luns => [], 'nvme-host-nqn' => 'nqn.test:n2' }),
n3 => encode_json({ time => $NOW - 181, luns => [] }),
n4 => encode_json({ time => $NOW, luns => [] }),
- n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
+ n5 => encode_json({
+ time => $NOW,
+ error => 'multipath-tools is not installed',
+ luns => [{
+ wwid => 'uuid.1',
+ native => 1,
+ state => 'degraded',
+ 'paths-total' => 2,
+ 'paths-active' => 1,
+ storages => ['nvme'],
+ }],
+ }),
);
$cluster->mock(get_node_kv => sub { \%raw });
my $entries = { '3600a' => { type => 'wwid' } };
@@ -480,6 +540,8 @@ subtest 'API returns final node and LUN states' => sub {
{ type => 'iscsi', 'managed-multipath' => 1, nodes => { departed => 1 } },
lvmgone => { type => 'lvm', base => 'gone:0.0.0.dm-uuid-mpath-3600e' },
scoped => { type => 'iscsi', 'managed-multipath' => 1, nodes => { n2 => 1 } },
+ nvme => { type => 'nvmeof', subsysnqn => 'nqn.test:sub' },
+ lvmns => { type => 'lvm', base => 'nvme:nvme-uuid.2', nodes => { n5 => 1 } },
},
} },
);
@@ -541,6 +603,37 @@ subtest 'API returns final node and LUN states' => sub {
'an entry for other nodes does not change that',
);
}
+ my $namespace = $lun{'uuid.1'};
+ ok($namespace->{native}, 'namespace marked as native');
+ ok(!exists($namespace->{'multipath-storage'}), 'a namespace has no requested map');
+ is_deeply(
+ { map { $_ => $namespace->{nodes}->{$_}->{state} } keys $namespace->{nodes}->%* },
+ {
+ n1 => 'optimal',
+ n2 => 'missing',
+ n3 => 'unknown',
+ n4 => 'unknown',
+ n5 => 'degraded',
+ },
+ 'namespace state of a node without multipathd counts',
+ );
+ is($lun->{nodes}->{n5}->{state}, 'unknown', 'no map state from a node without multipathd');
+ is_deeply(
+ $lun{'uuid.2'},
+ {
+ wwid => 'uuid.2',
+ explicit => 0,
+ native => 1,
+ 'used-by-storage' => 'lvmns',
+ state => 'missing',
+ nodes => { n5 => { state => 'missing' } },
+ },
+ 'base namespace that a node without multipathd does not report',
+ );
+ $raw{n5} = encode_json({ time => $NOW, error => 'too many LUNs to report' });
+ $res = PVE::API2::Multipath->status({});
+ %lun = map { $_->{wwid} => $_ } $res->{luns}->@*;
+ is($lun{'uuid.2'}->{nodes}->{n5}->{state}, 'unknown', 'node without a list');
eval {
PVE::JSONSchema::validate(
$res,
next prev parent reply other threads:[~2026-11-18 20:45 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` Thomas Lamprecht [this message]
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118204302.3100813-10-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.