From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2F2541FF0C0 for ; Wed, 18 Nov 2026 07:57:25 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7DBD621A19; Wed, 18 Nov 2026 07:54:13 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v6 storage 13/42] multipath: publish per-node status Date: Wed, 18 Nov 2026 07:52:30 +0100 Message-ID: <20261118065302.3033937-14-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118065302.3033937-1-t.lamprecht@proxmox.com> References: <20261118065302.3033937-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794984787762 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.179 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: APNK5WESQ7BBQJARXJTRG75CXTPNYZ25 X-Message-ID-Hash: APNK5WESQ7BBQJARXJTRG75CXTPNYZ25 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Path health differs between nodes. Add the function for the status daemon to call on each node. It applies the multipath configuration and publishes the state of the maps of the node in the cluster key-value store. Besides the maps, a report has the iSCSI initiator name, which an admin needs for the access rules on the array, and the error of a failed configuration run. If multipath is not usable, the report has the reason instead of the maps. A node can have hundreds of maps that something else set up, for example a storage plugin with one LUN per guest disk. But a value of the key-value store is limited to 32 KiB. So a map only gets a row if the node manages it, if it is the base volume of an LVM storage or if it holds the volume group of a storage. The other maps are only counted. A report that is still too large drops the list of maps and reports an error instead. An unchanged report is sent again once a minute, so that readers can tell a stale one. The volume groups come from the hint file of LVM, where a volume group on a partition of a map counts for the map. LVM leaves the file empty while it sees duplicate PVs. A node that leaves maps out then says so in its report, as it cannot tell whether one of them holds the volume group of a storage. The file can also be outdated. After the LVM hints are turned off, it stays until the node reboots or someone removes it. And it lacks a physical volume that another node added to a shared volume group until LVM scans the devices of the node again. Signed-off-by: Thomas Lamprecht --- src/PVE/Makefile | 1 + src/PVE/Multipath/Status.pm | 99 ++++++++++++++++ src/PVE/Storage/ISCSIPlugin.pm | 16 +++ src/PVE/Storage/LVMPlugin.pm | 24 +++- src/test/run_multipath_api_tests.pl | 176 ++++++++++++++++++++++++++++ src/test/run_san_library_tests.pl | 55 +++++++++ 6 files changed, 370 insertions(+), 1 deletion(-) create mode 100644 src/PVE/Multipath/Status.pm diff --git a/src/PVE/Makefile b/src/PVE/Makefile index 87d5c41f..13fecbf8 100644 --- a/src/PVE/Makefile +++ b/src/PVE/Makefile @@ -9,6 +9,7 @@ install: install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm install -D -m 0644 Multipath/Generator.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Generator.pm install -D -m 0644 Multipath/Identity.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Identity.pm + install -D -m 0644 Multipath/Status.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Status.pm install -D -m 0644 CephConfig.pm ${DESTDIR}${PERLDIR}/PVE/CephConfig.pm install -D -m 0644 GuestImport.pm ${DESTDIR}${PERLDIR}/PVE/GuestImport.pm make -C Storage install diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm new file mode 100644 index 00000000..cd049795 --- /dev/null +++ b/src/PVE/Multipath/Status.pm @@ -0,0 +1,99 @@ +package PVE::Multipath::Status; + +use v5.36; + +use JSON qw(decode_json); + +use PVE::Cluster; +use PVE::INotify; + +use PVE::Multipath; +use PVE::Multipath::Generator; + +my $json = JSON->new->utf8->canonical; +my $KEY = 'multipath'; + +sub publish($report) { + my $node = PVE::INotify::nodename(); + my $now = time(); + my $data = { %$report, time => $now }; + my $encoded = $json->encode($data); + if (length($encoded) >= 32 * 1024) { + $data = { %$data, error => 'too many LUNs to report' }; + delete $data->{luns}; + $encoded = $json->encode($data); + } + my $raw = PVE::Cluster::get_node_kv($KEY, $node); + my $previous = eval { decode_json($raw->{$node} // '') }; + if (ref($previous) eq 'HASH') { + my $age = $now - ($previous->{time} // 0); + $previous->{time} = $now; + return if $age >= 0 && $age < 60 && $json->encode($previous) eq $encoded; + } + PVE::Cluster::broadcast_node_kv($KEY, $encoded); +} + +sub update_node_status() { + my $apply = PVE::Multipath::Generator::sync(); + my $report = { luns => [] }; + $report->{'apply-error'} = $apply->{error} if defined($apply->{error}); + require PVE::Storage::ISCSIPlugin; + my $initiator = eval { PVE::Storage::ISCSIPlugin::read_initiator_info() }; + $report->{'iscsi-initiator-name'} = $initiator->{name} + if $initiator && defined($initiator->{name}); + eval { + die "multipath-tools is not installed\n" if !PVE::Multipath::is_supported(); + die "multipathd is not running\n" if !PVE::Multipath::is_running(); + require PVE::Storage::LVMPlugin; + my $pv_vgs = PVE::Storage::LVMPlugin::cached_pv_volume_groups(); + # A node can have many maps that something else set up, for example a storage plugin with + # one LUN per guest disk. Only the maps that the node manages, that are the base volume of a + # storage or hold its volume group get a row, the others are counted. + require PVE::Storage; + my $wanted = $apply->{wwids}; + my $storages = eval { PVE::Storage::config()->{ids} }; + my (%storage_vgs, %base); + for my $scfg (values %{ $storages // {} }) { + $storage_vgs{ $scfg->{vgname} } = 1 if defined($scfg->{vgname}); + $base{$1} = 1 + if ($scfg->{base} // '') =~ m/(?:^|\.)dm-uuid-(?:part\d+-)?mpath-(\S+)$/; + } + # without the wanted WWIDs or the storages nothing tells the maps apart + my $all = !$wanted || !$storages; + for my $map (PVE::Multipath::get_maps()->@*) { + my $wwid = $map->{wwid} // next; + my $vg = defined($map->{sysfs}) ? ($pv_vgs // {})->{"/dev/$map->{sysfs}"} : undef; + if (!$all && !$wanted->{$wwid} && !$base{$wwid} && !$storage_vgs{ $vg // '' }) { + $report->{'other-maps'}++; + next; + } + my $info = ($wanted // {})->{$wwid} // {}; + my $lun = { + wwid => $wwid, + 'paths-total' => $map->{'paths-total'}, + 'paths-active' => $map->{'paths-active'}, + state => PVE::Multipath::derive_health( + $map->{'paths-active'}, + $map->{'paths-total'}, + $info->{'paths-expected'}, + ), + }; + $lun->{size} = $map->{size} if defined($map->{size}); + $lun->{name} = $map->{name} if defined($map->{name}) && $map->{name} ne $wwid; + $lun->{storages} = $info->{storages} if $info->{storages}; + $lun->{vgname} = $vg if defined($vg); + push $report->{luns}->@*, $lun; + } + # the volume groups come from the hint file of LVM, which it does not always have + $report->{'vgs-unknown'} = 1 if !$pv_vgs && $report->{'other-maps'}; + }; + if (my $error = $@) { + chomp $error; + $report->{error} = $error; + delete $report->{luns}; + } + eval { publish($report) }; + warn "multipath: cannot publish node status - $@" if $@; +} + +1; diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm index 70ac2346..9136c7c9 100644 --- a/src/PVE/Storage/ISCSIPlugin.pm +++ b/src/PVE/Storage/ISCSIPlugin.pm @@ -21,8 +21,24 @@ use base qw(PVE::Storage::Plugin); # iscsi helper function my $ISCSIADM = '/usr/bin/iscsiadm'; +our $INITIATOR_NAME_FILE = '/etc/iscsi/initiatorname.iscsi'; our $SYSFS = '/sys'; +sub read_initiator_info { + my $content = eval { PVE::Tools::file_get_contents($INITIATOR_NAME_FILE, 64 * 1024) }; + return undef if !defined($content); + + my $res = {}; + for my $line (split(/\n/, $content)) { + if ($line =~ m/^\s*InitiatorName\s*=\s*(\S+)\s*$/) { + $res->{name} = $1; + } elsif ($line =~ m/^\s*InitiatorAlias\s*=\s*(.+?)\s*$/) { + $res->{alias} = $1; + } + } + return defined($res->{name}) ? $res : undef; +} + my $found_iscsi_adm_exe; my sub assert_iscsi_support { my ($noerr) = @_; diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index f9c72764..451e20ba 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -19,7 +19,7 @@ use PVE::INotify; use PVE::JSONSchema qw(get_standard_option); use PVE::Multipath; use PVE::RESTEnvironment qw(log_warn); -use PVE::Tools qw(run_command file_read_firstline trim); +use PVE::Tools qw(run_command file_get_contents file_read_firstline trim); use PVE::Storage::Common; use PVE::Storage::Plugin; @@ -140,6 +140,28 @@ sub lvm_pv_info { return $pvinfo; } +sub cached_pv_volume_groups { + my $data = eval { file_get_contents('/run/lvm/hints') } // ''; + # LVM leaves the file without content when devices change and while it sees duplicate PVs + return undef if $data !~ m/^hints_version: 2\./m; + my $res = {}; + while ($data =~ m/^scan:(\/dev\/\S+) pvid:\S+ devn:\d+:\d+ vg:([a-zA-Z0-9+_.-]+)$/mg) { + next if $2 eq '-'; + my ($device, $vg) = (abs_path($1) // $1, $2); + $res->{$device} = $vg; + # a volume group on a partition of a multipath map counts for the map as well + my $sysdir = '/sys/class/block/' . basename($device); + my $map = PVE::Multipath::parse_dm_uuid(file_read_firstline("$sysdir/dm/uuid")); + next if !$map || !$map->{partition}; + PVE::Tools::dir_glob_foreach( + "$sysdir/slaves", + 'dm-\d+', + sub { $res->{"/dev/$_[0]"} //= $vg }, + ); + } + return $res; +} + sub clear_first_sector { my ($dev) = shift; diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl index a3d7f3e6..7d671458 100755 --- a/src/test/run_multipath_api_tests.pl +++ b/src/test/run_multipath_api_tests.pl @@ -9,9 +9,11 @@ BEGIN { } use lib ('.', '..'); use File::Temp qw(tempdir); +use JSON qw(decode_json); use Test::More; use Test::MockModule; use PVE::Multipath::Generator; +use PVE::Storage; subtest 'add-only generation and failed reload throttle' => sub { my $dir = tempdir('multipath-test-XXXX', DIR => '.', CLEANUP => 1); @@ -126,4 +128,178 @@ subtest 'storage LUNs are the identified paths of the storage target' => sub { ); }; +subtest 'an unconfigured node does not write drop-ins or reload' => sub { + my $dir = tempdir('multipath-empty-XXXX', DIR => '.', CLEANUP => 1); + local $PVE::Multipath::Generator::DEFAULTS_DROPIN = "$dir/defaults"; + local $PVE::Multipath::Generator::OVERRIDES_DROPIN = "$dir/overrides"; + local $PVE::Multipath::Generator::ALIASES_DROPIN = "$dir/aliases"; + local $PVE::Multipath::Generator::RECONFIGURE_PENDING = "$dir/pending"; + my $config = Test::MockModule->new('PVE::Multipath::Config'); + $config->mock(read_config => sub { { ids => {} } }); + $config->mock(read_overrides => sub { '' }); + my $storage = Test::MockModule->new('PVE::Storage'); + my $identity = Test::MockModule->new('PVE::Multipath::Identity'); + $identity->mock(collect => sub { die "unexpected identity collection\n" }); + my $mp = Test::MockModule->new('PVE::Multipath'); + $mp->mock(is_supported => sub { 1 }); + $mp->mock(is_running => sub { 1 }); + $mp->mock(get_maps => sub { [] }); + $mp->mock(list_etc_multipath_wwids => sub { [] }); + my $generator = Test::MockModule->new('PVE::Multipath::Generator'); + my @changes; + + for my $function (qw(file_set_contents make_path)) { + $generator->mock($function => sub { push @changes, $function }); + } + for my $function (qw(add_wwid reconfigure)) { + $mp->mock($function => sub { push @changes, $function }); + } + my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin'); + $iscsi->mock(read_initiator_info => sub { {} }); + my $status = Test::MockModule->new('PVE::Multipath::Status'); + my $report; + $status->mock(publish => sub { $report = $_[0] }); + for my $ids ({}, { plain => { type => 'iscsi', target => 'iqn.test' } }) { + $storage->mock(config => sub { { ids => $ids } }); + PVE::Multipath::Status::update_node_status(); + ok(!exists($report->{error}), 'running daemon reports successfully'); + ok(!exists($report->{'apply-error'}), 'unconfigured application succeeds'); + is_deeply(\@changes, [], 'no write, mkdir, allow-list addition or reload'); + is_deeply([glob("$dir/*")], [], 'no generated files or pending marker'); + } +}; + +subtest 'node reports use read-back, refresh and a bounded error report' => sub { + my $cluster = Test::MockModule->new('PVE::Cluster'); + my $node = Test::MockModule->new('PVE::INotify'); + $node->mock(nodename => sub { 'n1' }); + my (%raw, @sent); + $cluster->mock(get_node_kv => sub { \%raw }); + $cluster->mock( + broadcast_node_kv => sub($key, $data) { + is($key, 'multipath', 'one node key'); + push @sent, decode_json($data); + $raw{n1} = $data; + }, + ); + my $report = { + luns => [{ + wwid => '3600a', + name => 'data', + state => 'optimal', + 'paths-total' => 2, + 'paths-active' => 2, + }], + }; + PVE::Multipath::Status::publish($report); + $NOW += 59; + PVE::Multipath::Status::publish($report); + is(scalar(@sent), 1, 'same report younger than 60 seconds is skipped'); + $NOW++; + PVE::Multipath::Status::publish($report); + is(scalar(@sent), 2, 'report refreshed at 60 seconds'); + %raw = (); + PVE::Multipath::Status::publish($report); + is(scalar(@sent), 3, 'lost pmxcfs value is republished without local send state'); + PVE::Multipath::Status::publish( + { luns => [('x' x 33000)], 'iscsi-initiator-name' => 'iqn.2026-11.test:n1' }); + is($sent[-1]->{error}, 'too many LUNs to report', 'oversized report is an error'); + ok(!exists($sent[-1]->{luns}), 'not an empty or partial LUN report'); + is( + $sent[-1]->{'iscsi-initiator-name'}, + 'iqn.2026-11.test:n1', + 'which keeps the node identity', + ); +}; + +subtest 'only managed maps and those under a storage get a row' => sub { + my $gen = Test::MockModule->new('PVE::Multipath::Generator'); + my $status = Test::MockModule->new('PVE::Multipath::Status'); + my $mp = Test::MockModule->new('PVE::Multipath'); + my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin'); + my $storage = Test::MockModule->new('PVE::Storage'); + my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); + my $report; + $status->mock(publish => sub { $report = $_[0] }); + $iscsi->mock(read_initiator_info => sub { undef }); + $mp->mock(is_supported => sub { 1 }, is_running => sub { 1 }); + $mp->mock( + get_maps => sub { + return [ + map { { + wwid => "3600$_", + sysfs => "dm-$_", + 'paths-active' => 2, + 'paths-total' => 2, + } } 1 .. 5 + ]; + }, + ); + $gen->mock(sync => sub { { wwids => { '36001' => {} } } }); + $lvm->mock( + cached_pv_volume_groups => sub { { '/dev/dm-2' => 'vgsan', '/dev/dm-3' => 'vgother' } }, + ); + my $ids = { + san => { type => 'lvm', vgname => 'vgsan' }, + based => + { type => 'lvm', vgname => 'vgbase', base => 'conn:0.0.0.dm-uuid-mpath-36004' }, + }; + $storage->mock(config => sub { { ids => $ids } }); + PVE::Multipath::Status::update_node_status(); + is_deeply( + [map { $_->{wwid} } $report->{luns}->@*], + ['36001', '36002', '36004'], + 'a wanted map, one with the volume group of a storage and a base volume', + ); + is($report->{'other-maps'}, 2, 'the other maps are counted'); + ok(!exists($report->{'vgs-unknown'}), 'volume groups known from the hints of LVM'); + $lvm->mock(cached_pv_volume_groups => sub { undef }); + PVE::Multipath::Status::update_node_status(); + is_deeply( + [map { $_->{wwid} } $report->{luns}->@*], + ['36001', '36004'], + 'without hints only the maps that the configuration names', + ); + is($report->{'other-maps'}, 3, 'the map of the volume group counts as other'); + ok($report->{'vgs-unknown'}, 'and the report says that volume groups are unknown'); + $storage->mock(config => sub { die "no configuration\n" }); + PVE::Multipath::Status::update_node_status(); + is(scalar($report->{luns}->@*), 5, 'all maps without the storage configuration'); + ok(!exists($report->{'other-maps'}), 'and no count'); + $storage->mock(config => sub { { ids => $ids } }); + $mp->mock(get_maps => sub { [] }); + PVE::Multipath::Status::update_node_status(); + ok(!exists($report->{'vgs-unknown'}), + 'unknown volume groups only matter for left out maps'); +}; + +subtest 'one entry point applies configuration and reports unavailable reasons' => sub { + my $gen = Test::MockModule->new('PVE::Multipath::Generator'); + my $status = Test::MockModule->new('PVE::Multipath::Status'); + my $mp = Test::MockModule->new('PVE::Multipath'); + my $iscsi = Test::MockModule->new('PVE::Storage::ISCSIPlugin'); + my ($report, @order); + $gen->mock(sync => sub { push @order, 'apply'; { error => 'reload failed' } }); + $status->mock(publish => sub { push @order, 'publish'; $report = $_[0] }); + $iscsi->mock(read_initiator_info => sub { { name => 'iqn.2025-01.example:node' } }); + $mp->mock(is_supported => sub { 0 }); + PVE::Multipath::Status::update_node_status(); + is($report->{error}, 'multipath-tools is not installed', 'missing tools reason only'); + is_deeply(\@order, [qw(apply publish)], 'single call owns ordering'); + is($report->{'apply-error'}, 'reload failed', 'apply failure is separate'); + $mp->mock(is_supported => sub { 1 }); + $mp->mock(is_running => sub { 0 }); + PVE::Multipath::Status::update_node_status(); + is($report->{error}, 'multipathd is not running', 'stopped daemon reason only'); + $mp->mock(is_running => sub { 1 }); + $mp->mock(get_maps => sub { die "bad JSON\n" }); + PVE::Multipath::Status::update_node_status(); + is($report->{error}, 'bad JSON', 'query error without unavailable prefix'); + $mp->mock(get_maps => + sub { [{ wwid => '3600a', name => 'data', 'paths-active' => 1, 'paths-total' => 2 }] }); + PVE::Multipath::Status::update_node_status(); + is($report->{luns}->[0]->{state}, 'degraded', 'all maps while the wanted ones are unknown'); + ok(!exists($report->{supported}), 'no separate capability state'); +}; + done_testing(); diff --git a/src/test/run_san_library_tests.pl b/src/test/run_san_library_tests.pl index 2ec2a651..f7ae68d1 100755 --- a/src/test/run_san_library_tests.pl +++ b/src/test/run_san_library_tests.pl @@ -50,4 +50,59 @@ subtest 'omitted optional parameters retain explicit undef behavior' => sub { } }; +subtest 'LVM hints are optional cached display metadata' => sub { + require PVE::Storage::LVMPlugin; + my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin'); + my $contents; + $lvm->mock( + file_get_contents => sub { + is($_[0], '/run/lvm/hints', 'only the existing LVM cache is read'); + die "No such file\n" if !defined($contents); + return $contents; + }, + ); + $lvm->mock(abs_path => sub { $_[0] eq '/dev/disk/by-id/test' ? '/dev/dm-0' : $_[0] }); + $lvm->mock(run_command => sub { die "unexpected device probe\n" }); + $lvm->redefine( + file_read_firstline => sub { $_[0] =~ m{/dm-3/dm/uuid$} ? 'part1-mpath-36001' : undef }, + ); + my $tools = Test::MockModule->new('PVE::Tools'); + $tools->redefine( + dir_glob_foreach => sub { $_[2]->('dm-2') if $_[0] eq '/sys/class/block/dm-3/slaves' }); + my $head = "# Created by vgs pid 1\nhints_version: 2.1\n"; + for my $case ( + [undef, undef, 'missing file'], + ['', undef, 'empty file'], + ["# Created empty by vgs pid 1\n", undef, 'file that LVM left without content'], + ["scan:/dev/sda pvid:abc devn:8:0 vg:data\n", undef, 'no version line'], + [ + "hints_version: 3.0\nscan:/dev/sda pvid:abc devn:8:0 vg:data\n", + undef, + 'other format version', + ], + [$head, {}, 'no PVs'], + ["${head}scan=/dev/sda pvid=abc devn=8:0 vg=data\n", {}, 'different format'], + ["${head}scan:/dev/sda pvid:abc devn:8:0 vg:-\n", {}, 'orphan PV'], + ["${head}scan:/dev/sda pvid:abc devn:8:0 vg:[unknown]\n", {}, 'unknown VG'], + [ + "${head}scan:/dev/sda pvid:abc devn:8:0 vg:data\n", + { '/dev/sda' => 'data' }, + 'cached VG', + ], + [ + "${head}scan:/dev/disk/by-id/test pvid:abc devn:253:0 vg:data\n", + { '/dev/dm-0' => 'data' }, + 'resolved device name', + ], + [ + "${head}scan:/dev/dm-3 pvid:abc devn:253:3 vg:data\n", + { '/dev/dm-3' => 'data', '/dev/dm-2' => 'data' }, + 'partition of a multipath map', + ], + ) { + $contents = $case->[0]; + is_deeply(PVE::Storage::LVMPlugin::cached_pv_volume_groups(), $case->[1], $case->[2]); + } +}; + done_testing();