From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id F3A561FF0DA for ; Thu, 19 Nov 2026 20:52:09 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 43ED221D21; Thu, 19 Nov 2026 20:48:37 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v8 storage 26/43] api: scan: report the process that has a SAN LUN open Date: Thu, 19 Nov 2026 20:46:31 +0100 Message-ID: <20261119194655.3163258-27-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119194655.3163258-1-t.lamprecht@proxmox.com> References: <20261119194655.3163258-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795117622842 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.151 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: URXFQZLKYRV5Q2WPXULZ46V5QBTNDLFZ X-Message-ID-Hash: URXFQZLKYRV5Q2WPXULZ46V5QBTNDLFZ X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The LUN inventory shows a LUN as unused if no storage references it and it has no signature. A guest can still have such a LUN as disk, by its device path or from a third-party storage plugin. Wiping the LUN or creating a volume group on it is then refused, as both check for a process that has the device open. But the list gave no hint of that. Report a process that has the LUN open for writing, also through another path or a partition of the LUN, and the virtual machine that the control group of the process names, for users who may see that guest. qemu-server starts each virtual machine in a control group named after its ID. pve-storage cannot ask qemu-server, which depends on it, and if that naming changed, the list would only name the process. A process that only reads does not count: the status query of an LVM storage reads its physical volumes every few seconds, so their LUNs would show up as in use at times. The list needs one pass over the open files of all processes. That took 2 ms on an idle lab node and 106 ms with 1000 more processes that have 64 files open each. Looking up the partitions of a LUN and its paths adds about 35 us per device, 0.1 s for 1000 LUNs with two paths. Signed-off-by: Thomas Lamprecht --- src/PVE/API2/Storage/Scan.pm | 27 ++++++++++++- src/PVE/CLI/pvesm.pm | 2 +- src/PVE/Storage/SAN.pm | 25 +++++++++++- src/test/run_san_api_tests.pl | 25 ++++++++++++ src/test/run_san_lun_scan_tests.pl | 61 ++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+), 3 deletions(-) diff --git a/src/PVE/API2/Storage/Scan.pm b/src/PVE/API2/Storage/Scan.pm index 343b073a..cf19f835 100644 --- a/src/PVE/API2/Storage/Scan.pm +++ b/src/PVE/API2/Storage/Scan.pm @@ -7,6 +7,7 @@ use warnings; use PVE::JSONSchema qw(get_standard_option); use PVE::RESTHandler; +use PVE::RPCEnvironment; use PVE::SafeSyslog; use PVE::Storage::LVMPlugin; @@ -534,6 +535,20 @@ __PACKAGE__->register_method({ type => 'string', optional => 1, }, + 'used-by-process' => { + description => "A process of the node that has the LUN, one of its paths" + . " or a partition open for writing, as PID followed by the command name" + . " in parentheses. A guest shows up like this if it has the LUN as disk" + . " by its device path or from a third-party storage plugin.", + type => 'string', + optional => 1, + }, + 'used-by-guest' => { + description => "The ID of the virtual machine that this process belongs" + . " to. Only returned with the VM.Audit privilege on it.", + type => 'integer', + optional => 1, + }, wwid => { description => "The LUN WWID.", type => 'string', @@ -590,7 +605,17 @@ __PACKAGE__->register_method({ }, }, code => sub { - return PVE::Storage::SAN::scan_luns(PVE::Storage::config()); + my $rpcenv = PVE::RPCEnvironment::get(); + my $authuser = $rpcenv->get_user(); + + my $rows = PVE::Storage::SAN::scan_luns(PVE::Storage::config()); + # without VM.Audit on the guest, its process still tells that the LUN is in use + for my $row (@$rows) { + my $vmid = $row->{'used-by-guest'} // next; + delete $row->{'used-by-guest'} + if !$rpcenv->check($authuser, "/vms/$vmid", ['VM.Audit'], 1); + } + return $rows; }, }); diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm index 95f66d83..8c3a2eda 100755 --- a/src/PVE/CLI/pvesm.pm +++ b/src/PVE/CLI/pvesm.pm @@ -745,7 +745,7 @@ our $cmddef = { my ($data, $schema, $options) = @_; my $columns = [ qw(devpath name size transport state paths-active paths-total paths-expected - multipath usage vgname used-by-storage wwid) + multipath usage vgname used-by-storage used-by-guest used-by-process wwid) ]; PVE::CLIFormatter::print_api_result($data, $schema, $columns, $options); }, diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm index f811fee0..3be15653 100644 --- a/src/PVE/Storage/SAN.pm +++ b/src/PVE/Storage/SAN.pm @@ -48,7 +48,7 @@ sub scan_luns($cfg) { PVE::Multipath::Config::wwids_for_node($mpcfg, PVE::INotify::nodename())->@*; my %listed = map { $_ => 1 } PVE::Multipath::list_etc_multipath_wwids()->@*; my $derived = PVE::Multipath::Generator::derived_wwids($cfg, $ids); - my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %targets, %unmapped); + my (%storage, %expected, %vg_owner, %target_owner, %base_owner, %targets, %paths, %unmapped); for my $sid (sort keys %$derived) { for my $wwid (keys $derived->{$sid}->{wwids}->%*) { @@ -119,6 +119,7 @@ sub scan_luns($cfg) { $row->{$key} //= $value if defined($value) && $value ne 'unknown'; } for my $member (map { basename($_) } ($disk->{slaves} // [$disk->{devpath}])->@*) { + $paths{$wwid}->{"/dev/$member"} = 1; my $session = $sessions->{$member} // next; $targets{$wwid}->{ $session->{target} } = 1; $row->{'iscsi-target'} //= $session->{target}; @@ -179,7 +180,29 @@ sub scan_luns($cfg) { $path->{address} //= $portal if length($portal // ''); } } + # A guest can have a LUN as disk by its device path or from a third-party storage plugin, also + # by one of its paths or partitions. The scans of LVM and udev only read a device, and are gone + # again a moment later. + for my $wwid (sort keys %rows) { + my %seen; + for my $device ($rows{$wwid}->{devpath}, sort keys(($paths{$wwid} // {})->%*)) { + next if $seen{$device}++; + my $parts = eval { lun_partitions({ devpath => $device, wwid => $wwid }) } // []; + $paths{$wwid}->{$_} = 1 for @$parts; + } + } + my %devices = map { $_ => 1 } (map { $_->{devpath} } values %rows), + map { keys %$_ } values %paths; + my $openers = PVE::Diskmanage::device_openers([sort keys %devices], writable => 1); for my $row (values %rows) { + my ($opener) = map { $openers->{$_} // () } abs_path($row->{devpath}) // $row->{devpath}, + sort keys(($paths{ $row->{wwid} } // {})->%*); + if ($opener) { + $row->{'used-by-process'} = "$opener->{pid} ($opener->{command})"; + # a virtual machine runs in a control group of its own that is named after its ID + my $cgroup = file_read_firstline("/proc/$opener->{pid}/cgroup") // ''; + $row->{'used-by-guest'} = int($1) if $cgroup =~ m{^0::/qemu\.slice/(\d+)\.scope$}; + } # the devices of a LUN whose requested map is missing do not tell the state of the LUN if (defined($row->{'paths-active'}) && $row->{multipath} ne 'pending') { $row->{state} = PVE::Multipath::derive_health( diff --git a/src/test/run_san_api_tests.pl b/src/test/run_san_api_tests.pl index c16d2bf2..c95ae474 100755 --- a/src/test/run_san_api_tests.pl +++ b/src/test/run_san_api_tests.pl @@ -71,6 +71,31 @@ for my $case ( ); } } +{ + # a user without VM.Audit on the guest still learns that a process uses the LUN + my $san = Test::MockModule->new('PVE::Storage::SAN'); + my $rpc = Test::MockModule->new('PVE::RPCEnvironment'); + $san->mock( + scan_luns => sub { + return [ + { wwid => '3600a', 'used-by-process' => '4242 (kvm)', 'used-by-guest' => 100 }, + { wwid => '3600b', 'used-by-process' => '4243 (kvm)', 'used-by-guest' => 101 }, + ]; + }, + ); + { + + package SANScanTestRPC; + sub get_user { 'auditor@pve' } + sub check { $_[2] eq '/vms/100' && $_[3]->[0] eq 'VM.Audit' && $_[4] } + } + $rpc->mock(get => sub { bless {}, 'SANScanTestRPC' }); + my $scan = PVE::API2::Storage::Scan->map_method_by_name('sanlunscan')->{code}; + my %rows = map { $_->{wwid} => $_ } $scan->({ node => 'test' })->@*; + is($rows{'3600a'}->{'used-by-guest'}, 100, 'a guest that the user may audit is named'); + ok(!exists($rows{'3600b'}->{'used-by-guest'}), 'another one is not'); + is($rows{'3600b'}->{'used-by-process'}, '4243 (kvm)', 'but its process is'); +} is_deeply( PVE::API2::ISCSI->map_method_by_name('index')->{code}->({}), [{ name => 'status' }, { name => 'target' }], diff --git a/src/test/run_san_lun_scan_tests.pl b/src/test/run_san_lun_scan_tests.pl index 8ee76a49..11bb294d 100755 --- a/src/test/run_san_lun_scan_tests.pl +++ b/src/test/run_san_lun_scan_tests.pl @@ -59,7 +59,9 @@ $scan->redefine( $code->($_) for grep { /^$regex$/ } qw(sda sdb sdc sdd sde sdf dm-0 loop0 nvme0n1); }, ); +my ($openers, @opener_query) = ({}); $disk->mock( + device_openers => sub { @opener_query = @_; return $openers }, get_disks => sub { ok($_[3], 'SAN inventory opts into iSCSI'); @listed = sort $_[0]->@*; @@ -97,6 +99,8 @@ $mp->mock( ); $scan->mock(abs_path => sub { $_[0] }); $scan->mock(stable_path => sub { '/dev/disk/by-id/scsi-' . $_[1] }); +my %partitions; +$scan->mock(lun_partitions => sub { [($partitions{ $_[0]->{devpath} } // [])->@*] }); my %device_state; $scan->mock( device_state => sub { exists($device_state{ $_[0] }) ? $device_state{ $_[0] } : 'running' }, @@ -118,6 +122,63 @@ is($row{'3600plain'}->{'used-by-storage'}, 'direct', 'a storage serving the LUNs my %plain = map { $_->{wwid} => $_ } PVE::Storage::SAN::scan_luns($cfg)->@*; is($plain{'3600plain'}->{'used-by-storage'}, undef, 'a connection storage does not'); } +{ + # a process has a map open by the device it resolves to + $scan->mock(abs_path => sub { $_[0] eq '/dev/mapper/lun' ? '/dev/dm-0' : $_[0] }); + my $cgroup = '0::/qemu.slice/100.scope'; + $scan->mock( + file_read_firstline => sub { + return $_[0] eq '/proc/4242/cgroup' ? $cgroup : PVE::Tools::file_read_firstline(@_); + }, + ); + local $openers->{'/dev/dm-0'} = { pid => 4242, command => 'kvm' }; + my %open = map { $_->{wwid} => $_ } PVE::Storage::SAN::scan_luns($cfg)->@*; + is($open{'3600map'}->{'used-by-process'}, '4242 (kvm)', 'a process with the map open'); + is($open{'3600map'}->{'used-by-guest'}, 100, 'that belongs to a virtual machine'); + ok(!exists($open{'3600plain'}->{'used-by-process'}), 'no process for the other LUN'); + my ($devices, %param) = @opener_query; + is_deeply( + [sort @$devices], + [qw(/dev/mapper/lun /dev/sdb /dev/sdc /dev/sdd /dev/sdf)], + 'asked once for all LUNs and their paths', + ); + ok($param{writable}, 'a process that only reads a LUN does not use it'); + { + local $openers->{'/dev/sdc'} = { pid => 4243, command => 'kvm' }; + local $openers->{'/dev/sdf'} = { pid => 4244, command => 'kvm' }; + my %path = map { $_->{wwid} => $_ } PVE::Storage::SAN::scan_luns($cfg)->@*; + is($path{'3600plain'}->{'used-by-process'}, '4243 (kvm)', 'a process with another path'); + is($path{'3600map'}->{'used-by-process'}, '4242 (kvm)', 'the map before its paths'); + } + { + # a kernel partition of a path, one that kpartx maps on the map, and one of a member + local $partitions{'/dev/sdc'} = ['/dev/sdc1']; + local $partitions{'/dev/mapper/lun'} = ['/dev/dm-5']; + local $partitions{'/dev/sdf'} = ['/dev/sdf1']; + delete local $openers->{'/dev/dm-0'}; + local $openers->{'/dev/sdc1'} = { pid => 4242, command => 'kvm' }; + local $openers->{'/dev/dm-5'} = { pid => 4242, command => 'kvm' }; + my %part = map { $_->{wwid} => $_ } PVE::Storage::SAN::scan_luns($cfg)->@*; + is($part{'3600plain'}->{'used-by-guest'}, 100, 'a guest that writes to a partition'); + is($part{'3600map'}->{'used-by-guest'}, 100, 'also to one that kpartx maps'); + ($devices) = @opener_query; + is_deeply( + [sort @$devices], + [qw(/dev/dm-5 /dev/mapper/lun /dev/sdb /dev/sdc /dev/sdc1 /dev/sdd /dev/sdf /dev/sdf1)], + 'the partitions in the same query', + ); + } + my $schema = PVE::API2::Storage::Scan->map_method_by_name('sanlunscan')->{returns}; + eval { PVE::JSONSchema::validate([values %open], $schema) }; + is($@, '', 'rows with a process match the public schema'); + + $cgroup = '0::/system.slice/100.scope'; + ($open{'3600map'}) = grep { $_->{wwid} eq '3600map' } PVE::Storage::SAN::scan_luns($cfg)->@*; + is($open{'3600map'}->{'used-by-process'}, '4242 (kvm)', 'a process of no guest'); + ok(!exists($open{'3600map'}->{'used-by-guest'}), 'has no guest ID'); + $scan->unmock('file_read_firstline'); + $scan->mock(abs_path => sub { $_[0] }); +} is_deeply( $row{'3600plain'}->{'path-list'}, [