From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v7 storage 23/43] disks: lvm: allow volume groups on SAN LUNs and multipath maps
Date: Wed, 18 Nov 2026 21:26:06 +0100 [thread overview]
Message-ID: <20261118202646.3017286-24-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118202646.3017286-1-t.lamprecht@proxmox.com>
The node disk API creates volume groups on local disks only. It
treats a Fibre Channel LUN as a local disk, refuses a multipath map
and adds the storage as local to the node. Accept the LUNs of the SAN
inventory and add their storage as shared for all nodes. Refuse an
existing storage ID for a LUN, and never join a storage that got
added with that ID meanwhile. Joining is meant for local volume groups
of the same name, and on a LUN would combine two different volume
groups in one shared storage. The partitions of a LUN, also the ones
that kpartx maps, stay local devices as before, so that an LVM storage
can still be added for a volume group on one of them.
Initializing a LUN, here or as base volume of an LVM storage that is
being added, takes a cluster-wide lock per WWID. It refuses a LUN that
a storage references through its volume group, as base volume, also
through a partition, or as iSCSI target serving images. The name of a
map can pass to another map while the lock is awaited. So the LUN is
held open by its stable name, and creation stops if a name used for
the checks and the LVM commands no longer leads to the held device.
The lookup of the LUN also resolves the requested name again after its
scan, and stops if the name leads to another device by then.
A LUN that a process of the node has open is refused too. A guest
that got the LUN as disk from a third-party storage plugin, or by its
device path, is neither a holder nor a storage reference, and its disk
can look empty. The check goes by the open files of the processes, so
it misses an open through another device node, for example the one of
a container. Also refused is a LUN whose other SCSI paths or their
partitions are mounted, held by a block device other than the map of
the LUN, or open for writing. An open for reading does not count, as
multipathd keeps the paths open read-only.
Co-authored-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/API2/Disks/LVM.pm | 71 +++++---
src/PVE/Diskmanage.pm | 45 +++++
src/PVE/Storage/LVMPlugin.pm | 81 ++++++++-
src/PVE/Storage/SAN.pm | 154 ++++++++++++++++
src/test/Makefile | 11 +-
src/test/disk_api_test.pm | 247 ++++++++++++++++++++++++++
src/test/run_disk_api_tests.pl | 11 ++
src/test/run_san_library_tests.pl | 27 +++
src/test/run_san_nvme_tests.pl | 104 +++++++++++
src/test/run_san_taint_tests.pl | 26 +++
src/test/run_san_write_check_tests.pl | 219 +++++++++++++++++++++++
src/test/run_test_lvm_create.pl | 46 +++++
12 files changed, 1006 insertions(+), 36 deletions(-)
create mode 100644 src/test/disk_api_test.pm
create mode 100755 src/test/run_disk_api_tests.pl
create mode 100755 src/test/run_san_nvme_tests.pl
create mode 100755 src/test/run_san_write_check_tests.pl
diff --git a/src/PVE/API2/Disks/LVM.pm b/src/PVE/API2/Disks/LVM.pm
index 6101743b..1fd68f1a 100644
--- a/src/PVE/API2/Disks/LVM.pm
+++ b/src/PVE/API2/Disks/LVM.pm
@@ -4,6 +4,7 @@ use strict;
use warnings;
use PVE::Storage::LVMPlugin;
+use PVE::Storage::SAN;
use PVE::Diskmanage;
use PVE::JSONSchema qw(get_standard_option);
use PVE::API2::Storage::Config;
@@ -130,10 +131,13 @@ __PACKAGE__->register_method({
name => get_standard_option('pve-storage-id'),
device => {
type => 'string',
- description => 'The block device you want to create the volume group on',
+ description => 'The block device you want to create the volume group on.'
+ . ' Multipath mapped devices are supported.',
},
add_storage => {
- description => "Configure storage using the Volume Group",
+ description => "Configure storage using the Volume Group. On a SAN LUN, this"
+ . " adds a new shared storage for all nodes. On a local disk, the storage is"
+ . " for this node, or this node joins a matching storage of that ID.",
type => 'boolean',
optional => 1,
default => 0,
@@ -152,48 +156,63 @@ __PACKAGE__->register_method({
my $node = $param->{node};
$dev = PVE::Diskmanage::verify_blockdev_path($dev);
- PVE::Diskmanage::assert_disk_unused($dev);
+ my $san = PVE::Storage::SAN::is_san_device($dev);
+ PVE::Diskmanage::assert_disk_unused($dev) if !$san;
my $storage_params = {
type => 'lvm',
vgname => $name,
storage => $name,
content => 'rootdir,images',
- shared => 0,
- nodes => $node,
+ shared => $san ? 1 : 0,
+ $san ? () : (nodes => $node),
};
my $verify_params = [qw(vgname)];
if ($param->{add_storage}) {
$rpcenv->check($user, "/storage", ['Datastore.Allocate']);
- # reserve the name and add as disabled, will be enabled below if creation works out
+ # only local volume groups of the same name may share a storage that exists already
+ die "storage ID '$name' already defined\n"
+ if $san && PVE::Storage::storage_config(PVE::Storage::config(), $name, 1);
+
+ # a dry run that checks the parameters, the storage is added after the volume group
PVE::API2::Storage::Config->create_or_update(
$name, $node, $storage_params, $verify_params, 1,
);
}
+ my $create_group = sub {
+ PVE::Diskmanage::assert_disk_unused($dev) if !$san;
+ die "volume group with name '${name}' already exists on node '${node}'\n"
+ if PVE::Storage::LVMPlugin::lvm_vgs()->{$name};
+
+ if (PVE::Diskmanage::is_partition($dev)) {
+ eval { PVE::Diskmanage::change_parttype($dev, '8E00'); };
+ warn $@ if $@;
+ }
+
+ # the given name, so that the LUN lookup notices if it passed to another device during
+ # its scan
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ $san ? $param->{device} : $dev,
+ $name,
+ $san,
+ );
+
+ PVE::Diskmanage::udevadm_trigger($dev);
+
+ if ($param->{add_storage} && $san) {
+ # must not join a storage that someone added with this ID in the meantime
+ PVE::API2::Storage::Config->create({%$storage_params});
+ } elsif ($param->{add_storage}) {
+ PVE::API2::Storage::Config->create_or_update(
+ $name, $node, $storage_params, $verify_params,
+ );
+ }
+ };
my $worker = sub {
- PVE::Diskmanage::locked_disk_action(sub {
- PVE::Diskmanage::assert_disk_unused($dev);
- die "volume group with name '${name}' already exists on node '${node}'\n"
- if PVE::Storage::LVMPlugin::lvm_vgs()->{$name};
-
- if (PVE::Diskmanage::is_partition($dev)) {
- eval { PVE::Diskmanage::change_parttype($dev, '8E00'); };
- warn $@ if $@;
- }
-
- PVE::Storage::LVMPlugin::lvm_create_volume_group($dev, $name);
-
- PVE::Diskmanage::udevadm_trigger($dev);
-
- if ($param->{add_storage}) {
- PVE::API2::Storage::Config->create_or_update(
- $name, $node, $storage_params, $verify_params,
- );
- }
- });
+ $san ? $create_group->() : PVE::Diskmanage::locked_disk_action($create_group);
};
return $rpcenv->fork_worker('lvmcreate', $name, $user, $worker);
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index c2c50bcc..29669a28 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -1165,6 +1165,51 @@ sub change_parttype {
run_command(['sgdisk', "-t${partnum}:${parttype}", $blockdev], errmsg => $err);
}
+# Returns the PID and command name of a process for each of the block devices that a process has
+# open, as hash by the resolved path of the device. With 'writable', a process that opened the
+# device only for reading does not count. Only detects an open through the device nodes in /dev
+# of the host, not one through the device node that a container has of its own.
+sub device_openers {
+ my ($devices, %param) = @_;
+
+ # the device number tells a device from a file of that name in another mount namespace
+ my %nodes;
+ for my $path (grep { defined($_) } map { abs_path($_) } @$devices) {
+ my $st = stat($path) // next;
+ $nodes{$path} = $st->rdev;
+ }
+ my $openers = {};
+ return $openers if !%nodes;
+
+ opendir(my $proc, '/proc') or return $openers;
+ for my $pid (grep { m/^\d+$/ && $_ != $$ } readdir($proc)) {
+ opendir(my $fds, "/proc/$pid/fd") or next;
+ for my $fd (readdir($fds)) {
+ my $target = readlink("/proc/$pid/fd/$fd") // next;
+ next if !defined($nodes{$target}) || $openers->{$target};
+ my $st = stat("/proc/$pid/fd/$fd") // next;
+ next if $st->rdev != $nodes{$target};
+ if ($param{writable}) {
+ # the permissions of the link are the access mode of the open file
+ my $link = lstat("/proc/$pid/fd/$fd") // next;
+ next if !($link->mode & S_IWUSR);
+ }
+ my $comm = file_read_firstline("/proc/$pid/comm") // 'unknown';
+ $openers->{$target} = { pid => $pid, command => $comm };
+ }
+ }
+ return $openers;
+}
+
+# Returns the PID and command name of a process that has one of the block devices open, if any.
+sub device_opener {
+ my (@devices) = @_;
+
+ my $openers = device_openers(\@devices);
+ my ($opener) = map { $openers->{$_} } sort keys %$openers;
+ return $opener ? "$opener->{pid} ($opener->{command})" : undef;
+}
+
# Wipes all labels and the first 200 MiB of a disk/partition (or the whole if it is smaller).
# If called with a partition, also sets the partition type to 0x83 'Linux filesystem'.
# Expected to be called with a result of verify_blockdev_path().
diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
index bc15f2b3..50c35ff2 100644
--- a/src/PVE/Storage/LVMPlugin.pm
+++ b/src/PVE/Storage/LVMPlugin.pm
@@ -162,6 +162,37 @@ sub cached_pv_volume_groups {
return $res;
}
+sub lun_volume_groups {
+ my ($devices) = @_;
+ my $output = '';
+ run_command(
+ [
+ '/sbin/pvs',
+ '--nohints',
+ '--readonly',
+ '--devices',
+ join(',', @$devices),
+ '--reportformat',
+ 'json',
+ '--options',
+ 'pv_name,vg_name',
+ ],
+ timeout => 10,
+ outfunc => sub { $output .= "$_[0]\n" },
+ );
+ my $data = decode_json($output);
+ my $pvs = $data->{report}->[0]->{pv};
+ die "unable to inspect volume groups - invalid pvs report\n" if ref($pvs) ne 'ARRAY';
+ my %vgs;
+ for my $pv (@$pvs) {
+ my $vg = $pv->{vg_name};
+ die "unable to inspect volume groups - missing volume group metadata\n"
+ if !defined($vg) || $vg eq '[unknown]';
+ $vgs{$vg} = 1 if length($vg);
+ }
+ return [sort keys %vgs];
+}
+
sub clear_first_sector {
my ($dev) = shift;
@@ -172,10 +203,17 @@ sub clear_first_sector {
}
}
-sub lvm_create_volume_group {
- my ($device, $vgname, $shared, $guard) = @_;
-
+my sub create_volume_group {
+ my ($device, $vgname, $lun, $guard) = @_;
$device = PVE::Diskmanage::verify_blockdev_path($device);
+ my $fh;
+ if ($lun) {
+ sysopen($fh, $lun->{'by-id'}, O_RDWR) or die "cannot open '$device' - $!\n";
+ }
+ my $check_name = sub {
+ PVE::Storage::SAN::assert_same_device($lun, $fh, $device) if $lun;
+ };
+ $check_name->();
my $refuse_data = sub {
my (@types) = @_;
die "device '$device' contains data ("
@@ -193,7 +231,8 @@ sub lvm_create_volume_group {
if !defined($size) || $size !~ m/^\d+$/ || !$size;
die "unable to probe '$device' - missing device mapper identity\n"
if basename($device) =~ m/^dm-\d+$/ && !defined($uuid);
- if (defined($uuid) && $uuid =~ m/^mpath-/) {
+ my $map = PVE::Multipath::parse_dm_uuid($uuid);
+ if ($map && !$map->{partition}) {
# A queueing map without a running path can stall even a read-only probe.
my $running = 0;
PVE::Tools::dir_glob_foreach(
@@ -220,6 +259,8 @@ sub lvm_create_volume_group {
}
my %signatures = map { $_->{type} => 1 } $probe->@*;
my $res = lvm_pv_info($device, $signatures{LVM2_member});
+ # also a volume group that exists already only counts if the probes looked at the held LUN
+ $check_name->();
# A PV without metadata copies can belong to a VG whose other devices are not visible. The
# sentinel is not a VG name or orphan proof, even if the use flag contradicts it.
@@ -262,26 +303,52 @@ sub lvm_create_volume_group {
}
$refuse_data->(keys %signatures) if %signatures;
+ # Only initializing the device needs the checks, an existing volume group on it can be added.
$guard->{check_storage}->($device) if $guard && $guard->{check_storage};
- clear_first_sector($device); # else pvcreate fails
+ if ($lun) {
+ my $exempt = $guard ? $guard->{exempt} : undef;
+ PVE::Storage::SAN::assert_lun_unreferenced(PVE::Storage::config(), $lun, $exempt);
+ PVE::Storage::SAN::assert_lun_paths_unused($lun);
+ # a guest with the LUN as disk has it open, without being a holder or a storage reference
+ my $opener = PVE::Diskmanage::device_opener($device);
+ die "cannot create volume group on '$device' - process $opener has it open\n"
+ if defined($opener);
+ }
+
+ $check_name->();
+ clear_first_sector($lun ? '/proc/' . $$ . '/fd/' . fileno($fh) : $device);
# we use --metadatasize 250k, which reseults in "pe_start = 512"
# so pe_start is aligned on a 128k boundary (advantage for SSDs)
my $cmd = ['/sbin/pvcreate', '--metadatasize', '250k', $device];
- run_command($cmd, errmsg => "pvcreate '$device' error");
+ $check_name->();
+ run_command($cmd, errmsg => "pvcreate '$device' error", $lun ? (timeout => 30) : ());
$cmd = ['/sbin/vgcreate', $vgname, $device];
# push @$cmd, '-c', 'y' if $shared; # we do not use this yet
+ $check_name->();
run_command(
$cmd,
+ $lun ? (timeout => 30) : (),
errmsg => "vgcreate $vgname $device error",
errfunc => $ignore_no_medium_warnings,
outfunc => $ignore_no_medium_warnings,
);
}
+sub lvm_create_volume_group {
+ my ($device, $vgname, $shared, $guard) = @_;
+ require PVE::Storage::SAN;
+ my $lun = PVE::Storage::SAN::lun_for_device($device);
+ return create_volume_group($device, $vgname, undef, $guard) if !$lun;
+ PVE::Storage::SAN::with_lun_lock(
+ $lun,
+ sub { create_volume_group($device, $vgname, $lun, $guard) },
+ );
+}
+
sub lvm_destroy_volume_group {
my ($vgname) = @_;
@@ -1162,7 +1229,7 @@ sub on_add_hook {
$path,
$scfg->{vgname},
$scfg->{shared},
- { check_storage => $check_target },
+ { check_storage => $check_target, exempt => $storeid },
);
}
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 6b92475b..eb34857a 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -5,6 +5,7 @@ use v5.36;
use Cwd qw(abs_path);
use File::Basename qw(basename);
+use PVE::Cluster;
use PVE::INotify;
use PVE::Tools qw(file_read_firstline dir_glob_foreach);
@@ -225,4 +226,157 @@ sub is_san_device($device, $info = undef) {
|| PVE::Diskmanage::is_remote_transport($info->{transport}) ? 1 : 0;
}
+sub lun_for_device($name) {
+ my $device = PVE::Diskmanage::verify_blockdev_path($name);
+ return undef if !is_san_device($device);
+ my $rows = scan_luns(PVE::Storage::config());
+ # the device behind a name such as a by-id one can pass to another LUN during the scan
+ die "cannot use '$device' - the device behind '$name' changed\n"
+ if (abs_path($name) // '') ne $device;
+ for my $lun (@$rows) {
+ next if (abs_path($lun->{devpath}) // '') ne $device;
+ die "cannot use '$device' - no link under /dev/disk/by-id names the WWID of the LUN\n"
+ if !$lun->{'by-id'};
+ # read from sysfs, but used in command lines and in the name of the cluster lock
+ ($lun->{devpath}) = $lun->{devpath} =~ m{\A(/dev/[A-Za-z0-9_.:+/-]+)\z}
+ or die "cannot use '$device' - unexpected device path\n";
+ ($lun->{wwid}) = $lun->{wwid} =~ m/\A([A-Za-z0-9_.:+=-]+)\z/
+ or die "cannot use '$device' - unexpected WWID\n";
+ return $lun;
+ }
+ die "cannot use '$device' - select an identified LUN or its multipath map\n";
+}
+
+sub lun_partitions($lun) {
+ my $device = PVE::Diskmanage::verify_blockdev_path($lun->{devpath});
+ my $name = basename($device);
+ my @parts;
+ dir_glob_foreach(
+ "/sys/class/block/$name",
+ "\Q$name\E.+",
+ sub($part, @) {
+ my ($path) = "/dev/$part" =~ m{\A(/dev/[A-Za-z0-9_.+-]+)\z};
+ push @parts, $path if defined($path) && -e "/sys/class/block/$name/$part/partition";
+ },
+ );
+ dir_glob_foreach(
+ "/sys/class/block/$name/holders",
+ 'dm-\\d+',
+ sub($part, @) {
+ my $map = PVE::Multipath::parse_dm_uuid(
+ file_read_firstline("/sys/class/block/$part/dm/uuid"));
+ my ($path) = "/dev/$part" =~ m{\A(/dev/dm-\d+)\z};
+ push @parts, $path
+ if defined($path) && $map && $map->{partition} && $map->{wwid} eq $lun->{wwid};
+ },
+ );
+ return \@parts;
+}
+
+sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+ my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
+ [$lun->{devpath}, lun_partitions($lun)->@*],
+ )->@*;
+ # the paths of a LUN can belong to more than one target
+ my $sessions = PVE::ISCSI::Topology::device_map();
+ my %targets =
+ map { $_->{target} => 1 } map { $sessions->{ basename($_) } // () } lun_paths($lun)->@*;
+ $targets{ $lun->{'iscsi-target'} } = 1 if defined($lun->{'iscsi-target'});
+ for my $sid (sort keys $cfg->{ids}->%*) {
+ next if defined($exempt) && $sid eq $exempt;
+ my $scfg = $cfg->{ids}->{$sid};
+ my $type = $scfg->{type};
+ if ($vgs{ $scfg->{vgname} // '' }) {
+ die "cannot use '$lun->{devpath}' - volume group is used by storage '$sid'\n";
+ }
+ if (
+ defined($scfg->{target})
+ && $targets{ $scfg->{target} }
+ && PVE::Storage::san_storage_serves_images($scfg)
+ ) {
+ die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
+ }
+ next if $type ne 'lvm' || !$scfg->{base};
+ my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base});
+ my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/;
+ my $matches = defined($wwid) && $wwid eq $lun->{wwid};
+ if (!$matches) {
+ my $path = eval { abs_path(PVE::Storage::path($cfg, $scfg->{base})) };
+ my $device = abs_path($lun->{devpath});
+ $matches = defined($path) && defined($device) && $path eq $device;
+ }
+ die "cannot use '$lun->{devpath}' - storage '$sid' uses this LUN as its base volume\n"
+ if $matches;
+ }
+ return;
+}
+
+# Serializes writes to a LUN cluster-wide. The storage configuration read in the code is current.
+sub with_lun_lock($lun, $code) {
+ my $key = $lun->{wwid} =~ s/([^A-Za-z0-9_.:-])/sprintf('=%02x', ord($1))/ger;
+ PVE::Cluster::cfs_lock_domain("san-$key", undef, $code);
+ die $@ if $@;
+}
+
+# Device names can pass to another device while a LUN waits for its lock or is worked on. All
+# names used for it have to mean the device that is held open.
+sub assert_same_device($lun, $fh, @names) {
+ my @held = stat($fh);
+ for my $name ($lun->{'by-id'}, $lun->{devpath}, @names) {
+ my @named = stat($name);
+ die "cannot use '$lun->{devpath}' - the device behind '$name' changed\n"
+ if !@held || !@named || $held[6] != $named[6];
+ }
+}
+
+sub lun_paths($lun) {
+ my $paths = PVE::Multipath::Identity::collect(all => 1)->{paths};
+ my %names = map { $_ => 1 } grep { $paths->{$_}->{wwid} eq $lun->{wwid} } keys %$paths;
+ my $device = basename(abs_path($lun->{devpath}) // $lun->{devpath});
+ dir_glob_foreach(
+ "/sys/class/block/$device/slaves",
+ 'sd[a-z]+',
+ sub($slave, @) { $names{$slave} = 1 },
+ );
+ return [map { m/^(sd[a-z]+)$/ ? "/dev/$1" : () } sort keys %names];
+}
+
+# The other paths of a LUN reach the same data, but neither the exclusive open of the LUN nor the
+# checks of its device cover them.
+sub assert_lun_paths_unused($lun) {
+ my $device = basename(abs_path($lun->{devpath}) // $lun->{devpath});
+ my %devices; # the paths and their partitions
+ for my $path (grep { basename($_) ne $device } lun_paths($lun)->@*) {
+ if (my $mounted = PVE::Diskmanage::is_mounted($path)) {
+ die "cannot use '$lun->{devpath}' - the LUN is mounted through '$mounted'\n";
+ }
+ my $name = basename($path);
+ $devices{$path} = 1;
+ dir_glob_foreach(
+ "/sys/class/block/$name",
+ "\Q$name\E\\d+",
+ sub($part, @) {
+ $devices{"/dev/$part"} = 1;
+ },
+ );
+ }
+ for my $path (sort keys %devices) {
+ dir_glob_foreach(
+ '/sys/class/block/' . basename($path) . '/holders',
+ '[^.].*',
+ sub($holder, @) {
+ die "cannot use '$lun->{devpath}' - another block device uses the LUN through"
+ . " '$path'\n"
+ if $holder ne $device;
+ },
+ );
+ }
+ my $openers = PVE::Diskmanage::device_openers([sort keys %devices], writable => 1);
+ for my $path (sort keys %$openers) {
+ my $opener = $openers->{$path};
+ die "cannot use '$lun->{devpath}' - process $opener->{pid} ($opener->{command}) has the"
+ . " LUN open through '$path'\n";
+ }
+}
+
1;
diff --git a/src/test/Makefile b/src/test/Makefile
index 2f6f4cd7..0d866d91 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
all: test
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api
test_zfspoolplugin: run_test_zfspoolplugin.pl
./run_test_zfspoolplugin.pl
@@ -9,9 +9,10 @@ test_lvmplugin: run_test_lvmplugin.pl run_test_lvm_throughput.pl
./run_test_lvmplugin.pl
./run_test_lvm_throughput.pl
-test_lvm_create: run_test_lvm_create.pl run_test_lvm_base_conflicts.pl
+test_lvm_create: run_test_lvm_create.pl run_test_lvm_base_conflicts.pl run_san_write_check_tests.pl
./run_test_lvm_create.pl
./run_test_lvm_base_conflicts.pl
+ ./run_san_write_check_tests.pl
test_lvm_lock_retry: run_test_lvm_lock_retry.pl
./run_test_lvm_lock_retry.pl
@@ -75,5 +76,9 @@ test_san_taint: run_san_taint_tests.pl
test_san_lun_scan: run_san_lun_scan_tests.pl
./run_san_lun_scan_tests.pl
-test_san_api: run_san_api_tests.pl
+test_san_api: run_san_api_tests.pl run_san_nvme_tests.pl
./run_san_api_tests.pl
+ ./run_san_nvme_tests.pl
+
+test_disk_api: run_disk_api_tests.pl
+ ./run_disk_api_tests.pl
diff --git a/src/test/disk_api_test.pm b/src/test/disk_api_test.pm
new file mode 100644
index 00000000..474ad663
--- /dev/null
+++ b/src/test/disk_api_test.pm
@@ -0,0 +1,247 @@
+package PVE::API2::Disks::Test;
+
+use v5.36;
+
+use lib qw(..);
+
+use PVE::API2::Disks;
+use PVE::API2::Disks::LVM;
+use PVE::Diskmanage;
+use PVE::Storage;
+use PVE::Storage::LVMPlugin;
+
+use Test::MockModule;
+use Test::More;
+
+my @transport_cases = (
+ ['none', undef],
+ ['unknown', 'unknown'],
+ ['ata', 'ata'],
+ ['sata', 'sata'],
+ ['sas', 'sas'],
+ ['spi', 'spi'],
+ ['sbp', 'sbp'],
+ ['usb', 'usb'],
+ ['fc', 'fc'],
+ ['fcoe', 'fcoe'],
+ ['iscsi', 'iscsi'],
+ ['nvme', 'nvme'],
+ ['virtio', 'virtio'],
+ ['nvme-fc', 'nvme-fc'],
+ ['nvme-tcp', 'nvme-tcp'],
+ ['nvme-rdma', 'nvme-rdma'],
+ ['nvme-loop', 'nvme-loop'],
+);
+
+my $disks = {};
+my @all;
+for my $case (@transport_cases) {
+ my ($name, $transport) = $case->@*;
+ my $devpath = "/dev/test-$name";
+ my $entry = {
+ devpath => $devpath,
+ gpt => 0,
+ osdid => -1,
+ };
+ $entry->{transport} = $transport if defined($transport);
+ $disks->{$name} = $entry;
+ push @all, $devpath;
+}
+
+my $diskmanage_module = Test::MockModule->new('PVE::Diskmanage', no_auto => 1);
+$diskmanage_module->mock('get_disks' => sub { return $disks });
+
+my $list_handler = PVE::API2::Disks->map_method_by_name('list')->{code};
+my $list_devpaths = sub($param) {
+ my $result = $list_handler->({ node => 'test-node', skipsmart => 1, $param->%* });
+ return [sort map { $_->{devpath} } $result->@*];
+};
+
+is_deeply(
+ $list_devpaths->({}), [sort @all], 'the disk list uses the default disk inventory',
+);
+
+is_deeply(
+ $list_devpaths->({ type => 'unused' }),
+ [sort @all],
+ 'unused disk selection preserves transport visibility',
+);
+
+$diskmanage_module->mock(
+ 'get_disks' => sub {
+ return {
+ 'fc-disk' => {
+ devpath => '/dev/fc-disk',
+ gpt => 0,
+ osdid => -1,
+ transport => 'fc',
+ },
+ 'fc-partition' => {
+ devpath => '/dev/fc-disk1',
+ gpt => 0,
+ osdid => -1,
+ parent => '/dev/fc-disk',
+ },
+ 'loop-disk' => {
+ devpath => '/dev/loop-disk',
+ gpt => 0,
+ osdid => -1,
+ transport => 'nvme-loop',
+ },
+ 'loop-partition' => {
+ devpath => '/dev/loop-disk1',
+ gpt => 0,
+ osdid => -1,
+ parent => '/dev/loop-disk',
+ },
+ };
+ },
+);
+
+is_deeply(
+ $list_devpaths->({ 'include-partitions' => 1 }),
+ ['/dev/fc-disk', '/dev/fc-disk1', '/dev/loop-disk', '/dev/loop-disk1'],
+ 'the plain list keeps FC disks and their partitions',
+);
+
+is_deeply(
+ $list_devpaths->({ 'include-partitions' => 1, type => 'unused' }),
+ ['/dev/fc-disk', '/dev/fc-disk1', '/dev/loop-disk', '/dev/loop-disk1'],
+ 'unused partitions preserve transport visibility',
+);
+
+subtest 'volume group creation leaves SAN locking to the LVM helper' => sub {
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $disk = $diskmanage_module;
+ my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+ my $rpc = Test::MockModule->new('PVE::RPCEnvironment');
+ my (@stack, @created);
+ my $lock = sub($name, $code) {
+ push @stack, $name;
+ my $res = eval { $code->() };
+ my $err = $@;
+ pop @stack;
+ die $err if $err;
+ return $res;
+ };
+ $storage->mock(lock_storage_config => sub { $lock->('storage.cfg', $_[0]) });
+ $disk->mock(locked_disk_action => sub { $lock->('disk', $_[0]) });
+ $disk->mock(verify_blockdev_path => sub { $_[0] });
+ $disk->mock(assert_disk_unused => sub { });
+ $disk->mock(is_partition => sub { 0 });
+ $disk->mock(udevadm_trigger => sub { });
+ $lvm->mock(lvm_vgs => sub { {} });
+ $lvm->mock(
+ lvm_create_volume_group => sub {
+ is(scalar(@_), 3, 'disk API does not exempt a storage');
+ push @created, [$_[0], join('>', @stack)];
+ },
+ );
+ {
+
+ package MultipathTestRPC;
+ sub get_user { 'root@pam' }
+ sub check { 1 }
+ sub fork_worker { $_[4]->(); return 'test-upid' }
+ }
+ $rpc->mock(get => sub { bless {}, 'MultipathTestRPC' });
+
+ my ($is_map, $is_san) = (1, 0);
+ my $san = Test::MockModule->new('PVE::Storage::SAN');
+ $san->mock(is_san_device => sub { $is_map || $is_san });
+ PVE::API2::Disks::LVM->create({ node => 'n1', name => 'vg1', device => '/dev/dm-3' });
+ ($is_map, $is_san) = (0, 1);
+ PVE::API2::Disks::LVM->create({ node => 'n1', name => 'vg2', device => '/dev/sdc' });
+ ($is_map, $is_san) = (0, 0);
+ PVE::API2::Disks::LVM->create({ node => 'n1', name => 'vg3', device => '/dev/sdb' });
+ is_deeply(
+ \@created,
+ [
+ ['/dev/dm-3', ''], ['/dev/sdc', ''], ['/dev/sdb', 'disk'],
+ ],
+ 'SAN writes use no local disk or storage configuration lock',
+ );
+ my $config_api = Test::MockModule->new('PVE::API2::Storage::Config');
+ my (@params, @registered);
+ $config_api->mock(
+ create_or_update => sub {
+ push @registered, $_[5] ? 'check' : 'create or update';
+ push @params, $_[3];
+ },
+ create => sub {
+ push @registered, 'create';
+ push @params, $_[1];
+ },
+ );
+ my $cfg =
+ { ids => { other => { type => 'lvm', vgname => 'other', nodes => { n2 => 1 } } } };
+ $storage->mock(config => sub { $cfg });
+ $is_san = 1;
+ PVE::API2::Disks::LVM->create({
+ node => 'n1',
+ name => 'vg4',
+ device => '/dev/sdc',
+ add_storage => 1,
+ });
+ is($params[-1]->{shared}, 1, 'SAN storage is shared');
+ ok(!exists($params[-1]->{nodes}), 'SAN storage covers all nodes');
+ is_deeply(
+ \@registered,
+ ['check', 'create'],
+ 'and only gets created: it never joins a storage added in the meantime',
+ );
+ @registered = ();
+ @created = ();
+ eval {
+ PVE::API2::Disks::LVM->create({
+ node => 'n1',
+ name => 'other',
+ device => '/dev/sdc',
+ add_storage => 1,
+ });
+ };
+ like($@, qr/already defined/, 'SAN volume group does not join the storage of another node');
+ is_deeply(\@created, [], 'existing storage ID prevents initialization');
+ $is_san = 0;
+ PVE::API2::Disks::LVM->create({
+ node => 'n1',
+ name => 'vg5',
+ device => '/dev/sdb',
+ add_storage => 1,
+ });
+ is($params[-1]->{shared}, 0, 'local storage is not shared');
+ is($params[-1]->{nodes}, 'n1', 'local storage remains node-restricted');
+ is_deeply(\@registered, ['check', 'create or update'], 'and may join one of its name');
+ @created = ();
+ $disk->mock(assert_disk_unused => sub { die "device is already in use\n" });
+ eval {
+ PVE::API2::Disks::LVM->create({
+ node => 'n1',
+ name => 'vg',
+ device => '/dev/sdc',
+ });
+ };
+ like($@, qr/already in use/, 'disk API rejects a used inventory entry');
+ is_deeply(\@created, [], 'used inventory prevents initialization');
+ $disk->mock(assert_disk_unused => sub { });
+ $lvm->mock(lvm_vgs => sub { { vg => {} } });
+ eval {
+ PVE::API2::Disks::LVM->create({
+ node => 'n1',
+ name => 'vg',
+ device => '/dev/sdc',
+ });
+ };
+ like(
+ $@,
+ qr/volume group with name 'vg' already exists/,
+ 'stale inventory does not permit an existing VG',
+ );
+ is_deeply(\@created, [], 'existing VG prevents initialization');
+ my @mocked = qw(
+ locked_disk_action verify_blockdev_path assert_disk_unused is_partition udevadm_trigger
+ );
+ $disk->unmock($_) for @mocked;
+};
+
+done_testing();
diff --git a/src/test/run_disk_api_tests.pl b/src/test/run_disk_api_tests.pl
new file mode 100755
index 00000000..75fc98c6
--- /dev/null
+++ b/src/test/run_disk_api_tests.pl
@@ -0,0 +1,11 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use TAP::Harness;
+
+my $harness = TAP::Harness->new({ verbosity => -2 });
+my $res = $harness->runtests("disk_api_test.pm");
+
+exit -1 if !$res || $res->{failed} || $res->{parse_errors};
diff --git a/src/test/run_san_library_tests.pl b/src/test/run_san_library_tests.pl
index f7ae68d1..d961b87c 100755
--- a/src/test/run_san_library_tests.pl
+++ b/src/test/run_san_library_tests.pl
@@ -105,4 +105,31 @@ subtest 'LVM hints are optional cached display metadata' => sub {
}
};
+subtest 'the names of a held LUN have to mean the held device' => sub {
+ require PVE::Storage::SAN;
+ open(my $fh, '<', '/dev/null') or die "cannot open /dev/null - $!\n";
+ my $lun = { 'by-id' => '/dev/null', devpath => '/dev/null' };
+ my $check = sub {
+ eval { PVE::Storage::SAN::assert_same_device(@_) };
+ return $@;
+ };
+ is($check->($lun, $fh, '/dev/null'), '', 'same device behind all names');
+ like(
+ $check->({ %$lun, 'by-id' => '/dev/zero' }, $fh),
+ qr/'\/dev\/zero' changed/,
+ 'by-id name',
+ );
+ like(
+ $check->({ %$lun, devpath => '/dev/zero' }, $fh),
+ qr/'\/dev\/zero' changed/,
+ 'device path',
+ );
+ like(
+ $check->($lun, $fh, '/dev/zero'),
+ qr/'\/dev\/zero' changed/,
+ 'name used for the commands',
+ );
+ like($check->($lun, $fh, '/dev/gone'), qr/'\/dev\/gone' changed/, 'name without a device');
+};
+
done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
new file mode 100755
index 00000000..9f243563
--- /dev/null
+++ b/src/test/run_san_nvme_tests.pl
@@ -0,0 +1,104 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+our ($backing, @opens);
+
+BEGIN {
+ *CORE::GLOBAL::sysopen = sub (*$$;$) {
+ push @opens, $_[1];
+ my $path = $_[1] =~ m{^/dev/disk/by-id/nvme-} ? $backing : $_[1];
+ return CORE::sysopen($_[0], $path, $_[2]);
+ };
+}
+use lib qw(..);
+use File::Temp qw(tempfile);
+use Test::More;
+use Test::MockModule;
+use PVE::Storage::SAN;
+
+my ($fh, $file) = tempfile('san-nvme-XXXX', DIR => '.', UNLINK => 1);
+close($fh);
+$backing = $file;
+my $device = '/dev/nvme0n1';
+my $link = '/dev/disk/by-id/nvme-eui.0123456789abcdef';
+my $lun =
+ { devpath => $device, 'by-id' => $link, wwid => 'eui.0123456789abcdef', multipath => 'native' };
+my $san = Test::MockModule->new('PVE::Storage::SAN');
+my $disk = Test::MockModule->new('PVE::Diskmanage');
+my $mp = Test::MockModule->new('PVE::Multipath');
+my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+my $storage = Test::MockModule->new('PVE::Storage');
+my $cluster = Test::MockModule->new('PVE::Cluster');
+my $cfg = { ids => {} };
+my @order;
+$storage->mock(config => sub { $cfg });
+$disk->mock(verify_blockdev_path => sub { $device });
+$disk->mock(is_partition => sub { 0 });
+$disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(is_iscsi_device => sub { 0 });
+$disk->mock(is_mounted => sub { 0 });
+$disk->mock(has_holder => sub { 0 });
+$disk->mock(device_opener => sub { undef });
+$mp->mock(device_map => sub { undef });
+$san->mock(abs_path => sub { $device });
+$san->mock(scan_luns => sub { [$lun] });
+$san->mock(lun_partitions => sub { [] });
+$san->mock(assert_same_device => sub { ok(defined(fileno($_[1])), 'held descriptor checked') });
+$lvm->mock(lun_volume_groups => sub { push @order, 'check'; [] });
+$cluster->mock(
+ cfs_lock_domain => sub {
+ is($_[0], 'san-eui.0123456789abcdef', 'native namespace uses the LUN lock');
+ push @order, 'lock';
+ $_[2]->();
+ push @order, 'unlock';
+ $@ = '';
+ },
+);
+is(PVE::Storage::SAN::stable_path($device, $lun->{wwid}), $link, 'native stable path');
+is_deeply(PVE::Storage::SAN::lun_for_device($link), $lun, 'native row selected for writes');
+my $model_link = '/dev/disk/by-id/nvme-NetApp_ONTAP_Controller_81Mkq_1';
+is_deeply(PVE::Storage::SAN::lun_for_device($model_link), $lun, 'any name of the namespace');
+{
+ # the name passes to another device while the LUNs are scanned
+ my $scanned = 0;
+ $san->mock(scan_luns => sub { $scanned = 1; [$lun] });
+ $san->mock(abs_path => sub { $_[0] eq $model_link && $scanned ? '/dev/nvme9n1' : $device });
+ eval { PVE::Storage::SAN::lun_for_device($model_link) };
+ like($@, qr/the device behind '\Q$model_link\E' changed/, 'a name that changed refuses');
+ $san->mock(scan_luns => sub { [$lun] });
+ $san->mock(abs_path => sub { $device });
+}
+$disk->mock(is_partition => sub { 1 });
+is(PVE::Storage::SAN::lun_for_device($link), undef, 'a partition is handled as a local device');
+$disk->mock(is_partition => sub { 0 });
+$cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/base volume/, 'native base reference refuses');
+$cfg->{ids} = {};
+@order = @opens = ();
+$lvm->mock(file_read_firstline => sub { $_[0] =~ m{/size$} ? 1024 : undef });
+$lvm->mock(lvm_pv_info => sub { undef });
+$lvm->mock(
+ clear_first_sector => sub { push @order, 'clear'; ok(-e $_[0], 'VG clear uses held fd') });
+$lvm->mock(
+ run_command => sub {
+ my ($cmd, %opts) = @_;
+ if ($cmd->[0] eq 'wipefs') {
+ $opts{outfunc}->('{"signatures":[]}');
+ } else {
+ is($cmd->[-1], $device, 'VG command uses the checked device');
+ push @order, $cmd->[0];
+ }
+ },
+);
+PVE::Storage::LVMPlugin::lvm_create_volume_group($link, 'data', 1);
+is_deeply(
+ \@order,
+ [qw(lock check clear /sbin/pvcreate /sbin/vgcreate unlock)],
+ 'native VG sequencing',
+);
+is_deeply(\@opens, [$link], 'VG creation holds the native by-id device');
+
+done_testing();
diff --git a/src/test/run_san_taint_tests.pl b/src/test/run_san_taint_tests.pl
index 35b11cd7..0ccd9e2f 100755
--- a/src/test/run_san_taint_tests.pl
+++ b/src/test/run_san_taint_tests.pl
@@ -31,4 +31,30 @@ for my $bad ('-bad', '1:2', '../dev', "3600a\n") {
like($@, qr/invalid WWID/, 'ambiguous identity refused');
}
+# the LUN that write operations get must be usable in command lines and lock names
+use PVE::Storage::SAN;
+my $san = Test::MockModule->new('PVE::Storage::SAN');
+my $tainted_dev = PVE::Tools::file_get_contents('/proc/self/comm') =~ s/.*//sr . '/dev/null';
+ok(tainted($tainted_dev), 'device path derived from a file is tainted');
+$san->mock(
+ is_san_device => sub { 1 },
+ scan_luns => sub {
+ return [{
+ devpath => $tainted_dev,
+ wwid => $wwid,
+ 'by-id' => '/dev/null',
+ usage => 'unused',
+ multipath => 'none',
+ }];
+ },
+);
+my $disks = Test::MockModule->new('PVE::Diskmanage');
+$disks->mock(verify_blockdev_path => sub { '/dev/null' });
+my $storage = Test::MockModule->new('PVE::Storage');
+$storage->mock(config => sub { { ids => {} } });
+my $lun = PVE::Storage::SAN::lun_for_device('/dev/null');
+ok(!tainted($lun->{devpath}), 'LUN device path is untainted');
+ok(!tainted($lun->{wwid}), 'LUN WWID is untainted');
+is($lun->{wwid}, '3600a', 'LUN WWID is unchanged');
+
done_testing();
diff --git a/src/test/run_san_write_check_tests.pl b/src/test/run_san_write_check_tests.pl
new file mode 100755
index 00000000..acd199a2
--- /dev/null
+++ b/src/test/run_san_write_check_tests.pl
@@ -0,0 +1,219 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+use lib qw(..);
+use Test::More;
+use Test::MockModule;
+use PVE::Storage::SAN;
+
+my $scan = Test::MockModule->new('PVE::Storage::SAN');
+my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
+my $storage = Test::MockModule->new('PVE::Storage');
+my $cluster = Test::MockModule->new('PVE::Cluster');
+my $lun = { wwid => '3600+abc', devpath => '/dev/dm-0', 'iscsi-target' => 'iqn.test' };
+# the LUN is exported through two targets, its row names only the first
+my $identity = Test::MockModule->new('PVE::Multipath::Identity');
+my $paths = sub {
+ {
+ paths => {
+ sdb => { wwid => '3600+abc', target => 'iqn.test' },
+ sdc => { wwid => '3600+abc', target => 'iqn.second' },
+ sdd => { wwid => '3600+other', target => 'iqn.other' },
+ },
+ };
+};
+$identity->mock(collect => $paths);
+my %sessions = (sdb => 'iqn.test', sdc => 'iqn.second', sdd => 'iqn.other');
+my $topology = Test::MockModule->new('PVE::ISCSI::Topology');
+$topology->mock(
+ device_map => sub {
+ return { map { $_ => { target => $sessions{$_} } } keys %sessions };
+ },
+);
+my @slaves;
+$scan->mock(dir_glob_foreach => sub { $_[2]->($_) for $_[0] =~ m{/dm-0/slaves$} ? @slaves : () });
+my $vgs = [];
+$scan->mock(lun_partitions => sub { [] });
+$lvm->mock(lun_volume_groups => sub { $vgs });
+$lvm->mock(cached_pv_volume_groups => sub { die "write check must not use cached PV metadata\n" });
+my $cfg = { ids => {} };
+$storage->mock(config => sub { $cfg });
+my $check = sub {
+ eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun, @_) };
+ return $@;
+};
+is($check->(), '', 'unreferenced LUN accepted');
+for my $type (qw(lvm lvmthin custom)) {
+ $vgs = ['data'];
+ $cfg->{ids} = {
+ data => {
+ type => $type,
+ vgname => 'data',
+ shared => 1,
+ disable => 1,
+ nodes => { other => 1 },
+ },
+ };
+ for my $cached_vg (undef, 'unrelated') {
+ $lun->{vgname} = $cached_vg;
+ $lun->{usage} = 'unused';
+ like($check->(), qr/used by storage 'data'/, "$type refusal ignores display VG and usage");
+ }
+ is($check->('data'), '', 'the storage being added can be exempted');
+}
+delete $lun->{vgname};
+delete $lun->{usage};
+$vgs = [];
+$cfg->{ids} = { direct => { type => 'iscsi', target => 'iqn.test', content => { images => 1 } } };
+like($check->(), qr/storage 'direct' serves images from this target$/, 'direct image use refuses');
+$cfg->{ids}->{direct}->{target} = 'iqn.second';
+like($check->(), qr/storage 'direct' serves images/, 'also through another target of the LUN');
+$cfg->{ids}->{direct}->{target} = 'iqn.other';
+is($check->(), '', 'a target of another LUN is no reference');
+{
+ # multipathd does not answer and udev knows the path by another identity
+ local $lun->{'iscsi-target'} = 'iqn.other';
+ $identity->mock(collect => sub { { paths => {} } });
+ like($check->(), qr/storage 'direct' serves images/, 'the target of the LUN itself counts');
+ $cfg->{ids}->{direct}->{target} = 'iqn.second';
+ is($check->(), '', 'which is the only one known then');
+ @slaves = qw(sdb sdc);
+ like($check->(), qr/storage 'direct' serves images/, 'unless a member of the map has another');
+ @slaves = ();
+ $identity->mock(collect => $paths);
+}
+$cfg->{ids}->{direct}->{target} = 'iqn.test';
+$cfg->{ids}->{direct}->{content} = { none => 1 };
+is($check->(), '', 'connection-only storage permits use');
+
+for my $type (qw(iscsidirect zfs)) {
+ local $cfg->{ids}->{other} = { type => $type, target => 'iqn.test', content => {} };
+ like(
+ $check->(),
+ qr/storage 'other' serves images from this target$/,
+ "$type storage of the target refuses",
+ );
+}
+$cfg->{ids}->{data} = { type => 'lvm', base => 'direct:0.0.1.dm-uuid-mpath-3600+abc' };
+like($check->(), qr/base volume/, 'base reference refuses');
+$cfg->{ids}->{data}->{base} = 'direct:0.0.1.scsi-3600+abc';
+like($check->(), qr/base volume/, 'scsi WWID in base name refuses');
+$cfg->{ids}->{data}->{base} = 'direct:0.0.1.dm-uuid-part1-mpath-3600+abc';
+like($check->(), qr/base volume/, 'a partition of the map as base refuses');
+$cfg->{ids}->{data}->{base} = 'direct:0.0.1.scsi-legacy';
+$storage->mock(path => sub { '/dev/disk/by-id/legacy' });
+$scan->mock(abs_path => sub { '/dev/dm-0' });
+like($check->(), qr/base volume/, 'resolved alias to the LUN refuses');
+$scan->mock(abs_path => sub { $_[0] eq $lun->{devpath} ? '/dev/dm-0' : '/dev/dm-1' });
+is($check->(), '', 'base on another device is no match');
+$scan->mock(abs_path => sub { undef });
+is($check->(), '', 'unresolvable base is no match');
+$storage->mock(path => sub { die "base unavailable\n" });
+is($check->(), '', 'unavailable base is no match');
+ok(!PVE::Storage::LVMPlugin->can('create_volume_group'), 'creation implementation is lexical');
+$cfg->{ids} = {};
+my @order;
+$cluster->mock(
+ cfs_lock_domain => sub {
+ is($_[0], 'san-3600=2babc', 'lock name escapes unsafe characters without collisions');
+ push @order, 'lock';
+ $_[2]->();
+ push @order, 'unlock';
+ $@ = '';
+ },
+);
+PVE::Storage::SAN::with_lun_lock($lun, sub { push @order, 'write' });
+is_deeply(\@order, [qw(lock write unlock)], 'the code runs under the lock of the LUN');
+
+# volume group creation checks the references only when it has to initialize the LUN
+my $disks = Test::MockModule->new('PVE::Diskmanage');
+$disks->mock(
+ verify_blockdev_path => sub { $_[0] },
+ is_mounted => sub { undef },
+ has_holder => sub { undef },
+ device_opener => sub { undef },
+);
+my ($signatures, $pvinfo) = ('', undef);
+$lun->{'by-id'} = '/dev/null';
+$scan->mock(
+ lun_for_device => sub { $lun },
+ assert_same_device => sub { },
+ assert_lun_unreferenced => sub { push @order, 'check'; die "referenced\n" },
+);
+$lvm->mock(
+ file_read_firstline => sub { $_[0] =~ m{/size$} ? '2048' : undef },
+ lvm_pv_info => sub { $pvinfo },
+ clear_first_sector => sub { push @order, 'clear' },
+ run_command => sub {
+ my ($cmd, %param) = @_;
+ return $param{outfunc}->(qq({"signatures":[$signatures]})) if $cmd->[0] eq 'wipefs';
+ push @order, $cmd->[0];
+ },
+);
+@order = ();
+$signatures = '{"type":"LVM2_member","offset":"0x218"}';
+$pvinfo = { vgname => 'data', in_use => '1' };
+eval {
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ '/dev/null',
+ 'data',
+ 1,
+ { exempt => 'new' },
+ );
+};
+is($@, '', 'an existing volume group can be added whatever references its LUN');
+is_deeply(\@order, [qw(lock unlock)], 'without reference check or write');
+eval {
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ '/dev/null',
+ 'other',
+ 1,
+ { exempt => 'new' },
+ );
+};
+like($@, qr/already used by volume group 'data'/, 'another volume group on the LUN refuses');
+@order = ();
+($signatures, $pvinfo) = ('', undef);
+eval {
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ '/dev/null',
+ 'data',
+ 1,
+ { exempt => 'new' },
+ );
+};
+like($@, qr/referenced/, 'initializing a referenced LUN refuses');
+is_deeply(\@order, [qw(lock check)], 'the check precedes the first write');
+$scan->mock(assert_lun_unreferenced =>
+ sub { is($_[2], 'new', 'new storage exempt'); push @order, 'check' });
+$scan->mock(assert_lun_paths_unused => sub { push @order, 'paths'; die "path in use\n" });
+@order = ();
+eval {
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ '/dev/null',
+ 'data',
+ 1,
+ { exempt => 'new' },
+ );
+};
+like($@, qr/path in use/, 'a LUN that is in use through another path refuses');
+is_deeply(\@order, [qw(lock check paths)], 'before the first write');
+$scan->mock(assert_lun_paths_unused => sub { push @order, 'paths' });
+@order = ();
+eval {
+ PVE::Storage::LVMPlugin::lvm_create_volume_group(
+ '/dev/null',
+ 'data',
+ 1,
+ { exempt => 'new' },
+ );
+};
+is($@, '', 'an unreferenced empty LUN is initialized');
+is_deeply(
+ \@order,
+ [qw(lock check paths clear /sbin/pvcreate /sbin/vgcreate unlock)],
+ 'checked once, then written under the lock',
+);
+
+done_testing();
diff --git a/src/test/run_test_lvm_create.pl b/src/test/run_test_lvm_create.pl
index d4be25b0..13058fe2 100755
--- a/src/test/run_test_lvm_create.pl
+++ b/src/test/run_test_lvm_create.pl
@@ -10,9 +10,12 @@ use Test::More;
use lib ('.', '..');
use PVE::Storage::LVMPlugin;
+use PVE::Storage::SAN;
my $lvm = Test::MockModule->new('PVE::Storage::LVMPlugin');
my $disk = Test::MockModule->new('PVE::Diskmanage');
+my $san = Test::MockModule->new('PVE::Storage::SAN');
+$san->mock(lun_for_device => sub { undef });
my $tools = Test::MockModule->new('PVE::Tools');
my $inotify = Test::MockModule->new('PVE::INotify');
$inotify->mock(nodename => sub { 'testnode' });
@@ -43,6 +46,7 @@ $lvm->mock(
if $_[0] =~ m{/(?:uevent|size)/partition$};
return $partitioned ? '1' : undef if $_[0] =~ m{/partition$};
return $holder_uuid if $_[0] =~ m{/dm-5/dm/uuid$};
+ return 'part1-mpath-3600abc' if $map eq 'partition' && $_[0] =~ m{/uuid$};
return $map ? 'mpath-3600abc' : undef if $_[0] =~ m{/uuid$};
return $running ? 'running' : 'offline' if $_[0] =~ m{/state$};
die "unexpected read $_[0]";
@@ -119,6 +123,10 @@ for my $case (
is_deeply(\@probes, [], "$case->[0]: no device reads");
is_deeply(\@writes, [], "$case->[0]: no writes");
}
+reset_case();
+($map, $running) = ('partition', 0);
+is(create(), '', 'the partition of a map has the map as member, not its paths');
+is(scalar(@writes), 3, 'so it is set up like a local partition');
reset_case();
my $storage_checks = 0;
@@ -170,6 +178,44 @@ like(
qr/contains data \(gpt\)/,
'a partition table of a map is data, kpartx may not map it',
);
+reset_case();
+($signatures, $partitioned) = ([{ type => 'PMBR' }, { type => 'gpt' }], 1);
+$san->mock(
+ lun_for_device =>
+ sub { { wwid => '3600abc', devpath => '/dev/sda', 'by-id' => '/dev/null' } },
+ with_lun_lock => sub { $_[1]->() },
+);
+like(
+ create(),
+ qr{the device behind '/dev/sda' changed},
+ 'a device name that no longer means the held LUN is refused',
+);
+is_deeply(\@writes, [], 'changed device: no writes');
+$san->mock(assert_same_device => sub { });
+like(create(), qr/contains data \(PMBR, gpt\)/, 'a partitioned LUN that is no map is data too');
+is_deeply(\@writes, [], 'partitioned LUN: no writes');
+reset_case();
+my $opener = '4711 (kvm)';
+$disk->mock(device_opener => sub { $opener });
+$san->mock(assert_lun_unreferenced => sub { });
+my $storage = Test::MockModule->new('PVE::Storage');
+$storage->mock(config => sub { { ids => {} } });
+like(create(), qr/process 4711 \(kvm\) has it open/, 'a LUN that a process has open is refused');
+is_deeply(\@writes, [], 'open LUN: no writes');
+$opener = undef;
+is(create(), '', 'a LUN that no process has open is initialized');
+is(scalar(@writes), 3, 'unused LUN: initialized');
+reset_case();
+$signatures = [{ type => 'LVM2_member' }];
+$label = '/dev/sda: LVM2 PV';
+$pv = '/dev/sda:1024:test:aaaaaa-bbbb-cccc-dddd-eeee-ffff-gggggg:1';
+my $checked = 0;
+$san->mock(assert_same_device => sub { die "device changed\n" if ++$checked == 2 });
+like(create(), qr/device changed/, 'an existing volume group only counts on the held LUN');
+$san->unmock('assert_lun_unreferenced');
+$san->unmock('assert_same_device');
+$san->unmock('with_lun_lock');
+$san->mock(lun_for_device => sub { undef });
my $uuid = 'aaaaaa-bbbb-cccc-dddd-eeee-ffff-gggggg';
for my $vg ('test', 'other', 'unknown', '') {
next prev parent reply other threads:[~2026-11-18 20:34 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 20:25 [PATCH v7 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 cluster 01/43] cfs: observe the multipath configuration files Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 02/43] diskmanage: collect disk transport type from lsblk Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 03/43] diskmanage: qualify NVMe over Fabrics transport Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 04/43] diskmanage: optionally include iSCSI session devices in disk enumeration Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 05/43] diskmanage: collect native NVMe multipath path state Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 06/43] iscsi: factor out session device map from device list Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 07/43] multipath: add a library around multipathd Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 08/43] diskmanage: add helper to list multipath devices Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 09/43] multipath: add a cluster-wide configuration Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 10/43] iscsi: add an option to manage storage LUNs with multipath Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 11/43] lvm: activate a missing base volume before using the volume group Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 12/43] multipath: apply the configuration on every node Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 13/43] multipath: publish per-node status Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 14/43] api: add endpoints for the cluster-wide multipath configuration Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 15/43] api: multipath: add the cluster-wide health status Thomas Lamprecht
2026-11-18 20:25 ` [PATCH v7 storage 16/43] diskmanage: look up the requested disks by name Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 17/43] diskmanage: read the udev data of all disks with one call Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 18/43] api: scan: add a SAN LUN inventory Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 19/43] pvesm: expose the " Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 20/43] api: add node-level iSCSI status Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 21/43] api: scan: name the portal of an iSCSI path Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 22/43] disks: lvm: drop unused imports Thomas Lamprecht
2026-11-18 20:26 ` Thomas Lamprecht [this message]
2026-11-18 20:26 ` [PATCH v7 storage 24/43] disks: lvm: refuse to destroy the volume group of a shared storage Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 25/43] disks: allow wiping a SAN LUN that no storage uses Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 storage 26/43] api: scan: report the process that has a SAN LUN open Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 docs 27/43] storage: document the cluster-wide multipath management Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 docs 28/43] storage: multipath: describe the use shown for an open LUN Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 29/43] pvestatd: apply the multipath configuration and report the node status Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 30/43] report: add the managed multipath configuration Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 31/43] api: cluster: add the multipath endpoints Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 32/43] api: nodes: expose the iSCSI status Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 33/43] ui: dc: add a SAN view Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 34/43] ui: storage: iscsi: add the multipath option Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 35/43] ui: lvm: enable shared when a base storage is selected Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 36/43] ui: node: add SAN LUNs panel Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 37/43] ui: node: san: wipe a LUN that no storage uses Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 38/43] ui: node: san: show iSCSI targets and sessions Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 39/43] ui: dc: san: add a guided setup for shared LVM on iSCSI Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 40/43] ui: dc: san: add a guided setup for shared LVM on Fibre Channel Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 41/43] ui: storage: point the iSCSI and LVM add dialogs to the SAN view Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 42/43] ui: san: show the guest or process that has a LUN open Thomas Lamprecht
2026-11-18 20:26 ` [PATCH v7 manager 43/43] ui: node: san: add a column for the target or subsystem of a LUN Thomas Lamprecht
2026-11-19 19:49 ` superseded: [PATCH v7 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118202646.3017286-24-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.