From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9E2921FF0E3 for ; Wed, 18 Nov 2026 21:45:54 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C57EA21851; Wed, 18 Nov 2026 21:44:02 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Date: Wed, 18 Nov 2026 21:42:21 +0100 Message-ID: <20261118204302.3100813-3-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com> References: <20261118204302.3100813-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795034587209 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.987 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6XF3JYQYMVWSIL2X7OHYWNMCBLDA3U27 X-Message-ID-Hash: 6XF3JYQYMVWSIL2X7OHYWNMCBLDA3U27 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: NVMe/TCP namespaces need a manual nvme-cli setup on every node so far. Add a storage type that connects one subsystem through each configured portal and offers its namespaces as raw volumes. Native NVMe multipath joins the paths and the kernel reconnects lost ones. A second storage for a subsystem is refused: the LUN list, the controller list and the node status attribute a subsystem to one storage. Resolving the path of a volume checks an existing device against the subsystem: a volume name can be the by-id name of any NVMe device, and not every user of a path activates the volume first. The plugin isolates its properties, so names like 'portals' belong to this storage type alone, and a third-party plugin that declares one, like 'subsysnqn', is not left out. As the first in-tree plugin that isolates, it makes the storage API check the parameters for adding and updating a storage of any type against the options of that type alone. A third-party plugin then has to list the sensitive properties it uses in options(), as the API change log says. Local disk setup refuses namespaces attached over a fabric, as for iSCSI LUNs. Wiping a namespace or creating a volume group on it is refused while a storage serves images from its subsystem. nvme-cli is only recommended, as no other storage type needs it. Signed-off-by: Thomas Lamprecht --- ApiChangeLog | 7 +- debian/control | 1 + src/PVE/Diskmanage.pm | 2 + src/PVE/Makefile | 1 + src/PVE/NVMeOF.pm | 244 +++++++++++++++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NVMeOFPlugin.pm | 223 +++++++++++++++++++++ src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SAN.pm | 13 ++ src/test/Makefile | 6 +- src/test/run_nvmeof_taint_tests.pl | 60 ++++++ src/test/run_nvmeof_tests.pl | 302 +++++++++++++++++++++++++++++ src/test/run_san_nvme_tests.pl | 16 ++ 14 files changed, 875 insertions(+), 5 deletions(-) create mode 100644 src/PVE/NVMeOF.pm create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm create mode 100755 src/test/run_nvmeof_taint_tests.pl create mode 100755 src/test/run_nvmeof_tests.pl diff --git a/ApiChangeLog b/ApiChangeLog index 10f39508..d7310fe4 100644 --- a/ApiChangeLog +++ b/ApiChangeLog @@ -101,9 +101,10 @@ Future changes should be documented in here. declares a property of that name is then not loaded until it renames it, so prefer names that are specific to the plugin, for example prefixed with its type. - As soon as one such plugin is installed, the API checks the parameters for adding and updating a - storage against the options of its type alone, for all storage types. So every plugin has to - list the sensitive properties that it uses in `options()`. + Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for + adding and updating a storage against the options of its type alone, for all storage types, + also for plugins of older API versions. So every plugin has to list the sensitive properties + that it uses in `options()`. Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of version 11 also applies to the plugin's own properties of these names. diff --git a/debian/control b/debian/control index d10dfb23..8b9b6455 100644 --- a/debian/control +++ b/debian/control @@ -55,6 +55,7 @@ Depends: bzip2, ${misc:Depends}, ${perl:Depends}, Recommends: multipath-tools, + nvme-cli, pve-esxi-import-tools (>= 0.6.0), skopeo, zfsutils-linux (>= 2.4.1-pve2), diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm index 8fda7afc..1ad91e06 100644 --- a/src/PVE/Diskmanage.pm +++ b/src/PVE/Diskmanage.pm @@ -108,6 +108,8 @@ sub disk_is_used { my $disklist = get_disks($dev, 1, 1); die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev}); + die "'$disk' is not a valid local disk\n" + if ($disklist->{$dev}->{transport} // '') =~ /^nvme-(?:tcp|rdma|fc)$/; return 1 if $disklist->{$dev}->{used}; return 0; diff --git a/src/PVE/Makefile b/src/PVE/Makefile index 13fecbf8..238eefd4 100644 --- a/src/PVE/Makefile +++ b/src/PVE/Makefile @@ -4,6 +4,7 @@ install: install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm + install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm new file mode 100644 index 00000000..ebaedf9b --- /dev/null +++ b/src/PVE/NVMeOF.pm @@ -0,0 +1,244 @@ +package PVE::NVMeOF; + +use v5.36; + +use Digest::SHA qw(sha1_hex); +use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC); +use File::Path qw(make_path); +use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton); + +use PVE::JSONSchema; +use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command); + +use PVE::Diskmanage; + +our $NVME = '/usr/sbin/nvme'; +our $SYSFS = '/sys'; +our $HOSTNQN_FILE = '/etc/nvme/hostnqn'; +our $HOSTID_FILE = '/etc/nvme/hostid'; +our $RUN_DIR = '/run/pve/nvmeof'; +our $NQN_PATTERN = 'nqn\.[^\s,=]+'; + +sub canonical_address($addr) { + if (defined(my $packed = inet_pton(AF_INET, $addr))) { + return inet_ntop(AF_INET, $packed); + } + if (defined(my $packed = inet_pton(AF_INET6, $addr))) { + return inet_ntop(AF_INET, substr($packed, 12)) + if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff"; + return inet_ntop(AF_INET6, $packed); + } + return lc($addr); +} + +# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal. +sub parse_portal($portal) { + my ($addr, $port); + if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) { + ($addr, $port) = ($1, $2); + } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) { + ($addr, $port) = ($1, $2); + } else { + return undef; + } + return undef if defined($port) && ($port < 1 || $port > 65535); + $addr = canonical_address($addr); + if (defined(my $packed = inet_pton(AF_INET, $addr))) { + my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0; + return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast; + } elsif (defined($packed = inet_pton(AF_INET6, $addr))) { + my $prefix = unpack('n', $packed); + return undef + if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00; + } else { + return undef; + } + return { traddr => $addr, trsvcid => int($port // 4420) }; +} + +PVE::JSONSchema::register_format( + 'pve-storage-nvme-portal', + sub($portal, $noerr = undef) { + return $portal if parse_portal($portal); + return undef if $noerr; + die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n"; + }, +); + +sub format_path($path) { + my $addr = $path->{traddr} =~ /:/ ? "[$path->{traddr}]" : $path->{traddr}; + return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr; +} + +sub storage_paths($scfg) { + return [map { parse_portal($_) // die "invalid portal '$_'\n" } + PVE::Tools::split_list($scfg->{portals})]; +} + +sub host_identity() { + my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/; + my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~ + /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i; + if (!defined($nqn) || length($nqn) > 223) { + die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the" + . " host identity\n"; + } + die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n" + if !defined($id) || $id =~ /^[0-]+$/; + return { nqn => $nqn, id => lc($id) }; +} + +sub is_supported() { return -x $NVME ? 1 : 0 } + +sub assert_support() { + die "nvme-cli is not installed\n" if !is_supported(); + run_command(['modprobe', 'nvme-tcp'], timeout => 10) + if !-e "$SYSFS/module/nvme_tcp"; + my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // ''; + die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y'; +} + +sub controllers() { + my $res = {}; + my $dir = "$SYSFS/class/nvme-fabrics/ctl"; + dir_glob_foreach( + $dir, + 'nvme[0-9]+', + sub($name, @) { + my $address = file_read_firstline("$dir/$name/address") // return; + my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address); + return if !defined($address{traddr}); + $res->{$name} = { + transport => file_read_firstline("$dir/$name/transport") // '', + subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '', + state => file_read_firstline("$dir/$name/state") // 'unknown', + traddr => canonical_address($address{traddr}), + trsvcid => $address{trsvcid}, + }; + }, + ); + return $res; +} + +sub subsystem_controllers($subsystem) { + my $controllers = controllers(); + return { + map { $_ => $controllers->{$_} } grep { + $controllers->{$_}->{subsystem} eq $subsystem + && $controllers->{$_}->{transport} eq 'tcp' + } keys %$controllers + }; +} + +sub path_controller($controllers, $path) { + for my $name (sort keys %$controllers) { + my $ctrl = $controllers->{$name}; + if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) { + return $name; + } + } + return undef; +} + +sub namespaces($subsystem) { + my $res = {}; + my $dir = "$SYSFS/class/nvme-subsystem"; + dir_glob_foreach( + $dir, + 'nvme-subsys[0-9]+', + sub($subsys, @) { + my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return; + return if $nqn ne $subsystem; + dir_glob_foreach( + "$dir/$subsys", + 'nvme[0-9]+n[0-9]+', + sub($dev, @) { + my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return; + my $size = file_read_firstline("$SYSFS/block/$dev/size") // return; + $res->{$dev} = { wwid => $wwid, size => int($size) * 512 }; + }, + ); + }, + ); + return $res; +} + +sub with_lock($code) { + my ($res, $timeout) = PVE::Tools::run_fork_with_timeout( + 35, + sub { + make_path($RUN_DIR); + open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n"; + flock($lock, LOCK_EX | LOCK_NB) + or die "NVMe over Fabrics is busy - another operation is still running\n"; + # A command still in the kernel must retain serialization after its caller times out. + my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n"; + fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC) + or die "cannot inherit NVMe lock - $!\n"; + return $code->(); + }, + ); + die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout; + return $res; +} + +my sub retry_file($scfg, $path) { + my ($key) = + sha1_hex(join( + "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~ + /\A([0-9a-f]{40})\z/; + return "$RUN_DIR/retry/$key"; +} + +sub in_cooldown($scfg, $path) { + my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0; + my $age = time() - $mtime; + return $age > -60 && $age < 60; +} + +sub note_connect_failure($scfg, $path) { + make_path("$RUN_DIR/retry"); + file_set_contents(retry_file($scfg, $path), ''); +} + +sub connect_path($scfg, $path) { + my ($nqn) = $scfg->{subsysnqn} =~ /\A($NQN_PATTERN)\z/; + if (!defined($nqn) || length($nqn) > 223 || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery') { + die "invalid subsystem NQN\n"; + } + my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/; + die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1; + return with_lock(sub { + return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path); + assert_support(); + my $host = host_identity(); + run_command( + [ + $NVME, + 'connect', + '--transport=tcp', + '--nqn', + $nqn, + '--traddr', + $path->{traddr}, + '--trsvcid', + $path->{trsvcid}, + '--hostnqn', + $host->{nqn}, + '--hostid', + $host->{id}, + '--ctrl-loss-tmo=-1', + '--fast_io_fail_tmo', + $fail, + ], + timeout => 30, + ); + # nvme-cli can exit successfully without having connected + die "no controller appeared\n" + if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path); + unlink(retry_file($scfg, $path)); + return; + }); +} + +1; diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index 1e9c670a..91283ff2 100755 --- a/src/PVE/Storage.pm +++ b/src/PVE/Storage.pm @@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin; use PVE::Storage::NFSPlugin; use PVE::Storage::CIFSPlugin; use PVE::Storage::ISCSIPlugin; +use PVE::Storage::NVMeOFPlugin; use PVE::Storage::RBDPlugin; use PVE::Storage::CephFSPlugin; use PVE::Storage::ISCSIDirectPlugin; @@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register(); PVE::Storage::NFSPlugin->register(); PVE::Storage::CIFSPlugin->register(); PVE::Storage::ISCSIPlugin->register(); +PVE::Storage::NVMeOFPlugin->register(); PVE::Storage::RBDPlugin->register(); PVE::Storage::CephFSPlugin->register(); PVE::Storage::ISCSIDirectPlugin->register(); diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile index a2df83c0..7f2ff6c8 100644 --- a/src/PVE/Storage/Makefile +++ b/src/PVE/Storage/Makefile @@ -6,6 +6,7 @@ SOURCES= \ NFSPlugin.pm \ CIFSPlugin.pm \ ISCSIPlugin.pm \ + NVMeOFPlugin.pm \ CephFSPlugin.pm \ RBDPlugin.pm \ ISCSIDirectPlugin.pm \ diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm new file mode 100644 index 00000000..cb514c1d --- /dev/null +++ b/src/PVE/Storage/NVMeOFPlugin.pm @@ -0,0 +1,223 @@ +package PVE::Storage::NVMeOFPlugin; + +use v5.36; + +use Cwd (); +use Time::HiRes (); + +use PVE::Network; + +use PVE::Diskmanage; +use PVE::NVMeOF; +use PVE::Storage::Plugin; + +use base qw(PVE::Storage::Plugin); + +sub type { return 'nvmeof' } + +sub plugindata { + return { + content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }], + select_existing => 1, + 'isolate-properties' => 1, + 'advanced-properties' => { + 'fast-io-fail-tmo' => 1, + }, + 'hidden-properties' => { + bwlimit => 1, + }, + 'sensitive-properties' => {}, + }; +} + +sub properties { + return { + subsysnqn => { + title => "Subsystem", + description => 'The NQN of the NVMe subsystem.', + type => 'string', + pattern => $PVE::NVMeOF::NQN_PATTERN, + maxLength => 223, + }, + portals => { + title => "Portals", + description => + 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses' + . ' with an optional port (default 4420).', + type => 'string', + format => 'pve-storage-nvme-portal-list', + }, + 'fast-io-fail-tmo' => { + title => "Fail I/O After (s)", + description => 'Seconds before I/O through a reconnecting path fails. -1 disables' + . ' this timeout; I/O can remain queued while paths reconnect.', + type => 'integer', + minimum => -1, + default => 120, + }, + }; +} + +sub options { + return { + subsysnqn => { fixed => 1 }, + portals => {}, + 'fast-io-fail-tmo' => { optional => 1 }, + nodes => { optional => 1 }, + disable => { optional => 1 }, + content => { optional => 1 }, + bwlimit => { optional => 1 }, + }; +} + +my sub assert_valid_config($scfg) { + die "the discovery subsystem cannot be used as storage\n" + if $scfg->{subsysnqn} eq 'nqn.2014-08.org.nvmexpress.discovery'; + my %seen; + for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) { + my $portal = PVE::NVMeOF::format_path($path); + die "portal '$portal' is listed more than once\n" if $seen{$portal}++; + } +} + +sub on_add_hook($class, $storeid, $scfg, %param) { + assert_valid_config($scfg); + # the LUN and controller lists and the node status attribute a subsystem to its one storage + my $ids = PVE::Storage::config()->{ids}; + for my $sid (sort keys %$ids) { + next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof'; + die "storage '$sid' already uses this subsystem\n" + if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn}; + } + return; +} + +sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) { + my $merged = { %$scfg, %$update }; + delete $merged->{$_} for ($delete // [])->@*; + assert_valid_config($merged); + return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive); +} + +sub parse_volname($class, $volname) { + if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) { + return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw'); + } + die "unable to parse NVMe over Fabrics volume name '$volname'\n"; +} + +# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a +# namespace of another subsystem. +sub assert_namespace($scfg, $volname, $real) { + my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!; + die "volume '$volname' is not an NVMe namespace\n" if !defined($dev); + my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$dev/device/subsysnqn") // ''; + die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n" + if $nqn ne $scfg->{subsysnqn}; +} + +sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) { + die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname); + my ($vtype, $name, $vmid) = $class->parse_volname($volname); + my $path = "/dev/disk/by-id/$name"; + # not every user of a path activates the volume first, and the base volume of an LVM storage + # has no device before its storage connects + my $real = Cwd::realpath($path); + assert_namespace($scfg, $volname, $real) if defined($real) && -e $real; + return wantarray ? ($path, $vmid, $vtype) : $path; +} + +sub create_base { die "can't create base images in NVMe over Fabrics storage\n" } +sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" } +sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" } +sub free_image { die "can't free space in NVMe over Fabrics storage\n" } +sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" } + +sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) { + my $res = $class->list_images($storeid, $scfg, $vmid); + $_->{content} = 'images' for @$res; + return $res; +} + +sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) { + return [] if defined($vmid) && !$vollist; + my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn}); + my $res = []; + for my $dev (sort keys %$namespaces) { + my $ns = $namespaces->{$dev}; + my $volid = "$storeid:nvme-$ns->{wwid}"; + next if $vollist && !grep { $_ eq $volid } @$vollist; + push @$res, + { + volid => $volid, + format => 'raw', + size => $ns->{size}, + vmid => 0, + }; + } + return $res; +} + +sub status($class, $storeid, $scfg, $cache = undef, @) { + my $active = grep { $_->{state} eq 'live' } + values PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*; + return (0, 0, 0, $active ? 1 : 0); +} + +sub activate_storage($class, $storeid, $scfg, $cache = undef, @) { + my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) { + next if defined(PVE::NVMeOF::path_controller($controllers, $path)); + next if PVE::NVMeOF::in_cooldown($scfg, $path); + eval { + die "no answer from portal\n" + if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2); + PVE::NVMeOF::connect_path($scfg, $path); + }; + if (my $err = $@) { + PVE::NVMeOF::note_connect_failure($scfg, $path); + warn "storage '$storeid': connecting to '" + . PVE::NVMeOF::format_path($path) + . "' failed - $err"; + } + } + return; +} + +sub deactivate_storage { return } + +sub check_connection($class, $storeid, $scfg, @) { + my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + return 1 if grep { $_->{state} eq 'live' } values %$controllers; + for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) { + next if defined(PVE::NVMeOF::path_controller($controllers, $path)); + next if PVE::NVMeOF::in_cooldown($scfg, $path); + return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2); + PVE::NVMeOF::note_connect_failure($scfg, $path); + } + return 0; +} + +sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) { + my $path = $class->filesystem_path($scfg, $volname, $snapname); + # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device + my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10); + while (!-b $path && Time::HiRes::time() < $deadline) { + last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*; + Time::HiRes::usleep(250_000); + } + my $real = Cwd::realpath($path); + die "volume '$volname' is not available on this node\n" if !$real || !-b $real; + assert_namespace($scfg, $volname, $real); +} + +sub volume_has_feature( + $class, $scfg, $feature, $storeid, $volname, + $snapname = undef, + $running = undef, @, +) { + return 1 if $feature eq 'copy' && !defined($snapname); + return undef; +} + +1; diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index 724c2d35..6e305580 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw( ); our @SHARED_STORAGE = ( - 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', + 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof', ); our $QCOW2_PREALLOCATION = { diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm index b025c03f..86469f5c 100644 --- a/src/PVE/Storage/SAN.pm +++ b/src/PVE/Storage/SAN.pm @@ -298,6 +298,11 @@ sub lun_partitions($lun) { } sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) { + my $name = basename($lun->{devpath}); + my $nqn = + $name =~ /^nvme\d+n\d+$/ + ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn") + : undef; my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups( [$lun->{devpath}, lun_partitions($lun)->@*], )->@*; @@ -320,6 +325,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) { ) { die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n"; } + if ( + $type eq 'nvmeof' + && defined($nqn) + && $scfg->{subsysnqn} eq $nqn + && PVE::Storage::san_storage_serves_images($scfg) + ) { + die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n"; + } next if $type ne 'lvm' || !$scfg->{base}; my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base}); my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/; diff --git a/src/test/Makefile b/src/test/Makefile index 279ba1ee..c88666f5 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -1,6 +1,6 @@ all: test -test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api +test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api test_nvmeof test_zfspoolplugin: run_test_zfspoolplugin.pl ./run_test_zfspoolplugin.pl @@ -83,3 +83,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl test_disk_api: run_disk_api_tests.pl ./run_disk_api_tests.pl + +test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl + ./run_nvmeof_tests.pl + perl -T ./run_nvmeof_taint_tests.pl diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl new file mode 100755 index 00000000..62405291 --- /dev/null +++ b/src/test/run_nvmeof_taint_tests.pl @@ -0,0 +1,60 @@ +#!/usr/bin/perl -T + +use v5.36; +use lib qw(..); +use File::Temp qw(tempdir); +use Scalar::Util qw(tainted); +use Test::More; +use Test::MockModule; +use PVE::NVMeOF; +use PVE::Tools qw(file_set_contents file_get_contents); + +$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin'; +delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; +my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1); +local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn"; +local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid"; +local $PVE::NVMeOF::RUN_DIR = "$dir/run"; +file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n"); +file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n"); +file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n"); +my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config")); +ok(tainted($nqn), 'configuration input is tainted'); +my $nvme = Test::MockModule->new('PVE::NVMeOF'); +$nvme->mock(with_lock => sub { $_[0]->() }); +my $commands = 0; +$nvme->mock( + subsystem_controllers => sub { + return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {}; + }, +); +$nvme->mock(assert_support => sub { }); +$nvme->mock( + run_command => sub($cmd, %opts) { + $commands++; + is( + scalar(grep { tainted($_) } @$cmd), + 0, + 'all connect operands are validated and untainted', + ); + PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5); + }, +); +my $path = PVE::NVMeOF::parse_portal($portal); +eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) }; +is($@, '', 'connect command can execute in taint mode'); +is($commands, 1, 'one connect'); +PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path); +ok( + PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path), + 'cooldown accepts validated file-backed configuration', +); +for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') { + eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) }; + like($@, qr/invalid subsystem/, 'invalid connect subsystem refused'); +} +file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n"); +eval { PVE::NVMeOF::host_identity() }; +like($@, qr/nonzero UUID/, 'zero host ID refused'); + +done_testing(); diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl new file mode 100755 index 00000000..b75f7a5a --- /dev/null +++ b/src/test/run_nvmeof_tests.pl @@ -0,0 +1,302 @@ +#!/usr/bin/perl + +use v5.36; +use lib qw(..); +use Fcntl qw(:flock); +use File::Path qw(make_path); +use File::Temp qw(tempdir); +use Test::MockModule; +use Test::More; + +use PVE::Tools qw(file_set_contents); + +use PVE::NVMeOF; +use PVE::Storage; + +my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1); +local $PVE::NVMeOF::RUN_DIR = "$dir/run"; +local $PVE::NVMeOF::SYSFS = "$dir/sys"; +my $class = 'PVE::Storage::NVMeOFPlugin'; +my $cfg = { + type => 'nvmeof', + subsysnqn => 'nqn.test', + portals => '192.0.2.1,[2001:db8::2]:4421', + content => { none => 1 }, +}; + +subtest 'configuration, portals and volumes' => sub { + is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered'); + is_deeply( + [sort keys $class->properties()->%*], + [qw(fast-io-fail-tmo portals subsysnqn)], + 'only three plugin-specific properties', + ); + my sub nvmeof_schema($schema) { + my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*; + my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types; + return $nvmeof->{properties} // {}; + } + my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema()); + is_deeply( + [sort grep { !$create->{$_}->{optional} } keys $create->%*], + [qw(portals subsysnqn)], + 'adding a storage requires the subsystem and the portals', + ); + my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema()); + ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed'); + my $parsed = PVE::Storage::Plugin->parse_config( + 'storage.cfg', + "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n", + ); + is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin'); + is_deeply( + $parsed->{ids}->{fabric}->{content}, + { none => 1 }, + 'connection-only content parses', + ); + is_deeply( + PVE::NVMeOF::parse_portal('192.0.2.1'), + { traddr => '192.0.2.1', trsvcid => 4420 }, + 'default I/O port', + ); + is_deeply( + PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'), + { traddr => '2001:db8::1', trsvcid => 4421 }, + 'canonical IPv6', + ); + + for my $bad ( + 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1', + ) { + ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad"); + } + eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); }; + like($@, qr/listed more than once/, 'duplicate portals refused'); + eval { + $class->on_add_hook( + 'fabric', + { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' }, + ); + }; + like($@, qr/discovery subsystem/, 'discovery subsystem refused'); + my $storage = Test::MockModule->new('PVE::Storage'); + my $ids = { fabric => $cfg, local => { type => 'dir' } }; + $storage->mock(config => sub { { ids => $ids } }); + is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way'); + eval { $class->on_add_hook('second', {%$cfg}) }; + like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem'); + is( + scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')), + '/dev/disk/by-id/nvme-eui.0123456789abcdef', + 'stable volume path', + ); + { + my $disk = Test::MockModule->new('PVE::Diskmanage'); + my $nqn = 'nqn.other'; + $disk->redefine(read_subsysnqn => sub { $nqn }); + my $check = sub { + eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) }; + $@; + }; + like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem'); + like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type'); + like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition'); + $nqn = 'nqn.test'; + is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem'); + my $cwd = Test::MockModule->new('Cwd'); + $cwd->redefine(realpath => sub { $dir }); + eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') }; + like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked'); + } + + for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') { + eval { $class->parse_volname($bad) }; + like($@, qr/unable to parse/, "invalid volume $bad"); + } + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } }); + is( + $class->list_volumes('fabric', $cfg)->[0]->{volid}, + 'fabric:nvme-eui.abc', + 'content none still offers bases', + ); + is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner'); + $nvme->mock(subsystem_controllers => sub { {} }); + eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') }; + like($@, qr/not available/, + 'missing namespace fails without waiting on absent controllers'); + my ($now, $queries) = (0, 0); + my $clock = Test::MockModule->new('Time::HiRes'); + $clock->redefine(time => sub { $now += 5 }, usleep => sub { }); + $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } }); + eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) }; + is($queries, 1, 'a missing namespace of a connected subsystem is waited for'); + eval { + $class->activate_volume( + 'fabric', + $cfg, + 'nvme-nonexistent-test', + undef, + { status_only => 1 }, + ); + }; + like($@, qr/not available/, 'in a status pass it fails as well'); + is($queries, 1, 'but without a wait'); + is( + ($class->parse_volname('nvme-eui.0123456789abcdef'))[0], + 'vm-vol', + 'namespaces are VM disks', + ); +}; + +subtest 'sysfs topology' => sub { + my $sys = $PVE::NVMeOF::SYSFS; + make_path( + "$sys/class/nvme-fabrics/ctl/nvme1", + "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1", + "$sys/block/nvme0n1", + ); + my %attrs = ( + address => 'traddr=192.0.2.1,trsvcid=4420', + transport => 'tcp', + subsysnqn => 'nqn.test', + state => 'connecting', + ); + file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs; + file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n"); + file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n"); + file_set_contents("$sys/block/nvme0n1/size", "8\n"); + is( + PVE::NVMeOF::controllers()->{nvme1}->{state}, + 'connecting', + 'reconnecting controller is still present', + ); + is( + PVE::NVMeOF::path_controller( + PVE::NVMeOF::subsystem_controllers('nqn.test'), + PVE::NVMeOF::parse_portal('192.0.2.1'), + ), + 'nvme1', + 'portal controller found', + ); + is_deeply( + PVE::NVMeOF::namespaces('nqn.test'), + { nvme0n1 => { wwid => 'eui.abc', size => 4096 } }, + 'namespace facts', + ); + is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded'); +}; + +subtest 'serialized commands inherit the lock' => sub { + my $value = PVE::NVMeOF::with_lock(sub { + my $out = ''; + PVE::Tools::run_command( + [ + '/usr/bin/perl', + '-MFcntl=:flock', + '-e', + 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*"); die "not inherited" if !@held; open(my $fh, ">>", $ARGV[0]) or die $!; die "lock missing" if flock($fh, LOCK_EX|LOCK_NB); print "locked\\n";', + "$PVE::NVMeOF::RUN_DIR/fabrics.lock", + ], + timeout => 5, + outfunc => sub { $out .= $_[0] }, + ); + return $out; + }); + is($value, 'locked', 'exec cannot acquire the lock held by its parent'); + open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!; + flock($lock, LOCK_EX) or die $!; + eval { + PVE::NVMeOF::with_lock(sub { die "must not count paths\n" }); + }; + like($@, qr/is busy/, 'busy operation never enters callback'); + close($lock); + + my $tools = Test::MockModule->new('PVE::Tools'); + my $timed_out = 1; + $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) }); + my $retained; + eval { + PVE::NVMeOF::with_lock(sub { + my ($path) = + grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*'); + my ($fd) = $path =~ m{/(\d+)$}; + open($retained, '>&', $fd) or die $!; + return 1; + }); + }; + like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed'); + $timed_out = 0; + eval { + PVE::NVMeOF::with_lock(sub { die "must not count paths\n" }); + }; + like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns'); + close($retained); + is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization'); +}; + +subtest 'connect options, kernel reconnect and cooldown' => sub { + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + my $net = Test::MockModule->new('PVE::Network'); + my @commands; + my $controllers = {}; + $nvme->mock(with_lock => sub { $_[0]->() }); + $nvme->mock(assert_support => sub { }); + $nvme->mock(host_identity => + sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } }); + $nvme->mock(subsystem_controllers => sub { $controllers }); + my $appear = 1; + $nvme->mock( + run_command => sub($cmd, %opts) { + is($opts{timeout}, 30, 'bounded connect'); + push @commands, $cmd; + my %arg = + map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd; + my $controller = + { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' }; + $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear; + }, + ); + $net->mock(tcp_ping => sub { 1 }); + $class->activate_storage('fabric', $cfg); + is(scalar(@commands), 2, 'one connect per missing portal'); + ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely'); + is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default'); + @commands = (); + $controllers = { + nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' }, + nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' }, + }; + $class->activate_storage('fabric', $cfg); + is_deeply(\@commands, [], + 'existing controllers are never reconnected, regardless of state'); + $class->deactivate_storage('fabric', $cfg); + is_deeply(\@commands, [], 'deactivation does not remove controllers'); + $controllers = {}; + my $path = PVE::NVMeOF::storage_paths($cfg)->[0]; + PVE::NVMeOF::note_connect_failure($cfg, $path); + ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown'); + $class->activate_storage('fabric', $cfg); + is(scalar(@commands), 1, 'only the other portal connects'); + ($controllers, $appear) = ({}, 0); + my $other = PVE::NVMeOF::storage_paths($cfg)->[1]; + my $warning = ''; + { + local $SIG{__WARN__} = sub { $warning .= $_[0] }; + $class->activate_storage('fabric', $cfg); + } + like($warning, qr/no controller appeared/, 'a connect without a controller is a failure'); + ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too'); +}; + +subtest 'local disk setup rejects fabric namespaces by transport' => sub { + my $disk = Test::MockModule->new('PVE::Diskmanage'); + for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) { + $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } }); + eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') }; + like($@, qr/not a valid local disk/, + "$transport refuses without storage configuration"); + } +}; + +done_testing(); diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl index 51236f87..c74b638a 100755 --- a/src/test/run_san_nvme_tests.pl +++ b/src/test/run_san_nvme_tests.pl @@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg }); $disk->mock(verify_blockdev_path => sub { $device }); $disk->mock(is_partition => sub { 0 }); $disk->mock(get_nvme_transport => sub { 'fc' }); +$disk->mock(read_subsysnqn => sub { 'nqn.test' }); $disk->mock(is_iscsi_device => sub { 0 }); $disk->mock(is_mounted => sub { 0 }); $disk->mock(has_holder => sub { 0 }); @@ -78,6 +79,21 @@ $disk->mock(is_partition => sub { 0 }); $cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' }; eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; like($@, qr/base volume/, 'native base reference refuses'); +$cfg->{ids} = + { fabric => + { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +like( + $@, + qr/serves images from this subsystem/, + 'direct namespace image use refuses even when disabled', +); +$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks'); +$cfg->{ids}->{fabric}->{content} = { none => 1 }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +is($@, '', 'connection-only storage permits namespace initialization'); $cfg->{ids} = {}; @order = @opens = (); $tools->mock(file_get_contents => sub { 1024 });