From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 0D2421FF0B2 for ; Sun, 22 Nov 2026 02:28:52 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 6E114217CC; Sun, 22 Nov 2026 02:27:19 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Date: Sun, 22 Nov 2026 02:25:39 +0100 Message-ID: <20261122012605.1848824-3-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261122012605.1848824-1-t.lamprecht@proxmox.com> References: <20261122012605.1848824-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795310771564 X-SPAM-LEVEL: Spam detection results: 0 AWL -1.003 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: VJ2ZQ45LJCCWOPI773QG33RVT5VGLTED X-Message-ID-Hash: VJ2ZQ45LJCCWOPI773QG33RVT5VGLTED X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: NVMe/TCP namespaces need a manual nvme-cli setup on every node so far. Add a storage type that connects one subsystem through each configured portal and offers its namespaces as raw volumes. Native NVMe multipath joins the paths and the kernel reconnects lost ones. Only one storage may use a subsystem, as the LUN list, the controller list and the node status attribute a subsystem to one storage. The activation fails if no usable path exists after connecting the missing portals, so that adding a storage tells why. As adding a storage holds the lock of the configuration, it tries no further portal after 15 seconds once a controller is live, and leaves the rest to the next activation. A portal that failed is retried after a minute, and adding a storage forgets those failures. Disk setup refuses namespaces attached over a fabric, as for iSCSI LUNs, and nvme-cli is only recommended, as no other storage type needs it. The plugin isolates its properties, so names like 'portals' belong to this storage type alone, and a third-party plugin that declares one of them is still loaded. As the first in-tree plugin that isolates, it makes the storage API check the parameters for adding and updating a storage of any type against the options of that type alone. A third-party plugin then has to list the sensitive properties it uses in options(), as the API change log says. Signed-off-by: Thomas Lamprecht --- ApiChangeLog | 9 +- debian/control | 1 + src/PVE/Diskmanage.pm | 7 +- src/PVE/Makefile | 1 + src/PVE/NVMeOF.pm | 271 ++++++++++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NVMeOFPlugin.pm | 266 +++++++++++++++++ src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SAN.pm | 13 + src/test/Makefile | 5 + src/test/run_nvmeof_taint_tests.pl | 62 ++++ src/test/run_nvmeof_tests.pl | 446 +++++++++++++++++++++++++++++ src/test/run_san_nvme_tests.pl | 16 ++ 14 files changed, 1096 insertions(+), 6 deletions(-) create mode 100644 src/PVE/NVMeOF.pm create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm create mode 100755 src/test/run_nvmeof_taint_tests.pl create mode 100755 src/test/run_nvmeof_tests.pl diff --git a/ApiChangeLog b/ApiChangeLog index a51578a1..1816e18a 100644 --- a/ApiChangeLog +++ b/ApiChangeLog @@ -101,10 +101,11 @@ Future changes should be documented in here. declares a property of that name is then not loaded until it renames it, so prefer names that are specific to the plugin, for example prefixed with its type. - As soon as one such plugin is installed, the API checks the parameters for adding and updating a - storage against the options of its type alone, for all storage types. So every plugin has to - list the sensitive properties that it uses in `options()`. - Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of + Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for + adding and updating a storage against the options of its type alone, for all storage types, + also for plugins of older API versions. So every plugin has to list the sensitive properties + that it uses in `options()`. + Set 'sensitive-properties' in an isolating plugin, even if empty. Without it, the fallback list of version 11 also applies to the plugin's own properties of these names. * Pass the guest identity to backup providers diff --git a/debian/control b/debian/control index fdcf8117..b5c2f529 100644 --- a/debian/control +++ b/debian/control @@ -55,6 +55,7 @@ Depends: bzip2, ${misc:Depends}, ${perl:Depends}, Recommends: multipath-tools, + nvme-cli, pve-esxi-import-tools (>= 0.6.0), skopeo, zfsutils-linux (>= 2.4.1-pve2), diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm index be3fb8fc..21857dc6 100644 --- a/src/PVE/Diskmanage.pm +++ b/src/PVE/Diskmanage.pm @@ -107,7 +107,12 @@ sub disk_is_used { my $disklist = get_disks($dev, 1, 1); - die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev}); + my $entry = $disklist->{$dev}; + die "'$disk' is not a valid local disk\n" if !defined($entry); + # a partition has the transport of its disk + my ($parent) = ($entry->{parent} // '') =~ m!^/dev/(.+)$!; + my $transport = (defined($parent) ? $disklist->{$parent} : $entry)->{transport} // ''; + die "'$disk' is not a valid local disk\n" if $transport =~ /^nvme-(?:tcp|rdma|fc)$/; return 1 if $disklist->{$dev}->{used}; return 0; diff --git a/src/PVE/Makefile b/src/PVE/Makefile index 13fecbf8..238eefd4 100644 --- a/src/PVE/Makefile +++ b/src/PVE/Makefile @@ -4,6 +4,7 @@ install: install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm + install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm new file mode 100644 index 00000000..45b3acfc --- /dev/null +++ b/src/PVE/NVMeOF.pm @@ -0,0 +1,271 @@ +package PVE::NVMeOF; + +use v5.36; + +use Digest::SHA qw(sha1_hex); +use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC); +use File::Path qw(make_path); +use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton); +use Time::HiRes (); + +use PVE::JSONSchema; +use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command); + +use PVE::Diskmanage; + +our $NVME = '/usr/sbin/nvme'; +our $SYSFS = '/sys'; +our $HOSTNQN_FILE = '/etc/nvme/hostnqn'; +our $HOSTID_FILE = '/etc/nvme/hostid'; +our $RUN_DIR = '/run/pve/nvmeof'; +our $LOCK_WAIT = 10; +our $NQN_PATTERN = 'nqn\.[^\s,=]+'; +our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery'; + +sub canonical_address($addr) { + if (defined(my $packed = inet_pton(AF_INET, $addr))) { + return inet_ntop(AF_INET, $packed); + } + if (defined(my $packed = inet_pton(AF_INET6, $addr))) { + return inet_ntop(AF_INET, substr($packed, 12)) + if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff"; + return inet_ntop(AF_INET6, $packed); + } + return lc($addr); +} + +# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal. +sub parse_portal($portal) { + my ($addr, $port); + if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) { + ($addr, $port) = ($1, $2); + } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) { + ($addr, $port) = ($1, $2); + } else { + return undef; + } + return undef if defined($port) && ($port < 1 || $port > 65535); + $addr = canonical_address($addr); + if (defined(my $packed = inet_pton(AF_INET, $addr))) { + my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0; + return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast; + } elsif (defined($packed = inet_pton(AF_INET6, $addr))) { + my $prefix = unpack('n', $packed); + return undef + if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00; + } else { + return undef; + } + return { traddr => $addr, trsvcid => int($port // 4420) }; +} + +PVE::JSONSchema::register_format( + 'pve-storage-nvme-portal', + sub($portal, $noerr = undef) { + return $portal if parse_portal($portal); + return undef if $noerr; + die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n"; + }, +); + +sub format_path($path) { + # an IPv6 address, not the address of a Fibre Channel port, which has a colon too + my $ipv6 = $path->{traddr} =~ /:/ && $path->{traddr} =~ /^[0-9a-fA-F:.]+$/; + my $addr = $ipv6 ? "[$path->{traddr}]" : $path->{traddr}; + return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr; +} + +sub storage_paths($scfg) { + return [map { parse_portal($_) // die "invalid portal '$_'\n" } + PVE::Tools::split_list($scfg->{portals})]; +} + +sub host_identity() { + my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/; + my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~ + /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i; + if (!defined($nqn) || length($nqn) > 223) { + die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the" + . " host identity\n"; + } + die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n" + if !defined($id) || $id =~ /^[0-]+$/; + return { nqn => $nqn, id => lc($id) }; +} + +sub is_supported() { return -x $NVME ? 1 : 0 } + +sub assert_support() { + die "nvme-cli is not installed\n" if !is_supported(); + run_command(['modprobe', 'nvme-tcp'], timeout => 10) + if !-e "$SYSFS/module/nvme_tcp"; + my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // ''; + die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y'; +} + +sub controllers() { + my $res = {}; + my $dir = "$SYSFS/class/nvme-fabrics/ctl"; + dir_glob_foreach( + $dir, + 'nvme[0-9]+', + sub($name, @) { + my $address = file_read_firstline("$dir/$name/address") // return; + my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address); + return if !defined($address{traddr}); + $res->{$name} = { + transport => file_read_firstline("$dir/$name/transport") // '', + subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '', + state => file_read_firstline("$dir/$name/state") // 'unknown', + traddr => canonical_address($address{traddr}), + trsvcid => $address{trsvcid}, + }; + }, + ); + return $res; +} + +sub subsystem_controllers($subsystem) { + my $controllers = controllers(); + return { + map { $_ => $controllers->{$_} } grep { + $controllers->{$_}->{subsystem} eq $subsystem + && $controllers->{$_}->{transport} eq 'tcp' + } keys %$controllers + }; +} + +# The NQN of a subsystem, untainted, or undef if it is invalid or the one of the discovery service. +sub subsystem_nqn($nqn) { + my ($res) = ($nqn // '') =~ /\A($NQN_PATTERN)\z/; + return undef if !defined($res) || length($res) > 223 || $res eq $DISCOVERY_NQN; + return $res; +} + +sub namespace_wwid($dev) { + return file_read_firstline("$SYSFS/block/$dev/wwid"); +} + +sub has_live_controller($controllers) { + return scalar(grep { $_->{state} eq 'live' } values %$controllers); +} + +sub path_controller($controllers, $path) { + for my $name (sort keys %$controllers) { + my $ctrl = $controllers->{$name}; + if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) { + return $name; + } + } + return undef; +} + +sub namespaces($subsystem) { + my $res = {}; + my $dir = "$SYSFS/class/nvme-subsystem"; + dir_glob_foreach( + $dir, + 'nvme-subsys[0-9]+', + sub($subsys, @) { + my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return; + return if $nqn ne $subsystem; + dir_glob_foreach( + "$dir/$subsys", + 'nvme[0-9]+n[0-9]+', + sub($dev, @) { + my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return; + my $size = file_read_firstline("$SYSFS/block/$dev/size") // return; + $res->{$dev} = { wwid => $wwid, size => int($size) * 512 }; + }, + ); + }, + ); + return $res; +} + +sub with_lock($code) { + my ($res, $timeout) = PVE::Tools::run_fork_with_timeout( + $LOCK_WAIT + 35, + sub { + make_path($RUN_DIR); + open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n"; + # a busy lock fails a connect and arms the cooldown of its portal, so wait a while + my $deadline = Time::HiRes::time() + $LOCK_WAIT; + while (!flock($lock, LOCK_EX | LOCK_NB)) { + die "NVMe over Fabrics is busy - another operation is still running\n" + if Time::HiRes::time() >= $deadline; + Time::HiRes::usleep(100_000); + } + # Commands inherit the lock, so that it stays held until they exit, even if the caller + # timed out while one was stuck in the kernel. + my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n"; + fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC) + or die "cannot inherit NVMe lock - $!\n"; + return $code->(); + }, + ); + die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout; + return $res; +} + +my sub retry_file($scfg, $path) { + my ($key) = + sha1_hex(join( + "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~ + /\A([0-9a-f]{40})\z/; + return "$RUN_DIR/retry/$key"; +} + +sub in_cooldown($scfg, $path) { + my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0; + my $age = time() - $mtime; + return $age > -60 && $age < 60; +} + +sub note_connect_failure($scfg, $path) { + make_path("$RUN_DIR/retry"); + file_set_contents(retry_file($scfg, $path), ''); +} + +sub clear_connect_failures($scfg) { + unlink(retry_file($scfg, $_)) for storage_paths($scfg)->@*; +} + +sub connect_path($scfg, $path) { + my $nqn = subsystem_nqn($scfg->{subsysnqn}) // die "invalid subsystem NQN\n"; + my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/; + die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1; + return with_lock(sub { + return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path); + assert_support(); + my $host = host_identity(); + run_command( + [ + $NVME, + 'connect', + '--transport=tcp', + '--nqn', + $nqn, + '--traddr', + $path->{traddr}, + '--trsvcid', + $path->{trsvcid}, + '--hostnqn', + $host->{nqn}, + '--hostid', + $host->{id}, + '--ctrl-loss-tmo=-1', + '--fast_io_fail_tmo', + $fail, + ], + timeout => 30, + ); + # nvme-cli can exit successfully without having connected + die "no controller appeared\n" + if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path); + unlink(retry_file($scfg, $path)); + return; + }); +} + +1; diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index c640cd8e..c77a6755 100755 --- a/src/PVE/Storage.pm +++ b/src/PVE/Storage.pm @@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin; use PVE::Storage::NFSPlugin; use PVE::Storage::CIFSPlugin; use PVE::Storage::ISCSIPlugin; +use PVE::Storage::NVMeOFPlugin; use PVE::Storage::RBDPlugin; use PVE::Storage::CephFSPlugin; use PVE::Storage::ISCSIDirectPlugin; @@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register(); PVE::Storage::NFSPlugin->register(); PVE::Storage::CIFSPlugin->register(); PVE::Storage::ISCSIPlugin->register(); +PVE::Storage::NVMeOFPlugin->register(); PVE::Storage::RBDPlugin->register(); PVE::Storage::CephFSPlugin->register(); PVE::Storage::ISCSIDirectPlugin->register(); diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile index a2df83c0..7f2ff6c8 100644 --- a/src/PVE/Storage/Makefile +++ b/src/PVE/Storage/Makefile @@ -6,6 +6,7 @@ SOURCES= \ NFSPlugin.pm \ CIFSPlugin.pm \ ISCSIPlugin.pm \ + NVMeOFPlugin.pm \ CephFSPlugin.pm \ RBDPlugin.pm \ ISCSIDirectPlugin.pm \ diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm new file mode 100644 index 00000000..76f73af0 --- /dev/null +++ b/src/PVE/Storage/NVMeOFPlugin.pm @@ -0,0 +1,266 @@ +package PVE::Storage::NVMeOFPlugin; + +use v5.36; + +use Cwd (); +use Time::HiRes (); + +use PVE::Network; + +use PVE::Diskmanage; +use PVE::NVMeOF; +use PVE::Storage::Plugin; + +use base qw(PVE::Storage::Plugin); + +# Adding a storage activates it under the lock of the storage configuration, so once a controller +# is live, no further portal is tried after this many seconds. Later activations connect the rest. +our $ACTIVATION_BUDGET = 15; + +sub type { return 'nvmeof' } + +sub plugindata { + return { + content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }], + select_existing => 1, + 'isolate-properties' => 1, + 'advanced-properties' => { + 'fast-io-fail-tmo' => 1, + }, + 'hidden-properties' => { + bwlimit => 1, + }, + 'sensitive-properties' => {}, + }; +} + +sub properties { + return { + subsysnqn => { + title => "Subsystem", + description => 'The NQN of the NVMe subsystem.', + type => 'string', + pattern => $PVE::NVMeOF::NQN_PATTERN, + maxLength => 223, + }, + portals => { + title => "Portals", + description => + 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses' + . ' with an optional port (default 4420).', + type => 'string', + format => 'pve-storage-nvme-portal-list', + }, + 'fast-io-fail-tmo' => { + title => "Fail I/O After (s)", + description => 'Seconds before I/O through a reconnecting path fails. -1 disables' + . ' this timeout; I/O can remain queued while paths reconnect.', + type => 'integer', + minimum => -1, + default => 120, + }, + }; +} + +sub options { + return { + subsysnqn => { fixed => 1 }, + portals => {}, + 'fast-io-fail-tmo' => { optional => 1 }, + nodes => { optional => 1 }, + disable => { optional => 1 }, + content => { optional => 1 }, + bwlimit => { optional => 1 }, + }; +} + +my sub assert_valid_config($scfg) { + die "the discovery subsystem cannot be used as storage\n" + if $scfg->{subsysnqn} eq $PVE::NVMeOF::DISCOVERY_NQN; + my $paths = PVE::NVMeOF::storage_paths($scfg); + # the list format also accepts a list of separators only + die "no portal given\n" if !@$paths; + my %seen; + for my $path (@$paths) { + my $portal = PVE::NVMeOF::format_path($path); + die "portal '$portal' is listed more than once\n" if $seen{$portal}++; + } +} + +sub on_add_hook($class, $storeid, $scfg, %param) { + assert_valid_config($scfg); + # the LUN and controller lists and the node status attribute a subsystem to its one storage + my $ids = PVE::Storage::config()->{ids}; + for my $sid (sort keys %$ids) { + next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof'; + die "storage '$sid' already uses this subsystem\n" + if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn}; + } + # adding the storage again, for example once the array allows the node, tries all portals + PVE::NVMeOF::clear_connect_failures($scfg); + return; +} + +sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) { + my $merged = { %$scfg, %$update }; + delete $merged->{$_} for ($delete // [])->@*; + assert_valid_config($merged); + return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive); +} + +sub parse_volname($class, $volname) { + if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) { + return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw'); + } + die "unable to parse NVMe over Fabrics volume name '$volname'\n"; +} + +# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a +# namespace of another subsystem. +sub assert_namespace($scfg, $volname, $real) { + my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!; + die "volume '$volname' is not an NVMe namespace\n" if !defined($dev); + my $nqn = PVE::Diskmanage::read_subsysnqn("$PVE::NVMeOF::SYSFS/block/$dev/device/subsysnqn") + // ''; + die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n" + if $nqn ne $scfg->{subsysnqn}; + # another by-id name of the subsystem, like model and serial, can name another namespace + my $wwid = PVE::NVMeOF::namespace_wwid($dev) // ''; + die "volume '$volname' is not named after the WWID of its namespace\n" + if $volname ne "nvme-$wwid"; +} + +sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) { + die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname); + my ($vtype, $name, $vmid) = $class->parse_volname($volname); + my $path = "/dev/disk/by-id/$name"; + # not every user of a path activates the volume first, and the base volume of an LVM storage + # has no device before its storage connects + my $real = Cwd::realpath($path); + assert_namespace($scfg, $volname, $real) if defined($real) && -e $real; + return wantarray ? ($path, $vmid, $vtype) : $path; +} + +sub create_base { die "can't create base images in NVMe over Fabrics storage\n" } +sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" } +sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" } +sub free_image { die "can't free space in NVMe over Fabrics storage\n" } +sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" } + +sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) { + my $res = $class->list_images($storeid, $scfg, $vmid); + $_->{content} = 'images' for @$res; + return $res; +} + +sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) { + return [] if defined($vmid) && !$vollist; + my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn}); + my $res = []; + for my $dev (sort keys %$namespaces) { + my $ns = $namespaces->{$dev}; + my $volid = "$storeid:nvme-$ns->{wwid}"; + next if $vollist && !grep { $_ eq $volid } @$vollist; + push @$res, + { + volid => $volid, + format => 'raw', + size => $ns->{size}, + vmid => 0, + }; + } + return $res; +} + +sub status($class, $storeid, $scfg, $cache = undef, @) { + my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + return (0, 0, 0, PVE::NVMeOF::has_live_controller($controllers) ? 1 : 0); +} + +sub activate_storage($class, $storeid, $scfg, $cache = undef, @) { + my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + my @missing = grep { !defined(PVE::NVMeOF::path_controller($controllers, $_)) } + PVE::NVMeOF::storage_paths($scfg)->@*; + return if !@missing; + # Without these no path connects, which is no reason to wait before the next attempt. A storage + # with a usable path still works, with fewer paths. + eval { + PVE::NVMeOF::assert_support(); + PVE::NVMeOF::host_identity(); + }; + if (my $err = $@) { + die $err if !PVE::NVMeOF::has_live_controller($controllers); + warn "storage '$storeid': cannot connect to the other portals - $err"; + return; + } + my @errors; + my $live = PVE::NVMeOF::has_live_controller($controllers); + my $deadline = Time::HiRes::time() + $ACTIVATION_BUDGET; + for my $path (@missing) { + my $portal = PVE::NVMeOF::format_path($path); + if ($live && Time::HiRes::time() >= $deadline) { + push @errors, "$portal: left for the next activation"; + next; + } + if (PVE::NVMeOF::in_cooldown($scfg, $path)) { + push @errors, "$portal: failed less than a minute ago"; + next; + } + eval { + die "no answer from portal\n" + if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2); + PVE::NVMeOF::connect_path($scfg, $path); + }; + if (my $err = $@) { + PVE::NVMeOF::note_connect_failure($scfg, $path); + warn "storage '$storeid': connecting to '$portal' failed - $err"; + chomp $err; + push @errors, "$portal: $err"; + } else { + $live = 1; + } + } + # some paths are enough, but without a usable one the namespaces cannot be used + $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + die "cannot connect to any portal - " . join('; ', @errors) . "\n" + if !PVE::NVMeOF::has_live_controller($controllers); + return; +} + +sub deactivate_storage { return } + +sub check_connection($class, $storeid, $scfg, @) { + my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn}); + return 1 if PVE::NVMeOF::has_live_controller($controllers); + for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) { + next if defined(PVE::NVMeOF::path_controller($controllers, $path)); + next if PVE::NVMeOF::in_cooldown($scfg, $path); + return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2); + PVE::NVMeOF::note_connect_failure($scfg, $path); + } + return 0; +} + +sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) { + my $path = $class->filesystem_path($scfg, $volname, $snapname); + # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device + my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10); + while (!-b $path && Time::HiRes::time() < $deadline) { + last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*; + Time::HiRes::usleep(250_000); + } + my $real = Cwd::realpath($path); + die "volume '$volname' is not available on this node\n" if !$real || !-b $real; + assert_namespace($scfg, $volname, $real); +} + +sub volume_has_feature( + $class, $scfg, $feature, $storeid, $volname, + $snapname = undef, + $running = undef, @, +) { + return 1 if $feature eq 'copy' && !defined($snapname); + return undef; +} + +1; diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index a97f361c..d61e78b6 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw( ); our @SHARED_STORAGE = ( - 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', + 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof', ); our $QCOW2_PREALLOCATION = { diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm index 0a9e441c..53c24c9e 100644 --- a/src/PVE/Storage/SAN.pm +++ b/src/PVE/Storage/SAN.pm @@ -302,6 +302,11 @@ sub lun_partitions($lun) { } sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) { + my $name = basename($lun->{devpath}); + my $nqn = + $name =~ /^nvme\d+n\d+$/ + ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn") + : undef; my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups( [$lun->{devpath}, lun_partitions($lun)->@*], )->@*; @@ -324,6 +329,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) { ) { die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n"; } + if ( + $type eq 'nvmeof' + && defined($nqn) + && $scfg->{subsysnqn} eq $nqn + && PVE::Storage::san_storage_serves_images($scfg) + ) { + die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n"; + } next if $type ne 'lvm' || !$scfg->{base}; my $base = PVE::Multipath::parse_base_volume($scfg->{base}); my $matches = $base diff --git a/src/test/Makefile b/src/test/Makefile index 3b902c43..ceabd991 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -17,6 +17,7 @@ test: \ test_lvmplugin \ test_multipath \ test_multipath_api \ + test_nvmeof \ test_ovf \ test_plugin \ test_remote_command \ @@ -115,3 +116,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl test_disk_api: run_disk_api_tests.pl ./run_disk_api_tests.pl + +test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl + ./run_nvmeof_tests.pl + perl -T ./run_nvmeof_taint_tests.pl diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl new file mode 100755 index 00000000..6fd37410 --- /dev/null +++ b/src/test/run_nvmeof_taint_tests.pl @@ -0,0 +1,62 @@ +#!/usr/bin/perl -T + +use v5.36; +use lib qw(..); +use File::Temp qw(tempdir); +use Scalar::Util qw(tainted); +use Test::MockModule; +use Test::More; + +use PVE::Tools qw(file_get_contents file_set_contents); + +use PVE::NVMeOF; + +$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin'; +delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; +my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1); +local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn"; +local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid"; +local $PVE::NVMeOF::RUN_DIR = "$dir/run"; +file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n"); +file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n"); +file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n"); +my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config")); +ok(tainted($nqn), 'configuration input is tainted'); +my $nvme = Test::MockModule->new('PVE::NVMeOF'); +$nvme->mock(with_lock => sub { $_[0]->() }); +my $commands = 0; +$nvme->mock( + subsystem_controllers => sub { + return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {}; + }, +); +$nvme->mock(assert_support => sub { }); +$nvme->mock( + run_command => sub($cmd, %opts) { + $commands++; + is( + scalar(grep { tainted($_) } @$cmd), + 0, + 'all connect operands are validated and untainted', + ); + PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5); + }, +); +my $path = PVE::NVMeOF::parse_portal($portal); +eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) }; +is($@, '', 'connect command can execute in taint mode'); +is($commands, 1, 'one connect'); +PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path); +ok( + PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path), + 'cooldown accepts validated file-backed configuration', +); +for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') { + eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) }; + like($@, qr/invalid subsystem/, 'invalid connect subsystem refused'); +} +file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n"); +eval { PVE::NVMeOF::host_identity() }; +like($@, qr/nonzero UUID/, 'zero host ID refused'); + +done_testing(); diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl new file mode 100755 index 00000000..cdf228c0 --- /dev/null +++ b/src/test/run_nvmeof_tests.pl @@ -0,0 +1,446 @@ +#!/usr/bin/perl + +use v5.36; +use lib qw(..); +use Fcntl qw(:flock); +use File::Path qw(make_path remove_tree); +use File::Temp qw(tempdir); +use POSIX (); +use Test::MockModule; +use Test::More; +use Time::HiRes (); + +use PVE::Tools qw(file_set_contents); + +use PVE::NVMeOF; +use PVE::Storage; + +my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1); +local $PVE::NVMeOF::RUN_DIR = "$dir/run"; +local $PVE::NVMeOF::SYSFS = "$dir/sys"; +my $class = 'PVE::Storage::NVMeOFPlugin'; +my $cfg = { + type => 'nvmeof', + subsysnqn => 'nqn.test', + portals => '192.0.2.1,[2001:db8::2]:4421', + content => { none => 1 }, +}; + +subtest 'configuration, portals and volumes' => sub { + is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered'); + is_deeply( + [sort keys $class->properties()->%*], + [qw(fast-io-fail-tmo portals subsysnqn)], + 'only three plugin-specific properties', + ); + my sub nvmeof_schema($schema) { + my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*; + my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types; + return $nvmeof->{properties} // {}; + } + my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema()); + is_deeply( + [sort grep { !$create->{$_}->{optional} } keys $create->%*], + [qw(portals subsysnqn)], + 'adding a storage requires the subsystem and the portals', + ); + my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema()); + ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed'); + my $parsed = PVE::Storage::Plugin->parse_config( + 'storage.cfg', + "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n", + ); + is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin'); + is_deeply( + $parsed->{ids}->{fabric}->{content}, + { none => 1 }, + 'connection-only content parses', + ); + is_deeply( + PVE::NVMeOF::parse_portal('192.0.2.1'), + { traddr => '192.0.2.1', trsvcid => 4420 }, + 'default I/O port', + ); + is_deeply( + PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'), + { traddr => '2001:db8::1', trsvcid => 4421 }, + 'canonical IPv6', + ); + is( + PVE::NVMeOF::format_path({ traddr => '2001:db8::1', trsvcid => 4421 }), + '[2001:db8::1]:4421', + 'an IPv6 portal in brackets', + ); + is( + PVE::NVMeOF::format_path({ traddr => 'nn-0x2000:pn-0x2001' }), + 'nn-0x2000:pn-0x2001', + 'the address of a Fibre Channel port as it is', + ); + + for my $bad ( + 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1', + ) { + ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad"); + } + eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); }; + like($@, qr/listed more than once/, 'duplicate portals refused'); + eval { $class->on_add_hook('fabric', { %$cfg, portals => ',,' }); }; + like($@, qr/no portal given/, 'a list without a portal refused'); + eval { + $class->on_add_hook( + 'fabric', + { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' }, + ); + }; + like($@, qr/discovery subsystem/, 'discovery subsystem refused'); + my $storage = Test::MockModule->new('PVE::Storage'); + my $ids = { fabric => $cfg, local => { type => 'dir' } }; + $storage->mock(config => sub { { ids => $ids } }); + my $first = PVE::NVMeOF::storage_paths($cfg)->[0]; + PVE::NVMeOF::note_connect_failure($cfg, $first); + is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way'); + ok(!PVE::NVMeOF::in_cooldown($cfg, $first), 'adding it tries all portals again'); + eval { $class->on_add_hook('second', {%$cfg}) }; + like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem'); + is( + scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')), + '/dev/disk/by-id/nvme-eui.0123456789abcdef', + 'stable volume path', + ); + { + my $disk = Test::MockModule->new('PVE::Diskmanage'); + my $nqn = 'nqn.other'; + $disk->redefine(read_subsysnqn => sub { $nqn }); + my $sysfs = Test::MockModule->new('PVE::NVMeOF'); + $sysfs->redefine(namespace_wwid => sub { 'x' }); + my $check = sub { + eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) }; + $@; + }; + like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem'); + like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type'); + like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition'); + $nqn = 'nqn.test'; + is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem'); + like( + $check->('nvme-Vendor_Model_123_1', '/dev/nvme0n1'), + qr/not named after the WWID/, + 'another by-id name of a namespace of the subsystem', + ); + my $cwd = Test::MockModule->new('Cwd'); + $cwd->redefine(realpath => sub { $dir }); + eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') }; + like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked'); + } + + for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') { + eval { $class->parse_volname($bad) }; + like($@, qr/unable to parse/, "invalid volume $bad"); + } + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } }); + is( + $class->list_volumes('fabric', $cfg)->[0]->{volid}, + 'fabric:nvme-eui.abc', + 'content none still offers bases', + ); + is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner'); + $nvme->mock(subsystem_controllers => sub { {} }); + eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') }; + like($@, qr/not available/, + 'missing namespace fails without waiting on absent controllers'); + my ($now, $queries) = (0, 0); + my $clock = Test::MockModule->new('Time::HiRes'); + $clock->redefine(time => sub { $now += 5 }, usleep => sub { }); + $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } }); + eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) }; + is($queries, 1, 'a missing namespace of a connected subsystem is waited for'); + eval { + $class->activate_volume( + 'fabric', + $cfg, + 'nvme-nonexistent-test', + undef, + { status_only => 1 }, + ); + }; + like($@, qr/not available/, 'in a status pass it fails as well'); + is($queries, 1, 'but without a wait'); + is( + ($class->parse_volname('nvme-eui.0123456789abcdef'))[0], + 'vm-vol', + 'namespaces are VM disks', + ); +}; + +subtest 'sysfs topology' => sub { + my $sys = $PVE::NVMeOF::SYSFS; + make_path( + "$sys/class/nvme-fabrics/ctl/nvme1", + "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1", + "$sys/block/nvme0n1", + ); + my %attrs = ( + address => 'traddr=192.0.2.1,trsvcid=4420', + transport => 'tcp', + subsysnqn => 'nqn.test', + state => 'connecting', + ); + file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs; + file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n"); + file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n"); + file_set_contents("$sys/block/nvme0n1/size", "8\n"); + is( + PVE::NVMeOF::controllers()->{nvme1}->{state}, + 'connecting', + 'reconnecting controller is still present', + ); + is( + PVE::NVMeOF::path_controller( + PVE::NVMeOF::subsystem_controllers('nqn.test'), + PVE::NVMeOF::parse_portal('192.0.2.1'), + ), + 'nvme1', + 'portal controller found', + ); + is_deeply( + PVE::NVMeOF::namespaces('nqn.test'), + { nvme0n1 => { wwid => 'eui.abc', size => 4096 } }, + 'namespace facts', + ); + is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded'); +}; + +subtest 'serialized commands inherit the lock' => sub { + local $PVE::NVMeOF::LOCK_WAIT = 0; + my $script = join( + ' ', + 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*");', + 'die "not inherited" if !@held;', + 'open(my $fh, ">>", $ARGV[0]) or die $!;', + 'die "lock missing" if flock($fh, LOCK_EX|LOCK_NB);', + 'print "locked\\n";', + ); + my $value = PVE::NVMeOF::with_lock(sub { + my $out = ''; + PVE::Tools::run_command( + [ + '/usr/bin/perl', + '-MFcntl=:flock', + '-e', + $script, + "$PVE::NVMeOF::RUN_DIR/fabrics.lock", + ], + timeout => 5, + outfunc => sub { $out .= $_[0] }, + ); + return $out; + }); + is($value, 'locked', 'exec cannot acquire the lock held by its parent'); + open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!; + flock($lock, LOCK_EX) or die $!; + eval { + PVE::NVMeOF::with_lock(sub { die "must not count paths\n" }); + }; + like($@, qr/is busy/, 'busy operation never enters callback'); + close($lock); + + my $tools = Test::MockModule->new('PVE::Tools'); + my $timed_out = 1; + $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) }); + my $retained; + eval { + PVE::NVMeOF::with_lock(sub { + my ($path) = + grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*'); + my ($fd) = $path =~ m{/(\d+)$}; + open($retained, '>&', $fd) or die $!; + return 1; + }); + }; + like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed'); + $timed_out = 0; + eval { + PVE::NVMeOF::with_lock(sub { die "must not count paths\n" }); + }; + like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns'); + close($retained); + is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization'); +}; + +subtest 'a short hold of the lock does not fail a connect' => sub { + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + my $net = Test::MockModule->new('PVE::Network'); + my $scfg = { %$cfg, subsysnqn => 'nqn.wait', portals => '192.0.2.3' }; + my $ctl = "$PVE::NVMeOF::SYSFS/class/nvme-fabrics/ctl/nvme9"; + $net->mock(tcp_ping => sub { 1 }); + $nvme->mock(assert_support => sub { }); + $nvme->mock(host_identity => + sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } }); + $nvme->mock( + run_command => sub(@) { + my %attrs = ( + address => 'traddr=192.0.2.3,trsvcid=4420', + transport => 'tcp', + subsysnqn => 'nqn.wait', + state => 'live', + ); + make_path($ctl); + file_set_contents("$ctl/$_", "$attrs{$_}\n") for keys %attrs; + }, + ); + pipe(my $held, my $signal) or die "pipe failed - $!\n"; + my $holder = fork() // die "fork failed - $!\n"; + if (!$holder) { + open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or POSIX::_exit(1); + flock($lock, LOCK_EX) or POSIX::_exit(1); + syswrite($signal, '1'); + Time::HiRes::sleep(0.5); + POSIX::_exit(0); + } + close($signal); + sysread($held, my $state, 1); + is($state, '1', 'another process holds the lock'); + my $warning = ''; + { + local $SIG{__WARN__} = sub { $warning .= $_[0] }; + $class->activate_storage('wait', $scfg); + } + waitpid($holder, 0); + is($warning, '', 'the connect waits for the lock'); + my $path = PVE::NVMeOF::storage_paths($scfg)->[0]; + ok(!PVE::NVMeOF::in_cooldown($scfg, $path), 'and arms no cooldown'); + is( + PVE::NVMeOF::path_controller(PVE::NVMeOF::subsystem_controllers('nqn.wait'), $path), + 'nvme9', + 'the portal is connected', + ); + remove_tree($ctl); +}; + +subtest 'connect options, kernel reconnect and cooldown' => sub { + my $nvme = Test::MockModule->new('PVE::NVMeOF'); + my $net = Test::MockModule->new('PVE::Network'); + my @commands; + my $controllers = {}; + $nvme->mock(with_lock => sub { $_[0]->() }); + $nvme->mock(assert_support => sub { }); + $nvme->mock(host_identity => + sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } }); + $nvme->mock(subsystem_controllers => sub { $controllers }); + my $appear = 1; + $nvme->mock( + run_command => sub($cmd, %opts) { + is($opts{timeout}, 30, 'bounded connect'); + push @commands, $cmd; + my %arg = + map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd; + my $controller = + { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' }; + $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear; + }, + ); + $net->mock(tcp_ping => sub { 1 }); + $class->activate_storage('fabric', $cfg); + is(scalar(@commands), 2, 'one connect per missing portal'); + ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely'); + is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default'); + { + local $PVE::Storage::NVMeOFPlugin::ACTIVATION_BUDGET = 0; + @commands = (); + $controllers = {}; + $class->activate_storage('fabric', $cfg); + is(scalar(@commands), 1, 'with a live controller, a spent time budget leaves the rest'); + ($controllers, $appear) = ({}, 0); + @commands = (); + { + local $SIG{__WARN__} = sub { }; + eval { $class->activate_storage('fabric', $cfg) }; + } + is(scalar(@commands), 2, 'without one, every portal is tried'); + unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*")); + $appear = 1; + } + @commands = (); + $controllers = { + nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' }, + nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' }, + }; + $class->activate_storage('fabric', $cfg); + is_deeply(\@commands, [], + 'existing controllers are never reconnected, regardless of state'); + $class->deactivate_storage('fabric', $cfg); + is_deeply(\@commands, [], 'deactivation does not remove controllers'); + $controllers = {}; + my $path = PVE::NVMeOF::storage_paths($cfg)->[0]; + PVE::NVMeOF::note_connect_failure($cfg, $path); + ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown'); + $class->activate_storage('fabric', $cfg); + is(scalar(@commands), 1, 'only the other portal connects'); + ($controllers, $appear) = ({}, 0); + my $other = PVE::NVMeOF::storage_paths($cfg)->[1]; + my $warning = ''; + { + local $SIG{__WARN__} = sub { $warning .= $_[0] }; + eval { $class->activate_storage('fabric', $cfg) }; + } + like($warning, qr/no controller appeared/, 'a connect without a controller is a failure'); + ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too'); + like( + $@, + qr/^cannot connect to any portal - .*failed less than a minute ago; .*no controller appeared/, + 'without any controller the activation fails with the reason of each portal', + ); + unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*")); + $nvme->mock(assert_support => sub { die "nvme-cli is not installed\n" }); + @commands = (); + eval { $class->activate_storage('fabric', $cfg) }; + is($@, "nvme-cli is not installed\n", 'a missing prerequisite fails the activation'); + is_deeply(\@commands, [], 'before any connect'); + ok(!PVE::NVMeOF::in_cooldown($cfg, $path), 'and without a cooldown'); + $controllers = { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'live' } }; + $warning = ''; + { + local $SIG{__WARN__} = sub { $warning .= $_[0] }; + is( + eval { $class->activate_storage('fabric', $cfg); 1 }, + 1, + 'but not with a usable path', + ); + } + like($warning, qr/cannot connect to the other portals - nvme-cli/, 'which it warns about'); + $nvme->mock(assert_support => sub { }); + $controllers = + { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' } }; + { + local $SIG{__WARN__} = sub { }; + eval { $class->activate_storage('fabric', $cfg) }; + } + like( + $@, + qr/^cannot connect to any portal/, + 'a reconnecting path is not usable when the missing one fails', + ); +}; + +subtest 'local disk setup rejects fabric namespaces by transport' => sub { + my $disk = Test::MockModule->new('PVE::Diskmanage'); + for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) { + $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } }); + eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') }; + like($@, qr/not a valid local disk/, + "$transport refuses without storage configuration"); + $disk->mock( + get_disks => sub { + { + nvme0n1 => { transport => $transport }, + nvme0n1p1 => { parent => '/dev/nvme0n1', type => 'partition' }, + }; + }, + ); + eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1p1') }; + like($@, qr/not a valid local disk/, "$transport refuses a partition of a namespace"); + } +}; + +done_testing(); diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl index 3245b774..2a9a2121 100755 --- a/src/test/run_san_nvme_tests.pl +++ b/src/test/run_san_nvme_tests.pl @@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg }); $disk->mock(verify_blockdev_path => sub { $device }); $disk->mock(is_partition => sub { 0 }); $disk->mock(get_nvme_transport => sub { 'fc' }); +$disk->mock(read_subsysnqn => sub { 'nqn.test' }); $disk->mock(is_iscsi_device => sub { 0 }); $disk->mock(is_mounted => sub { 0 }); $disk->mock(has_holder => sub { 0 }); @@ -86,6 +87,21 @@ for my $base (qw(fabric nvme-fabric)) { eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; like($@, qr/base volume/, "native base reference on storage '$base' refuses"); } +$cfg->{ids} = + { fabric => + { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +like( + $@, + qr/serves images from this subsystem/, + 'direct namespace image use refuses even when disabled', +); +$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks'); +$cfg->{ids}->{fabric}->{content} = { none => 1 }; +eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) }; +is($@, '', 'connection-only storage permits namespace initialization'); $cfg->{ids} = {}; @order = @opens = (); $tools->mock(file_get_contents => sub { 1024 });