From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems
Date: Wed, 18 Nov 2026 21:42:21 +0100 [thread overview]
Message-ID: <20261118204302.3100813-3-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com>
NVMe/TCP namespaces need a manual nvme-cli setup on every node so
far. Add a storage type that connects one subsystem through each
configured portal and offers its namespaces as raw volumes. Native
NVMe multipath joins the paths and the kernel reconnects lost ones.
A second storage for a subsystem is refused: the LUN list, the
controller list and the node status attribute a subsystem to one
storage. Resolving the path of a volume checks an existing device
against the subsystem: a volume name can be the by-id name of any NVMe
device, and not every user of a path activates the volume first.
The plugin isolates its properties, so names like 'portals' belong to
this storage type alone, and a third-party plugin that declares one,
like 'subsysnqn', is not left out. As the first in-tree plugin that
isolates, it makes the storage API check the parameters for adding and
updating a storage of any type against the options of that type alone.
A third-party plugin then has to list the sensitive properties it uses
in options(), as the API change log says.
Local disk setup refuses namespaces attached over a fabric, as for
iSCSI LUNs. Wiping a namespace or creating a volume group on it is
refused while a storage serves images from its subsystem. nvme-cli
is only recommended, as no other storage type needs it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
ApiChangeLog | 7 +-
debian/control | 1 +
src/PVE/Diskmanage.pm | 2 +
src/PVE/Makefile | 1 +
src/PVE/NVMeOF.pm | 244 +++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 223 +++++++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 13 ++
src/test/Makefile | 6 +-
src/test/run_nvmeof_taint_tests.pl | 60 ++++++
src/test/run_nvmeof_tests.pl | 302 +++++++++++++++++++++++++++++
src/test/run_san_nvme_tests.pl | 16 ++
14 files changed, 875 insertions(+), 5 deletions(-)
create mode 100644 src/PVE/NVMeOF.pm
create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
create mode 100755 src/test/run_nvmeof_taint_tests.pl
create mode 100755 src/test/run_nvmeof_tests.pl
diff --git a/ApiChangeLog b/ApiChangeLog
index 10f39508..d7310fe4 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -101,9 +101,10 @@ Future changes should be documented in here.
declares a property of that name is then not loaded until it renames it, so prefer names that
are specific to the plugin, for example prefixed with its type.
- As soon as one such plugin is installed, the API checks the parameters for adding and updating a
- storage against the options of its type alone, for all storage types. So every plugin has to
- list the sensitive properties that it uses in `options()`.
+ Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for
+ adding and updating a storage against the options of its type alone, for all storage types,
+ also for plugins of older API versions. So every plugin has to list the sensitive properties
+ that it uses in `options()`.
Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of
version 11 also applies to the plugin's own properties of these names.
diff --git a/debian/control b/debian/control
index d10dfb23..8b9b6455 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
${misc:Depends},
${perl:Depends},
Recommends: multipath-tools,
+ nvme-cli,
pve-esxi-import-tools (>= 0.6.0),
skopeo,
zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 8fda7afc..1ad91e06 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -108,6 +108,8 @@ sub disk_is_used {
my $disklist = get_disks($dev, 1, 1);
die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+ die "'$disk' is not a valid local disk\n"
+ if ($disklist->{$dev}->{transport} // '') =~ /^nvme-(?:tcp|rdma|fc)$/;
return 1 if $disklist->{$dev}->{used};
return 0;
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
install:
install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+ install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..ebaedf9b
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,244 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+use PVE::Diskmanage;
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+
+sub canonical_address($addr) {
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ return inet_ntop(AF_INET, $packed);
+ }
+ if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+ return inet_ntop(AF_INET, substr($packed, 12))
+ if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+ return inet_ntop(AF_INET6, $packed);
+ }
+ return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+ my ($addr, $port);
+ if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) {
+ ($addr, $port) = ($1, $2);
+ } else {
+ return undef;
+ }
+ return undef if defined($port) && ($port < 1 || $port > 65535);
+ $addr = canonical_address($addr);
+ if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+ my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0;
+ return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast;
+ } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+ my $prefix = unpack('n', $packed);
+ return undef
+ if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+ } else {
+ return undef;
+ }
+ return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+ 'pve-storage-nvme-portal',
+ sub($portal, $noerr = undef) {
+ return $portal if parse_portal($portal);
+ return undef if $noerr;
+ die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+ },
+);
+
+sub format_path($path) {
+ my $addr = $path->{traddr} =~ /:/ ? "[$path->{traddr}]" : $path->{traddr};
+ return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+ return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+ PVE::Tools::split_list($scfg->{portals})];
+}
+
+sub host_identity() {
+ my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+ my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+ /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+ if (!defined($nqn) || length($nqn) > 223) {
+ die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the"
+ . " host identity\n";
+ }
+ die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+ if !defined($id) || $id =~ /^[0-]+$/;
+ return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+ die "nvme-cli is not installed\n" if !is_supported();
+ run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+ if !-e "$SYSFS/module/nvme_tcp";
+ my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+ die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+ dir_glob_foreach(
+ $dir,
+ 'nvme[0-9]+',
+ sub($name, @) {
+ my $address = file_read_firstline("$dir/$name/address") // return;
+ my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+ return if !defined($address{traddr});
+ $res->{$name} = {
+ transport => file_read_firstline("$dir/$name/transport") // '',
+ subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+ state => file_read_firstline("$dir/$name/state") // 'unknown',
+ traddr => canonical_address($address{traddr}),
+ trsvcid => $address{trsvcid},
+ };
+ },
+ );
+ return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+ my $controllers = controllers();
+ return {
+ map { $_ => $controllers->{$_} } grep {
+ $controllers->{$_}->{subsystem} eq $subsystem
+ && $controllers->{$_}->{transport} eq 'tcp'
+ } keys %$controllers
+ };
+}
+
+sub path_controller($controllers, $path) {
+ for my $name (sort keys %$controllers) {
+ my $ctrl = $controllers->{$name};
+ if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) {
+ return $name;
+ }
+ }
+ return undef;
+}
+
+sub namespaces($subsystem) {
+ my $res = {};
+ my $dir = "$SYSFS/class/nvme-subsystem";
+ dir_glob_foreach(
+ $dir,
+ 'nvme-subsys[0-9]+',
+ sub($subsys, @) {
+ my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+ return if $nqn ne $subsystem;
+ dir_glob_foreach(
+ "$dir/$subsys",
+ 'nvme[0-9]+n[0-9]+',
+ sub($dev, @) {
+ my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+ my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+ $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+ },
+ );
+ },
+ );
+ return $res;
+}
+
+sub with_lock($code) {
+ my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+ 35,
+ sub {
+ make_path($RUN_DIR);
+ open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+ flock($lock, LOCK_EX | LOCK_NB)
+ or die "NVMe over Fabrics is busy - another operation is still running\n";
+ # A command still in the kernel must retain serialization after its caller times out.
+ my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+ fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+ or die "cannot inherit NVMe lock - $!\n";
+ return $code->();
+ },
+ );
+ die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+ return $res;
+}
+
+my sub retry_file($scfg, $path) {
+ my ($key) =
+ sha1_hex(join(
+ "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~
+ /\A([0-9a-f]{40})\z/;
+ return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+ my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+ my $age = time() - $mtime;
+ return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+ make_path("$RUN_DIR/retry");
+ file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub connect_path($scfg, $path) {
+ my ($nqn) = $scfg->{subsysnqn} =~ /\A($NQN_PATTERN)\z/;
+ if (!defined($nqn) || length($nqn) > 223 || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery') {
+ die "invalid subsystem NQN\n";
+ }
+ my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/;
+ die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+ return with_lock(sub {
+ return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ assert_support();
+ my $host = host_identity();
+ run_command(
+ [
+ $NVME,
+ 'connect',
+ '--transport=tcp',
+ '--nqn',
+ $nqn,
+ '--traddr',
+ $path->{traddr},
+ '--trsvcid',
+ $path->{trsvcid},
+ '--hostnqn',
+ $host->{nqn},
+ '--hostid',
+ $host->{id},
+ '--ctrl-loss-tmo=-1',
+ '--fast_io_fail_tmo',
+ $fail,
+ ],
+ timeout => 30,
+ );
+ # nvme-cli can exit successfully without having connected
+ die "no controller appeared\n"
+ if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+ unlink(retry_file($scfg, $path));
+ return;
+ });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 1e9c670a..91283ff2 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin;
use PVE::Storage::NFSPlugin;
use PVE::Storage::CIFSPlugin;
use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
use PVE::Storage::RBDPlugin;
use PVE::Storage::CephFSPlugin;
use PVE::Storage::ISCSIDirectPlugin;
@@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register();
PVE::Storage::NFSPlugin->register();
PVE::Storage::CIFSPlugin->register();
PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
PVE::Storage::RBDPlugin->register();
PVE::Storage::CephFSPlugin->register();
PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
NFSPlugin.pm \
CIFSPlugin.pm \
ISCSIPlugin.pm \
+ NVMeOFPlugin.pm \
CephFSPlugin.pm \
RBDPlugin.pm \
ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..cb514c1d
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,223 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use v5.36;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Network;
+
+use PVE::Diskmanage;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+ return {
+ content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }],
+ select_existing => 1,
+ 'isolate-properties' => 1,
+ 'advanced-properties' => {
+ 'fast-io-fail-tmo' => 1,
+ },
+ 'hidden-properties' => {
+ bwlimit => 1,
+ },
+ 'sensitive-properties' => {},
+ };
+}
+
+sub properties {
+ return {
+ subsysnqn => {
+ title => "Subsystem",
+ description => 'The NQN of the NVMe subsystem.',
+ type => 'string',
+ pattern => $PVE::NVMeOF::NQN_PATTERN,
+ maxLength => 223,
+ },
+ portals => {
+ title => "Portals",
+ description =>
+ 'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+ . ' with an optional port (default 4420).',
+ type => 'string',
+ format => 'pve-storage-nvme-portal-list',
+ },
+ 'fast-io-fail-tmo' => {
+ title => "Fail I/O After (s)",
+ description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+ . ' this timeout; I/O can remain queued while paths reconnect.',
+ type => 'integer',
+ minimum => -1,
+ default => 120,
+ },
+ };
+}
+
+sub options {
+ return {
+ subsysnqn => { fixed => 1 },
+ portals => {},
+ 'fast-io-fail-tmo' => { optional => 1 },
+ nodes => { optional => 1 },
+ disable => { optional => 1 },
+ content => { optional => 1 },
+ bwlimit => { optional => 1 },
+ };
+}
+
+my sub assert_valid_config($scfg) {
+ die "the discovery subsystem cannot be used as storage\n"
+ if $scfg->{subsysnqn} eq 'nqn.2014-08.org.nvmexpress.discovery';
+ my %seen;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ my $portal = PVE::NVMeOF::format_path($path);
+ die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+ }
+}
+
+sub on_add_hook($class, $storeid, $scfg, %param) {
+ assert_valid_config($scfg);
+ # the LUN and controller lists and the node status attribute a subsystem to its one storage
+ my $ids = PVE::Storage::config()->{ids};
+ for my $sid (sort keys %$ids) {
+ next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof';
+ die "storage '$sid' already uses this subsystem\n"
+ if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn};
+ }
+ return;
+}
+
+sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) {
+ my $merged = { %$scfg, %$update };
+ delete $merged->{$_} for ($delete // [])->@*;
+ assert_valid_config($merged);
+ return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname($class, $volname) {
+ if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) {
+ return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw');
+ }
+ die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a
+# namespace of another subsystem.
+sub assert_namespace($scfg, $volname, $real) {
+ my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!;
+ die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+ my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$dev/device/subsysnqn") // '';
+ die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+ if $nqn ne $scfg->{subsysnqn};
+}
+
+sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) {
+ die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+ my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+ my $path = "/dev/disk/by-id/$name";
+ # not every user of a path activates the volume first, and the base volume of an LVM storage
+ # has no device before its storage connects
+ my $real = Cwd::realpath($path);
+ assert_namespace($scfg, $volname, $real) if defined($real) && -e $real;
+ return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) {
+ my $res = $class->list_images($storeid, $scfg, $vmid);
+ $_->{content} = 'images' for @$res;
+ return $res;
+}
+
+sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) {
+ return [] if defined($vmid) && !$vollist;
+ my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+ my $res = [];
+ for my $dev (sort keys %$namespaces) {
+ my $ns = $namespaces->{$dev};
+ my $volid = "$storeid:nvme-$ns->{wwid}";
+ next if $vollist && !grep { $_ eq $volid } @$vollist;
+ push @$res,
+ {
+ volid => $volid,
+ format => 'raw',
+ size => $ns->{size},
+ vmid => 0,
+ };
+ }
+ return $res;
+}
+
+sub status($class, $storeid, $scfg, $cache = undef, @) {
+ my $active = grep { $_->{state} eq 'live' }
+ values PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ return (0, 0, 0, $active ? 1 : 0);
+}
+
+sub activate_storage($class, $storeid, $scfg, $cache = undef, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ eval {
+ die "no answer from portal\n"
+ if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::connect_path($scfg, $path);
+ };
+ if (my $err = $@) {
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ warn "storage '$storeid': connecting to '"
+ . PVE::NVMeOF::format_path($path)
+ . "' failed - $err";
+ }
+ }
+ return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection($class, $storeid, $scfg, @) {
+ my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+ return 1 if grep { $_->{state} eq 'live' } values %$controllers;
+ for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+ next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+ next if PVE::NVMeOF::in_cooldown($scfg, $path);
+ return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+ PVE::NVMeOF::note_connect_failure($scfg, $path);
+ }
+ return 0;
+}
+
+sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) {
+ my $path = $class->filesystem_path($scfg, $volname, $snapname);
+ # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device
+ my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10);
+ while (!-b $path && Time::HiRes::time() < $deadline) {
+ last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+ Time::HiRes::usleep(250_000);
+ }
+ my $real = Cwd::realpath($path);
+ die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+ assert_namespace($scfg, $volname, $real);
+}
+
+sub volume_has_feature(
+ $class, $scfg, $feature, $storeid, $volname,
+ $snapname = undef,
+ $running = undef, @,
+) {
+ return 1 if $feature eq 'copy' && !defined($snapname);
+ return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 724c2d35..6e305580 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
);
our @SHARED_STORAGE = (
- 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+ 'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
);
our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index b025c03f..86469f5c 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -298,6 +298,11 @@ sub lun_partitions($lun) {
}
sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+ my $name = basename($lun->{devpath});
+ my $nqn =
+ $name =~ /^nvme\d+n\d+$/
+ ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+ : undef;
my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
[$lun->{devpath}, lun_partitions($lun)->@*],
)->@*;
@@ -320,6 +325,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
) {
die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
}
+ if (
+ $type eq 'nvmeof'
+ && defined($nqn)
+ && $scfg->{subsysnqn} eq $nqn
+ && PVE::Storage::san_storage_serves_images($scfg)
+ ) {
+ die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+ }
next if $type ne 'lvm' || !$scfg->{base};
my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base});
my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/;
diff --git a/src/test/Makefile b/src/test/Makefile
index 279ba1ee..c88666f5 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
all: test
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api test_nvmeof
test_zfspoolplugin: run_test_zfspoolplugin.pl
./run_test_zfspoolplugin.pl
@@ -83,3 +83,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
test_disk_api: run_disk_api_tests.pl
./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+ ./run_nvmeof_tests.pl
+ perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..62405291
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,60 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::More;
+use Test::MockModule;
+use PVE::NVMeOF;
+use PVE::Tools qw(file_set_contents file_get_contents);
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+ subsystem_controllers => sub {
+ return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+ },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+ run_command => sub($cmd, %opts) {
+ $commands++;
+ is(
+ scalar(grep { tainted($_) } @$cmd),
+ 0,
+ 'all connect operands are validated and untainted',
+ );
+ PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+ },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+ PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+ 'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+ eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+ like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..b75f7a5a
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,302 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Fcntl qw(:flock);
+use File::Path qw(make_path);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Tools qw(file_set_contents);
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+ type => 'nvmeof',
+ subsysnqn => 'nqn.test',
+ portals => '192.0.2.1,[2001:db8::2]:4421',
+ content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+ is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+ is_deeply(
+ [sort keys $class->properties()->%*],
+ [qw(fast-io-fail-tmo portals subsysnqn)],
+ 'only three plugin-specific properties',
+ );
+ my sub nvmeof_schema($schema) {
+ my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*;
+ my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types;
+ return $nvmeof->{properties} // {};
+ }
+ my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema());
+ is_deeply(
+ [sort grep { !$create->{$_}->{optional} } keys $create->%*],
+ [qw(portals subsysnqn)],
+ 'adding a storage requires the subsystem and the portals',
+ );
+ my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema());
+ ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed');
+ my $parsed = PVE::Storage::Plugin->parse_config(
+ 'storage.cfg',
+ "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n",
+ );
+ is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+ is_deeply(
+ $parsed->{ids}->{fabric}->{content},
+ { none => 1 },
+ 'connection-only content parses',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ { traddr => '192.0.2.1', trsvcid => 4420 },
+ 'default I/O port',
+ );
+ is_deeply(
+ PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+ { traddr => '2001:db8::1', trsvcid => 4421 },
+ 'canonical IPv6',
+ );
+
+ for my $bad (
+ 'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+ ) {
+ ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+ }
+ eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); };
+ like($@, qr/listed more than once/, 'duplicate portals refused');
+ eval {
+ $class->on_add_hook(
+ 'fabric',
+ { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+ );
+ };
+ like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+ my $storage = Test::MockModule->new('PVE::Storage');
+ my $ids = { fabric => $cfg, local => { type => 'dir' } };
+ $storage->mock(config => sub { { ids => $ids } });
+ is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way');
+ eval { $class->on_add_hook('second', {%$cfg}) };
+ like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem');
+ is(
+ scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+ '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+ 'stable volume path',
+ );
+ {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ my $nqn = 'nqn.other';
+ $disk->redefine(read_subsysnqn => sub { $nqn });
+ my $check = sub {
+ eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) };
+ $@;
+ };
+ like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem');
+ like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type');
+ like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition');
+ $nqn = 'nqn.test';
+ is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem');
+ my $cwd = Test::MockModule->new('Cwd');
+ $cwd->redefine(realpath => sub { $dir });
+ eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') };
+ like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked');
+ }
+
+ for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+ eval { $class->parse_volname($bad) };
+ like($@, qr/unable to parse/, "invalid volume $bad");
+ }
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } });
+ is(
+ $class->list_volumes('fabric', $cfg)->[0]->{volid},
+ 'fabric:nvme-eui.abc',
+ 'content none still offers bases',
+ );
+ is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+ $nvme->mock(subsystem_controllers => sub { {} });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+ like($@, qr/not available/,
+ 'missing namespace fails without waiting on absent controllers');
+ my ($now, $queries) = (0, 0);
+ my $clock = Test::MockModule->new('Time::HiRes');
+ $clock->redefine(time => sub { $now += 5 }, usleep => sub { });
+ $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } });
+ eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) };
+ is($queries, 1, 'a missing namespace of a connected subsystem is waited for');
+ eval {
+ $class->activate_volume(
+ 'fabric',
+ $cfg,
+ 'nvme-nonexistent-test',
+ undef,
+ { status_only => 1 },
+ );
+ };
+ like($@, qr/not available/, 'in a status pass it fails as well');
+ is($queries, 1, 'but without a wait');
+ is(
+ ($class->parse_volname('nvme-eui.0123456789abcdef'))[0],
+ 'vm-vol',
+ 'namespaces are VM disks',
+ );
+};
+
+subtest 'sysfs topology' => sub {
+ my $sys = $PVE::NVMeOF::SYSFS;
+ make_path(
+ "$sys/class/nvme-fabrics/ctl/nvme1",
+ "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+ "$sys/block/nvme0n1",
+ );
+ my %attrs = (
+ address => 'traddr=192.0.2.1,trsvcid=4420',
+ transport => 'tcp',
+ subsysnqn => 'nqn.test',
+ state => 'connecting',
+ );
+ file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+ file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+ file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+ file_set_contents("$sys/block/nvme0n1/size", "8\n");
+ is(
+ PVE::NVMeOF::controllers()->{nvme1}->{state},
+ 'connecting',
+ 'reconnecting controller is still present',
+ );
+ is(
+ PVE::NVMeOF::path_controller(
+ PVE::NVMeOF::subsystem_controllers('nqn.test'),
+ PVE::NVMeOF::parse_portal('192.0.2.1'),
+ ),
+ 'nvme1',
+ 'portal controller found',
+ );
+ is_deeply(
+ PVE::NVMeOF::namespaces('nqn.test'),
+ { nvme0n1 => { wwid => 'eui.abc', size => 4096 } },
+ 'namespace facts',
+ );
+ is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+ my $value = PVE::NVMeOF::with_lock(sub {
+ my $out = '';
+ PVE::Tools::run_command(
+ [
+ '/usr/bin/perl',
+ '-MFcntl=:flock',
+ '-e',
+ 'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*"); die "not inherited" if !@held; open(my $fh, ">>", $ARGV[0]) or die $!; die "lock missing" if flock($fh, LOCK_EX|LOCK_NB); print "locked\\n";',
+ "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+ ],
+ timeout => 5,
+ outfunc => sub { $out .= $_[0] },
+ );
+ return $out;
+ });
+ is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+ open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+ flock($lock, LOCK_EX) or die $!;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'busy operation never enters callback');
+ close($lock);
+
+ my $tools = Test::MockModule->new('PVE::Tools');
+ my $timed_out = 1;
+ $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+ my $retained;
+ eval {
+ PVE::NVMeOF::with_lock(sub {
+ my ($path) =
+ grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+ my ($fd) = $path =~ m{/(\d+)$};
+ open($retained, '>&', $fd) or die $!;
+ return 1;
+ });
+ };
+ like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+ $timed_out = 0;
+ eval {
+ PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+ };
+ like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+ close($retained);
+ is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+ my $nvme = Test::MockModule->new('PVE::NVMeOF');
+ my $net = Test::MockModule->new('PVE::Network');
+ my @commands;
+ my $controllers = {};
+ $nvme->mock(with_lock => sub { $_[0]->() });
+ $nvme->mock(assert_support => sub { });
+ $nvme->mock(host_identity =>
+ sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+ $nvme->mock(subsystem_controllers => sub { $controllers });
+ my $appear = 1;
+ $nvme->mock(
+ run_command => sub($cmd, %opts) {
+ is($opts{timeout}, 30, 'bounded connect');
+ push @commands, $cmd;
+ my %arg =
+ map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+ my $controller =
+ { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' };
+ $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear;
+ },
+ );
+ $net->mock(tcp_ping => sub { 1 });
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 2, 'one connect per missing portal');
+ ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+ is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+ @commands = ();
+ $controllers = {
+ nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+ nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+ };
+ $class->activate_storage('fabric', $cfg);
+ is_deeply(\@commands, [],
+ 'existing controllers are never reconnected, regardless of state');
+ $class->deactivate_storage('fabric', $cfg);
+ is_deeply(\@commands, [], 'deactivation does not remove controllers');
+ $controllers = {};
+ my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+ PVE::NVMeOF::note_connect_failure($cfg, $path);
+ ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+ $class->activate_storage('fabric', $cfg);
+ is(scalar(@commands), 1, 'only the other portal connects');
+ ($controllers, $appear) = ({}, 0);
+ my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+ my $warning = '';
+ {
+ local $SIG{__WARN__} = sub { $warning .= $_[0] };
+ $class->activate_storage('fabric', $cfg);
+ }
+ like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+ ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+ my $disk = Test::MockModule->new('PVE::Diskmanage');
+ for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+ $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+ eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+ like($@, qr/not a valid local disk/,
+ "$transport refuses without storage configuration");
+ }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 51236f87..c74b638a 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
$disk->mock(verify_blockdev_path => sub { $device });
$disk->mock(is_partition => sub { 0 });
$disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
$disk->mock(is_iscsi_device => sub { 0 });
$disk->mock(is_mounted => sub { 0 });
$disk->mock(has_holder => sub { 0 });
@@ -78,6 +79,21 @@ $disk->mock(is_partition => sub { 0 });
$cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' };
eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
like($@, qr/base volume/, 'native base reference refuses');
+$cfg->{ids} =
+ { fabric =>
+ { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+ $@,
+ qr/serves images from this subsystem/,
+ 'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks');
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
$cfg->{ids} = {};
@order = @opens = ();
$tools->mock(file_get_contents => sub { 1024 });
next prev parent reply other threads:[~2026-11-18 20:45 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` Thomas Lamprecht [this message]
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118204302.3100813-3-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox