public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems
Date: Sun, 22 Nov 2026 02:25:39 +0100	[thread overview]
Message-ID: <20261122012605.1848824-3-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261122012605.1848824-1-t.lamprecht@proxmox.com>

NVMe/TCP namespaces need a manual nvme-cli setup on every node so far.
Add a storage type that connects one subsystem through each configured
portal and offers its namespaces as raw volumes. Native NVMe multipath
joins the paths and the kernel reconnects lost ones. Only one storage
may use a subsystem, as the LUN list, the controller list and the node
status attribute a subsystem to one storage.

The activation fails if no usable path exists after connecting the
missing portals, so that adding a storage tells why. As adding a
storage holds the lock of the configuration, it tries no further portal
after 15 seconds once a controller is live, and leaves the rest to the
next activation. A portal that failed is retried after a minute, and
adding a storage forgets those failures. Disk setup refuses namespaces
attached over a fabric, as for iSCSI LUNs, and nvme-cli is only
recommended, as no other storage type needs it.

The plugin isolates its properties, so names like 'portals' belong to
this storage type alone, and a third-party plugin that declares one of
them is still loaded. As the first in-tree plugin that isolates, it
makes the storage API check the parameters for adding and updating a
storage of any type against the options of that type alone. A
third-party plugin then has to list the sensitive properties it uses in
options(), as the API change log says.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 ApiChangeLog                       |   9 +-
 debian/control                     |   1 +
 src/PVE/Diskmanage.pm              |   7 +-
 src/PVE/Makefile                   |   1 +
 src/PVE/NVMeOF.pm                  | 271 ++++++++++++++++++
 src/PVE/Storage.pm                 |   2 +
 src/PVE/Storage/Makefile           |   1 +
 src/PVE/Storage/NVMeOFPlugin.pm    | 266 +++++++++++++++++
 src/PVE/Storage/Plugin.pm          |   2 +-
 src/PVE/Storage/SAN.pm             |  13 +
 src/test/Makefile                  |   5 +
 src/test/run_nvmeof_taint_tests.pl |  62 ++++
 src/test/run_nvmeof_tests.pl       | 446 +++++++++++++++++++++++++++++
 src/test/run_san_nvme_tests.pl     |  16 ++
 14 files changed, 1096 insertions(+), 6 deletions(-)
 create mode 100644 src/PVE/NVMeOF.pm
 create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
 create mode 100755 src/test/run_nvmeof_taint_tests.pl
 create mode 100755 src/test/run_nvmeof_tests.pl

diff --git a/ApiChangeLog b/ApiChangeLog
index a51578a1..1816e18a 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -101,10 +101,11 @@ Future changes should be documented in here.
   declares a property of that name is then not loaded until it renames it, so prefer names that
   are specific to the plugin, for example prefixed with its type.
 
-  As soon as one such plugin is installed, the API checks the parameters for adding and updating a
-  storage against the options of its type alone, for all storage types. So every plugin has to
-  list the sensitive properties that it uses in `options()`.
-  Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of
+  Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for
+  adding and updating a storage against the options of its type alone, for all storage types,
+  also for plugins of older API versions. So every plugin has to list the sensitive properties
+  that it uses in `options()`.
+  Set 'sensitive-properties' in an isolating plugin, even if empty. Without it, the fallback list of
   version 11 also applies to the plugin's own properties of these names.
 
 * Pass the guest identity to backup providers
diff --git a/debian/control b/debian/control
index fdcf8117..b5c2f529 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
          ${misc:Depends},
          ${perl:Depends},
 Recommends: multipath-tools,
+            nvme-cli,
             pve-esxi-import-tools (>= 0.6.0),
             skopeo,
             zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index be3fb8fc..21857dc6 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -107,7 +107,12 @@ sub disk_is_used {
 
     my $disklist = get_disks($dev, 1, 1);
 
-    die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+    my $entry = $disklist->{$dev};
+    die "'$disk' is not a valid local disk\n" if !defined($entry);
+    # a partition has the transport of its disk
+    my ($parent) = ($entry->{parent} // '') =~ m!^/dev/(.+)$!;
+    my $transport = (defined($parent) ? $disklist->{$parent} : $entry)->{transport} // '';
+    die "'$disk' is not a valid local disk\n" if $transport =~ /^nvme-(?:tcp|rdma|fc)$/;
     return 1 if $disklist->{$dev}->{used};
 
     return 0;
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
 install:
 	install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
 	install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+	install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
 	install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
 	install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
 	install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..45b3acfc
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,271 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+use Time::HiRes ();
+
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+use PVE::Diskmanage;
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $LOCK_WAIT = 10;
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+our $DISCOVERY_NQN = 'nqn.2014-08.org.nvmexpress.discovery';
+
+sub canonical_address($addr) {
+    if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+        return inet_ntop(AF_INET, $packed);
+    }
+    if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+        return inet_ntop(AF_INET, substr($packed, 12))
+            if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+        return inet_ntop(AF_INET6, $packed);
+    }
+    return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+    my ($addr, $port);
+    if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) {
+        ($addr, $port) = ($1, $2);
+    } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) {
+        ($addr, $port) = ($1, $2);
+    } else {
+        return undef;
+    }
+    return undef if defined($port) && ($port < 1 || $port > 65535);
+    $addr = canonical_address($addr);
+    if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+        my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0;
+        return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast;
+    } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+        my $prefix = unpack('n', $packed);
+        return undef
+            if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+    } else {
+        return undef;
+    }
+    return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+    'pve-storage-nvme-portal',
+    sub($portal, $noerr = undef) {
+        return $portal if parse_portal($portal);
+        return undef if $noerr;
+        die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+    },
+);
+
+sub format_path($path) {
+    # an IPv6 address, not the address of a Fibre Channel port, which has a colon too
+    my $ipv6 = $path->{traddr} =~ /:/ && $path->{traddr} =~ /^[0-9a-fA-F:.]+$/;
+    my $addr = $ipv6 ? "[$path->{traddr}]" : $path->{traddr};
+    return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+    return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+        PVE::Tools::split_list($scfg->{portals})];
+}
+
+sub host_identity() {
+    my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+    my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+        /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+    if (!defined($nqn) || length($nqn) > 223) {
+        die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the"
+            . " host identity\n";
+    }
+    die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+        if !defined($id) || $id =~ /^[0-]+$/;
+    return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+    die "nvme-cli is not installed\n" if !is_supported();
+    run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+        if !-e "$SYSFS/module/nvme_tcp";
+    my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+    die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+    my $res = {};
+    my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+    dir_glob_foreach(
+        $dir,
+        'nvme[0-9]+',
+        sub($name, @) {
+            my $address = file_read_firstline("$dir/$name/address") // return;
+            my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+            return if !defined($address{traddr});
+            $res->{$name} = {
+                transport => file_read_firstline("$dir/$name/transport") // '',
+                subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+                state => file_read_firstline("$dir/$name/state") // 'unknown',
+                traddr => canonical_address($address{traddr}),
+                trsvcid => $address{trsvcid},
+            };
+        },
+    );
+    return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+    my $controllers = controllers();
+    return {
+        map { $_ => $controllers->{$_} } grep {
+            $controllers->{$_}->{subsystem} eq $subsystem
+                && $controllers->{$_}->{transport} eq 'tcp'
+        } keys %$controllers
+    };
+}
+
+# The NQN of a subsystem, untainted, or undef if it is invalid or the one of the discovery service.
+sub subsystem_nqn($nqn) {
+    my ($res) = ($nqn // '') =~ /\A($NQN_PATTERN)\z/;
+    return undef if !defined($res) || length($res) > 223 || $res eq $DISCOVERY_NQN;
+    return $res;
+}
+
+sub namespace_wwid($dev) {
+    return file_read_firstline("$SYSFS/block/$dev/wwid");
+}
+
+sub has_live_controller($controllers) {
+    return scalar(grep { $_->{state} eq 'live' } values %$controllers);
+}
+
+sub path_controller($controllers, $path) {
+    for my $name (sort keys %$controllers) {
+        my $ctrl = $controllers->{$name};
+        if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) {
+            return $name;
+        }
+    }
+    return undef;
+}
+
+sub namespaces($subsystem) {
+    my $res = {};
+    my $dir = "$SYSFS/class/nvme-subsystem";
+    dir_glob_foreach(
+        $dir,
+        'nvme-subsys[0-9]+',
+        sub($subsys, @) {
+            my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+            return if $nqn ne $subsystem;
+            dir_glob_foreach(
+                "$dir/$subsys",
+                'nvme[0-9]+n[0-9]+',
+                sub($dev, @) {
+                    my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+                    my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+                    $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+                },
+            );
+        },
+    );
+    return $res;
+}
+
+sub with_lock($code) {
+    my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+        $LOCK_WAIT + 35,
+        sub {
+            make_path($RUN_DIR);
+            open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+            # a busy lock fails a connect and arms the cooldown of its portal, so wait a while
+            my $deadline = Time::HiRes::time() + $LOCK_WAIT;
+            while (!flock($lock, LOCK_EX | LOCK_NB)) {
+                die "NVMe over Fabrics is busy - another operation is still running\n"
+                    if Time::HiRes::time() >= $deadline;
+                Time::HiRes::usleep(100_000);
+            }
+            # Commands inherit the lock, so that it stays held until they exit, even if the caller
+            # timed out while one was stuck in the kernel.
+            my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+            fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+                or die "cannot inherit NVMe lock - $!\n";
+            return $code->();
+        },
+    );
+    die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+    return $res;
+}
+
+my sub retry_file($scfg, $path) {
+    my ($key) =
+        sha1_hex(join(
+            "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~
+        /\A([0-9a-f]{40})\z/;
+    return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+    my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+    my $age = time() - $mtime;
+    return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+    make_path("$RUN_DIR/retry");
+    file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub clear_connect_failures($scfg) {
+    unlink(retry_file($scfg, $_)) for storage_paths($scfg)->@*;
+}
+
+sub connect_path($scfg, $path) {
+    my $nqn = subsystem_nqn($scfg->{subsysnqn}) // die "invalid subsystem NQN\n";
+    my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/;
+    die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+    return with_lock(sub {
+        return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+        assert_support();
+        my $host = host_identity();
+        run_command(
+            [
+                $NVME,
+                'connect',
+                '--transport=tcp',
+                '--nqn',
+                $nqn,
+                '--traddr',
+                $path->{traddr},
+                '--trsvcid',
+                $path->{trsvcid},
+                '--hostnqn',
+                $host->{nqn},
+                '--hostid',
+                $host->{id},
+                '--ctrl-loss-tmo=-1',
+                '--fast_io_fail_tmo',
+                $fail,
+            ],
+            timeout => 30,
+        );
+        # nvme-cli can exit successfully without having connected
+        die "no controller appeared\n"
+            if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+        unlink(retry_file($scfg, $path));
+        return;
+    });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index c640cd8e..c77a6755 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin;
 use PVE::Storage::NFSPlugin;
 use PVE::Storage::CIFSPlugin;
 use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
 use PVE::Storage::RBDPlugin;
 use PVE::Storage::CephFSPlugin;
 use PVE::Storage::ISCSIDirectPlugin;
@@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register();
 PVE::Storage::NFSPlugin->register();
 PVE::Storage::CIFSPlugin->register();
 PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
 PVE::Storage::RBDPlugin->register();
 PVE::Storage::CephFSPlugin->register();
 PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
 	NFSPlugin.pm \
 	CIFSPlugin.pm \
 	ISCSIPlugin.pm \
+	NVMeOFPlugin.pm \
 	CephFSPlugin.pm \
 	RBDPlugin.pm \
 	ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..76f73af0
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,266 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use v5.36;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Network;
+
+use PVE::Diskmanage;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+# Adding a storage activates it under the lock of the storage configuration, so once a controller
+# is live, no further portal is tried after this many seconds. Later activations connect the rest.
+our $ACTIVATION_BUDGET = 15;
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+    return {
+        content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }],
+        select_existing => 1,
+        'isolate-properties' => 1,
+        'advanced-properties' => {
+            'fast-io-fail-tmo' => 1,
+        },
+        'hidden-properties' => {
+            bwlimit => 1,
+        },
+        'sensitive-properties' => {},
+    };
+}
+
+sub properties {
+    return {
+        subsysnqn => {
+            title => "Subsystem",
+            description => 'The NQN of the NVMe subsystem.',
+            type => 'string',
+            pattern => $PVE::NVMeOF::NQN_PATTERN,
+            maxLength => 223,
+        },
+        portals => {
+            title => "Portals",
+            description =>
+                'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+                . ' with an optional port (default 4420).',
+            type => 'string',
+            format => 'pve-storage-nvme-portal-list',
+        },
+        'fast-io-fail-tmo' => {
+            title => "Fail I/O After (s)",
+            description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+                . ' this timeout; I/O can remain queued while paths reconnect.',
+            type => 'integer',
+            minimum => -1,
+            default => 120,
+        },
+    };
+}
+
+sub options {
+    return {
+        subsysnqn => { fixed => 1 },
+        portals => {},
+        'fast-io-fail-tmo' => { optional => 1 },
+        nodes => { optional => 1 },
+        disable => { optional => 1 },
+        content => { optional => 1 },
+        bwlimit => { optional => 1 },
+    };
+}
+
+my sub assert_valid_config($scfg) {
+    die "the discovery subsystem cannot be used as storage\n"
+        if $scfg->{subsysnqn} eq $PVE::NVMeOF::DISCOVERY_NQN;
+    my $paths = PVE::NVMeOF::storage_paths($scfg);
+    # the list format also accepts a list of separators only
+    die "no portal given\n" if !@$paths;
+    my %seen;
+    for my $path (@$paths) {
+        my $portal = PVE::NVMeOF::format_path($path);
+        die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+    }
+}
+
+sub on_add_hook($class, $storeid, $scfg, %param) {
+    assert_valid_config($scfg);
+    # the LUN and controller lists and the node status attribute a subsystem to its one storage
+    my $ids = PVE::Storage::config()->{ids};
+    for my $sid (sort keys %$ids) {
+        next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof';
+        die "storage '$sid' already uses this subsystem\n"
+            if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn};
+    }
+    # adding the storage again, for example once the array allows the node, tries all portals
+    PVE::NVMeOF::clear_connect_failures($scfg);
+    return;
+}
+
+sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) {
+    my $merged = { %$scfg, %$update };
+    delete $merged->{$_} for ($delete // [])->@*;
+    assert_valid_config($merged);
+    return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname($class, $volname) {
+    if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) {
+        return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw');
+    }
+    die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a
+# namespace of another subsystem.
+sub assert_namespace($scfg, $volname, $real) {
+    my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!;
+    die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+    my $nqn = PVE::Diskmanage::read_subsysnqn("$PVE::NVMeOF::SYSFS/block/$dev/device/subsysnqn")
+        // '';
+    die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+        if $nqn ne $scfg->{subsysnqn};
+    # another by-id name of the subsystem, like model and serial, can name another namespace
+    my $wwid = PVE::NVMeOF::namespace_wwid($dev) // '';
+    die "volume '$volname' is not named after the WWID of its namespace\n"
+        if $volname ne "nvme-$wwid";
+}
+
+sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) {
+    die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+    my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+    my $path = "/dev/disk/by-id/$name";
+    # not every user of a path activates the volume first, and the base volume of an LVM storage
+    # has no device before its storage connects
+    my $real = Cwd::realpath($path);
+    assert_namespace($scfg, $volname, $real) if defined($real) && -e $real;
+    return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) {
+    my $res = $class->list_images($storeid, $scfg, $vmid);
+    $_->{content} = 'images' for @$res;
+    return $res;
+}
+
+sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) {
+    return [] if defined($vmid) && !$vollist;
+    my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+    my $res = [];
+    for my $dev (sort keys %$namespaces) {
+        my $ns = $namespaces->{$dev};
+        my $volid = "$storeid:nvme-$ns->{wwid}";
+        next if $vollist && !grep { $_ eq $volid } @$vollist;
+        push @$res,
+            {
+                volid => $volid,
+                format => 'raw',
+                size => $ns->{size},
+                vmid => 0,
+            };
+    }
+    return $res;
+}
+
+sub status($class, $storeid, $scfg, $cache = undef, @) {
+    my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    return (0, 0, 0, PVE::NVMeOF::has_live_controller($controllers) ? 1 : 0);
+}
+
+sub activate_storage($class, $storeid, $scfg, $cache = undef, @) {
+    my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    my @missing = grep { !defined(PVE::NVMeOF::path_controller($controllers, $_)) }
+        PVE::NVMeOF::storage_paths($scfg)->@*;
+    return if !@missing;
+    # Without these no path connects, which is no reason to wait before the next attempt. A storage
+    # with a usable path still works, with fewer paths.
+    eval {
+        PVE::NVMeOF::assert_support();
+        PVE::NVMeOF::host_identity();
+    };
+    if (my $err = $@) {
+        die $err if !PVE::NVMeOF::has_live_controller($controllers);
+        warn "storage '$storeid': cannot connect to the other portals - $err";
+        return;
+    }
+    my @errors;
+    my $live = PVE::NVMeOF::has_live_controller($controllers);
+    my $deadline = Time::HiRes::time() + $ACTIVATION_BUDGET;
+    for my $path (@missing) {
+        my $portal = PVE::NVMeOF::format_path($path);
+        if ($live && Time::HiRes::time() >= $deadline) {
+            push @errors, "$portal: left for the next activation";
+            next;
+        }
+        if (PVE::NVMeOF::in_cooldown($scfg, $path)) {
+            push @errors, "$portal: failed less than a minute ago";
+            next;
+        }
+        eval {
+            die "no answer from portal\n"
+                if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+            PVE::NVMeOF::connect_path($scfg, $path);
+        };
+        if (my $err = $@) {
+            PVE::NVMeOF::note_connect_failure($scfg, $path);
+            warn "storage '$storeid': connecting to '$portal' failed - $err";
+            chomp $err;
+            push @errors, "$portal: $err";
+        } else {
+            $live = 1;
+        }
+    }
+    # some paths are enough, but without a usable one the namespaces cannot be used
+    $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    die "cannot connect to any portal - " . join('; ', @errors) . "\n"
+        if !PVE::NVMeOF::has_live_controller($controllers);
+    return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection($class, $storeid, $scfg, @) {
+    my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    return 1 if PVE::NVMeOF::has_live_controller($controllers);
+    for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+        next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+        next if PVE::NVMeOF::in_cooldown($scfg, $path);
+        return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+        PVE::NVMeOF::note_connect_failure($scfg, $path);
+    }
+    return 0;
+}
+
+sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) {
+    my $path = $class->filesystem_path($scfg, $volname, $snapname);
+    # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device
+    my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10);
+    while (!-b $path && Time::HiRes::time() < $deadline) {
+        last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+        Time::HiRes::usleep(250_000);
+    }
+    my $real = Cwd::realpath($path);
+    die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+    assert_namespace($scfg, $volname, $real);
+}
+
+sub volume_has_feature(
+    $class, $scfg, $feature, $storeid, $volname,
+    $snapname = undef,
+    $running = undef, @,
+) {
+    return 1 if $feature eq 'copy' && !defined($snapname);
+    return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index a97f361c..d61e78b6 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
 );
 
 our @SHARED_STORAGE = (
-    'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+    'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
 );
 
 our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index 0a9e441c..53c24c9e 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -302,6 +302,11 @@ sub lun_partitions($lun) {
 }
 
 sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+    my $name = basename($lun->{devpath});
+    my $nqn =
+        $name =~ /^nvme\d+n\d+$/
+        ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+        : undef;
     my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
         [$lun->{devpath}, lun_partitions($lun)->@*],
     )->@*;
@@ -324,6 +329,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
         ) {
             die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
         }
+        if (
+            $type eq 'nvmeof'
+            && defined($nqn)
+            && $scfg->{subsysnqn} eq $nqn
+            && PVE::Storage::san_storage_serves_images($scfg)
+        ) {
+            die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+        }
         next if $type ne 'lvm' || !$scfg->{base};
         my $base = PVE::Multipath::parse_base_volume($scfg->{base});
         my $matches = $base
diff --git a/src/test/Makefile b/src/test/Makefile
index 3b902c43..ceabd991 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -17,6 +17,7 @@ test: \
     test_lvmplugin \
     test_multipath \
     test_multipath_api \
+    test_nvmeof \
     test_ovf \
     test_plugin \
     test_remote_command \
@@ -115,3 +116,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
 
 test_disk_api: run_disk_api_tests.pl
 	./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+	./run_nvmeof_tests.pl
+	perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..6fd37410
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,62 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Tools qw(file_get_contents file_set_contents);
+
+use PVE::NVMeOF;
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+    subsystem_controllers => sub {
+        return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+    },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+    run_command => sub($cmd, %opts) {
+        $commands++;
+        is(
+            scalar(grep { tainted($_) } @$cmd),
+            0,
+            'all connect operands are validated and untainted',
+        );
+        PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+    },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+    PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+    'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+    eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+    like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..cdf228c0
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,446 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Fcntl qw(:flock);
+use File::Path qw(make_path remove_tree);
+use File::Temp qw(tempdir);
+use POSIX ();
+use Test::MockModule;
+use Test::More;
+use Time::HiRes ();
+
+use PVE::Tools qw(file_set_contents);
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+    type => 'nvmeof',
+    subsysnqn => 'nqn.test',
+    portals => '192.0.2.1,[2001:db8::2]:4421',
+    content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+    is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+    is_deeply(
+        [sort keys $class->properties()->%*],
+        [qw(fast-io-fail-tmo portals subsysnqn)],
+        'only three plugin-specific properties',
+    );
+    my sub nvmeof_schema($schema) {
+        my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*;
+        my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types;
+        return $nvmeof->{properties} // {};
+    }
+    my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema());
+    is_deeply(
+        [sort grep { !$create->{$_}->{optional} } keys $create->%*],
+        [qw(portals subsysnqn)],
+        'adding a storage requires the subsystem and the portals',
+    );
+    my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema());
+    ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed');
+    my $parsed = PVE::Storage::Plugin->parse_config(
+        'storage.cfg',
+        "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n",
+    );
+    is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+    is_deeply(
+        $parsed->{ids}->{fabric}->{content},
+        { none => 1 },
+        'connection-only content parses',
+    );
+    is_deeply(
+        PVE::NVMeOF::parse_portal('192.0.2.1'),
+        { traddr => '192.0.2.1', trsvcid => 4420 },
+        'default I/O port',
+    );
+    is_deeply(
+        PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+        { traddr => '2001:db8::1', trsvcid => 4421 },
+        'canonical IPv6',
+    );
+    is(
+        PVE::NVMeOF::format_path({ traddr => '2001:db8::1', trsvcid => 4421 }),
+        '[2001:db8::1]:4421',
+        'an IPv6 portal in brackets',
+    );
+    is(
+        PVE::NVMeOF::format_path({ traddr => 'nn-0x2000:pn-0x2001' }),
+        'nn-0x2000:pn-0x2001',
+        'the address of a Fibre Channel port as it is',
+    );
+
+    for my $bad (
+        'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+    ) {
+        ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+    }
+    eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); };
+    like($@, qr/listed more than once/, 'duplicate portals refused');
+    eval { $class->on_add_hook('fabric', { %$cfg, portals => ',,' }); };
+    like($@, qr/no portal given/, 'a list without a portal refused');
+    eval {
+        $class->on_add_hook(
+            'fabric',
+            { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+        );
+    };
+    like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+    my $storage = Test::MockModule->new('PVE::Storage');
+    my $ids = { fabric => $cfg, local => { type => 'dir' } };
+    $storage->mock(config => sub { { ids => $ids } });
+    my $first = PVE::NVMeOF::storage_paths($cfg)->[0];
+    PVE::NVMeOF::note_connect_failure($cfg, $first);
+    is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way');
+    ok(!PVE::NVMeOF::in_cooldown($cfg, $first), 'adding it tries all portals again');
+    eval { $class->on_add_hook('second', {%$cfg}) };
+    like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem');
+    is(
+        scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+        '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+        'stable volume path',
+    );
+    {
+        my $disk = Test::MockModule->new('PVE::Diskmanage');
+        my $nqn = 'nqn.other';
+        $disk->redefine(read_subsysnqn => sub { $nqn });
+        my $sysfs = Test::MockModule->new('PVE::NVMeOF');
+        $sysfs->redefine(namespace_wwid => sub { 'x' });
+        my $check = sub {
+            eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) };
+            $@;
+        };
+        like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem');
+        like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type');
+        like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition');
+        $nqn = 'nqn.test';
+        is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem');
+        like(
+            $check->('nvme-Vendor_Model_123_1', '/dev/nvme0n1'),
+            qr/not named after the WWID/,
+            'another by-id name of a namespace of the subsystem',
+        );
+        my $cwd = Test::MockModule->new('Cwd');
+        $cwd->redefine(realpath => sub { $dir });
+        eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') };
+        like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked');
+    }
+
+    for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+        eval { $class->parse_volname($bad) };
+        like($@, qr/unable to parse/, "invalid volume $bad");
+    }
+    my $nvme = Test::MockModule->new('PVE::NVMeOF');
+    $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } });
+    is(
+        $class->list_volumes('fabric', $cfg)->[0]->{volid},
+        'fabric:nvme-eui.abc',
+        'content none still offers bases',
+    );
+    is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+    $nvme->mock(subsystem_controllers => sub { {} });
+    eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+    like($@, qr/not available/,
+        'missing namespace fails without waiting on absent controllers');
+    my ($now, $queries) = (0, 0);
+    my $clock = Test::MockModule->new('Time::HiRes');
+    $clock->redefine(time => sub { $now += 5 }, usleep => sub { });
+    $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } });
+    eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) };
+    is($queries, 1, 'a missing namespace of a connected subsystem is waited for');
+    eval {
+        $class->activate_volume(
+            'fabric',
+            $cfg,
+            'nvme-nonexistent-test',
+            undef,
+            { status_only => 1 },
+        );
+    };
+    like($@, qr/not available/, 'in a status pass it fails as well');
+    is($queries, 1, 'but without a wait');
+    is(
+        ($class->parse_volname('nvme-eui.0123456789abcdef'))[0],
+        'vm-vol',
+        'namespaces are VM disks',
+    );
+};
+
+subtest 'sysfs topology' => sub {
+    my $sys = $PVE::NVMeOF::SYSFS;
+    make_path(
+        "$sys/class/nvme-fabrics/ctl/nvme1",
+        "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+        "$sys/block/nvme0n1",
+    );
+    my %attrs = (
+        address => 'traddr=192.0.2.1,trsvcid=4420',
+        transport => 'tcp',
+        subsysnqn => 'nqn.test',
+        state => 'connecting',
+    );
+    file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+    file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+    file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+    file_set_contents("$sys/block/nvme0n1/size", "8\n");
+    is(
+        PVE::NVMeOF::controllers()->{nvme1}->{state},
+        'connecting',
+        'reconnecting controller is still present',
+    );
+    is(
+        PVE::NVMeOF::path_controller(
+            PVE::NVMeOF::subsystem_controllers('nqn.test'),
+            PVE::NVMeOF::parse_portal('192.0.2.1'),
+        ),
+        'nvme1',
+        'portal controller found',
+    );
+    is_deeply(
+        PVE::NVMeOF::namespaces('nqn.test'),
+        { nvme0n1 => { wwid => 'eui.abc', size => 4096 } },
+        'namespace facts',
+    );
+    is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+    local $PVE::NVMeOF::LOCK_WAIT = 0;
+    my $script = join(
+        ' ',
+        'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*");',
+        'die "not inherited" if !@held;',
+        'open(my $fh, ">>", $ARGV[0]) or die $!;',
+        'die "lock missing" if flock($fh, LOCK_EX|LOCK_NB);',
+        'print "locked\\n";',
+    );
+    my $value = PVE::NVMeOF::with_lock(sub {
+        my $out = '';
+        PVE::Tools::run_command(
+            [
+                '/usr/bin/perl',
+                '-MFcntl=:flock',
+                '-e',
+                $script,
+                "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+            ],
+            timeout => 5,
+            outfunc => sub { $out .= $_[0] },
+        );
+        return $out;
+    });
+    is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+    open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+    flock($lock, LOCK_EX) or die $!;
+    eval {
+        PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+    };
+    like($@, qr/is busy/, 'busy operation never enters callback');
+    close($lock);
+
+    my $tools = Test::MockModule->new('PVE::Tools');
+    my $timed_out = 1;
+    $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+    my $retained;
+    eval {
+        PVE::NVMeOF::with_lock(sub {
+            my ($path) =
+                grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+            my ($fd) = $path =~ m{/(\d+)$};
+            open($retained, '>&', $fd) or die $!;
+            return 1;
+        });
+    };
+    like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+    $timed_out = 0;
+    eval {
+        PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+    };
+    like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+    close($retained);
+    is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'a short hold of the lock does not fail a connect' => sub {
+    my $nvme = Test::MockModule->new('PVE::NVMeOF');
+    my $net = Test::MockModule->new('PVE::Network');
+    my $scfg = { %$cfg, subsysnqn => 'nqn.wait', portals => '192.0.2.3' };
+    my $ctl = "$PVE::NVMeOF::SYSFS/class/nvme-fabrics/ctl/nvme9";
+    $net->mock(tcp_ping => sub { 1 });
+    $nvme->mock(assert_support => sub { });
+    $nvme->mock(host_identity =>
+        sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+    $nvme->mock(
+        run_command => sub(@) {
+            my %attrs = (
+                address => 'traddr=192.0.2.3,trsvcid=4420',
+                transport => 'tcp',
+                subsysnqn => 'nqn.wait',
+                state => 'live',
+            );
+            make_path($ctl);
+            file_set_contents("$ctl/$_", "$attrs{$_}\n") for keys %attrs;
+        },
+    );
+    pipe(my $held, my $signal) or die "pipe failed - $!\n";
+    my $holder = fork() // die "fork failed - $!\n";
+    if (!$holder) {
+        open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or POSIX::_exit(1);
+        flock($lock, LOCK_EX) or POSIX::_exit(1);
+        syswrite($signal, '1');
+        Time::HiRes::sleep(0.5);
+        POSIX::_exit(0);
+    }
+    close($signal);
+    sysread($held, my $state, 1);
+    is($state, '1', 'another process holds the lock');
+    my $warning = '';
+    {
+        local $SIG{__WARN__} = sub { $warning .= $_[0] };
+        $class->activate_storage('wait', $scfg);
+    }
+    waitpid($holder, 0);
+    is($warning, '', 'the connect waits for the lock');
+    my $path = PVE::NVMeOF::storage_paths($scfg)->[0];
+    ok(!PVE::NVMeOF::in_cooldown($scfg, $path), 'and arms no cooldown');
+    is(
+        PVE::NVMeOF::path_controller(PVE::NVMeOF::subsystem_controllers('nqn.wait'), $path),
+        'nvme9',
+        'the portal is connected',
+    );
+    remove_tree($ctl);
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+    my $nvme = Test::MockModule->new('PVE::NVMeOF');
+    my $net = Test::MockModule->new('PVE::Network');
+    my @commands;
+    my $controllers = {};
+    $nvme->mock(with_lock => sub { $_[0]->() });
+    $nvme->mock(assert_support => sub { });
+    $nvme->mock(host_identity =>
+        sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+    $nvme->mock(subsystem_controllers => sub { $controllers });
+    my $appear = 1;
+    $nvme->mock(
+        run_command => sub($cmd, %opts) {
+            is($opts{timeout}, 30, 'bounded connect');
+            push @commands, $cmd;
+            my %arg =
+                map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+            my $controller =
+                { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' };
+            $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear;
+        },
+    );
+    $net->mock(tcp_ping => sub { 1 });
+    $class->activate_storage('fabric', $cfg);
+    is(scalar(@commands), 2, 'one connect per missing portal');
+    ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+    is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+    {
+        local $PVE::Storage::NVMeOFPlugin::ACTIVATION_BUDGET = 0;
+        @commands = ();
+        $controllers = {};
+        $class->activate_storage('fabric', $cfg);
+        is(scalar(@commands), 1, 'with a live controller, a spent time budget leaves the rest');
+        ($controllers, $appear) = ({}, 0);
+        @commands = ();
+        {
+            local $SIG{__WARN__} = sub { };
+            eval { $class->activate_storage('fabric', $cfg) };
+        }
+        is(scalar(@commands), 2, 'without one, every portal is tried');
+        unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*"));
+        $appear = 1;
+    }
+    @commands = ();
+    $controllers = {
+        nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+        nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+    };
+    $class->activate_storage('fabric', $cfg);
+    is_deeply(\@commands, [],
+        'existing controllers are never reconnected, regardless of state');
+    $class->deactivate_storage('fabric', $cfg);
+    is_deeply(\@commands, [], 'deactivation does not remove controllers');
+    $controllers = {};
+    my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+    PVE::NVMeOF::note_connect_failure($cfg, $path);
+    ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+    $class->activate_storage('fabric', $cfg);
+    is(scalar(@commands), 1, 'only the other portal connects');
+    ($controllers, $appear) = ({}, 0);
+    my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+    my $warning = '';
+    {
+        local $SIG{__WARN__} = sub { $warning .= $_[0] };
+        eval { $class->activate_storage('fabric', $cfg) };
+    }
+    like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+    ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+    like(
+        $@,
+        qr/^cannot connect to any portal - .*failed less than a minute ago; .*no controller appeared/,
+        'without any controller the activation fails with the reason of each portal',
+    );
+    unlink(glob("$PVE::NVMeOF::RUN_DIR/retry/*"));
+    $nvme->mock(assert_support => sub { die "nvme-cli is not installed\n" });
+    @commands = ();
+    eval { $class->activate_storage('fabric', $cfg) };
+    is($@, "nvme-cli is not installed\n", 'a missing prerequisite fails the activation');
+    is_deeply(\@commands, [], 'before any connect');
+    ok(!PVE::NVMeOF::in_cooldown($cfg, $path), 'and without a cooldown');
+    $controllers = { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'live' } };
+    $warning = '';
+    {
+        local $SIG{__WARN__} = sub { $warning .= $_[0] };
+        is(
+            eval { $class->activate_storage('fabric', $cfg); 1 },
+            1,
+            'but not with a usable path',
+        );
+    }
+    like($warning, qr/cannot connect to the other portals - nvme-cli/, 'which it warns about');
+    $nvme->mock(assert_support => sub { });
+    $controllers =
+        { nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' } };
+    {
+        local $SIG{__WARN__} = sub { };
+        eval { $class->activate_storage('fabric', $cfg) };
+    }
+    like(
+        $@,
+        qr/^cannot connect to any portal/,
+        'a reconnecting path is not usable when the missing one fails',
+    );
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+    my $disk = Test::MockModule->new('PVE::Diskmanage');
+    for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+        $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+        eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+        like($@, qr/not a valid local disk/,
+            "$transport refuses without storage configuration");
+        $disk->mock(
+            get_disks => sub {
+                {
+                    nvme0n1 => { transport => $transport },
+                    nvme0n1p1 => { parent => '/dev/nvme0n1', type => 'partition' },
+                };
+            },
+        );
+        eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1p1') };
+        like($@, qr/not a valid local disk/, "$transport refuses a partition of a namespace");
+    }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 3245b774..2a9a2121 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
 $disk->mock(verify_blockdev_path => sub { $device });
 $disk->mock(is_partition => sub { 0 });
 $disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
 $disk->mock(is_iscsi_device => sub { 0 });
 $disk->mock(is_mounted => sub { 0 });
 $disk->mock(has_holder => sub { 0 });
@@ -86,6 +87,21 @@ for my $base (qw(fabric nvme-fabric)) {
     eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
     like($@, qr/base volume/, "native base reference on storage '$base' refuses");
 }
+$cfg->{ids} =
+    { fabric =>
+        { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+    $@,
+    qr/serves images from this subsystem/,
+    'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks');
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
 $cfg->{ids} = {};
 @order = @opens = ();
 $tools->mock(file_get_contents => sub { 1024 });




  parent reply	other threads:[~2026-11-22  1:28 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-22  1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22  1:25 ` Thomas Lamprecht [this message]
2026-11-22  1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122012605.1848824-3-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal