all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems
Date: Wed, 18 Nov 2026 21:42:21 +0100	[thread overview]
Message-ID: <20261118204302.3100813-3-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com>

NVMe/TCP namespaces need a manual nvme-cli setup on every node so
far. Add a storage type that connects one subsystem through each
configured portal and offers its namespaces as raw volumes. Native
NVMe multipath joins the paths and the kernel reconnects lost ones.
A second storage for a subsystem is refused: the LUN list, the
controller list and the node status attribute a subsystem to one
storage. Resolving the path of a volume checks an existing device
against the subsystem: a volume name can be the by-id name of any NVMe
device, and not every user of a path activates the volume first.

The plugin isolates its properties, so names like 'portals' belong to
this storage type alone, and a third-party plugin that declares one,
like 'subsysnqn', is not left out. As the first in-tree plugin that
isolates, it makes the storage API check the parameters for adding and
updating a storage of any type against the options of that type alone.
A third-party plugin then has to list the sensitive properties it uses
in options(), as the API change log says.

Local disk setup refuses namespaces attached over a fabric, as for
iSCSI LUNs. Wiping a namespace or creating a volume group on it is
refused while a storage serves images from its subsystem. nvme-cli
is only recommended, as no other storage type needs it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 ApiChangeLog                       |   7 +-
 debian/control                     |   1 +
 src/PVE/Diskmanage.pm              |   2 +
 src/PVE/Makefile                   |   1 +
 src/PVE/NVMeOF.pm                  | 244 +++++++++++++++++++++++
 src/PVE/Storage.pm                 |   2 +
 src/PVE/Storage/Makefile           |   1 +
 src/PVE/Storage/NVMeOFPlugin.pm    | 223 +++++++++++++++++++++
 src/PVE/Storage/Plugin.pm          |   2 +-
 src/PVE/Storage/SAN.pm             |  13 ++
 src/test/Makefile                  |   6 +-
 src/test/run_nvmeof_taint_tests.pl |  60 ++++++
 src/test/run_nvmeof_tests.pl       | 302 +++++++++++++++++++++++++++++
 src/test/run_san_nvme_tests.pl     |  16 ++
 14 files changed, 875 insertions(+), 5 deletions(-)
 create mode 100644 src/PVE/NVMeOF.pm
 create mode 100644 src/PVE/Storage/NVMeOFPlugin.pm
 create mode 100755 src/test/run_nvmeof_taint_tests.pl
 create mode 100755 src/test/run_nvmeof_tests.pl

diff --git a/ApiChangeLog b/ApiChangeLog
index 10f39508..d7310fe4 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -101,9 +101,10 @@ Future changes should be documented in here.
   declares a property of that name is then not loaded until it renames it, so prefer names that
   are specific to the plugin, for example prefixed with its type.
 
-  As soon as one such plugin is installed, the API checks the parameters for adding and updating a
-  storage against the options of its type alone, for all storage types. So every plugin has to
-  list the sensitive properties that it uses in `options()`.
+  Since the in-tree 'nvmeof' plugin isolates its properties, the API checks the parameters for
+  adding and updating a storage against the options of its type alone, for all storage types,
+  also for plugins of older API versions. So every plugin has to list the sensitive properties
+  that it uses in `options()`.
   Set 'sensitive-properties' in such a plugin, even if empty. Without it, the fallback list of
   version 11 also applies to the plugin's own properties of these names.
 
diff --git a/debian/control b/debian/control
index d10dfb23..8b9b6455 100644
--- a/debian/control
+++ b/debian/control
@@ -55,6 +55,7 @@ Depends: bzip2,
          ${misc:Depends},
          ${perl:Depends},
 Recommends: multipath-tools,
+            nvme-cli,
             pve-esxi-import-tools (>= 0.6.0),
             skopeo,
             zfsutils-linux (>= 2.4.1-pve2),
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index 8fda7afc..1ad91e06 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -108,6 +108,8 @@ sub disk_is_used {
     my $disklist = get_disks($dev, 1, 1);
 
     die "'$disk' is not a valid local disk\n" if !defined($disklist->{$dev});
+    die "'$disk' is not a valid local disk\n"
+        if ($disklist->{$dev}->{transport} // '') =~ /^nvme-(?:tcp|rdma|fc)$/;
     return 1 if $disklist->{$dev}->{used};
 
     return 0;
diff --git a/src/PVE/Makefile b/src/PVE/Makefile
index 13fecbf8..238eefd4 100644
--- a/src/PVE/Makefile
+++ b/src/PVE/Makefile
@@ -4,6 +4,7 @@
 install:
 	install -D -m 0644 Storage.pm ${DESTDIR}${PERLDIR}/PVE/Storage.pm
 	install -D -m 0644 Diskmanage.pm ${DESTDIR}${PERLDIR}/PVE/Diskmanage.pm
+	install -D -m 0644 NVMeOF.pm ${DESTDIR}${PERLDIR}/PVE/NVMeOF.pm
 	install -D -m 0644 ISCSI/Topology.pm ${DESTDIR}${PERLDIR}/PVE/ISCSI/Topology.pm
 	install -D -m 0644 Multipath.pm ${DESTDIR}${PERLDIR}/PVE/Multipath.pm
 	install -D -m 0644 Multipath/Config.pm ${DESTDIR}${PERLDIR}/PVE/Multipath/Config.pm
diff --git a/src/PVE/NVMeOF.pm b/src/PVE/NVMeOF.pm
new file mode 100644
index 00000000..ebaedf9b
--- /dev/null
+++ b/src/PVE/NVMeOF.pm
@@ -0,0 +1,244 @@
+package PVE::NVMeOF;
+
+use v5.36;
+
+use Digest::SHA qw(sha1_hex);
+use Fcntl qw(:flock F_GETFD F_SETFD FD_CLOEXEC);
+use File::Path qw(make_path);
+use Socket qw(AF_INET AF_INET6 inet_ntop inet_pton);
+
+use PVE::JSONSchema;
+use PVE::Tools qw(dir_glob_foreach file_read_firstline file_set_contents run_command);
+
+use PVE::Diskmanage;
+
+our $NVME = '/usr/sbin/nvme';
+our $SYSFS = '/sys';
+our $HOSTNQN_FILE = '/etc/nvme/hostnqn';
+our $HOSTID_FILE = '/etc/nvme/hostid';
+our $RUN_DIR = '/run/pve/nvmeof';
+our $NQN_PATTERN = 'nqn\.[^\s,=]+';
+
+sub canonical_address($addr) {
+    if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+        return inet_ntop(AF_INET, $packed);
+    }
+    if (defined(my $packed = inet_pton(AF_INET6, $addr))) {
+        return inet_ntop(AF_INET, substr($packed, 12))
+            if substr($packed, 0, 12) eq ("\0" x 10) . "\xff\xff";
+        return inet_ntop(AF_INET6, $packed);
+    }
+    return lc($addr);
+}
+
+# Link-local IPv6 needs a node-specific interface and cannot identify a shared configured portal.
+sub parse_portal($portal) {
+    my ($addr, $port);
+    if ($portal =~ m/^\[([0-9a-fA-F:.]+)\](?::([0-9]+))?$/) {
+        ($addr, $port) = ($1, $2);
+    } elsif ($portal =~ m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)(?::([0-9]+))?$/) {
+        ($addr, $port) = ($1, $2);
+    } else {
+        return undef;
+    }
+    return undef if defined($port) && ($port < 1 || $port > 65535);
+    $addr = canonical_address($addr);
+    if (defined(my $packed = inet_pton(AF_INET, $addr))) {
+        my $multicast = (unpack('C', $packed) & 0xf0) == 0xe0;
+        return undef if $addr eq '0.0.0.0' || $addr eq '255.255.255.255' || $multicast;
+    } elsif (defined($packed = inet_pton(AF_INET6, $addr))) {
+        my $prefix = unpack('n', $packed);
+        return undef
+            if $addr eq '::' || ($prefix & 0xffc0) == 0xfe80 || ($prefix & 0xff00) == 0xff00;
+    } else {
+        return undef;
+    }
+    return { traddr => $addr, trsvcid => int($port // 4420) };
+}
+
+PVE::JSONSchema::register_format(
+    'pve-storage-nvme-portal',
+    sub($portal, $noerr = undef) {
+        return $portal if parse_portal($portal);
+        return undef if $noerr;
+        die "value is not an IPv4 address or a bracketed IPv6 address with optional port\n";
+    },
+);
+
+sub format_path($path) {
+    my $addr = $path->{traddr} =~ /:/ ? "[$path->{traddr}]" : $path->{traddr};
+    return defined($path->{trsvcid}) ? "$addr:$path->{trsvcid}" : $addr;
+}
+
+sub storage_paths($scfg) {
+    return [map { parse_portal($_) // die "invalid portal '$_'\n" }
+        PVE::Tools::split_list($scfg->{portals})];
+}
+
+sub host_identity() {
+    my ($nqn) = (file_read_firstline($HOSTNQN_FILE) // '') =~ /\A($NQN_PATTERN)\z/;
+    my ($id) = (file_read_firstline($HOSTID_FILE) // '') =~
+        /\A([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\z/i;
+    if (!defined($nqn) || length($nqn) > 223) {
+        die "invalid or missing NVMe host NQN in '$HOSTNQN_FILE' - install nvme-cli and check the"
+            . " host identity\n";
+    }
+    die "invalid or missing NVMe host ID in '$HOSTID_FILE' - expected a nonzero UUID\n"
+        if !defined($id) || $id =~ /^[0-]+$/;
+    return { nqn => $nqn, id => lc($id) };
+}
+
+sub is_supported() { return -x $NVME ? 1 : 0 }
+
+sub assert_support() {
+    die "nvme-cli is not installed\n" if !is_supported();
+    run_command(['modprobe', 'nvme-tcp'], timeout => 10)
+        if !-e "$SYSFS/module/nvme_tcp";
+    my $multipath = file_read_firstline("$SYSFS/module/nvme_core/parameters/multipath") // '';
+    die "native NVMe multipath is disabled - enable nvme_core.multipath\n" if $multipath ne 'Y';
+}
+
+sub controllers() {
+    my $res = {};
+    my $dir = "$SYSFS/class/nvme-fabrics/ctl";
+    dir_glob_foreach(
+        $dir,
+        'nvme[0-9]+',
+        sub($name, @) {
+            my $address = file_read_firstline("$dir/$name/address") // return;
+            my %address = map { split(/=/, $_, 2) } grep { /=/ } split(/,/, $address);
+            return if !defined($address{traddr});
+            $res->{$name} = {
+                transport => file_read_firstline("$dir/$name/transport") // '',
+                subsystem => PVE::Diskmanage::read_subsysnqn("$dir/$name/subsysnqn") // '',
+                state => file_read_firstline("$dir/$name/state") // 'unknown',
+                traddr => canonical_address($address{traddr}),
+                trsvcid => $address{trsvcid},
+            };
+        },
+    );
+    return $res;
+}
+
+sub subsystem_controllers($subsystem) {
+    my $controllers = controllers();
+    return {
+        map { $_ => $controllers->{$_} } grep {
+            $controllers->{$_}->{subsystem} eq $subsystem
+                && $controllers->{$_}->{transport} eq 'tcp'
+        } keys %$controllers
+    };
+}
+
+sub path_controller($controllers, $path) {
+    for my $name (sort keys %$controllers) {
+        my $ctrl = $controllers->{$name};
+        if ($ctrl->{traddr} eq $path->{traddr} && ($ctrl->{trsvcid} // '') eq $path->{trsvcid}) {
+            return $name;
+        }
+    }
+    return undef;
+}
+
+sub namespaces($subsystem) {
+    my $res = {};
+    my $dir = "$SYSFS/class/nvme-subsystem";
+    dir_glob_foreach(
+        $dir,
+        'nvme-subsys[0-9]+',
+        sub($subsys, @) {
+            my $nqn = PVE::Diskmanage::read_subsysnqn("$dir/$subsys/subsysnqn") // return;
+            return if $nqn ne $subsystem;
+            dir_glob_foreach(
+                "$dir/$subsys",
+                'nvme[0-9]+n[0-9]+',
+                sub($dev, @) {
+                    my $wwid = file_read_firstline("$SYSFS/block/$dev/wwid") // return;
+                    my $size = file_read_firstline("$SYSFS/block/$dev/size") // return;
+                    $res->{$dev} = { wwid => $wwid, size => int($size) * 512 };
+                },
+            );
+        },
+    );
+    return $res;
+}
+
+sub with_lock($code) {
+    my ($res, $timeout) = PVE::Tools::run_fork_with_timeout(
+        35,
+        sub {
+            make_path($RUN_DIR);
+            open(my $lock, '>>', "$RUN_DIR/fabrics.lock") or die "cannot open NVMe lock - $!\n";
+            flock($lock, LOCK_EX | LOCK_NB)
+                or die "NVMe over Fabrics is busy - another operation is still running\n";
+            # A command still in the kernel must retain serialization after its caller times out.
+            my $flags = fcntl($lock, F_GETFD, 0) // die "cannot read NVMe lock flags - $!\n";
+            fcntl($lock, F_SETFD, $flags & ~FD_CLOEXEC)
+                or die "cannot inherit NVMe lock - $!\n";
+            return $code->();
+        },
+    );
+    die "NVMe over Fabrics operation timed out - it may still be running\n" if $timeout;
+    return $res;
+}
+
+my sub retry_file($scfg, $path) {
+    my ($key) =
+        sha1_hex(join(
+            "\0", $scfg->{subsysnqn}, format_path($path), $scfg->{'fast-io-fail-tmo'} // 120)) =~
+        /\A([0-9a-f]{40})\z/;
+    return "$RUN_DIR/retry/$key";
+}
+
+sub in_cooldown($scfg, $path) {
+    my $mtime = (stat(retry_file($scfg, $path)))[9] // return 0;
+    my $age = time() - $mtime;
+    return $age > -60 && $age < 60;
+}
+
+sub note_connect_failure($scfg, $path) {
+    make_path("$RUN_DIR/retry");
+    file_set_contents(retry_file($scfg, $path), '');
+}
+
+sub connect_path($scfg, $path) {
+    my ($nqn) = $scfg->{subsysnqn} =~ /\A($NQN_PATTERN)\z/;
+    if (!defined($nqn) || length($nqn) > 223 || $nqn eq 'nqn.2014-08.org.nvmexpress.discovery') {
+        die "invalid subsystem NQN\n";
+    }
+    my ($fail) = ($scfg->{'fast-io-fail-tmo'} // 120) =~ /\A(-?[0-9]+)\z/;
+    die "invalid fast I/O fail timeout\n" if !defined($fail) || $fail < -1;
+    return with_lock(sub {
+        return if path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+        assert_support();
+        my $host = host_identity();
+        run_command(
+            [
+                $NVME,
+                'connect',
+                '--transport=tcp',
+                '--nqn',
+                $nqn,
+                '--traddr',
+                $path->{traddr},
+                '--trsvcid',
+                $path->{trsvcid},
+                '--hostnqn',
+                $host->{nqn},
+                '--hostid',
+                $host->{id},
+                '--ctrl-loss-tmo=-1',
+                '--fast_io_fail_tmo',
+                $fail,
+            ],
+            timeout => 30,
+        );
+        # nvme-cli can exit successfully without having connected
+        die "no controller appeared\n"
+            if !path_controller(subsystem_controllers($scfg->{subsysnqn}), $path);
+        unlink(retry_file($scfg, $path));
+        return;
+    });
+}
+
+1;
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 1e9c670a..91283ff2 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -33,6 +33,7 @@ use PVE::Storage::LvmThinPlugin;
 use PVE::Storage::NFSPlugin;
 use PVE::Storage::CIFSPlugin;
 use PVE::Storage::ISCSIPlugin;
+use PVE::Storage::NVMeOFPlugin;
 use PVE::Storage::RBDPlugin;
 use PVE::Storage::CephFSPlugin;
 use PVE::Storage::ISCSIDirectPlugin;
@@ -58,6 +59,7 @@ PVE::Storage::LvmThinPlugin->register();
 PVE::Storage::NFSPlugin->register();
 PVE::Storage::CIFSPlugin->register();
 PVE::Storage::ISCSIPlugin->register();
+PVE::Storage::NVMeOFPlugin->register();
 PVE::Storage::RBDPlugin->register();
 PVE::Storage::CephFSPlugin->register();
 PVE::Storage::ISCSIDirectPlugin->register();
diff --git a/src/PVE/Storage/Makefile b/src/PVE/Storage/Makefile
index a2df83c0..7f2ff6c8 100644
--- a/src/PVE/Storage/Makefile
+++ b/src/PVE/Storage/Makefile
@@ -6,6 +6,7 @@ SOURCES= \
 	NFSPlugin.pm \
 	CIFSPlugin.pm \
 	ISCSIPlugin.pm \
+	NVMeOFPlugin.pm \
 	CephFSPlugin.pm \
 	RBDPlugin.pm \
 	ISCSIDirectPlugin.pm \
diff --git a/src/PVE/Storage/NVMeOFPlugin.pm b/src/PVE/Storage/NVMeOFPlugin.pm
new file mode 100644
index 00000000..cb514c1d
--- /dev/null
+++ b/src/PVE/Storage/NVMeOFPlugin.pm
@@ -0,0 +1,223 @@
+package PVE::Storage::NVMeOFPlugin;
+
+use v5.36;
+
+use Cwd ();
+use Time::HiRes ();
+
+use PVE::Network;
+
+use PVE::Diskmanage;
+use PVE::NVMeOF;
+use PVE::Storage::Plugin;
+
+use base qw(PVE::Storage::Plugin);
+
+sub type { return 'nvmeof' }
+
+sub plugindata {
+    return {
+        content => [{ images => 1, 'vm-vol' => 1, none => 1 }, { images => 1 }],
+        select_existing => 1,
+        'isolate-properties' => 1,
+        'advanced-properties' => {
+            'fast-io-fail-tmo' => 1,
+        },
+        'hidden-properties' => {
+            bwlimit => 1,
+        },
+        'sensitive-properties' => {},
+    };
+}
+
+sub properties {
+    return {
+        subsysnqn => {
+            title => "Subsystem",
+            description => 'The NQN of the NVMe subsystem.',
+            type => 'string',
+            pattern => $PVE::NVMeOF::NQN_PATTERN,
+            maxLength => 223,
+        },
+        portals => {
+            title => "Portals",
+            description =>
+                'TCP portals of the NVMe subsystem, as IPv4 or bracketed IPv6 addresses'
+                . ' with an optional port (default 4420).',
+            type => 'string',
+            format => 'pve-storage-nvme-portal-list',
+        },
+        'fast-io-fail-tmo' => {
+            title => "Fail I/O After (s)",
+            description => 'Seconds before I/O through a reconnecting path fails. -1 disables'
+                . ' this timeout; I/O can remain queued while paths reconnect.',
+            type => 'integer',
+            minimum => -1,
+            default => 120,
+        },
+    };
+}
+
+sub options {
+    return {
+        subsysnqn => { fixed => 1 },
+        portals => {},
+        'fast-io-fail-tmo' => { optional => 1 },
+        nodes => { optional => 1 },
+        disable => { optional => 1 },
+        content => { optional => 1 },
+        bwlimit => { optional => 1 },
+    };
+}
+
+my sub assert_valid_config($scfg) {
+    die "the discovery subsystem cannot be used as storage\n"
+        if $scfg->{subsysnqn} eq 'nqn.2014-08.org.nvmexpress.discovery';
+    my %seen;
+    for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+        my $portal = PVE::NVMeOF::format_path($path);
+        die "portal '$portal' is listed more than once\n" if $seen{$portal}++;
+    }
+}
+
+sub on_add_hook($class, $storeid, $scfg, %param) {
+    assert_valid_config($scfg);
+    # the LUN and controller lists and the node status attribute a subsystem to its one storage
+    my $ids = PVE::Storage::config()->{ids};
+    for my $sid (sort keys %$ids) {
+        next if $sid eq $storeid || $ids->{$sid}->{type} ne 'nvmeof';
+        die "storage '$sid' already uses this subsystem\n"
+            if $ids->{$sid}->{subsysnqn} eq $scfg->{subsysnqn};
+    }
+    return;
+}
+
+sub on_update_hook_full($class, $storeid, $scfg, $update, $delete, $sensitive, @) {
+    my $merged = { %$scfg, %$update };
+    delete $merged->{$_} for ($delete // [])->@*;
+    assert_valid_config($merged);
+    return $class->SUPER::on_update_hook_full($storeid, $scfg, $update, $delete, $sensitive);
+}
+
+sub parse_volname($class, $volname) {
+    if ($volname =~ m!^(nvme-[a-zA-Z0-9_.:+-]+)$! && $volname !~ /-part[0-9]+$/) {
+        return ($class->guest_volume_vtype('raw'), $1, undef, undef, undef, undef, 'raw');
+    }
+    die "unable to parse NVMe over Fabrics volume name '$volname'\n";
+}
+
+# The name of a volume can be the by-id name of any NVMe device, also of a local disk or of a
+# namespace of another subsystem.
+sub assert_namespace($scfg, $volname, $real) {
+    my ($dev) = $real =~ m!^/dev/(nvme[0-9]+n[0-9]+)$!;
+    die "volume '$volname' is not an NVMe namespace\n" if !defined($dev);
+    my $nqn = PVE::Diskmanage::read_subsysnqn("/sys/block/$dev/device/subsysnqn") // '';
+    die "volume '$volname' is not part of subsystem '$scfg->{subsysnqn}'\n"
+        if $nqn ne $scfg->{subsysnqn};
+}
+
+sub filesystem_path($class, $scfg, $volname, $snapname = undef, @) {
+    die "snapshot is not possible on NVMe over Fabrics storage\n" if defined($snapname);
+    my ($vtype, $name, $vmid) = $class->parse_volname($volname);
+    my $path = "/dev/disk/by-id/$name";
+    # not every user of a path activates the volume first, and the base volume of an LVM storage
+    # has no device before its storage connects
+    my $real = Cwd::realpath($path);
+    assert_namespace($scfg, $volname, $real) if defined($real) && -e $real;
+    return wantarray ? ($path, $vmid, $vtype) : $path;
+}
+
+sub create_base { die "can't create base images in NVMe over Fabrics storage\n" }
+sub clone_image { die "can't clone images in NVMe over Fabrics storage\n" }
+sub alloc_image { die "can't allocate space in NVMe over Fabrics storage\n" }
+sub free_image { die "can't free space in NVMe over Fabrics storage\n" }
+sub volume_resize { die "volume resize is not possible on NVMe over Fabrics storage\n" }
+
+sub list_volumes($class, $storeid, $scfg, $vmid = undef, $content_types = undef, @) {
+    my $res = $class->list_images($storeid, $scfg, $vmid);
+    $_->{content} = 'images' for @$res;
+    return $res;
+}
+
+sub list_images($class, $storeid, $scfg, $vmid = undef, $vollist = undef, $cache = undef, @) {
+    return [] if defined($vmid) && !$vollist;
+    my $namespaces = PVE::NVMeOF::namespaces($scfg->{subsysnqn});
+    my $res = [];
+    for my $dev (sort keys %$namespaces) {
+        my $ns = $namespaces->{$dev};
+        my $volid = "$storeid:nvme-$ns->{wwid}";
+        next if $vollist && !grep { $_ eq $volid } @$vollist;
+        push @$res,
+            {
+                volid => $volid,
+                format => 'raw',
+                size => $ns->{size},
+                vmid => 0,
+            };
+    }
+    return $res;
+}
+
+sub status($class, $storeid, $scfg, $cache = undef, @) {
+    my $active = grep { $_->{state} eq 'live' }
+        values PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+    return (0, 0, 0, $active ? 1 : 0);
+}
+
+sub activate_storage($class, $storeid, $scfg, $cache = undef, @) {
+    my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+        next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+        next if PVE::NVMeOF::in_cooldown($scfg, $path);
+        eval {
+            die "no answer from portal\n"
+                if !PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+            PVE::NVMeOF::connect_path($scfg, $path);
+        };
+        if (my $err = $@) {
+            PVE::NVMeOF::note_connect_failure($scfg, $path);
+            warn "storage '$storeid': connecting to '"
+                . PVE::NVMeOF::format_path($path)
+                . "' failed - $err";
+        }
+    }
+    return;
+}
+
+sub deactivate_storage { return }
+
+sub check_connection($class, $storeid, $scfg, @) {
+    my $controllers = PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn});
+    return 1 if grep { $_->{state} eq 'live' } values %$controllers;
+    for my $path (PVE::NVMeOF::storage_paths($scfg)->@*) {
+        next if defined(PVE::NVMeOF::path_controller($controllers, $path));
+        next if PVE::NVMeOF::in_cooldown($scfg, $path);
+        return 1 if PVE::Network::tcp_ping($path->{traddr}, $path->{trsvcid}, 2);
+        PVE::NVMeOF::note_connect_failure($scfg, $path);
+    }
+    return 0;
+}
+
+sub activate_volume($class, $storeid, $scfg, $volname, $snapname = undef, $cache = undef, @) {
+    my $path = $class->filesystem_path($scfg, $volname, $snapname);
+    # as for the multipath map of an iSCSI LUN, a status pass does not wait for the device
+    my $deadline = Time::HiRes::time() + ($cache && $cache->{status_only} ? 0 : 10);
+    while (!-b $path && Time::HiRes::time() < $deadline) {
+        last if !keys PVE::NVMeOF::subsystem_controllers($scfg->{subsysnqn})->%*;
+        Time::HiRes::usleep(250_000);
+    }
+    my $real = Cwd::realpath($path);
+    die "volume '$volname' is not available on this node\n" if !$real || !-b $real;
+    assert_namespace($scfg, $volname, $real);
+}
+
+sub volume_has_feature(
+    $class, $scfg, $feature, $storeid, $volname,
+    $snapname = undef,
+    $running = undef, @,
+) {
+    return 1 if $feature eq 'copy' && !defined($snapname);
+    return undef;
+}
+
+1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 724c2d35..6e305580 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -42,7 +42,7 @@ our @COMMON_TAR_FLAGS = qw(
 );
 
 our @SHARED_STORAGE = (
-    'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs',
+    'iscsi', 'nfs', 'cifs', 'rbd', 'cephfs', 'iscsidirect', 'zfs', 'drbd', 'pbs', 'nvmeof',
 );
 
 our $QCOW2_PREALLOCATION = {
diff --git a/src/PVE/Storage/SAN.pm b/src/PVE/Storage/SAN.pm
index b025c03f..86469f5c 100644
--- a/src/PVE/Storage/SAN.pm
+++ b/src/PVE/Storage/SAN.pm
@@ -298,6 +298,11 @@ sub lun_partitions($lun) {
 }
 
 sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
+    my $name = basename($lun->{devpath});
+    my $nqn =
+        $name =~ /^nvme\d+n\d+$/
+        ? PVE::Diskmanage::read_subsysnqn("/sys/block/$name/device/subsysnqn")
+        : undef;
     my %vgs = map { $_ => 1 } PVE::Storage::LVMPlugin::lun_volume_groups(
         [$lun->{devpath}, lun_partitions($lun)->@*],
     )->@*;
@@ -320,6 +325,14 @@ sub assert_lun_unreferenced($cfg, $lun, $exempt = undef) {
         ) {
             die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this target\n";
         }
+        if (
+            $type eq 'nvmeof'
+            && defined($nqn)
+            && $scfg->{subsysnqn} eq $nqn
+            && PVE::Storage::san_storage_serves_images($scfg)
+        ) {
+            die "cannot use '$lun->{devpath}' - storage '$sid' serves images from this subsystem\n";
+        }
         next if $type ne 'lvm' || !$scfg->{base};
         my (undef, $volname) = PVE::Storage::Plugin::parse_volume_id($scfg->{base});
         my ($wwid) = $volname =~ m/(?:^|\.)(?:dm-uuid-(?:part\d+-)?mpath|scsi|nvme)-(.+)$/;
diff --git a/src/test/Makefile b/src/test/Makefile
index 279ba1ee..c88666f5 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -1,6 +1,6 @@
 all: test
 
-test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api
+test: test_iscsi test_zfspoolplugin test_lvmplugin test_lvm_create test_lvm_lock_retry test_lvm_base_volume test_lvm_hints test_disklist test_bwlimit test_plugin test_ovf test_volume_access test_remote_command test_sensitive_properties test_import_stream test_multipath test_san_libraries test_lun_identity test_lvm_base_activation test_multipath_api test_san_taint test_san_lun_scan test_san_api test_disk_api test_nvmeof
 
 test_zfspoolplugin: run_test_zfspoolplugin.pl
 	./run_test_zfspoolplugin.pl
@@ -83,3 +83,7 @@ test_san_api: run_san_api_tests.pl run_san_wipe_tests.pl run_san_nvme_tests.pl
 
 test_disk_api: run_disk_api_tests.pl
 	./run_disk_api_tests.pl
+
+test_nvmeof: run_nvmeof_tests.pl run_nvmeof_taint_tests.pl
+	./run_nvmeof_tests.pl
+	perl -T ./run_nvmeof_taint_tests.pl
diff --git a/src/test/run_nvmeof_taint_tests.pl b/src/test/run_nvmeof_taint_tests.pl
new file mode 100755
index 00000000..62405291
--- /dev/null
+++ b/src/test/run_nvmeof_taint_tests.pl
@@ -0,0 +1,60 @@
+#!/usr/bin/perl -T
+
+use v5.36;
+use lib qw(..);
+use File::Temp qw(tempdir);
+use Scalar::Util qw(tainted);
+use Test::More;
+use Test::MockModule;
+use PVE::NVMeOF;
+use PVE::Tools qw(file_set_contents file_get_contents);
+
+$ENV{PATH} = '/usr/sbin:/usr/bin:/sbin:/bin';
+delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
+my $dir = tempdir('nvmeof-taint-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::HOSTNQN_FILE = "$dir/hostnqn";
+local $PVE::NVMeOF::HOSTID_FILE = "$dir/hostid";
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+file_set_contents($PVE::NVMeOF::HOSTNQN_FILE, "nqn.host\n");
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "12345678-1234-1234-1234-123456789abc\n");
+file_set_contents("$dir/config", "nqn.test\n192.0.2.1\n120\n");
+my ($nqn, $portal, $fail) = split(/\n/, file_get_contents("$dir/config"));
+ok(tainted($nqn), 'configuration input is tainted');
+my $nvme = Test::MockModule->new('PVE::NVMeOF');
+$nvme->mock(with_lock => sub { $_[0]->() });
+my $commands = 0;
+$nvme->mock(
+    subsystem_controllers => sub {
+        return $commands ? { nvme0 => { traddr => '192.0.2.1', trsvcid => 4420 } } : {};
+    },
+);
+$nvme->mock(assert_support => sub { });
+$nvme->mock(
+    run_command => sub($cmd, %opts) {
+        $commands++;
+        is(
+            scalar(grep { tainted($_) } @$cmd),
+            0,
+            'all connect operands are validated and untainted',
+        );
+        PVE::Tools::run_command(['/usr/bin/true', @$cmd], timeout => 5);
+    },
+);
+my $path = PVE::NVMeOF::parse_portal($portal);
+eval { PVE::NVMeOF::connect_path({ subsysnqn => $nqn, 'fast-io-fail-tmo' => $fail }, $path) };
+is($@, '', 'connect command can execute in taint mode');
+is($commands, 1, 'one connect');
+PVE::NVMeOF::note_connect_failure({ subsysnqn => $nqn }, $path);
+ok(
+    PVE::NVMeOF::in_cooldown({ subsysnqn => $nqn }, $path),
+    'cooldown accepts validated file-backed configuration',
+);
+for my $bad ('nqn.bad,nqn=other', "nqn.test\n", 'nqn.2014-08.org.nvmexpress.discovery') {
+    eval { PVE::NVMeOF::connect_path({ subsysnqn => $bad }, $path) };
+    like($@, qr/invalid subsystem/, 'invalid connect subsystem refused');
+}
+file_set_contents($PVE::NVMeOF::HOSTID_FILE, "00000000-0000-0000-0000-000000000000\n");
+eval { PVE::NVMeOF::host_identity() };
+like($@, qr/nonzero UUID/, 'zero host ID refused');
+
+done_testing();
diff --git a/src/test/run_nvmeof_tests.pl b/src/test/run_nvmeof_tests.pl
new file mode 100755
index 00000000..b75f7a5a
--- /dev/null
+++ b/src/test/run_nvmeof_tests.pl
@@ -0,0 +1,302 @@
+#!/usr/bin/perl
+
+use v5.36;
+use lib qw(..);
+use Fcntl qw(:flock);
+use File::Path qw(make_path);
+use File::Temp qw(tempdir);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Tools qw(file_set_contents);
+
+use PVE::NVMeOF;
+use PVE::Storage;
+
+my $dir = tempdir('nvmeof-test-XXXX', DIR => '.', CLEANUP => 1);
+local $PVE::NVMeOF::RUN_DIR = "$dir/run";
+local $PVE::NVMeOF::SYSFS = "$dir/sys";
+my $class = 'PVE::Storage::NVMeOFPlugin';
+my $cfg = {
+    type => 'nvmeof',
+    subsysnqn => 'nqn.test',
+    portals => '192.0.2.1,[2001:db8::2]:4421',
+    content => { none => 1 },
+};
+
+subtest 'configuration, portals and volumes' => sub {
+    is(PVE::Storage::Plugin->lookup('nvmeof'), $class, 'plugin registered');
+    is_deeply(
+        [sort keys $class->properties()->%*],
+        [qw(fast-io-fail-tmo portals subsysnqn)],
+        'only three plugin-specific properties',
+    );
+    my sub nvmeof_schema($schema) {
+        my @types = map { ($_->{oneOf} // [])->@* } ($schema->{allOf} // [])->@*;
+        my ($nvmeof) = grep { $_->{'instance-type'} eq 'nvmeof' } @types;
+        return $nvmeof->{properties} // {};
+    }
+    my $create = nvmeof_schema(PVE::Storage::Plugin->createSchema());
+    is_deeply(
+        [sort grep { !$create->{$_}->{optional} } keys $create->%*],
+        [qw(portals subsysnqn)],
+        'adding a storage requires the subsystem and the portals',
+    );
+    my $update = nvmeof_schema(PVE::Storage::Plugin->updateSchema());
+    ok($update->{portals} && !$update->{subsysnqn}, 'the subsystem cannot be changed');
+    my $parsed = PVE::Storage::Plugin->parse_config(
+        'storage.cfg',
+        "nvmeof: fabric\n\tsubsysnqn nqn.test\n\tportals 192.0.2.1\n\tcontent none\n",
+    );
+    is($parsed->{ids}->{fabric}->{subsysnqn}, 'nqn.test', 'storage.cfg accepts the plugin');
+    is_deeply(
+        $parsed->{ids}->{fabric}->{content},
+        { none => 1 },
+        'connection-only content parses',
+    );
+    is_deeply(
+        PVE::NVMeOF::parse_portal('192.0.2.1'),
+        { traddr => '192.0.2.1', trsvcid => 4420 },
+        'default I/O port',
+    );
+    is_deeply(
+        PVE::NVMeOF::parse_portal('[2001:0db8::1]:4421'),
+        { traddr => '2001:db8::1', trsvcid => 4421 },
+        'canonical IPv6',
+    );
+
+    for my $bad (
+        'host.example', '1.2.3.4:0', '1.2.3.4:65536', '[fe80::1]', '[::]', '224.0.0.1',
+    ) {
+        ok(!PVE::NVMeOF::parse_portal($bad), "invalid portal $bad");
+    }
+    eval { $class->on_add_hook('fabric', { %$cfg, portals => '192.0.2.1,192.0.2.1:4420' }); };
+    like($@, qr/listed more than once/, 'duplicate portals refused');
+    eval {
+        $class->on_add_hook(
+            'fabric',
+            { %$cfg, subsysnqn => 'nqn.2014-08.org.nvmexpress.discovery' },
+        );
+    };
+    like($@, qr/discovery subsystem/, 'discovery subsystem refused');
+    my $storage = Test::MockModule->new('PVE::Storage');
+    my $ids = { fabric => $cfg, local => { type => 'dir' } };
+    $storage->mock(config => sub { { ids => $ids } });
+    is(eval { $class->on_add_hook('fabric', $cfg); 1 }, 1, 'a storage is not in its own way');
+    eval { $class->on_add_hook('second', {%$cfg}) };
+    like($@, qr/storage 'fabric' already uses this subsystem/, 'one storage per subsystem');
+    is(
+        scalar($class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef')),
+        '/dev/disk/by-id/nvme-eui.0123456789abcdef',
+        'stable volume path',
+    );
+    {
+        my $disk = Test::MockModule->new('PVE::Diskmanage');
+        my $nqn = 'nqn.other';
+        $disk->redefine(read_subsysnqn => sub { $nqn });
+        my $check = sub {
+            eval { PVE::Storage::NVMeOFPlugin::assert_namespace($cfg, @_) };
+            $@;
+        };
+        like($check->('nvme-x', '/dev/nvme0n1'), qr/not part of subsystem/, 'other subsystem');
+        like($check->('nvme-x', '/dev/sda'), qr/not an NVMe namespace/, 'other device type');
+        like($check->('nvme-x', '/dev/nvme0n1p1'), qr/not an NVMe namespace/, 'partition');
+        $nqn = 'nqn.test';
+        is($check->('nvme-x', '/dev/nvme0n1'), '', 'namespace of the subsystem');
+        my $cwd = Test::MockModule->new('Cwd');
+        $cwd->redefine(realpath => sub { $dir });
+        eval { $class->filesystem_path($cfg, 'nvme-eui.0123456789abcdef') };
+        like($@, qr/not an NVMe namespace/, 'the path of an existing device is checked');
+    }
+
+    for my $bad ('nvme-../../sda', 'nvme-eui.abc-part1', 'sda') {
+        eval { $class->parse_volname($bad) };
+        like($@, qr/unable to parse/, "invalid volume $bad");
+    }
+    my $nvme = Test::MockModule->new('PVE::NVMeOF');
+    $nvme->mock(namespaces => sub { { nvme0n1 => { wwid => 'eui.abc', size => 4096 } } });
+    is(
+        $class->list_volumes('fabric', $cfg)->[0]->{volid},
+        'fabric:nvme-eui.abc',
+        'content none still offers bases',
+    );
+    is_deeply($class->list_images('fabric', $cfg, 100), [], 'namespaces have no VM owner');
+    $nvme->mock(subsystem_controllers => sub { {} });
+    eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test') };
+    like($@, qr/not available/,
+        'missing namespace fails without waiting on absent controllers');
+    my ($now, $queries) = (0, 0);
+    my $clock = Test::MockModule->new('Time::HiRes');
+    $clock->redefine(time => sub { $now += 5 }, usleep => sub { });
+    $nvme->mock(subsystem_controllers => sub { $queries++; { nvme1 => {} } });
+    eval { $class->activate_volume('fabric', $cfg, 'nvme-nonexistent-test', undef, {}) };
+    is($queries, 1, 'a missing namespace of a connected subsystem is waited for');
+    eval {
+        $class->activate_volume(
+            'fabric',
+            $cfg,
+            'nvme-nonexistent-test',
+            undef,
+            { status_only => 1 },
+        );
+    };
+    like($@, qr/not available/, 'in a status pass it fails as well');
+    is($queries, 1, 'but without a wait');
+    is(
+        ($class->parse_volname('nvme-eui.0123456789abcdef'))[0],
+        'vm-vol',
+        'namespaces are VM disks',
+    );
+};
+
+subtest 'sysfs topology' => sub {
+    my $sys = $PVE::NVMeOF::SYSFS;
+    make_path(
+        "$sys/class/nvme-fabrics/ctl/nvme1",
+        "$sys/class/nvme-subsystem/nvme-subsys0/nvme0n1",
+        "$sys/block/nvme0n1",
+    );
+    my %attrs = (
+        address => 'traddr=192.0.2.1,trsvcid=4420',
+        transport => 'tcp',
+        subsysnqn => 'nqn.test',
+        state => 'connecting',
+    );
+    file_set_contents("$sys/class/nvme-fabrics/ctl/nvme1/$_", "$attrs{$_}\n") for keys %attrs;
+    file_set_contents("$sys/class/nvme-subsystem/nvme-subsys0/subsysnqn", "nqn.test\n");
+    file_set_contents("$sys/block/nvme0n1/wwid", "eui.abc\n");
+    file_set_contents("$sys/block/nvme0n1/size", "8\n");
+    is(
+        PVE::NVMeOF::controllers()->{nvme1}->{state},
+        'connecting',
+        'reconnecting controller is still present',
+    );
+    is(
+        PVE::NVMeOF::path_controller(
+            PVE::NVMeOF::subsystem_controllers('nqn.test'),
+            PVE::NVMeOF::parse_portal('192.0.2.1'),
+        ),
+        'nvme1',
+        'portal controller found',
+    );
+    is_deeply(
+        PVE::NVMeOF::namespaces('nqn.test'),
+        { nvme0n1 => { wwid => 'eui.abc', size => 4096 } },
+        'namespace facts',
+    );
+    is_deeply(PVE::NVMeOF::namespaces('nqn.other'), {}, 'other subsystem excluded');
+};
+
+subtest 'serialized commands inherit the lock' => sub {
+    my $value = PVE::NVMeOF::with_lock(sub {
+        my $out = '';
+        PVE::Tools::run_command(
+            [
+                '/usr/bin/perl',
+                '-MFcntl=:flock',
+                '-e',
+                'my @held = grep { (readlink($_) // "") =~ /fabrics.lock$/ } glob("/proc/self/fd/*"); die "not inherited" if !@held; open(my $fh, ">>", $ARGV[0]) or die $!; die "lock missing" if flock($fh, LOCK_EX|LOCK_NB); print "locked\\n";',
+                "$PVE::NVMeOF::RUN_DIR/fabrics.lock",
+            ],
+            timeout => 5,
+            outfunc => sub { $out .= $_[0] },
+        );
+        return $out;
+    });
+    is($value, 'locked', 'exec cannot acquire the lock held by its parent');
+    open(my $lock, '>>', "$PVE::NVMeOF::RUN_DIR/fabrics.lock") or die $!;
+    flock($lock, LOCK_EX) or die $!;
+    eval {
+        PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+    };
+    like($@, qr/is busy/, 'busy operation never enters callback');
+    close($lock);
+
+    my $tools = Test::MockModule->new('PVE::Tools');
+    my $timed_out = 1;
+    $tools->mock(run_fork_with_timeout => sub { return ($_[1]->(), $timed_out) });
+    my $retained;
+    eval {
+        PVE::NVMeOF::with_lock(sub {
+            my ($path) =
+                grep { (readlink($_) // '') =~ /fabrics.lock$/ } glob('/proc/self/fd/*');
+            my ($fd) = $path =~ m{/(\d+)$};
+            open($retained, '>&', $fd) or die $!;
+            return 1;
+        });
+    };
+    like($@, qr/timed out.*may still be running/, 'timeout does not claim deletion completed');
+    $timed_out = 0;
+    eval {
+        PVE::NVMeOF::with_lock(sub { die "must not count paths\n" });
+    };
+    like($@, qr/is busy/, 'retained descriptor keeps serialization after the caller returns');
+    close($retained);
+    is(PVE::NVMeOF::with_lock(sub { 1 }), 1, 'last descriptor close releases serialization');
+};
+
+subtest 'connect options, kernel reconnect and cooldown' => sub {
+    my $nvme = Test::MockModule->new('PVE::NVMeOF');
+    my $net = Test::MockModule->new('PVE::Network');
+    my @commands;
+    my $controllers = {};
+    $nvme->mock(with_lock => sub { $_[0]->() });
+    $nvme->mock(assert_support => sub { });
+    $nvme->mock(host_identity =>
+        sub { { nqn => 'nqn.host', id => '12345678-1234-1234-1234-123456789abc' } });
+    $nvme->mock(subsystem_controllers => sub { $controllers });
+    my $appear = 1;
+    $nvme->mock(
+        run_command => sub($cmd, %opts) {
+            is($opts{timeout}, 30, 'bounded connect');
+            push @commands, $cmd;
+            my %arg =
+                map { $cmd->[$_] => $cmd->[$_ + 1] } grep { $cmd->[$_] =~ /^--tr/ } 0 .. $#$cmd;
+            my $controller =
+                { traddr => $arg{'--traddr'}, trsvcid => $arg{'--trsvcid'}, state => 'live' };
+            $controllers->{ 'nvme' . scalar(@commands) } = $controller if $appear;
+        },
+    );
+    $net->mock(tcp_ping => sub { 1 });
+    $class->activate_storage('fabric', $cfg);
+    is(scalar(@commands), 2, 'one connect per missing portal');
+    ok(grep({ $_ eq '--ctrl-loss-tmo=-1' } $commands[0]->@*), 'kernel reconnects indefinitely');
+    is_deeply([@{ $commands[0] }[-2, -1]], ['--fast_io_fail_tmo', 120], 'fast failure default');
+    @commands = ();
+    $controllers = {
+        nvme1 => { traddr => '192.0.2.1', trsvcid => 4420, state => 'connecting' },
+        nvme2 => { traddr => '2001:db8::2', trsvcid => 4421, state => 'live' },
+    };
+    $class->activate_storage('fabric', $cfg);
+    is_deeply(\@commands, [],
+        'existing controllers are never reconnected, regardless of state');
+    $class->deactivate_storage('fabric', $cfg);
+    is_deeply(\@commands, [], 'deactivation does not remove controllers');
+    $controllers = {};
+    my $path = PVE::NVMeOF::storage_paths($cfg)->[0];
+    PVE::NVMeOF::note_connect_failure($cfg, $path);
+    ok(PVE::NVMeOF::in_cooldown($cfg, $path), 'failure starts cooldown');
+    $class->activate_storage('fabric', $cfg);
+    is(scalar(@commands), 1, 'only the other portal connects');
+    ($controllers, $appear) = ({}, 0);
+    my $other = PVE::NVMeOF::storage_paths($cfg)->[1];
+    my $warning = '';
+    {
+        local $SIG{__WARN__} = sub { $warning .= $_[0] };
+        $class->activate_storage('fabric', $cfg);
+    }
+    like($warning, qr/no controller appeared/, 'a connect without a controller is a failure');
+    ok(PVE::NVMeOF::in_cooldown($cfg, $other), 'which starts the cooldown too');
+};
+
+subtest 'local disk setup rejects fabric namespaces by transport' => sub {
+    my $disk = Test::MockModule->new('PVE::Diskmanage');
+    for my $transport (qw(nvme-tcp nvme-rdma nvme-fc)) {
+        $disk->mock(get_disks => sub { { nvme0n1 => { transport => $transport } } });
+        eval { PVE::Diskmanage::assert_disk_unused('/dev/nvme0n1') };
+        like($@, qr/not a valid local disk/,
+            "$transport refuses without storage configuration");
+    }
+};
+
+done_testing();
diff --git a/src/test/run_san_nvme_tests.pl b/src/test/run_san_nvme_tests.pl
index 51236f87..c74b638a 100755
--- a/src/test/run_san_nvme_tests.pl
+++ b/src/test/run_san_nvme_tests.pl
@@ -38,6 +38,7 @@ $storage->mock(config => sub { $cfg });
 $disk->mock(verify_blockdev_path => sub { $device });
 $disk->mock(is_partition => sub { 0 });
 $disk->mock(get_nvme_transport => sub { 'fc' });
+$disk->mock(read_subsysnqn => sub { 'nqn.test' });
 $disk->mock(is_iscsi_device => sub { 0 });
 $disk->mock(is_mounted => sub { 0 });
 $disk->mock(has_holder => sub { 0 });
@@ -78,6 +79,21 @@ $disk->mock(is_partition => sub { 0 });
 $cfg->{ids}->{data} = { type => 'lvm', base => 'fabric:nvme-eui.0123456789abcdef' };
 eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
 like($@, qr/base volume/, 'native base reference refuses');
+$cfg->{ids} =
+    { fabric =>
+        { type => 'nvmeof', subsysnqn => 'nqn.test', disable => 1, content => { images => 1 } } };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like(
+    $@,
+    qr/serves images from this subsystem/,
+    'direct namespace image use refuses even when disabled',
+);
+$cfg->{ids}->{fabric}->{content} = { 'vm-vol' => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+like($@, qr/serves images from this subsystem/, 'also when restricted to VM disks');
+$cfg->{ids}->{fabric}->{content} = { none => 1 };
+eval { PVE::Storage::SAN::assert_lun_unreferenced($cfg, $lun) };
+is($@, '', 'connection-only storage permits namespace initialization');
 $cfg->{ids} = {};
 @order = @opens = ();
 $tools->mock(file_get_contents => sub { 1024 });




  parent reply	other threads:[~2026-11-18 20:45 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` Thomas Lamprecht [this message]
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118204302.3100813-3-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal