public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages
Date: Sun, 22 Nov 2026 02:25:47 +0100	[thread overview]
Message-ID: <20261122012605.1848824-11-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261122012605.1848824-1-t.lamprecht@proxmox.com>

Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of the
storage type. The introduction lists it among the network storages, and
the LVM chapter names it next to iSCSI as base storage.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 pve-gui.adoc               |  12 ++-
 pve-intro.adoc             |   3 +-
 pve-storage-lvm.adoc       |  10 +-
 pve-storage-multipath.adoc |   3 +-
 pve-storage-nvmeof.adoc    | 189 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  24 +++--
 6 files changed, 222 insertions(+), 19 deletions(-)
 create mode 100644 pve-storage-nvmeof.adoc

diff --git a/pve-gui.adoc b/pve-gui.adoc
index 9a1f70b..a514eca 100644
--- a/pve-gui.adoc
+++ b/pve-gui.adoc
@@ -233,10 +233,11 @@ On the datacenter level, you can access cluster-wide settings and information.
 * *Options:* view and manage cluster-wide default settings.
 
 * *Storage:* provides an interface for managing cluster storage. The *Add* menu
-   offers guided setups for shared LVM on iSCSI and Fibre Channel LUNs.
+   offers guided setups for shared LVM on iSCSI and Fibre Channel LUNs and on
+   NVMe over Fabrics namespaces.
 
-* *SAN:* shows the state of the SAN LUNs on all nodes and manages the multipath
-   configuration.
+* *SAN:* shows the state of the SAN LUNs and NVMe namespaces on all nodes and
+   manages the multipath configuration.
 
 * *Backup:* schedule backup jobs. This operates cluster wide, so it doesn't
    matter where the VMs/containers are on your cluster when scheduling.
@@ -288,8 +289,9 @@ The top header has useful buttons such as 'Reboot', 'Shutdown', 'Shell',
 
 * *Disks:* get an overview of the attached disks, and manage how they are used.
 
-* *SAN:* get an overview of the SAN LUNs of the node, with their paths and
-  usage. Below it, *iSCSI* shows the iSCSI targets and sessions.
+* *SAN:* get an overview of the SAN LUNs and NVMe namespaces of the node, with
+  their paths and usage. Below it, *iSCSI* shows the iSCSI targets and sessions,
+  and *NVMe over Fabrics* the connected controllers.
 
 * *Ceph:* is only used if you have installed a Ceph server on your
    host. In this case, you can manage your Ceph cluster and see the status
diff --git a/pve-intro.adoc b/pve-intro.adoc
index c2a0b6a..baec59f 100644
--- a/pve-intro.adoc
+++ b/pve-intro.adoc
@@ -103,9 +103,10 @@ the cluster have direct access to VM disk images.
 
 The following network storage types are supported:
 
-* LVM group (network backing with iSCSI targets)
+* LVM group (network backing with iSCSI targets or NVMe over Fabrics)
 * iSCSI target
 * iSCSI LUNs (direct use)
+* NVMe over Fabrics subsystem
 * NFS share
 * CIFS share
 * Ceph RBD
diff --git a/pve-storage-lvm.adoc b/pve-storage-lvm.adoc
index 92019fc..8b9564c 100644
--- a/pve-storage-lvm.adoc
+++ b/pve-storage-lvm.adoc
@@ -34,7 +34,8 @@ LVM volume group name. This must point to an existing volume group.
 
 Base volume. This volume is automatically activated before accessing
 the storage. This is mostly useful when the LVM volume group resides
-on a remote iSCSI server.
+on a remote iSCSI server or on an xref:storage_nvmeof[NVMe over Fabrics]
+namespace.
 
 `saferemove`::
 
@@ -143,9 +144,10 @@ A benefit of LVM is that it can be used with shared storage.
 For example, an iSCSI LUN. The backend implements proper cluster-wide locking if
 the storage is marked as shared in the configuration.
 
-TIP: *Shared LVM on iSCSI* or *Shared LVM on Fibre Channel* in *Datacenter ->
-Storage -> Add* in the web UI creates a shared LVM storage on a SAN LUN, see
-xref:storage_multipath_setup[Multipath].
+TIP: *Shared LVM on iSCSI*, *Shared LVM on Fibre Channel* or *Shared LVM on NVMe
+over Fabrics* in *Datacenter -> Storage -> Add* in the web UI creates a shared
+LVM storage on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace].
 
 NOTE: If you extend a shared volume group onto another device on one node, the
 other nodes may only see the added physical volume after LVM has scanned their
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index 3e07eba..45ab3a8 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -15,7 +15,8 @@ Fibre Channel.
 {pve} sets up the maps on all nodes and shows their state under *Datacenter ->
 SAN*. An iSCSI storage with *Multipath* enabled requests the maps of its LUNs, a
 Fibre Channel LUN needs a multipath entry. NVMe namespaces use the multipath
-support of the kernel instead and are not covered here.
+support of the kernel instead, see xref:storage_nvmeof[NVMe over Fabrics], and
+are not covered here.
 
 Prerequisites
 ~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..2a5aaa6
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,189 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage, also
+  nodes that will not use it. A node without support for this storage type
+  removes such storages from the cluster-wide storage configuration when it
+  writes that configuration, for example when a storage is added or edited on
+  that node.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+  package recommends it, but installations without recommended packages need it
+  added separately. Its installation creates the node's host NQN in
+  `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+  image share them. Give each a new identity before it connects, with
+  `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+  *Datacenter -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+  addresses that every node can reach; host names are not accepted.
+. Use a storage network that only trusted hosts can reach. The connections use
+  neither authentication nor TLS.
+. Keep native NVMe multipath enabled, which is the kernel default
+  (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. On a node that can reach the array, select *Shared LVM on NVMe over Fabrics*
+  under *Datacenter -> Storage -> Add*. The *Add Shared LVM* menu of
+  *Datacenter -> SAN* has the same setup.
+. Enter the *ID* of the new storage, a discovery portal and the subsystem, and
+  check the portals. If no NVMe over Fabrics storage connects the nodes to that
+  subsystem yet, the setup adds one named after the ID with a `-base` suffix, or
+  as set under *Advanced*. This storage connects to the subsystem with `content
+  none`, while the LVM storage manages guest disks. Clear the nodes in the list
+  of host NQNs that should not use the storages. Continuing creates or reuses
+  the NVMe over Fabrics storage before you select a namespace.
+. Select the namespace after checking that no host uses it and that no data on
+  it must be kept. The volume group is named after the ID unless set under
+  *Advanced*.
+. Check the state of the namespace on each node of the storage, which the
+  setup shows once the storage is added, as described for LUNs in
+  xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+  check its access to the namespace on the array before placing guests on the
+  storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace shows `1/1`, which has no path
+redundancy.
+Portals can also be changed later in the edit dialog of the NVMe over Fabrics
+storage; each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the NVMe over
+Fabrics storage. Without a discovery portal, enter the subsystem NQN and its
+portals as they are. Enable *Use namespaces directly* there to use whole
+namespaces as guest disks, or add an *LVM* storage on a namespace later, with
+the NVMe over Fabrics storage as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <nvmeof-storage> --subsysnqn <nqn> --portals <portals> \
+    --content none
+# pvesm list <nvmeof-storage>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<nvmeof-storage>:nvme-<wwid>`.
+Adding the LVM storage creates the volume group if it does not exist yet.
+Creating it is refused if the namespace holds data that {pve} recognizes, such
+as partitions, a file system or a volume group, or if another storage references
+it. To reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later, and only one storage
+can use a subsystem.
+
+portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace have been lost for
+that long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+     subsysnqn nqn.2010-06.com.example:array1
+     portals 192.0.2.10,192.0.2.11
+     content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types  |Image formats  |Shared |Snapshots |Clones
+|images none    |raw            |yes    |no        |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace under *Node -> SAN* to inspect its paths.
+Its *State* counts the usable paths out of the portals of the storage, for
+example `1/2` while one of two paths is unusable, and shows *Failed* without a
+usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal,
+with their *State*: *Connected*, or *Reconnecting* while the kernel retries a
+lost path. A node sends a xref:notification_events[multipath notification] when
+a namespace loses or regains paths, as for a multipath map. A node that cannot
+query `multipathd` sends no such notifications, also not for its namespaces,
+although *Datacenter -> SAN* still shows their state.
+
+Each node of an enabled storage connects to the portals that it is not yet
+connected to. A portal that failed is retried after a minute. Removing a portal,
+or disabling or removing the storage, does not disconnect anything. To remove
+such a path, select its controller under *Node -> SAN -> NVMe over Fabrics* on
+each node and press *Disconnect*, or run
+`pvesh delete /nodes/<node>/san/nvmeof/controller/<controller>`.
+This is refused if a namespace that is mounted, in use by another device or as
+swap, open in a process such as a running guest, or referenced by a storage
+would lose its last usable path. For a controller that is no usable path itself,
+such as one that keeps reconnecting to a deleted subsystem, only the local uses
+count, not a storage reference. The process check does not find a process that
+opened the namespace through another device node, for example one in a
+container. A controller with namespaces that do not use native NVMe multipath is
+refused as well. A storage that still lists the portal connects to it again.
+Namespaces of a subsystem that no storage uses then show fewer paths, but none
+as missing, as nothing tells how many paths they should have.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS.
+* NVMe/FC namespaces need no storage of this type, and the guided setups and
+  the cluster-wide status do not cover them. *Node -> SAN* lists them. Create
+  their volume group under *Node -> Disks -> LVM*; with *Add Storage* enabled,
+  this adds a shared LVM storage.
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 8f1b24e..0bea1d2 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
 |LVM-thin       |lvmthin     |block  |no    |yes      |yes
 |iSCSI/kernel   |iscsi       |block  |yes^3^|yes^4^   |yes
 |iSCSI/libiscsi |iscsidirect |block  |yes^3^|yes^4^   |yes
+|NVMe-oF        |nvmeof      |block  |yes^3^|yes^4^   |yes
 |FC/SAS         |native^6^   |block  |yes^3^|yes^4^   |yes
 |Ceph/RBD       |rbd         |block  |yes   |yes      |yes
 |ZFS over iSCSI |zfs         |block  |yes   |yes      |yes
@@ -96,11 +97,13 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
 hours. If your setup is affected, create and remove snapshots while the VM is
 shut down, expecting a long task duration.
 
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster. FC and SAS LUNs need a
-xref:storage_multipath_entries[multipath entry].
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+FC and SAS LUNs need a xref:storage_multipath_entries[multipath entry].
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
 
 ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
 These snapshots use separate volumes for the snapshot data and layer them.  For
@@ -154,9 +157,10 @@ is available on all nodes, but it is physically different and can have
 totally different content.
 
 Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use *Shared LVM on iSCSI* or
-*Shared LVM on Fibre Channel* there instead. It creates the storages that are
-needed and enables multipath for the LUN.
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use *Shared LVM on iSCSI*, *Shared
+LVM on Fibre Channel* or *Shared LVM on NVMe over Fabrics* there instead. It
+creates the storages that are needed and enables multipath for a LUN.
 
 
 Storage Pools
@@ -507,6 +511,8 @@ See Also
 
 * link:/wiki/Storage:_Multipath[Storage: Multipath]
 
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
 * link:/wiki/Storage:_LVM[Storage: LVM]
 
 * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -551,6 +557,8 @@ include::pve-storage-iscsidirect.adoc[]
 
 include::pve-storage-multipath.adoc[]
 
+include::pve-storage-nvmeof.adoc[]
+
 include::pve-storage-rbd.adoc[]
 
 include::pve-storage-cephfs.adoc[]




  parent reply	other threads:[~2026-11-22  1:28 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-22  1:25 [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22  1:25 ` Thomas Lamprecht [this message]
2026-11-22  1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122012605.1848824-11-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal