all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages
Date: Wed, 18 Nov 2026 21:42:29 +0100	[thread overview]
Message-ID: <20261118204302.3100813-11-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com>

Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of the
storage type.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 pve-storage-multipath.adoc |   4 +-
 pve-storage-nvmeof.adoc    | 167 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  20 +++--
 3 files changed, 183 insertions(+), 8 deletions(-)
 create mode 100644 pve-storage-nvmeof.adoc

diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index d74fa23..caeae9b 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node.
 {pve} configures maps cluster-wide and shows their state under *Datacenter ->
 Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its
 LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe
-namespaces use kernel-native multipath instead; the procedures below cover SCSI
-LUNs.
+namespaces use kernel-native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs.
 
 Prerequisites
 ~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..aa464b6
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,167 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+  package recommends it, but installations without recommended packages need it
+  added separately. Its installation creates the node's host NQN in
+  `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+  image share them. Give each a new identity before it connects, with
+  `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+  *Datacenter -> Storage -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+  addresses that every node can reach; host names are not accepted.
+. Use a storage network that only trusted hosts can reach. The connections use
+  neither authentication nor TLS.
+. Keep native NVMe multipath enabled, which is the kernel default
+  (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach
+  the array and select *Guided Setup -> LVM on NVMe over Fabrics*.
+. Enter a discovery portal, select the subsystem, check the portals and name
+  the connection storage. This storage connects to the subsystem with
+  `content none`, while the LVM storage manages guest disks. Continuing creates
+  or reuses the connection storage before you select a namespace.
+. Select the namespace, choose the storage ID and volume group, then confirm.
+  Leave *Nodes* empty for all nodes.
+. Check the state of the namespace on each node of the storage, which the
+  setup shows once the storage is added, as described for LUNs in
+  xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+  check its access to the namespace on the array before placing guests on the
+  storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace reads *OK (1 path)*.
+Portals can also be changed later in the edit dialog of the connection storage;
+each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection
+storage. Enable *Use namespaces directly* there to use whole namespaces as guest
+disks, or add an *LVM* storage on a namespace later, with the connection storage
+as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <connection> --subsysnqn <nqn> --portals <portals> --content none
+# pvesm list <connection>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<connection>:nvme-<wwid>`. Adding
+the LVM storage creates the volume group if it does not exist yet. Creating it
+is refused if the namespace holds data that {pve} recognizes, such as
+partitions, a file system or a volume group, or if another storage references
+it. To reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later, and only one storage
+can use a subsystem.
+
+portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace have been lost for
+that long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+     subsysnqn nqn.2010-06.com.example:array1
+     portals 192.0.2.10,192.0.2.11
+     content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types  |Image formats  |Shared |Snapshots |Clones
+|images none    |raw            |yes    |no        |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace there to inspect its paths.
+Its *State* is *Degraded* while a path is unusable or fewer paths exist than
+the storage has portals, and *Failed* without a usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal:
+*Connected*, or *Reconnecting* while the kernel retries a lost path.
+
+An enabled storage connects the portals a node has no controller for. A portal
+that failed is retried after a minute. Removing a portal, or disabling or
+removing the storage, does not disconnect anything. To remove such a path,
+select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and
+press *Disconnect*, or run
+`pvesh delete /nodes/<node>/nvmeof/controller/<controller>`. This is refused if
+a namespace that is mounted, in use by another device, open in a process such
+as a running guest, or referenced by a storage would lose its last usable path.
+The process check does not find one that opened the namespace through another
+device node. A controller with namespaces that do not use native NVMe multipath
+is refused as well. A storage that still lists the portal connects to it again.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS.
+* NVMe/FC namespaces need no storage of this type, and the guided setups and
+  the cluster-wide status do not cover them. *Node -> SAN* lists them. Create
+  their volume group under *Node -> Disks -> LVM*; with *Add Storage* enabled,
+  this adds a shared LVM storage.
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index f7de231..5cf0df2 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
 |LVM-thin       |lvmthin     |block  |no    |yes      |yes
 |iSCSI/kernel   |iscsi       |block  |yes^3^|yes^4^   |yes
 |iSCSI/libiscsi |iscsidirect |block  |yes^3^|yes^4^   |yes
+|NVMe-oF        |nvmeof      |block  |yes^3^|yes^4^   |yes
 |FC/SAS         |native^6^   |block  |yes^3^|yes^4^   |yes
 |Ceph/RBD       |rbd         |block  |yes   |yes      |yes
 |ZFS over iSCSI |zfs         |block  |yes   |yes      |yes
@@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
 hours. If your setup is affected, create and remove snapshots while the VM is
 shut down, expecting a long task duration.
 
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster.
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
 
 ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
 These snapshots use separate volumes for the snapshot data and layer them.  For
@@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have
 totally different content.
 
 Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use *Guided Setup* under
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use *Guided Setup* under
 *Datacenter -> Storage -> SAN* instead. It creates the storages that are needed
-and enables multipath for the LUN.
+and enables multipath for a LUN.
 
 
 Storage Pools
@@ -506,6 +510,8 @@ See Also
 
 * link:/wiki/Storage:_Multipath[Storage: Multipath]
 
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
 * link:/wiki/Storage:_LVM[Storage: LVM]
 
 * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -550,6 +556,8 @@ include::pve-storage-iscsidirect.adoc[]
 
 include::pve-storage-multipath.adoc[]
 
+include::pve-storage-nvmeof.adoc[]
+
 include::pve-storage-rbd.adoc[]
 
 include::pve-storage-cephfs.adoc[]




  parent reply	other threads:[~2026-11-18 20:44 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 20:42 [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-18 20:42 ` Thomas Lamprecht [this message]
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118204302.3100813-11-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal