public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH docs 10/17] storage: document NVMe over Fabrics storages
Date: Fri, 13 Nov 2026 04:04:38 +0100	[thread overview]
Message-ID: <20261113030541.1591528-11-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261113030541.1591528-1-t.lamprecht@proxmox.com>

Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of this
first version.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 pve-storage-multipath.adoc |   4 +-
 pve-storage-nvmeof.adoc    | 161 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  20 +++--
 3 files changed, 177 insertions(+), 8 deletions(-)
 create mode 100644 pve-storage-nvmeof.adoc

diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index d4503a7..c7b2143 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node.
 {pve} configures maps cluster-wide and shows their state under *Datacenter ->
 Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its
 LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe
-namespaces use kernel-native multipath instead; the procedures below cover SCSI
-LUNs.
+namespaces use kernel-native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs.
 
 Prerequisites
 ~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..d2faf0f
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,161 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+  package recommends it, but installations without recommended packages need it
+  added separately. Its installation creates the node's host NQN in
+  `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+  image share them. Give each a new identity before it connects, with
+  `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+  *Datacenter -> Storage -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+  addresses that every node can reach; host names are not accepted.
+. Keep native NVMe multipath enabled, which is the kernel default
+  (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach
+  the array and select *Add Storage -> LVM on NVMe over Fabrics*.
+. Enter a discovery portal, select the subsystem, check the portals and name
+  the connection storage. This storage connects to the subsystem with
+  `content none`, while the LVM storage manages guest disks. Continuing creates
+  or reuses the connection storage before you select a namespace.
+. Select the namespace, choose the storage ID and volume group, then confirm.
+  Leave *Nodes* empty for all nodes.
+. Check the state of the namespace on each node of the storage, which the
+  setup shows once the storage is added, as described for LUNs in
+  xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+  check its access to the namespace on the array before placing guests on the
+  storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace reads *OK (1 path)*.
+Portals can also be changed later in the edit dialog of the connection storage;
+each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection
+storage. Enable *Use namespaces directly* there to use whole namespaces as guest
+disks, or add an *LVM* storage on a namespace later, with the connection storage
+as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <connection> --subsysnqn <nqn> --nvme-portals <portals> --content none
+# pvesm list <connection>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<connection>:nvme-<wwid>`. Adding
+the LVM storage creates the volume group if it does not exist yet. Creating it
+is refused if the namespace holds data or another storage references it. To
+reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later.
+
+nvme-portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+nvme-fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace were lost for that
+long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+     subsysnqn nqn.2010-06.com.example:array1
+     nvme-portals 192.0.2.10,192.0.2.11
+     content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types  |Image formats  |Shared |Snapshots |Clones
+|images none    |raw            |yes    |no        |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace there to inspect its paths.
+Its *State* is *Degraded* while a path is unusable or fewer paths exist than
+the storage has portals, and *Failed* without a usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal:
+*Connected*, or *Reconnecting* while the kernel retries a lost path.
+
+An enabled storage connects the portals a node has no controller for. A portal
+that failed is retried after a minute. Removing a portal, or disabling or
+removing the storage, does not disconnect anything. To remove such a path,
+select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and
+press *Disconnect*, or run
+`pvesh delete /nodes/<node>/nvmeof/controller/<controller>`. This is refused if
+a namespace that is mounted, in use by another device, or referenced by a
+storage would lose its last usable path. A storage that still lists the portal
+connects to it again.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS. Use a
+  storage network that only trusted hosts can reach.
+* NVMe/FC namespaces need no storage of this type. Set them up with
+  *Datacenter -> Storage -> SAN -> Add Storage -> LVM on Fibre Channel*, see
+  xref:storage_multipath_setup[Multipath].
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 5668a41..442ac2d 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
 |LVM-thin       |lvmthin     |block  |no    |yes      |yes
 |iSCSI/kernel   |iscsi       |block  |yes^3^|yes^4^   |yes
 |iSCSI/libiscsi |iscsidirect |block  |yes^3^|yes^4^   |yes
+|NVMe-oF        |nvmeof      |block  |yes^3^|yes^4^   |yes
 |FC/SAS         |native^6^   |block  |yes^3^|yes^4^   |yes
 |Ceph/RBD       |rbd         |block  |yes   |yes      |yes
 |ZFS over iSCSI |zfs         |block  |yes   |yes      |yes
@@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
 hours. If your setup is affected, create and remove snapshots while the VM is
 shut down, expecting a long task duration.
 
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster.
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
 
 ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
 These snapshots use separate volumes for the snapshot data and layer them.  For
@@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have
 totally different content.
 
 Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use the guided setup under
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use the guided setup under
 *Datacenter -> Storage -> SAN -> Add Storage* instead. It creates the storages
-that are needed and enables multipath for the LUN.
+that are needed and enables multipath for a LUN.
 
 
 Storage Pools
@@ -427,6 +431,8 @@ See Also
 
 * link:/wiki/Storage:_Multipath[Storage: Multipath]
 
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
 * link:/wiki/Storage:_LVM[Storage: LVM]
 
 * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -471,6 +477,8 @@ include::pve-storage-iscsidirect.adoc[]
 
 include::pve-storage-multipath.adoc[]
 
+include::pve-storage-nvmeof.adoc[]
+
 include::pve-storage-rbd.adoc[]
 
 include::pve-storage-cephfs.adoc[]




  parent reply	other threads:[~2026-11-13  3:31 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13  3:04 ` Thomas Lamprecht [this message]
2026-11-13  3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113030541.1591528-11-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal