From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH docs 10/17] storage: document NVMe over Fabrics storages
Date: Fri, 13 Nov 2026 04:04:38 +0100 [thread overview]
Message-ID: <20261113030541.1591528-11-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261113030541.1591528-1-t.lamprecht@proxmox.com>
Add a chapter for the nvmeof storage type: prerequisites, setting up
shared LVM on a namespace with the guided setup or by hand, the
properties, checking and disconnecting paths, and the limits of this
first version.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
pve-storage-multipath.adoc | 4 +-
pve-storage-nvmeof.adoc | 161 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 20 +++--
3 files changed, 177 insertions(+), 8 deletions(-)
create mode 100644 pve-storage-nvmeof.adoc
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
index d4503a7..c7b2143 100644
--- a/pve-storage-multipath.adoc
+++ b/pve-storage-multipath.adoc
@@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node.
{pve} configures maps cluster-wide and shows their state under *Datacenter ->
Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its
LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe
-namespaces use kernel-native multipath instead; the procedures below cover SCSI
-LUNs.
+namespaces use kernel-native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs.
Prerequisites
~~~~~~~~~~~~~
diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc
new file mode 100644
index 0000000..d2faf0f
--- /dev/null
+++ b/pve-storage-nvmeof.adoc
@@ -0,0 +1,161 @@
+[[storage_nvmeof]]
+NVMe over Fabrics Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: NVMe over Fabrics
+endif::wiki[]
+
+Storage pool type: `nvmeof`
+
+NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe
+subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of
+LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces
+directly as disk images.
+
+Each node connects one controller per portal of the storage. The kernel
+combines these paths into one device per namespace with its native NVMe
+multipath and reconnects lost paths on its own. `multipath-tools` is not
+involved.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+. Upgrade all nodes before adding the first NVMe over Fabrics storage.
+. Install `nvme-cli` on every participating node. The `libpve-storage-perl`
+ package recommends it, but installations without recommended packages need it
+ added separately. Its installation creates the node's host NQN in
+ `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one
+ image share them. Give each a new identity before it connects, with
+ `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`.
+. Allow each node's host NQN on the array. The names are shown under
+ *Datacenter -> Storage -> SAN* and in the guided setup.
+. Present the same empty namespace to all intended nodes. Portals are IP
+ addresses that every node can reach; host names are not accepted.
+. Keep native NVMe multipath enabled, which is the kernel default
+ (`nvme_core.multipath=Y`).
+
+[[storage_nvmeof_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach
+ the array and select *Add Storage -> LVM on NVMe over Fabrics*.
+. Enter a discovery portal, select the subsystem, check the portals and name
+ the connection storage. This storage connects to the subsystem with
+ `content none`, while the LVM storage manages guest disks. Continuing creates
+ or reuses the connection storage before you select a namespace.
+. Select the namespace, choose the storage ID and volume group, then confirm.
+ Leave *Nodes* empty for all nodes.
+. Check the state of the namespace on each node of the storage, which the
+ setup shows once the storage is added, as described for LUNs in
+ xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*,
+ check its access to the namespace on the array before placing guests on the
+ storage.
+
+The discovery fills in *Portals*. Some targets, such as the Linux kernel
+target, report only the portal that was asked: add the others to the list.
+With a single portal, the state of the namespace reads *OK (1 path)*.
+Portals can also be changed later in the edit dialog of the connection storage;
+each node then connects new ones on its own.
+
+*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection
+storage. Enable *Use namespaces directly* there to use whole namespaces as guest
+disks, or add an *LVM* storage on a namespace later, with the connection storage
+as *Base storage*.
+
+To set it up by hand, run these commands on a node that can reach the array:
+
+----
+# pvesm scan nvmeof <portal>
+# pvesm add nvmeof <connection> --subsysnqn <nqn> --nvme-portals <portals> --content none
+# pvesm list <connection>
+# pvesm add lvm <storage> --vgname <vg> --base <volume-id> --shared 1
+----
+
+Use the complete volume ID from the listing, `<connection>:nvme-<wwid>`. Adding
+the LVM storage creates the volume group if it does not exist yet. Creating it
+is refused if the namespace holds data or another storage references it. To
+reuse a namespace, wipe it as described for LUNs in
+xref:storage_multipath_setup[Multipath].
+
+Configuration
+~~~~~~~~~~~~~
+
+The backend supports the common storage properties `content`, `nodes`,
+`disable`, and the following NVMe over Fabrics specific properties:
+
+subsysnqn::
+
+The NQN of the NVMe subsystem. It cannot be changed later.
+
+nvme-portals::
+
+Comma-separated list of the subsystem's TCP portals, one per path. Each is an
+IPv4 address or an IPv6 address in brackets, with an optional port (default
+4420).
+
+nvme-fast-io-fail-tmo::
+
+Seconds a path may be reconnecting before I/O sent to it fails. The default is
+120: guests get I/O errors once all paths of a namespace were lost for that
+long. With `-1`, I/O waits until a path returns. A change applies to newly
+connected controllers only.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+nvmeof: fabric
+ subsysnqn nqn.2010-06.com.example:array1
+ nvme-portals 192.0.2.10,192.0.2.11
+ content none
+----
+
+.Storage features for backend `nvmeof`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types |Image formats |Shared |Snapshots |Clones
+|images none |raw |yes |no |no
+|==============================================================================
+
+{pve} does not create or delete namespaces; manage them on the array. A volume
+is named `nvme-<wwid>` after its namespace, like its link in `/dev/disk/by-id/`.
+
+Checking and Disconnecting Paths
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics
+storages among the LUNs, with their state on each node of the storage and
+*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a
+node. Expand a namespace there to inspect its paths.
+Its *State* is *Degraded* while a path is unusable or fewer paths exist than
+the storage has portals, and *Failed* without a usable path. *Node -> SAN ->
+NVMe over Fabrics* lists the node's controllers, one per connected portal:
+*Connected*, or *Reconnecting* while the kernel retries a lost path.
+
+An enabled storage connects the portals a node has no controller for. A portal
+that failed is retried after a minute. Removing a portal, or disabling or
+removing the storage, does not disconnect anything. To remove such a path,
+select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and
+press *Disconnect*, or run
+`pvesh delete /nodes/<node>/nvmeof/controller/<controller>`. This is refused if
+a namespace that is mounted, in use by another device, or referenced by a
+storage would lose its last usable path. A storage that still lists the portal
+connects to it again.
+
+Limits
+~~~~~~
+
+* Only the TCP transport is supported, without authentication or TLS. Use a
+ storage network that only trusted hosts can reach.
+* NVMe/FC namespaces need no storage of this type. Set them up with
+ *Datacenter -> Storage -> SAN -> Add Storage -> LVM on Fibre Channel*, see
+ xref:storage_multipath_setup[Multipath].
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 5668a41..442ac2d 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device).
|LVM-thin |lvmthin |block |no |yes |yes
|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes
|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes
+|NVMe-oF |nvmeof |block |yes^3^|yes^4^ |yes
|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes
|Ceph/RBD |rbd |block |yes |yes |yes
|ZFS over iSCSI |zfs |block |yes |yes |yes
@@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
hours. If your setup is affected, create and remove snapshots while the VM is
shut down, expecting a long task duration.
-^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
-a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
-reachable over several paths; xref:storage_multipath[Multipath] describes how
-{pve} manages them in a cluster.
+^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block
+storage as shared storage in a cluster, LVM is used to split the LUN or
+namespace into virtual disks. A LUN is usually reachable over several paths;
+xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster.
+NVMe-oF namespaces use the kernel's native multipath instead, see
+xref:storage_nvmeof[NVMe over Fabrics].
^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
These snapshots use separate volumes for the snapshot data and layer them. For
@@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have
totally different content.
Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM
-on a xref:storage_multipath_setup[SAN LUN], use the guided setup under
+on a xref:storage_multipath_setup[SAN LUN] or an
+xref:storage_nvmeof_setup[NVMe namespace], use the guided setup under
*Datacenter -> Storage -> SAN -> Add Storage* instead. It creates the storages
-that are needed and enables multipath for the LUN.
+that are needed and enables multipath for a LUN.
Storage Pools
@@ -427,6 +431,8 @@ See Also
* link:/wiki/Storage:_Multipath[Storage: Multipath]
+* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics]
+
* link:/wiki/Storage:_LVM[Storage: LVM]
* link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -471,6 +477,8 @@ include::pve-storage-iscsidirect.adoc[]
include::pve-storage-multipath.adoc[]
+include::pve-storage-nvmeof.adoc[]
+
include::pve-storage-rbd.adoc[]
include::pve-storage-cephfs.adoc[]
next prev parent reply other threads:[~2026-11-13 3:31 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13 3:04 ` Thomas Lamprecht [this message]
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113030541.1591528-11-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.