From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 7956B1FF0E3 for ; Wed, 18 Nov 2026 21:44:55 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 93DA42197C; Wed, 18 Nov 2026 21:43:34 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Date: Wed, 18 Nov 2026 21:42:29 +0100 Message-ID: <20261118204302.3100813-11-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118204302.3100813-1-t.lamprecht@proxmox.com> References: <20261118204302.3100813-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795034587716 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.388 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PWWYSZ54QBZM6WQBQZZEKPW3ZF65D3NS X-Message-ID-Hash: PWWYSZ54QBZM6WQBQZZEKPW3ZF65D3NS X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add a chapter for the nvmeof storage type: prerequisites, setting up shared LVM on a namespace with the guided setup or by hand, the properties, checking and disconnecting paths, and the limits of the storage type. Signed-off-by: Thomas Lamprecht --- pve-storage-multipath.adoc | 4 +- pve-storage-nvmeof.adoc | 167 +++++++++++++++++++++++++++++++++++++ pvesm.adoc | 20 +++-- 3 files changed, 183 insertions(+), 8 deletions(-) create mode 100644 pve-storage-nvmeof.adoc diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc index d74fa23..caeae9b 100644 --- a/pve-storage-multipath.adoc +++ b/pve-storage-multipath.adoc @@ -14,8 +14,8 @@ Identifier (WWID) identifies the same LUN on every node. {pve} configures maps cluster-wide and shows their state under *Datacenter -> Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe -namespaces use kernel-native multipath instead; the procedures below cover SCSI -LUNs. +namespaces use kernel-native multipath instead, see +xref:storage_nvmeof[NVMe over Fabrics]; the procedures below cover SCSI LUNs. Prerequisites ~~~~~~~~~~~~~ diff --git a/pve-storage-nvmeof.adoc b/pve-storage-nvmeof.adoc new file mode 100644 index 0000000..aa464b6 --- /dev/null +++ b/pve-storage-nvmeof.adoc @@ -0,0 +1,167 @@ +[[storage_nvmeof]] +NVMe over Fabrics Backend +------------------------- +ifdef::wiki[] +:pve-toplevel: +:title: Storage: NVMe over Fabrics +endif::wiki[] + +Storage pool type: `nvmeof` + +NVMe over Fabrics (NVMe-oF) gives a node access to the namespaces of an NVMe +subsystem on a storage array over TCP. Namespaces are the NVMe counterpart of +LUNs. Use one as the base volume of a shared LVM storage, or use the namespaces +directly as disk images. + +Each node connects one controller per portal of the storage. The kernel +combines these paths into one device per namespace with its native NVMe +multipath and reconnects lost paths on its own. `multipath-tools` is not +involved. + +Prerequisites +~~~~~~~~~~~~~ + +. Upgrade all nodes before adding the first NVMe over Fabrics storage. +. Install `nvme-cli` on every participating node. The `libpve-storage-perl` + package recommends it, but installations without recommended packages need it + added separately. Its installation creates the node's host NQN in + `/etc/nvme/hostnqn` and host ID in `/etc/nvme/hostid`. Nodes cloned from one + image share them. Give each a new identity before it connects, with + `nvme gen-hostnqn > /etc/nvme/hostnqn` and `uuidgen > /etc/nvme/hostid`. +. Allow each node's host NQN on the array. The names are shown under + *Datacenter -> Storage -> SAN* and in the guided setup. +. Present the same empty namespace to all intended nodes. Portals are IP + addresses that every node can reach; host names are not accepted. +. Use a storage network that only trusted hosts can reach. The connections use + neither authentication nor TLS. +. Keep native NVMe multipath enabled, which is the kernel default + (`nvme_core.multipath=Y`). + +[[storage_nvmeof_setup]] +Setting Up Shared LVM +~~~~~~~~~~~~~~~~~~~~~ + +. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can reach + the array and select *Guided Setup -> LVM on NVMe over Fabrics*. +. Enter a discovery portal, select the subsystem, check the portals and name + the connection storage. This storage connects to the subsystem with + `content none`, while the LVM storage manages guest disks. Continuing creates + or reuses the connection storage before you select a namespace. +. Select the namespace, choose the storage ID and volume group, then confirm. + Leave *Nodes* empty for all nodes. +. Check the state of the namespace on each node of the storage, which the + setup shows once the storage is added, as described for LUNs in + xref:storage_multipath_setup[Multipath]. For a node that stays *Missing*, + check its access to the namespace on the array before placing guests on the + storage. + +The discovery fills in *Portals*. Some targets, such as the Linux kernel +target, report only the portal that was asked: add the others to the list. +With a single portal, the state of the namespace reads *OK (1 path)*. +Portals can also be changed later in the edit dialog of the connection storage; +each node then connects new ones on its own. + +*Datacenter -> Storage -> Add -> NVMe over Fabrics* adds only the connection +storage. Enable *Use namespaces directly* there to use whole namespaces as guest +disks, or add an *LVM* storage on a namespace later, with the connection storage +as *Base storage*. + +To set it up by hand, run these commands on a node that can reach the array: + +---- +# pvesm scan nvmeof +# pvesm add nvmeof --subsysnqn --portals --content none +# pvesm list +# pvesm add lvm --vgname --base --shared 1 +---- + +Use the complete volume ID from the listing, `:nvme-`. Adding +the LVM storage creates the volume group if it does not exist yet. Creating it +is refused if the namespace holds data that {pve} recognizes, such as +partitions, a file system or a volume group, or if another storage references +it. To reuse a namespace, wipe it as described for LUNs in +xref:storage_multipath_setup[Multipath]. + +Configuration +~~~~~~~~~~~~~ + +The backend supports the common storage properties `content`, `nodes`, +`disable`, and the following NVMe over Fabrics specific properties: + +subsysnqn:: + +The NQN of the NVMe subsystem. It cannot be changed later, and only one storage +can use a subsystem. + +portals:: + +Comma-separated list of the subsystem's TCP portals, one per path. Each is an +IPv4 address or an IPv6 address in brackets, with an optional port (default +4420). + +fast-io-fail-tmo:: + +Seconds a path may be reconnecting before I/O sent to it fails. The default is +120: guests get I/O errors once all paths of a namespace have been lost for +that long. With `-1`, I/O waits until a path returns. A change applies to newly +connected controllers only. + +.Configuration Example (`/etc/pve/storage.cfg`) +---- +nvmeof: fabric + subsysnqn nqn.2010-06.com.example:array1 + portals 192.0.2.10,192.0.2.11 + content none +---- + +.Storage features for backend `nvmeof` +[width="100%",cols="m,m,3*d",options="header"] +|============================================================================== +|Content types |Image formats |Shared |Snapshots |Clones +|images none |raw |yes |no |no +|============================================================================== + +{pve} does not create or delete namespaces; manage them on the array. A volume +is named `nvme-` after its namespace, like its link in `/dev/disk/by-id/`. + +Checking and Disconnecting Paths +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Datacenter -> Storage -> SAN* lists the namespaces of NVMe over Fabrics +storages among the LUNs, with their state on each node of the storage and +*Multipath* shown as *Native*, as does *Node -> SAN* for the namespaces of a +node. Expand a namespace there to inspect its paths. +Its *State* is *Degraded* while a path is unusable or fewer paths exist than +the storage has portals, and *Failed* without a usable path. *Node -> SAN -> +NVMe over Fabrics* lists the node's controllers, one per connected portal: +*Connected*, or *Reconnecting* while the kernel retries a lost path. + +An enabled storage connects the portals a node has no controller for. A portal +that failed is retried after a minute. Removing a portal, or disabling or +removing the storage, does not disconnect anything. To remove such a path, +select its controller under *Node -> SAN -> NVMe over Fabrics* on each node and +press *Disconnect*, or run +`pvesh delete /nodes//nvmeof/controller/`. This is refused if +a namespace that is mounted, in use by another device, open in a process such +as a running guest, or referenced by a storage would lose its last usable path. +The process check does not find one that opened the namespace through another +device node. A controller with namespaces that do not use native NVMe multipath +is refused as well. A storage that still lists the portal connects to it again. + +Limits +~~~~~~ + +* Only the TCP transport is supported, without authentication or TLS. +* NVMe/FC namespaces need no storage of this type, and the guided setups and + the cluster-wide status do not cover them. *Node -> SAN* lists them. Create + their volume group under *Node -> Disks -> LVM*; with *Add Storage* enabled, + this adds a shared LVM storage. + +ifdef::wiki[] + +See Also +~~~~~~~~ + +* link:/wiki/Storage[Storage] + +endif::wiki[] diff --git a/pvesm.adoc b/pvesm.adoc index f7de231..5cf0df2 100644 --- a/pvesm.adoc +++ b/pvesm.adoc @@ -79,6 +79,7 @@ nodes that can be accessed as RBD (RADOS Block Device). |LVM-thin |lvmthin |block |no |yes |yes |iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes |iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes +|NVMe-oF |nvmeof |block |yes^3^|yes^4^ |yes |FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes |Ceph/RBD |rbd |block |yes |yes |yes |ZFS over iSCSI |zfs |block |yes |yes |yes @@ -96,10 +97,12 @@ TiB sized), these operations may take several minutes, or in extreme cases, even hours. If your setup is affected, create and remove snapshots while the VM is shut down, expecting a long task duration. -^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in -a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually -reachable over several paths; xref:storage_multipath[Multipath] describes how -{pve} manages them in a cluster. +^3^: With iSCSI, NVMe over Fabrics (NVMe-oF), Fibre Channel (FC), or SAS block +storage as shared storage in a cluster, LVM is used to split the LUN or +namespace into virtual disks. A LUN is usually reachable over several paths; +xref:storage_multipath[Multipath] describes how {pve} manages them in a cluster. +NVMe-oF namespaces use the kernel's native multipath instead, see +xref:storage_nvmeof[NVMe over Fabrics]. ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs. These snapshots use separate volumes for the snapshot data and layer them. For @@ -153,9 +156,10 @@ is available on all nodes, but it is physically different and can have totally different content. Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM -on a xref:storage_multipath_setup[SAN LUN], use *Guided Setup* under +on a xref:storage_multipath_setup[SAN LUN] or an +xref:storage_nvmeof_setup[NVMe namespace], use *Guided Setup* under *Datacenter -> Storage -> SAN* instead. It creates the storages that are needed -and enables multipath for the LUN. +and enables multipath for a LUN. Storage Pools @@ -506,6 +510,8 @@ See Also * link:/wiki/Storage:_Multipath[Storage: Multipath] +* link:/wiki/Storage:_NVMe_over_Fabrics[Storage: NVMe over Fabrics] + * link:/wiki/Storage:_LVM[Storage: LVM] * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin] @@ -550,6 +556,8 @@ include::pve-storage-iscsidirect.adoc[] include::pve-storage-multipath.adoc[] +include::pve-storage-nvmeof.adoc[] + include::pve-storage-rbd.adoc[] include::pve-storage-cephfs.adoc[]