public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage
@ 2026-11-13  3:04 Thomas Lamprecht
  2026-11-13  3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
                   ` (17 more replies)
  0 siblings, 18 replies; 19+ messages in thread
From: Thomas Lamprecht @ 2026-11-13  3:04 UTC (permalink / raw)
  To: pve-devel

A storage type for NVMe subsystems reached over TCP, on top of v4 of the
SAN series [0]: namespaces as base volumes of shared LVM, or directly
as disk images, with the kernel's native multipath.

This is a deliberately small first version: TCP only, without
authentication or TLS, so access rests on the host NQN list of the
target and a separate storage network. It relies on the kernel to
reconnect lost paths and on the SAN series for the LUN inventory, the
guided setup and the checks before a namespace is written. The plugin
connects what is missing and removes nothing, also not when a portal or
the storage is removed: disconnecting a controller is an explicit
action per node.

Two prerequisites from the chapter: nodes cloned from one image need
distinct host NQNs before they connect. And as with every new storage
type, all nodes need the upgrade before the first such storage is
added, as a node that does not know the type drops its definitions when
it writes storage.cfg. I ran into that in the lab.

What it adds:

 - storage type 'nvmeof' with a subsystem NQN and a list of TCP
   portals. Each node connects the portals it has no controller for and
   leaves existing controllers as they are. New controllers get
   ctrl_loss_tmo=-1, so the kernel keeps reconnecting, and fail I/O on
   a lost path after 120 seconds unless configured otherwise. Volumes
   are named nvme-<wwid>.
 - LVM accepts such a volume as base, with the reference check and the
   per-LUN lock of the SAN series.
 - discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'.
 - /nodes/{node}/nvmeof: host identity, controller list, and removal
   of one controller. That is refused if a namespace that is mounted,
   has a holder or is referenced by a storage would lose its last
   usable path. A storage that still lists the portal connects again.
 - the node report and with it the cluster-wide status of the SAN
   series include the namespaces of nvmeof storages, with their state
   on each node of the storage. They do not need multipathd.
 - web UI: the Add Storage menu of the SAN panel gets a third entry for
   shared LVM on a namespace, which ends with the state of the namespace
   on every node like the other two; Datacenter > Storage > Add gets the
   dialog of the storage type, with subsystem discovery and the note
   pointing to the SAN panel; Node > SAN lists the controllers; the SAN
   panel shows the host NQN of each node.
 - nvme-cli becomes a Recommends of libpve-storage-perl.

Namespaces already show up in the LUN list of Node > SAN through the
SAN series, with their paths.

Patches:

 - storage 01-09: SMART skip for fabrics namespaces, the plugin,
   LVM base, discovery, namespace attribution in the LUN inventory, the
   node API, the host NQN and the namespaces in the node report
 - docs 10: the storage chapter
 - manager 11-17: API mount, storage type and its dialog, LVM base
   selector, guided setup, controller list, host NQN column, native
   multipath in the LUN list of the SAN panel

I tested this on a six-node nested cluster against a Linux nvmet target
and a NetApp AFF A20, each with two portals: the guided setup with a new
and with a reused connection storage and with a node that has no access
at first, one path down on both targets, a NetApp path cut and a stopped
multipathd as the cluster-wide status shows them, the plain dialogs on
the nvmet target, a guest disk on a NetApp namespace through snapshot,
rollback, live migration and a path cut under I/O, wiping a namespace
and its refusals on the NetApp, and connecting after a node reboot on
both.

[0]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/

pve-storage:

Thomas Lamprecht (9):
      diskmanage: skip SMART queries for NVMe fabric namespaces
      nvmeof: add a storage for NVMe over Fabrics subsystems
      lvm: allow a volume group on an NVMe over Fabrics namespace
      api: scan: add NVMe over Fabrics discovery
      pvesm: add a scan command for NVMe over Fabrics
      api: scan: san-luns: attribute NVMe namespaces to storage
      api: add node-level NVMe over Fabrics status and controller removal
      multipath: report the NVMe host NQN of each node
      multipath: report NVMe over Fabrics namespaces in the cluster status

 debian/control                      |   1 +
 src/PVE/API2/Makefile               |   1 +
 src/PVE/API2/Multipath.pm           |   8 +-
 src/PVE/API2/NVMeOF.pm              | 182 ++++++++++++++
 src/PVE/API2/Storage/Scan.pm        |  59 +++++
 src/PVE/CLI/pvesm.pm                |  13 +
 src/PVE/Diskmanage.pm               |  30 ++-
 src/PVE/Makefile                    |   1 +
 src/PVE/Multipath/Status.pm         |  64 ++++-
 src/PVE/NVMeOF.pm                   | 352 ++++++++++++++++++++++++++++
 src/PVE/Storage.pm                  |   2 +
 src/PVE/Storage/LVMPlugin.pm        |   3 +-
 src/PVE/Storage/Makefile            |   1 +
 src/PVE/Storage/NVMeOFPlugin.pm     | 204 ++++++++++++++++
 src/PVE/Storage/Plugin.pm           |   2 +-
 src/PVE/Storage/SAN.pm              |  38 ++-
 src/test/Makefile                   |   7 +-
 src/test/disklist_test.pm           |  14 ++
 src/test/run_multipath_api_tests.pl | 103 +++++++-
 src/test/run_nvmeof_api_tests.pl    | 151 ++++++++++++
 src/test/run_nvmeof_taint_tests.pl  |  65 +++++
 src/test/run_nvmeof_tests.pl        | 309 ++++++++++++++++++++++++
 src/test/run_san_lun_scan_tests.pl  |  33 ++-
 src/test/run_san_nvme_tests.pl      |  30 +++
 24 files changed, 1642 insertions(+), 31 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      storage: document NVMe over Fabrics storages

 pve-storage-multipath.adoc |   4 +-
 pve-storage-nvmeof.adoc    | 161 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  20 +++--
 3 files changed, 177 insertions(+), 8 deletions(-)

pve-manager:

Thomas Lamprecht (7):
      api: nodes: mount the NVMe over Fabrics endpoints
      ui: storage: add NVMe over Fabrics storage
      ui: lvm: offer NVMe over Fabrics storages as base storage
      ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
      ui: node: san: list NVMe over Fabrics controllers
      ui: dc: san: show the NVMe host NQN of each node
      ui: dc: san: show that a namespace uses native multipath

 PVE/API2/Nodes.pm                             |   7 +
 www/manager6/Makefile                         |   4 +-
 www/manager6/SANUtils.js                      |   7 +
 www/manager6/Utils.js                         |   7 +
 www/manager6/dc/SAN.js                        |  41 ++++--
 www/manager6/dc/StorageView.js                |   5 +-
 www/manager6/node/Config.js                   |   9 +-
 www/manager6/node/NVMeOFControllers.js        | 115 ++++++++++++++++
 www/manager6/storage/IScsiEdit.js             |   5 +-
 www/manager6/storage/LVMEdit.js               |  12 +-
 www/manager6/storage/NVMeOFEdit.js            | 130 ++++++++++++++++++
 www/manager6/storage/wizard/LunSelect.js      |  29 +++-
 www/manager6/storage/wizard/NodeIdentities.js |  45 +++---
 .../storage/wizard/{IScsi.js => Target.js}    | 129 +++++++++++------
 www/manager6/storage/wizard/Wizard.js         |  43 +++---
 www/manager6/window/SANLunState.js            |  23 +++-
 16 files changed, 499 insertions(+), 112 deletions(-)




^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2026-11-18 20:46 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13  3:04 [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13  3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13  3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13  3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal