all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage
Date: Wed, 18 Nov 2026 21:42:19 +0100	[thread overview]
Message-ID: <20261118204302.3100813-1-t.lamprecht@proxmox.com> (raw)

A storage type for NVMe subsystems reached over TCP, on top of v7 of the
SAN series [0]: namespaces as base volumes of shared LVM, or directly as
disk images, with the kernel's native multipath.

This is v2 [1]. Besides following the SAN series, it adds three
checks, listed first below.

This is a deliberately small first version: TCP only, without
authentication or TLS, so access rests on the host NQN list of the
target and a separate storage network. It relies on the kernel to
reconnect lost paths and on the SAN series for the LUN inventory, the
guided setup and the checks before a namespace is written. The plugin
connects what is missing and removes nothing, also not when a portal or
the storage is removed: disconnecting a controller is an explicit action
per node.

Two prerequisites from the chapter: nodes cloned from one image need
distinct host NQNs before they connect. And as with every new storage
type, all nodes need the upgrade before the first such storage is added,
as a node that does not know the type drops its definitions when it
writes storage.cfg. I ran into that in the lab.

What it adds:

 - storage type 'nvmeof' with a subsystem NQN and a list of TCP portals,
   one storage per subsystem. Each node connects the portals it has no
   controller for and leaves existing controllers as they are. New
   controllers get ctrl_loss_tmo=-1, so the kernel keeps reconnecting,
   and fail I/O on a lost path after 120 seconds unless configured
   otherwise. Volumes are named nvme-<wwid>.
 - LVM accepts such a volume as base, with the reference check and the
   per-LUN lock of the SAN series.
 - discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'.
 - /nodes/{node}/nvmeof: host identity, controller list, and removal of
   one controller. That is refused if a namespace that is mounted, has a
   holder, is open in a process or is referenced by a storage would lose
   its last usable path. As in the SAN series, the process check misses
   an open through another device node. A storage that still lists the
   portal connects again.
 - the node report and with it the cluster-wide status of the SAN series
   include the namespaces of nvmeof storages, with their state on each
   node of the storage. They do not need multipathd.
 - web UI: the Guided Setup menu of the SAN view gets a third entry for
   shared LVM on a namespace, which ends with the state of the namespace
   on every node like the other two; Datacenter > Storage > Add gets the
   dialog of the storage type, with subsystem discovery and the note
   pointing to the SAN view; Node > SAN lists the controllers, next to
   the namespaces that the SAN series lists with their paths; the SAN
   view shows the host NQN of each node.
 - nvme-cli becomes a Recommends of libpve-storage-perl.

Patches:

 - storage 01-09: SMART skip for TCP and RDMA namespaces, the
   plugin, LVM base, discovery, namespace attribution in the LUN
   inventory, the node API, the host NQN and the namespaces in the node
   report
 - docs 10: the storage chapter
 - manager 11-17: API mount, storage type and its dialog, LVM base
   selector, guided setup, controller list, host NQN column, native
   multipath in the LUN list of the SAN view

Changes since v1 [1]:

 - the controller removal also refuses when a process has a namespace
   open, and refuses a controller whose namespaces do not use native
   NVMe multipath, as its checks go by their head devices
 - resolving a volume ID to its path checks an existing device against
   the subsystem of the storage, not only activation: a volume name can
   be the by-id name of any NVMe device, and a live import, for example,
   reads its source without activating it
 - a second storage for the same subsystem is refused. The LUN list, the
   controller list and the node report take the first storage they find,
   so namespaces showed as unused next to a connection storage. One
   storage lists all portals instead
 - the discovery in the add dialog can run on another node, as for iSCSI
 - the chapter lists the trusted storage network as a prerequisite
 - NVMe/FC is no longer offered in the Fibre Channel setup, which covers
   SCSI LUNs only; the chapter says how to set up such a namespace
 - the controller list counts the namespaces in one pass: 8.4 seconds
   for 1000 controllers before, 0.07 now, found by a plugin vendor
 - the status update does not wait for a missing namespace, as in the
   SAN series, and the hint of the namespace step speaks of a namespace
   in use
 - the subsystem and ID of a namespace in the LUN inventory, and the
   subsystem in the target column of the LUN list of a node, come from
   v7 of the SAN series now, which shows them for any namespace
 - rebased onto v7 of the SAN series and plugin API 17, with 'vm-vol'
   content and the plugin metadata, and with the namespace variants of
   the setup texts that v7 changed
 - the plugin isolates its properties, so they need no prefix:
   'nvme-portals' and 'nvme-fast-io-fail-tmo' are now 'portals' and
   'fast-io-fail-tmo', and a third-party plugin can declare these names
   and 'subsysnqn' for itself. Being a new type, it has no
   configurations to carry over. As the first in-tree plugin that
   isolates, it makes the storage API check and describe the create and
   update parameters per type, as an installed isolating third-party
   plugin already does. For the in-tree types, the web UI and pvesm work
   as before. A third-party plugin now has to list the sensitive
   properties it uses in options() on every installation, which the
   TrueNAS, Lightbits, simplyblock and Pure Storage plugins do. The API
   viewer lists 186 parameters for creating a storage instead of 66, one
   per property and type, and the pvesm man page grows from 880 to 1108
   lines. pve-api-types does not generate these two calls
 - the namespace ID left the content listing, where nothing read it;
   style fixes, and the plugin module uses v5.36 and signatures

When applying, pve-manager needs a versioned dependency on the
libpve-storage-perl with this series, as its node API loads the new
PVE::API2::NVMeOF, and on the pve-doc-generator (build) and pve-docs
with the NVMe over Fabrics chapter, for its help links.

For v1 I tested this on a six-node nested cluster against a Linux nvmet
target and a NetApp AFF A20, each with two portals: both guided setup
variants, also with a node without access at first, path faults and a
stopped multipathd in the cluster-wide status, a guest on shared LVM on
a NetApp namespace through snapshot, rollback, live migration and a path
cut under I/O, the wipe and its refusals, and reconnecting after a node
reboot.

For v2 I upgraded that cluster and ran the new refusals (a process on
the only path, a second storage for a subsystem from CLI, API and
dialog), the discovery through another node, and a volume named after a
namespace of another subsystem. After the rebase onto v6 of the SAN
series, the guided setup ran again on the nvmet target, and the guest of
the SAN series' test had a data disk on such a namespace through
snapshots, live migrations and a node reboot. The controller list showed
1000 controllers in the web UI. The refusal for a controller outside of
native multipath and the status update that does not wait have unit
tests only. With the isolated properties, I added, edited and removed
such a storage from the CLI and the web UI, ran the guided setup up to a
new connection storage, and removed a NetApp controller on one node,
which connected again from the portals of its storage. On the build on
top of v7 of the SAN series, a guest with a partition of a namespace as
disk showed in the LUN list and kept the wipe from running.

[0]: https://lore.proxmox.com/pve-devel/20261118202646.3017286-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/

pve-storage:

Thomas Lamprecht (9):
      diskmanage: skip SMART queries for NVMe fabric namespaces
      nvmeof: add a storage for NVMe over Fabrics subsystems
      lvm: allow a volume group on an NVMe over Fabrics namespace
      api: scan: add NVMe over Fabrics discovery
      pvesm: add a scan command for NVMe over Fabrics
      api: scan: san-luns: attribute NVMe namespaces to storage
      api: add node-level NVMe over Fabrics status and controller removal
      multipath: report the NVMe host NQN of each node
      multipath: report NVMe over Fabrics namespaces in the cluster status

 ApiChangeLog                        |   7 +-
 debian/control                      |   1 +
 src/PVE/API2/Makefile               |   1 +
 src/PVE/API2/Multipath.pm           |  13 +-
 src/PVE/API2/NVMeOF.pm              | 182 +++++++++++++
 src/PVE/API2/Storage/Scan.pm        |  58 ++++-
 src/PVE/CLI/pvesm.pm                |  13 +
 src/PVE/Diskmanage.pm               |  22 +-
 src/PVE/Makefile                    |   1 +
 src/PVE/Multipath/Status.pm         |  63 ++++-
 src/PVE/NVMeOF.pm                   | 380 ++++++++++++++++++++++++++++
 src/PVE/Storage.pm                  |   2 +
 src/PVE/Storage/LVMPlugin.pm        |   3 +-
 src/PVE/Storage/Makefile            |   1 +
 src/PVE/Storage/NVMeOFPlugin.pm     | 223 ++++++++++++++++
 src/PVE/Storage/Plugin.pm           |   2 +-
 src/PVE/Storage/SAN.pm              |  29 ++-
 src/test/Makefile                   |   7 +-
 src/test/disklist_test.pm           |  14 +
 src/test/run_multipath_api_tests.pl | 107 +++++++-
 src/test/run_nvmeof_api_tests.pl    | 178 +++++++++++++
 src/test/run_nvmeof_taint_tests.pl  |  65 +++++
 src/test/run_nvmeof_tests.pl        | 375 +++++++++++++++++++++++++++
 src/test/run_san_lun_scan_tests.pl  |  28 ++
 src/test/run_san_nvme_tests.pl      |  33 +++
 25 files changed, 1770 insertions(+), 38 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      storage: document NVMe over Fabrics storages

 pve-storage-multipath.adoc |   4 +-
 pve-storage-nvmeof.adoc    | 167 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  20 +++--
 3 files changed, 183 insertions(+), 8 deletions(-)

pve-manager:

Thomas Lamprecht (7):
      api: nodes: mount the NVMe over Fabrics endpoints
      ui: storage: add NVMe over Fabrics storage
      ui: lvm: offer NVMe over Fabrics storages as base storage
      ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
      ui: node: san: list NVMe over Fabrics controllers
      ui: dc: san: show the NVMe host NQN of each node
      ui: dc: san: show that a namespace uses native multipath

 PVE/API2/Nodes.pm                             |   7 +
 www/manager6/Makefile                         |   4 +-
 www/manager6/SANUtils.js                      |   6 +
 www/manager6/Utils.js                         |   7 +
 www/manager6/dc/SAN.js                        |  41 +++--
 www/manager6/dc/StorageView.js                |   5 +-
 www/manager6/node/Config.js                   |   9 +-
 www/manager6/node/NVMeOFControllers.js        | 115 +++++++++++++
 www/manager6/storage/IScsiEdit.js             |   5 +-
 www/manager6/storage/LVMEdit.js               |  12 +-
 www/manager6/storage/NVMeOFEdit.js            | 140 ++++++++++++++++
 www/manager6/storage/wizard/LunSelect.js      |  34 ++--
 www/manager6/storage/wizard/NodeIdentities.js |  45 +++---
 .../storage/wizard/{IScsi.js => Target.js}    | 153 ++++++++++++------
 www/manager6/storage/wizard/Wizard.js         |  43 ++---
 www/manager6/window/SANLunState.js            |  23 ++-
 16 files changed, 529 insertions(+), 120 deletions(-)




             reply	other threads:[~2026-11-18 20:44 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 20:42 Thomas Lamprecht [this message]
2026-11-18 20:42 ` [PATCH v2 storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-18 20:42 ` [PATCH v2 manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118204302.3100813-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal