From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage
Date: Fri, 13 Nov 2026 04:04:28 +0100 [thread overview]
Message-ID: <20261113030541.1591528-1-t.lamprecht@proxmox.com> (raw)
A storage type for NVMe subsystems reached over TCP, on top of v4 of the
SAN series [0]: namespaces as base volumes of shared LVM, or directly
as disk images, with the kernel's native multipath.
This is a deliberately small first version: TCP only, without
authentication or TLS, so access rests on the host NQN list of the
target and a separate storage network. It relies on the kernel to
reconnect lost paths and on the SAN series for the LUN inventory, the
guided setup and the checks before a namespace is written. The plugin
connects what is missing and removes nothing, also not when a portal or
the storage is removed: disconnecting a controller is an explicit
action per node.
Two prerequisites from the chapter: nodes cloned from one image need
distinct host NQNs before they connect. And as with every new storage
type, all nodes need the upgrade before the first such storage is
added, as a node that does not know the type drops its definitions when
it writes storage.cfg. I ran into that in the lab.
What it adds:
- storage type 'nvmeof' with a subsystem NQN and a list of TCP
portals. Each node connects the portals it has no controller for and
leaves existing controllers as they are. New controllers get
ctrl_loss_tmo=-1, so the kernel keeps reconnecting, and fail I/O on
a lost path after 120 seconds unless configured otherwise. Volumes
are named nvme-<wwid>.
- LVM accepts such a volume as base, with the reference check and the
per-LUN lock of the SAN series.
- discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'.
- /nodes/{node}/nvmeof: host identity, controller list, and removal
of one controller. That is refused if a namespace that is mounted,
has a holder or is referenced by a storage would lose its last
usable path. A storage that still lists the portal connects again.
- the node report and with it the cluster-wide status of the SAN
series include the namespaces of nvmeof storages, with their state
on each node of the storage. They do not need multipathd.
- web UI: the Add Storage menu of the SAN panel gets a third entry for
shared LVM on a namespace, which ends with the state of the namespace
on every node like the other two; Datacenter > Storage > Add gets the
dialog of the storage type, with subsystem discovery and the note
pointing to the SAN panel; Node > SAN lists the controllers; the SAN
panel shows the host NQN of each node.
- nvme-cli becomes a Recommends of libpve-storage-perl.
Namespaces already show up in the LUN list of Node > SAN through the
SAN series, with their paths.
Patches:
- storage 01-09: SMART skip for fabrics namespaces, the plugin,
LVM base, discovery, namespace attribution in the LUN inventory, the
node API, the host NQN and the namespaces in the node report
- docs 10: the storage chapter
- manager 11-17: API mount, storage type and its dialog, LVM base
selector, guided setup, controller list, host NQN column, native
multipath in the LUN list of the SAN panel
I tested this on a six-node nested cluster against a Linux nvmet target
and a NetApp AFF A20, each with two portals: the guided setup with a new
and with a reused connection storage and with a node that has no access
at first, one path down on both targets, a NetApp path cut and a stopped
multipathd as the cluster-wide status shows them, the plain dialogs on
the nvmet target, a guest disk on a NetApp namespace through snapshot,
rollback, live migration and a path cut under I/O, wiping a namespace
and its refusals on the NetApp, and connecting after a node reboot on
both.
[0]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/
pve-storage:
Thomas Lamprecht (9):
diskmanage: skip SMART queries for NVMe fabric namespaces
nvmeof: add a storage for NVMe over Fabrics subsystems
lvm: allow a volume group on an NVMe over Fabrics namespace
api: scan: add NVMe over Fabrics discovery
pvesm: add a scan command for NVMe over Fabrics
api: scan: san-luns: attribute NVMe namespaces to storage
api: add node-level NVMe over Fabrics status and controller removal
multipath: report the NVMe host NQN of each node
multipath: report NVMe over Fabrics namespaces in the cluster status
debian/control | 1 +
src/PVE/API2/Makefile | 1 +
src/PVE/API2/Multipath.pm | 8 +-
src/PVE/API2/NVMeOF.pm | 182 ++++++++++++++
src/PVE/API2/Storage/Scan.pm | 59 +++++
src/PVE/CLI/pvesm.pm | 13 +
src/PVE/Diskmanage.pm | 30 ++-
src/PVE/Makefile | 1 +
src/PVE/Multipath/Status.pm | 64 ++++-
src/PVE/NVMeOF.pm | 352 ++++++++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/LVMPlugin.pm | 3 +-
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 204 ++++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 38 ++-
src/test/Makefile | 7 +-
src/test/disklist_test.pm | 14 ++
src/test/run_multipath_api_tests.pl | 103 +++++++-
src/test/run_nvmeof_api_tests.pl | 151 ++++++++++++
src/test/run_nvmeof_taint_tests.pl | 65 +++++
src/test/run_nvmeof_tests.pl | 309 ++++++++++++++++++++++++
src/test/run_san_lun_scan_tests.pl | 33 ++-
src/test/run_san_nvme_tests.pl | 30 +++
24 files changed, 1642 insertions(+), 31 deletions(-)
pve-docs:
Thomas Lamprecht (1):
storage: document NVMe over Fabrics storages
pve-storage-multipath.adoc | 4 +-
pve-storage-nvmeof.adoc | 161 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 20 +++--
3 files changed, 177 insertions(+), 8 deletions(-)
pve-manager:
Thomas Lamprecht (7):
api: nodes: mount the NVMe over Fabrics endpoints
ui: storage: add NVMe over Fabrics storage
ui: lvm: offer NVMe over Fabrics storages as base storage
ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics
ui: node: san: list NVMe over Fabrics controllers
ui: dc: san: show the NVMe host NQN of each node
ui: dc: san: show that a namespace uses native multipath
PVE/API2/Nodes.pm | 7 +
www/manager6/Makefile | 4 +-
www/manager6/SANUtils.js | 7 +
www/manager6/Utils.js | 7 +
www/manager6/dc/SAN.js | 41 ++++--
www/manager6/dc/StorageView.js | 5 +-
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 115 ++++++++++++++++
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/LVMEdit.js | 12 +-
www/manager6/storage/NVMeOFEdit.js | 130 ++++++++++++++++++
www/manager6/storage/wizard/LunSelect.js | 29 +++-
www/manager6/storage/wizard/NodeIdentities.js | 45 +++---
.../storage/wizard/{IScsi.js => Target.js} | 129 +++++++++++------
www/manager6/storage/wizard/Wizard.js | 43 +++---
www/manager6/window/SANLunState.js | 23 +++-
16 files changed, 499 insertions(+), 112 deletions(-)
next reply other threads:[~2026-11-13 3:26 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 3:04 Thomas Lamprecht [this message]
2026-11-13 3:04 ` [PATCH storage 01/17] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 02/17] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 03/17] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 04/17] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 05/17] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 06/17] api: scan: san-luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 07/17] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 08/17] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH storage 09/17] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-13 3:04 ` [PATCH docs 10/17] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 11/17] api: nodes: mount the NVMe over Fabrics endpoints Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 12/17] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 13/17] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 14/17] ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 15/17] ui: node: san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 16/17] ui: dc: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-13 3:04 ` [PATCH manager 17/17] ui: dc: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-18 20:44 ` superseded: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113030541.1591528-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.