From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage
Date: Sun, 22 Nov 2026 02:25:37 +0100 [thread overview]
Message-ID: <20261122012605.1848824-1-t.lamprecht@proxmox.com> (raw)
A storage type for NVMe subsystems over TCP on top of the SAN series
[0] (libpve-storage-perl 9.1.22, pve-manager 9.2.36, pve-docs 9.2.20):
namespaces as base volumes for shared LVM or directly as disk images,
using the kernel's native multipath.
This is v3. Besides review fixes for v2 [2], it follows the SAN rework
on master: the guided setup lives under Datacenter > Storage > Add and
asks for one ID, the node API moved under /nodes/{node}/san, and
pvestatd's path notifications cover namespaces too.
This is a deliberately small first version: TCP only, without
authentication or TLS, relying on the array's host NQN allow-list and
a separate storage network. The kernel handles path failover and
reconnects, while the plugin only connects missing portals and never
disconnects on portal or storage removal; disconnecting a controller
is an explicit per-node action. Two prerequisites from the docs: cloned
nodes need distinct host NQNs before connecting, and all nodes must be
upgraded before adding the first nvmeof storage (older nodes drop
unknown storage types when writing storage.cfg).
What it adds:
- Storage type 'nvmeof' (isolated properties, one storage per
subsystem) connecting configured TCP portals with ctrl_loss_tmo=-1
and a configurable fast-io-fail-tmo (default 120s), exposing
namespaces as nvme-<wwid> volumes.
- LVM support for nvmeof namespaces as base volumes, reusing the SAN
reference checks and per-LUN locking.
- Discovery via /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof', and
node management via /nodes/{node}/san/nvmeof (host identity,
controller list, and guarded per-controller disconnect refusing the
last usable path of any mounted, held, swap, open, or storage-used
namespace; as in the SAN series, the open-process check misses opens
through another device node).
- Cluster status and path-loss notifications covering nvmeof
namespaces. Status reporting works without multipathd; notifications
still require multipathd to be queryable.
- Web UI: guided 'Shared LVM on NVMe over Fabrics' setup under
Datacenter > Storage > Add (shared with iSCSI), nvmeof storage
dialog with discovery, Node > SAN controller list, and host NQNs in
Datacenter > SAN.
- pvereport host NQN and subsystem output, plus nvme-cli as a
Recommends of libpve-storage-perl.
Changes since v2 [2]:
- Follow the SAN rework: single-ID wizard deriving '<ID>-base', node
selection in the host NQN list, portal suggestions without default
port 4420, early-close cleanup, and /nodes/{node}/san/nvmeof.
- Notify on lost and restored namespace paths in pvestatd.
- Tighten activation and locking: fail only if no controller is live
(reporting per-portal errors) and warn otherwise; cap extra portal
connects at 15s while holding storage.cfg's lock; wait up to 10s for
the node fabrics lock; clear portal cooldowns on storage creation.
- Enforce WWID-based volume names (model/serial by-id links can alias
within a subsystem) and refuse local disk setup on fabric namespace
partitions.
- Harden Disconnect: also refuse the last usable path of swap
namespaces, and pass expected subsystem and portal from the UI to
catch reused controller numbers.
- Skip the NVMe path walk on nodes without nvmeof storages, and add
NVMe-oF info to pvereport.
- Validation and UI fixes: require at least one portal, trim and clear
wizard portals on discovery portal change, ignore empty or stale
scans, skip SMART on NVMe/FC too, leave Add Multipath Entry blank
for namespaces, and parse LVM base volume IDs first (fixing IDs like
'nvme-a').
- Docs: explain upgrading all nodes first, document local-only checks
for unusable controllers, and list nvmeof in the intro and LVM
chapters.
The ZFS over NVMe/TCP plugin on the list [3] brings its own initiator.
Because 'nvmeof' isolates its properties, names do not clash; sharing
connect helpers can follow once both are in.
Packaging: pve-manager needs versioned Depends on libpve-storage-perl
and pve-docs (plus Build-Depends on pve-doc-generator);
libpve-storage-perl should break pve-manager without patch 18.
Testing builds on the v1 [1] and v2 [2] runs on a six-node nested
cluster (dual-portal Linux nvmet and NetApp AFF A20). For v3 on master,
I re-ran both guided setups from Add (new nvmet at 1/1, reused NetApp
at 2/2), cut one NetApp path for a minute and both for 40s under guest
I/O (1/2 and Failed 0/2, storages inactive, I/O requeued cleanly, full
recovery), live-migrated a guest with disks on both storages, and
disconnected controllers via the API on every node. After the final
review fixes, I also verified model/serial by-id refusals, Disconnect
refusals for a mismatched subsystem or in-use LVM namespace, and UI
controller disconnect.
The 15s activation cap, swap and partition refusals, non-NQN subsystem
names, and skipped path walk are covered by unit tests only. Earlier on
v3, storage status reconnected a controller once another process
released the fabrics lock after five seconds.
[0]: https://lore.proxmox.com/pve-devel/20261119194655.3163258-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/
[2]: https://lore.proxmox.com/pve-devel/20261118204302.3100813-1-t.lamprecht@proxmox.com/
[3]: https://lore.proxmox.com/pve-devel/20261005002609.571-1-joaquinvarela@neatech.ar/
pve-storage:
Thomas Lamprecht (9):
diskmanage: skip SMART queries for NVMe fabric namespaces
nvmeof: add a storage for NVMe over Fabrics subsystems
lvm: allow a volume group on an NVMe over Fabrics namespace
api: scan: add NVMe over Fabrics discovery
pvesm: add a scan command for NVMe over Fabrics
api: san luns: attribute NVMe namespaces to storage
api: add node-level NVMe over Fabrics status and controller removal
multipath: report the NVMe host NQN of each node
multipath: report NVMe over Fabrics namespaces in the cluster status
ApiChangeLog | 9 +-
debian/control | 1 +
src/PVE/API2/Makefile | 1 +
src/PVE/API2/NVMeOF.pm | 199 +++++++++++
src/PVE/API2/SAN/Cluster.pm | 22 +-
src/PVE/API2/SAN/Node.pm | 17 +-
src/PVE/API2/Storage/Scan.pm | 49 +++
src/PVE/CLI/pvesm.pm | 13 +
src/PVE/Diskmanage.pm | 29 +-
src/PVE/Makefile | 1 +
src/PVE/Multipath/Status.pm | 72 +++-
src/PVE/NVMeOF.pm | 445 +++++++++++++++++++++++
src/PVE/Storage.pm | 2 +
src/PVE/Storage/LVMPlugin.pm | 3 +-
src/PVE/Storage/Makefile | 1 +
src/PVE/Storage/NVMeOFPlugin.pm | 266 ++++++++++++++
src/PVE/Storage/Plugin.pm | 2 +-
src/PVE/Storage/SAN.pm | 32 +-
src/test/Makefile | 6 +
src/test/disklist_test.pm | 14 +
src/test/run_multipath_api_tests.pl | 132 ++++++-
src/test/run_nvmeof_api_tests.pl | 207 +++++++++++
src/test/run_nvmeof_taint_tests.pl | 67 ++++
src/test/run_nvmeof_tests.pl | 533 ++++++++++++++++++++++++++++
src/test/run_san_api_tests.pl | 4 +-
src/test/run_san_lun_scan_tests.pl | 41 +++
src/test/run_san_nvme_tests.pl | 33 ++
27 files changed, 2160 insertions(+), 41 deletions(-)
pve-docs:
Thomas Lamprecht (1):
storage: document NVMe over Fabrics storages
pve-gui.adoc | 12 ++-
pve-intro.adoc | 3 +-
pve-storage-lvm.adoc | 10 +-
pve-storage-multipath.adoc | 3 +-
pve-storage-nvmeof.adoc | 189 +++++++++++++++++++++++++++++++++++++
pvesm.adoc | 24 +++--
6 files changed, 222 insertions(+), 19 deletions(-)
pve-manager:
Thomas Lamprecht (8):
report: add the NVMe over Fabrics host NQN and subsystems
ui: storage: add NVMe over Fabrics storage
ui: lvm: offer NVMe over Fabrics storages as base storage
ui: storage: add a guided setup for shared LVM on NVMe over Fabrics
ui: node san: list NVMe over Fabrics controllers
ui: san: show the NVMe host NQN of each node
ui: san: show that a namespace uses native multipath
pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps
PVE/MultipathNotification.pm | 5 +-
PVE/Report.pm | 6 +
PVE/Service/pvestatd.pm | 2 +-
test/MultipathNotification_test.pl | 3 +
www/manager6/Makefile | 4 +-
www/manager6/SANUtils.js | 7 +-
www/manager6/Utils.js | 7 +
www/manager6/dc/SAN.js | 49 +++-
www/manager6/dc/StorageView.js | 23 +-
www/manager6/node/Config.js | 9 +-
www/manager6/node/NVMeOFControllers.js | 152 ++++++++++++
www/manager6/storage/IScsiEdit.js | 5 +-
www/manager6/storage/LVMEdit.js | 12 +-
www/manager6/storage/NVMeOFEdit.js | 150 +++++++++++
www/manager6/storage/wizard/LunSelect.js | 35 ++-
www/manager6/storage/wizard/NodeIdentities.js | 57 +++--
.../storage/wizard/{IScsi.js => Target.js} | 232 +++++++++++++-----
www/manager6/storage/wizard/Wizard.js | 60 +++--
www/manager6/window/SANLunState.js | 38 ++-
19 files changed, 699 insertions(+), 157 deletions(-)
next reply other threads:[~2026-11-22 1:27 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-22 1:25 Thomas Lamprecht [this message]
2026-11-22 1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22 1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122012605.1848824-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox