public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage
Date: Sun, 22 Nov 2026 02:25:37 +0100	[thread overview]
Message-ID: <20261122012605.1848824-1-t.lamprecht@proxmox.com> (raw)

A storage type for NVMe subsystems over TCP on top of the SAN series
[0] (libpve-storage-perl 9.1.22, pve-manager 9.2.36, pve-docs 9.2.20):
namespaces as base volumes for shared LVM or directly as disk images,
using the kernel's native multipath.

This is v3. Besides review fixes for v2 [2], it follows the SAN rework
on master: the guided setup lives under Datacenter > Storage > Add and
asks for one ID, the node API moved under /nodes/{node}/san, and
pvestatd's path notifications cover namespaces too.

This is a deliberately small first version: TCP only, without
authentication or TLS, relying on the array's host NQN allow-list and
a separate storage network. The kernel handles path failover and
reconnects, while the plugin only connects missing portals and never
disconnects on portal or storage removal; disconnecting a controller
is an explicit per-node action. Two prerequisites from the docs: cloned
nodes need distinct host NQNs before connecting, and all nodes must be
upgraded before adding the first nvmeof storage (older nodes drop
unknown storage types when writing storage.cfg).

What it adds:

 - Storage type 'nvmeof' (isolated properties, one storage per
   subsystem) connecting configured TCP portals with ctrl_loss_tmo=-1
   and a configurable fast-io-fail-tmo (default 120s), exposing
   namespaces as nvme-<wwid> volumes.
 - LVM support for nvmeof namespaces as base volumes, reusing the SAN
   reference checks and per-LUN locking.
 - Discovery via /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof', and
   node management via /nodes/{node}/san/nvmeof (host identity,
   controller list, and guarded per-controller disconnect refusing the
   last usable path of any mounted, held, swap, open, or storage-used
   namespace; as in the SAN series, the open-process check misses opens
   through another device node).
 - Cluster status and path-loss notifications covering nvmeof
   namespaces. Status reporting works without multipathd; notifications
   still require multipathd to be queryable.
 - Web UI: guided 'Shared LVM on NVMe over Fabrics' setup under
   Datacenter > Storage > Add (shared with iSCSI), nvmeof storage
   dialog with discovery, Node > SAN controller list, and host NQNs in
   Datacenter > SAN.
 - pvereport host NQN and subsystem output, plus nvme-cli as a
   Recommends of libpve-storage-perl.

Changes since v2 [2]:

 - Follow the SAN rework: single-ID wizard deriving '<ID>-base', node
   selection in the host NQN list, portal suggestions without default
   port 4420, early-close cleanup, and /nodes/{node}/san/nvmeof.
 - Notify on lost and restored namespace paths in pvestatd.
 - Tighten activation and locking: fail only if no controller is live
   (reporting per-portal errors) and warn otherwise; cap extra portal
   connects at 15s while holding storage.cfg's lock; wait up to 10s for
   the node fabrics lock; clear portal cooldowns on storage creation.
 - Enforce WWID-based volume names (model/serial by-id links can alias
   within a subsystem) and refuse local disk setup on fabric namespace
   partitions.
 - Harden Disconnect: also refuse the last usable path of swap
   namespaces, and pass expected subsystem and portal from the UI to
   catch reused controller numbers.
 - Skip the NVMe path walk on nodes without nvmeof storages, and add
   NVMe-oF info to pvereport.
 - Validation and UI fixes: require at least one portal, trim and clear
   wizard portals on discovery portal change, ignore empty or stale
   scans, skip SMART on NVMe/FC too, leave Add Multipath Entry blank
   for namespaces, and parse LVM base volume IDs first (fixing IDs like
   'nvme-a').
 - Docs: explain upgrading all nodes first, document local-only checks
   for unusable controllers, and list nvmeof in the intro and LVM
   chapters.

The ZFS over NVMe/TCP plugin on the list [3] brings its own initiator.
Because 'nvmeof' isolates its properties, names do not clash; sharing
connect helpers can follow once both are in.

Packaging: pve-manager needs versioned Depends on libpve-storage-perl
and pve-docs (plus Build-Depends on pve-doc-generator);
libpve-storage-perl should break pve-manager without patch 18.

Testing builds on the v1 [1] and v2 [2] runs on a six-node nested
cluster (dual-portal Linux nvmet and NetApp AFF A20). For v3 on master,
I re-ran both guided setups from Add (new nvmet at 1/1, reused NetApp
at 2/2), cut one NetApp path for a minute and both for 40s under guest
I/O (1/2 and Failed 0/2, storages inactive, I/O requeued cleanly, full
recovery), live-migrated a guest with disks on both storages, and
disconnected controllers via the API on every node. After the final
review fixes, I also verified model/serial by-id refusals, Disconnect
refusals for a mismatched subsystem or in-use LVM namespace, and UI
controller disconnect.

The 15s activation cap, swap and partition refusals, non-NQN subsystem
names, and skipped path walk are covered by unit tests only. Earlier on
v3, storage status reconnected a controller once another process
released the fabrics lock after five seconds.

[0]: https://lore.proxmox.com/pve-devel/20261119194655.3163258-1-t.lamprecht@proxmox.com/
[1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/
[2]: https://lore.proxmox.com/pve-devel/20261118204302.3100813-1-t.lamprecht@proxmox.com/
[3]: https://lore.proxmox.com/pve-devel/20261005002609.571-1-joaquinvarela@neatech.ar/

pve-storage:

Thomas Lamprecht (9):
      diskmanage: skip SMART queries for NVMe fabric namespaces
      nvmeof: add a storage for NVMe over Fabrics subsystems
      lvm: allow a volume group on an NVMe over Fabrics namespace
      api: scan: add NVMe over Fabrics discovery
      pvesm: add a scan command for NVMe over Fabrics
      api: san luns: attribute NVMe namespaces to storage
      api: add node-level NVMe over Fabrics status and controller removal
      multipath: report the NVMe host NQN of each node
      multipath: report NVMe over Fabrics namespaces in the cluster status

 ApiChangeLog                        |   9 +-
 debian/control                      |   1 +
 src/PVE/API2/Makefile               |   1 +
 src/PVE/API2/NVMeOF.pm              | 199 +++++++++++
 src/PVE/API2/SAN/Cluster.pm         |  22 +-
 src/PVE/API2/SAN/Node.pm            |  17 +-
 src/PVE/API2/Storage/Scan.pm        |  49 +++
 src/PVE/CLI/pvesm.pm                |  13 +
 src/PVE/Diskmanage.pm               |  29 +-
 src/PVE/Makefile                    |   1 +
 src/PVE/Multipath/Status.pm         |  72 +++-
 src/PVE/NVMeOF.pm                   | 445 +++++++++++++++++++++++
 src/PVE/Storage.pm                  |   2 +
 src/PVE/Storage/LVMPlugin.pm        |   3 +-
 src/PVE/Storage/Makefile            |   1 +
 src/PVE/Storage/NVMeOFPlugin.pm     | 266 ++++++++++++++
 src/PVE/Storage/Plugin.pm           |   2 +-
 src/PVE/Storage/SAN.pm              |  32 +-
 src/test/Makefile                   |   6 +
 src/test/disklist_test.pm           |  14 +
 src/test/run_multipath_api_tests.pl | 132 ++++++-
 src/test/run_nvmeof_api_tests.pl    | 207 +++++++++++
 src/test/run_nvmeof_taint_tests.pl  |  67 ++++
 src/test/run_nvmeof_tests.pl        | 533 ++++++++++++++++++++++++++++
 src/test/run_san_api_tests.pl       |   4 +-
 src/test/run_san_lun_scan_tests.pl  |  41 +++
 src/test/run_san_nvme_tests.pl      |  33 ++
 27 files changed, 2160 insertions(+), 41 deletions(-)

pve-docs:

Thomas Lamprecht (1):
      storage: document NVMe over Fabrics storages

 pve-gui.adoc               |  12 ++-
 pve-intro.adoc             |   3 +-
 pve-storage-lvm.adoc       |  10 +-
 pve-storage-multipath.adoc |   3 +-
 pve-storage-nvmeof.adoc    | 189 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  24 +++--
 6 files changed, 222 insertions(+), 19 deletions(-)

pve-manager:

Thomas Lamprecht (8):
      report: add the NVMe over Fabrics host NQN and subsystems
      ui: storage: add NVMe over Fabrics storage
      ui: lvm: offer NVMe over Fabrics storages as base storage
      ui: storage: add a guided setup for shared LVM on NVMe over Fabrics
      ui: node san: list NVMe over Fabrics controllers
      ui: san: show the NVMe host NQN of each node
      ui: san: show that a namespace uses native multipath
      pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps

 PVE/MultipathNotification.pm                  |   5 +-
 PVE/Report.pm                                 |   6 +
 PVE/Service/pvestatd.pm                       |   2 +-
 test/MultipathNotification_test.pl            |   3 +
 www/manager6/Makefile                         |   4 +-
 www/manager6/SANUtils.js                      |   7 +-
 www/manager6/Utils.js                         |   7 +
 www/manager6/dc/SAN.js                        |  49 +++-
 www/manager6/dc/StorageView.js                |  23 +-
 www/manager6/node/Config.js                   |   9 +-
 www/manager6/node/NVMeOFControllers.js        | 152 ++++++++++++
 www/manager6/storage/IScsiEdit.js             |   5 +-
 www/manager6/storage/LVMEdit.js               |  12 +-
 www/manager6/storage/NVMeOFEdit.js            | 150 +++++++++++
 www/manager6/storage/wizard/LunSelect.js      |  35 ++-
 www/manager6/storage/wizard/NodeIdentities.js |  57 +++--
 .../storage/wizard/{IScsi.js => Target.js}    | 232 +++++++++++++-----
 www/manager6/storage/wizard/Wizard.js         |  60 +++--
 www/manager6/window/SANLunState.js            |  38 ++-
 19 files changed, 699 insertions(+), 157 deletions(-)




             reply	other threads:[~2026-11-22  1:27 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-22  1:25 Thomas Lamprecht [this message]
2026-11-22  1:25 ` [PATCH v3 storage 01/18] diskmanage: skip SMART queries for NVMe fabric namespaces Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 02/18] nvmeof: add a storage for NVMe over Fabrics subsystems Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 03/18] lvm: allow a volume group on an NVMe over Fabrics namespace Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 04/18] api: scan: add NVMe over Fabrics discovery Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 05/18] pvesm: add a scan command for NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 06/18] api: san luns: attribute NVMe namespaces to storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 07/18] api: add node-level NVMe over Fabrics status and controller removal Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 08/18] multipath: report the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 storage 09/18] multipath: report NVMe over Fabrics namespaces in the cluster status Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 docs 10/18] storage: document NVMe over Fabrics storages Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 11/18] report: add the NVMe over Fabrics host NQN and subsystems Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 12/18] ui: storage: add NVMe over Fabrics storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 13/18] ui: lvm: offer NVMe over Fabrics storages as base storage Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 14/18] ui: storage: add a guided setup for shared LVM on NVMe over Fabrics Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 15/18] ui: node san: list NVMe over Fabrics controllers Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 16/18] ui: san: show the NVMe host NQN of each node Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 17/18] ui: san: show that a namespace uses native multipath Thomas Lamprecht
2026-11-22  1:25 ` [PATCH v3 manager 18/18] pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122012605.1848824-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal