From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 8127F1FF0E3 for ; Wed, 18 Nov 2026 21:44:20 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1D26F218D9; Wed, 18 Nov 2026 21:43:33 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v2 storage,docs,manager 00/17] NVMe over Fabrics storage Date: Wed, 18 Nov 2026 21:42:19 +0100 Message-ID: <20261118204302.3100813-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795034587088 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.013 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WD3D6R2I5FB7QN6LQBEQJPAJKCG3XLC6 X-Message-ID-Hash: WD3D6R2I5FB7QN6LQBEQJPAJKCG3XLC6 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A storage type for NVMe subsystems reached over TCP, on top of v7 of the SAN series [0]: namespaces as base volumes of shared LVM, or directly as disk images, with the kernel's native multipath. This is v2 [1]. Besides following the SAN series, it adds three checks, listed first below. This is a deliberately small first version: TCP only, without authentication or TLS, so access rests on the host NQN list of the target and a separate storage network. It relies on the kernel to reconnect lost paths and on the SAN series for the LUN inventory, the guided setup and the checks before a namespace is written. The plugin connects what is missing and removes nothing, also not when a portal or the storage is removed: disconnecting a controller is an explicit action per node. Two prerequisites from the chapter: nodes cloned from one image need distinct host NQNs before they connect. And as with every new storage type, all nodes need the upgrade before the first such storage is added, as a node that does not know the type drops its definitions when it writes storage.cfg. I ran into that in the lab. What it adds: - storage type 'nvmeof' with a subsystem NQN and a list of TCP portals, one storage per subsystem. Each node connects the portals it has no controller for and leaves existing controllers as they are. New controllers get ctrl_loss_tmo=-1, so the kernel keeps reconnecting, and fail I/O on a lost path after 120 seconds unless configured otherwise. Volumes are named nvme-. - LVM accepts such a volume as base, with the reference check and the per-LUN lock of the SAN series. - discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'. - /nodes/{node}/nvmeof: host identity, controller list, and removal of one controller. That is refused if a namespace that is mounted, has a holder, is open in a process or is referenced by a storage would lose its last usable path. As in the SAN series, the process check misses an open through another device node. A storage that still lists the portal connects again. - the node report and with it the cluster-wide status of the SAN series include the namespaces of nvmeof storages, with their state on each node of the storage. They do not need multipathd. - web UI: the Guided Setup menu of the SAN view gets a third entry for shared LVM on a namespace, which ends with the state of the namespace on every node like the other two; Datacenter > Storage > Add gets the dialog of the storage type, with subsystem discovery and the note pointing to the SAN view; Node > SAN lists the controllers, next to the namespaces that the SAN series lists with their paths; the SAN view shows the host NQN of each node. - nvme-cli becomes a Recommends of libpve-storage-perl. Patches: - storage 01-09: SMART skip for TCP and RDMA namespaces, the plugin, LVM base, discovery, namespace attribution in the LUN inventory, the node API, the host NQN and the namespaces in the node report - docs 10: the storage chapter - manager 11-17: API mount, storage type and its dialog, LVM base selector, guided setup, controller list, host NQN column, native multipath in the LUN list of the SAN view Changes since v1 [1]: - the controller removal also refuses when a process has a namespace open, and refuses a controller whose namespaces do not use native NVMe multipath, as its checks go by their head devices - resolving a volume ID to its path checks an existing device against the subsystem of the storage, not only activation: a volume name can be the by-id name of any NVMe device, and a live import, for example, reads its source without activating it - a second storage for the same subsystem is refused. The LUN list, the controller list and the node report take the first storage they find, so namespaces showed as unused next to a connection storage. One storage lists all portals instead - the discovery in the add dialog can run on another node, as for iSCSI - the chapter lists the trusted storage network as a prerequisite - NVMe/FC is no longer offered in the Fibre Channel setup, which covers SCSI LUNs only; the chapter says how to set up such a namespace - the controller list counts the namespaces in one pass: 8.4 seconds for 1000 controllers before, 0.07 now, found by a plugin vendor - the status update does not wait for a missing namespace, as in the SAN series, and the hint of the namespace step speaks of a namespace in use - the subsystem and ID of a namespace in the LUN inventory, and the subsystem in the target column of the LUN list of a node, come from v7 of the SAN series now, which shows them for any namespace - rebased onto v7 of the SAN series and plugin API 17, with 'vm-vol' content and the plugin metadata, and with the namespace variants of the setup texts that v7 changed - the plugin isolates its properties, so they need no prefix: 'nvme-portals' and 'nvme-fast-io-fail-tmo' are now 'portals' and 'fast-io-fail-tmo', and a third-party plugin can declare these names and 'subsysnqn' for itself. Being a new type, it has no configurations to carry over. As the first in-tree plugin that isolates, it makes the storage API check and describe the create and update parameters per type, as an installed isolating third-party plugin already does. For the in-tree types, the web UI and pvesm work as before. A third-party plugin now has to list the sensitive properties it uses in options() on every installation, which the TrueNAS, Lightbits, simplyblock and Pure Storage plugins do. The API viewer lists 186 parameters for creating a storage instead of 66, one per property and type, and the pvesm man page grows from 880 to 1108 lines. pve-api-types does not generate these two calls - the namespace ID left the content listing, where nothing read it; style fixes, and the plugin module uses v5.36 and signatures When applying, pve-manager needs a versioned dependency on the libpve-storage-perl with this series, as its node API loads the new PVE::API2::NVMeOF, and on the pve-doc-generator (build) and pve-docs with the NVMe over Fabrics chapter, for its help links. For v1 I tested this on a six-node nested cluster against a Linux nvmet target and a NetApp AFF A20, each with two portals: both guided setup variants, also with a node without access at first, path faults and a stopped multipathd in the cluster-wide status, a guest on shared LVM on a NetApp namespace through snapshot, rollback, live migration and a path cut under I/O, the wipe and its refusals, and reconnecting after a node reboot. For v2 I upgraded that cluster and ran the new refusals (a process on the only path, a second storage for a subsystem from CLI, API and dialog), the discovery through another node, and a volume named after a namespace of another subsystem. After the rebase onto v6 of the SAN series, the guided setup ran again on the nvmet target, and the guest of the SAN series' test had a data disk on such a namespace through snapshots, live migrations and a node reboot. The controller list showed 1000 controllers in the web UI. The refusal for a controller outside of native multipath and the status update that does not wait have unit tests only. With the isolated properties, I added, edited and removed such a storage from the CLI and the web UI, ran the guided setup up to a new connection storage, and removed a NetApp controller on one node, which connected again from the portals of its storage. On the build on top of v7 of the SAN series, a guest with a partition of a namespace as disk showed in the LUN list and kept the wipe from running. [0]: https://lore.proxmox.com/pve-devel/20261118202646.3017286-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/ pve-storage: Thomas Lamprecht (9): diskmanage: skip SMART queries for NVMe fabric namespaces nvmeof: add a storage for NVMe over Fabrics subsystems lvm: allow a volume group on an NVMe over Fabrics namespace api: scan: add NVMe over Fabrics discovery pvesm: add a scan command for NVMe over Fabrics api: scan: san-luns: attribute NVMe namespaces to storage api: add node-level NVMe over Fabrics status and controller removal multipath: report the NVMe host NQN of each node multipath: report NVMe over Fabrics namespaces in the cluster status ApiChangeLog | 7 +- debian/control | 1 + src/PVE/API2/Makefile | 1 + src/PVE/API2/Multipath.pm | 13 +- src/PVE/API2/NVMeOF.pm | 182 +++++++++++++ src/PVE/API2/Storage/Scan.pm | 58 ++++- src/PVE/CLI/pvesm.pm | 13 + src/PVE/Diskmanage.pm | 22 +- src/PVE/Makefile | 1 + src/PVE/Multipath/Status.pm | 63 ++++- src/PVE/NVMeOF.pm | 380 ++++++++++++++++++++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/LVMPlugin.pm | 3 +- src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NVMeOFPlugin.pm | 223 ++++++++++++++++ src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SAN.pm | 29 ++- src/test/Makefile | 7 +- src/test/disklist_test.pm | 14 + src/test/run_multipath_api_tests.pl | 107 +++++++- src/test/run_nvmeof_api_tests.pl | 178 +++++++++++++ src/test/run_nvmeof_taint_tests.pl | 65 +++++ src/test/run_nvmeof_tests.pl | 375 +++++++++++++++++++++++++++ src/test/run_san_lun_scan_tests.pl | 28 ++ src/test/run_san_nvme_tests.pl | 33 +++ 25 files changed, 1770 insertions(+), 38 deletions(-) pve-docs: Thomas Lamprecht (1): storage: document NVMe over Fabrics storages pve-storage-multipath.adoc | 4 +- pve-storage-nvmeof.adoc | 167 +++++++++++++++++++++++++++++++++++++ pvesm.adoc | 20 +++-- 3 files changed, 183 insertions(+), 8 deletions(-) pve-manager: Thomas Lamprecht (7): api: nodes: mount the NVMe over Fabrics endpoints ui: storage: add NVMe over Fabrics storage ui: lvm: offer NVMe over Fabrics storages as base storage ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics ui: node: san: list NVMe over Fabrics controllers ui: dc: san: show the NVMe host NQN of each node ui: dc: san: show that a namespace uses native multipath PVE/API2/Nodes.pm | 7 + www/manager6/Makefile | 4 +- www/manager6/SANUtils.js | 6 + www/manager6/Utils.js | 7 + www/manager6/dc/SAN.js | 41 +++-- www/manager6/dc/StorageView.js | 5 +- www/manager6/node/Config.js | 9 +- www/manager6/node/NVMeOFControllers.js | 115 +++++++++++++ www/manager6/storage/IScsiEdit.js | 5 +- www/manager6/storage/LVMEdit.js | 12 +- www/manager6/storage/NVMeOFEdit.js | 140 ++++++++++++++++ www/manager6/storage/wizard/LunSelect.js | 34 ++-- www/manager6/storage/wizard/NodeIdentities.js | 45 +++--- .../storage/wizard/{IScsi.js => Target.js} | 153 ++++++++++++------ www/manager6/storage/wizard/Wizard.js | 43 ++--- www/manager6/window/SANLunState.js | 23 ++- 16 files changed, 529 insertions(+), 120 deletions(-)