From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 0DB651FF0B7 for ; Fri, 13 Nov 2026 04:26:03 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C0DCE2193A; Fri, 13 Nov 2026 04:21:59 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH storage,docs,manager 00/17] NVMe over Fabrics storage Date: Fri, 13 Nov 2026 04:04:28 +0100 Message-ID: <20261113030541.1591528-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794539147891 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.083 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YO2WANQHQZD64T22RDVMDBFGAX6RAH7I X-Message-ID-Hash: YO2WANQHQZD64T22RDVMDBFGAX6RAH7I X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A storage type for NVMe subsystems reached over TCP, on top of v4 of the SAN series [0]: namespaces as base volumes of shared LVM, or directly as disk images, with the kernel's native multipath. This is a deliberately small first version: TCP only, without authentication or TLS, so access rests on the host NQN list of the target and a separate storage network. It relies on the kernel to reconnect lost paths and on the SAN series for the LUN inventory, the guided setup and the checks before a namespace is written. The plugin connects what is missing and removes nothing, also not when a portal or the storage is removed: disconnecting a controller is an explicit action per node. Two prerequisites from the chapter: nodes cloned from one image need distinct host NQNs before they connect. And as with every new storage type, all nodes need the upgrade before the first such storage is added, as a node that does not know the type drops its definitions when it writes storage.cfg. I ran into that in the lab. What it adds: - storage type 'nvmeof' with a subsystem NQN and a list of TCP portals. Each node connects the portals it has no controller for and leaves existing controllers as they are. New controllers get ctrl_loss_tmo=-1, so the kernel keeps reconnecting, and fail I/O on a lost path after 120 seconds unless configured otherwise. Volumes are named nvme-. - LVM accepts such a volume as base, with the reference check and the per-LUN lock of the SAN series. - discovery: /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof'. - /nodes/{node}/nvmeof: host identity, controller list, and removal of one controller. That is refused if a namespace that is mounted, has a holder or is referenced by a storage would lose its last usable path. A storage that still lists the portal connects again. - the node report and with it the cluster-wide status of the SAN series include the namespaces of nvmeof storages, with their state on each node of the storage. They do not need multipathd. - web UI: the Add Storage menu of the SAN panel gets a third entry for shared LVM on a namespace, which ends with the state of the namespace on every node like the other two; Datacenter > Storage > Add gets the dialog of the storage type, with subsystem discovery and the note pointing to the SAN panel; Node > SAN lists the controllers; the SAN panel shows the host NQN of each node. - nvme-cli becomes a Recommends of libpve-storage-perl. Namespaces already show up in the LUN list of Node > SAN through the SAN series, with their paths. Patches: - storage 01-09: SMART skip for fabrics namespaces, the plugin, LVM base, discovery, namespace attribution in the LUN inventory, the node API, the host NQN and the namespaces in the node report - docs 10: the storage chapter - manager 11-17: API mount, storage type and its dialog, LVM base selector, guided setup, controller list, host NQN column, native multipath in the LUN list of the SAN panel I tested this on a six-node nested cluster against a Linux nvmet target and a NetApp AFF A20, each with two portals: the guided setup with a new and with a reused connection storage and with a node that has no access at first, one path down on both targets, a NetApp path cut and a stopped multipathd as the cluster-wide status shows them, the plain dialogs on the nvmet target, a guest disk on a NetApp namespace through snapshot, rollback, live migration and a path cut under I/O, wiping a namespace and its refusals on the NetApp, and connecting after a node reboot on both. [0]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/ pve-storage: Thomas Lamprecht (9): diskmanage: skip SMART queries for NVMe fabric namespaces nvmeof: add a storage for NVMe over Fabrics subsystems lvm: allow a volume group on an NVMe over Fabrics namespace api: scan: add NVMe over Fabrics discovery pvesm: add a scan command for NVMe over Fabrics api: scan: san-luns: attribute NVMe namespaces to storage api: add node-level NVMe over Fabrics status and controller removal multipath: report the NVMe host NQN of each node multipath: report NVMe over Fabrics namespaces in the cluster status debian/control | 1 + src/PVE/API2/Makefile | 1 + src/PVE/API2/Multipath.pm | 8 +- src/PVE/API2/NVMeOF.pm | 182 ++++++++++++++ src/PVE/API2/Storage/Scan.pm | 59 +++++ src/PVE/CLI/pvesm.pm | 13 + src/PVE/Diskmanage.pm | 30 ++- src/PVE/Makefile | 1 + src/PVE/Multipath/Status.pm | 64 ++++- src/PVE/NVMeOF.pm | 352 ++++++++++++++++++++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/LVMPlugin.pm | 3 +- src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NVMeOFPlugin.pm | 204 ++++++++++++++++ src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SAN.pm | 38 ++- src/test/Makefile | 7 +- src/test/disklist_test.pm | 14 ++ src/test/run_multipath_api_tests.pl | 103 +++++++- src/test/run_nvmeof_api_tests.pl | 151 ++++++++++++ src/test/run_nvmeof_taint_tests.pl | 65 +++++ src/test/run_nvmeof_tests.pl | 309 ++++++++++++++++++++++++ src/test/run_san_lun_scan_tests.pl | 33 ++- src/test/run_san_nvme_tests.pl | 30 +++ 24 files changed, 1642 insertions(+), 31 deletions(-) pve-docs: Thomas Lamprecht (1): storage: document NVMe over Fabrics storages pve-storage-multipath.adoc | 4 +- pve-storage-nvmeof.adoc | 161 +++++++++++++++++++++++++++++++++++++ pvesm.adoc | 20 +++-- 3 files changed, 177 insertions(+), 8 deletions(-) pve-manager: Thomas Lamprecht (7): api: nodes: mount the NVMe over Fabrics endpoints ui: storage: add NVMe over Fabrics storage ui: lvm: offer NVMe over Fabrics storages as base storage ui: dc: san: add a guided setup for shared LVM on NVMe over Fabrics ui: node: san: list NVMe over Fabrics controllers ui: dc: san: show the NVMe host NQN of each node ui: dc: san: show that a namespace uses native multipath PVE/API2/Nodes.pm | 7 + www/manager6/Makefile | 4 +- www/manager6/SANUtils.js | 7 + www/manager6/Utils.js | 7 + www/manager6/dc/SAN.js | 41 ++++-- www/manager6/dc/StorageView.js | 5 +- www/manager6/node/Config.js | 9 +- www/manager6/node/NVMeOFControllers.js | 115 ++++++++++++++++ www/manager6/storage/IScsiEdit.js | 5 +- www/manager6/storage/LVMEdit.js | 12 +- www/manager6/storage/NVMeOFEdit.js | 130 ++++++++++++++++++ www/manager6/storage/wizard/LunSelect.js | 29 +++- www/manager6/storage/wizard/NodeIdentities.js | 45 +++--- .../storage/wizard/{IScsi.js => Target.js} | 129 +++++++++++------ www/manager6/storage/wizard/Wizard.js | 43 +++--- www/manager6/window/SANLunState.js | 23 +++- 16 files changed, 499 insertions(+), 112 deletions(-)