From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id CF4461FF0B2 for ; Sun, 22 Nov 2026 02:27:08 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 435AD2179A; Sun, 22 Nov 2026 02:26:22 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v3 storage,docs,manager 00/18] NVMe over Fabrics storage Date: Sun, 22 Nov 2026 02:25:37 +0100 Message-ID: <20261122012605.1848824-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795310771438 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.004 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: EUSRWXLRRVYQAOKS4QCRPODOLVN7BRQ5 X-Message-ID-Hash: EUSRWXLRRVYQAOKS4QCRPODOLVN7BRQ5 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A storage type for NVMe subsystems over TCP on top of the SAN series [0] (libpve-storage-perl 9.1.22, pve-manager 9.2.36, pve-docs 9.2.20): namespaces as base volumes for shared LVM or directly as disk images, using the kernel's native multipath. This is v3. Besides review fixes for v2 [2], it follows the SAN rework on master: the guided setup lives under Datacenter > Storage > Add and asks for one ID, the node API moved under /nodes/{node}/san, and pvestatd's path notifications cover namespaces too. This is a deliberately small first version: TCP only, without authentication or TLS, relying on the array's host NQN allow-list and a separate storage network. The kernel handles path failover and reconnects, while the plugin only connects missing portals and never disconnects on portal or storage removal; disconnecting a controller is an explicit per-node action. Two prerequisites from the docs: cloned nodes need distinct host NQNs before connecting, and all nodes must be upgraded before adding the first nvmeof storage (older nodes drop unknown storage types when writing storage.cfg). What it adds: - Storage type 'nvmeof' (isolated properties, one storage per subsystem) connecting configured TCP portals with ctrl_loss_tmo=-1 and a configurable fast-io-fail-tmo (default 120s), exposing namespaces as nvme- volumes. - LVM support for nvmeof namespaces as base volumes, reusing the SAN reference checks and per-LUN locking. - Discovery via /nodes/{node}/scan/nvmeof and 'pvesm scan nvmeof', and node management via /nodes/{node}/san/nvmeof (host identity, controller list, and guarded per-controller disconnect refusing the last usable path of any mounted, held, swap, open, or storage-used namespace; as in the SAN series, the open-process check misses opens through another device node). - Cluster status and path-loss notifications covering nvmeof namespaces. Status reporting works without multipathd; notifications still require multipathd to be queryable. - Web UI: guided 'Shared LVM on NVMe over Fabrics' setup under Datacenter > Storage > Add (shared with iSCSI), nvmeof storage dialog with discovery, Node > SAN controller list, and host NQNs in Datacenter > SAN. - pvereport host NQN and subsystem output, plus nvme-cli as a Recommends of libpve-storage-perl. Changes since v2 [2]: - Follow the SAN rework: single-ID wizard deriving '-base', node selection in the host NQN list, portal suggestions without default port 4420, early-close cleanup, and /nodes/{node}/san/nvmeof. - Notify on lost and restored namespace paths in pvestatd. - Tighten activation and locking: fail only if no controller is live (reporting per-portal errors) and warn otherwise; cap extra portal connects at 15s while holding storage.cfg's lock; wait up to 10s for the node fabrics lock; clear portal cooldowns on storage creation. - Enforce WWID-based volume names (model/serial by-id links can alias within a subsystem) and refuse local disk setup on fabric namespace partitions. - Harden Disconnect: also refuse the last usable path of swap namespaces, and pass expected subsystem and portal from the UI to catch reused controller numbers. - Skip the NVMe path walk on nodes without nvmeof storages, and add NVMe-oF info to pvereport. - Validation and UI fixes: require at least one portal, trim and clear wizard portals on discovery portal change, ignore empty or stale scans, skip SMART on NVMe/FC too, leave Add Multipath Entry blank for namespaces, and parse LVM base volume IDs first (fixing IDs like 'nvme-a'). - Docs: explain upgrading all nodes first, document local-only checks for unusable controllers, and list nvmeof in the intro and LVM chapters. The ZFS over NVMe/TCP plugin on the list [3] brings its own initiator. Because 'nvmeof' isolates its properties, names do not clash; sharing connect helpers can follow once both are in. Packaging: pve-manager needs versioned Depends on libpve-storage-perl and pve-docs (plus Build-Depends on pve-doc-generator); libpve-storage-perl should break pve-manager without patch 18. Testing builds on the v1 [1] and v2 [2] runs on a six-node nested cluster (dual-portal Linux nvmet and NetApp AFF A20). For v3 on master, I re-ran both guided setups from Add (new nvmet at 1/1, reused NetApp at 2/2), cut one NetApp path for a minute and both for 40s under guest I/O (1/2 and Failed 0/2, storages inactive, I/O requeued cleanly, full recovery), live-migrated a guest with disks on both storages, and disconnected controllers via the API on every node. After the final review fixes, I also verified model/serial by-id refusals, Disconnect refusals for a mismatched subsystem or in-use LVM namespace, and UI controller disconnect. The 15s activation cap, swap and partition refusals, non-NQN subsystem names, and skipped path walk are covered by unit tests only. Earlier on v3, storage status reconnected a controller once another process released the fabrics lock after five seconds. [0]: https://lore.proxmox.com/pve-devel/20261119194655.3163258-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20261113030541.1591528-1-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20261118204302.3100813-1-t.lamprecht@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20261005002609.571-1-joaquinvarela@neatech.ar/ pve-storage: Thomas Lamprecht (9): diskmanage: skip SMART queries for NVMe fabric namespaces nvmeof: add a storage for NVMe over Fabrics subsystems lvm: allow a volume group on an NVMe over Fabrics namespace api: scan: add NVMe over Fabrics discovery pvesm: add a scan command for NVMe over Fabrics api: san luns: attribute NVMe namespaces to storage api: add node-level NVMe over Fabrics status and controller removal multipath: report the NVMe host NQN of each node multipath: report NVMe over Fabrics namespaces in the cluster status ApiChangeLog | 9 +- debian/control | 1 + src/PVE/API2/Makefile | 1 + src/PVE/API2/NVMeOF.pm | 199 +++++++++++ src/PVE/API2/SAN/Cluster.pm | 22 +- src/PVE/API2/SAN/Node.pm | 17 +- src/PVE/API2/Storage/Scan.pm | 49 +++ src/PVE/CLI/pvesm.pm | 13 + src/PVE/Diskmanage.pm | 29 +- src/PVE/Makefile | 1 + src/PVE/Multipath/Status.pm | 72 +++- src/PVE/NVMeOF.pm | 445 +++++++++++++++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/LVMPlugin.pm | 3 +- src/PVE/Storage/Makefile | 1 + src/PVE/Storage/NVMeOFPlugin.pm | 266 ++++++++++++++ src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SAN.pm | 32 +- src/test/Makefile | 6 + src/test/disklist_test.pm | 14 + src/test/run_multipath_api_tests.pl | 132 ++++++- src/test/run_nvmeof_api_tests.pl | 207 +++++++++++ src/test/run_nvmeof_taint_tests.pl | 67 ++++ src/test/run_nvmeof_tests.pl | 533 ++++++++++++++++++++++++++++ src/test/run_san_api_tests.pl | 4 +- src/test/run_san_lun_scan_tests.pl | 41 +++ src/test/run_san_nvme_tests.pl | 33 ++ 27 files changed, 2160 insertions(+), 41 deletions(-) pve-docs: Thomas Lamprecht (1): storage: document NVMe over Fabrics storages pve-gui.adoc | 12 ++- pve-intro.adoc | 3 +- pve-storage-lvm.adoc | 10 +- pve-storage-multipath.adoc | 3 +- pve-storage-nvmeof.adoc | 189 +++++++++++++++++++++++++++++++++++++ pvesm.adoc | 24 +++-- 6 files changed, 222 insertions(+), 19 deletions(-) pve-manager: Thomas Lamprecht (8): report: add the NVMe over Fabrics host NQN and subsystems ui: storage: add NVMe over Fabrics storage ui: lvm: offer NVMe over Fabrics storages as base storage ui: storage: add a guided setup for shared LVM on NVMe over Fabrics ui: node san: list NVMe over Fabrics controllers ui: san: show the NVMe host NQN of each node ui: san: show that a namespace uses native multipath pvestatd: notify on NVMe over Fabrics namespaces as on multipath maps PVE/MultipathNotification.pm | 5 +- PVE/Report.pm | 6 + PVE/Service/pvestatd.pm | 2 +- test/MultipathNotification_test.pl | 3 + www/manager6/Makefile | 4 +- www/manager6/SANUtils.js | 7 +- www/manager6/Utils.js | 7 + www/manager6/dc/SAN.js | 49 +++- www/manager6/dc/StorageView.js | 23 +- www/manager6/node/Config.js | 9 +- www/manager6/node/NVMeOFControllers.js | 152 ++++++++++++ www/manager6/storage/IScsiEdit.js | 5 +- www/manager6/storage/LVMEdit.js | 12 +- www/manager6/storage/NVMeOFEdit.js | 150 +++++++++++ www/manager6/storage/wizard/LunSelect.js | 35 ++- www/manager6/storage/wizard/NodeIdentities.js | 57 +++-- .../storage/wizard/{IScsi.js => Target.js} | 232 +++++++++++++----- www/manager6/storage/wizard/Wizard.js | 60 +++-- www/manager6/window/SANLunState.js | 38 ++- 19 files changed, 699 insertions(+), 157 deletions(-)