From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 864671FF0DA for ; Thu, 19 Nov 2026 20:49:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D0F1021A44; Thu, 19 Nov 2026 20:48:25 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v8 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Date: Thu, 19 Nov 2026 20:46:05 +0100 Message-ID: <20261119194655.3163258-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795117620972 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.003 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KO2JFEYKA6J2EV52WCATQY2RWNZDFUWW X-Message-ID-Hash: KO2JFEYKA6J2EV52WCATQY2RWNZDFUWW X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is v8 of the combined and reduced series of the cluster-wide multipath work [0][1] and Dietmar's RFC for guided remote storage setup and SAN visibility [2]. If nothing grave comes up I plan to apply this as is. v8 is v7 [9] plus what an independent review of v7 turned up and held up when checked against the code and the lab, i.e.: - A WWID that multipath refuses, for example because the overrides blacklist it, no longer stops the other WWIDs of a node or the reload of the configuration. The node reports it as an error and tries it again after five minutes, or after the overrides change; a multipath that hangs ends the additions of that pass. - The overrides also refuse a '{' that is not on the line of its section name: multipath only warns about that and applies the section, so a multipaths block could get past the check. - A drop-in is only replaced if its first line is exactly the generated one, no longer also with the header of the unreleased v1 to v3. - A wipe that finished no longer ends as a failed task if udev does not settle in time afterwards. - The guided setups are one class per transport, instead of functions on a namespace, and the note of the target step names the Next button through a placeholder for translators. - The setup procedure of the chapter keeps its steps short, with the Fibre Channel details and keeping a volume group after the list. It names multipath_component_detection and says that a portal a node cannot reach keeps the LUNs of the storage Degraded there. - A base volume is read as volume ID first, so a storage ID like 'scsi-a' no longer confuses which LUN an LVM storage has as base in the LUN list and the status. - Smaller ones: descriptions for the fields of the cluster-wide status, if blocks instead of long post-ifs, clearer comments, test descriptions and commit messages, and the chapter says why all nodes need the upgrade first. Blockbridge tested v7 with their plugin again, thanks! On their suggestion, Node > SAN shows the namespace ID of an NVMe namespace in the LUN column, now named LUN/NSID. Their note that a namespace their plugin connected looks like a single-path one once a path is disconnected led to two small changes: an explicit multipath entry for an NVMe namespace is refused, as it showed the namespace as Missing on every node and could not give it an expected path count, and the chapter says that NVMe namespaces have none. A plugin method for the expected paths of plugin volumes could follow as an addition. What I left as is: points decided for v5 [3] to v7, such as the time budget of the cluster lock for slow commands, storage IDs for node auditors and a version check for mixed clusters, and master's behavior of the other node disk endpoints, which keep allowing a Fibre Channel LUN as local disk of one node. Overview of what an admin gets with this: - Datacenter > Storage > SAN, a new entry below Storage, shows the multipath state of every node and of the LUNs that Proxmox VE manages or has a storage on, and edits the multipath entries and array-specific overrides. I explicitly chose this over the client/plugin side of Storage, as we do not have e.g. Ceph or ZFS/LVM/... creation in there either. We might want to rethink and consolidate our storage interface vs. management story in the future, but IMO a partial switch now is not worth it, that would add even more potential for confusion than the new SAN view. - Its Guided Setup menu leads through shared LVM on a LUN: for iSCSI it connects or reuses a target, with the initiator names of its nodes at hand, for Fibre Channel it takes a LUN the nodes already see. Both set up multipath and end with the state of the LUN on every node. - Datacenter > Storage > Add keeps its dialogs. The iSCSI one gets a Multipath option and the LVM one enables Shared for a base storage, and both point to the SAN view. - Node > SAN lists the LUNs of a node with their paths and usage, can wipe a LUN that no storage uses, and shows the iSCSI sessions. The rules the series follows since v4: nodes add WWIDs and maps but never remove them, a write is checked where it happens, lists do not probe devices, there are no new iSCSI operations, the guided setup makes the API calls an admin would make by hand, and multipath management is turned on by the admin. The v4 cover [7] has the reasons. Two things an admin will likely miss: there is no notification when a LUN loses path redundancy, the state only shows on the SAN pages and in the API. And Fibre Channel starts at a LUN the nodes already see; the chapter gives the commands to list WWPNs and rescan the HBAs. How it works: - /etc/pve/multipath.cfg holds explicit entries per WWID, /etc/pve/multipath-overrides.conf array settings; patch 01 registers both. An iSCSI storage with the 'managed-multipath' option supplies its LUNs without entries. The option is not named 'multipath', as third-party plugins use that name, and the storage layer leaves out a third-party plugin that declares a property of another plugin. - pvestatd calls one function of pve-storage per pass. It adds the WWIDs the node needs to /etc/multipath/wwids, writes the drop-ins /etc/multipath/conf.d/pve-*.conf, reloads multipathd after a change, and publishes the state of the maps that Proxmox VE manages or that a storage builds on through the node key-value store. - /cluster/multipath/status aggregates these reports; /cluster/multipath/{wwid,overrides} edit the two files; /nodes/{node}/scan/san-luns lists the LUNs of a node and /nodes/{node}/iscsi its sessions. - The volumes of an iSCSI storage with the option are named after the multipath map, so an LVM storage uses them through 'base' as before. - Volume group creation and wipe run under a cluster lock per LUN. They refuse a LUN that a storage uses or a process of the node has open, also through another path. Opens through another device node, such as one of a container, and users on other hosts are not seen. Patches: - cluster 01: register the two configuration files - storage 02-26: helpers for the disk and iSCSI session inventory (mostly Dietmar's), the multipath library and configuration, the iSCSI option, the per-node apply and status with their API, the disk list of many disks, the LUN inventory, the read-only iSCSI status and the portal of an iSCSI path, volume groups on LUNs, the wipe and the process that has a LUN open - docs 27-28: the multipath chapter, rewritten around tasks, and the paragraph for a LUN that a process has open - manager 29-43: the pvestatd call, report and API mounts, then the views, the two dialog changes, the two guided setups, the note in the add dialogs, and the guest or process and the target of a LUN Other changes since v7 [9]: - rebased onto current master When applying, versioned dependencies have to be set: pve-manager on libpve-storage-perl, and libpve-storage-perl (also as build dependency) on the libpve-cluster-perl with patch 01, as PVE::Storage registers the two files when it is loaded. pve-cluster with the same patch keeps the version counters of the two files; with an older pmxcfs they are read and parsed again on every request. pve-manager also needs the pve-doc-generator (build) and pve-docs with the multipath chapter, for its help links. With these versions set, the chapter's request to upgrade all nodes before enabling the management could name them. If the bound for LVM status commands [4] gets applied too, whichever is applied second has to pass the status cache to the two lvm_vgs() calls of the base volume activation, as they can block a status pass otherwise. Both mark the cache of the status update with 'status_only', in the same line. The per-node target and portal series [5][6] are independent of this one. It takes the LUNs of an iSCSI storage from the sessions of its one target, so if a storage can have several targets per node, whichever is applied second has to cover that. I tested v8 on the lab of v4 [7] to v7, upgraded from the v7 build, with the NVMe over Fabrics series and [4] on top. A WWID that the overrides blacklist next to another let the other be added and its map assembled, with the refusal shown for each node in the SAN view; the two forms of a '{' apart from its section name were refused; a drop-in with the old header stayed and the node said why. The iSCSI setup on the NetApp, the Fibre Channel setup and the NVMe over Fabrics setup ran to the end with the new classes, and two wipes ended as OK tasks. With five refused WWIDs, every node named three and counted the other two, and lifting the blacklist of one let it be added in the next pass. An unquoted value with braces in the overrides got the hint to quote it. An LVM storage on a LUN of a second iSCSI storage named 'scsi-n02' showed as its user in the LUN list and the cluster status, and the wipe of the LUN was refused. For the rest, the tests of v7 [9] and v6 [8] still apply. [0]: https://lore.proxmox.com/pve-devel/20260626121000.2095591-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20260703124707.1172980-2-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20260731102156.3947857-1-dietmar@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20261116012355.3187417-1-t.lamprecht@proxmox.com/ [4]: https://lore.proxmox.com/pve-devel/20261111015217.941220-1-t.lamprecht@proxmox.com/ [5]: https://lore.proxmox.com/pve-devel/20260430173220.441001-1-m.limbeck@proxmox.com/ [6]: https://lore.proxmox.com/pve-devel/20260804090819.2136483-1-dietmar@proxmox.com/ [7]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/ [8]: https://lore.proxmox.com/pve-devel/20261118065302.3033937-1-t.lamprecht@proxmox.com/ [9]: https://lore.proxmox.com/pve-devel/20261118202646.3017286-1-t.lamprecht@proxmox.com/ pve-cluster: Thomas Lamprecht (1): cfs: observe the multipath configuration files src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-storage: Dietmar Maurer (8): diskmanage: collect disk transport type from lsblk diskmanage: qualify NVMe over Fabrics transport diskmanage: optionally include iSCSI session devices in disk enumeration diskmanage: collect native NVMe multipath path state iscsi: factor out session device map from device list diskmanage: add helper to list multipath devices pvesm: expose the SAN LUN inventory api: add node-level iSCSI status Thomas Lamprecht (17): multipath: add a library around multipathd multipath: add a cluster-wide configuration iscsi: add an option to manage storage LUNs with multipath lvm: activate a missing base volume before using the volume group multipath: apply the configuration on every node multipath: publish per-node status api: add endpoints for the cluster-wide multipath configuration api: multipath: add the cluster-wide health status diskmanage: look up the requested disks by name diskmanage: read the udev data of all disks with one call api: scan: add a SAN LUN inventory api: scan: name the portal of an iSCSI path disks: lvm: drop unused imports disks: lvm: allow volume groups on SAN LUNs and multipath maps disks: lvm: refuse to destroy the volume group of a shared storage disks: allow wiping a SAN LUN that no storage uses api: scan: report the process that has a SAN LUN open debian/control | 3 +- src/PVE/API2/Disks.pm | 47 +- src/PVE/API2/Disks/LVM.pm | 81 +- src/PVE/API2/ISCSI.pm | 166 +++++ src/PVE/API2/Makefile | 2 + src/PVE/API2/Multipath.pm | 537 +++++++++++++ src/PVE/API2/Storage/Scan.pm | 244 ++++++ src/PVE/CLI/pvesm.pm | 15 + src/PVE/Diskmanage.pm | 386 ++++++++-- src/PVE/ISCSI/Topology.pm | 89 +++ src/PVE/Makefile | 6 + src/PVE/Multipath.pm | 213 ++++++ src/PVE/Multipath/Config.pm | 215 ++++++ src/PVE/Multipath/Generator.pm | 156 ++++ src/PVE/Multipath/Identity.pm | 44 ++ src/PVE/Multipath/Status.pm | 229 ++++++ src/PVE/Storage.pm | 2 +- src/PVE/Storage/ISCSIPlugin.pm | 219 ++++-- src/PVE/Storage/LVMPlugin.pm | 158 +++- src/PVE/Storage/Makefile | 3 +- src/PVE/Storage/SAN.pm | 463 ++++++++++++ src/test/Makefile | 37 +- src/test/disk_api_test.pm | 314 ++++++++ src/test/disk_tests/iscsi/disklist | 1 + .../disk_tests/iscsi/disklist_expected.json | 19 + src/test/disk_tests/iscsi/lsblk | 5 + src/test/disk_tests/iscsi/sda/device/model | 1 + src/test/disk_tests/iscsi/sda/device/vendor | 1 + .../disk_tests/iscsi/sda/queue/rotational | 1 + src/test/disk_tests/iscsi/sda/size | 1 + src/test/disk_tests/iscsi/sda_udevadm | 21 + src/test/disk_tests/multipath/disklist | 11 + .../multipath/disklist_expected.json | 107 +++ src/test/disk_tests/multipath/dm-0/dm/name | 1 + src/test/disk_tests/multipath/dm-0/dm/uuid | 1 + src/test/disk_tests/multipath/dm-0/size | 1 + src/test/disk_tests/multipath/dm-0_slaves | 4 + src/test/disk_tests/multipath/dm-1/dm/name | 1 + src/test/disk_tests/multipath/dm-1/dm/uuid | 1 + src/test/disk_tests/multipath/dm-1/size | 1 + src/test/disk_tests/multipath/dm-1_slaves | 4 + src/test/disk_tests/multipath/dm-2/dm/name | 1 + src/test/disk_tests/multipath/dm-2/dm/uuid | 1 + src/test/disk_tests/multipath/dm-2/size | 1 + src/test/disk_tests/multipath/dm-2_slaves | 0 src/test/disk_tests/multipath/dm-3/dm/name | 1 + src/test/disk_tests/multipath/dm-3/dm/uuid | 1 + src/test/disk_tests/multipath/dm-3/size | 1 + src/test/disk_tests/multipath/dm-3_slaves | 0 src/test/disk_tests/multipath/dm-4/dm/name | 1 + src/test/disk_tests/multipath/dm-4/dm/uuid | 1 + src/test/disk_tests/multipath/dm-4/size | 1 + src/test/disk_tests/multipath/dm-4_slaves | 0 src/test/disk_tests/multipath/lsblk | 77 ++ src/test/disk_tests/multipath/lvs | 0 src/test/disk_tests/multipath/mounts | 1 + .../multipath/multipath_expected.json | 59 ++ src/test/disk_tests/multipath/multipathd | 124 +++ .../multipath/multipathd_status_expected.json | 40 + .../nvme-subsystem/nvme-subsys0/nvme3/address | 1 + .../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme3/state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/address | 1 + .../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/state | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/address | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/state | 1 + .../disk_tests/multipath/nvme8n1/device/model | 1 + .../multipath/nvme8n1/device/transport | 1 + .../multipath/nvme8n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme8n1/size | 1 + src/test/disk_tests/multipath/nvme8n1_udevadm | 14 + .../disk_tests/multipath/nvme9n1/device/model | 1 + .../multipath/nvme9n1/device/nvme5/transport | 1 + .../multipath/nvme9n1/device/vendor | 1 + .../multipath/nvme9n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme9n1/size | 1 + src/test/disk_tests/multipath/nvme9n1_udevadm | 12 + .../multipath/nvme_path_status_expected.json | 19 + src/test/disk_tests/multipath/partlist | 0 src/test/disk_tests/multipath/pvs | 1 + .../disk_tests/multipath/sda/device/model | 1 + .../disk_tests/multipath/sda/device/vendor | 1 + .../disk_tests/multipath/sda/holders/holder | 1 + .../disk_tests/multipath/sda/queue/rotational | 1 + src/test/disk_tests/multipath/sda/size | 1 + src/test/disk_tests/multipath/sda_udevadm | 18 + .../disk_tests/multipath/sdb/device/model | 1 + .../disk_tests/multipath/sdb/device/vendor | 1 + .../disk_tests/multipath/sdb/holders/holder | 1 + .../disk_tests/multipath/sdb/queue/rotational | 1 + src/test/disk_tests/multipath/sdb/size | 1 + src/test/disk_tests/multipath/sdb_udevadm | 18 + .../disk_tests/multipath/sdc/device/model | 1 + .../disk_tests/multipath/sdc/device/vendor | 1 + .../disk_tests/multipath/sdc/holders/holder | 1 + .../disk_tests/multipath/sdc/queue/rotational | 1 + src/test/disk_tests/multipath/sdc/size | 1 + src/test/disk_tests/multipath/sdc_udevadm | 18 + .../disk_tests/multipath/sdd/device/model | 1 + .../disk_tests/multipath/sdd/device/vendor | 1 + .../disk_tests/multipath/sdd/holders/holder | 1 + .../disk_tests/multipath/sdd/queue/rotational | 1 + src/test/disk_tests/multipath/sdd/size | 1 + src/test/disk_tests/multipath/sdd_udevadm | 18 + src/test/disk_tests/multipath/zpool | 1 + .../disk_tests/sas/disklist_expected.json | 1 + src/test/disk_tests/sas/lsblk | 5 + src/test/disklist_test.pm | 179 ++++- src/test/multipath-three-states.json | 52 ++ src/test/plugin_properties_test.pm | 1 + src/test/run_disk_api_tests.pl | 11 + src/test/run_iscsi_activation_tests.pl | 109 +++ src/test/run_lun_identity_tests.pl | 57 ++ src/test/run_multipath_api_tests.pl | 705 ++++++++++++++++++ src/test/run_multipath_tests.pl | 194 +++++ src/test/run_san_api_tests.pl | 124 +++ src/test/run_san_library_tests.pl | 135 ++++ src/test/run_san_lun_scan_tests.pl | 338 +++++++++ src/test/run_san_nvme_tests.pl | 122 +++ src/test/run_san_taint_tests.pl | 60 ++ src/test/run_san_wipe_tests.pl | 271 +++++++ src/test/run_san_write_check_tests.pl | 219 ++++++ src/test/run_test_lvm_base_activation.pl | 121 +++ src/test/run_test_lvm_base_volume.pl | 16 +- src/test/run_test_lvm_create.pl | 46 ++ 126 files changed, 6846 insertions(+), 197 deletions(-) pve-docs: Thomas Lamprecht (2): storage: document the cluster-wide multipath management storage: multipath: describe the use shown for an open LUN pmxcfs.adoc | 2 + pve-storage-iscsi.adoc | 11 ++ pve-storage-iscsidirect.adoc | 2 +- pve-storage-lvm.adoc | 3 + pve-storage-multipath.adoc | 329 +++++++++++++++++++++++++++++++++++ pvesm.adoc | 18 +- 6 files changed, 359 insertions(+), 6 deletions(-) pve-manager: Dietmar Maurer (2): api: nodes: expose the iSCSI status ui: node: san: show iSCSI targets and sessions Thomas Lamprecht (13): pvestatd: apply the multipath configuration and report the node status report: add the managed multipath configuration api: cluster: add the multipath endpoints ui: dc: add a SAN view ui: storage: iscsi: add the multipath option ui: lvm: enable shared when a base storage is selected ui: node: add SAN LUNs panel ui: node: san: wipe a LUN that no storage uses ui: dc: san: add a guided setup for shared LVM on iSCSI ui: dc: san: add a guided setup for shared LVM on Fibre Channel ui: storage: point the iSCSI and LVM add dialogs to the SAN view ui: san: show the guest or process that has a LUN open ui: node: san: add a column for the target or subsystem of a LUN PVE/API2/Cluster.pm | 7 + PVE/API2/Nodes.pm | 7 + PVE/Report.pm | 3 + PVE/Service/pvestatd.pm | 7 + www/manager6/Makefile | 12 + www/manager6/SANUtils.js | 239 +++++++++++ www/manager6/Utils.js | 19 + www/manager6/dc/Config.js | 23 +- www/manager6/dc/MultipathWwidEdit.js | 81 ++++ www/manager6/dc/SAN.js | 398 ++++++++++++++++++ www/manager6/node/Config.js | 24 ++ www/manager6/node/ISCSITargets.js | 150 +++++++ www/manager6/node/LVM.js | 5 + www/manager6/node/SANLuns.js | 233 ++++++++++ www/manager6/storage/Base.js | 5 + www/manager6/storage/IScsiEdit.js | 46 ++ www/manager6/storage/LVMEdit.js | 6 + www/manager6/storage/wizard/FibreChannel.js | 196 +++++++++ www/manager6/storage/wizard/IScsi.js | 306 ++++++++++++++ www/manager6/storage/wizard/LunSelect.js | 224 ++++++++++ www/manager6/storage/wizard/NodeIdentities.js | 94 +++++ www/manager6/storage/wizard/Wizard.js | 295 +++++++++++++ www/manager6/window/SANLunState.js | 119 ++++++ www/manager6/window/SANLunWipe.js | 64 +++ 24 files changed, 2557 insertions(+), 6 deletions(-)