From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id DFDED1FF0C0 for ; Wed, 18 Nov 2026 07:55:52 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DD6FE21897; Wed, 18 Nov 2026 07:54:08 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v6 cluster,storage,docs,manager 00/42] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Date: Wed, 18 Nov 2026 07:52:17 +0100 Message-ID: <20261118065302.3033937-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794984786887 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WLCKL5C7P3TNH3GFOFBBVUBW4EPEB556 X-Message-ID-Hash: WLCKL5C7P3TNH3GFOFBBVUBW4EPEB556 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is v6 of the combined and reduced series of the cluster-wide multipath work [0][1] and Dietmar's RFC for guided remote storage setup and SAN visibility [2]. v6 is v5 [3] rebased onto current master, where plugin API 17 with the volume types, the plugin metadata for the web UI and the property isolation for plugins landed underneath, plus what an independent review of v5 turned up: - The status update no longer waits for the missing map of an LVM base volume, nor asks a hanging multipathd for its paths. With three such storages a pass took 32 seconds instead of 1.2 in my lab. Activating a volume for a guest still waits. - An existing volume group on a partition of a multipath map can be added again, also with the Multipath option of the iSCSI storage. - The node disk API only creates the storage of a volume group on a LUN; v5 could join a storage added meanwhile with another volume group. - The Fibre Channel setup waits for the volume group task itself and cannot be closed meanwhile. Closing the task window ended it with a volume group and no storage. - The setup menu also needs Datastore.Audit or Datastore.AllocateSpace. - The first status pass removed a drop-in with one of the generated names under /etc/multipath/conf.d, also one that Proxmox VE had not written. Such a file now stays, and the node reports an error while it needs that name. - The volume group creation and the wipe also refuse a LUN that is in use through another of its SCSI paths, for example by a guest started before the map existed. A name in the request has to lead to the same device after the LUN lookup still. - A node only reported the first of the storages that request a map. - Node > Disks > LVM creates a shared storage on a SAN LUN, on a single path for a Fibre Channel LUN without a map. Its dialog now points to the guided setup. - Enable Multipath under Node > SAN opens the entry dialog, where the admin sees and chooses the nodes, instead of adding an entry for all. - Smaller ones: partition bases and LUNs reached through two targets are attributed to their storage, Node > SAN shows the path counts of a degraded LUN, and the SAN view names the nodes in Unknown. - let the disk wipe take by-id, dm-N and mapper names for any device, so it also wiped a logical volume with a guest disk by such a name. These names are now accepted for a LUN only. BlockBridge was very reactive here and tested v5 with 1000 LUNs or namespaces on a node, one per guest disk. For them LUN inventory took over ten seconds. New patches 16 and 17 look the disks up by name and read the udev data with one call, which brought it to under a second for 1000 NVMe/TCP namespaces in my lab. The test also showed the disks of running guests as unused, as no storage references them. New patches 25, 27, 41 and 42 show the guest or process that has a LUN open for writing, and a hidden column for the iSCSI target, which names the volume for a plugin with one target per volume. They come last in their repositories, so that they can be left out; the NVMe over Fabrics series then applies up to its last manager patch. A LUN that stays attached with nothing holding it, for example after a failed detach, still shows as unused: telling whose it is needs the plugin, and I left that for later. What I left as is for now: lsblk for the usage of a LUN, as it opens no device without a udev entry (measured again); 'auto_resize grow_only' for all maps of a node, as multipath has no other form of it, documented with how to turn it off; a node that drops the new option when it writes storage.cfg before it is upgraded, which the chapter covers; and an empty volume group that another node wipes right after its creation. Overview of what an admin gets with this: - Datacenter > Storage > SAN, a new entry below Storage, shows the multipath state of every node and of the LUNs that Proxmox VE manages or has a storage on, and edits the multipath entries and array-specific overrides. I explicitly chose this over the client/plugin side of Storage, as we do not have e.g. Ceph or ZFS/LVM/... creation in there either. We might want to rethink and consolidate our storage interface vs. management story in the future, but IMO a partial switch now is not worth it, that would add even more potential for confusion than the new SAN view. - Its Guided Setup menu leads through shared LVM on a LUN: for iSCSI it connects or reuses a target, with the initiator names of all nodes at hand, for Fibre Channel it takes a LUN the nodes already see. Both set up multipath and end with the state of the LUN on every node. - Datacenter > Storage > Add keeps its dialogs. The iSCSI one gets a Multipath option and the LVM one enables Shared for a base storage, and both point to the SAN view. - Node > SAN lists the LUNs of a node with their paths and usage, can wipe a LUN that no storage uses, and shows the iSCSI sessions. The rules the series follows since v4: nodes add WWIDs and maps but never remove them, a write is checked where it happens, lists do not probe devices, there are no new iSCSI operations, the guided setup makes the API calls an admin would make by hand, and multipath management is turned on by the admin. The v4 cover [7] has the reasons. Two things an admin will likely miss: there is no notification when a LUN loses path redundancy, the state only shows on the SAN pages and in the API. And Fibre Channel starts at a LUN the nodes already see; the chapter gives the commands to list WWPNs and rescan the HBAs. How it works: - /etc/pve/multipath.cfg holds explicit entries per WWID, /etc/pve/multipath-overrides.conf array settings; patch 01 registers both. An iSCSI storage with the 'managed-multipath' option supplies its LUNs without entries. The option is not named 'multipath', as third-party plugins use that name, and the storage layer leaves out a third-party plugin that declares a property of another plugin. - pvestatd calls one function of pve-storage per pass. It adds the WWIDs the node needs to /etc/multipath/wwids, writes the drop-ins /etc/multipath/conf.d/pve-*.conf, reloads multipathd after a change, and publishes the state of the maps that Proxmox VE manages or that a storage builds on through the node key-value store. - /cluster/multipath/status aggregates these reports; /cluster/multipath/{wwid,overrides} edit the two files; /nodes/{node}/scan/san-luns lists the LUNs of a node and /nodes/{node}/iscsi its sessions. - The volumes of an iSCSI storage with the option are named after the multipath map, so an LVM storage uses them through 'base' as before. - Volume group creation and wipe run under a cluster lock per LUN. They refuse a LUN that a storage uses or a process of the node has open, also through another path. Opens through another device node, such as one of a container, and users on other hosts are not seen. Patches: - cluster 01: register the two configuration files - storage 02-25: helpers for the disk and iSCSI session inventory (mostly Dietmar's), the multipath library and configuration, the iSCSI option, the per-node apply and status with their API, the disk list of many disks, the LUN inventory, the read-only iSCSI status, volume groups on LUNs, the wipe and the process that has a LUN open - docs 26-27: the multipath chapter, rewritten around tasks, and the paragraph for a LUN that a process has open - manager 28-42: the pvestatd call, report and API mounts, then the views, the two dialog changes, the two guided setups, the note in the add dialogs, and the guest or process and the target of a LUN Other changes since v5 [3]: - rebased onto current master in all four repositories, with the plugin metadata for the iSCSI option and its entry in the pinned property list - the setup menu is named Guided Setup instead of Add Storage, as three different things in two views were called add - search fields in the LUN lists and the iSCSI tree, which on a node also match path devices and WWPNs, for example one from the kernel log - the process check compares device numbers, and the volume group check refuses a volume group that any storage names - style fixes, warnings of three test files, and shorter commit messages When applying, versioned dependencies have to be set: pve-manager on libpve-storage-perl, and libpve-storage-perl (also as build dependency) on the libpve-cluster-perl with patch 01, as PVE::Storage registers the two files when it is loaded. pve-manager also needs the pve-doc-generator (build) and pve-docs with the multipath chapter, for its help links. If the bound for LVM status commands [4] gets applied too, whichever is applied second has to pass the status cache to the two lvm_vgs() calls of the base volume activation, as they can block a status pass otherwise. Both mark the cache of the status update with 'status_only', in the same line. The per-node target and portal series [5][6] are independent of this one. It takes the LUNs of an iSCSI storage from the sessions of its one target, so if a storage can have several targets per node, whichever is applied second has to cover that. I tested on the lab of v4 [7] and v5: a six-node nested cluster with a NetApp AFF A20 (three iSCSI portals, NVMe/TCP), Linux LIO targets and software FCoE, upgraded from the v5 build, with the NVMe over Fabrics series and [4] on top. Most review fixes ran as drills against the fault they fix, a few have unit tests only. Both guided setups ended optimal on six nodes, the Fibre Channel one also with a failing task, delayed and cut-off status requests and Escape pressed during the wait. A guest on NetApp, Fibre Channel and NVMe/TCP disks went through snapshots, live migrations, a path cut under I/O, an online resize and a node reboot with checksums compared. The scale test ran with 1000 NVMe/TCP namespaces and with 1000 iSCSI LUNs with maps; the disk list and the LUN inventory gave the same result with the one udev call and with a query per disk, also on the other five nodes. Guests with a NetApp map, a Fibre Channel LUN or an NVMe/TCP namespace as disk, also on one path of a map, showed in the LUN lists, and the wipe and the volume group creation refused these LUNs. Most of this ran on builds before the last fixes, which got drills of their own. I looked at each changed view and dialog in the web UI. Not repeated for v6: a path cut on Fibre Channel, the CLI procedure, a LUN that one node is not allowed to see, all paths down with queued I/O, the SAN views with a stopped multipathd, the multipath status of a node with 1000 foreign maps and the series installed alone. [0]: https://lore.proxmox.com/pve-devel/20260626121000.2095591-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20260703124707.1172980-2-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20260731102156.3947857-1-dietmar@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20261116012355.3187417-1-t.lamprecht@proxmox.com/ [4]: https://lore.proxmox.com/pve-devel/20261111015217.941220-1-t.lamprecht@proxmox.com/ [5]: https://lore.proxmox.com/pve-devel/20260430173220.441001-1-m.limbeck@proxmox.com/ [6]: https://lore.proxmox.com/pve-devel/20260804090819.2136483-1-dietmar@proxmox.com/ [7]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/ pve-cluster: Thomas Lamprecht (1): cfs: observe the multipath configuration files src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-storage: Dietmar Maurer (8): diskmanage: collect disk transport type from lsblk diskmanage: qualify NVMe over fabrics transport diskmanage: optionally include iSCSI session devices in disk enumeration diskmanage: collect native NVMe multipath path state iscsi: factor out session device map from device list diskmanage: add helper to list multipath devices pvesm: expose the SAN LUN inventory api: add node-level iSCSI status Thomas Lamprecht (16): multipath: add a library around multipathd multipath: add a cluster-wide configuration iscsi: add an option to manage storage LUNs with multipath lvm: activate a missing base volume before using the volume group multipath: apply the configuration on every node multipath: publish per-node status api: add endpoints for the cluster-wide multipath configuration api: multipath: add the cluster-wide health status diskmanage: look up the requested disks by name diskmanage: read the udev data of all disks with one call api: scan: add a SAN LUN inventory disks: lvm: drop unused imports disks: lvm: allow volume groups on SAN LUNs and multipath maps disks: lvm: refuse to destroy the volume group of a shared storage disks: allow wiping a SAN LUN that no storage uses api: scan: report the process that has a SAN LUN open debian/control | 3 +- src/PVE/API2/Disks.pm | 46 +- src/PVE/API2/Disks/LVM.pm | 77 ++- src/PVE/API2/ISCSI.pm | 166 ++++++ src/PVE/API2/Makefile | 2 + src/PVE/API2/Multipath.pm | 451 +++++++++++++++ src/PVE/API2/Storage/Scan.pm | 216 ++++++++ src/PVE/CLI/pvesm.pm | 14 + src/PVE/Diskmanage.pm | 375 +++++++++++-- src/PVE/ISCSI/Topology.pm | 88 +++ src/PVE/Makefile | 6 + src/PVE/Multipath.pm | 213 ++++++++ src/PVE/Multipath/Config.pm | 204 +++++++ src/PVE/Multipath/Generator.pm | 120 ++++ src/PVE/Multipath/Identity.pm | 44 ++ src/PVE/Multipath/Status.pm | 215 ++++++++ src/PVE/Storage.pm | 2 +- src/PVE/Storage/ISCSIPlugin.pm | 219 +++++--- src/PVE/Storage/LVMPlugin.pm | 151 ++++- src/PVE/Storage/Makefile | 3 +- src/PVE/Storage/SAN.pm | 409 ++++++++++++++ src/test/Makefile | 37 +- src/test/disk_api_test.pm | 314 +++++++++++ src/test/disk_tests/iscsi/disklist | 1 + .../disk_tests/iscsi/disklist_expected.json | 19 + src/test/disk_tests/iscsi/lsblk | 5 + src/test/disk_tests/iscsi/sda/device/model | 1 + src/test/disk_tests/iscsi/sda/device/vendor | 1 + .../disk_tests/iscsi/sda/queue/rotational | 1 + src/test/disk_tests/iscsi/sda/size | 1 + src/test/disk_tests/iscsi/sda_udevadm | 21 + src/test/disk_tests/multipath/disklist | 11 + .../multipath/disklist_expected.json | 107 ++++ src/test/disk_tests/multipath/dm-0/dm/name | 1 + src/test/disk_tests/multipath/dm-0/dm/uuid | 1 + src/test/disk_tests/multipath/dm-0/size | 1 + src/test/disk_tests/multipath/dm-0_slaves | 4 + src/test/disk_tests/multipath/dm-1/dm/name | 1 + src/test/disk_tests/multipath/dm-1/dm/uuid | 1 + src/test/disk_tests/multipath/dm-1/size | 1 + src/test/disk_tests/multipath/dm-1_slaves | 4 + src/test/disk_tests/multipath/dm-2/dm/name | 1 + src/test/disk_tests/multipath/dm-2/dm/uuid | 1 + src/test/disk_tests/multipath/dm-2/size | 1 + src/test/disk_tests/multipath/dm-2_slaves | 0 src/test/disk_tests/multipath/dm-3/dm/name | 1 + src/test/disk_tests/multipath/dm-3/dm/uuid | 1 + src/test/disk_tests/multipath/dm-3/size | 1 + src/test/disk_tests/multipath/dm-3_slaves | 0 src/test/disk_tests/multipath/dm-4/dm/name | 1 + src/test/disk_tests/multipath/dm-4/dm/uuid | 1 + src/test/disk_tests/multipath/dm-4/size | 1 + src/test/disk_tests/multipath/dm-4_slaves | 0 src/test/disk_tests/multipath/lsblk | 77 +++ src/test/disk_tests/multipath/lvs | 0 src/test/disk_tests/multipath/mounts | 1 + .../multipath/multipath_expected.json | 59 ++ src/test/disk_tests/multipath/multipathd | 124 +++++ .../multipath/multipathd_status_expected.json | 40 ++ .../nvme-subsystem/nvme-subsys0/nvme3/address | 1 + .../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme3/state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/address | 1 + .../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/state | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/address | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/state | 1 + .../disk_tests/multipath/nvme8n1/device/model | 1 + .../multipath/nvme8n1/device/transport | 1 + .../multipath/nvme8n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme8n1/size | 1 + src/test/disk_tests/multipath/nvme8n1_udevadm | 14 + .../disk_tests/multipath/nvme9n1/device/model | 1 + .../multipath/nvme9n1/device/transport | 1 + .../multipath/nvme9n1/device/vendor | 1 + .../multipath/nvme9n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme9n1/size | 1 + src/test/disk_tests/multipath/nvme9n1_udevadm | 12 + .../multipath/nvme_path_status_expected.json | 19 + src/test/disk_tests/multipath/partlist | 0 src/test/disk_tests/multipath/pvs | 1 + .../disk_tests/multipath/sda/device/model | 1 + .../disk_tests/multipath/sda/device/vendor | 1 + .../disk_tests/multipath/sda/holders/holder | 1 + .../disk_tests/multipath/sda/queue/rotational | 1 + src/test/disk_tests/multipath/sda/size | 1 + src/test/disk_tests/multipath/sda_udevadm | 18 + .../disk_tests/multipath/sdb/device/model | 1 + .../disk_tests/multipath/sdb/device/vendor | 1 + .../disk_tests/multipath/sdb/holders/holder | 1 + .../disk_tests/multipath/sdb/queue/rotational | 1 + src/test/disk_tests/multipath/sdb/size | 1 + src/test/disk_tests/multipath/sdb_udevadm | 18 + .../disk_tests/multipath/sdc/device/model | 1 + .../disk_tests/multipath/sdc/device/vendor | 1 + .../disk_tests/multipath/sdc/holders/holder | 1 + .../disk_tests/multipath/sdc/queue/rotational | 1 + src/test/disk_tests/multipath/sdc/size | 1 + src/test/disk_tests/multipath/sdc_udevadm | 18 + .../disk_tests/multipath/sdd/device/model | 1 + .../disk_tests/multipath/sdd/device/vendor | 1 + .../disk_tests/multipath/sdd/holders/holder | 1 + .../disk_tests/multipath/sdd/queue/rotational | 1 + src/test/disk_tests/multipath/sdd/size | 1 + src/test/disk_tests/multipath/sdd_udevadm | 18 + src/test/disk_tests/multipath/zpool | 1 + .../disk_tests/sas/disklist_expected.json | 1 + src/test/disk_tests/sas/lsblk | 5 + src/test/disklist_test.pm | 175 +++++- src/test/multipath-three-states.json | 52 ++ src/test/plugin_properties_test.pm | 1 + src/test/run_disk_api_tests.pl | 11 + src/test/run_iscsi_activation_tests.pl | 109 ++++ src/test/run_lun_identity_tests.pl | 57 ++ src/test/run_multipath_api_tests.pl | 515 ++++++++++++++++++ src/test/run_multipath_tests.pl | 169 ++++++ src/test/run_san_api_tests.pl | 94 ++++ src/test/run_san_library_tests.pl | 135 +++++ src/test/run_san_lun_scan_tests.pl | 252 +++++++++ src/test/run_san_nvme_tests.pl | 120 ++++ src/test/run_san_taint_tests.pl | 60 ++ src/test/run_san_wipe_tests.pl | 255 +++++++++ src/test/run_san_write_check_tests.pl | 205 +++++++ src/test/run_test_lvm_base_activation.pl | 121 ++++ src/test/run_test_lvm_base_volume.pl | 12 +- src/test/run_test_lvm_create.pl | 46 ++ 126 files changed, 6224 insertions(+), 194 deletions(-) pve-docs: Thomas Lamprecht (2): storage: document the cluster-wide multipath management storage: multipath: describe the use shown for an open LUN pmxcfs.adoc | 2 + pve-storage-iscsi.adoc | 11 ++ pve-storage-iscsidirect.adoc | 2 +- pve-storage-lvm.adoc | 3 + pve-storage-multipath.adoc | 309 +++++++++++++++++++++++++++++++++++ pvesm.adoc | 18 +- 6 files changed, 339 insertions(+), 6 deletions(-) pve-manager: Dietmar Maurer (2): api: nodes: expose the iSCSI status ui: node: san: show iSCSI targets and sessions Thomas Lamprecht (13): pvestatd: apply the multipath configuration and report the node status report: add the managed multipath configuration api: cluster: add the multipath endpoints ui: dc: add a SAN view ui: storage: iscsi: add the multipath option ui: lvm: enable shared when a base storage is selected ui: node: add SAN LUNs panel ui: node: san: wipe a LUN that no storage uses ui: dc: san: add a guided setup for shared LVM on iSCSI ui: dc: san: add a guided setup for shared LVM on Fibre Channel ui: storage: point the iSCSI and LVM add dialogs to the SAN view ui: san: show the guest or process that has a LUN open ui: node: san: add a column for the iSCSI target of a LUN PVE/API2/Cluster.pm | 7 + PVE/API2/Nodes.pm | 7 + PVE/Report.pm | 3 + PVE/Service/pvestatd.pm | 7 + www/manager6/Makefile | 12 + www/manager6/SANUtils.js | 202 ++++++++++ www/manager6/Utils.js | 19 + www/manager6/dc/Config.js | 23 +- www/manager6/dc/MultipathWwidEdit.js | 81 ++++ www/manager6/dc/SAN.js | 374 ++++++++++++++++++ www/manager6/node/Config.js | 24 ++ www/manager6/node/ISCSITargets.js | 150 +++++++ www/manager6/node/LVM.js | 3 + www/manager6/node/SANLuns.js | 229 +++++++++++ www/manager6/storage/Base.js | 5 + www/manager6/storage/IScsiEdit.js | 46 +++ www/manager6/storage/LVMEdit.js | 6 + www/manager6/storage/wizard/FibreChannel.js | 175 ++++++++ www/manager6/storage/wizard/IScsi.js | 274 +++++++++++++ www/manager6/storage/wizard/LunSelect.js | 177 +++++++++ www/manager6/storage/wizard/NodeIdentities.js | 83 ++++ www/manager6/storage/wizard/Wizard.js | 299 ++++++++++++++ www/manager6/window/SANLunState.js | 115 ++++++ www/manager6/window/SANLunWipe.js | 52 +++ 24 files changed, 2367 insertions(+), 6 deletions(-)