From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 747521FF0C0 for ; Wed, 18 Nov 2026 21:28:32 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4C6AF2167E; Wed, 18 Nov 2026 21:27:58 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v7 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Date: Wed, 18 Nov 2026 21:25:43 +0100 Message-ID: <20261118202646.3017286-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795033611042 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.018 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: K5J5RPUAYDAVQEG7UWEPCVNNJIBDW3KO X-Message-ID-Hash: K5J5RPUAYDAVQEG7UWEPCVNNJIBDW3KO X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is v7 of the combined and reduced series of the cluster-wide multipath work [0][1] and Dietmar's RFC for guided remote storage setup and SAN visibility [2]. v7 is v6 [8] plus what some independent reviews and further lab testing of v6 turned up, i.e.: - The checks before a wipe or volume group creation take the members of a map from the kernel too, not only those whose WWID multipathd or udev reports, so a guest on such a path or its target is found. The LUN list names a guest that writes to a partition of a LUN, which the wipe refused but the list showed as unused. - The web UI no longer falls back to a device name like /dev/sdX for a wipe or for the Fibre Channel volume group: a name that passed to another LUN in between would lead there. A LUN without a name under /dev/disk/by-id cannot be wiped from the list, which the node refused anyway, and the Fibre Channel setup waits for that name of the map. - A drop-in is only replaced if its first line is the generated one, v6 also took "# Not managed by Proxmox VE", and a node checks all drop-ins before it changes one. The overrides refuse text after a '{' and a '}' that is not alone on its line: multipath ignores the rest of such a line, and with it the sections after it. - A LUN that a storage requests, but that no node has assembled, shows as Missing in the SAN view instead of not at all. An LVM base named after the scsi- link of a LUN is attributed to its storage. - pvestatd applies the multipath configuration right before the storage status, so the node and guest status do not wait for it. - The iSCSI setup offers the LUNs that the connection storage lists, also one that the scan attributes to another target of the array, finds a LUN by name, WWID or path, and says which storage blocks the reuse of a target and why. The initiator names and the window at the end follow the nodes of the new storage. The Fibre Channel recovery texts name the volume group and storage ID. - Enable Multipath under Node > SAN is offered for Fibre Channel LUNs only, not for an NVMe namespace without native multipath. - The LUN list names a guest only to users who may see it. - Smaller ones: the status summary is described as the worst known state, Expected Paths as the highest of the entry and the portal counts, the Node > Disks > LVM dialog says that Add Storage on a SAN LUN creates a shared storage for all nodes, error messages name what to use instead, and the chapter says what a reader of a LUN and a kernel partition of it mean for the checks. What I left as is: findings that repeat points decided for v5 [3], except the first-target filter of the iSCSI setup, which the content listing now makes cheap to fix, and master's behavior, such as a kernel partition of a LUN that is selected on its own, which the chapter now mentions. wipefs on the claimed device does not re-read the partition table, but udev does once the wipe closes the device, as the lab showed. Overview of what an admin gets with this: - Datacenter > Storage > SAN, a new entry below Storage, shows the multipath state of every node and of the LUNs that Proxmox VE manages or has a storage on, and edits the multipath entries and array-specific overrides. I explicitly chose this over the client/plugin side of Storage, as we do not have e.g. Ceph or ZFS/LVM/... creation in there either. We might want to rethink and consolidate our storage interface vs. management story in the future, but IMO a partial switch now is not worth it, that would add even more potential for confusion than the new SAN view. - Its Guided Setup menu leads through shared LVM on a LUN: for iSCSI it connects or reuses a target, with the initiator names of its nodes at hand, for Fibre Channel it takes a LUN the nodes already see. Both set up multipath and end with the state of the LUN on every node. - Datacenter > Storage > Add keeps its dialogs. The iSCSI one gets a Multipath option and the LVM one enables Shared for a base storage, and both point to the SAN view. - Node > SAN lists the LUNs of a node with their paths and usage, can wipe a LUN that no storage uses, and shows the iSCSI sessions. The rules the series follows since v4: nodes add WWIDs and maps but never remove them, a write is checked where it happens, lists do not probe devices, there are no new iSCSI operations, the guided setup makes the API calls an admin would make by hand, and multipath management is turned on by the admin. The v4 cover [7] has the reasons. Two things an admin will likely miss: there is no notification when a LUN loses path redundancy, the state only shows on the SAN pages and in the API. And Fibre Channel starts at a LUN the nodes already see; the chapter gives the commands to list WWPNs and rescan the HBAs. How it works: - /etc/pve/multipath.cfg holds explicit entries per WWID, /etc/pve/multipath-overrides.conf array settings; patch 01 registers both. An iSCSI storage with the 'managed-multipath' option supplies its LUNs without entries. The option is not named 'multipath', as third-party plugins use that name, and the storage layer leaves out a third-party plugin that declares a property of another plugin. - pvestatd calls one function of pve-storage per pass. It adds the WWIDs the node needs to /etc/multipath/wwids, writes the drop-ins /etc/multipath/conf.d/pve-*.conf, reloads multipathd after a change, and publishes the state of the maps that Proxmox VE manages or that a storage builds on through the node key-value store. - /cluster/multipath/status aggregates these reports; /cluster/multipath/{wwid,overrides} edit the two files; /nodes/{node}/scan/san-luns lists the LUNs of a node and /nodes/{node}/iscsi its sessions. - The volumes of an iSCSI storage with the option are named after the multipath map, so an LVM storage uses them through 'base' as before. - Volume group creation and wipe run under a cluster lock per LUN. They refuse a LUN that a storage uses or a process of the node has open, also through another path. Opens through another device node, such as one of a container, and users on other hosts are not seen. Patches: - cluster 01: register the two configuration files - storage 02-26: helpers for the disk and iSCSI session inventory (mostly Dietmar's), the multipath library and configuration, the iSCSI option, the per-node apply and status with their API, the disk list of many disks, the LUN inventory, the read-only iSCSI status and the portal of an iSCSI path, volume groups on LUNs, the wipe and the process that has a LUN open - docs 27-28: the multipath chapter, rewritten around tasks, and the paragraph for a LUN that a process has open - manager 29-43: the pvestatd call, report and API mounts, then the views, the two dialog changes, the two guided setups, the note in the add dialogs, and the guest or process and the target of a LUN Other changes since v6 [8]: - rebased onto current master; for pve-manager and pve-docs that has a commit each that ends the technology preview status of snapshots as volume chains, as they are ready for production use. The guided setup and the chapter no longer call them a technology preview either - the node list of Datacenter > Storage > SAN links each node to its Node > SAN view, like the content of a storage links a volume to its guest - Blockbridge tested v6 with their storage plugin, which runs its own sessions, thanks! From that: Node > SAN shows a LUN without a multipath map with its devices as paths, in the state that the kernel reports for each, so a LUN with a single path also shows whether it is up. Its Multipath column says Single path for one device, and an iSCSI path names the portal of its session. The target column also shows the subsystem of an NVMe namespace, and the LUN lists sort by the names and uses they show - the Wipe button under Node > SAN sits apart from the other buttons, and its dialog also asks to confirm that no other host uses the LUN, as the checks before the wipe only see such a use through a storage - the subject of patch 3 spells NVMe over Fabrics like the rest When applying, versioned dependencies have to be set: pve-manager on libpve-storage-perl, and libpve-storage-perl (also as build dependency) on the libpve-cluster-perl with patch 01, as PVE::Storage registers the two files when it is loaded. pve-cluster with the same patch keeps the version counters of the two files; with an older pmxcfs they are read and parsed again on every request. pve-manager also needs the pve-doc-generator (build) and pve-docs with the multipath chapter, for its help links. If the bound for LVM status commands [4] gets applied too, whichever is applied second has to pass the status cache to the two lvm_vgs() calls of the base volume activation, as they can block a status pass otherwise. Both mark the cache of the status update with 'status_only', in the same line. The per-node target and portal series [5][6] are independent of this one. It takes the LUNs of an iSCSI storage from the sessions of its one target, so if a storage can have several targets per node, whichever is applied second has to cover that. I tested v7 on the lab of v4 [7] to v6, upgraded from the v6 build, with the NVMe over Fabrics series and [4] on top. The checks of the fixes ran there: a drop-in with a near-match first line stayed and the node reported the conflict without touching another file, overrides on one line were refused, a guest with a partition of an NVMe/TCP namespace as disk showed in the LUN list and kept the wipe from running, a storage administrator without VM.Audit got the process but not the guest, and the button states under Node > SAN were as described. The iSCSI setup ran on the NetApp with a search, two of six nodes and the reuse of the connection storage, the target conflict with two storages, and the Fibre Channel setup with a new entry, once to the end and once with the task status lost. The link of a node in the SAN view and the storage dialogs and setup without the technology preview hint I checked on the final build. A node logged in to an iSCSI target without a map, as a plugin with its own sessions does, showed OK (1 path) and Single path, then Failed with the path blocked when its portal was cut, and with a second portal Optimal and Degraded; the LUN list took 0.6 s during the cut. The partition lookup of the LUN list costs about 34 us per device, so 0.1 s for 1000 LUNs with two paths. I looked at each changed view and dialog. For the rest, the tests of v6 [8] still apply; I did not repeat the guest, path cut of maps and scale tests. [0]: https://lore.proxmox.com/pve-devel/20260626121000.2095591-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20260703124707.1172980-2-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20260731102156.3947857-1-dietmar@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20261116012355.3187417-1-t.lamprecht@proxmox.com/ [4]: https://lore.proxmox.com/pve-devel/20261111015217.941220-1-t.lamprecht@proxmox.com/ [5]: https://lore.proxmox.com/pve-devel/20260430173220.441001-1-m.limbeck@proxmox.com/ [6]: https://lore.proxmox.com/pve-devel/20260804090819.2136483-1-dietmar@proxmox.com/ [7]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/ [8]: https://lore.proxmox.com/pve-devel/20261118065302.3033937-1-t.lamprecht@proxmox.com/ pve-cluster: Thomas Lamprecht (1): cfs: observe the multipath configuration files src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-storage: Dietmar Maurer (8): diskmanage: collect disk transport type from lsblk diskmanage: qualify NVMe over Fabrics transport diskmanage: optionally include iSCSI session devices in disk enumeration diskmanage: collect native NVMe multipath path state iscsi: factor out session device map from device list diskmanage: add helper to list multipath devices pvesm: expose the SAN LUN inventory api: add node-level iSCSI status Thomas Lamprecht (17): multipath: add a library around multipathd multipath: add a cluster-wide configuration iscsi: add an option to manage storage LUNs with multipath lvm: activate a missing base volume before using the volume group multipath: apply the configuration on every node multipath: publish per-node status api: add endpoints for the cluster-wide multipath configuration api: multipath: add the cluster-wide health status diskmanage: look up the requested disks by name diskmanage: read the udev data of all disks with one call api: scan: add a SAN LUN inventory api: scan: name the portal of an iSCSI path disks: lvm: drop unused imports disks: lvm: allow volume groups on SAN LUNs and multipath maps disks: lvm: refuse to destroy the volume group of a shared storage disks: allow wiping a SAN LUN that no storage uses api: scan: report the process that has a SAN LUN open debian/control | 3 +- src/PVE/API2/Disks.pm | 46 +- src/PVE/API2/Disks/LVM.pm | 80 ++- src/PVE/API2/ISCSI.pm | 166 +++++ src/PVE/API2/Makefile | 2 + src/PVE/API2/Multipath.pm | 451 ++++++++++++++ src/PVE/API2/Storage/Scan.pm | 244 ++++++++ src/PVE/CLI/pvesm.pm | 15 + src/PVE/Diskmanage.pm | 383 ++++++++++-- src/PVE/ISCSI/Topology.pm | 89 +++ src/PVE/Makefile | 6 + src/PVE/Multipath.pm | 213 +++++++ src/PVE/Multipath/Config.pm | 209 +++++++ src/PVE/Multipath/Generator.pm | 128 ++++ src/PVE/Multipath/Identity.pm | 44 ++ src/PVE/Multipath/Status.pm | 228 +++++++ src/PVE/Storage.pm | 2 +- src/PVE/Storage/ISCSIPlugin.pm | 219 ++++--- src/PVE/Storage/LVMPlugin.pm | 156 ++++- src/PVE/Storage/Makefile | 3 +- src/PVE/Storage/SAN.pm | 457 ++++++++++++++ src/test/Makefile | 37 +- src/test/disk_api_test.pm | 314 ++++++++++ src/test/disk_tests/iscsi/disklist | 1 + .../disk_tests/iscsi/disklist_expected.json | 19 + src/test/disk_tests/iscsi/lsblk | 5 + src/test/disk_tests/iscsi/sda/device/model | 1 + src/test/disk_tests/iscsi/sda/device/vendor | 1 + .../disk_tests/iscsi/sda/queue/rotational | 1 + src/test/disk_tests/iscsi/sda/size | 1 + src/test/disk_tests/iscsi/sda_udevadm | 21 + src/test/disk_tests/multipath/disklist | 11 + .../multipath/disklist_expected.json | 107 ++++ src/test/disk_tests/multipath/dm-0/dm/name | 1 + src/test/disk_tests/multipath/dm-0/dm/uuid | 1 + src/test/disk_tests/multipath/dm-0/size | 1 + src/test/disk_tests/multipath/dm-0_slaves | 4 + src/test/disk_tests/multipath/dm-1/dm/name | 1 + src/test/disk_tests/multipath/dm-1/dm/uuid | 1 + src/test/disk_tests/multipath/dm-1/size | 1 + src/test/disk_tests/multipath/dm-1_slaves | 4 + src/test/disk_tests/multipath/dm-2/dm/name | 1 + src/test/disk_tests/multipath/dm-2/dm/uuid | 1 + src/test/disk_tests/multipath/dm-2/size | 1 + src/test/disk_tests/multipath/dm-2_slaves | 0 src/test/disk_tests/multipath/dm-3/dm/name | 1 + src/test/disk_tests/multipath/dm-3/dm/uuid | 1 + src/test/disk_tests/multipath/dm-3/size | 1 + src/test/disk_tests/multipath/dm-3_slaves | 0 src/test/disk_tests/multipath/dm-4/dm/name | 1 + src/test/disk_tests/multipath/dm-4/dm/uuid | 1 + src/test/disk_tests/multipath/dm-4/size | 1 + src/test/disk_tests/multipath/dm-4_slaves | 0 src/test/disk_tests/multipath/lsblk | 77 +++ src/test/disk_tests/multipath/lvs | 0 src/test/disk_tests/multipath/mounts | 1 + .../multipath/multipath_expected.json | 59 ++ src/test/disk_tests/multipath/multipathd | 124 ++++ .../multipath/multipathd_status_expected.json | 40 ++ .../nvme-subsystem/nvme-subsys0/nvme3/address | 1 + .../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme3/state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/address | 1 + .../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/state | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/address | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/state | 1 + .../disk_tests/multipath/nvme8n1/device/model | 1 + .../multipath/nvme8n1/device/transport | 1 + .../multipath/nvme8n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme8n1/size | 1 + src/test/disk_tests/multipath/nvme8n1_udevadm | 14 + .../disk_tests/multipath/nvme9n1/device/model | 1 + .../multipath/nvme9n1/device/nvme5/transport | 1 + .../multipath/nvme9n1/device/vendor | 1 + .../multipath/nvme9n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme9n1/size | 1 + src/test/disk_tests/multipath/nvme9n1_udevadm | 12 + .../multipath/nvme_path_status_expected.json | 19 + src/test/disk_tests/multipath/partlist | 0 src/test/disk_tests/multipath/pvs | 1 + .../disk_tests/multipath/sda/device/model | 1 + .../disk_tests/multipath/sda/device/vendor | 1 + .../disk_tests/multipath/sda/holders/holder | 1 + .../disk_tests/multipath/sda/queue/rotational | 1 + src/test/disk_tests/multipath/sda/size | 1 + src/test/disk_tests/multipath/sda_udevadm | 18 + .../disk_tests/multipath/sdb/device/model | 1 + .../disk_tests/multipath/sdb/device/vendor | 1 + .../disk_tests/multipath/sdb/holders/holder | 1 + .../disk_tests/multipath/sdb/queue/rotational | 1 + src/test/disk_tests/multipath/sdb/size | 1 + src/test/disk_tests/multipath/sdb_udevadm | 18 + .../disk_tests/multipath/sdc/device/model | 1 + .../disk_tests/multipath/sdc/device/vendor | 1 + .../disk_tests/multipath/sdc/holders/holder | 1 + .../disk_tests/multipath/sdc/queue/rotational | 1 + src/test/disk_tests/multipath/sdc/size | 1 + src/test/disk_tests/multipath/sdc_udevadm | 18 + .../disk_tests/multipath/sdd/device/model | 1 + .../disk_tests/multipath/sdd/device/vendor | 1 + .../disk_tests/multipath/sdd/holders/holder | 1 + .../disk_tests/multipath/sdd/queue/rotational | 1 + src/test/disk_tests/multipath/sdd/size | 1 + src/test/disk_tests/multipath/sdd_udevadm | 18 + src/test/disk_tests/multipath/zpool | 1 + .../disk_tests/sas/disklist_expected.json | 1 + src/test/disk_tests/sas/lsblk | 5 + src/test/disklist_test.pm | 179 +++++- src/test/multipath-three-states.json | 52 ++ src/test/plugin_properties_test.pm | 1 + src/test/run_disk_api_tests.pl | 11 + src/test/run_iscsi_activation_tests.pl | 109 ++++ src/test/run_lun_identity_tests.pl | 57 ++ src/test/run_multipath_api_tests.pl | 584 ++++++++++++++++++ src/test/run_multipath_tests.pl | 175 ++++++ src/test/run_san_api_tests.pl | 124 ++++ src/test/run_san_library_tests.pl | 135 ++++ src/test/run_san_lun_scan_tests.pl | 334 ++++++++++ src/test/run_san_nvme_tests.pl | 120 ++++ src/test/run_san_taint_tests.pl | 60 ++ src/test/run_san_wipe_tests.pl | 271 ++++++++ src/test/run_san_write_check_tests.pl | 219 +++++++ src/test/run_test_lvm_base_activation.pl | 121 ++++ src/test/run_test_lvm_base_volume.pl | 16 +- src/test/run_test_lvm_create.pl | 46 ++ 126 files changed, 6566 insertions(+), 197 deletions(-) pve-docs: Thomas Lamprecht (2): storage: document the cluster-wide multipath management storage: multipath: describe the use shown for an open LUN pmxcfs.adoc | 2 + pve-storage-iscsi.adoc | 11 ++ pve-storage-iscsidirect.adoc | 2 +- pve-storage-lvm.adoc | 3 + pve-storage-multipath.adoc | 313 +++++++++++++++++++++++++++++++++++ pvesm.adoc | 18 +- 6 files changed, 343 insertions(+), 6 deletions(-) pve-manager: Dietmar Maurer (2): api: nodes: expose the iSCSI status ui: node: san: show iSCSI targets and sessions Thomas Lamprecht (13): pvestatd: apply the multipath configuration and report the node status report: add the managed multipath configuration api: cluster: add the multipath endpoints ui: dc: add a SAN view ui: storage: iscsi: add the multipath option ui: lvm: enable shared when a base storage is selected ui: node: add SAN LUNs panel ui: node: san: wipe a LUN that no storage uses ui: dc: san: add a guided setup for shared LVM on iSCSI ui: dc: san: add a guided setup for shared LVM on Fibre Channel ui: storage: point the iSCSI and LVM add dialogs to the SAN view ui: san: show the guest or process that has a LUN open ui: node: san: add a column for the target or subsystem of a LUN PVE/API2/Cluster.pm | 7 + PVE/API2/Nodes.pm | 7 + PVE/Report.pm | 3 + PVE/Service/pvestatd.pm | 7 + www/manager6/Makefile | 12 + www/manager6/SANUtils.js | 232 ++++++++++ www/manager6/Utils.js | 19 + www/manager6/dc/Config.js | 23 +- www/manager6/dc/MultipathWwidEdit.js | 81 ++++ www/manager6/dc/SAN.js | 400 ++++++++++++++++++ www/manager6/node/Config.js | 24 ++ www/manager6/node/ISCSITargets.js | 150 +++++++ www/manager6/node/LVM.js | 5 + www/manager6/node/SANLuns.js | 233 ++++++++++ www/manager6/storage/Base.js | 5 + www/manager6/storage/IScsiEdit.js | 46 ++ www/manager6/storage/LVMEdit.js | 6 + www/manager6/storage/wizard/FibreChannel.js | 187 ++++++++ www/manager6/storage/wizard/IScsi.js | 297 +++++++++++++ www/manager6/storage/wizard/LunSelect.js | 224 ++++++++++ www/manager6/storage/wizard/NodeIdentities.js | 94 ++++ www/manager6/storage/wizard/Wizard.js | 298 +++++++++++++ www/manager6/window/SANLunState.js | 119 ++++++ www/manager6/window/SANLunWipe.js | 64 +++ 24 files changed, 2537 insertions(+), 6 deletions(-)