From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 873CD1FF0AF for ; Fri, 06 Nov 2026 01:52:05 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id ACE372181A; Fri, 06 Nov 2026 01:50:59 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v3 proxmox,proxmox-perl-rs,cluster,storage,docs,manager 00/76] SAN: cluster-wide multipath, SAN views and a guided storage setup Date: Fri, 6 Nov 2026 00:30:35 +0100 Message-ID: <20261105233336.2178064-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793926177219 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.330 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 7CDQ5RFWTOXAG6OH4PSM2MX2QDRF34ES X-Message-ID-Hash: 7CDQ5RFWTOXAG6OH4PSM2MX2QDRF34ES X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This v3 combines the cluster-wide multipath series [0][1] with Dietmar's RFC for guided remote storage setup and SAN visibility [2]. What it adds: - cluster-wide dm-multipath management through /etc/pve/multipath.cfg: iSCSI storages with the new 'multipath' option supply their LUNs; explicit entries cover individual LUNs, for example Fibre Channel. pvestatd applies the configuration on each node without removing allow-list entries added by hand or for boot from SAN. Array-specific settings go in /etc/pve/multipath-overrides.conf. - SAN views: Datacenter > SAN shows node status and initiator names for target access rules. Its Multipath view groups path health per LUN, including expected paths, report age, and unresolved identities. Node > SAN lists LUNs, paths, usage, and iSCSI targets and sessions, with actions to enable multipath or wipe a LUN as root@pam. - guided setup for NFS, SMB/CIFS, iSCSI, Fibre Channel, and ZFS over iSCSI. For shared LVM over iSCSI, it creates or reuses the connection storage, waits for LUN discovery, and creates a volume group on the selected LUN, using its map when multipath is enabled. Partial setups retain their settings and retry only missing steps; unanswered requests are checked before retrying. - guards for destructive actions: SAN wipes and volume-group creation check cluster storage references, local users, and device identities. Before a wipe, the web UI also asks the other nodes for users of the LUN. Wipes keep the checked device references and recheck identities between writes. Inconclusive checks refuse the operation. Session logout is refused if it would remove the last usable path of a LUN in use. Cluster locks do not cover hosts outside the cluster. Patches 04-11 are independent fixes and could go in first, with their documentation and pvestatd parts in patches 43 and 46: status-only activation and bounded LVM queries, shared VGs missed by LVM hints, per-target rescans, LUNs listed per iSCSI session, validation before volume-group creation, and a target check that refuses composite devices. The smaller fixes for SCSI padding, iSCSI discovery errors, iSER and offload sessions, LVM base volumes and the iSCSI edit dialog are already applied. Changes since v2: - combined with Dietmar's RFC; multipath is an iSCSI storage option rather than a separate per-LUN storage type - Node > SAN offers Enable Multipath; the datacenter view uses one row per LUN with expandable node details, not a column per node (Dominik) - addressed Dominik's API and code remarks: add_wwid accepts alias and settings, alias changes use the update endpoint, config helpers are renamed, group_by_tpg is available, and the overrides example is fixed - preserved existing allow-list entries, addressing Maximiliano's migration concern without a warning on first use - added expected path counts, stalled-node detection, withdrawn LUN status, unresolved identities, and duplicate-PV hints for clones - added LVM and wipe guards, bounded status commands, the node iSCSI API, and pvenode commands - added an admin guide chapter, including removal, growth, and clone procedures tested on real hardware - moved the parsing of multipathd map status and the health derivation to a new proxmox-multipath crate, used through pve-rs bindings Testing used a five-node nested cluster attached to a NetApp AFF A20 through two independent paths over iSCSI and NVMe/TCP, plus host-side Linux LIO and nvmet targets. Scenarios covered path, LIF, and whole-array outages with guests running; LUN unmapping with queued I/O; revoked node access; online growth; snapshot clones; reboot; and adding 30 LUNs at once. Wizard and wipe tests covered lost responses, stuck tasks, denied logins, and LVM metadata written by other nodes. The screens were also reviewed from four user perspectives. The bounded status commands and the wizard's LUN identity check also have focused regression tests. Fibre Channel and offload iSCSI initiators, which can run several sessions on one SCSI host, were not tested on hardware. Open points: - path-redundancy loss is visible in the web UI and API but sends no notification (can be more easily done once Lukas' notification matcher rework has landed) - an optimal LUN is marked incomplete while its state on an expected node is unknown, but there is no count of the current reports per LUN - LUN removal is documented, not guided - can be added later. - if thin_check takes longer than the shared status-command budget, that status pass reports no LVM or LVM-thin status. Later passes retry the activation; starting a guest activates the pool without this budget - pvcreate and vgcreate receive a checked, identity-bearing by-id name, but resolve it themselves, leaving a short race after the final check - integration with the per-node target and portal mapping work [3][4] Apply in patch order. The pve-rs bindings need the proxmox-multipath crate packaged first. pve-storage needs pve-cluster with the multipath configuration files registered and libpve-rs-perl with the bindings. pve-manager needs the matching pve-storage APIs and status-command helper, plus the pve-docs and pve-doc-generator changes. Feature-specific dependency bumps are deferred until the providing package versions are known and must be added before shipping. This includes a Breaks of libpve-storage-perl on older pve-manager, whose Ceph OSD creation would accept the iSCSI LUNs that the disk list now includes (patch 45). [0]: https://lore.proxmox.com/pve-devel/20260626121000.2095591-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20260703124707.1172980-2-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20260731102156.3947857-1-dietmar@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20260430173220.441001-1-m.limbeck@proxmox.com/ [4]: https://lore.proxmox.com/pve-devel/20260804090819.2136483-1-dietmar@proxmox.com/ proxmox: Thomas Lamprecht (1): add proxmox-multipath crate Cargo.toml | 2 + proxmox-multipath/Cargo.toml | 17 ++ proxmox-multipath/debian/changelog | 7 + proxmox-multipath/debian/control | 38 +++ proxmox-multipath/debian/copyright | 18 ++ proxmox-multipath/debian/debcargo.toml | 7 + proxmox-multipath/src/lib.rs | 531 +++++++++++++++++++++++++++++++++ 7 files changed, 620 insertions(+) proxmox-perl-rs: Thomas Lamprecht (1): pve-rs: add PVE::RS::Multipath bindings for multipathd status parsing pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 2 ++ pve-rs/debian/control | 1 + pve-rs/src/bindings/mod.rs | 3 ++ pve-rs/src/bindings/multipath.rs | 40 ++++++++++++++++++++++ pve-rs/test/Makefile | 1 + pve-rs/test/multipath.pl | 74 ++++++++++++++++++++++++++++++++++++++++ 7 files changed, 122 insertions(+) pve-cluster: Thomas Lamprecht (1): cfs: observe the multipath configuration files src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-storage: Dietmar Maurer (12): diskmanage: collect disk transport type from lsblk diskmanage: add helper to list multipath devices diskmanage: qualify NVMe over fabrics transport disks: list: add include-remote parameter diskmanage: include iSCSI session devices in disk enumeration diskmanage: link multipath member disks to their map device iscsi: factor out session device map from device list api: scan: add a SAN LUN inventory pvesm: expose the SAN LUN inventory diskmanage: collect native NVMe multipath path state api: scan: san-luns: report path state api: add node-level iSCSI status and maintenance Thomas Lamprecht (27): storage: distinguish status-only activation in plugins lvm: find a shared volume group created by another node storage: bound LVM status commands iscsi: rate-limit session rescans per target iscsi: list the LUNs of each session, not of its SCSI host lvm: refuse to create a volume group on a device that holds data lvm: refuse a base volume of a target that serves disk images iscsi: refuse virtual devices other than multipath maps as volumes disks: lvm: allow volume groups on SAN LUNs and multipath maps multipath: add a library around multipathd diskmanage: collect local block device users api: iscsi: add session logout with last-path protection multipath: add a cluster-wide configuration iscsi: add an option to manage storage LUNs with multipath lvm: activate a missing base volume before using the volume group multipath: apply the configuration on every node iscsi: share the initiator identity reader multipath: publish per-node status api: add endpoints for the cluster-wide multipath configuration api: multipath: add the cluster-wide health status multipath: collect identities of non-iSCSI SCSI paths api: scan: san-luns: report the multipath state of each LUN api: scan: flag duplicate physical volumes on distinct LUNs api: disks: recheck local users and retain wipe device references iscsi: name plain LUNs after their identity link api: disks: guard SAN wipes with stable identities and storage checks api: scan: refresh SAN usage with bounded probes ApiChangeLog | 24 + debian/control | 2 + src/PVE/API2/Disks.pm | 76 +- src/PVE/API2/Disks/LVM.pm | 74 +- src/PVE/API2/ISCSI.pm | 423 +++++++ src/PVE/API2/Makefile | 2 + src/PVE/API2/Multipath.pm | 689 +++++++++++ src/PVE/API2/Storage/Scan.pm | 247 ++++ src/PVE/CLI/pvesm.pm | 13 + src/PVE/Diskmanage.pm | 1007 +++++++++++++++- src/PVE/Makefile | 6 + src/PVE/Multipath.pm | 375 ++++++ src/PVE/Multipath/ClusterConfig.pm | 64 + src/PVE/Multipath/Config.pm | 451 +++++++ src/PVE/Multipath/Generator.pm | 314 +++++ src/PVE/Multipath/Identity.pm | 402 +++++++ src/PVE/Multipath/Status.pm | 486 ++++++++ src/PVE/Storage.pm | 272 ++++- src/PVE/Storage/ISCSIPlugin.pm | 695 +++++++++-- src/PVE/Storage/LVMPlugin.pm | 323 ++++- src/PVE/Storage/LvmThinPlugin.pm | 39 +- src/PVE/Storage/Makefile | 4 +- src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/SANScan.pm | 429 +++++++ src/PVE/Storage/StatusCommand.pm | 241 ++++ src/test/Makefile | 61 +- src/test/disk_api_test.pm | 194 +++ src/test/disk_tests/iscsi/disklist | 1 + src/test/disk_tests/iscsi/disklist_expected.json | 19 + src/test/disk_tests/iscsi/lsblk | 5 + src/test/disk_tests/iscsi/sda/device/model | 1 + src/test/disk_tests/iscsi/sda/device/vendor | 1 + src/test/disk_tests/iscsi/sda/queue/rotational | 1 + src/test/disk_tests/iscsi/sda/size | 1 + src/test/disk_tests/iscsi/sda_udevadm | 21 + src/test/disk_tests/multipath/disklist | 11 + .../disk_tests/multipath/disklist_expected.json | 111 ++ src/test/disk_tests/multipath/dm-0/dm/name | 1 + src/test/disk_tests/multipath/dm-0/dm/uuid | 1 + src/test/disk_tests/multipath/dm-0/size | 1 + src/test/disk_tests/multipath/dm-0_slaves | 4 + src/test/disk_tests/multipath/dm-1/dm/name | 1 + src/test/disk_tests/multipath/dm-1/dm/uuid | 1 + src/test/disk_tests/multipath/dm-1/size | 1 + src/test/disk_tests/multipath/dm-1_slaves | 4 + src/test/disk_tests/multipath/dm-2/dm/name | 1 + src/test/disk_tests/multipath/dm-2/dm/uuid | 1 + src/test/disk_tests/multipath/dm-2/size | 1 + src/test/disk_tests/multipath/dm-2_slaves | 1 + src/test/disk_tests/multipath/dm-3/dm/name | 1 + src/test/disk_tests/multipath/dm-3/dm/uuid | 1 + src/test/disk_tests/multipath/dm-3/size | 1 + src/test/disk_tests/multipath/dm-3_slaves | 1 + src/test/disk_tests/multipath/dm-4/dm/name | 1 + src/test/disk_tests/multipath/dm-4/dm/uuid | 1 + src/test/disk_tests/multipath/dm-4/size | 1 + src/test/disk_tests/multipath/dm-4_slaves | 1 + src/test/disk_tests/multipath/lsblk | 77 ++ src/test/disk_tests/multipath/lvs | 0 src/test/disk_tests/multipath/mounts | 1 + .../disk_tests/multipath/multipath_expected.json | 59 + src/test/disk_tests/multipath/multipathd | 124 ++ .../multipath/multipathd_status_expected.json | 42 + .../nvme-subsystem/nvme-subsys0/nvme3/address | 1 + .../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme3/state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/address | 1 + .../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/state | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/address | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/state | 1 + src/test/disk_tests/multipath/nvme8n1/device/model | 1 + .../disk_tests/multipath/nvme8n1/device/transport | 1 + .../disk_tests/multipath/nvme8n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme8n1/size | 1 + src/test/disk_tests/multipath/nvme8n1_udevadm | 14 + src/test/disk_tests/multipath/nvme9n1/device/model | 1 + .../disk_tests/multipath/nvme9n1/device/transport | 1 + .../disk_tests/multipath/nvme9n1/device/vendor | 1 + .../disk_tests/multipath/nvme9n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme9n1/size | 1 + src/test/disk_tests/multipath/nvme9n1_udevadm | 12 + .../multipath/nvme_path_status_expected.json | 19 + src/test/disk_tests/multipath/partlist | 0 src/test/disk_tests/multipath/pvs | 1 + src/test/disk_tests/multipath/sda/device/model | 1 + src/test/disk_tests/multipath/sda/device/vendor | 1 + src/test/disk_tests/multipath/sda/holders/holder | 1 + src/test/disk_tests/multipath/sda/queue/rotational | 1 + src/test/disk_tests/multipath/sda/size | 1 + src/test/disk_tests/multipath/sda_udevadm | 18 + src/test/disk_tests/multipath/sdb/device/model | 1 + src/test/disk_tests/multipath/sdb/device/vendor | 1 + src/test/disk_tests/multipath/sdb/holders/holder | 1 + src/test/disk_tests/multipath/sdb/queue/rotational | 1 + src/test/disk_tests/multipath/sdb/size | 1 + src/test/disk_tests/multipath/sdb_udevadm | 18 + src/test/disk_tests/multipath/sdc/device/model | 1 + src/test/disk_tests/multipath/sdc/device/vendor | 1 + src/test/disk_tests/multipath/sdc/holders/holder | 1 + src/test/disk_tests/multipath/sdc/queue/rotational | 1 + src/test/disk_tests/multipath/sdc/size | 1 + src/test/disk_tests/multipath/sdc_udevadm | 18 + src/test/disk_tests/multipath/sdd/device/model | 1 + src/test/disk_tests/multipath/sdd/device/vendor | 1 + src/test/disk_tests/multipath/sdd/holders/holder | 1 + src/test/disk_tests/multipath/sdd/queue/rotational | 1 + src/test/disk_tests/multipath/sdd/size | 1 + src/test/disk_tests/multipath/sdd_udevadm | 18 + src/test/disk_tests/multipath/zpool | 1 + src/test/disk_tests/sas/disklist_expected.json | 1 + src/test/disk_tests/sas/lsblk | 5 + src/test/disklist_test.pm | 117 +- src/test/lvm-fixtures/README.md | 20 + .../lvm-fixtures/lab1-nomda-orphan-derived.colon | 2 + src/test/lvm-fixtures/lab1-nomda-orphan.txt | 23 + src/test/run_block_device_users_tests.pl | 172 +++ src/test/run_disk_api_tests.pl | 11 + src/test/run_iscsi_activation_tests.pl | 171 +++ src/test/run_iscsi_device_list_tests.pl | 78 ++ src/test/run_iscsi_logout_tests.pl | 271 +++++ src/test/run_iscsi_rescan_tests.pl | 109 ++ src/test/run_iscsi_session_tests.pl | 15 +- src/test/run_iscsi_volume_tests.pl | 81 +- src/test/run_lun_identity_tests.pl | 1262 ++++++++++++++++++++ src/test/run_multipath_api_tests.pl | 693 +++++++++++ src/test/run_multipath_tests.pl | 1240 +++++++++++++++++++ src/test/run_san_api_tests.pl | 180 +++ src/test/run_san_library_tests.pl | 77 ++ src/test/run_san_lun_scan_tests.pl | 945 +++++++++++++++ src/test/run_san_lvm_create_tests.pl | 611 ++++++++++ src/test/run_san_taint_tests.pl | 80 ++ src/test/run_san_wipe_tests.pl | 673 +++++++++++ src/test/run_status_command_contention_tests.pl | 520 ++++++++ src/test/run_status_command_tests.pl | 189 +++ src/test/run_test_lvm_base_activation.pl | 130 ++ src/test/run_test_lvm_base_conflicts.pl | 62 + src/test/run_test_lvm_base_volume.pl | 8 +- src/test/run_test_lvm_create.pl | 243 ++++ src/test/run_test_lvm_hints.pl | 110 ++ src/test/run_test_lvm_status.pl | 220 ++++ src/test/run_wipe_partition_tests.pl | 387 ++++++ src/test/run_wipe_readiness_tests.pl | 138 +++ src/test/run_wipe_reference_tests.pl | 243 ++++ src/test/run_wipedisk_tests.pl | 110 ++ 145 files changed, 16554 insertions(+), 215 deletions(-) pve-docs: Thomas Lamprecht (2): storage: lvm: note the rescan after extending a shared volume group storage: document the cluster-wide multipath management pve-storage-iscsi.adoc | 13 +- pve-storage-lvm.adoc | 9 + pve-storage-multipath.adoc | 527 +++++++++++++++++++++++++++++++++++++++++++++ pvesm.adoc | 13 +- 4 files changed, 554 insertions(+), 8 deletions(-) pve-manager: Dietmar Maurer (10): ui: storage: allow changing the node for NFS/CIFS scans ui: storage: add guided remote storage wizard with NFS support ui: storage wizard: add SMB/CIFS support ui: storage wizard: add ZFS over iSCSI support ui: dc: storage: add remote storage wizard entry to the add menu ui: node: add SAN LUNs panel ui: san luns: show multipath path state api: nodes: expose iSCSI status and maintenance pvenode: add iscsi commands ui: node: san: add iSCSI target and session management Thomas Lamprecht (22): api: ceph: osd: keep iSCSI LUNs out of OSD creation pvestatd: track pending storage status commands across re-exec ui: storage: factor out the scan picker error mask ui: storage: nfs: show scan errors and empty results ui: storage wizard: preserve settings after partial setup ui: storage wizard: add Fibre Channel shared LVM setup ui: storage wizard: reconcile unanswered storage creation pvestatd: apply the multipath configuration and report the node status api: cluster: add the multipath endpoints ui: dc: add a cluster-wide multipath view ui: storage: iscsi: add the multipath option ui: san luns: show multipath state and manage LUN entries ui: storage wizard: add iSCSI support ui: storage wizard: iscsi: show cluster initiator names ui: storage wizard: iscsi: enable multipath for shared LVM ui: node: san: wipe a LUN that no storage uses ui: storage wizard: iscsi: reuse existing target connections ui: storage wizard: fibre channel: enable multipath for a LUN ui: storage wizard: let root@pam wipe a LUN with data ui: storage wizard: do not offer LUNs in use for a new volume group ui: san: show the local users of a LUN in its usage tooltip ui: san: check the other nodes for users before wiping a LUN PVE/API2/Ceph/OSD.pm | 3 + PVE/API2/Cluster.pm | 7 + PVE/API2/Nodes.pm | 7 + PVE/CLI/pvenode.pm | 48 ++ PVE/Service/pvestatd.pm | 34 + debian/control | 1 + test/Makefile | 10 + test/SANUtils_test.js | 204 +++++ test/StorageWizard_test.js | 128 +++ www/manager6/Makefile | 21 + www/manager6/SANUtils.js | 579 +++++++++++++ www/manager6/Utils.js | 47 ++ www/manager6/dc/Config.js | 28 +- www/manager6/dc/Multipath.js | 535 ++++++++++++ www/manager6/dc/MultipathWwidEdit.js | 233 ++++++ www/manager6/dc/SAN.js | 326 ++++++++ www/manager6/dc/StorageView.js | 13 +- www/manager6/node/Config.js | 24 + www/manager6/node/ISCSITargets.js | 615 ++++++++++++++ www/manager6/node/SANLuns.js | 771 ++++++++++++++++++ www/manager6/panel/SANNotice.js | 26 + www/manager6/storage/CIFSEdit.js | 29 +- www/manager6/storage/IScsiEdit.js | 38 + www/manager6/storage/NFSEdit.js | 13 + www/manager6/storage/wizard/CIFS.js | 118 +++ www/manager6/storage/wizard/CommonSettings.js | 75 ++ www/manager6/storage/wizard/FCRecovery.js | 242 ++++++ www/manager6/storage/wizard/FibreChannel.js | 1083 +++++++++++++++++++++++++ www/manager6/storage/wizard/IScsi.js | 707 ++++++++++++++++ www/manager6/storage/wizard/LunPicker.js | 101 +++ www/manager6/storage/wizard/LunSelect.js | 485 +++++++++++ www/manager6/storage/wizard/NFS.js | 89 ++ www/manager6/storage/wizard/NodeIdentities.js | 123 +++ www/manager6/storage/wizard/SanLvm.js | 179 ++++ www/manager6/storage/wizard/WipeStatus.js | 34 + www/manager6/storage/wizard/Wizard.js | 970 ++++++++++++++++++++++ www/manager6/storage/wizard/ZFSOverISCSI.js | 180 ++++ www/manager6/window/SANLunWipe.js | 331 ++++++++ 38 files changed, 8432 insertions(+), 25 deletions(-)