From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id EC3201FF0AF for ; Fri, 20 Nov 2026 09:48:06 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 36B53216BB; Fri, 20 Nov 2026 09:48:05 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: Re: [PATCH v8 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Date: Fri, 20 Nov 2026 09:45:20 +0100 Message-ID: <179516247284.2632472.8451368514027474084.b4-ty@b4> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261119194655.3163258-1-t.lamprecht@proxmox.com> References: <20261119194655.3163258-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795164479322 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.024 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TNRHLY53AECRVBPYGUEBUO4B6ECBPS4Z X-Message-ID-Hash: TNRHLY53AECRVBPYGUEBUO4B6ECBPS4Z X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 19 Nov 2026 20:46:05 +0100, Thomas Lamprecht wrote: > This is v8 of the combined and reduced series of the cluster-wide > multipath work [0][1] and Dietmar's RFC for guided remote storage setup > and SAN visibility [2]. > > If nothing grave comes up I plan to apply this as is. > > v8 is v7 [9] plus what an independent review of v7 turned up and held > up when checked against the code and the lab, i.e.: > > [...] Applied, big thanks here to Blockbridge [0], which where very reactive and tested basically every version here, ensuring a much better end result! [0]: https://www.blockbridge.com Compared to v8 as posted, I folded in fixes for quite a few more findings, but mostly rather targeted edge cases: - If the multipath map of a LUN was not set up on a node, for example because multipathd failed there, LVM on that node hid the PV on the LUN, so a wipe from there could destroy the volume group of a shared storage. The checks before a wipe or a volume group creation now see such PVs, and also read the LVM label directly. - The wipe and the volume group creation checked the LUN through its name, which can lead to another device by the time they write. Both now check the device they hold open. - Creating a volume group through the disk API on a LUN that already had one of the same name, for example from another node, just succeeded, so two guided setups on one LUN could end with two storages on the same volume group. This is refused now; only adding a storage with the LUN as base volume takes over an existing volume group. - A wipe of a multipath map without any working path could hang in its checks until the cluster lock timed out. It is now refused right away, as the volume group creation already did. - An entry with the name of a map instead of a WWID was accepted by 'multipath -a' without an error, but never showed up as added, so the node reloaded multipathd on every pass. Such an entry is now reported as refused. - The status report that each node publishes only had room for about 180 LUNs. It is stored compressed now, which raises that to about 700. - The LUN list, the checks and the status each worked out on their own which LUN an LVM storage uses as base, and did not always agree: a base with a partition suffix, for example, did not protect its LUN from a wipe. They now use one helper. - The cluster status still expected the LUN of a disabled LVM storage on its nodes, so the LUN showed as missing everywhere once its iSCSI sessions were gone. A disabled storage now expects nothing. - A hand-edited overrides file skipped the syntax check that the API does, so it could hold a multipaths section that the API refuses. Such a file now stops the apply with an error, instead of being installed. - A node without iscsiadm, but in the scope of an iSCSI storage with multipath management, could neither apply its Fibre Channel entries nor show Node > SAN. It now only asks iscsiadm about targets that have paths on that node. - Adding a second entry for the same WWID now shows the error at the WWID field, instead of as a generic error. - If the base volume of an LVM storage could not be activated, a status pass repeated slow LVM scans for it. It now stops at the first failure. - In the web UI, a LUN that is unknown on all nodes no longer lists every node in its state, the Target/Subsystem header is one string for translators, and the iSCSI setup says where to allow the initiator names. - The multipath chapter is shorter and simpler, and also covers the options of a multipath entry and that nothing notifies about a lost path. The web UI and pvestatd chapters mention the SAN panels and what pvestatd does for multipath. - Plus some rewording of comments, test descriptions and commit messages. Still open (just for sake of completness): - Nothing notifies about a lost path yet, so an admin has to look at the SAN view. A notification could build on the status that every node already reports. - Storage plugins cannot tell how many paths a volume should have, so a namespace that a plugin connected looks single-path once one of its paths is gone. A plugin method for the expected paths could add that. - NVMe over Fabrics support follows as v3 of its own series, on top of this. [pve-cluster] [01/43] cfs: observe the multipath configuration files commit: f196907143173a904a24a42f7c2a9f7f78c183c4 [pve-storage] [02/43] diskmanage: collect disk transport type from lsblk commit: d5167f815b6344718506e4a5449096ed9518bf4c [03/43] diskmanage: qualify NVMe over Fabrics transport commit: 968f1549f8acc3874d253de8a476664438a18d16 [04/43] diskmanage: optionally include iSCSI session devices in disk enumeration commit: 08c04c8d7a077ff03f4a6fbdab466037ed6e85dd [05/43] diskmanage: collect native NVMe multipath path state commit: 6e52aaf06509dc8314030793e1b76f785ee7367a [06/43] iscsi: factor out session device map from device list commit: e9e65e672d19cc5d1f33c13a5175cb1eee90fd34 [07/43] multipath: add a library around multipathd commit: dfe34094562cfc5de060ff835b6e75debc2be12a [08/43] diskmanage: add helper to list multipath devices commit: d3e5f28dae60dac121a46a42d63527e5db6f03b5 [09/43] multipath: add a cluster-wide configuration commit: 59948403ba368b3b771f4860f359c62d1347bd3e [10/43] iscsi: add an option to manage storage LUNs with multipath commit: 55dd1de9a1d8017b23ffe53c2815ce5fcfc5a217 [11/43] lvm: activate a missing base volume before using the volume group commit: 44bc0e3e4e4b261fc4cc76e47dc233f0b4768694 [12/43] multipath: apply the configuration on every node commit: 51b8a5c820b9ba68d511022350eab4e9cc614937 [13/43] multipath: publish per-node status commit: e72b1be2793c36e6991d87fe7707ba5288d05509 [14/43] api: add endpoints for the cluster-wide multipath configuration commit: f0d87e448c207bf166fd000bb880c35c551bfba5 [15/43] api: multipath: add the cluster-wide health status commit: c98ecf288ff4423d5064e51d19e28734b99a6040 [16/43] diskmanage: look up the requested disks by name commit: 8222e318de382ea22f554998dd876b67e8c08ee0 [17/43] diskmanage: read the udev data of all disks with one call commit: 298dcfaec978ae611e6c54d155bd1cdd80e6d7fb [18/43] api: scan: add a SAN LUN inventory commit: 2e5f0afad07ffd1845c7d0940e9662df48c72836 [19/43] pvesm: expose the SAN LUN inventory commit: 73091d3621e51ad5fccb316d2efd902c0570212d [20/43] api: add node-level iSCSI status commit: cdf8763f784d2eddadfacaa576d39b64b0c8dea4 [21/43] api: scan: name the portal of an iSCSI path commit: ad2ccb2ead41c6f4675d9a8d9e382dd59863d7f6 [22/43] disks: lvm: drop unused imports commit: 946c8422ed805512515864fe80b6aaae340315ec [23/43] disks: lvm: allow volume groups on SAN LUNs and multipath maps commit: 6a98c4ba61fb133a6dc82bf20931417f5c0c5ed8 [24/43] disks: lvm: refuse to destroy the volume group of a shared storage commit: 679d241b7bb4c016704b2fd6ed7983a4983efd98 [25/43] disks: allow wiping a SAN LUN that no storage uses commit: c5529caa6377f03654d121573659e1cfc0235074 [26/43] api: scan: report the process that has a SAN LUN open commit: 8e9935f595a664290e6edd4ae8c23b28fe631018 [pve-docs] [27/43] storage: document the cluster-wide multipath management commit: 5704d152297524600925469f2065e66d21d49300 [28/43] storage: multipath: describe the use shown for an open LUN commit: 64861a0de0a5e086f3f5557a6e9165cf75ff8996 [pve-manager] [29/43] pvestatd: apply the multipath configuration and report the node status commit: ec55d57a3288630c0690dea4e8d63280e70d58c3 [30/43] report: add the managed multipath configuration commit: a87eab65140ac4ae098ee4304830d3b9815565a1 [31/43] api: cluster: add the multipath endpoints commit: 0a6f615a5594135103c3f6aaedeaeefb08664114 [32/43] api: nodes: expose the iSCSI status commit: faaaadcf6f29c7cd7b1d938d88f0957ceba4d5f8 [33/43] ui: dc: add a SAN view commit: a335adb38a9b2b9d0a05a1e3afc720bf9a0897c1 [34/43] ui: storage: iscsi: add the multipath option commit: ca0bfb7e4b079e328f60037ba784ec2354c836b4 [35/43] ui: lvm: enable shared when a base storage is selected commit: 6cfb3b8ac945ae4b7b0b07ac122da7c5d5f20584 [36/43] ui: node: add SAN LUNs panel commit: d079fb4b3bd3126df29684822083922e2469afde [37/43] ui: node: san: wipe a LUN that no storage uses commit: 966f798adacd34d5b726ae9e6265a8a02a7ec5c5 [38/43] ui: node: san: show iSCSI targets and sessions commit: 56482f5dd485d37a06f33329216c375e135d5b33 [39/43] ui: dc: san: add a guided setup for shared LVM on iSCSI commit: e2094f57fff7b8058060b0fb7d0410c2ab3ec744 [40/43] ui: dc: san: add a guided setup for shared LVM on Fibre Channel commit: 39162712816a5fc715d00a07d86f683b9a005bf9 [41/43] ui: storage: point the iSCSI and LVM add dialogs to the SAN view commit: ab30edfcf78d6b6017ebb82354a20402dab22d3b [42/43] ui: san: show the guest or process that has a LUN open commit: 188f844baad0ce05b701f0f1a9018ff05f207a25 [43/43] ui: node: san: add a column for the target or subsystem of a LUN commit: 215871e61ea8ab696d5a4400707d2ea18fb46fa2