From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id B97B01FF0C0 for ; Wed, 18 Nov 2026 07:59:33 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 0099B21BE1; Wed, 18 Nov 2026 07:54:20 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v6 docs 26/42] storage: document the cluster-wide multipath management Date: Wed, 18 Nov 2026 07:52:43 +0100 Message-ID: <20261118065302.3033937-27-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118065302.3033937-1-t.lamprecht@proxmox.com> References: <20261118065302.3033937-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794984788623 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.373 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QCM4JO7J37O2LKKEZ6UAFB2ATFMT6IH3 X-Message-ID-Hash: QCM4JO7J37O2LKKEZ6UAFB2ATFMT6IH3 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add a chapter for multipath on iSCSI and Fibre Channel SANs, written around the tasks an admin has: prerequisites, setting up shared LVM with the guided setup or by hand, adding, growing and removing a LUN, reading the states during a path failure, array-specific settings and moving over from a hand-made multipath.conf. Removing a LUN stays a manual procedure, as Proxmox VE never removes a WWID or a map by itself, so the chapter spells it out including how to get out of queued I/O. Signed-off-by: Thomas Lamprecht --- pmxcfs.adoc | 2 + pve-storage-iscsi.adoc | 11 ++ pve-storage-iscsidirect.adoc | 2 +- pve-storage-lvm.adoc | 3 + pve-storage-multipath.adoc | 301 +++++++++++++++++++++++++++++++++++ pvesm.adoc | 18 ++- 6 files changed, 331 insertions(+), 6 deletions(-) create mode 100644 pve-storage-multipath.adoc diff --git a/pmxcfs.adoc b/pmxcfs.adoc index 1128425..2ab3fb3 100644 --- a/pmxcfs.adoc +++ b/pmxcfs.adoc @@ -105,6 +105,8 @@ Files |`ha/manager_status` | JSON-formatted information regarding HA services on the cluster |`ha/resources.cfg` | Resources managed by high availability, and their current state |`ha/rules.cfg` | Rules putting constraints on the HA manager's scheduling of HA resources +|`multipath-overrides.conf` | Hardware-specific multipath settings, in `multipath.conf` format, applied on all nodes +|`multipath.cfg` | {pve} cluster-wide multipath configuration (LUN entries) |`nodes//config` | Node-specific configuration |`nodes//lxc/.conf` | VM configuration data for LXC containers |`nodes//openvz/` | Prior to {pve} 4.0, used for container configuration data (deprecated, removed soon) diff --git a/pve-storage-iscsi.adoc b/pve-storage-iscsi.adoc index a02ed46..2dcc196 100644 --- a/pve-storage-iscsi.adoc +++ b/pve-storage-iscsi.adoc @@ -23,6 +23,10 @@ resources. Low-level iscsi management task can be done using the `iscsiadm` tool. +TIP: To set up shared LVM on a LUN of an iSCSI target, use the guided setup +under *Datacenter -> Storage -> SAN* in the web UI, see +xref:storage_multipath_setup[Multipath]. + Configuration ~~~~~~~~~~~~~ @@ -38,6 +42,13 @@ target:: iSCSI target. +managed-multipath:: + +Combine the paths to each LUN of the target into one `dm-multipath` map on every +node the storage is enabled on, and monitor the maps cluster-wide. A shared LVM +storage on a target with several portals needs multipath, through this option +or set up by hand. See xref:storage_multipath[Multipath]. + .Configuration Example (`/etc/pve/storage.cfg`) ---- diff --git a/pve-storage-iscsidirect.adoc b/pve-storage-iscsidirect.adoc index ae722e8..97b0979 100644 --- a/pve-storage-iscsidirect.adoc +++ b/pve-storage-iscsidirect.adoc @@ -21,7 +21,7 @@ Configuration ~~~~~~~~~~~~~ The user mode iSCSI backend uses the same configuration options as the -Open-iSCSI backend. +Open-iSCSI backend, except for `managed-multipath`. .Configuration Example (`/etc/pve/storage.cfg`) ---- diff --git a/pve-storage-lvm.adoc b/pve-storage-lvm.adoc index b148832..66d896e 100644 --- a/pve-storage-lvm.adoc +++ b/pve-storage-lvm.adoc @@ -143,6 +143,9 @@ A benefit of LVM is that it can be used with shared storage. For example, an iSCSI LUN. The backend implements proper cluster-wide locking if the storage is marked as shared in the configuration. +TIP: The guided setup under *Datacenter -> Storage -> SAN* in the web UI creates +a shared LVM storage on a SAN LUN, see xref:storage_multipath_setup[Multipath]. + NOTE: If you extend a shared volume group onto another device on one node, the other nodes may only see the added physical volume after LVM has scanned their devices again, for example with `pvscan --cache`. This command also resets the diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc new file mode 100644 index 0000000..ac8b4cf --- /dev/null +++ b/pve-storage-multipath.adoc @@ -0,0 +1,301 @@ +[[storage_multipath]] +Multipath +--------- +ifdef::wiki[] +:pve-toplevel: +:title: Storage: Multipath +endif::wiki[] + +Use multipath for shared LVM on an iSCSI or Fibre Channel SAN with redundant +connections. Device-mapper multipath combines the paths to a LUN into one block +device, called a map, so I/O can continue when a path fails. Its World Wide +Identifier (WWID) identifies the same LUN on every node. + +{pve} configures maps cluster-wide and shows their state under *Datacenter -> +Storage -> SAN*. An iSCSI storage with *Multipath* enabled requests maps for its +LUNs; Fibre Channel LUNs need entries in `/etc/pve/multipath.cfg`. NVMe +namespaces use kernel-native multipath instead; the procedures below cover SCSI +LUNs. + +Prerequisites +~~~~~~~~~~~~~ + +. Upgrade all nodes before enabling cluster-wide multipath management. +. Install `multipath-tools` on every participating node. The + `libpve-storage-perl` package recommends it, but installations without + recommended packages need it added separately. A node without it shows + *Multipath unavailable* with the reason under *Datacenter -> Storage -> SAN*. + Plain iSCSI and LVM without the multipath option still work without it. +. Allow each node's iSCSI initiator name on the array; the names are shown under + *Datacenter -> Storage -> SAN* and in the guided setup. Install `open-iscsi` + if needed. + For Fibre Channel, configure zoning and LUN masking for the WWPNs of each + node's HBA ports, which `cat /sys/class/fc_host/host*/port_name` lists. +. Present the same empty LUN to all intended nodes over redundant paths. A node + does not scan for a Fibre Channel LUN that is mapped to it later, so rescan + the Fibre Channel hosts on each node: ++ +---- +# for host in /sys/class/fc_host/host*; do echo '- - -' > /sys/class/scsi_host/${host##*/}/scan; done +---- ++ +If the LUN still does not show up, check the target ports of the array on the +node. This lists the WWPN, state and roles of each remote port; those of the +array need the state `Online` and the role `FCP Target`. A `Blocked` port needs +its connection restored before a rescan can reach it: ++ +---- +# for port in /sys/class/fc_remote_ports/rport-*; do echo $(cat $port/{port_name,port_state,roles}); done +---- + +[[storage_multipath_setup]] +Setting Up Shared LVM +~~~~~~~~~~~~~~~~~~~~ + +. Open *Datacenter -> Storage -> SAN* in the web UI on a node that can access + the LUN and select *Guided Setup -> LVM on iSCSI* or *LVM on Fibre Channel*. +. For iSCSI, enter a portal, select the target and name the connection storage. + This storage connects to the target with `content none` and *Multipath* + enabled, while the LVM storage manages guest disks. Continuing creates or + reuses the connection storage before you select a LUN. +. Select the LUN, choose the storage ID and volume group, then confirm. For + Fibre Channel, the setup adds the multipath entry of the LUN if it has none. + It then waits for the map on the node serving the web UI and sets *Expected + Paths* of that entry to the number of paths of the map, if that is at least + two. Existing entries and maps stay as they are; a map that was set up by + hand gets no entry, so check that the other nodes have it as well. Set + *Expected Paths* in the entry if it is missing or does not match the intended + topology, for example because a fabric was down at that time or nodes have + different path counts. Leave *Nodes* empty for all nodes, or for all nodes of + a reused connection storage. Virtual machines on the storage can only + have snapshots with *Allow Snapshots as Volume-Chain*, a technology preview + that applies to disks created after it is set; see + xref:pvesm_lvm_config[LVM configuration]. +. Check the state of the LUN on each node of the storage, which the setup + shows once the storage is added. Before that, only the node serving the web + UI looked at the LUN; the LUN list names it as *Scan node*. The other nodes + set the LUN up with their next status updates and show *Pending* during the + first minute. For a node that stays *Missing*, check its access to the LUN + on the array before placing guests on the storage. *OK (1 path)* means that + the node reaches the LUN over a single path. The *State* column of the view + shows the same later. + +A LUN that a storage uses cannot be selected, and the setup refuses a LUN that +contains data. To reuse such a LUN, first stop all its users on all hosts and +remove its storage references, then use *Wipe* under *Node -> SAN* as +`root@pam`. This destroys its data. Before the setup or the wipe writes to a +LUN, the node checks for processes that have it open, also through another of +its paths, for example a guest that uses it as a disk. This does not find every +user: not one that opened the LUN through another device node, and none on other +hosts. To keep a volume group that a LUN already holds, do not wipe the LUN: add +the volume group under *Datacenter -> Storage -> Add -> LVM* with *Shared* +enabled. The usage shown for a LUN is cached per node, so other nodes can still +show the old content after a wipe. If setup fails, check the error, the node's +task log and *Datacenter -> Storage* before retrying; closing the dialog does +not undo changes. + +The dialogs under *Datacenter -> Storage -> Add* can do the same for iSCSI in +two steps: add an *iSCSI* storage with *Multipath* enabled and *Use LUNs +directly* cleared, then an *LVM* storage with it as *Base storage* and the LUN +as *Base volume*. Selecting a base storage enables *Shared*; keep it enabled. + +To set up iSCSI by hand, run these commands on a node with access to the target: + +---- +# pvesm add iscsi --portal --target \ + --content none --managed-multipath 1 +# pvesm list --content images +# pvesm add lvm --vgname --base --shared 1 +---- + +Use the complete volume ID from the listing, including the connection storage +prefix and the `dm-uuid-mpath-` suffix. Adding the LVM storage creates the +volume group. Keep its base volume: it connects the target before LVM +activation. + +For Fibre Channel, add an entry with the number of paths each node should +have, wait for the scan to show a by-id path with `dm-uuid-mpath-`, then +create the volume group and shared storage: + +---- +# pvesh create /cluster/multipath/wwid --wwid --paths-expected +# pvesm scan san-luns --output-format json +# pvesh create /nodes//disks/lvm --name \ + --device /dev/disk/by-id/dm-uuid-mpath- --add_storage 1 +---- + +These examples allow all nodes. For a subset, set `--nodes ` on the +entry or iSCSI storage and on the LVM storage (`pvesm set --nodes +`), using a comma-separated list. `pvesh` waits for the creation task; +check its output for errors before using the storage. + +Adding Another LUN +~~~~~~~~~~~~~~~~~~ + +. Present the new LUN to the same nodes on the array. Enabled iSCSI storages + rescan their sessions automatically; for Fibre Channel, rescan the hosts as + shown in the prerequisites. +. Run the guided setup again. For iSCSI, select the same target to reuse its + connection storage. If it lacks *Multipath*, enable that option in its edit + dialog first, observing the migration advice below. +. Select the new LUN and create a separate LVM storage. By hand, repeat the + iSCSI listing and LVM command above, or the Fibre Channel procedure. + +[[storage_multipath_grow]] +Growing a LUN +~~~~~~~~~~~~~ + +. Grow the LUN on the array. Never shrink a LUN in use. +. Wait for the iSCSI session rescan, or rescan each SCSI path on every Fibre + Channel node with `echo 1 > /sys/block//device/rescan`, where `` + is a member device such as `sdc`, not the map. +. Check the map size on every node with `multipath -ll ` or under + *Node -> SAN*. The generated configuration sets `auto_resize grow_only`: + the map grows once all its paths report the new size. If necessary, run + `multipathd resize map ` after all paths have grown. +. Run `pvresize /dev/disk/by-id/dm-uuid-mpath-` on one node only, after + every node's map has grown. A smaller map cannot access the added space. + +[[storage_multipath_remove]] +Removing a LUN +~~~~~~~~~~~~~~ + +{pve} never flushes maps or removes WWIDs from `/etc/multipath/wwids`. +Removing a configuration entry is not enough. A map that queues I/O without +paths can block guests and LVM commands, so disable queueing before unmapping: + +. Stop all users of the LUN on all hosts, or move their data elsewhere. Remove + the LVM storage with `pvesm remove ` and deactivate its volume group + on every node with `vgchange -an `. Neither command erases the data. +. Set an existing entry to fail I/O with + `pvesh set /cluster/multipath/wwid/ --no-path-retry fail`. + If none exists, create one with + `pvesh create /cluster/multipath/wwid --wwid --no-path-retry fail`. + Add `--nodes ` if only a subset of nodes sees the LUN. +. Wait until `multipath -ll ` on every participating node no longer lists + `queue_if_no_path` in the map's features. Wait for offline nodes to return. + Record the map name and member devices, and each member's identifier from + `/sys/block//device/wwid`. +. Unmap the LUN on the array. +. On every node, flush the map with `multipath -f `. If it is busy, retry + after a few seconds; if it stays busy, find and stop its remaining users. Do + not force removal. For each recorded path, + check that its identifier is unchanged, then delete it with + `echo 1 > /sys/block//device/delete`. Skip paths already gone; stop if + an identifier changed. +. Remove the explicit entry with `pvesh delete /cluster/multipath/wwid/`. + Removing it earlier makes the nodes apply the default retry policy of the + array again, which can queue I/O on a map that is not flushed yet and block + whatever opens it. +. On every node, wait until its next status update has removed the WWID from + `/etc/multipath/conf.d/pve-aliases.conf`, which takes about ten seconds, then + run `multipath -w ` to remove it from the allow-list. Running it before + that update can let the update add the WWID again. Check that + `grep -c '^//$' /etc/multipath/wwids` prints 0, and again after the + next status update. + +If the LUN was already unmapped and I/O is stuck, run +`multipathd disablequeueing map ` on each affected node, then continue +with the removal procedure. + +[[storage_multipath_disconnect]] +Removing an iSCSI connection storage does not log out its sessions. Keep it +while other LUNs of the target are in use; session maintenance is done with +`iscsiadm`, not the read-only *Node -> SAN -> iSCSI* view. + +Checking Path Failures +~~~~~~~~~~~~~~~~~~~~~ + +*Datacenter -> Storage -> SAN* shows node availability, configuration errors +and LUN states. It lists the LUNs that {pve} manages or uses: those with an +explicit entry, those of an iSCSI storage with *Multipath* enabled, and those +that are the base volume of a storage or hold its volume group. Other maps of a +node, for example the ones of a third-party storage plugin, are only counted +under *Other Maps*; *Node -> SAN* lists them as well. The volume groups come +from the hint file of LVM. While LVM has none, for example because it sees +duplicate PVs, a node cannot tell which of its other maps hold the volume group +of a storage and leaves them all out. A LUN that is only listed for its volume +group then shows *Unknown* for that node, or is not listed if no node reports +it. + +The *State* of a LUN is the worst state among its nodes. For a state other than +*Optimal*, it names the nodes in that state, unless all nodes are in it. Nodes +in *Unknown* do not count for it and are named after it. Hover over it for the +states and path counts of all nodes. *Multipath* describes how the map is +managed, not its health: by an explicit entry, by the storage that requests it, +or *Unmanaged* for a map that {pve} did not set up. Under *Node -> SAN*, a +degraded or failed LUN shows its usable and expected paths; expand it to +inspect individual paths. +*Node -> SAN -> iSCSI* shows targets and sessions. + +*Optimal*:: All known paths are usable and no expected path is missing. +*OK (1 path)*:: The only path is usable, and there is no other path to fail +over to. The API reports this as `optimal` with one path. +*Degraded*:: Some paths are unusable or fewer paths exist than expected. +*Failed*:: No usable path remains; I/O queues or fails according to the retry +policy. +*Missing*:: A node with a current report is expected to have the LUN but does +not. +*Unknown*:: No usable current observation is available for that node. + +During a path failure, inspect the degraded LUN's paths and restore the failed +connection. The state returns to *Optimal* when the paths recover. Check node +availability too: unknown observations do not worsen a known LUN state. + +The status report of a node is limited to 32 KiB, which is enough for about 180 +listed LUNs with typical names. A node with more reports `too many LUNs to +report` and no LUNs: those that the configuration expects on it show *Unknown* +for that node, and LUNs that only this node reports are missing from the list. + +An iSCSI storage expects one path per portal. For a Fibre Channel LUN, set +*Expected Paths* in its multipath entry, as the guided setup does. Without it, +a node only shows as *Degraded* while its lost paths are still listed. +Depending on the settings for the array, the kernel removes a path that stays +down longer than `dev_loss_tmo`, and the map then shows *Optimal* with fewer +paths, or *OK (1 path)* for the last one. + +[[storage_multipath_config]] +Array Settings and Existing Configurations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the built-in device profiles unless the array vendor requires changes. +*Edit Overrides* under *Datacenter -> Storage -> SAN* edits +`/etc/pve/multipath-overrides.conf` for all nodes. For example, to set a retry +count for one array model (replace the vendor and product expressions): + +---- +devices { + device { + vendor "" + product "" + no_path_retry 12 + } +} +---- + +The generated configuration sets `auto_resize grow_only` for the growing of +LUNs. `multipath` only has this setting for all maps of a node, so it also +applies to maps that {pve} does not manage, for example the ones of a +third-party storage plugin: they grow once all their paths have grown. To keep +the resize manual on all nodes, add this to the overrides: + +---- +defaults { + auto_resize never +} +---- + +To move from a hand-made `/etc/multipath.conf`: + +. Back up the local configuration on every node. Move shared array settings to + the overrides file; add WWID entries under *Datacenter -> Storage -> SAN* for + Fibre Channel LUNs or per-LUN aliases and `no-path-retry` settings. +. Enable *Multipath* on existing iSCSI storages. Keep existing LVM base volumes. + Check `multipathd show config` on every node before removing duplicate local + settings. Do not edit the generated `/etc/multipath/conf.d/pve-*.conf` files. +. Wait until *Datacenter -> Storage -> SAN* shows the LUNs of these storages on + every node. Then check active volume dependencies with `lsblk` on each node. + Enabling multipath does not move active volumes off individual paths. Stop and + restart affected guests, or migrate them away and back, before relying on path + redundancy. Keep LVM component detection enabled and follow the + xref:storage_lvm[shared LVM autoactivation guidance]. diff --git a/pvesm.adoc b/pvesm.adoc index 07a7917..f7de231 100644 --- a/pvesm.adoc +++ b/pvesm.adoc @@ -96,11 +96,10 @@ TiB sized), these operations may take several minutes, or in extreme cases, even hours. If your setup is affected, create and remove snapshots while the VM is shut down, expecting a long task duration. -^3^: With iSCSI, FibreChannel (FC), or SAS block storage as shared storage in a -cluster, LVM is used to split the LUN into virtual disks. The -https://pve.proxmox.com/wiki/Multipath[Multipath Guide] explains how to -configure it in a cluster, including multipathing which is usually a -requirement in these scenarios. +^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in +a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually +reachable over several paths; xref:storage_multipath[Multipath] describes how +{pve} manages them in a cluster. ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs. These snapshots use separate volumes for the snapshot data and layer them. For @@ -153,6 +152,11 @@ also useful for local storage types. In this case such local storage is available on all nodes, but it is physically different and can have totally different content. +Add storage under *Datacenter -> Storage -> Add* in the web UI. For shared LVM +on a xref:storage_multipath_setup[SAN LUN], use *Guided Setup* under +*Datacenter -> Storage -> SAN* instead. It creates the storages that are needed +and enables multipath for the LUN. + Storage Pools ~~~~~~~~~~~~~ @@ -500,6 +504,8 @@ See Also * link:/wiki/Storage:_iSCSI[Storage: iSCSI] +* link:/wiki/Storage:_Multipath[Storage: Multipath] + * link:/wiki/Storage:_LVM[Storage: LVM] * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin] @@ -542,6 +548,8 @@ include::pve-storage-iscsi.adoc[] include::pve-storage-iscsidirect.adoc[] +include::pve-storage-multipath.adoc[] + include::pve-storage-rbd.adoc[] include::pve-storage-cephfs.adoc[]