[PATCH v3 docs 44/76] storage: document the cluster-wide multipath management

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Document managed multipath setup, health states, and upgrade
requirements so administrators can distinguish incomplete configuration
from unusable paths.

Cover shared LVM setup and recovery in the Remote Storage wizard,
including device checks, unanswered requests, task completion, and
readiness before guest use. Explain safe connection-storage removal,
growing LUNs, and presenting them again without queued I/O blocking
the operation.

Require content none in the iSCSI guidance for LVM, rather than merely
recommending it: initialization is refused while a storage uses the
target for disk images.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 pve-storage-iscsi.adoc     |  13 +-
 pve-storage-multipath.adoc | 527 +++++++++++++++++++++++++++++++++++++
 pvesm.adoc                 |  13 +-
 3 files changed, 545 insertions(+), 8 deletions(-)
 create mode 100644 pve-storage-multipath.adoc

diff --git a/pve-storage-iscsi.adoc b/pve-storage-iscsi.adoc
index aea26d2..67055f1 100644
--- a/pve-storage-iscsi.adoc
+++ b/pve-storage-iscsi.adoc
@@ -38,6 +38,13 @@ target::
 
 iSCSI target.
 
+multipath::
+
+Combine the paths to each LUN of the target into one `dm-multipath` map on every
+node the storage is enabled on, and monitor the maps cluster-wide. A shared LVM
+storage on a target with several portals needs this option. See
+xref:storage_multipath[Multipath].
+
 
 .Configuration Example (`/etc/pve/storage.cfg`)
 ----
@@ -47,9 +54,9 @@ iscsi: mynas
      content none
 ----
 
-TIP: If you want to use LVM on top of iSCSI, it make sense to set
-`content none`. That way it is not possible to create VMs using iSCSI
-LUNs directly.
+NOTE: To use LVM on top of iSCSI, set `content none`. When you add an LVM
+storage with a LUN as its base volume, {pve} refuses the operation if any
+configured storage also uses that target for disk images.
 
 
 File naming conventions
diff --git a/pve-storage-multipath.adoc b/pve-storage-multipath.adoc
new file mode 100644
index 0000000..6f6dc1a
--- /dev/null
+++ b/pve-storage-multipath.adoc
@@ -0,0 +1,527 @@
+[[storage_multipath]]
+Multipath
+---------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: Multipath
+endif::wiki[]
+
+A storage area network (SAN) usually exports each logical unit (LUN) over
+several paths, for example over two iSCSI portals or two Fibre Channel ports.
+Linux sees every path as a separate block device. Device-mapper multipath
+(`dm-multipath`) combines the paths of a LUN into one device, called a
+multipath map, keeps I/O flowing while a path is down, and spreads the load over
+the paths.
+
+{pve} manages `dm-multipath` cluster-wide: it decides which LUNs get a map,
+sets the name and options of each map, and monitors every path on every node. A
+LUN is identified by its World Wide Identifier (WWID), which is the same on
+every node. The configuration is stored in the cluster file system, every node
+generates its local `multipath-tools` configuration from it, and
+*Datacenter -> SAN -> Multipath* shows the health of the maps across the
+cluster.
+
+NOTE: NVMe over Fabrics namespaces use native NVMe multipath in the kernel
+instead of `dm-multipath`. They need no multipath configuration and do not
+appear under *Datacenter -> SAN -> Multipath*. They show up under
+*Node -> SAN*.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+Install `multipath-tools` on every node that accesses the SAN:
+
+----
+# apt install multipath-tools
+----
+
+A node without the package does not take part. *Datacenter -> SAN* shows it as
+*Not installed*, which counts as a problem once a LUN is expected on the node.
+
+Upgrade every node of the cluster before you enable the multipath option of a
+storage. A node with an older version does not know the option and removes it
+from every iSCSI storage the next time it writes the storage configuration,
+even for an unrelated change. The other nodes then stop managing the LUNs of
+those storages. {pve} therefore refuses to enable the option until every
+cluster node has reported multipath support, which a node does while it is
+online. It also refuses while `multipath-tools` is not installed on a node the
+storage is available on, as the volumes of the storage cannot be activated
+there without their maps. Upgrade a node before it joins a cluster that uses the
+option, as the check covers only the nodes that are members when you enable it.
+
+Managing LUNs
+~~~~~~~~~~~~~
+
+{pve} assembles maps only for the LUNs on its allow-list, so unrelated disks,
+the boot disk, and LUNs that other software uses directly stay untouched. A LUN
+gets onto the allow-list in one of two ways, and the *Source* column under
+*Datacenter -> SAN -> Multipath* shows which one applies:
+
+iSCSI storages::
+
+Enable *Multipath* in the edit dialog of the iSCSI storage or in the
+*Remote Storage* wizard, see
+xref:storage_multipath_setup[Setting Up Shared LVM]. Every node the storage is
+enabled on then assembles one map per LUN over the paths through the portals of
+the target. The storage lists its LUNs as multipath maps afterwards, and an LVM
+storage created on top of it uses the map as its base volume, see
+xref:storage_lvm[LVM].
++
+Without the option, {pve} logs in to every portal of the target as well, and
+each LUN shows up once per portal as a separate device. LVM ignores a volume
+group that it finds on several devices, so a shared LVM storage on a target with
+several portals needs the option. The option cannot be disabled while an LVM
+storage uses one of the maps as its base volume.
++
+You do not need to remove and add an existing LVM storage again when you enable
+the option: it can keep its base volume. LVM finds the physical volume by its
+metadata, not by the device name. Keep `multipath_component_detection` enabled
+in the LVM configuration (the default), so that LVM ignores the path devices of
+the LUN and uses the map. Logical volumes that are already active on a single
+path device stay there when you enable the option; before you rely on path
+redundancy, run `lsblk` on each node to see which device they use. Deactivating
+and activating a logical volume again moves it to the map, for example by
+migrating its guest to another node and back.
++
+The same can happen when a node logs in to the target for the first time, for
+example after it joined the cluster, or when the target presents a new LUN: the
+node adds a LUN to its allow-list only after it found it, and until then LVM can
+activate logical volumes with autoactivation enabled on a path device. {pve} 9
+creates new logical volumes for guests with autoactivation disabled. For older
+guest volumes, run
+`/usr/share/pve-manager/migrations/pve-lvm-disable-autoactivation`; for other
+logical volumes, run `lvchange --setautoactivation n <vg>/<lv>`.
++
+If the volume group is not visible yet, for example right after a node booted,
+activating a volume of the LVM storage also activates the base volume and waits
+briefly for the map. For a base volume that is still named by the SCSI
+identifier of the LUN, a timeout only logs a warning and activation continues;
+activation can still fail if the volume group remains unavailable.
+
+Other LUNs::
+
+LUNs attached over Fibre Channel or SAS, and LUNs of an iSCSI target without an
+iSCSI storage, need an explicit entry in the cluster-wide multipath
+configuration, shown as *Explicit entry* in the *Source* column. For Fibre
+Channel, the *Remote Storage* wizard can add the entry together with the volume
+group and the LVM storage, see
+xref:storage_multipath_setup[Setting Up Shared LVM]. To add an entry by hand:
++
+. Under *Node -> SAN*, select the LUN and choose *Enable Multipath*.
+. Review *Nodes*. For a new entry, the dialog preselects this node, the nodes
+  that see the LUN, and the nodes it could not check. Adjust the selection to
+  the nodes that should use the LUN. Optionally set an *Alias*, which becomes
+  the map name on every selected node.
+. Wait for the map to assemble, which takes up to about ten seconds, as every
+  node applies the configuration periodically.
+. Create the volume group on the map with `pvesh create
+  /nodes/<node>/disks/lvm --name <vg> --device
+  /dev/disk/by-id/dm-uuid-mpath-<wwid>` and check its output for errors. Then
+  add the shared LVM storage with
+  `pvesm add lvm <storage> --vgname <vg> --shared 1`.
+
+A map that {pve} does not manage, for example one from a hand-made
+configuration that predates the upgrade, shows up as *Unmanaged* under
+*Node -> SAN*. Select it and choose *Manage* to add it to the cluster-wide
+configuration.
+
+A node keeps a LUN on its allow-list while an explicit entry or an enabled
+storage requires it there. Once neither does, the node removes the LUN from its
+allow-list and `multipathd` releases the map, unless the LUN was added to the
+allow-list outside of {pve}. However, the node keeps the LUN on its allow-list
+while something on the node still uses the map, for example a volume group, a
+file system, or a partition table, as the users would otherwise fall back to a
+single path after the next reboot. The datacenter view and the SAN
+view of the node show such a LUN as *Kept while in use* until the node can
+confirm that the map is unused, including that no data signatures remain. To
+keep the LUN on the allow-list after it is no longer in use, choose *Manage* in
+the SAN view of the node, which adds an explicit entry for it.
+
+{pve} refuses to remove an explicit entry, or to restrict it to fewer nodes,
+while an LVM storage uses the map as its base volume on a node that would then
+drop the LUN from its allow-list. Remove or change that storage first. An LVM
+storage without a base volume, as used for Fibre Channel, is not linked to the
+LUN in the configuration. It is protected only because every node keeps the LUN
+on its allow-list while the volume group is on the map, as described above.
+
+[[storage_multipath_setup]]
+Setting Up Shared LVM
+~~~~~~~~~~~~~~~~~~~~~
+
+The *Remote Storage* wizard under *Datacenter -> Storage -> Add* sets up shared
+LVM storage on a LUN. For iSCSI, it creates the volume group from the node that
+the web interface is connected to. That node must reach the target and be
+included in any node restriction. For Fibre Channel, it uses the selected scan
+node and can enable multipath for the selected LUN, create a volume group or use
+an existing one, and add the LVM storage.
+
+For iSCSI, the wizard first adds or reuses a connection storage: an iSCSI
+storage with `content none` that connects the nodes to the target without
+offering its LUNs as disk images. *Connect and Select LUN* adds the connection
+storage and lists its LUNs. If a connection storage for the target already
+exists, the button is *Select LUN* and lists the LUNs of that storage. Select a
+LUN, then choose *Create Volume Group*. The LVM storage gets the LUN as its
+base volume, so that a node connects to the target before it activates the
+volume group, for example for guests that start at boot. Keep the base volume
+when you add such a storage by hand.
+
+The wizard offers only LUNs that no configured storage uses for data, that are
+neither mounted nor used by another device on the node, and on which the node
+finds no partitions, file system, LVM, or other signature. The node checks the
+device and its users again right before it creates the volume group. If no
+configured storage uses a device for data and nothing on the node uses it,
+`root@pam` can wipe it for reuse: in the wizard after selecting *Allow wiping
+LUNs with data*, or under *Node -> SAN*.
+
+Before you put guests on a new shared storage, check under
+*Datacenter -> SAN -> Multipath* that every node of the storage sees the LUN
+with all its paths.
+
+If a step fails after the wizard has created something, you can no longer
+change the settings in the wizard. *Retry Setup* runs the remaining steps again.
+If creating the LVM storage on iSCSI fails, retries use the same LUN and do not
+return to device selection, as the attempt may have written to it. If the wizard
+could not confirm that it created a storage, *Check and Retry* checks whether
+the storage exists with the expected settings and creates it only if it is
+missing. Closing the wizard does not remove the storages it already created.
+
+If the wizard cannot tell whether a wipe finished, it blocks the device until
+the outcome is known. *Check Task Status* waits for the same wipe task again; it
+does not start another wipe. If the wizard does not know the task ID, check the
+*Task History* of the node before you choose *Dismiss* or use the device.
+
+If the node does not answer while the Fibre Channel wizard creates the volume
+group, the wizard stops, as the task may still be running. Check the
+*Task History* of the node to see whether the task created the volume group.
+Once the task has finished, run the wizard again. If the volume group exists,
+the wizard offers it; select it to complete the setup.
+
+NOTE: When you use the API, a returned task ID only means that the node
+accepted the task, not that the task succeeded. Wait for the task to finish and
+check its exit status before you use the device. `pvesh` runs the task in the
+foreground; check its output for errors.
+
+Health
+~~~~~~
+
+*Datacenter -> SAN* shows one row per node with the state of its multipath
+service, the number of its managed LUNs in each state, and its iSCSI initiator
+name, which the targets of its iSCSI storages have to allow. A node that has not
+reported shows *No report*. A missing `multipath-tools` package or a stopped
+`multipathd` counts as a problem only on a node that should see a LUN.
+
+*Datacenter -> SAN -> Multipath* lists every managed LUN with its state on each
+node:
+
+*Optimal*:: Every path is active and no expected path is missing.
+*Degraded*:: Some paths are down, or fewer paths are present than the storage
+configuration expects.
+*Failed*:: No active path remains.
+*Missing*:: The node should see the LUN but has not assembled its map.
+*Unknown*:: The node has no current report. The view also shows why: the node
+is offline, its report is stale, `multipath-tools` is not installed,
+`multipathd` is not running or did not answer, or the node has too many LUNs to
+list them. A node that runs an older version, or has not reported since its
+cluster file system started, has no report at all.
+
+If the status updates of a node stall, for example while a status query waits
+for I/O to a LUN that lost every path, the view keeps a *Degraded*, *Missing*,
+or *Failed* state from the last report of the node, but shows *Unknown* instead
+of *Optimal*, as that report may be long outdated.
+
+A LUN of an iSCSI storage that no node finds on the target anymore, and that no
+node has a map of, shows *Not currently observed* instead of *Missing*. This
+usually means that the LUN was unmapped on the storage system, on purpose or by
+mistake, so check which one it was. The storage keeps the LUN on the allow-list,
+see xref:storage_multipath_remove[Removing a LUN]. A LUN stays *Missing* if it
+has an explicit entry or an LVM storage uses it. It also stays *Missing* while
+one of its nodes cannot tell whether the LUN is gone: the node runs an older
+version, could not apply the configuration, has no current report or stalled
+status updates, has not completed login and the initial LUN scan on every
+portal of the target, or cannot identify every LUN of the target.
+
+The *Cluster Health* of a LUN is the worst state among the nodes that should see
+it. Nodes without a current report do not count, so *Optimal* means that every
+node with a current report is fine, not that every node has reported. While the
+state on one of these nodes is unknown, the view shows *Optimal (incomplete)*. A
+node that should see the LUN but cannot manage multipath, because
+`multipath-tools` is not installed or `multipathd` is not running or does not
+answer, makes the LUN *Degraded*. An explicit entry without a node restriction
+expects the LUN on every node of the cluster.
+
+Nodes report the state of their LUNs through the cluster file system, which
+limits the size of a report. A node with more than about 200 LUNs reports only
+the number of its LUNs in each state, and the datacenter view refers to the SAN
+view of that node for details.
+
+For an iSCSI storage, {pve} knows how many paths a LUN should have: one per
+portal of the target. If several storages provide the same LUN, the LUN is
+expected on the nodes of all of them, with the highest path count among them.
+For Fibre Channel, {pve} only knows the paths that are currently present. If the
+transport has already removed a failed path, the map shows as *Optimal* with
+fewer paths, so compare the path count with what the SAN should provide.
+
+A node learns the portals of an iSCSI target when it first logs in to it. A
+portal that is added to the target later is neither used nor expected until the
+node discovers the target again. To start using a new portal, run the discovery
+on every node, against any portal of the target:
+
+----
+# iscsiadm --mode discovery --type sendtargets --portal <portal>
+----
+
+The next activation of the storage then also logs in to the new portal.
+
+Above the list, *Datacenter -> SAN -> Multipath* shows the nodes that could not
+apply the configuration, so a node that drifts from the configured state does
+not go unnoticed. It also shows nodes that cannot identify some LUNs of their
+iSCSI storages, because the identifiers of their paths are not available or
+could not be verified, for example while `multipathd` starts. Such a LUN is left
+out until the node can identify it.
+
+*Node -> SAN* lists each LUN of the node with its users and the state of its
+paths, and shows the problems of the node above the list. There, `root@pam` can
+also wipe a LUN that no configured storage uses for data, for example before
+reusing it. The wipe dialog first asks the other nodes what uses the LUN and
+refuses while one of them reports a user. *Node -> SAN -> iSCSI* shows the iSCSI
+targets and sessions of the node and its initiator name.
+
+Permissions
+~~~~~~~~~~~
+
+The multipath views and changes need these privileges:
+
+* Viewing the multipath status: `Sys.Audit` on `/` or `Datastore.Allocate` on
+  `/storage`.
+* Viewing the SAN views of a node: `Sys.Audit` on `/nodes/<node>` or
+  `Datastore.Allocate` on `/storage`.
+* Changing the cluster-wide multipath configuration through the API:
+  `Sys.Modify` on `/`.
+* Using *Enable Multipath*, *Manage*, or *Edit* in the multipath views:
+  `Sys.Modify` and `Sys.Audit` on `/`. The editors read the configuration before
+  changing it.
+* Changing the multipath option of a storage: `Datastore.Allocate` on
+  `/storage`.
+
+[[storage_multipath_grow]]
+Growing a LUN
+~~~~~~~~~~~~~
+
+{pve} lets a map grow with its LUN once every path of the map reports the new
+size, see `auto-resize` under xref:storage_multipath_config[Configuration]. To
+grow a LUN that carries a shared LVM volume group:
+
+. Grow the LUN on the storage system. Never shrink a LUN that is in use.
+. Make every path on every node report the new size. For iSCSI, {pve} rescans
+  the sessions of an active storage about every ten seconds, so this happens on
+  its own; a path of a session that is down follows once the session recovers.
+  For Fibre Channel, rescan each path on every node with
+  `echo 1 > /sys/block/<path>/device/rescan`, where `<path>` is a path device
+  of the map, such as `sdc`. NVMe over Fabrics targets notify the hosts
+  themselves.
+. Check the size of the map on every node, under *Node -> SAN* or with
+  `multipath -ll <wwid>`. If the map of a node did not follow, run
+  `multipathd resize map <wwid>` on that node.
+. Only then, on one node, grow the physical volume with
+  `pvresize /dev/disk/by-id/dm-uuid-mpath-<wwid>`. A node whose map is still
+  smaller cannot activate logical volumes in the added space.
+
+[[storage_multipath_remove]]
+Removing a LUN
+~~~~~~~~~~~~~~
+
+The device profiles of many storage systems make `dm-multipath` queue I/O while
+a map has no path, some of them without a time limit, so that a controller
+failover causes no I/O errors. If such a LUN is unmapped on the storage system,
+every process that reads its map blocks until a path returns or the retries run
+out. This includes LVM commands, which scan every device. The map cannot be
+removed while processes wait for it. The LVM status commands share a 15-second
+time limit. If they fail or time out, {pve} shows the affected LVM storages as
+not active on that node. The rest of the status update can take longer. Guests,
+backups, and other LVM commands can still hang. Therefore, turn off queueing
+before you unmap a LUN:
+
+. Stop the guests that use volumes on the LUN or move their disks to another
+  storage, and remove the LVM storage. Removing the storage entry does not stop
+  its users and keeps the volume group with its data.
+. Deactivate the volume group on every node with `vgchange -an <vg>`. Do not
+  remove the volume group or the physical volume unless you want to destroy the
+  data; removing them is not part of this procedure.
+. Make the map fail I/O instead of queueing it. If the LUN has an explicit
+  entry, note its `no-path-retry` setting and set it to `fail`, for example with
+  `pvesh set /cluster/multipath/wwid/<wwid> --no-path-retry fail`. Otherwise,
+  create a temporary entry for the nodes that see the LUN:
++
+----
+# pvesh create /cluster/multipath/wwid --wwid <wwid> --nodes <nodes> \
+    --no-path-retry fail
+----
+
+. On every node that sees the LUN, check in the output of `multipath -ll <wwid>`
+  that the `features` of the map no longer list `queue_if_no_path`. If a node is
+  offline, wait until it is back and check it as well before you continue.
+. On every node, note the path devices of the map that `multipath -ll <wwid>`
+  lists, and for each of them the identifier in `/sys/block/<path>/device/wwid`.
+  Then unmap the LUN on the storage system.
+. On every node:
+.. Remove the map with `multipath -f <wwid>`. If the map is still in use, do not
+   force the removal: find out what uses it, and retry after you have deleted
+   the path devices.
+.. For each noted path device, check that `/sys/block/<path>/device/wwid` still
+   shows the noted identifier, then delete the device with
+   `echo 1 > /sys/block/<path>/device/delete`. Skip a path device that is gone.
+   If one shows another identifier, stop: its name now belongs to another LUN.
+.. Check that `dmsetup ls` no longer lists the map.
+. Remove the temporary entry with `pvesh delete /cluster/multipath/wwid/<wwid>`,
+  or set `no-path-retry` of the explicit entry back to its previous value. If it
+  was not set before, unset it with
+  `pvesh set /cluster/multipath/wwid/<wwid> --delete no-path-retry`.
+
+If the LUN was already unmapped while its map queued I/O, processes that read
+the map hang, and changing the entry does not stop the queueing while they hold
+the map open. Run `multipathd disablequeueing map <wwid>` on every node that has
+the map to fail the queued I/O, then continue with noting the path devices and
+removing the map.
+
+An iSCSI storage keeps every LUN it has found on the allow-list for as long as
+the storage exists, as it cannot tell a LUN that was withdrawn on purpose from
+one that is only temporarily not visible. The datacenter view therefore keeps
+showing an unmapped LUN as *Not currently observed*. The allow-list entry is
+harmless, and if the LUN is presented again, it is managed again right away.
+Once you remove the storage or disable its multipath option, each node removes
+the allow-list entries that {pve} added, as soon as no other storage or
+explicit entry needs them and the map is unused. Allow-list entries that {pve}
+did not add stay in place.
+
+If the nodes lost a LUN, for example because it was unmapped by mistake, its
+paths report a size of zero after it is presented again, until they are
+rescanned. For iSCSI, the periodic session rescan of {pve} repairs the paths
+that still exist, and `iscsiadm --mode session --rescan` finds the LUNs whose
+paths were deleted. For Fibre Channel, rescan the remaining paths as described
+in xref:storage_multipath_grow[Growing a LUN], and rescan the host adapters to
+find the deleted paths, for example with `rescan-scsi-bus.sh` from the
+`sg3-utils` package.
+
+[[storage_multipath_disconnect]]
+Removing a Connection Storage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Removing an iSCSI storage leaves its sessions connected and does not erase any
+data. To remove such a connection storage and disconnect the nodes safely:
+
+. Find out what uses each LUN of the target, not only the one that an LVM
+  storage uses. Stop these users or move their data to another storage.
+. Remove the LVM storages on the connection storage, and deactivate their
+  volume groups on every node with `vgchange -an <vg>`. Removing a storage does
+  not deactivate its volumes or remove its data.
+. Remove the connection storage. Make sure that no other enabled storage uses
+  the same target, as it would connect again.
+. On every node, choose *Logout* for the sessions under *Node -> SAN -> iSCSI*.
+  Once the iSCSI sessions are gone, *Remove* deletes the saved target records.
+  A logout is refused if it would take away the last usable path of a LUN that
+  is in use; stop what uses it instead of forcing the removal.
+
+Snapshot Clones
+~~~~~~~~~~~~~~~
+
+Do not present a snapshot clone of a LUN that carries an LVM physical volume to
+the nodes that use the original LUN. LVM then sees the same physical volume
+twice and refuses to activate logical volumes of the volume group or to change
+it, so guests on that storage cannot start or migrate. The SAN view of a node
+flags such LUNs as *duplicate PV*, but that check relies on the information udev
+recorded and can miss a duplicate. To access the data of a clone, present it to
+another host, or give it a new identity with `vgimportclone` on a host that
+sees only the clone before presenting it to the nodes.
+
+[[storage_multipath_config]]
+Configuration
+~~~~~~~~~~~~~
+
+The cluster-wide configuration is stored in `/etc/pve/multipath.cfg`. It has an
+optional `defaults` section for global `multipathd` settings and one `wwid`
+section for each explicit entry. From it, every node generates drop-in files
+with the prefix `pve-` under `/etc/multipath/conf.d/`, and maintains its
+allow-list in `/etc/multipath/wwids` with the LUNs of its multipath-enabled
+storages and the explicit entries that apply to the node. A hand-made
+`/etc/multipath.conf` and allow-list entries that {pve} did not add stay in
+place.
+
+.Example (`/etc/pve/multipath.cfg`)
+----
+defaults: defaults
+        no-path-retry queue
+
+wwid: 3600140558631413f06d4840826b34782
+        alias san1-lun0
+        nodes pve1,pve2
+        path-selector service-time 0
+----
+
+The `defaults` section takes `find-multipaths`, `user-friendly-names`,
+`polling-interval`, `auto-resize`, `no-path-retry`, `path-grouping-policy`,
+`failback`, and `path-selector`. Keep `find-multipaths` at `strict`, so that
+`multipathd` assembles maps only for LUNs on the allow-list, and
+`user-friendly-names` at `no`, so that maps without an explicit alias use their
+WWIDs as names, consistently across nodes and without a node-local bindings
+file. `auto-resize` defaults to `grow_only`, so a map grows with its LUN, see
+xref:storage_multipath_grow[Growing a LUN]; `grow_shrink` also shrinks it, and
+`never` leaves resizing to you.
+
+The built-in device profiles of `multipath-tools` set `no-path-retry` and the
+path settings for many storage systems, and these profiles take precedence over
+a `defaults` section. {pve} therefore writes `no-path-retry`,
+`path-grouping-policy`, `failback`, and `path-selector` from its `defaults`
+section into an `overrides` section, which takes precedence over the device
+profiles. Leave these settings unset to keep the values from the profile of your
+storage system.
+
+A `wwid` section takes `alias`, `nodes`, and the per-LUN settings
+`no-path-retry`, `path-grouping-policy`, `failback`, `path-selector`,
+`rr-min-io-rq`, and `rr-weight`, which take precedence over the defaults and
+the device profiles. See the `multipath.conf` manual page for the meaning of
+each setting.
+
+NOTE: For storage systems with Asymmetric Logical Unit Access (ALUA), whose
+paths all use the `alua` or `sysfs` prioritizer, `multipathd` detects the path
+grouping policy by itself and ignores a configured `path-grouping-policy`, even
+that of a `wwid` section. Setting `path-grouping-policy` in the `defaults`
+section turns this detection off. For a policy that only a `wwid` section sets,
+turn the detection off with `detect_pgpolicy no` in an `overrides` section of
+`/etc/pve/multipath-overrides.conf`.
+
+NOTE: A LUN is identified by the WWID that `multipathd` computes for its paths.
+If you change how `multipathd` computes it, for example with `uid_attrs`, the
+WWIDs of the affected LUNs can change once `multipathd` restarts. The entries of
+their old WWIDs then stay on the allow-list until their storage is removed or
+its multipath option is disabled, see
+xref:storage_multipath_remove[Removing a LUN].
+
+Hardware-specific settings, that is, the `devices`, `overrides`, and
+`blacklist` sections of `multipath.conf`, belong in
+`/etc/pve/multipath-overrides.conf`, written as in `multipath.conf`. Edit them
+with *Edit Overrides* under *Datacenter -> SAN -> Multipath*; every node applies
+them like the rest of the configuration. An `overrides` section in this file
+takes precedence over the managed defaults, but a `devices` section does not
+take precedence over the settings that {pve} writes as overrides.
+
+The API under `/cluster/multipath` manages the configuration and reports the
+health, for example:
+
+----
+# pvesh create /cluster/multipath/wwid --nodes pve1,pve2 \
+    --wwid 3600140558631413f06d4840826b34782
+# pvesh get /cluster/multipath/status
+----
+
+ifdef::wiki[]
+
+See Also
+~~~~~~~~
+
+* link:/wiki/Storage[Storage]
+
+* link:/wiki/Storage:_iSCSI[Storage: iSCSI]
+
+* link:/wiki/Storage:_LVM[Storage: LVM]
+
+endif::wiki[]
diff --git a/pvesm.adoc b/pvesm.adoc
index 5bd24b2..6f4e776 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -96,11 +96,10 @@ TiB sized), these operations may take several minutes, or in extreme cases, even
 hours. If your setup is affected, create and remove snapshots while the VM is
 shut down, expecting a long task duration.
 
-^3^: With iSCSI, FibreChannel (FC), or SAS block storage as shared storage in a
-cluster, LVM is used to split the LUN into virtual disks. The
-https://pve.proxmox.com/wiki/Multipath[Multipath Guide] explains how to
-configure it in a cluster, including multipathing which is usually a
-requirement in these scenarios.
+^3^: With iSCSI, Fibre Channel (FC), or SAS block storage as shared storage in
+a cluster, LVM is used to split the LUN into virtual disks. A LUN is usually
+reachable over several paths; xref:storage_multipath[Multipath] describes how
+{pve} manages them in a cluster.
 
 ^4^: Since {pve} 9, snapshots as a volume chain have been available for VMs.
 These snapshots use separate volumes for the snapshot data and layer them.  For
@@ -421,6 +420,8 @@ See Also
 
 * link:/wiki/Storage:_iSCSI[Storage: iSCSI]
 
+* link:/wiki/Storage:_Multipath[Storage: Multipath]
+
 * link:/wiki/Storage:_LVM[Storage: LVM]
 
 * link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
@@ -463,6 +464,8 @@ include::pve-storage-iscsi.adoc[]
 
 include::pve-storage-iscsidirect.adoc[]
 
+include::pve-storage-multipath.adoc[]
+
 include::pve-storage-rbd.adoc[]
 
 include::pve-storage-cephfs.adoc[]
-- 
2.47.3