From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 2C4071FF14C for ; Fri, 26 Jun 2026 14:11:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 93556F6E6; Fri, 26 Jun 2026 14:10:54 +0200 (CEST) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH storage,cluster,manager 0/13] multipath: cluster-wide config, storage and health overview Date: Fri, 26 Jun 2026 14:07:30 +0200 Message-ID: <20260626121000.2095591-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1782475801051 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.145 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 25QRFUJW5HNFJOH7THMLGEBWWZKCTRZB X-Message-ID-Hash: 25QRFUJW5HNFJOH7THMLGEBWWZKCTRZB X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is a proof of concept for better multipath support in PVE. I talked it over with Friedrich back in May and put it together to get early feedback, also because there is other multipath and iSCSI work going on (for example Mira's storage mapping series). Today multipath is set up by hand on each node, with nothing managing it cluster-wide. This series tries to improve that by adding: - cluster-wide config: a new /etc/pve/multipath.cfg in pmxcfs, kept as a SectionConfig: a 'defaults' section for the global multipathd knobs plus one 'wwid' section per allow-listed LUN holding its optional alias and any per-LUN knobs. Free-form hardware overrides live in a separate /etc/pve/multipath-overrides.conf. pvestatd renders both into the local multipathd drop-ins, so you set it up once for the whole cluster. Map names stay WWID-based and equal on every node (user_friendly_names no, find_multipaths strict). - multipath as storage: a new 'multipath' storage type exposes the maps as raw volumes by WWID at the stable /dev/disk/by-id path. An LVM storage can use it as its 'base', so a shared volume group gets path redundancy with no manual device setup. This standalone type is deliberately provisional: the cleaner long-term shape is multipath as a capability toggled on the transport plugins (iSCSI/FC/NVMe-oF) rather than a peer storage type. It is kept standalone here to keep the POC self-contained, and because FC has no PVE transport plugin to hang such a capability off. - health overview: each node publishes its per-WWID map health into the pmxcfs KV store, and /cluster/multipath/status turns that into a per-WWID by per-node matrix with a rolled-up cluster-state. The web UI adds a Datacenter "Multipath" panel (table plus config editor) and a read-only per-node view under Disks. This matrix is the most generalizable piece; it is really a per-resource, per-node health roll-up and could become a small shared primitive that other features reuse. Everything keys on the (global) WWID, never the node-local sdX or mpathN names. Repo (build-)dependencies: pve-manager -> pve-storage -> pve-cluster. I tested it on a three-node cluster against an iSCSI target with two portals: the config reaches all nodes, the matrix flags a path fault on one node (rolling that LUN up to 'degraded') while the rest stay healthy, and a guest on the shared volume group migrates between nodes without copying its disk. Still a fairly simple test, so more would be needed to be sure nothing is off. Open points: - Health shows the paths a map has right now, so a node that fully lost a path (removed, not just failed) still looks fine on its last remaining path. The series does surface a node that lost all paths (one that is expected but silent) as missing, but not the "down to one of two" case; catching lost redundancy properly needs a notion of how many paths to expect, and I would like input on how to model that. The "expected nodes" set is derived here from where a multipath storage is enabled; sourcing it from the storage mapping series instead would be cleaner. - Upgrade and adoption: PVE rewrites /etc/multipath/wwids to match its allow-list, so it drops WWIDs that PVE did not add, which is risky on nodes whose multipath was set up by hand or that boot from SAN. There is also no migration from an existing multipath.conf, and a no-touch guarantee for boot-from-SAN devices needs thought. - The 'multipath' storage type is provisional. The alternative is to make path coalescing a property of the transport storages (multipath on iscsi, and in-kernel ANA on a future nvme-of) with LVM using the transport as its base, instead of a separate type. That gives fewer storages to set up for iSCSI, a natural home for NVMe-oF, and no extra peer type next to iscsi, at the cost of touching the transport plugins and the ongoing iSCSI/NVMe-oF work (Dietmar). I lean towards it as the target and would like opinions. - The pure logic (config parsing, health derivation, status aggregation) could move to rust crate(s) used from Perl via perlmod, with the cluster and multipathd glue wrapper code staying in Perl. - Whether the per-node trigger should stay in pvestatd or move to its own service or timer. - How this should fit with the storage mapping work. pve-storage: Thomas Lamprecht (7): multipath: add helper library and managed configuration api: disks: add read-only multipath status endpoint api: multipath: add cluster-wide configuration endpoints multipath: add storage plugin for multipath LUNs lvm: allow a multipath storage as the base device multipath: broadcast per-node map health to the cluster KV store api: multipath: add cluster-wide health status endpoint src/PVE/API2/Disks.pm | 7 + src/PVE/API2/Disks/Makefile | 1 + src/PVE/API2/Disks/Multipath.pm | 206 ++++++++++++++ src/PVE/API2/Makefile | 1 + src/PVE/API2/Multipath.pm | 538 +++++++++++++++++++++++++++++++++++++ src/PVE/Makefile | 4 + src/PVE/Multipath.pm | 447 ++++++++++++++++++++++++++++++ src/PVE/Multipath/ClusterConfig.pm | 55 ++++ src/PVE/Multipath/Config.pm | 361 +++++++++++++++++++++++++ src/PVE/Multipath/Generator.pm | 147 ++++++++++ src/PVE/Storage.pm | 2 + src/PVE/Storage/LVMPlugin.pm | 7 +- src/PVE/Storage/Makefile | 3 +- src/PVE/Storage/MultipathPlugin.pm | 186 +++++++++++++ src/test/Makefile | 5 +- src/test/run_multipath_tests.pl | 423 +++++++++++++++++++++++++++++ 16 files changed, 2388 insertions(+), 5 deletions(-) pve-cluster: Thomas Lamprecht (1): pmxcfs: track cluster-wide multipath configuration src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-manager: Thomas Lamprecht (5): pvestatd: apply the cluster-wide multipath config on each node api: cluster: mount the multipath configuration endpoint pvestatd: broadcast multipath map health to the cluster ui: dc: add multipath health matrix and config editor ui: node: show multipath maps and their paths under Disks PVE/API2/Cluster.pm | 7 + PVE/Service/pvestatd.pm | 14 ++ www/manager6/Makefile | 2 + www/manager6/Utils.js | 25 +++ www/manager6/dc/Config.js | 6 + www/manager6/dc/Multipath.js | 371 +++++++++++++++++++++++++++++++++++++++++ www/manager6/node/Config.js | 7 + www/manager6/node/Multipath.js | 163 ++++++++++++++++++ 8 files changed, 595 insertions(+)