From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id A57B91FF143 for ; Sat, 25 Apr 2026 14:18:07 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 11359E7; Sat, 25 Apr 2026 14:18:06 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH manager 1/1] fix #7518: ceph: mon: advertise a single address per monitor Date: Sat, 25 Apr 2026 20:15:28 +0800 Message-ID: <20260425121528.924363-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1777119243267 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.326 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: A37CKLDAWYEYL5M3DZSGNNLFPHGOUPRO X-Message-ID-Hash: A37CKLDAWYEYL5M3DZSGNNLFPHGOUPRO X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: When public_network lists multiple subnets (added in 0b6a2838 to support clusters spanning more than one subnet) and the local node has IPs in more than one of them, $find_mon_ips collected one address per subnet and the monaddr loop emitted both a v2 and v1 entry for every IP. This produced addrvecs of the shape: [v2:ip1:3300, v1:ip1:6789, v2:ip2:3300, v1:ip2:6789] The kernel libceph decoder rejects these entries with "another match of type N in addrvec", because it permits at most one entry of each messenger type per monitor. Userspace has tolerated this since Nautilus by picking the first matching entry, but no consumer of the addrvec ever falls back to later entries, so the additional addresses provide no functional benefit. The mismatch manifests as CephFS mount failures on the kernel client while RBD continues to work. Now pick a single IP from the listed networks (first match in the order given by public_network), and warn that the other subnets must be reachable via routing for clients on them. The change is limited to the auto-detection path; explicit 'mon-address' values are honored as before. The comma-split parser for public_network introduced in 0b6a2838 is preserved, so the original failure from #2422 stays fixed. Existing clusters with multi-entry monmaps from earlier pve-manager versions are unaffected by this change but can fix their monmap manually: ceph mon set-addrs '[v2::3300/0,v1::6789/0]' Signed-off-by: Kefu Chai --- PVE/API2/Ceph/MON.pm | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm index 18407b1c..01de9eac 100644 --- a/PVE/API2/Ceph/MON.pm +++ b/PVE/API2/Ceph/MON.pm @@ -58,7 +58,14 @@ my $find_mon_ips = sub { my $res = []; - if (!scalar(@{$overwrite_ips})) { # auto-select one address for each public network + if (!scalar(@{$overwrite_ips})) { + # auto-select a single address even when public_network lists + # several subnets: the kernel libceph decoder rejects monmap + # entries with more than one address per messenger type (#7518), + # and the userspace messenger has only ever used the first match. + # other subnets remain reachable via routing. + my $candidates = []; + my $candidate_origin = {}; for my $net (@{$public_nets}) { my $allowed_ips = PVE::Network::get_local_ip_from_cidr($net); $allowed_ips = PVE::Network::unique_ips($allowed_ips); @@ -67,13 +74,24 @@ my $find_mon_ips = sub { "No active IP found for the requested ceph public network '$net' on node '$node'\n" if scalar(@$allowed_ips) < 1; - if (scalar(@$allowed_ips) == 1) { - push @{$res}, $allowed_ips->[0]; - } else { - die "Multiple IPs for ceph public network '$net' detected on $node:\n" - . join("\n", @$allowed_ips) - . "\nuse 'mon-address' to specify one of them.\n"; - } + die "Multiple IPs for ceph public network '$net' detected on $node:\n" + . join("\n", @$allowed_ips) + . "\nuse 'mon-address' to specify one of them.\n" + if scalar(@$allowed_ips) > 1; + + push @{$candidates}, $allowed_ips->[0]; + $candidate_origin->{ $allowed_ips->[0] } = $net; + } + + push @{$res}, $candidates->[0]; + + if (scalar(@{$candidates}) > 1) { + my $picked = $candidates->[0]; + my $picked_net = $candidate_origin->{$picked}; + warn "Multiple candidate IPs for monitor on $node across networks ($pubnet);" + . " picked $picked from '$picked_net'." + . " Other subnets must be reachable via routing for clients on them." + . " Use 'mon-address' to override.\n"; } } else { # check if overwrite IPs are active and in any of the public networks my $allowed_list = []; -- 2.47.3