From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 6003F1FF0A7 for ; Wed, 16 Sep 2026 07:22:01 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 25EFD213AC; Wed, 16 Sep 2026 07:21:57 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH v2 manager 1/1] fix #7518: ceph: mon: advertise a single address per monitor Date: Wed, 16 Sep 2026 13:21:45 +0800 Message-ID: <20260916052145.1524871-2-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260916052145.1524871-1-k.chai@proxmox.com> References: <20260916052145.1524871-1-k.chai@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789536097811 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.133 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: BI3YLCV54B6W53Z3GMGVKPZJPTXEXXM7 X-Message-ID-Hash: BI3YLCV54B6W53Z3GMGVKPZJPTXEXXM7 X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: $find_mon_ips picks one local IP per subnet listed in public_network, and the monaddr loop emits a v2 and a v1 entry for each of them. A monitor on a node with addresses in two subnets ends up with: [v2:ip1:3300, v1:ip1:6789, v2:ip2:3300, v1:ip2:6789] Ceph uses at most one entry per messenger type, and no client reads past the first match. Userspace has always skipped the rest, and the kernel client does the same since 5a87925539ac ("libceph: tolerate addrvecs with multiple entries of the same type"), which went to stable. Older kernels reject the whole monmap with "another match of type N in addrvec", so CephFS failed to mount while RBD kept working. Pick a single IP from the listed networks instead, the first match in the order given by public_network, and warn that clients in the other subnets need routing to reach it. Only the auto-detection path changes. An explicit 'mon-address' is honored as before, and the comma-split parser added in 0b6a2838 stays, so #2422 remains fixed. Existing monmaps keep their addresses and can be corrected with: ceph mon set-addrs '[v2::3300/0,v1::6789/0]' Signed-off-by: Kefu Chai --- PVE/API2/Ceph/MON.pm | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm index 68abd502..bc08e33c 100644 --- a/PVE/API2/Ceph/MON.pm +++ b/PVE/API2/Ceph/MON.pm @@ -58,7 +58,14 @@ my $find_mon_ips = sub { my $res = []; - if (!scalar(@{$overwrite_ips})) { # auto-select one address for each public network + if (!scalar(@{$overwrite_ips})) { + # auto-select a single address even when public_network lists + # several subnets: the kernel libceph decoder rejects monmap + # entries with more than one address per messenger type (#7518), + # and the userspace messenger has only ever used the first match. + # other subnets remain reachable via routing. + my $candidates = []; + my $candidate_origin = {}; for my $net (@{$public_nets}) { my $allowed_ips = PVE::Network::get_local_ip_from_cidr($net); $allowed_ips = PVE::Network::unique_ips($allowed_ips); @@ -67,13 +74,24 @@ my $find_mon_ips = sub { "No active IP found for the requested ceph public network '$net' on node '$node'\n" if scalar(@$allowed_ips) < 1; - if (scalar(@$allowed_ips) == 1) { - push @{$res}, $allowed_ips->[0]; - } else { - die "Multiple IPs for ceph public network '$net' detected on $node:\n" - . join("\n", @$allowed_ips) - . "\nuse 'mon-address' to specify one of them.\n"; - } + die "Multiple IPs for ceph public network '$net' detected on $node:\n" + . join("\n", @$allowed_ips) + . "\nuse 'mon-address' to specify one of them.\n" + if scalar(@$allowed_ips) > 1; + + push @{$candidates}, $allowed_ips->[0]; + $candidate_origin->{ $allowed_ips->[0] } = $net; + } + + push @{$res}, $candidates->[0]; + + if (scalar(@{$candidates}) > 1) { + my $picked = $candidates->[0]; + my $picked_net = $candidate_origin->{$picked}; + warn "Multiple candidate IPs for monitor on $node across networks ($pubnet);" + . " picked $picked from '$picked_net'." + . " Other subnets must be reachable via routing for clients on them." + . " Use 'mon-address' to override.\n"; } } else { # check if overwrite IPs are active and in any of the public networks my $allowed_list = []; -- 2.47.3