[PATCH v2 manager 0/1] fix #7518: ceph: mon: advertise a single address per monitor

[PATCH v2 manager 0/1] fix #7518: ceph: mon: advertise a single address per monitor

[Kefu Chai]

pveceph mon create can put two addresses of the same messenger type
into a monitor's addrvec. Older kernels refuse to decode that, so
CephFS fails to mount. This makes it pick one address per monitor.

The kernel side is fixed in the meantime, so v2 only rewords the
commit message. The code is unchanged.

Changes since v1:
 - say that current kernels tolerate such addrvecs since 5a87925539ac
   ("libceph: tolerate addrvecs with multiple entries of the same
   type"), which went to stable, and that the reasons to still pick a
   single address are older kernels and that no client reads past the
   first match
 - no code change

v1: https://lore.proxmox.com/pve-devel/20260425121528.924363-1-k.chai@proxmox.com/

Kefu Chai (1):
  fix #7518: ceph: mon: advertise a single address per monitor

 PVE/API2/Ceph/MON.pm | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

-- 
2.47.3

[PATCH v2 manager 1/1] fix #7518: ceph: mon: advertise a single address per monitor

[Kefu Chai]

$find_mon_ips picks one local IP per subnet listed in public_network,
and the monaddr loop emits a v2 and a v1 entry for each of them. A
monitor on a node with addresses in two subnets ends up with:

    [v2:ip1:3300, v1:ip1:6789, v2:ip2:3300, v1:ip2:6789]

Ceph uses at most one entry per messenger type, and no client reads
past the first match. Userspace has always skipped the rest, and the
kernel client does the same since 5a87925539ac ("libceph: tolerate
addrvecs with multiple entries of the same type"), which went to
stable. Older kernels reject the whole monmap with "another match of
type N in addrvec", so CephFS failed to mount while RBD kept working.

Pick a single IP from the listed networks instead, the first match in
the order given by public_network, and warn that clients in the other
subnets need routing to reach it. Only the auto-detection path
changes. An explicit 'mon-address' is honored as before, and the
comma-split parser added in 0b6a2838 stays, so #2422 remains fixed.

Existing monmaps keep their addresses and can be corrected with:

    ceph mon set-addrs <id> '[v2:<ip>:3300/0,v1:<ip>:6789/0]'

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
 PVE/API2/Ceph/MON.pm | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm
index 68abd502..bc08e33c 100644
--- a/PVE/API2/Ceph/MON.pm
+++ b/PVE/API2/Ceph/MON.pm
@@ -58,7 +58,14 @@ my $find_mon_ips = sub {
 
     my $res = [];
 
-    if (!scalar(@{$overwrite_ips})) { # auto-select one address for each public network
+    if (!scalar(@{$overwrite_ips})) {
+        # auto-select a single address even when public_network lists
+        # several subnets: the kernel libceph decoder rejects monmap
+        # entries with more than one address per messenger type (#7518),
+        # and the userspace messenger has only ever used the first match.
+        # other subnets remain reachable via routing.
+        my $candidates = [];
+        my $candidate_origin = {};
         for my $net (@{$public_nets}) {
             my $allowed_ips = PVE::Network::get_local_ip_from_cidr($net);
             $allowed_ips = PVE::Network::unique_ips($allowed_ips);
@@ -67,13 +74,24 @@ my $find_mon_ips = sub {
                 "No active IP found for the requested ceph public network '$net' on node '$node'\n"
                 if scalar(@$allowed_ips) < 1;
 
-            if (scalar(@$allowed_ips) == 1) {
-                push @{$res}, $allowed_ips->[0];
-            } else {
-                die "Multiple IPs for ceph public network '$net' detected on $node:\n"
-                    . join("\n", @$allowed_ips)
-                    . "\nuse 'mon-address' to specify one of them.\n";
-            }
+            die "Multiple IPs for ceph public network '$net' detected on $node:\n"
+                . join("\n", @$allowed_ips)
+                . "\nuse 'mon-address' to specify one of them.\n"
+                if scalar(@$allowed_ips) > 1;
+
+            push @{$candidates}, $allowed_ips->[0];
+            $candidate_origin->{ $allowed_ips->[0] } = $net;
+        }
+
+        push @{$res}, $candidates->[0];
+
+        if (scalar(@{$candidates}) > 1) {
+            my $picked = $candidates->[0];
+            my $picked_net = $candidate_origin->{$picked};
+            warn "Multiple candidate IPs for monitor on $node across networks ($pubnet);"
+                . " picked $picked from '$picked_net'."
+                . " Other subnets must be reachable via routing for clients on them."
+                . " Use 'mon-address' to override.\n";
         }
     } else { # check if overwrite IPs are active and in any of the public networks
         my $allowed_list = [];
-- 
2.47.3