[PATCH manager 1/1] fix #7518: ceph: mon: advertise a single address per monitor

[PATCH manager 1/1] fix #7518: ceph: mon: advertise a single address per monitor

[Kefu Chai]

When public_network lists multiple subnets (added in 0b6a2838 to
support clusters spanning more than one subnet) and the local node
has IPs in more than one of them, $find_mon_ips collected one address
per subnet and the monaddr loop emitted both a v2 and v1 entry for
every IP. This produced addrvecs of the shape:

    [v2:ip1:3300, v1:ip1:6789, v2:ip2:3300, v1:ip2:6789]

The kernel libceph decoder rejects these entries with "another match
of type N in addrvec", because it permits at most one entry of each
messenger type per monitor. Userspace has tolerated this since
Nautilus by picking the first matching entry, but no consumer of the
addrvec ever falls back to later entries, so the additional addresses
provide no functional benefit. The mismatch manifests as CephFS mount
failures on the kernel client while RBD continues to work.

Now pick a single IP from the listed networks (first match in the
order given by public_network), and warn that the other subnets must
be reachable via routing for clients on them. The change is limited
to the auto-detection path; explicit 'mon-address' values are honored
as before.

The comma-split parser for public_network introduced in 0b6a2838 is
preserved, so the original failure from #2422 stays fixed.

Existing clusters with multi-entry monmaps from earlier pve-manager
versions are unaffected by this change but can fix their monmap
manually:

    ceph mon set-addrs <id> '[v2:<ip>:3300/0,v1:<ip>:6789/0]'

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
 PVE/API2/Ceph/MON.pm | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm
index 18407b1c..01de9eac 100644
--- a/PVE/API2/Ceph/MON.pm
+++ b/PVE/API2/Ceph/MON.pm
@@ -58,7 +58,14 @@ my $find_mon_ips = sub {
 
     my $res = [];
 
-    if (!scalar(@{$overwrite_ips})) { # auto-select one address for each public network
+    if (!scalar(@{$overwrite_ips})) {
+        # auto-select a single address even when public_network lists
+        # several subnets: the kernel libceph decoder rejects monmap
+        # entries with more than one address per messenger type (#7518),
+        # and the userspace messenger has only ever used the first match.
+        # other subnets remain reachable via routing.
+        my $candidates = [];
+        my $candidate_origin = {};
         for my $net (@{$public_nets}) {
             my $allowed_ips = PVE::Network::get_local_ip_from_cidr($net);
             $allowed_ips = PVE::Network::unique_ips($allowed_ips);
@@ -67,13 +74,24 @@ my $find_mon_ips = sub {
                 "No active IP found for the requested ceph public network '$net' on node '$node'\n"
                 if scalar(@$allowed_ips) < 1;
 
-            if (scalar(@$allowed_ips) == 1) {
-                push @{$res}, $allowed_ips->[0];
-            } else {
-                die "Multiple IPs for ceph public network '$net' detected on $node:\n"
-                    . join("\n", @$allowed_ips)
-                    . "\nuse 'mon-address' to specify one of them.\n";
-            }
+            die "Multiple IPs for ceph public network '$net' detected on $node:\n"
+                . join("\n", @$allowed_ips)
+                . "\nuse 'mon-address' to specify one of them.\n"
+                if scalar(@$allowed_ips) > 1;
+
+            push @{$candidates}, $allowed_ips->[0];
+            $candidate_origin->{ $allowed_ips->[0] } = $net;
+        }
+
+        push @{$res}, $candidates->[0];
+
+        if (scalar(@{$candidates}) > 1) {
+            my $picked = $candidates->[0];
+            my $picked_net = $candidate_origin->{$picked};
+            warn "Multiple candidate IPs for monitor on $node across networks ($pubnet);"
+                . " picked $picked from '$picked_net'."
+                . " Other subnets must be reachable via routing for clients on them."
+                . " Use 'mon-address' to override.\n";
         }
     } else { # check if overwrite IPs are active and in any of the public networks
         my $allowed_list = [];
-- 
2.47.3