[PATCH pve-manager v1] fix #5327: api: ceph: support multiple cluster networks on OSD creation

[PATCH pve-manager v1] fix #5327: api: ceph: support multiple cluster networks on OSD creation

[Max R. Carrara]

Both the `cluster_network` and `public_network` fields in `ceph.conf`
may contain a list of comma-separated CIDRs [0].

When a user has more than one CIDR specified in `cluster_network` --
or `public_network`, which is used as a fallback if `cluster_network`
is empty -- our check here passes the entire string to our networking
helpers without splitting it first.

To fix this, split the `$osd_network` string that we obtain in a
rather lenient manner and try to resolve the node's local IP for each
CIDR.

Additionally, `warn` if resolving a local IP fails, but do not error
out as long as at least one IP can be resolved. OSD creation can
therefore still succeed if part of the network is misconfigured. This
can be useful in situations where the user has to e.g. work on or fix
things on their Ceph cluster if their network happens to struggle at
the same time, or they're dealing with other configuration issues.

[0] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#network-config-settings

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=5327
Reported-by: Alexander Zeidler <a.zeidler@proxmox.com>
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/API2/Ceph/OSD.pm | 43 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 36 insertions(+), 7 deletions(-)

diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm
index f6581564..c267adc8 100644
--- a/PVE/API2/Ceph/OSD.pm
+++ b/PVE/API2/Ceph/OSD.pm
@@ -344,14 +344,43 @@ __PACKAGE__->register_method({
         $osd_network //= $ceph_conf->{global}->{public_network}; # fallback

         if ($osd_network) { # check only if something is configured
-            my $cluster_net_ips = PVE::Network::get_local_ip_from_cidr($osd_network);
-            if (scalar(@$cluster_net_ips) < 1) {
-                my $osd_net_obj = PVE::Network::IP_from_cidr($osd_network);
-                my $osd_base_cidr = $osd_net_obj->{ip} . "/" . $osd_net_obj->{prefixlen};
+            # Multiple subnets may be specified for both cluster_network and public_network:
+            # https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#confval-cluster_network
+            #
+            # Split them in a more lenient manner and filter out empty strings
+            # for additional resilience
+            my $cidrs = [grep { $_ } split(/,+\s*/, $osd_network)];

-                die
-                    "No address from ceph cluster network (${osd_base_cidr}) found on node '$nodename'. "
-                    . "Check your network config.\n";
+            my $cluster_net_ips = [];
+
+            for my $cidr ($cidrs->@*) {
+                next if !$cidr;
+
+                my $ips = eval { PVE::Network::get_local_ip_from_cidr($cidr) };
+
+                if (my $err = $@) {
+                    warn "Failed to resolve local IP from CIDR '$cidr' in Ceph configuration"
+                        . " - $err\n";
+                    next;
+                }
+
+                push($cluster_net_ips->@*, $ips->@*);
+            }
+
+            if (scalar($cluster_net_ips->@*) < 1) {
+                my $osd_base_cidrs = [];
+                for my $cidr ($cidrs->@*) {
+                    my $osd_net_obj = PVE::Network::IP_from_cidr($cidr);
+                    push(
+                        $osd_base_cidrs->@*,
+                        $osd_net_obj->{ip} . "/" . $osd_net_obj->{prefixlen},
+                    );
+                }
+
+                my $base_cidrs_formatted = join(', ', $osd_base_cidrs->@*);
+
+                die "No address from Ceph cluster network (${base_cidrs_formatted}) found"
+                    . " on node '$nodename'. Check your network config.\n";
             }
         }

--
2.47.3