[PATCH pve-network 08/16] evpn zone: add secondary-controllers and rt filtering

[Stefan Hanreich <s.hanreich@proxmox.com>]

With the ability to define multiple EVPN controllers, enable the
possibility to assign multiple EVPN controllers to a single zone. This
can be used to announce the L2VPN EVPN routes of a single zone to
multiple peer groups. By having to specify the controllers responsible
for announcing the routes explicitly, one can selectively announce the
routes for a given zone to specific peer groups. Because the EVPN
controller is used to auto-derive VTEP IPs (and the MTU of the
VNets), based on the peer definition in the EVPN controller, it is not
sufficient to extend the existing field and convert it to a list.
Instead, the old controller field now defines the 'primary' controller
for the zone, which is utilized to generate the VTEP IPs and MTU. All
other secondary controllers only announce the L2VPN EVPN routes -
without having any effect on VTEP IP auto-derivation and MTU.

Since the generated FRR configuration uses `advertise-all-vni`, which
advertise all routes from all zones by default (instead of importing
VRFs / route-targets explicitly), it is required to filter outgoing
EVPN routes in order to preserve backwards-compatibility. Otherwise an
EVPN controller configured on a node would announce the routes from
all zones, irregardless of the assigned zones. To prevent this,
outgoing routes for EVPN controllers need to be filtered, based on the
Route Targets of the zones and vnets. This is implemented via route
maps and extcommunity lists. An EVPN controller will only announce
routes with a route target that matches the route target of a zone /
vnet.

To prevent accidental breakage and preserve full
backwards-compatibility, the base case (one EVPN controller in the
cluster) is special cased. Since it was only possible to configure a
single EVPN controller, and EVPN zones need a controller assigned, it
is not required to filter outgoing routes when there is only one EVPN
controller configured. This, in turn, means that only if there are
multiple EVPN controllers configured, RT filtering for outgoing routes
takes place. This makes the new behavior opt-in and the generated FRR
configuration fully backwards-compatible. This is evidenced by the
fact that all existing test cases still work without any changes.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 src/PVE/API2/Network/SDN/Zones.pm             | 11 +++
 src/PVE/Network/SDN.pm                        |  1 +
 src/PVE/Network/SDN/Controllers.pm            | 71 +++++++++++++++++++
 src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 53 +++++++++++++-
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 39 ++++++++--
 5 files changed, 167 insertions(+), 8 deletions(-)

diff --git a/src/PVE/API2/Network/SDN/Zones.pm b/src/PVE/API2/Network/SDN/Zones.pm
index 8d829a9..cfdc195 100644
--- a/src/PVE/API2/Network/SDN/Zones.pm
+++ b/src/PVE/API2/Network/SDN/Zones.pm
@@ -202,6 +202,17 @@ my $ZONE_PROPERTIES = {
         description => "Disable auto mac learning. VLAN zone only.",
         optional => 1,
     },
+    'secondary-controllers' => {
+        type => 'array',
+        description => 'Additional controllers.',
+        items => {
+            type => 'string',
+            minLength => 2,
+            maxLength => 64,
+            pattern => '[a-zA-Z][a-zA-Z0-9_-]*[a-zA-Z0-9]',
+        },
+        optional => 1,
+    },
 };
 
 __PACKAGE__->register_method({
diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
index 2b3dead..58a21f2 100644
--- a/src/PVE/Network/SDN.pm
+++ b/src/PVE/Network/SDN.pm
@@ -508,6 +508,7 @@ sub encode_value {
         || $key eq 'entries'
         || $key eq 'match'
         || $key eq 'set'
+        || $key eq 'secondary-controllers'
     ) {
         if (ref($value) eq 'HASH') {
             return join(',', sort keys(%$value));
diff --git a/src/PVE/Network/SDN/Controllers.pm b/src/PVE/Network/SDN/Controllers.pm
index 3c18552..6167fa2 100644
--- a/src/PVE/Network/SDN/Controllers.pm
+++ b/src/PVE/Network/SDN/Controllers.pm
@@ -103,23 +103,41 @@ sub generate_frr_config {
         }
     }
 
+    my $allowed_communities = {};
+
     foreach my $id (sort keys %{ $controller_cfg->{ids} }) {
         my $plugin_config = $controller_cfg->{ids}->{$id};
         my $plugin = PVE::Network::SDN::Controllers::Plugin->lookup($plugin_config->{type});
         $plugin->generate_frr_config($plugin_config, $controller_cfg, $id, $uplinks, $frr_config);
+
+        if ($plugin_config->{type} eq 'evpn') {
+            $allowed_communities->{$id} = [];
+        }
     }
 
     foreach my $id (sort keys %{ $zone_cfg->{ids} }) {
         my $plugin_config = $zone_cfg->{ids}->{$id};
+
         my $controllerid = $plugin_config->{controller};
         next if !$controllerid;
+
         my $controller = $controller_cfg->{ids}->{$controllerid};
+        my $route_target = "$controller->{asn}:$plugin_config->{'vrf-vxlan'}";
+
         if ($controller) {
             my $controller_plugin =
                 PVE::Network::SDN::Controllers::Plugin->lookup($controller->{type});
             $controller_plugin->generate_zone_frr_config(
                 $plugin_config, $controller, $controller_cfg, $id, $uplinks, $frr_config,
             );
+
+            push $allowed_communities->{$controllerid}->@*, $route_target;
+        }
+
+        if ($plugin_config->{'secondary-controllers'}) {
+            for my $id ($plugin_config->{'secondary-controllers'}->@*) {
+                push $allowed_communities->{$id}->@*, $route_target;
+            }
         }
     }
 
@@ -133,12 +151,65 @@ sub generate_frr_config {
         next if !$controllerid;
         my $controller = $controller_cfg->{ids}->{$controllerid};
 
+        my $route_target = "$controller->{asn}:$plugin_config->{'tag'}";
+
         if ($controller) {
             my $controller_plugin =
                 PVE::Network::SDN::Controllers::Plugin->lookup($controller->{type});
             $controller_plugin->generate_vnet_frr_config(
                 $plugin_config, $controller, $zone, $zoneid, $id, $frr_config,
             );
+
+            push $allowed_communities->{$controllerid}->@*, $route_target;
+        }
+
+        if ($zone->{'secondary-controllers'}) {
+            for my $id ($zone->{'secondary-controllers'}->@*) {
+                push $allowed_communities->{$id}->@*, $route_target;
+            }
+        }
+    }
+
+    if (!PVE::Network::SDN::Controllers::EvpnPlugin::skip_route_target_filtering($controller_cfg)) {
+        $frr_config->{frr}->{bgp}->{ext_community_lists} = {};
+
+        for my $controller_id (sort keys $allowed_communities->%*) {
+            my $route_targets = $allowed_communities->{$controller_id};
+            my $community_list_name = "pve_controller_$controller_id";
+
+            if (scalar($route_targets->@*)) {
+                my @entries = map { {
+                    action => 'permit',
+                    match_entry => {
+                        type => 'rt',
+                        value => $_,
+                    },
+                } } $route_targets->@*;
+
+                $frr_config->{frr}->{bgp}->{ext_community_lists}->{$community_list_name} = {
+                    type => 'standard',
+                    entries => \@entries,
+                };
+            } else {
+                # Since it's impossible to create empty community lists create a
+                # community list with one deny entry instead. This works,
+                # because the default verdict is to deny any extcommunity that
+                # doesn't match an entry in the community list. So this
+                # extcommunity-list effectively blocks *every* route.
+
+                $frr_config->{frr}->{bgp}->{ext_community_lists}->{$community_list_name} = {
+                    type => 'standard',
+                    entries => [
+                        {
+                            action => 'deny',
+                            match_entry => {
+                                type => 'rt',
+                                value => "0:0",
+                            },
+                        },
+                    ],
+                };
+            }
         }
     }
 }
diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
index db844ff..e9bee33 100644
--- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
@@ -68,6 +68,32 @@ sub options {
     };
 }
 
+=head3 skip_route_target_filtering(\%controller_config)
+
+When multiple EVPN controllers are configured for multiple zones, outgoing
+routes need to get filtered, since otherwise EVPN controllers would simply
+announce all routes for all zones located on the node. To preserve
+backwards-compatibility and unnecessary filtering when only one EVPN controller
+is defined, this function can be used to check whether route target filtering
+should be performed or not.
+
+=cut
+
+sub skip_route_target_filtering {
+    my ($controller_config) = @_;
+
+    my $evpn_controller = undef;
+
+    for my $controller (values $controller_config->{ids}->%*) {
+        next if $controller->{type} ne 'evpn';
+
+        return 0 if $evpn_controller;
+        $evpn_controller = $controller;
+    }
+
+    return 1;
+}
+
 # Plugin implementation
 sub generate_frr_config {
     my ($class, $plugin_config, $controller_cfg, $id, $uplinks, $config) = @_;
@@ -228,9 +254,26 @@ sub generate_frr_config {
     }
 
     if (!$config->{frr}->{routemaps}->{$route_map_out}) {
-        my $entry = { seq => 1, action => "permit" };
-        $entry->{call} = $plugin_config->{'route-map-out'} if $plugin_config->{'route-map-out'};
+        my $entry = {
+            seq => 1,
+            action => "permit",
+        };
+
+        # only filter outgoing routes if there are multiple EVPN controllers, to
+        # preserve backwards-compatibility
+        if (!skip_route_target_filtering($controller_cfg)) {
+            $entry->{matches} = [
+                {
+                    key => 'extcommunity',
+                    value => {
+                        name => "pve_controller_$id",
+                        mode => 'any',
+                    },
+                },
+            ];
+        }
 
+        $entry->{call} = $plugin_config->{'route-map-out'} if $plugin_config->{'route-map-out'};
         push($config->{frr}->{routemaps}->{$route_map_out}->@*, $entry);
     }
 
@@ -538,8 +581,14 @@ sub on_delete_hook {
     # verify that zone is associated to this controller
     foreach my $id (keys %{ $zone_cfg->{ids} }) {
         my $zone = $zone_cfg->{ids}->{$id};
+
         die "controller $controllerid is used by $id"
             if (defined($zone->{controller}) && $zone->{controller} eq $controllerid);
+
+        for my $secondary_controller ($zone->{'secondary-controllers'}->@*) {
+            die "controller $controllerid is used by $id"
+                if $secondary_controller eq $controllerid;
+        }
     }
 }
 
diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
index 8e7ddfd..5df7e5f 100644
--- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
@@ -53,6 +53,17 @@ sub properties {
             type => 'string',
             description => 'Controller for this zone.',
         },
+        'secondary-controllers' => {
+            type => 'array',
+            description => 'Additional controllers.',
+            items => {
+                type => 'string',
+                minLength => 2,
+                maxLength => 64,
+                pattern => '[a-zA-Z][a-zA-Z0-9_-]*[a-zA-Z0-9]',
+            },
+            optional => 1,
+        },
         'mac' => {
             type => 'string',
             description => "Anycast logical router mac address.",
@@ -97,6 +108,7 @@ sub options {
         nodes => { optional => 1 },
         'vrf-vxlan' => { optional => 0 },
         controller => { optional => 0 },
+        'secondary-controllers' => { optional => 1 },
         exitnodes => { optional => 1 },
         'exitnodes-local-routing' => { optional => 1 },
         'exitnodes-primary' => { optional => 1 },
@@ -348,13 +360,28 @@ sub generate_sdn_config {
 sub on_update_hook {
     my ($class, $zoneid, $zone_cfg, $controller_cfg) = @_;
 
+    my $controllers = {};
+
     # verify that controller exist
-    my $controller = $zone_cfg->{ids}->{$zoneid}->{controller};
-    if (!defined($controller_cfg->{ids}->{$controller})) {
-        die "controller $controller don't exist";
-    } else {
-        die "$controller is not a evpn controller type"
-            if $controller_cfg->{ids}->{$controller}->{type} ne 'evpn';
+    my $zone = $zone_cfg->{ids}->{$zoneid};
+    my $controller = $zone->{controller};
+
+    $controllers->{$controller} = undef;
+
+    for my $secondary_controller ($zone->{'secondary-controllers'}->@*) {
+        die "can not configure the same controller twice"
+            if exists($controllers->{$secondary_controller});
+
+        $controllers->{$secondary_controller} = undef;
+    }
+
+    for my $controller_id (keys $controllers->%*) {
+        if (!defined($controller_cfg->{ids}->{$controller_id})) {
+            die "controller $controller don't exist";
+        } else {
+            die "$controller_id is not a evpn controller type"
+                if $controller_cfg->{ids}->{$controller_id}->{type} ne 'evpn';
+        }
     }
 
     #vrf-vxlan need to be defined
-- 
2.47.3