[PATCH] sdn: evpn: allow vlan-aware vnets

[PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann]

EVPN vnets rejected the vlanaware flag outright, so a guest could not be
attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with "vm
vlans are not allowed on vnet <vnet>" because the vnet bridge has VLAN
filtering disabled.

Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-4094)
just like the VXLAN zone does, so the guest VLAN tags are carried
transparently over the vnet's VNI: a guest on an untagged port sends
802.1q frames that get VXLAN encapsulated as-is, and a guest on a port
with a VLAN tag receives them with the tag popped.

Such a vnet is a VLAN bundle service, and FRR does not implement the
VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with an
Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not carried in
the route. FRR does advertise those MACs -- it does not restrict itself
to the VNI's access VLAN -- but a receiver has no way to tell which VLAN
they belong to.

Between FRR peers that is harmless. The receiver installs the route into
the VNI's access VLAN where it is inert, and the entry that actually
forwards the traffic comes from data plane learning:

  bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1
  bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1

Against a third-party VTEP it is not harmless, and it breaks differently
depending on how the peer maps the Ethernet Tag. Measured in a four-VTEP
fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the
route-map deny below removed:

  - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports the
    tag-0 route as a control-plane MAC in the VNI's bridge domain,

      C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11)

    which bypasses the hairpin that restores the inner VLAN tag.
    Broadcast survives -- an ARP request is flooded through the hairpin
    and arrives correctly tagged -- but the unicast that follows is
    dropped and never reaches the guest. 100% loss to every FRR-hosted
    guest, in both directions.

  - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN
    port's bridge-pvid instead of the VLAN carrying the traffic, as an
    EXTERNAL bridge host entry, and then blackholes that MAC outright --
    including for traffic in the VLAN it really belongs to. Withdrawing
    the route does not clear the entry, the VXLAN interface has to be
    bounced.

  - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag to VID
    mapping (vni 1 / vid 5 / tag 0), so the route lands in the correct
    VLAN and forwards normally.

So keep MAC learning enabled and ARP/ND suppression disabled on the VXLAN
interface of such a vnet, and additionally deny the type-2 routes for its
VNI in the outgoing VTEP route-map, so no peer is fed MAC routes that
carry no usable VLAN. BUM traffic is unaffected, it is still head-end
replicated via the type-3 routes installed by the controller, and the
tagged VLANs are covered by data plane learning on both sides.

Subnets stay mutually exclusive with vlanaware, which is already
enforced by the vnet and subnet plugins, so this does not affect the
L3/gateway setup of an EVPN zone.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
---
 .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33 ++++++++++++++++++++++
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 21 ++++++++++----
 2 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
--- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
@@ -586,6 +586,39 @@
 sub generate_vnet_frr_config {
     my ($class, $plugin_config, $controller, $zone, $zoneid, $vnetid, $config) = @_;
 
+    # A vlan-aware vnet carries the guest VLAN tags inside a single VNI, but FRR
+    # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN
+    # the MAC was learned on is not carried in the route. A receiver installs such
+    # a route into the VNI's access VLAN, which is the wrong VLAN for every tagged
+    # MAC. Between FRR peers the entry is inert (the tagged VLANs are forwarded via
+    # data plane learning), but third-party VTEPs break on it in different ways:
+    # Cisco NX-OS imports it as a control-plane MAC in the VNI's bridge domain,
+    # which bypasses the QinQ hairpin that restores the inner tag -- flooded
+    # traffic still works, the unicast that follows it is dropped. RouterOS files
+    # it under the VXLAN port's bridge-pvid and blackholes that MAC outright.
+    # BIRD is unaffected, it maps the Ethernet Tag to a VLAN explicitly.
+    # Suppress the type-2 routes for these vnets, BUM traffic is unaffected as it
+    # is still head-end replicated via the type-3 routes.
+    if ($plugin_config->{vlanaware}) {
+        my $route_map_out = 'MAP_VTEP_OUT';
+        $route_map_out .= "_$controller->{'peer-group-name'}"
+            if $controller->{'peer-group-name'};
+
+        # seq is renumbered by array order in PVE::Network::SDN::Frr, so unshift
+        # to be evaluated before the permit entry that terminates the route-map
+        unshift(
+            @{ $config->{frr}->{routemaps}->{$route_map_out} },
+            {
+                seq => 1,
+                action => 'deny',
+                matches => [
+                    { key => 'evpn route-type', value => 'macip' },
+                    { key => 'evpn vni', value => $plugin_config->{tag} },
+                ],
+            },
+        );
+    }
+
     my $exitnodes = $zone->{'exitnodes'};
     my $exitnodes_local_routing = $zone->{'exitnodes-local-routing'};
 
diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
index 0e79707..e597120 100644
--- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
@@ -218,15 +218,24 @@ sub generate_sdn_config {
     }
     $mtu = $plugin_config->{mtu} if $plugin_config->{mtu};
 
+    # a vlan-aware vnet transports the guest VLAN tags transparently over a single
+    # VNI. The EVPN control plane only knows about the VNI's access VLAN, so it can
+    # neither advertise nor install MAC entries for the other VLANs. Fall back to
+    # data plane learning (and no ARP/ND suppression) for those vnets, BUM traffic
+    # is still replicated via the type-3 routes installed by the controller.
+    my $vlanaware = $vnet->{vlanaware};
+
     #vxlan interface
     my $vxlan_iface = "vxlan_$vnetid";
     my @iface_config = ();
     push @iface_config, "vxlan-id $tag";
     push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;
     push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
-    push @iface_config, "bridge-learning off";
-    push @iface_config, "bridge-arp-nd-suppress on"
-        if !$plugin_config->{'disable-arp-nd-suppression'};
+    if (!$vlanaware) {
+        push @iface_config, "bridge-learning off";
+        push @iface_config, "bridge-arp-nd-suppress on"
+            if !$plugin_config->{'disable-arp-nd-suppression'};
+    }
 
     push @iface_config, "mtu $mtu" if $mtu;
     push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config->{$vxlan_iface};
@@ -299,6 +308,10 @@ sub generate_sdn_config {
     push @iface_config, "bridge_ports $vxlan_iface";
     push @iface_config, "bridge_stp off";
     push @iface_config, "bridge_fd 0";
+    if ($vlanaware) {
+        push @iface_config, "bridge-vlan-aware yes";
+        push @iface_config, "bridge-vids 2-4094";
+    }
     push @iface_config, "mtu $mtu" if $mtu;
     push @iface_config, "alias $alias" if $alias;
     push @iface_config, "ip-forward on" if $enable_forward_v4;
@@ -423,8 +436,6 @@ sub vnet_update_hook {
 
     raise_param_exc({ tag => "missing vxlan tag" }) if !defined($tag);
     raise_param_exc({ tag => "vxlan tag max value is 16777216" }) if $tag > 16777216;
-    raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be enabled with evpn" })
-        if $vnet->{vlanaware};
 
     # verify that tag is not already defined globally (vxlan-id are unique)
     foreach my $id (keys %{ $vnet_cfg->{ids} }) {
-- 
2.47.3

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Stefan Hanreich]

Thanks for your contribution!

We were discussing internally if this approach is the right one. Currently,
with the EVPN zone it is possible to create an L3 EVI and via the VXLAN
zone it is possible to create a standalone L2 EVI, but this is currently a
bit of a hack.

We're strongly considering improving upon this workaround and making
it the officially supported and documented way of creating standalone L2
EVIs, as well as integrating this nicer into our stack to make it more
discoverable. Does utilizing a VLAN-aware VXLAN zone with an EVPN controller
and then attaching a route map to the EVPN controller work for your use case
as well? We know of quite a few people that are using this setup and are
considering it when extending the SDN stack. That's also part of why we want
to go down that route.

Main problems I currently see are that it is not possible to create a VXLAN
zone without any peers. Learning VTEP IPs via type-3 routes works perfectly
fine though - it is just necessary to enter any peer IP address (which is
awkward to say the least). Also, the handling of route distribution happens
implicitly instead of explicitly, so adding the option of assigning EVPN
controllers to VXLAN zones explicitly would be preferred over the status
quo - but we'd have to find a way that makes this backwards compatible.

Do you think this is a tenable solution for your use-case? Do you see any
other issues, particularly with your specific setup?


On 9/4/26 11:27 AM, Thomas Glanzmann wrote:
> EVPN vnets rejected the vlanaware flag outright, so a guest could not be
> attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with "vm
> vlans are not allowed on vnet <vnet>" because the vnet bridge has VLAN
> filtering disabled.
> 
> Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-4094)
> just like the VXLAN zone does, so the guest VLAN tags are carried
> transparently over the vnet's VNI: a guest on an untagged port sends
> 802.1q frames that get VXLAN encapsulated as-is, and a guest on a port
> with a VLAN tag receives them with the tag popped.> Such a vnet is a VLAN bundle service, and FRR does not implement the
> VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with an
> Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not carried in
> the route. FRR does advertise those MACs -- it does not restrict itself
> to the VNI's access VLAN -- but a receiver has no way to tell which VLAN
> they belong to.
> 
> Between FRR peers that is harmless. The receiver installs the route into
> the VNI's access VLAN where it is inert, and the entry that actually
> forwards the traffic comes from data plane learning:
> 
>   bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1
>   bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1
> 
> Against a third-party VTEP it is not harmless, and it breaks differently
> depending on how the peer maps the Ethernet Tag. Measured in a four-VTEP
> fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the
> route-map deny below removed:
> 
>   - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports the
>     tag-0 route as a control-plane MAC in the VNI's bridge domain,
> 
>       C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11)
> 
>     which bypasses the hairpin that restores the inner VLAN tag.
>     Broadcast survives -- an ARP request is flooded through the hairpin
>     and arrives correctly tagged -- but the unicast that follows is
>     dropped and never reaches the guest. 100% loss to every FRR-hosted
>     guest, in both directions.
> 
>   - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN
>     port's bridge-pvid instead of the VLAN carrying the traffic, as an
>     EXTERNAL bridge host entry, and then blackholes that MAC outright --
>     including for traffic in the VLAN it really belongs to. Withdrawing
>     the route does not clear the entry, the VXLAN interface has to be
>     bounced.
> 
>   - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag to VID
>     mapping (vni 1 / vid 5 / tag 0), so the route lands in the correct
>     VLAN and forwards normally.
> 
> So keep MAC learning enabled and ARP/ND suppression disabled on the VXLAN
> interface of such a vnet, and additionally deny the type-2 routes for its
> VNI in the outgoing VTEP route-map, so no peer is fed MAC routes that
> carry no usable VLAN. BUM traffic is unaffected, it is still head-end
> replicated via the type-3 routes installed by the controller, and the
> tagged VLANs are covered by data plane learning on both sides.
> Subnets stay mutually exclusive with vlanaware, which is already
> enforced by the vnet and subnet plugins, so this does not affect the
> L3/gateway setup of an EVPN zone.
> 
> Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
> ---
>  .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33 ++++++++++++++++++++++
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 21 ++++++++++----
>  2 files changed, 49 insertions(+), 5 deletions(-)
> 
> diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> --- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> +++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> @@ -586,6 +586,39 @@
>  sub generate_vnet_frr_config {
>      my ($class, $plugin_config, $controller, $zone, $zoneid, $vnetid, $config) = @_;
>  
> +    # A vlan-aware vnet carries the guest VLAN tags inside a single VNI, but FRR
> +    # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN
> +    # the MAC was learned on is not carried in the route. A receiver installs such
> +    # a route into the VNI's access VLAN, which is the wrong VLAN for every tagged
> +    # MAC. Between FRR peers the entry is inert (the tagged VLANs are forwarded via
> +    # data plane learning), but third-party VTEPs break on it in different ways:
> +    # Cisco NX-OS imports it as a control-plane MAC in the VNI's bridge domain,
> +    # which bypasses the QinQ hairpin that restores the inner tag -- flooded
> +    # traffic still works, the unicast that follows it is dropped. RouterOS files
> +    # it under the VXLAN port's bridge-pvid and blackholes that MAC outright.
> +    # BIRD is unaffected, it maps the Ethernet Tag to a VLAN explicitly.
> +    # Suppress the type-2 routes for these vnets, BUM traffic is unaffected as it
> +    # is still head-end replicated via the type-3 routes.
> +    if ($plugin_config->{vlanaware}) {
> +        my $route_map_out = 'MAP_VTEP_OUT';
> +        $route_map_out .= "_$controller->{'peer-group-name'}"
> +            if $controller->{'peer-group-name'};
> +
> +        # seq is renumbered by array order in PVE::Network::SDN::Frr, so unshift
> +        # to be evaluated before the permit entry that terminates the route-map
> +        unshift(
> +            @{ $config->{frr}->{routemaps}->{$route_map_out} },
> +            {
> +                seq => 1,
> +                action => 'deny',
> +                matches => [
> +                    { key => 'evpn route-type', value => 'macip' },
> +                    { key => 'evpn vni', value => $plugin_config->{tag} },
> +                ],
> +            },
> +        );
> +    }
> +
>      my $exitnodes = $zone->{'exitnodes'};
>      my $exitnodes_local_routing = $zone->{'exitnodes-local-routing'};
>  
> diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> index 0e79707..e597120 100644
> --- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> +++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> @@ -218,15 +218,24 @@ sub generate_sdn_config {
>      }
>      $mtu = $plugin_config->{mtu} if $plugin_config->{mtu};
>  
> +    # a vlan-aware vnet transports the guest VLAN tags transparently over a single
> +    # VNI. The EVPN control plane only knows about the VNI's access VLAN, so it can
> +    # neither advertise nor install MAC entries for the other VLANs. Fall back to
> +    # data plane learning (and no ARP/ND suppression) for those vnets, BUM traffic
> +    # is still replicated via the type-3 routes installed by the controller.
> +    my $vlanaware = $vnet->{vlanaware};
> +
>      #vxlan interface
>      my $vxlan_iface = "vxlan_$vnetid";
>      my @iface_config = ();
>      push @iface_config, "vxlan-id $tag";
>      push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;
>      push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
> -    push @iface_config, "bridge-learning off";
> -    push @iface_config, "bridge-arp-nd-suppress on"
> -        if !$plugin_config->{'disable-arp-nd-suppression'};
> +    if (!$vlanaware) {
> +        push @iface_config, "bridge-learning off";
> +        push @iface_config, "bridge-arp-nd-suppress on"
> +            if !$plugin_config->{'disable-arp-nd-suppression'};
> +    }
>  
>      push @iface_config, "mtu $mtu" if $mtu;
>      push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config->{$vxlan_iface};
> @@ -299,6 +308,10 @@ sub generate_sdn_config {
>      push @iface_config, "bridge_ports $vxlan_iface";
>      push @iface_config, "bridge_stp off";
>      push @iface_config, "bridge_fd 0";
> +    if ($vlanaware) {
> +        push @iface_config, "bridge-vlan-aware yes";
> +        push @iface_config, "bridge-vids 2-4094";
> +    }
>      push @iface_config, "mtu $mtu" if $mtu;
>      push @iface_config, "alias $alias" if $alias;
>      push @iface_config, "ip-forward on" if $enable_forward_v4;
> @@ -423,8 +436,6 @@ sub vnet_update_hook {
>  
>      raise_param_exc({ tag => "missing vxlan tag" }) if !defined($tag);
>      raise_param_exc({ tag => "vxlan tag max value is 16777216" }) if $tag > 16777216;
> -    raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be enabled with evpn" })
> -        if $vnet->{vlanaware};
>  
>      # verify that tag is not already defined globally (vxlan-id are unique)
>      foreach my $id (keys %{ $vnet_cfg->{ids} }) {

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann]

Hello Stefan,

> Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then
> attaching a route map to the EVPN controller work for your use case as
> well?

Yes. I rebuilt my setup that way and it does everything the patch did, so
consider the patch withdrawn in favour of your approach - with one
packaging gap that has to be closed first, see below.

I tested it against three foreign VTEP implementations. Versions:

  Proxmox     PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches),
              libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3,
              ifupdown2 3.3.0-1+pmx12       VTEP 10.101.0.18
  MikroTik    CRS304-4XG, RouterOS 7.25beta3            VTEP 172.31.0.4
  Cisco       N9K-C9372TX, NX-OS 9.3(16)                VTEP 172.31.0.7
  BIRD        3.3.2 (Debian bird3)                      VTEP 172.31.0.118

One stretched L2 segment 10.0.0.0/24 over VNI 1, customer VLAN 5 carried
tagged inside the VXLAN and popped at each edge port, iBGP AS 65000, RT
65000:1, full mesh. Tenant endpoints: 10.0.0.1 is a guest on the Proxmox
host (net0 bridge=e1,tag=5), 10.0.0.3 hangs off the MikroTik, 10.0.0.4 off
the N9K, 10.0.0.5 is the BIRD box itself.

The whole Proxmox side, no patched Perl anywhere:

  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 10 --action deny \
      --match key=route-type,value=macip --match key=vni,value=1
  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 20 --action permit
  pvesh create /cluster/sdn/controllers --controller evpnctl --type evpn \
      --asn 65000 --route-map-out nomacip \
      --peers 10.101.0.18,172.31.0.4,172.31.0.7,172.31.0.118
  pvesh create /cluster/sdn/zones --zone vx1 --type vxlan \
      --peers 10.101.0.18
  pvesh create /cluster/sdn/vnets --vnet e1 --zone vx1 --tag 1 --vlanaware 1
  pvesh set /cluster/sdn

Result: 12/12 directed pairs at 0% loss, and the MTU is exact to the byte
(ping -M do -s 1338 passes end to end, -s 1340 fails - the underlay here
crosses a WireGuard tunnel with MTU 1420 and the customer tag costs 4 more
bytes inside the VXLAN). The tag really is transported, tcpdump on the
underlay port of the Proxmox host:

  10.101.0.18.42551 > 172.31.0.4.4789: VXLAN, flags [I] (0x08), vni 1
  bc:24:11:ec:33:f2 > a0:36:9f:20:37:d8, ethertype 802.1Q (0x8100),
      length 102: vlan 5, p 0, ethertype IPv4,
      10.0.0.1 > 10.0.0.3: ICMP echo request

Point by point against what the patch did:

  - bridge-vlan-aware yes / bridge-vids 2-4094 on the vnet bridge: the
    VXLAN zone plugin already does this for a vlanaware vnet.
  - keep MAC learning on and ARP/ND suppression off on the VXLAN
    interface: the VXLAN zone never emitted bridge-learning off or
    bridge-arp-nd-suppress on in the first place, so there is nothing to
    special-case.
  - lifting the "vlan-aware option can't be enabled with evpn" check: not
    needed, that raise_param_exc only lives in
    Zones/EvpnPlugin.pm::vnet_update_hook, and the VXLAN zone's
    vnet_update_hook has no equivalent.
  - denying the type-2 routes: route-map-out on the EVPN controller. It
    renders as a call inside the generated map, which is exactly what is
    wanted:

      route-map MAP_VTEP_OUT permit 1
       call nomacip
      exit
      route-map nomacip deny 10
       match evpn route-type macip
       match evpn vni 1
      exit
      route-map nomacip permit 20
      exit

So the answer to your question is yes, and I prefer your shape to mine: it
gets the L2 EVI without a VRF, and the filtering is explicit and visible in
the configuration instead of hardcoded in a plugin.

I re-measured that the route-map is actually load-bearing, because a test
that only ever passes proves nothing. Removing just the route-map from the
controller and re-applying:

  pvesh set /cluster/sdn/controllers/evpnctl --delete route-map-out
  pvesh set /cluster/sdn

  advertised to 172.31.0.4 WITHOUT the route-map:
     [2]:[0]:[48]:[bc:24:11:ec:33:f2]      <- Ethernet Tag 0
     [3]:[0]:[32]:[10.101.0.18]
  advertised WITH it:
     [3]:[0]:[32]:[10.101.0.18]

Per implementation, with the route-map removed:

  Cisco NX-OS - breaks, both directions, 100% loss to that guest while the
  other 10 of 12 pairs stay up. The tag-0 route is imported as a
  control-plane MAC in the VNI's bridge domain and beats the data plane
  entry that the QinQ hairpin learned in VLAN 5:

     C 1000  bc24.11ec.33f2  dynamic  nve1(10.101.0.18)
     *    5  bc24.11ec.33f2  dynamic  Eth1/3

  Putting route-map-out back heals all 12 pairs with no other action.

  MikroTik RouterOS - no longer fatal on 7.25beta3, which is a change from
  what I reported for 7.24.1. It still files the route under the VXLAN
  port's bridge-pvid as an EXTERNAL host entry:

     D E mac-address=BC:24:11:EC:33:F2 vid=1 interface=vxlan1
         remote-ip=10.101.0.18

  but the MAC is no longer blackholed - a lookup miss in VLAN 5 floods to
  vxlan1 and the frames arrive, 0% loss throughout. The stale-entry half of
  the defect is still there though: that entry survived the withdrawal of
  the route when I put the route-map back, and needed an
  /interface/vxlan/disable + enable to clear.

  BIRD - unaffected, as expected. Its EVPN protocol has an explicit
  Ethernet Tag to VID mapping (vni 1 / vid 5 / tag 0), so the route lands
  in the right VLAN. Worth noting for completeness that BIRD's own tag-0
  type-2 does show up on the N9K in the same wrong place
  (C 1000 aac4.3347.8840 nve1(172.31.0.118)) and is harmless there only
  because that MAC is the bridge's own and never sources tenant traffic -
  it is not evidence that NX-OS tolerates tag-0 type-2.

> Do you see any other issues, particularly with your specific setup?

Three, one of them a blocker.

1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one
   thing that stops your proposal from working out of the box, and it fails
   silently. generate_sdn_config computes $ifaceip via
   find_local_ip_interface_peers() (or from the fabric node) and then never
   uses it for the interface:

     my @iface_config = ();
     push @iface_config, "vxlan-id $tag";
     for my $address (sort @peers) {
         next if $address eq $ifaceip;
         push @iface_config, "vxlan_remoteip $address";
     }
     push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
     push @iface_config, "mtu $mtu" if $mtu;

   Zones/EvpnPlugin.pm has the line the VXLAN zone is missing:

     push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;

   ifupdown2 only logs "vxlan_e1: missing vxlan-local-tunnelip" at warning
   level, and FRR then reports:

     VNI: 1
      Type: L2
      Vlan: 0
      VxLAN interface: vxlan_e1
      Local VTEP IP: (null)
      No remote VTEPs known for this VNI

   No type-3 is originated, no VTEP is learned, nothing forwards, and all
   BGP sessions look perfectly healthy while this is the case. This does
   not matter for a plain VXLAN zone, because static vxlan_remoteip
   head-end replication works fine without a local tunnel IP - it bites
   only the moment an EVPN controller is supposed to drive that zone,
   which is precisely the setup you want to make official. I worked around
   it with

     # /etc/network/interfaces.d/zz-vxlan-local-tunnelip
     iface vxlan_e1
             vxlan-local-tunnelip 10.101.0.18

   which ifupdown2 merges with the generated stanza and which survives
   pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1,
   Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with
   flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm
   already has would fix it; I am happy to send that as a patch if you
   want it.

2. The peers requirement you already mentioned is real - on_update_hook
   demands exactly one of peers / fabric. There is a tidier form than
   entering an arbitrary peer though: list ONLY the local IP. The remoteip
   loop skips the local address, so no static vxlan_remoteip is generated
   at all and every VTEP is learned purely from type-3, which also avoids
   mixing static head-end replication with BGP-learned flood entries. That
   reduces peers to a pure "which of my addresses is the VTEP IP"
   declaration - which, if you are redesigning this anyway, is probably
   the thing the zone actually wants to say, and it could equally come
   from the controller or the fabric.

3. The controller route-map is per peer-group, not per VNI, so it applies
   to every zone on that controller. That is why my nomacip entry carries
   an explicit match evpn vni 1 - without it, an L3 EVI sharing the same
   controller would lose its type-2 routes too. It works, but it means the
   user has to know to scope the deny by VNI by hand, and it is the one
   place where the explicit route-map is less safe than the hardcoded
   per-vnet deny in my patch. Some documentation, or a per-vnet hook that
   adds the scoped entry for vlan-aware vnets, would help here.

One smaller note: with a single EVPN controller
skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare
permit 1 plus the call; with several controllers the generated
extcommunity match is added alongside the call, and I have not tested that
combination.

So: tenable, yes, and I would rather have your version than mine. If the
vxlan-local-tunnelip line lands, the setup works with no patched Perl at
all, which I have now verified against NX-OS, RouterOS and BIRD at the
same time.

I also ordered a Juniper QFX5100-48T-AFO and an Arista DCS-7050TX-64-R
in order to conduct more tests.

Cheers,
        Thomas