Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann <thomas@glanzmann.de>]

Hello Stefan,

> Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then
> attaching a route map to the EVPN controller work for your use case as
> well?

Yes. I rebuilt my setup that way and it does everything the patch did, so
consider the patch withdrawn in favour of your approach - with one
packaging gap that has to be closed first, see below.

I tested it against three foreign VTEP implementations. Versions:

  Proxmox     PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches),
              libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3,
              ifupdown2 3.3.0-1+pmx12       VTEP 10.101.0.18
  MikroTik    CRS304-4XG, RouterOS 7.25beta3            VTEP 172.31.0.4
  Cisco       N9K-C9372TX, NX-OS 9.3(16)                VTEP 172.31.0.7
  BIRD        3.3.2 (Debian bird3)                      VTEP 172.31.0.118

One stretched L2 segment 10.0.0.0/24 over VNI 1, customer VLAN 5 carried
tagged inside the VXLAN and popped at each edge port, iBGP AS 65000, RT
65000:1, full mesh. Tenant endpoints: 10.0.0.1 is a guest on the Proxmox
host (net0 bridge=e1,tag=5), 10.0.0.3 hangs off the MikroTik, 10.0.0.4 off
the N9K, 10.0.0.5 is the BIRD box itself.

The whole Proxmox side, no patched Perl anywhere:

  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 10 --action deny \
      --match key=route-type,value=macip --match key=vni,value=1
  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 20 --action permit
  pvesh create /cluster/sdn/controllers --controller evpnctl --type evpn \
      --asn 65000 --route-map-out nomacip \
      --peers 10.101.0.18,172.31.0.4,172.31.0.7,172.31.0.118
  pvesh create /cluster/sdn/zones --zone vx1 --type vxlan \
      --peers 10.101.0.18
  pvesh create /cluster/sdn/vnets --vnet e1 --zone vx1 --tag 1 --vlanaware 1
  pvesh set /cluster/sdn

Result: 12/12 directed pairs at 0% loss, and the MTU is exact to the byte
(ping -M do -s 1338 passes end to end, -s 1340 fails - the underlay here
crosses a WireGuard tunnel with MTU 1420 and the customer tag costs 4 more
bytes inside the VXLAN). The tag really is transported, tcpdump on the
underlay port of the Proxmox host:

  10.101.0.18.42551 > 172.31.0.4.4789: VXLAN, flags [I] (0x08), vni 1
  bc:24:11:ec:33:f2 > a0:36:9f:20:37:d8, ethertype 802.1Q (0x8100),
      length 102: vlan 5, p 0, ethertype IPv4,
      10.0.0.1 > 10.0.0.3: ICMP echo request

Point by point against what the patch did:

  - bridge-vlan-aware yes / bridge-vids 2-4094 on the vnet bridge: the
    VXLAN zone plugin already does this for a vlanaware vnet.
  - keep MAC learning on and ARP/ND suppression off on the VXLAN
    interface: the VXLAN zone never emitted bridge-learning off or
    bridge-arp-nd-suppress on in the first place, so there is nothing to
    special-case.
  - lifting the "vlan-aware option can't be enabled with evpn" check: not
    needed, that raise_param_exc only lives in
    Zones/EvpnPlugin.pm::vnet_update_hook, and the VXLAN zone's
    vnet_update_hook has no equivalent.
  - denying the type-2 routes: route-map-out on the EVPN controller. It
    renders as a call inside the generated map, which is exactly what is
    wanted:

      route-map MAP_VTEP_OUT permit 1
       call nomacip
      exit
      route-map nomacip deny 10
       match evpn route-type macip
       match evpn vni 1
      exit
      route-map nomacip permit 20
      exit

So the answer to your question is yes, and I prefer your shape to mine: it
gets the L2 EVI without a VRF, and the filtering is explicit and visible in
the configuration instead of hardcoded in a plugin.

I re-measured that the route-map is actually load-bearing, because a test
that only ever passes proves nothing. Removing just the route-map from the
controller and re-applying:

  pvesh set /cluster/sdn/controllers/evpnctl --delete route-map-out
  pvesh set /cluster/sdn

  advertised to 172.31.0.4 WITHOUT the route-map:
     [2]:[0]:[48]:[bc:24:11:ec:33:f2]      <- Ethernet Tag 0
     [3]:[0]:[32]:[10.101.0.18]
  advertised WITH it:
     [3]:[0]:[32]:[10.101.0.18]

Per implementation, with the route-map removed:

  Cisco NX-OS - breaks, both directions, 100% loss to that guest while the
  other 10 of 12 pairs stay up. The tag-0 route is imported as a
  control-plane MAC in the VNI's bridge domain and beats the data plane
  entry that the QinQ hairpin learned in VLAN 5:

     C 1000  bc24.11ec.33f2  dynamic  nve1(10.101.0.18)
     *    5  bc24.11ec.33f2  dynamic  Eth1/3

  Putting route-map-out back heals all 12 pairs with no other action.

  MikroTik RouterOS - no longer fatal on 7.25beta3, which is a change from
  what I reported for 7.24.1. It still files the route under the VXLAN
  port's bridge-pvid as an EXTERNAL host entry:

     D E mac-address=BC:24:11:EC:33:F2 vid=1 interface=vxlan1
         remote-ip=10.101.0.18

  but the MAC is no longer blackholed - a lookup miss in VLAN 5 floods to
  vxlan1 and the frames arrive, 0% loss throughout. The stale-entry half of
  the defect is still there though: that entry survived the withdrawal of
  the route when I put the route-map back, and needed an
  /interface/vxlan/disable + enable to clear.

  BIRD - unaffected, as expected. Its EVPN protocol has an explicit
  Ethernet Tag to VID mapping (vni 1 / vid 5 / tag 0), so the route lands
  in the right VLAN. Worth noting for completeness that BIRD's own tag-0
  type-2 does show up on the N9K in the same wrong place
  (C 1000 aac4.3347.8840 nve1(172.31.0.118)) and is harmless there only
  because that MAC is the bridge's own and never sources tenant traffic -
  it is not evidence that NX-OS tolerates tag-0 type-2.

> Do you see any other issues, particularly with your specific setup?

Three, one of them a blocker.

1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one
   thing that stops your proposal from working out of the box, and it fails
   silently. generate_sdn_config computes $ifaceip via
   find_local_ip_interface_peers() (or from the fabric node) and then never
   uses it for the interface:

     my @iface_config = ();
     push @iface_config, "vxlan-id $tag";
     for my $address (sort @peers) {
         next if $address eq $ifaceip;
         push @iface_config, "vxlan_remoteip $address";
     }
     push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
     push @iface_config, "mtu $mtu" if $mtu;

   Zones/EvpnPlugin.pm has the line the VXLAN zone is missing:

     push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;

   ifupdown2 only logs "vxlan_e1: missing vxlan-local-tunnelip" at warning
   level, and FRR then reports:

     VNI: 1
      Type: L2
      Vlan: 0
      VxLAN interface: vxlan_e1
      Local VTEP IP: (null)
      No remote VTEPs known for this VNI

   No type-3 is originated, no VTEP is learned, nothing forwards, and all
   BGP sessions look perfectly healthy while this is the case. This does
   not matter for a plain VXLAN zone, because static vxlan_remoteip
   head-end replication works fine without a local tunnel IP - it bites
   only the moment an EVPN controller is supposed to drive that zone,
   which is precisely the setup you want to make official. I worked around
   it with

     # /etc/network/interfaces.d/zz-vxlan-local-tunnelip
     iface vxlan_e1
             vxlan-local-tunnelip 10.101.0.18

   which ifupdown2 merges with the generated stanza and which survives
   pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1,
   Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with
   flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm
   already has would fix it; I am happy to send that as a patch if you
   want it.

2. The peers requirement you already mentioned is real - on_update_hook
   demands exactly one of peers / fabric. There is a tidier form than
   entering an arbitrary peer though: list ONLY the local IP. The remoteip
   loop skips the local address, so no static vxlan_remoteip is generated
   at all and every VTEP is learned purely from type-3, which also avoids
   mixing static head-end replication with BGP-learned flood entries. That
   reduces peers to a pure "which of my addresses is the VTEP IP"
   declaration - which, if you are redesigning this anyway, is probably
   the thing the zone actually wants to say, and it could equally come
   from the controller or the fabric.

3. The controller route-map is per peer-group, not per VNI, so it applies
   to every zone on that controller. That is why my nomacip entry carries
   an explicit match evpn vni 1 - without it, an L3 EVI sharing the same
   controller would lose its type-2 routes too. It works, but it means the
   user has to know to scope the deny by VNI by hand, and it is the one
   place where the explicit route-map is less safe than the hardcoded
   per-vnet deny in my patch. Some documentation, or a per-vnet hook that
   adds the scoped entry for vlan-aware vnets, would help here.

One smaller note: with a single EVPN controller
skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare
permit 1 plus the call; with several controllers the generated
extcommunity match is added alongside the call, and I have not tested that
combination.

So: tenable, yes, and I would rather have your version than mine. If the
vxlan-local-tunnelip line lands, the setup works with no patched Perl at
all, which I have now verified against NX-OS, RouterOS and BIRD at the
same time.

I also ordered a Juniper QFX5100-48T-AFO and an Arista DCS-7050TX-64-R
in order to conduct more tests.

Cheers,
        Thomas