Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann <thomas@glanzmann.de>]

Hello Gabriel,

> Yes, this should be fixed with https://github.com/FRRouting/frr/pull/22555.
> Note that `show evpn vni x` shows the zebra vtep, and zebra can't find a
> vtep as there is no vxlan-local-tunnelip set on the interface. bgpd will
> fallback to the router-ip, so `show bgp l2vpn evpn route` should show routes
> with the correct derived vtep ip.

You are right and I was wrong. I re-measured it today and my "no type-3 is
originated, no VTEP is learned, nothing forwards" was a misattribution -
please disregard that part of my last mail. Details below, since I think the
exact failure mode is worth having in the archive.

Setup unchanged from my last mail: PVE 9.2.11 with stock libpve-network-perl
1.6.7, frr 10.6.1-1+pve3, ifupdown2 3.3.0-1+pmx12 (VTEP 10.101.0.18), against
MikroTik CRS304-4XG on RouterOS 7.25beta3 (172.31.0.4), Cisco N9K-C9372TX on
NX-OS 9.3(16) (172.31.0.7) and BIRD 3.3.2 (172.31.0.118). The fix is indeed in
the package I tested with:

  frr (10.6.1-1+pve3) trixie; urgency=medium
    * fix VTEP derivation for VXLAN interfaces without a local IP address

  a5ad7bd "frr: backport VXLAN local VTEP fallback fix"
    debian/patches/upstream/0028-bgpd-zebra-fix-VXLAN-interface-local-IP-fallback.patch
    debian/patches/upstream/0029-tests-add-EVPN-L2VNI-unspecified-VTEP-topotest.patch

1. THE FALLBACK WORKS - my workaround file is gone for good

I removed /etc/network/interfaces.d/zz-vxlan-local-tunnelip, so there is now
no vxlan-local-tunnelip anywhere on the node, and the kernel device really has
no local address:

  # ip -d link show vxlan_e1
  21: vxlan_e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 master e1
      vxlan id 1 srcport 0 0 dstport 4789 ttl auto ageing 300

  # vtysh -c 'show evpn vni 1'
   VNI: 1 / Type: L2 / Vlan: 1 / Bridge: e1
   Local VTEP IP: (null)                <- exactly as you describe
   Remote VTEPs for this VNI:
    172.31.0.7   flood: HER
    172.31.0.4   flood: HER
    172.31.0.118 flood: HER

  # vtysh -c 'show bgp l2vpn evpn neighbors 172.31.0.7 advertised-routes'
   *> [3]:[0]:[32]:[10.101.0.18]        <- derived from bgp router-id
   Total number of prefixes 1

and on the wire the outer header carries the same address, with the customer
tag inside the VNI where it belongs (this one is the N9K leg, i.e. the QinVNI
peer that is fussiest about all of this):

  10.101.0.18.42551 > 172.31.0.7.4789: VXLAN, flags [I] (0x08), vni 1
  bc:24:11:ec:33:f2 > 58:11:22:b6:19:fa, ethertype 802.1Q (0x8100),
      vlan 5, p 0, ethertype IPv4, 10.0.0.1 > 10.0.0.4: ICMP echo request

All 12 directed pairs over {10.0.0.1 PVE VM, .3 MikroTik, .4 N9K, .5 BIRD} at
0% loss, `ping -M do -s 1338` passes and `-s 1340` fails end to end, so the
1366-byte tenant MTU is still exact to the byte. The N9K learns the guest MAC
where it should, on the data-plane port and in the customer VLAN, not as a
control-plane MAC in the VNI bridge domain:

  # show mac address-table
  *    5     bc24.11ec.33f2   dynamic  0  F  F  Eth1/3

and it survives a full `pvesh set /cluster/sdn` (regenerated interfaces + frr,
still 0% loss). So the proposed VLAN-aware-VXLAN-zone + EVPN-controller +
route-map setup does work out of the box on stock packages, with no patched
Perl and no local tunnel IP, against all three foreign implementations. The
"packaging gap" in my last mail does not exist; there is nothing to fix in
VxlanPlugin.pm.

One packaging-adjacent trap while re-testing, in case someone else chases it:
after deleting a vxlan-local-tunnelip stanza, the FIRST `ifreload -a` still
programs the old address, because the module-global vxlan._vxlan_local_tunnelip
is also populated from the saved old ifaceobjs in get_dependent_ifacenames().
A second `ifreload -a` clears it. I mistook that for "the workaround is load
bearing" more than once.

2. WHAT I ACTUALLY SAW - `Vlan: 0`, and it is not about the tunnel IP

What I reported as "nothing forwards" came with `VNI: 1 / Vlan: 0 / Local VTEP
IP: (null)`. The `Vlan: 0` is real and reproducible, but it is independent of
vxlan-local-tunnelip. Control:

  systemctl restart frr, WITHOUT vxlan-local-tunnelip -> Vlan: 0
  systemctl restart frr, WITH    vxlan-local-tunnelip -> Vlan: 0

i.e. after an frr restart zebra does not re-resolve the bridge VLAN for the
L2VNI. A subsequent `ifreload -a` does not heal it either; bouncing the vxlan
interface (`ifdown vxlan_e1 && ifup vxlan_e1`) does, and then it is `Vlan: 1`
again. In the `Vlan: 0` state the remote VTEPs stay learned and forwarding
stays at 0% loss in my topology, so it is not fatal here - but zebra does
advertise the local MAC as an Ethernet-Tag-0 type-2 in that state, which is
precisely the route the route-map on the controller has to filter, so on a
QinVNI peer such as the N9K it would matter (with the route-map in place it is
filtered on the way out, which is why it stayed invisible). Tell me if you want
that as a separate report, I can produce a clean reproducer.

3. THE ifupdown2 WARNING - patch attached

> The ifupdown2 warning is only printed on the syntax-check, which gets
> executed when using e.g. `ifreload -s`. We don't do that anywhere AFAICS.
> Maybe we should patch this warning out as it's technically false.

Confirmed on both counts. `_module_syntax_check()` in ifupdownmain.py is only
reached under the `syntaxcheck` flag, and PVE only ever calls `ifreload -a`
(API2/Network.pm) and `ifquery -a -c -o json` (SDN.pm, SDN/Zones.pm), neither
of which takes that path.

One thing to add though: it is not only a cosmetic warning. vxlan.py's
syntax_check() *returns False* on the missing attribute, so `ifreload -s -a`
exits non-zero on a perfectly good configuration - which is what an admin
validating a config before applying it would see.

Patch attached as git format-patch against the ifupdown2 packaging repo
(new debian/patches/pve/0016-*.patch plus series entry). It drops the whole
block rather than just the log line, for the return-value reason above; the
clagd anycast check below it is unaffected and copes with local == None.

Tested by applying it to /usr/share/ifupdown2/addons/vxlan.py on the node:

  before: warning: vxlan_e1: missing vxlan-local-tunnelip ; rc=1
  after:  (gone)

4. A SECOND FALSE POSITIVE IN THE SAME SYNTAX CHECK

With the patch applied, `ifreload -s -a` still exits 1 on this configuration,
because of bridge.py::_error_vxlan_in_vlan_aware_br():

  warning: vxlan_e1: `bridge-access` attribute is mandatory when vxlan device
           (vxlan_e1) is part of vlan aware bridge (e1)

That is a log_error(), so it also sets ifaceStatus.ERROR on the interface. It
is equally false for the setup we are discussing: a VLAN bundle service maps
a whole VLAN range into one VNI, so the vxlan port must be a trunk in the
vlan-aware bridge and must NOT have bridge-access. It is exactly the config
that the VXLAN zone plugin generates for a vlanaware vnet, so any EVPN-driven
VLAN-aware zone hits it. I left it out of the attached patch since you only
asked about the tunnel-ip one - say the word and I will send a second patch
(it needs `and not <vlan-aware trunk>` rather than a plain removal, since the
check is legitimate for the VNI-per-VLAN case).

Cheers,
        Thomas