[PATCH] sdn: evpn: allow vlan-aware vnets

[PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann]

EVPN vnets rejected the vlanaware flag outright, so a guest could not be
attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with "vm
vlans are not allowed on vnet <vnet>" because the vnet bridge has VLAN
filtering disabled.

Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-4094)
just like the VXLAN zone does, so the guest VLAN tags are carried
transparently over the vnet's VNI: a guest on an untagged port sends
802.1q frames that get VXLAN encapsulated as-is, and a guest on a port
with a VLAN tag receives them with the tag popped.

Such a vnet is a VLAN bundle service, and FRR does not implement the
VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with an
Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not carried in
the route. FRR does advertise those MACs -- it does not restrict itself
to the VNI's access VLAN -- but a receiver has no way to tell which VLAN
they belong to.

Between FRR peers that is harmless. The receiver installs the route into
the VNI's access VLAN where it is inert, and the entry that actually
forwards the traffic comes from data plane learning:

  bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1
  bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1

Against a third-party VTEP it is not harmless, and it breaks differently
depending on how the peer maps the Ethernet Tag. Measured in a four-VTEP
fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the
route-map deny below removed:

  - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports the
    tag-0 route as a control-plane MAC in the VNI's bridge domain,

      C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11)

    which bypasses the hairpin that restores the inner VLAN tag.
    Broadcast survives -- an ARP request is flooded through the hairpin
    and arrives correctly tagged -- but the unicast that follows is
    dropped and never reaches the guest. 100% loss to every FRR-hosted
    guest, in both directions.

  - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN
    port's bridge-pvid instead of the VLAN carrying the traffic, as an
    EXTERNAL bridge host entry, and then blackholes that MAC outright --
    including for traffic in the VLAN it really belongs to. Withdrawing
    the route does not clear the entry, the VXLAN interface has to be
    bounced.

  - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag to VID
    mapping (vni 1 / vid 5 / tag 0), so the route lands in the correct
    VLAN and forwards normally.

So keep MAC learning enabled and ARP/ND suppression disabled on the VXLAN
interface of such a vnet, and additionally deny the type-2 routes for its
VNI in the outgoing VTEP route-map, so no peer is fed MAC routes that
carry no usable VLAN. BUM traffic is unaffected, it is still head-end
replicated via the type-3 routes installed by the controller, and the
tagged VLANs are covered by data plane learning on both sides.

Subnets stay mutually exclusive with vlanaware, which is already
enforced by the vnet and subnet plugins, so this does not affect the
L3/gateway setup of an EVPN zone.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
---
 .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33 ++++++++++++++++++++++
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 21 ++++++++++----
 2 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
--- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
@@ -586,6 +586,39 @@
 sub generate_vnet_frr_config {
     my ($class, $plugin_config, $controller, $zone, $zoneid, $vnetid, $config) = @_;
 
+    # A vlan-aware vnet carries the guest VLAN tags inside a single VNI, but FRR
+    # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN
+    # the MAC was learned on is not carried in the route. A receiver installs such
+    # a route into the VNI's access VLAN, which is the wrong VLAN for every tagged
+    # MAC. Between FRR peers the entry is inert (the tagged VLANs are forwarded via
+    # data plane learning), but third-party VTEPs break on it in different ways:
+    # Cisco NX-OS imports it as a control-plane MAC in the VNI's bridge domain,
+    # which bypasses the QinQ hairpin that restores the inner tag -- flooded
+    # traffic still works, the unicast that follows it is dropped. RouterOS files
+    # it under the VXLAN port's bridge-pvid and blackholes that MAC outright.
+    # BIRD is unaffected, it maps the Ethernet Tag to a VLAN explicitly.
+    # Suppress the type-2 routes for these vnets, BUM traffic is unaffected as it
+    # is still head-end replicated via the type-3 routes.
+    if ($plugin_config->{vlanaware}) {
+        my $route_map_out = 'MAP_VTEP_OUT';
+        $route_map_out .= "_$controller->{'peer-group-name'}"
+            if $controller->{'peer-group-name'};
+
+        # seq is renumbered by array order in PVE::Network::SDN::Frr, so unshift
+        # to be evaluated before the permit entry that terminates the route-map
+        unshift(
+            @{ $config->{frr}->{routemaps}->{$route_map_out} },
+            {
+                seq => 1,
+                action => 'deny',
+                matches => [
+                    { key => 'evpn route-type', value => 'macip' },
+                    { key => 'evpn vni', value => $plugin_config->{tag} },
+                ],
+            },
+        );
+    }
+
     my $exitnodes = $zone->{'exitnodes'};
     my $exitnodes_local_routing = $zone->{'exitnodes-local-routing'};
 
diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
index 0e79707..e597120 100644
--- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
+++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
@@ -218,15 +218,24 @@ sub generate_sdn_config {
     }
     $mtu = $plugin_config->{mtu} if $plugin_config->{mtu};
 
+    # a vlan-aware vnet transports the guest VLAN tags transparently over a single
+    # VNI. The EVPN control plane only knows about the VNI's access VLAN, so it can
+    # neither advertise nor install MAC entries for the other VLANs. Fall back to
+    # data plane learning (and no ARP/ND suppression) for those vnets, BUM traffic
+    # is still replicated via the type-3 routes installed by the controller.
+    my $vlanaware = $vnet->{vlanaware};
+
     #vxlan interface
     my $vxlan_iface = "vxlan_$vnetid";
     my @iface_config = ();
     push @iface_config, "vxlan-id $tag";
     push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;
     push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
-    push @iface_config, "bridge-learning off";
-    push @iface_config, "bridge-arp-nd-suppress on"
-        if !$plugin_config->{'disable-arp-nd-suppression'};
+    if (!$vlanaware) {
+        push @iface_config, "bridge-learning off";
+        push @iface_config, "bridge-arp-nd-suppress on"
+            if !$plugin_config->{'disable-arp-nd-suppression'};
+    }
 
     push @iface_config, "mtu $mtu" if $mtu;
     push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config->{$vxlan_iface};
@@ -299,6 +308,10 @@ sub generate_sdn_config {
     push @iface_config, "bridge_ports $vxlan_iface";
     push @iface_config, "bridge_stp off";
     push @iface_config, "bridge_fd 0";
+    if ($vlanaware) {
+        push @iface_config, "bridge-vlan-aware yes";
+        push @iface_config, "bridge-vids 2-4094";
+    }
     push @iface_config, "mtu $mtu" if $mtu;
     push @iface_config, "alias $alias" if $alias;
     push @iface_config, "ip-forward on" if $enable_forward_v4;
@@ -423,8 +436,6 @@ sub vnet_update_hook {
 
     raise_param_exc({ tag => "missing vxlan tag" }) if !defined($tag);
     raise_param_exc({ tag => "vxlan tag max value is 16777216" }) if $tag > 16777216;
-    raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be enabled with evpn" })
-        if $vnet->{vlanaware};
 
     # verify that tag is not already defined globally (vxlan-id are unique)
     foreach my $id (keys %{ $vnet_cfg->{ids} }) {
-- 
2.47.3

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Stefan Hanreich]

Thanks for your contribution!

We were discussing internally if this approach is the right one. Currently,
with the EVPN zone it is possible to create an L3 EVI and via the VXLAN
zone it is possible to create a standalone L2 EVI, but this is currently a
bit of a hack.

We're strongly considering improving upon this workaround and making
it the officially supported and documented way of creating standalone L2
EVIs, as well as integrating this nicer into our stack to make it more
discoverable. Does utilizing a VLAN-aware VXLAN zone with an EVPN controller
and then attaching a route map to the EVPN controller work for your use case
as well? We know of quite a few people that are using this setup and are
considering it when extending the SDN stack. That's also part of why we want
to go down that route.

Main problems I currently see are that it is not possible to create a VXLAN
zone without any peers. Learning VTEP IPs via type-3 routes works perfectly
fine though - it is just necessary to enter any peer IP address (which is
awkward to say the least). Also, the handling of route distribution happens
implicitly instead of explicitly, so adding the option of assigning EVPN
controllers to VXLAN zones explicitly would be preferred over the status
quo - but we'd have to find a way that makes this backwards compatible.

Do you think this is a tenable solution for your use-case? Do you see any
other issues, particularly with your specific setup?


On 9/4/26 11:27 AM, Thomas Glanzmann wrote:
> EVPN vnets rejected the vlanaware flag outright, so a guest could not be
> attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with "vm
> vlans are not allowed on vnet <vnet>" because the vnet bridge has VLAN
> filtering disabled.
> 
> Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-4094)
> just like the VXLAN zone does, so the guest VLAN tags are carried
> transparently over the vnet's VNI: a guest on an untagged port sends
> 802.1q frames that get VXLAN encapsulated as-is, and a guest on a port
> with a VLAN tag receives them with the tag popped.> Such a vnet is a VLAN bundle service, and FRR does not implement the
> VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with an
> Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not carried in
> the route. FRR does advertise those MACs -- it does not restrict itself
> to the VNI's access VLAN -- but a receiver has no way to tell which VLAN
> they belong to.
> 
> Between FRR peers that is harmless. The receiver installs the route into
> the VNI's access VLAN where it is inert, and the entry that actually
> forwards the traffic comes from data plane learning:
> 
>   bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1
>   bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1
> 
> Against a third-party VTEP it is not harmless, and it breaks differently
> depending on how the peer maps the Ethernet Tag. Measured in a four-VTEP
> fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the
> route-map deny below removed:
> 
>   - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports the
>     tag-0 route as a control-plane MAC in the VNI's bridge domain,
> 
>       C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11)
> 
>     which bypasses the hairpin that restores the inner VLAN tag.
>     Broadcast survives -- an ARP request is flooded through the hairpin
>     and arrives correctly tagged -- but the unicast that follows is
>     dropped and never reaches the guest. 100% loss to every FRR-hosted
>     guest, in both directions.
> 
>   - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN
>     port's bridge-pvid instead of the VLAN carrying the traffic, as an
>     EXTERNAL bridge host entry, and then blackholes that MAC outright --
>     including for traffic in the VLAN it really belongs to. Withdrawing
>     the route does not clear the entry, the VXLAN interface has to be
>     bounced.
> 
>   - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag to VID
>     mapping (vni 1 / vid 5 / tag 0), so the route lands in the correct
>     VLAN and forwards normally.
> 
> So keep MAC learning enabled and ARP/ND suppression disabled on the VXLAN
> interface of such a vnet, and additionally deny the type-2 routes for its
> VNI in the outgoing VTEP route-map, so no peer is fed MAC routes that
> carry no usable VLAN. BUM traffic is unaffected, it is still head-end
> replicated via the type-3 routes installed by the controller, and the
> tagged VLANs are covered by data plane learning on both sides.
> Subnets stay mutually exclusive with vlanaware, which is already
> enforced by the vnet and subnet plugins, so this does not affect the
> L3/gateway setup of an EVPN zone.
> 
> Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
> ---
>  .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33 ++++++++++++++++++++++
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 21 ++++++++++----
>  2 files changed, 49 insertions(+), 5 deletions(-)
> 
> diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> --- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> +++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> @@ -586,6 +586,39 @@
>  sub generate_vnet_frr_config {
>      my ($class, $plugin_config, $controller, $zone, $zoneid, $vnetid, $config) = @_;
>  
> +    # A vlan-aware vnet carries the guest VLAN tags inside a single VNI, but FRR
> +    # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN
> +    # the MAC was learned on is not carried in the route. A receiver installs such
> +    # a route into the VNI's access VLAN, which is the wrong VLAN for every tagged
> +    # MAC. Between FRR peers the entry is inert (the tagged VLANs are forwarded via
> +    # data plane learning), but third-party VTEPs break on it in different ways:
> +    # Cisco NX-OS imports it as a control-plane MAC in the VNI's bridge domain,
> +    # which bypasses the QinQ hairpin that restores the inner tag -- flooded
> +    # traffic still works, the unicast that follows it is dropped. RouterOS files
> +    # it under the VXLAN port's bridge-pvid and blackholes that MAC outright.
> +    # BIRD is unaffected, it maps the Ethernet Tag to a VLAN explicitly.
> +    # Suppress the type-2 routes for these vnets, BUM traffic is unaffected as it
> +    # is still head-end replicated via the type-3 routes.
> +    if ($plugin_config->{vlanaware}) {
> +        my $route_map_out = 'MAP_VTEP_OUT';
> +        $route_map_out .= "_$controller->{'peer-group-name'}"
> +            if $controller->{'peer-group-name'};
> +
> +        # seq is renumbered by array order in PVE::Network::SDN::Frr, so unshift
> +        # to be evaluated before the permit entry that terminates the route-map
> +        unshift(
> +            @{ $config->{frr}->{routemaps}->{$route_map_out} },
> +            {
> +                seq => 1,
> +                action => 'deny',
> +                matches => [
> +                    { key => 'evpn route-type', value => 'macip' },
> +                    { key => 'evpn vni', value => $plugin_config->{tag} },
> +                ],
> +            },
> +        );
> +    }
> +
>      my $exitnodes = $zone->{'exitnodes'};
>      my $exitnodes_local_routing = $zone->{'exitnodes-local-routing'};
>  
> diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> index 0e79707..e597120 100644
> --- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> +++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> @@ -218,15 +218,24 @@ sub generate_sdn_config {
>      }
>      $mtu = $plugin_config->{mtu} if $plugin_config->{mtu};
>  
> +    # a vlan-aware vnet transports the guest VLAN tags transparently over a single
> +    # VNI. The EVPN control plane only knows about the VNI's access VLAN, so it can
> +    # neither advertise nor install MAC entries for the other VLANs. Fall back to
> +    # data plane learning (and no ARP/ND suppression) for those vnets, BUM traffic
> +    # is still replicated via the type-3 routes installed by the controller.
> +    my $vlanaware = $vnet->{vlanaware};
> +
>      #vxlan interface
>      my $vxlan_iface = "vxlan_$vnetid";
>      my @iface_config = ();
>      push @iface_config, "vxlan-id $tag";
>      push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;
>      push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
> -    push @iface_config, "bridge-learning off";
> -    push @iface_config, "bridge-arp-nd-suppress on"
> -        if !$plugin_config->{'disable-arp-nd-suppression'};
> +    if (!$vlanaware) {
> +        push @iface_config, "bridge-learning off";
> +        push @iface_config, "bridge-arp-nd-suppress on"
> +            if !$plugin_config->{'disable-arp-nd-suppression'};
> +    }
>  
>      push @iface_config, "mtu $mtu" if $mtu;
>      push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config->{$vxlan_iface};
> @@ -299,6 +308,10 @@ sub generate_sdn_config {
>      push @iface_config, "bridge_ports $vxlan_iface";
>      push @iface_config, "bridge_stp off";
>      push @iface_config, "bridge_fd 0";
> +    if ($vlanaware) {
> +        push @iface_config, "bridge-vlan-aware yes";
> +        push @iface_config, "bridge-vids 2-4094";
> +    }
>      push @iface_config, "mtu $mtu" if $mtu;
>      push @iface_config, "alias $alias" if $alias;
>      push @iface_config, "ip-forward on" if $enable_forward_v4;
> @@ -423,8 +436,6 @@ sub vnet_update_hook {
>  
>      raise_param_exc({ tag => "missing vxlan tag" }) if !defined($tag);
>      raise_param_exc({ tag => "vxlan tag max value is 16777216" }) if $tag > 16777216;
> -    raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be enabled with evpn" })
> -        if $vnet->{vlanaware};
>  
>      # verify that tag is not already defined globally (vxlan-id are unique)
>      foreach my $id (keys %{ $vnet_cfg->{ids} }) {

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann]

Hello Stefan,

> Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then
> attaching a route map to the EVPN controller work for your use case as
> well?

Yes. I rebuilt my setup that way and it does everything the patch did, so
consider the patch withdrawn in favour of your approach - with one
packaging gap that has to be closed first, see below.

I tested it against three foreign VTEP implementations. Versions:

  Proxmox     PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches),
              libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3,
              ifupdown2 3.3.0-1+pmx12       VTEP 10.101.0.18
  MikroTik    CRS304-4XG, RouterOS 7.25beta3            VTEP 172.31.0.4
  Cisco       N9K-C9372TX, NX-OS 9.3(16)                VTEP 172.31.0.7
  BIRD        3.3.2 (Debian bird3)                      VTEP 172.31.0.118

One stretched L2 segment 10.0.0.0/24 over VNI 1, customer VLAN 5 carried
tagged inside the VXLAN and popped at each edge port, iBGP AS 65000, RT
65000:1, full mesh. Tenant endpoints: 10.0.0.1 is a guest on the Proxmox
host (net0 bridge=e1,tag=5), 10.0.0.3 hangs off the MikroTik, 10.0.0.4 off
the N9K, 10.0.0.5 is the BIRD box itself.

The whole Proxmox side, no patched Perl anywhere:

  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 10 --action deny \
      --match key=route-type,value=macip --match key=vni,value=1
  pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \
      --order 20 --action permit
  pvesh create /cluster/sdn/controllers --controller evpnctl --type evpn \
      --asn 65000 --route-map-out nomacip \
      --peers 10.101.0.18,172.31.0.4,172.31.0.7,172.31.0.118
  pvesh create /cluster/sdn/zones --zone vx1 --type vxlan \
      --peers 10.101.0.18
  pvesh create /cluster/sdn/vnets --vnet e1 --zone vx1 --tag 1 --vlanaware 1
  pvesh set /cluster/sdn

Result: 12/12 directed pairs at 0% loss, and the MTU is exact to the byte
(ping -M do -s 1338 passes end to end, -s 1340 fails - the underlay here
crosses a WireGuard tunnel with MTU 1420 and the customer tag costs 4 more
bytes inside the VXLAN). The tag really is transported, tcpdump on the
underlay port of the Proxmox host:

  10.101.0.18.42551 > 172.31.0.4.4789: VXLAN, flags [I] (0x08), vni 1
  bc:24:11:ec:33:f2 > a0:36:9f:20:37:d8, ethertype 802.1Q (0x8100),
      length 102: vlan 5, p 0, ethertype IPv4,
      10.0.0.1 > 10.0.0.3: ICMP echo request

Point by point against what the patch did:

  - bridge-vlan-aware yes / bridge-vids 2-4094 on the vnet bridge: the
    VXLAN zone plugin already does this for a vlanaware vnet.
  - keep MAC learning on and ARP/ND suppression off on the VXLAN
    interface: the VXLAN zone never emitted bridge-learning off or
    bridge-arp-nd-suppress on in the first place, so there is nothing to
    special-case.
  - lifting the "vlan-aware option can't be enabled with evpn" check: not
    needed, that raise_param_exc only lives in
    Zones/EvpnPlugin.pm::vnet_update_hook, and the VXLAN zone's
    vnet_update_hook has no equivalent.
  - denying the type-2 routes: route-map-out on the EVPN controller. It
    renders as a call inside the generated map, which is exactly what is
    wanted:

      route-map MAP_VTEP_OUT permit 1
       call nomacip
      exit
      route-map nomacip deny 10
       match evpn route-type macip
       match evpn vni 1
      exit
      route-map nomacip permit 20
      exit

So the answer to your question is yes, and I prefer your shape to mine: it
gets the L2 EVI without a VRF, and the filtering is explicit and visible in
the configuration instead of hardcoded in a plugin.

I re-measured that the route-map is actually load-bearing, because a test
that only ever passes proves nothing. Removing just the route-map from the
controller and re-applying:

  pvesh set /cluster/sdn/controllers/evpnctl --delete route-map-out
  pvesh set /cluster/sdn

  advertised to 172.31.0.4 WITHOUT the route-map:
     [2]:[0]:[48]:[bc:24:11:ec:33:f2]      <- Ethernet Tag 0
     [3]:[0]:[32]:[10.101.0.18]
  advertised WITH it:
     [3]:[0]:[32]:[10.101.0.18]

Per implementation, with the route-map removed:

  Cisco NX-OS - breaks, both directions, 100% loss to that guest while the
  other 10 of 12 pairs stay up. The tag-0 route is imported as a
  control-plane MAC in the VNI's bridge domain and beats the data plane
  entry that the QinQ hairpin learned in VLAN 5:

     C 1000  bc24.11ec.33f2  dynamic  nve1(10.101.0.18)
     *    5  bc24.11ec.33f2  dynamic  Eth1/3

  Putting route-map-out back heals all 12 pairs with no other action.

  MikroTik RouterOS - no longer fatal on 7.25beta3, which is a change from
  what I reported for 7.24.1. It still files the route under the VXLAN
  port's bridge-pvid as an EXTERNAL host entry:

     D E mac-address=BC:24:11:EC:33:F2 vid=1 interface=vxlan1
         remote-ip=10.101.0.18

  but the MAC is no longer blackholed - a lookup miss in VLAN 5 floods to
  vxlan1 and the frames arrive, 0% loss throughout. The stale-entry half of
  the defect is still there though: that entry survived the withdrawal of
  the route when I put the route-map back, and needed an
  /interface/vxlan/disable + enable to clear.

  BIRD - unaffected, as expected. Its EVPN protocol has an explicit
  Ethernet Tag to VID mapping (vni 1 / vid 5 / tag 0), so the route lands
  in the right VLAN. Worth noting for completeness that BIRD's own tag-0
  type-2 does show up on the N9K in the same wrong place
  (C 1000 aac4.3347.8840 nve1(172.31.0.118)) and is harmless there only
  because that MAC is the bridge's own and never sources tenant traffic -
  it is not evidence that NX-OS tolerates tag-0 type-2.

> Do you see any other issues, particularly with your specific setup?

Three, one of them a blocker.

1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one
   thing that stops your proposal from working out of the box, and it fails
   silently. generate_sdn_config computes $ifaceip via
   find_local_ip_interface_peers() (or from the fabric node) and then never
   uses it for the interface:

     my @iface_config = ();
     push @iface_config, "vxlan-id $tag";
     for my $address (sort @peers) {
         next if $address eq $ifaceip;
         push @iface_config, "vxlan_remoteip $address";
     }
     push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
     push @iface_config, "mtu $mtu" if $mtu;

   Zones/EvpnPlugin.pm has the line the VXLAN zone is missing:

     push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip;

   ifupdown2 only logs "vxlan_e1: missing vxlan-local-tunnelip" at warning
   level, and FRR then reports:

     VNI: 1
      Type: L2
      Vlan: 0
      VxLAN interface: vxlan_e1
      Local VTEP IP: (null)
      No remote VTEPs known for this VNI

   No type-3 is originated, no VTEP is learned, nothing forwards, and all
   BGP sessions look perfectly healthy while this is the case. This does
   not matter for a plain VXLAN zone, because static vxlan_remoteip
   head-end replication works fine without a local tunnel IP - it bites
   only the moment an EVPN controller is supposed to drive that zone,
   which is precisely the setup you want to make official. I worked around
   it with

     # /etc/network/interfaces.d/zz-vxlan-local-tunnelip
     iface vxlan_e1
             vxlan-local-tunnelip 10.101.0.18

   which ifupdown2 merges with the generated stanza and which survives
   pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1,
   Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with
   flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm
   already has would fix it; I am happy to send that as a patch if you
   want it.

2. The peers requirement you already mentioned is real - on_update_hook
   demands exactly one of peers / fabric. There is a tidier form than
   entering an arbitrary peer though: list ONLY the local IP. The remoteip
   loop skips the local address, so no static vxlan_remoteip is generated
   at all and every VTEP is learned purely from type-3, which also avoids
   mixing static head-end replication with BGP-learned flood entries. That
   reduces peers to a pure "which of my addresses is the VTEP IP"
   declaration - which, if you are redesigning this anyway, is probably
   the thing the zone actually wants to say, and it could equally come
   from the controller or the fabric.

3. The controller route-map is per peer-group, not per VNI, so it applies
   to every zone on that controller. That is why my nomacip entry carries
   an explicit match evpn vni 1 - without it, an L3 EVI sharing the same
   controller would lose its type-2 routes too. It works, but it means the
   user has to know to scope the deny by VNI by hand, and it is the one
   place where the explicit route-map is less safe than the hardcoded
   per-vnet deny in my patch. Some documentation, or a per-vnet hook that
   adds the scoped entry for vlan-aware vnets, would help here.

One smaller note: with a single EVPN controller
skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare
permit 1 plus the call; with several controllers the generated
extcommunity match is added alongside the call, and I have not tested that
combination.

So: tenable, yes, and I would rather have your version than mine. If the
vxlan-local-tunnelip line lands, the setup works with no patched Perl at
all, which I have now verified against NX-OS, RouterOS and BIRD at the
same time.

I also ordered a Juniper QFX5100-48T-AFO and an Arista DCS-7050TX-64-R
in order to conduct more tests.

Cheers,
        Thomas

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[DERUMIER, Alexandre]

Hi,

they are also another way (that work with vxlan, but I'm not sure with
evpn),

https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-515/Network-Virtualization/VXLAN-Devices/
https://blog.vyos.io/evpn-vxlan-enhancements-introducing-single-vxlan-device-support
https://github.com/FRRouting/frr/pull/12364

it's mapping vlan tags from a vlan aware bridge to vnis, and transport
them in a single vxlan interface.


It was pretty new and buggy 5year ago, so I never tried to implemented
it,
maybe it could be interesting to look at it.


Le mardi 08 septembre 2026 à 11:45 +0200, Stefan Hanreich a écrit :
> Thanks for your contribution!
> 
> We were discussing internally if this approach is the right one.
> Currently,
> with the EVPN zone it is possible to create an L3 EVI and via the
> VXLAN
> zone it is possible to create a standalone L2 EVI, but this is
> currently a
> bit of a hack.
> 
> We're strongly considering improving upon this workaround and making
> it the officially supported and documented way of creating standalone
> L2
> EVIs, as well as integrating this nicer into our stack to make it
> more
> discoverable. Does utilizing a VLAN-aware VXLAN zone with an EVPN
> controller
> and then attaching a route map to the EVPN controller work for your
> use case
> as well? We know of quite a few people that are using this setup and
> are
> considering it when extending the SDN stack. That's also part of why
> we want
> to go down that route.
> 
> Main problems I currently see are that it is not possible to create a
> VXLAN
> zone without any peers. Learning VTEP IPs via type-3 routes works
> perfectly
> fine though - it is just necessary to enter any peer IP address
> (which is
> awkward to say the least). Also, the handling of route distribution
> happens
> implicitly instead of explicitly, so adding the option of assigning
> EVPN
> controllers to VXLAN zones explicitly would be preferred over the
> status
> quo - but we'd have to find a way that makes this backwards
> compatible.
> 
> Do you think this is a tenable solution for your use-case? Do you see
> any
> other issues, particularly with your specific setup?
> 
> 
> On 9/4/26 11:27 AM, Thomas Glanzmann wrote:
> > EVPN vnets rejected the vlanaware flag outright, so a guest could
> > not be
> > attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with
> > "vm
> > vlans are not allowed on vnet <vnet>" because the vnet bridge has
> > VLAN
> > filtering disabled.
> > 
> > Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-
> > 4094)
> > just like the VXLAN zone does, so the guest VLAN tags are carried
> > transparently over the vnet's VNI: a guest on an untagged port
> > sends
> > 802.1q frames that get VXLAN encapsulated as-is, and a guest on a
> > port
> > with a VLAN tag receives them with the tag popped.> Such a vnet is
> > a VLAN bundle service, and FRR does not implement the
> > VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with
> > an
> > Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not
> > carried in
> > the route. FRR does advertise those MACs -- it does not restrict
> > itself
> > to the VNI's access VLAN -- but a receiver has no way to tell which
> > VLAN
> > they belong to.
> > 
> > Between FRR peers that is harmless. The receiver installs the route
> > into
> > the VNI's access VLAN where it is inert, and the entry that
> > actually
> > forwards the traffic comes from data plane learning:
> > 
> >   bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1
> >   bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1
> > 
> > Against a third-party VTEP it is not harmless, and it breaks
> > differently
> > depending on how the peer maps the Ethernet Tag. Measured in a
> > four-VTEP
> > fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the
> > route-map deny below removed:
> > 
> >   - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports
> > the
> >     tag-0 route as a control-plane MAC in the VNI's bridge domain,
> > 
> >       C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11)
> > 
> >     which bypasses the hairpin that restores the inner VLAN tag.
> >     Broadcast survives -- an ARP request is flooded through the
> > hairpin
> >     and arrives correctly tagged -- but the unicast that follows is
> >     dropped and never reaches the guest. 100% loss to every FRR-
> > hosted
> >     guest, in both directions.
> > 
> >   - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN
> >     port's bridge-pvid instead of the VLAN carrying the traffic, as
> > an
> >     EXTERNAL bridge host entry, and then blackholes that MAC
> > outright --
> >     including for traffic in the VLAN it really belongs to.
> > Withdrawing
> >     the route does not clear the entry, the VXLAN interface has to
> > be
> >     bounced.
> > 
> >   - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag
> > to VID
> >     mapping (vni 1 / vid 5 / tag 0), so the route lands in the
> > correct
> >     VLAN and forwards normally.
> > 
> > So keep MAC learning enabled and ARP/ND suppression disabled on the
> > VXLAN
> > interface of such a vnet, and additionally deny the type-2 routes
> > for its
> > VNI in the outgoing VTEP route-map, so no peer is fed MAC routes
> > that
> > carry no usable VLAN. BUM traffic is unaffected, it is still head-
> > end
> > replicated via the type-3 routes installed by the controller, and
> > the
> > tagged VLANs are covered by data plane learning on both sides.
> > Subnets stay mutually exclusive with vlanaware, which is already
> > enforced by the vnet and subnet plugins, so this does not affect
> > the
> > L3/gateway setup of an EVPN zone.
> > 
> > Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
> > ---
> >  .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33
> > ++++++++++++++++++++++
> >  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 21 ++++++++++----
> >  2 files changed, 49 insertions(+), 5 deletions(-)
> > 
> > diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> > b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> > --- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> > +++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm
> > @@ -586,6 +586,39 @@
> >  sub generate_vnet_frr_config {
> >      my ($class, $plugin_config, $controller, $zone, $zoneid,
> > $vnetid, $config) = @_;
> >  
> > +    # A vlan-aware vnet carries the guest VLAN tags inside a
> > single VNI, but FRR
> > +    # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of
> > 0, so the VLAN
> > +    # the MAC was learned on is not carried in the route. A
> > receiver installs such
> > +    # a route into the VNI's access VLAN, which is the wrong VLAN
> > for every tagged
> > +    # MAC. Between FRR peers the entry is inert (the tagged VLANs
> > are forwarded via
> > +    # data plane learning), but third-party VTEPs break on it in
> > different ways:
> > +    # Cisco NX-OS imports it as a control-plane MAC in the VNI's
> > bridge domain,
> > +    # which bypasses the QinQ hairpin that restores the inner tag
> > -- flooded
> > +    # traffic still works, the unicast that follows it is dropped.
> > RouterOS files
> > +    # it under the VXLAN port's bridge-pvid and blackholes that
> > MAC outright.
> > +    # BIRD is unaffected, it maps the Ethernet Tag to a VLAN
> > explicitly.
> > +    # Suppress the type-2 routes for these vnets, BUM traffic is
> > unaffected as it
> > +    # is still head-end replicated via the type-3 routes.
> > +    if ($plugin_config->{vlanaware}) {
> > +        my $route_map_out = 'MAP_VTEP_OUT';
> > +        $route_map_out .= "_$controller->{'peer-group-name'}"
> > +            if $controller->{'peer-group-name'};
> > +
> > +        # seq is renumbered by array order in
> > PVE::Network::SDN::Frr, so unshift
> > +        # to be evaluated before the permit entry that terminates
> > the route-map
> > +        unshift(
> > +            @{ $config->{frr}->{routemaps}->{$route_map_out} },
> > +            {
> > +                seq => 1,
> > +                action => 'deny',
> > +                matches => [
> > +                    { key => 'evpn route-type', value => 'macip'
> > },
> > +                    { key => 'evpn vni', value => $plugin_config-
> > >{tag} },
> > +                ],
> > +            },
> > +        );
> > +    }
> > +
> >      my $exitnodes = $zone->{'exitnodes'};
> >      my $exitnodes_local_routing = $zone->{'exitnodes-local-
> > routing'};
> >  
> > diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> > b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> > index 0e79707..e597120 100644
> > --- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> > +++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm
> > @@ -218,15 +218,24 @@ sub generate_sdn_config {
> >      }
> >      $mtu = $plugin_config->{mtu} if $plugin_config->{mtu};
> >  
> > +    # a vlan-aware vnet transports the guest VLAN tags
> > transparently over a single
> > +    # VNI. The EVPN control plane only knows about the VNI's
> > access VLAN, so it can
> > +    # neither advertise nor install MAC entries for the other
> > VLANs. Fall back to
> > +    # data plane learning (and no ARP/ND suppression) for those
> > vnets, BUM traffic
> > +    # is still replicated via the type-3 routes installed by the
> > controller.
> > +    my $vlanaware = $vnet->{vlanaware};
> > +
> >      #vxlan interface
> >      my $vxlan_iface = "vxlan_$vnetid";
> >      my @iface_config = ();
> >      push @iface_config, "vxlan-id $tag";
> >      push @iface_config, "vxlan-local-tunnelip $ifaceip" if
> > $ifaceip;
> >      push @iface_config, "vxlan-port $vxlanport" if $vxlanport;
> > -    push @iface_config, "bridge-learning off";
> > -    push @iface_config, "bridge-arp-nd-suppress on"
> > -        if !$plugin_config->{'disable-arp-nd-suppression'};
> > +    if (!$vlanaware) {
> > +        push @iface_config, "bridge-learning off";
> > +        push @iface_config, "bridge-arp-nd-suppress on"
> > +            if !$plugin_config->{'disable-arp-nd-suppression'};
> > +    }
> >  
> >      push @iface_config, "mtu $mtu" if $mtu;
> >      push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config-
> > >{$vxlan_iface};
> > @@ -299,6 +308,10 @@ sub generate_sdn_config {
> >      push @iface_config, "bridge_ports $vxlan_iface";
> >      push @iface_config, "bridge_stp off";
> >      push @iface_config, "bridge_fd 0";
> > +    if ($vlanaware) {
> > +        push @iface_config, "bridge-vlan-aware yes";
> > +        push @iface_config, "bridge-vids 2-4094";
> > +    }
> >      push @iface_config, "mtu $mtu" if $mtu;
> >      push @iface_config, "alias $alias" if $alias;
> >      push @iface_config, "ip-forward on" if $enable_forward_v4;
> > @@ -423,8 +436,6 @@ sub vnet_update_hook {
> >  
> >      raise_param_exc({ tag => "missing vxlan tag" }) if
> > !defined($tag);
> >      raise_param_exc({ tag => "vxlan tag max value is 16777216" })
> > if $tag > 16777216;
> > -    raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be
> > enabled with evpn" })
> > -        if $vnet->{vlanaware};
> >  
> >      # verify that tag is not already defined globally (vxlan-id
> > are unique)
> >      foreach my $id (keys %{ $vnet_cfg->{ids} }) {
> 
> 
> 

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Stefan Hanreich]

On 9/9/26 6:52 AM, DERUMIER, Alexandre wrote:
> Hi,
> 
> they are also another way (that work with vxlan, but I'm not sure with
> evpn),
> 
> https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-515/Network-Virtualization/VXLAN-Devices/
> https://blog.vyos.io/evpn-vxlan-enhancements-introducing-single-vxlan-device-support
> https://github.com/FRRouting/frr/pull/12364
> 
> it's mapping vlan tags from a vlan aware bridge to vnis, and transport
> them in a single vxlan interface.
> 
> 
> It was pretty new and buggy 5year ago, so I never tried to implemented
> it,
> maybe it could be interesting to look at it.

Yes, we were discussing this as well in order to implement the VLAN-based
approach - @Gabriel has tested it internally already and it seemed to
work quite well from what he said. For the other approach with VLAN
tags inside the VXLAN packet the current workaround is needed though.

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Stefan Hanreich]

On 9/8/26 10:56 PM, Thomas Glanzmann wrote:
> Hello Stefan,
> 
>> Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then
>> attaching a route map to the EVPN controller work for your use case as
>> well?
> 
> Yes. I rebuilt my setup that way and it does everything the patch did, so
> consider the patch withdrawn in favour of your approach - with one
> packaging gap that has to be closed first, see below.
> 
> I tested it against three foreign VTEP implementations. Versions:
> 
>   Proxmox     PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches),
>               libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3,
>               ifupdown2 3.3.0-1+pmx12       VTEP 10.101.0.18
>   MikroTik    CRS304-4XG, RouterOS 7.25beta3            VTEP 172.31.0.4
>   Cisco       N9K-C9372TX, NX-OS 9.3(16)                VTEP 172.31.0.7
>   BIRD        3.3.2 (Debian bird3)                      VTEP 172.31.0.118
> 

[...]

> So the answer to your question is yes, and I prefer your shape to mine: it
> gets the L2 EVI without a VRF, and the filtering is explicit and visible in
> the configuration instead of hardcoded in a plugin.

Thanks for testing! I personally also prefer the solution with the explicit 
route map for that reason.

[...]

>> Do you see any other issues, particularly with your specific setup?
> 
> Three, one of them a blocker.
> 
> 1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one
>    thing that stops your proposal from working out of the box, and it fails
>    silently. generate_sdn_config computes $ifaceip via
>    find_local_ip_interface_peers() (or from the fabric node) and then never
>    uses it for the interface:

[...]

> 
>    No type-3 is originated, no VTEP is learned, nothing forwards, and all
>    BGP sessions look perfectly healthy while this is the case. This does
>    not matter for a plain VXLAN zone, because static vxlan_remoteip
>    head-end replication works fine without a local tunnel IP - it bites
>    only the moment an EVPN controller is supposed to drive that zone,
>    which is precisely the setup you want to make official. I worked around
>    it with
> 
>      # /etc/network/interfaces.d/zz-vxlan-local-tunnelip
>      iface vxlan_e1
>              vxlan-local-tunnelip 10.101.0.18
> 
>    which ifupdown2 merges with the generated stanza and which survives
>    pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1,
>    Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with
>    flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm
>    already has would fix it; I am happy to send that as a patch if you
>    want it.

That should have already been fixed in the FRR version used in your tests
(10.6.1-1+pve3) [1] as that was a bug introduced in version 10.6. @Gabriel
can you take a look?

[...]

> One smaller note: with a single EVPN controller
> skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare
> permit 1 plus the call; with several controllers the generated
> extcommunity match is added alongside the call, and I have not tested that
> combination.
> 
> So: tenable, yes, and I would rather have your version than mine. If the
> vxlan-local-tunnelip line lands, the setup works with no patched Perl at
> all, which I have now verified against NX-OS, RouterOS and BIRD at the
> same time.

Yes, with multiple EVPN controllers this would most likely break out of the
box - since only Route Targets from EVPN zones / vnets are considered when
generating the extcommunity list for the RT filter. It is possible to override
this behavior with a custom route map as well, even if it is just a blank
permit. This could be more ergonomic though since it is required to build 
the RT filter manually with Route Map entries...


[1] https://lore.proxmox.com/pve-devel/20260714111858.232230-1-g.goller@proxmox.com/

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Gabriel Goller]

On 09.09.2026 10:40, Stefan Hanreich wrote:
> 
> 
> On 9/8/26 10:56 PM, Thomas Glanzmann wrote:
> > Hello Stefan,
> > 
> >> Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then
> >> attaching a route map to the EVPN controller work for your use case as
> >> well?
> > 
> > Yes. I rebuilt my setup that way and it does everything the patch did, so
> > consider the patch withdrawn in favour of your approach - with one
> > packaging gap that has to be closed first, see below.
> > 
> > I tested it against three foreign VTEP implementations. Versions:
> > 
> >   Proxmox     PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches),
> >               libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3,
> >               ifupdown2 3.3.0-1+pmx12       VTEP 10.101.0.18
> >   MikroTik    CRS304-4XG, RouterOS 7.25beta3            VTEP 172.31.0.4
> >   Cisco       N9K-C9372TX, NX-OS 9.3(16)                VTEP 172.31.0.7
> >   BIRD        3.3.2 (Debian bird3)                      VTEP 172.31.0.118
> > 
> 
> [...]
> 
> > So the answer to your question is yes, and I prefer your shape to mine: it
> > gets the L2 EVI without a VRF, and the filtering is explicit and visible in
> > the configuration instead of hardcoded in a plugin.
> 
> Thanks for testing! I personally also prefer the solution with the explicit 
> route map for that reason.
> 
> [...]
> 
> >> Do you see any other issues, particularly with your specific setup?
> > 
> > Three, one of them a blocker.
> > 
> > 1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one
> >    thing that stops your proposal from working out of the box, and it fails
> >    silently. generate_sdn_config computes $ifaceip via
> >    find_local_ip_interface_peers() (or from the fabric node) and then never
> >    uses it for the interface:
> 
> [...]
> 
> > 
> >    No type-3 is originated, no VTEP is learned, nothing forwards, and all
> >    BGP sessions look perfectly healthy while this is the case. This does
> >    not matter for a plain VXLAN zone, because static vxlan_remoteip
> >    head-end replication works fine without a local tunnel IP - it bites
> >    only the moment an EVPN controller is supposed to drive that zone,
> >    which is precisely the setup you want to make official. I worked around
> >    it with
> > 
> >      # /etc/network/interfaces.d/zz-vxlan-local-tunnelip
> >      iface vxlan_e1
> >              vxlan-local-tunnelip 10.101.0.18
> > 
> >    which ifupdown2 merges with the generated stanza and which survives
> >    pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1,
> >    Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with
> >    flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm
> >    already has would fix it; I am happy to send that as a patch if you
> >    want it.
> 
> That should have already been fixed in the FRR version used in your tests
> (10.6.1-1+pve3) [1] as that was a bug introduced in version 10.6. @Gabriel
> can you take a look?

Yes, this should be fixed with https://github.com/FRRouting/frr/pull/22555.
Note that `show evpn vni x` shows the zebra vtep, and zebra can't find a vtep as
there is no vxlan-local-tunnelip set on the interface. bgpd will fallback to the
router-ip, so `show bgp l2vpn evpn route` should show routes with the correct
derived vtep ip.

The ifupdown2 warning is only printed on the syntax-check, which gets executed
when using e.g. `ifreload -s`. We don't do that anywhere AFAICS. Maybe we should
patch this warning out as it's technically false. No vxlan-local-tunnelip does
make sense when e.g. using source routing (ip src) or multiple local addresses
(migration).

> [...]
> 
> > One smaller note: with a single EVPN controller
> > skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare
> > permit 1 plus the call; with several controllers the generated
> > extcommunity match is added alongside the call, and I have not tested that
> > combination.
> > 
> > So: tenable, yes, and I would rather have your version than mine. If the
> > vxlan-local-tunnelip line lands, the setup works with no patched Perl at
> > all, which I have now verified against NX-OS, RouterOS and BIRD at the
> > same time.
> 
> Yes, with multiple EVPN controllers this would most likely break out of the
> box - since only Route Targets from EVPN zones / vnets are considered when
> generating the extcommunity list for the RT filter. It is possible to override
> this behavior with a custom route map as well, even if it is just a blank
> permit. This could be more ergonomic though since it is required to build 
> the RT filter manually with Route Map entries...
> 
> 
> [1] https://lore.proxmox.com/pve-devel/20260714111858.232230-1-g.goller@proxmox.com/
> 

Re: [PATCH] sdn: evpn: allow vlan-aware vnets

[Thomas Glanzmann]

Hello Gabriel,

> Yes, this should be fixed with https://github.com/FRRouting/frr/pull/22555.
> Note that `show evpn vni x` shows the zebra vtep, and zebra can't find a
> vtep as there is no vxlan-local-tunnelip set on the interface. bgpd will
> fallback to the router-ip, so `show bgp l2vpn evpn route` should show routes
> with the correct derived vtep ip.

You are right and I was wrong. I re-measured it today and my "no type-3 is
originated, no VTEP is learned, nothing forwards" was a misattribution -
please disregard that part of my last mail. Details below, since I think the
exact failure mode is worth having in the archive.

Setup unchanged from my last mail: PVE 9.2.11 with stock libpve-network-perl
1.6.7, frr 10.6.1-1+pve3, ifupdown2 3.3.0-1+pmx12 (VTEP 10.101.0.18), against
MikroTik CRS304-4XG on RouterOS 7.25beta3 (172.31.0.4), Cisco N9K-C9372TX on
NX-OS 9.3(16) (172.31.0.7) and BIRD 3.3.2 (172.31.0.118). The fix is indeed in
the package I tested with:

  frr (10.6.1-1+pve3) trixie; urgency=medium
    * fix VTEP derivation for VXLAN interfaces without a local IP address

  a5ad7bd "frr: backport VXLAN local VTEP fallback fix"
    debian/patches/upstream/0028-bgpd-zebra-fix-VXLAN-interface-local-IP-fallback.patch
    debian/patches/upstream/0029-tests-add-EVPN-L2VNI-unspecified-VTEP-topotest.patch

1. THE FALLBACK WORKS - my workaround file is gone for good

I removed /etc/network/interfaces.d/zz-vxlan-local-tunnelip, so there is now
no vxlan-local-tunnelip anywhere on the node, and the kernel device really has
no local address:

  # ip -d link show vxlan_e1
  21: vxlan_e1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 master e1
      vxlan id 1 srcport 0 0 dstport 4789 ttl auto ageing 300

  # vtysh -c 'show evpn vni 1'
   VNI: 1 / Type: L2 / Vlan: 1 / Bridge: e1
   Local VTEP IP: (null)                <- exactly as you describe
   Remote VTEPs for this VNI:
    172.31.0.7   flood: HER
    172.31.0.4   flood: HER
    172.31.0.118 flood: HER

  # vtysh -c 'show bgp l2vpn evpn neighbors 172.31.0.7 advertised-routes'
   *> [3]:[0]:[32]:[10.101.0.18]        <- derived from bgp router-id
   Total number of prefixes 1

and on the wire the outer header carries the same address, with the customer
tag inside the VNI where it belongs (this one is the N9K leg, i.e. the QinVNI
peer that is fussiest about all of this):

  10.101.0.18.42551 > 172.31.0.7.4789: VXLAN, flags [I] (0x08), vni 1
  bc:24:11:ec:33:f2 > 58:11:22:b6:19:fa, ethertype 802.1Q (0x8100),
      vlan 5, p 0, ethertype IPv4, 10.0.0.1 > 10.0.0.4: ICMP echo request

All 12 directed pairs over {10.0.0.1 PVE VM, .3 MikroTik, .4 N9K, .5 BIRD} at
0% loss, `ping -M do -s 1338` passes and `-s 1340` fails end to end, so the
1366-byte tenant MTU is still exact to the byte. The N9K learns the guest MAC
where it should, on the data-plane port and in the customer VLAN, not as a
control-plane MAC in the VNI bridge domain:

  # show mac address-table
  *    5     bc24.11ec.33f2   dynamic  0  F  F  Eth1/3

and it survives a full `pvesh set /cluster/sdn` (regenerated interfaces + frr,
still 0% loss). So the proposed VLAN-aware-VXLAN-zone + EVPN-controller +
route-map setup does work out of the box on stock packages, with no patched
Perl and no local tunnel IP, against all three foreign implementations. The
"packaging gap" in my last mail does not exist; there is nothing to fix in
VxlanPlugin.pm.

One packaging-adjacent trap while re-testing, in case someone else chases it:
after deleting a vxlan-local-tunnelip stanza, the FIRST `ifreload -a` still
programs the old address, because the module-global vxlan._vxlan_local_tunnelip
is also populated from the saved old ifaceobjs in get_dependent_ifacenames().
A second `ifreload -a` clears it. I mistook that for "the workaround is load
bearing" more than once.

2. WHAT I ACTUALLY SAW - `Vlan: 0`, and it is not about the tunnel IP

What I reported as "nothing forwards" came with `VNI: 1 / Vlan: 0 / Local VTEP
IP: (null)`. The `Vlan: 0` is real and reproducible, but it is independent of
vxlan-local-tunnelip. Control:

  systemctl restart frr, WITHOUT vxlan-local-tunnelip -> Vlan: 0
  systemctl restart frr, WITH    vxlan-local-tunnelip -> Vlan: 0

i.e. after an frr restart zebra does not re-resolve the bridge VLAN for the
L2VNI. A subsequent `ifreload -a` does not heal it either; bouncing the vxlan
interface (`ifdown vxlan_e1 && ifup vxlan_e1`) does, and then it is `Vlan: 1`
again. In the `Vlan: 0` state the remote VTEPs stay learned and forwarding
stays at 0% loss in my topology, so it is not fatal here - but zebra does
advertise the local MAC as an Ethernet-Tag-0 type-2 in that state, which is
precisely the route the route-map on the controller has to filter, so on a
QinVNI peer such as the N9K it would matter (with the route-map in place it is
filtered on the way out, which is why it stayed invisible). Tell me if you want
that as a separate report, I can produce a clean reproducer.

3. THE ifupdown2 WARNING - patch attached

> The ifupdown2 warning is only printed on the syntax-check, which gets
> executed when using e.g. `ifreload -s`. We don't do that anywhere AFAICS.
> Maybe we should patch this warning out as it's technically false.

Confirmed on both counts. `_module_syntax_check()` in ifupdownmain.py is only
reached under the `syntaxcheck` flag, and PVE only ever calls `ifreload -a`
(API2/Network.pm) and `ifquery -a -c -o json` (SDN.pm, SDN/Zones.pm), neither
of which takes that path.

One thing to add though: it is not only a cosmetic warning. vxlan.py's
syntax_check() *returns False* on the missing attribute, so `ifreload -s -a`
exits non-zero on a perfectly good configuration - which is what an admin
validating a config before applying it would see.

Patch attached as git format-patch against the ifupdown2 packaging repo
(new debian/patches/pve/0016-*.patch plus series entry). It drops the whole
block rather than just the log line, for the return-value reason above; the
clagd anycast check below it is unaffected and copes with local == None.

Tested by applying it to /usr/share/ifupdown2/addons/vxlan.py on the node:

  before: warning: vxlan_e1: missing vxlan-local-tunnelip ; rc=1
  after:  (gone)

4. A SECOND FALSE POSITIVE IN THE SAME SYNTAX CHECK

With the patch applied, `ifreload -s -a` still exits 1 on this configuration,
because of bridge.py::_error_vxlan_in_vlan_aware_br():

  warning: vxlan_e1: `bridge-access` attribute is mandatory when vxlan device
           (vxlan_e1) is part of vlan aware bridge (e1)

That is a log_error(), so it also sets ifaceStatus.ERROR on the interface. It
is equally false for the setup we are discussing: a VLAN bundle service maps
a whole VLAN range into one VNI, so the vxlan port must be a trunk in the
vlan-aware bridge and must NOT have bridge-access. It is exactly the config
that the VXLAN zone plugin generates for a vlanaware vnet, so any EVPN-driven
VLAN-aware zone hits it. I left it out of the attached patch since you only
asked about the tunnel-ip one - say the word and I will send a second patch
(it needs `and not <vlan-aware trunk>` rather than a plain removal, since the
check is legitimate for the VNI-per-VLAN case).

Cheers,
        Thomas