From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id BD8A51FF0A5 for ; Fri, 04 Sep 2026 11:27:48 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id F09A0215C2; Fri, 04 Sep 2026 11:27:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=glanzmann.de; s=infra26101010; t=1788472144; bh=lO+mt5MnDUfqU4783SE5hfKdkdezkMmih8Vcq2dJk5w=; h=Date:From:To:Subject:From; b=en/cflK3kDlwBVSK/0bmqUvc7NhW9cBlriojWv5xfa9W2tQ6mWVtWEN8zSwQ+r3tq TJ+ROZJkYScHVB+CvAr5x2bvhIpPtOXeOO2TNNafP9GCqG+gKitdg+1QDfss32ZJgN kwy6xJ9dVQHJqtViyN6fPL17JkqKQSfAmlB4iFLodjs9q36pJNI7P91l5dHvsaOVyS rKBMc/2fvK9OMYNB8+rRXzFzZBnj8Ye3PWi/CcKM0C3WqLIu9B7Waci04xGSBRgYf5 wb6ArGeGo0fmJjFNQbagPkwcRhGXwL4/nPDYOj1ktIkiJIrYmxw6mkr6OUVjkRX/A5 T52CD/HcVzIWfKpV6Q/V71tQN+FhaEY+yz32HLsRtxdCvyhahL4ACGBAW9VO+JiXET 09qnh6qGTnQIR+xsXXoTMT+NFNG1tx5arJPLIZDjMHnL+/gtxK10MBmve6aXDUd2vC vdEe7RYRbLxYhCjEF344rX69OtWYNIxEqn9kwfcI96sUDCLLTqPDqhJBGMBMGOxKoq 4UT0308UcI7kBlcSMy6Snxw+Ct9nR5IKGK2qfrfWQXbGGkl81dqWqYYzo+kf6F695y 8zQ2iFtHguhTSWzOuDd2IeiN6Gz7cTl7WyNxvXC+7sT/gWBabajcSWRFh3PMdI/UJN nGX8HAu1PbJIVJMZgtwmRouM= Date: Thu, 3 Sep 2026 23:49:04 +0200 From: Thomas Glanzmann To: pve-devel@lists.proxmox.com Subject: [PATCH] sdn: evpn: allow vlan-aware vnets Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: thomas@glanzmann.de X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: ULS3FHXGYFDYK4ZQWHVRUHX3Y44DRU4S X-Message-ID-Hash: ULS3FHXGYFDYK4ZQWHVRUHX3Y44DRU4S X-Mailman-Approved-At: Fri, 04 Sep 2026 11:27:28 +0200 X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: EVPN vnets rejected the vlanaware flag outright, so a guest could not be attached to an EVPN vnet with a VLAN tag -- tap_plug bails out with "vm vlans are not allowed on vnet " because the vnet bridge has VLAN filtering disabled. Lift the restriction and generate a VLAN-aware vnet bridge (vids 2-4094) just like the VXLAN zone does, so the guest VLAN tags are carried transparently over the vnet's VNI: a guest on an untagged port sends 802.1q frames that get VXLAN encapsulated as-is, and a guest on a port with a VLAN tag receives them with the tag popped. Such a vnet is a VLAN bundle service, and FRR does not implement the VLAN-aware variant of it: it originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN a MAC was learned on is not carried in the route. FRR does advertise those MACs -- it does not restrict itself to the VNI's access VLAN -- but a receiver has no way to tell which VLAN they belong to. Between FRR peers that is harmless. The receiver installs the route into the VNI's access VLAN where it is inert, and the entry that actually forwards the traffic comes from data plane learning: bc:24:11:c7:71:9c dev vxlan_e1 vlan 1 extern_learn master e1 bc:24:11:c7:71:9c dev vxlan_e1 vlan 5 master e1 Against a third-party VTEP it is not harmless, and it breaks differently depending on how the peer maps the Ethernet Tag. Measured in a four-VTEP fabric (VNI 1, customer VLAN 5 carried tagged inside it) with the route-map deny below removed: - Cisco NX-OS 9.3, vlan-based VNI fed by a QinQ hairpin: imports the tag-0 route as a control-plane MAC in the VNI's bridge domain, C 1000 bc24.11f1.9236 dynamic nve1(10.101.0.11) which bypasses the hairpin that restores the inner VLAN tag. Broadcast survives -- an ARP request is flooded through the hairpin and arrives correctly tagged -- but the unicast that follows is dropped and never reaches the guest. 100% loss to every FRR-hosted guest, in both directions. - MikroTik RouterOS 7.24.1: files the tag-0 route under the VXLAN port's bridge-pvid instead of the VLAN carrying the traffic, as an EXTERNAL bridge host entry, and then blackholes that MAC outright -- including for traffic in the VLAN it really belongs to. Withdrawing the route does not clear the entry, the VXLAN interface has to be bounced. - BIRD 3.3.2: unaffected. Its EVPN protocol has an explicit tag to VID mapping (vni 1 / vid 5 / tag 0), so the route lands in the correct VLAN and forwards normally. So keep MAC learning enabled and ARP/ND suppression disabled on the VXLAN interface of such a vnet, and additionally deny the type-2 routes for its VNI in the outgoing VTEP route-map, so no peer is fed MAC routes that carry no usable VLAN. BUM traffic is unaffected, it is still head-end replicated via the type-3 routes installed by the controller, and the tagged VLANs are covered by data plane learning on both sides. Subnets stay mutually exclusive with vlanaware, which is already enforced by the vnet and subnet plugins, so this does not affect the L3/gateway setup of an EVPN zone. Signed-off-by: Thomas Glanzmann --- .../PVE/Network/SDN/Controllers/EvpnPlugin.pm | 33 ++++++++++++++++++++++ src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 21 ++++++++++---- 2 files changed, 49 insertions(+), 5 deletions(-) diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm --- a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm +++ b/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm @@ -586,6 +586,39 @@ sub generate_vnet_frr_config { my ($class, $plugin_config, $controller, $zone, $zoneid, $vnetid, $config) = @_; + # A vlan-aware vnet carries the guest VLAN tags inside a single VNI, but FRR + # originates type-2 (MAC/IP) routes with an Ethernet Tag ID of 0, so the VLAN + # the MAC was learned on is not carried in the route. A receiver installs such + # a route into the VNI's access VLAN, which is the wrong VLAN for every tagged + # MAC. Between FRR peers the entry is inert (the tagged VLANs are forwarded via + # data plane learning), but third-party VTEPs break on it in different ways: + # Cisco NX-OS imports it as a control-plane MAC in the VNI's bridge domain, + # which bypasses the QinQ hairpin that restores the inner tag -- flooded + # traffic still works, the unicast that follows it is dropped. RouterOS files + # it under the VXLAN port's bridge-pvid and blackholes that MAC outright. + # BIRD is unaffected, it maps the Ethernet Tag to a VLAN explicitly. + # Suppress the type-2 routes for these vnets, BUM traffic is unaffected as it + # is still head-end replicated via the type-3 routes. + if ($plugin_config->{vlanaware}) { + my $route_map_out = 'MAP_VTEP_OUT'; + $route_map_out .= "_$controller->{'peer-group-name'}" + if $controller->{'peer-group-name'}; + + # seq is renumbered by array order in PVE::Network::SDN::Frr, so unshift + # to be evaluated before the permit entry that terminates the route-map + unshift( + @{ $config->{frr}->{routemaps}->{$route_map_out} }, + { + seq => 1, + action => 'deny', + matches => [ + { key => 'evpn route-type', value => 'macip' }, + { key => 'evpn vni', value => $plugin_config->{tag} }, + ], + }, + ); + } + my $exitnodes = $zone->{'exitnodes'}; my $exitnodes_local_routing = $zone->{'exitnodes-local-routing'}; diff --git a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm index 0e79707..e597120 100644 --- a/src/PVE/Network/SDN/Zones/EvpnPlugin.pm +++ b/src/PVE/Network/SDN/Zones/EvpnPlugin.pm @@ -218,15 +218,24 @@ sub generate_sdn_config { } $mtu = $plugin_config->{mtu} if $plugin_config->{mtu}; + # a vlan-aware vnet transports the guest VLAN tags transparently over a single + # VNI. The EVPN control plane only knows about the VNI's access VLAN, so it can + # neither advertise nor install MAC entries for the other VLANs. Fall back to + # data plane learning (and no ARP/ND suppression) for those vnets, BUM traffic + # is still replicated via the type-3 routes installed by the controller. + my $vlanaware = $vnet->{vlanaware}; + #vxlan interface my $vxlan_iface = "vxlan_$vnetid"; my @iface_config = (); push @iface_config, "vxlan-id $tag"; push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip; push @iface_config, "vxlan-port $vxlanport" if $vxlanport; - push @iface_config, "bridge-learning off"; - push @iface_config, "bridge-arp-nd-suppress on" - if !$plugin_config->{'disable-arp-nd-suppression'}; + if (!$vlanaware) { + push @iface_config, "bridge-learning off"; + push @iface_config, "bridge-arp-nd-suppress on" + if !$plugin_config->{'disable-arp-nd-suppression'}; + } push @iface_config, "mtu $mtu" if $mtu; push(@{ $config->{$vxlan_iface} }, @iface_config) if !$config->{$vxlan_iface}; @@ -299,6 +308,10 @@ sub generate_sdn_config { push @iface_config, "bridge_ports $vxlan_iface"; push @iface_config, "bridge_stp off"; push @iface_config, "bridge_fd 0"; + if ($vlanaware) { + push @iface_config, "bridge-vlan-aware yes"; + push @iface_config, "bridge-vids 2-4094"; + } push @iface_config, "mtu $mtu" if $mtu; push @iface_config, "alias $alias" if $alias; push @iface_config, "ip-forward on" if $enable_forward_v4; @@ -423,8 +436,6 @@ sub vnet_update_hook { raise_param_exc({ tag => "missing vxlan tag" }) if !defined($tag); raise_param_exc({ tag => "vxlan tag max value is 16777216" }) if $tag > 16777216; - raise_param_exc({ 'vlan-aware' => "vlan-aware option can't be enabled with evpn" }) - if $vnet->{vlanaware}; # verify that tag is not already defined globally (vxlan-id are unique) foreach my $id (keys %{ $vnet_cfg->{ids} }) { -- 2.47.3