From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 5081A1FF0B2 for ; Tue, 08 Sep 2026 22:56:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D444A21599; Tue, 08 Sep 2026 22:56:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=glanzmann.de; s=infra26101010; t=1788900988; bh=h0N/9b3o8Qn7j5Xp9fsAsJNzKgK4XN4AzpWdHE3Et9k=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=oyU/dokpED7rV8+IhijKwbBDN/CzZ1at14WYfyj5cNbYfWQzPhVaLmptvlJtSjs9C R/iMcD2534bvHbdrFe6huo6YrzFPwtngE1bVSVc4+XVpEmtPYIeb3wxkAUG8WTBkBF M+OOifBrljDMLvW6xqStlwHLqJk5zKAFsshCoGeSBC3gtOrH1yLRNllFIYPDSigfcz rvnnmbuKelCxqkjzBVlvSlV9c6OKV5NR8WQ45R5ka+Mk1uXNaT9JJhnrHkAKdgYkrA H4kKiA97uiZJJzrT8qMJzL3CXohlSfqeEjzCMfe7RWGLo4zrQifIhXoPuZUl3Nu8j8 j9fE61DUD2TMHRX3K8tn+CJFi43lr2CpYC74e0EORRwh31lyOZFWJfGEqEQ8Xj9ZYw 0jV+UKzMkCHC3cGZAYzuoH7tRrLJkZ7XYGqCEOqQFic57pRBfySFER3ymFoNj933LL htI2vcG7XJb9f/MzU/LDlcn4RfFd33KgMMGaeJJddF3ITQG4hgN7vmlXxlFcer7jVT kDtGIId3EsE+Oe8Kg2lBaBhQj5gfom755RXzP1Nsmo8WpJzyfyXdKh+bquHVTKA17X P3B3XMVE8PS80LTfEdKRFG1lpos+EAlaT0hFXGrAyYa/hcz9t//AMSSyfiMyn4SRS3 zcCLrAYhuy8Fby7ctMJkp+OU= Date: Tue, 8 Sep 2026 22:56:28 +0200 From: Thomas Glanzmann To: Stefan Hanreich Subject: Re: [PATCH] sdn: evpn: allow vlan-aware vnets Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL -0.150 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5CBWVP3GZXHQ3WSGSDBQU7OA6H5RKGI2 X-Message-ID-Hash: 5CBWVP3GZXHQ3WSGSDBQU7OA6H5RKGI2 X-MailFrom: thomas@glanzmann.de X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hello Stefan, > Does utilizing a VLAN-aware VXLAN zone with an EVPN controller and then > attaching a route map to the EVPN controller work for your use case as > well? Yes. I rebuilt my setup that way and it does everything the patch did, so consider the patch withdrawn in favour of your approach - with one packaging gap that has to be closed first, see below. I tested it against three foreign VTEP implementations. Versions: Proxmox PVE 9.2.11, libpve-network-perl 1.6.7 (stock, no patches), libpve-rs-perl 0.15.3, frr 10.6.1-1+pve3, ifupdown2 3.3.0-1+pmx12 VTEP 10.101.0.18 MikroTik CRS304-4XG, RouterOS 7.25beta3 VTEP 172.31.0.4 Cisco N9K-C9372TX, NX-OS 9.3(16) VTEP 172.31.0.7 BIRD 3.3.2 (Debian bird3) VTEP 172.31.0.118 One stretched L2 segment 10.0.0.0/24 over VNI 1, customer VLAN 5 carried tagged inside the VXLAN and popped at each edge port, iBGP AS 65000, RT 65000:1, full mesh. Tenant endpoints: 10.0.0.1 is a guest on the Proxmox host (net0 bridge=e1,tag=5), 10.0.0.3 hangs off the MikroTik, 10.0.0.4 off the N9K, 10.0.0.5 is the BIRD box itself. The whole Proxmox side, no patched Perl anywhere: pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \ --order 10 --action deny \ --match key=route-type,value=macip --match key=vni,value=1 pvesh create /cluster/sdn/route-maps/entries --route-map-id nomacip \ --order 20 --action permit pvesh create /cluster/sdn/controllers --controller evpnctl --type evpn \ --asn 65000 --route-map-out nomacip \ --peers 10.101.0.18,172.31.0.4,172.31.0.7,172.31.0.118 pvesh create /cluster/sdn/zones --zone vx1 --type vxlan \ --peers 10.101.0.18 pvesh create /cluster/sdn/vnets --vnet e1 --zone vx1 --tag 1 --vlanaware 1 pvesh set /cluster/sdn Result: 12/12 directed pairs at 0% loss, and the MTU is exact to the byte (ping -M do -s 1338 passes end to end, -s 1340 fails - the underlay here crosses a WireGuard tunnel with MTU 1420 and the customer tag costs 4 more bytes inside the VXLAN). The tag really is transported, tcpdump on the underlay port of the Proxmox host: 10.101.0.18.42551 > 172.31.0.4.4789: VXLAN, flags [I] (0x08), vni 1 bc:24:11:ec:33:f2 > a0:36:9f:20:37:d8, ethertype 802.1Q (0x8100), length 102: vlan 5, p 0, ethertype IPv4, 10.0.0.1 > 10.0.0.3: ICMP echo request Point by point against what the patch did: - bridge-vlan-aware yes / bridge-vids 2-4094 on the vnet bridge: the VXLAN zone plugin already does this for a vlanaware vnet. - keep MAC learning on and ARP/ND suppression off on the VXLAN interface: the VXLAN zone never emitted bridge-learning off or bridge-arp-nd-suppress on in the first place, so there is nothing to special-case. - lifting the "vlan-aware option can't be enabled with evpn" check: not needed, that raise_param_exc only lives in Zones/EvpnPlugin.pm::vnet_update_hook, and the VXLAN zone's vnet_update_hook has no equivalent. - denying the type-2 routes: route-map-out on the EVPN controller. It renders as a call inside the generated map, which is exactly what is wanted: route-map MAP_VTEP_OUT permit 1 call nomacip exit route-map nomacip deny 10 match evpn route-type macip match evpn vni 1 exit route-map nomacip permit 20 exit So the answer to your question is yes, and I prefer your shape to mine: it gets the L2 EVI without a VRF, and the filtering is explicit and visible in the configuration instead of hardcoded in a plugin. I re-measured that the route-map is actually load-bearing, because a test that only ever passes proves nothing. Removing just the route-map from the controller and re-applying: pvesh set /cluster/sdn/controllers/evpnctl --delete route-map-out pvesh set /cluster/sdn advertised to 172.31.0.4 WITHOUT the route-map: [2]:[0]:[48]:[bc:24:11:ec:33:f2] <- Ethernet Tag 0 [3]:[0]:[32]:[10.101.0.18] advertised WITH it: [3]:[0]:[32]:[10.101.0.18] Per implementation, with the route-map removed: Cisco NX-OS - breaks, both directions, 100% loss to that guest while the other 10 of 12 pairs stay up. The tag-0 route is imported as a control-plane MAC in the VNI's bridge domain and beats the data plane entry that the QinQ hairpin learned in VLAN 5: C 1000 bc24.11ec.33f2 dynamic nve1(10.101.0.18) * 5 bc24.11ec.33f2 dynamic Eth1/3 Putting route-map-out back heals all 12 pairs with no other action. MikroTik RouterOS - no longer fatal on 7.25beta3, which is a change from what I reported for 7.24.1. It still files the route under the VXLAN port's bridge-pvid as an EXTERNAL host entry: D E mac-address=BC:24:11:EC:33:F2 vid=1 interface=vxlan1 remote-ip=10.101.0.18 but the MAC is no longer blackholed - a lookup miss in VLAN 5 floods to vxlan1 and the frames arrive, 0% loss throughout. The stale-entry half of the defect is still there though: that entry survived the withdrawal of the route when I put the route-map back, and needed an /interface/vxlan/disable + enable to clear. BIRD - unaffected, as expected. Its EVPN protocol has an explicit Ethernet Tag to VID mapping (vni 1 / vid 5 / tag 0), so the route lands in the right VLAN. Worth noting for completeness that BIRD's own tag-0 type-2 does show up on the N9K in the same wrong place (C 1000 aac4.3347.8840 nve1(172.31.0.118)) and is harmless there only because that MAC is the bridge's own and never sources tenant traffic - it is not evidence that NX-OS tolerates tag-0 type-2. > Do you see any other issues, particularly with your specific setup? Three, one of them a blocker. 1. Zones/VxlanPlugin.pm never emits vxlan-local-tunnelip. This is the one thing that stops your proposal from working out of the box, and it fails silently. generate_sdn_config computes $ifaceip via find_local_ip_interface_peers() (or from the fabric node) and then never uses it for the interface: my @iface_config = (); push @iface_config, "vxlan-id $tag"; for my $address (sort @peers) { next if $address eq $ifaceip; push @iface_config, "vxlan_remoteip $address"; } push @iface_config, "vxlan-port $vxlanport" if $vxlanport; push @iface_config, "mtu $mtu" if $mtu; Zones/EvpnPlugin.pm has the line the VXLAN zone is missing: push @iface_config, "vxlan-local-tunnelip $ifaceip" if $ifaceip; ifupdown2 only logs "vxlan_e1: missing vxlan-local-tunnelip" at warning level, and FRR then reports: VNI: 1 Type: L2 Vlan: 0 VxLAN interface: vxlan_e1 Local VTEP IP: (null) No remote VTEPs known for this VNI No type-3 is originated, no VTEP is learned, nothing forwards, and all BGP sessions look perfectly healthy while this is the case. This does not matter for a plain VXLAN zone, because static vxlan_remoteip head-end replication works fine without a local tunnel IP - it bites only the moment an EVPN controller is supposed to drive that zone, which is precisely the setup you want to make official. I worked around it with # /etc/network/interfaces.d/zz-vxlan-local-tunnelip iface vxlan_e1 vxlan-local-tunnelip 10.101.0.18 which ifupdown2 merges with the generated stanza and which survives pvesh set /cluster/sdn. Immediately after adding it: Vlan: 1, Local VTEP IP: 10.101.0.18, and all three remote VTEPs appear with flood: HER. Adding the same push to VxlanPlugin.pm as EvpnPlugin.pm already has would fix it; I am happy to send that as a patch if you want it. 2. The peers requirement you already mentioned is real - on_update_hook demands exactly one of peers / fabric. There is a tidier form than entering an arbitrary peer though: list ONLY the local IP. The remoteip loop skips the local address, so no static vxlan_remoteip is generated at all and every VTEP is learned purely from type-3, which also avoids mixing static head-end replication with BGP-learned flood entries. That reduces peers to a pure "which of my addresses is the VTEP IP" declaration - which, if you are redesigning this anyway, is probably the thing the zone actually wants to say, and it could equally come from the controller or the fabric. 3. The controller route-map is per peer-group, not per VNI, so it applies to every zone on that controller. That is why my nomacip entry carries an explicit match evpn vni 1 - without it, an L3 EVI sharing the same controller would lose its type-2 routes too. It works, but it means the user has to know to scope the deny by VNI by hand, and it is the one place where the explicit route-map is less safe than the hardcoded per-vnet deny in my patch. Some documentation, or a per-vnet hook that adds the scoped entry for vlan-aware vnets, would help here. One smaller note: with a single EVPN controller skip_route_target_filtering() returns true, so MAP_VTEP_OUT is a bare permit 1 plus the call; with several controllers the generated extcommunity match is added alongside the call, and I have not tested that combination. So: tenable, yes, and I would rather have your version than mine. If the vxlan-local-tunnelip line lands, the setup works with no patched Perl at all, which I have now verified against NX-OS, RouterOS and BIRD at the same time. I also ordered a Juniper QFX5100-48T-AFO and an Arista DCS-7050TX-64-R in order to conduct more tests. Cheers, Thomas