From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 313001FF18C for ; Tue, 14 Apr 2026 18:35:07 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D62391FB68; Tue, 14 Apr 2026 18:34:06 +0200 (CEST) From: Stefan Hanreich To: pve-devel@lists.proxmox.com Subject: [RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality Date: Tue, 14 Apr 2026 18:32:57 +0200 Message-ID: <20260414163315.419384-1-s.hanreich@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776184326087 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.692 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: CEY3ZRIU7F4KTA72ZCKKLO2J334E6DYO X-Message-ID-Hash: CEY3ZRIU7F4KTA72ZCKKLO2J334E6DYO X-MailFrom: s.hanreich@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This patch series is based on the route-maps / prefix-lists series [1]. ## Introduction This patch series lifts several existing restrictions on the EVPN controller: * Allow defining multiple EVPN controllers in a cluster (which is effectively defining multiple EVPN peer-groups in the FRR configuration) * Allow restricting EVPN controllers to specific nodes * Allow overriding the type of BGP session for EVPN controllers ## Motivation The functionality implemented in this patch series allows exit nodes to act essentially as a ASBR router for the whole cluster or, alternatively, certain PVE nodes as route-server. The core motivation behind this patch series is that enables the possibility to define a second EVPN controller that handles advertising routes to another AS, independent of the intra-AS peering. This is particularly useful with the route-maps / prefix-lists series that allows attaching route-maps to EVPN controllers. That way users can create route-maps for the EVPN controller that connects to another AS (filtering routes, manipulating next-hops, ..). This patch series enables the PVE stack to implement setups similar to the Inter-AS Option B/C (with full-mesh peering inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the VPNv4 AF) [2]. Due to the modular nature of all the components involved, it is possible to replace every component in a complex setup with a physical router / route reflector / ... - giving users flexibility in how they want to build their network. By providing virtual equivalents of every critical component, it is also possible for us to create and manage a DCI EVPN cluster completely via the Proxmox VE stack and its API. Users can still choose to use physical equivalents - with the Proxmox VE stack only handling announcing the routes for an EVPN zone. By implementing and providing this functionality via the API, there is less need for users to run their own hand-crafted frr.conf[.local] files for implementing different types of EVPN setups or resort to running custom, external, FRR instances for mangling routes and then re-advertising them to an uplink. In addition, this API can be used internally or by other products (e.g. PDM) to setup more complex setups automatically, but afterwards still provide users with the ability to customize the pre-generated setups, since a lot of options are more explicit now. ## Future Work Future work should additionally allow specific PVE nodes to act as route reflectors for the cluster. This is similar to how e.g. Calico implements route reflectors [3]. This would make it possible to have the whole route handling manageable via the Proxmox VE stack, without the scaling issues attached to full-mesh peering and without the nodes itself having to be in the data path. This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with the PVE exit nodes acting as RRs and either the PVE node or an external router acting as ASBR. Inter-AS peering between the Route Reflectors of different clusters in the Option C could then happen directly e.g. via the Wireguard fabric and/or exchanging the loopback IPs of the RRs via the ASBRs. This patch series adds basic documentation to the newly introduced options. With the addition of this and other recent patch series, a lot of possibilities for configuring the EVPN stack are exposed to users, that can be leveraged to create sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage the new features and which setups are recommended by us, we will create documentation for example setups and. Since this depends not only on this patch series and the progress we make in the coming weeks, they will be added as a separate patch series - if everything goes as planned - depending on the state of the EVPN stack and the outstanding patch series, as well as the progress we make on PDM side. ## Discussion / Open questions for Reviewers A primary controller still needs to be assigned to an EVPN zone, as that is required for generating the VTEP IPs as well as auto-generating the MTU for the VNet. This means that the controller responsible for announcing the EVPN routes *inside* the cluster needs to be selected, in order for the SDN stack to generate correct configurations. This could be a bit confusing to users if they want to utilize multiple controllers, but aren't sure which one to pick as their 'primary' controller. Simply allowing users to configure multiple EVPN controllers for a zone doesn't work - since we cannot know which controller to use for auto-generating VTEP IPs. The 'legacy' mode might better be called 'Auto', since it automatically inherits the BGP session from a given BGP controller. This could also be used in conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP session established there, while still giving users the possibility to run iBGP sessions instead. ## Backwards Compatibility Great care has been taken to ensure that existing EVPN controllers / zones and BGP controller still work the same as before, every existing test case still works the same without any adjustments. This is possible because existing SDN configs should just be a subset of the now possible EVPN configurations. The new capabilities can only be used with completely new configuration options that need to be set explicitly. Additionally, when generating the FRR configuration, the case of only one EVPN controller existing is special-cased and generating filtering of outgoing routes does not take place then. Backwards Compatibility has been verified via the integration tests, new additional integration tests as well as the e2e tests. ## FRR issues This series suffers from a bug in the FRR version that is currently contained in our repositories (10.4.1), but has been fixed in a subsequent release (10.6). When leaking routes into the default routing table (which happens when defining PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get assigned the wrong ASN (1). This breaks the route filtering implemented in this patch series when multiple EVPN controllers are configured. Existing EVPN setups are *not* affected by this, since this has been special cased and no route filtering takes place. The bug should be addressed before merging this patch series, either by backporting the fix or upgrading the existing FRR version. ## Dependencies proxmox-frr depends on proxmox-frr-templates proxmox-ve-config depends on proxmox-frr proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version pve-network depends on proxmox-perl-rs pve-manager depends on pve-network [1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/ [2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10 [3] www.tigera.io/blog/configuring-route-reflectors-in-calico/ proxmox-ve-rs: Stefan Hanreich (4): frr: add local-as setting frr: add support for extcommunity lists frr-templates: render local-as setting frr-templates: render community lists in templates .../templates/bgp_router.jinja | 4 + proxmox-frr-templates/templates/bgpd.jinja | 6 + proxmox-frr/src/ser/bgp.rs | 103 +++++++++++++++++- proxmox-frr/src/ser/mod.rs | 23 +++- proxmox-frr/src/ser/route_map.rs | 43 ++++++-- 5 files changed, 166 insertions(+), 13 deletions(-) pve-network: Stefan Hanreich (9): evpn controller: make nodes configurable evpn controller: allow multiple evpn controllers in a cluster evpn controller: add bgp-mode setting evpn zone: add secondary-controllers and rt filtering evpn controller: add ebgp-multihop setting test: evpn: add test for ibgp + ebgp evpn controller test: evpn: add legacy test tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session tests: test route filtering mechanism with multiple zones/controllers src/PVE/API2/Network/SDN/Controllers.pm | 13 ++ src/PVE/API2/Network/SDN/Zones.pm | 11 + src/PVE/Network/SDN.pm | 1 + src/PVE/Network/SDN/Controllers.pm | 71 +++++++ src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 22 +- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++--- src/PVE/Network/SDN/Controllers/Plugin.pm | 82 ++++++++ src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 39 +++- .../expected_controller_config | 87 ++++++++ .../expected_sdn_interfaces | 41 ++++ .../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces | 11 + .../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config | 59 ++++++ .../expected_controller_config | 66 ++++++ .../expected_sdn_interfaces | 41 ++++ .../evpn_cluster_ibgp_uplink_ebgp/interfaces | 7 + .../evpn_cluster_ibgp_uplink_ebgp/sdn_config | 51 +++++ .../expected_controller_config | 78 ++++++++ .../expected_sdn_interfaces | 81 ++++++++ .../interfaces | 7 + .../sdn_config | 67 +++++++ .../legacy_mode/expected_controller_config | 61 ++++++ .../evpn/legacy_mode/expected_sdn_interfaces | 41 ++++ src/test/zones/evpn/legacy_mode/interfaces | 7 + src/test/zones/evpn/legacy_mode/sdn_config | 51 +++++ 24 files changed, 1143 insertions(+), 40 deletions(-) create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces create mode 100644 src/test/zones/evpn/legacy_mode/interfaces create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config pve-manager: Stefan Hanreich (1): sdn: evpn: zone: controller: add new advanced fields www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++ www/manager6/sdn/zones/EvpnEdit.js | 14 ++++++++- 2 files changed, 52 insertions(+), 1 deletion(-) pve-docs: Stefan Hanreich (2): sdn: evpn: document new zone / controller options sdn: fix typo in bgp controller pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 70 insertions(+), 4 deletions(-) Summary over all repositories: 32 files changed, 1431 insertions(+), 58 deletions(-) -- Generated by murpp 0.11.0