Re: [RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality

Hannes Laimer
2026-04-16T16:23:14Z

This looks very good to me!
The cover-letter is very well written and is very helpful for giving
context, an overview and reasoning for the changes.

I also did some basic testing, nothing too extensive, but I didn't
notice anything off.

With my comment on path 06/16 addressed consider this

Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>

note: FRR does allow things like `4200000001:1000`, and that
wouldn't fit into the `u16`. FRR does accept[1] these 4-octet[2] ASs,
expanding the u16 to a u32 in `ExtCommunityRouteTarget` should be
enough. There shouldn't be much of a downside (I think)

[1]
https://github.com/FRRouting/frr/blob/3b419ef76b8c0dfe8eb47097955f673f462c05e3/bgpd/bgp_ecommunity.c#L807
[2] https://www.rfc-editor.org/rfc/rfc5668

On 2026-04-14 18:33, Stefan Hanreich wrote:
> This patch series is based on the route-maps / prefix-lists series [1].
>
> ## Introduction
>
> This patch series lifts several existing restrictions on the EVPN controller:
>
> * Allow defining multiple EVPN controllers in a cluster (which is effectively
> defining multiple EVPN peer-groups in the FRR configuration)
> * Allow restricting EVPN controllers to specific nodes
> * Allow overriding the type of BGP session for EVPN controllers
>
>
> ## Motivation
>
> The functionality implemented in this patch series allows exit nodes to act
> essentially as a ASBR router for the whole cluster or, alternatively, certain
> PVE nodes as route-server. The core motivation behind this patch series is that
> enables the possibility to define a second EVPN controller that handles
> advertising routes to another AS, independent of the intra-AS peering. This is
> particularly useful with the route-maps / prefix-lists series that allows
> attaching route-maps to EVPN controllers. That way users can create route-maps
> for the EVPN controller that connects to another AS (filtering routes,
> manipulating next-hops, ..). This patch series enables the PVE stack to
> implement setups similar to the Inter-AS Option B/C (with full-mesh peering
> inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the
> VPNv4 AF) [2].
>
> Due to the modular nature of all the components involved, it is possible to
> replace every component in a complex setup with a physical router / route
> reflector / ... - giving users flexibility in how they want to build their
> network. By providing virtual equivalents of every critical component, it is
> also possible for us to create and manage a DCI EVPN cluster completely via the
> Proxmox VE stack and its API. Users can still choose to use physical equivalents
> - with the Proxmox VE stack only handling announcing the routes for an EVPN
> zone.
>
> By implementing and providing this functionality via the API, there is less need
> for users to run their own hand-crafted frr.conf[.local] files for implementing
> different types of EVPN setups or resort to running custom, external, FRR
> instances for mangling routes and then re-advertising them to an uplink. In
> addition, this API can be used internally or by other products (e.g. PDM) to
> setup more complex setups automatically, but afterwards still provide users with
> the ability to customize the pre-generated setups, since a lot of options are
> more explicit now.
>
>
> ## Future Work
>
> Future work should additionally allow specific PVE nodes to act as route
> reflectors for the cluster. This is similar to how e.g. Calico implements route
> reflectors [3]. This would make it possible to have the whole route handling
> manageable via the Proxmox VE stack, without the scaling issues attached to
> full-mesh peering and without the nodes itself having to be in the data path.
> This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with
> the PVE exit nodes acting as RRs and either the PVE node or an external router
> acting as ASBR. Inter-AS peering between the Route Reflectors of different
> clusters in the Option C could then happen directly e.g. via the Wireguard
> fabric and/or exchanging the loopback IPs of the RRs via the ASBRs.
>
> This patch series adds basic documentation to the newly introduced options. With
> the addition of this and other recent patch series, a lot of possibilities for
> configuring the EVPN stack are exposed to users, that can be leveraged to create
> sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage
> the new features and which setups are recommended by us, we will create
> documentation for example setups and. Since this depends not only on this patch
> series and the progress we make in the coming weeks, they will be added as a
> separate patch series - if everything goes as planned - depending on the state
> of the EVPN stack and the outstanding patch series, as well as the progress we
> make on PDM side.
>
>
> ## Discussion / Open questions for Reviewers
>
> A primary controller still needs to be assigned to an EVPN zone, as that is
> required for generating the VTEP IPs as well as auto-generating the MTU for the
> VNet. This means that the controller responsible for announcing the EVPN routes
> *inside* the cluster needs to be selected, in order for the SDN stack to
> generate correct configurations. This could be a bit confusing to users if they
> want to utilize multiple controllers, but aren't sure which one to pick as their
> 'primary' controller. Simply allowing users to configure multiple EVPN
> controllers for a zone doesn't work - since we cannot know which controller to
> use for auto-generating VTEP IPs.
>
> The 'legacy' mode might better be called 'Auto', since it automatically inherits
> the BGP session from a given BGP controller. This could also be used in
> conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP
> session established there, while still giving users the possibility to run iBGP
> sessions instead.
>
>
> ## Backwards Compatibility
>
> Great care has been taken to ensure that existing EVPN controllers / zones and
> BGP controller still work the same as before, every existing test case still
> works the same without any adjustments. This is possible because existing SDN
> configs should just be a subset of the now possible EVPN configurations. The new
> capabilities can only be used with completely new configuration options that
> need to be set explicitly. Additionally, when generating the FRR configuration,
> the case of only one EVPN controller existing is special-cased and generating
> filtering of outgoing routes does not take place then.
>
> Backwards Compatibility has been verified via the integration tests, new
> additional integration tests as well as the e2e tests.
>
>
> ## FRR issues
>
> This series suffers from a bug in the FRR version that is currently contained in
> our repositories (10.4.1), but has been fixed in a subsequent release (10.6).
> When leaking routes into the default routing table (which happens when defining
> PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get
> assigned the wrong ASN (1). This breaks the route filtering implemented in this
> patch series when multiple EVPN controllers are configured. Existing EVPN setups
> are *not* affected by this, since this has been special cased and no route
> filtering takes place.
>
> The bug should be addressed before merging this patch series, either by
> backporting the fix or upgrading the existing FRR version.
>
>
> ## Dependencies
>
> proxmox-frr depends on proxmox-frr-templates
> proxmox-ve-config depends on proxmox-frr
> proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version
> pve-network depends on proxmox-perl-rs
> pve-manager depends on pve-network
>
>
> [1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/
> [2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10
> [3] www.tigera.io/blog/configuring-route-reflectors-in-calico/
>
>
> proxmox-ve-rs:
>
> Stefan Hanreich (4):
> frr: add local-as setting
> frr: add support for extcommunity lists
> frr-templates: render local-as setting
> frr-templates: render community lists in templates
>
> .../templates/bgp_router.jinja | 4 +
> proxmox-frr-templates/templates/bgpd.jinja | 6 +
> proxmox-frr/src/ser/bgp.rs | 103 +++++++++++++++++-
> proxmox-frr/src/ser/mod.rs | 23 +++-
> proxmox-frr/src/ser/route_map.rs | 43 ++++++--
> 5 files changed, 166 insertions(+), 13 deletions(-)
>
>
> pve-network:
>
> Stefan Hanreich (9):
> evpn controller: make nodes configurable
> evpn controller: allow multiple evpn controllers in a cluster
> evpn controller: add bgp-mode setting
> evpn zone: add secondary-controllers and rt filtering
> evpn controller: add ebgp-multihop setting
> test: evpn: add test for ibgp + ebgp evpn controller
> test: evpn: add legacy test
> tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session
> tests: test route filtering mechanism with multiple zones/controllers
>
> src/PVE/API2/Network/SDN/Controllers.pm | 13 ++
> src/PVE/API2/Network/SDN/Zones.pm | 11 +
> src/PVE/Network/SDN.pm | 1 +
> src/PVE/Network/SDN/Controllers.pm | 71 +++++++
> src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 22 +-
> src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++---
> src/PVE/Network/SDN/Controllers/Plugin.pm | 82 ++++++++
> src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 39 +++-
> .../expected_controller_config | 87 ++++++++
> .../expected_sdn_interfaces | 41 ++++
> .../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces | 11 +
> .../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config | 59 ++++++
> .../expected_controller_config | 66 ++++++
> .../expected_sdn_interfaces | 41 ++++
> .../evpn_cluster_ibgp_uplink_ebgp/interfaces | 7 +
> .../evpn_cluster_ibgp_uplink_ebgp/sdn_config | 51 +++++
> .../expected_controller_config | 78 ++++++++
> .../expected_sdn_interfaces | 81 ++++++++
> .../interfaces | 7 +
> .../sdn_config | 67 +++++++
> .../legacy_mode/expected_controller_config | 61 ++++++
> .../evpn/legacy_mode/expected_sdn_interfaces | 41 ++++
> src/test/zones/evpn/legacy_mode/interfaces | 7 +
> src/test/zones/evpn/legacy_mode/sdn_config | 51 +++++
> 24 files changed, 1143 insertions(+), 40 deletions(-)
> create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config
> create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces
> create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces
> create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config
> create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config
> create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
> create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces
> create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config
> create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config
> create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
> create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces
> create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config
> create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config
> create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces
> create mode 100644 src/test/zones/evpn/legacy_mode/interfaces
> create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config
>
>
> pve-manager:
>
> Stefan Hanreich (1):
> sdn: evpn: zone: controller: add new advanced fields
>
> www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++
> www/manager6/sdn/zones/EvpnEdit.js | 14 ++++++++-
> 2 files changed, 52 insertions(+), 1 deletion(-)
>
>
> pve-docs:
>
> Stefan Hanreich (2):
> sdn: evpn: document new zone / controller options
> sdn: fix typo in bgp controller
>
> pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 70 insertions(+), 4 deletions(-)
>
>
> Summary over all repositories:
> 32 files changed, 1431 insertions(+), 58 deletions(-)
>