[RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality

Stefan Hanreich
2026-04-14T16:35:07Z

This patch series is based on the route-maps / prefix-lists series [1].

## Introduction

This patch series lifts several existing restrictions on the EVPN controller:

* Allow defining multiple EVPN controllers in a cluster (which is effectively
defining multiple EVPN peer-groups in the FRR configuration)
* Allow restricting EVPN controllers to specific nodes
* Allow overriding the type of BGP session for EVPN controllers

## Motivation

The functionality implemented in this patch series allows exit nodes to act
essentially as a ASBR router for the whole cluster or, alternatively, certain
PVE nodes as route-server. The core motivation behind this patch series is that
enables the possibility to define a second EVPN controller that handles
advertising routes to another AS, independent of the intra-AS peering. This is
particularly useful with the route-maps / prefix-lists series that allows
attaching route-maps to EVPN controllers. That way users can create route-maps
for the EVPN controller that connects to another AS (filtering routes,
manipulating next-hops, ..). This patch series enables the PVE stack to
implement setups similar to the Inter-AS Option B/C (with full-mesh peering
inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the
VPNv4 AF) [2].

Due to the modular nature of all the components involved, it is possible to
replace every component in a complex setup with a physical router / route
reflector / ... - giving users flexibility in how they want to build their
network. By providing virtual equivalents of every critical component, it is
also possible for us to create and manage a DCI EVPN cluster completely via the
Proxmox VE stack and its API. Users can still choose to use physical equivalents
- with the Proxmox VE stack only handling announcing the routes for an EVPN
zone.

By implementing and providing this functionality via the API, there is less need
for users to run their own hand-crafted frr.conf[.local] files for implementing
different types of EVPN setups or resort to running custom, external, FRR
instances for mangling routes and then re-advertising them to an uplink. In
addition, this API can be used internally or by other products (e.g. PDM) to
setup more complex setups automatically, but afterwards still provide users with
the ability to customize the pre-generated setups, since a lot of options are
more explicit now.

## Future Work

Future work should additionally allow specific PVE nodes to act as route
reflectors for the cluster. This is similar to how e.g. Calico implements route
reflectors [3]. This would make it possible to have the whole route handling
manageable via the Proxmox VE stack, without the scaling issues attached to
full-mesh peering and without the nodes itself having to be in the data path.
This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with
the PVE exit nodes acting as RRs and either the PVE node or an external router
acting as ASBR. Inter-AS peering between the Route Reflectors of different
clusters in the Option C could then happen directly e.g. via the Wireguard
fabric and/or exchanging the loopback IPs of the RRs via the ASBRs.

This patch series adds basic documentation to the newly introduced options. With
the addition of this and other recent patch series, a lot of possibilities for
configuring the EVPN stack are exposed to users, that can be leveraged to create
sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage
the new features and which setups are recommended by us, we will create
documentation for example setups and. Since this depends not only on this patch
series and the progress we make in the coming weeks, they will be added as a
separate patch series - if everything goes as planned - depending on the state
of the EVPN stack and the outstanding patch series, as well as the progress we
make on PDM side.

## Discussion / Open questions for Reviewers

A primary controller still needs to be assigned to an EVPN zone, as that is
required for generating the VTEP IPs as well as auto-generating the MTU for the
VNet. This means that the controller responsible for announcing the EVPN routes
*inside* the cluster needs to be selected, in order for the SDN stack to
generate correct configurations. This could be a bit confusing to users if they
want to utilize multiple controllers, but aren't sure which one to pick as their
'primary' controller. Simply allowing users to configure multiple EVPN
controllers for a zone doesn't work - since we cannot know which controller to
use for auto-generating VTEP IPs.

The 'legacy' mode might better be called 'Auto', since it automatically inherits
the BGP session from a given BGP controller. This could also be used in
conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP
session established there, while still giving users the possibility to run iBGP
sessions instead.

## Backwards Compatibility

Great care has been taken to ensure that existing EVPN controllers / zones and
BGP controller still work the same as before, every existing test case still
works the same without any adjustments. This is possible because existing SDN
configs should just be a subset of the now possible EVPN configurations. The new
capabilities can only be used with completely new configuration options that
need to be set explicitly. Additionally, when generating the FRR configuration,
the case of only one EVPN controller existing is special-cased and generating
filtering of outgoing routes does not take place then.

Backwards Compatibility has been verified via the integration tests, new
additional integration tests as well as the e2e tests.

## FRR issues

This series suffers from a bug in the FRR version that is currently contained in
our repositories (10.4.1), but has been fixed in a subsequent release (10.6).
When leaking routes into the default routing table (which happens when defining
PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get
assigned the wrong ASN (1). This breaks the route filtering implemented in this
patch series when multiple EVPN controllers are configured. Existing EVPN setups
are *not* affected by this, since this has been special cased and no route
filtering takes place.

The bug should be addressed before merging this patch series, either by
backporting the fix or upgrading the existing FRR version.

## Dependencies

proxmox-frr depends on proxmox-frr-templates
proxmox-ve-config depends on proxmox-frr
proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version
pve-network depends on proxmox-perl-rs
pve-manager depends on pve-network

[1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/
[2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10
[3] www.tigera.io/blog/configuring-route-reflectors-in-calico/

proxmox-ve-rs:

Stefan Hanreich (4):
frr: add local-as setting
frr: add support for extcommunity lists
frr-templates: render local-as setting
frr-templates: render community lists in templates

.../templates/bgp_router.jinja | 4 +
proxmox-frr-templates/templates/bgpd.jinja | 6 +
proxmox-frr/src/ser/bgp.rs | 103 +++++++++++++++++-
proxmox-frr/src/ser/mod.rs | 23 +++-
proxmox-frr/src/ser/route_map.rs | 43 ++++++--
5 files changed, 166 insertions(+), 13 deletions(-)

pve-network:

Stefan Hanreich (9):
evpn controller: make nodes configurable
evpn controller: allow multiple evpn controllers in a cluster
evpn controller: add bgp-mode setting
evpn zone: add secondary-controllers and rt filtering
evpn controller: add ebgp-multihop setting
test: evpn: add test for ibgp + ebgp evpn controller
test: evpn: add legacy test
tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session
tests: test route filtering mechanism with multiple zones/controllers

src/PVE/API2/Network/SDN/Controllers.pm | 13 ++
src/PVE/API2/Network/SDN/Zones.pm | 11 +
src/PVE/Network/SDN.pm | 1 +
src/PVE/Network/SDN/Controllers.pm | 71 +++++++
src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 22 +-
src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++---
src/PVE/Network/SDN/Controllers/Plugin.pm | 82 ++++++++
src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 39 +++-
.../expected_controller_config | 87 ++++++++
.../expected_sdn_interfaces | 41 ++++
.../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces | 11 +
.../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config | 59 ++++++
.../expected_controller_config | 66 ++++++
.../expected_sdn_interfaces | 41 ++++
.../evpn_cluster_ibgp_uplink_ebgp/interfaces | 7 +
.../evpn_cluster_ibgp_uplink_ebgp/sdn_config | 51 +++++
.../expected_controller_config | 78 ++++++++
.../expected_sdn_interfaces | 81 ++++++++
.../interfaces | 7 +
.../sdn_config | 67 +++++++
.../legacy_mode/expected_controller_config | 61 ++++++
.../evpn/legacy_mode/expected_sdn_interfaces | 41 ++++
src/test/zones/evpn/legacy_mode/interfaces | 7 +
src/test/zones/evpn/legacy_mode/sdn_config | 51 +++++
24 files changed, 1143 insertions(+), 40 deletions(-)
create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config
create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces
create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config
create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config
create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces
create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config
create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config
create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces
create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config
create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config
create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/legacy_mode/interfaces
create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config

pve-manager:

Stefan Hanreich (1):
sdn: evpn: zone: controller: add new advanced fields

www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++
www/manager6/sdn/zones/EvpnEdit.js | 14 ++++++++-
2 files changed, 52 insertions(+), 1 deletion(-)

pve-docs:

Stefan Hanreich (2):
sdn: evpn: document new zone / controller options
sdn: fix typo in bgp controller

pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 70 insertions(+), 4 deletions(-)

Summary over all repositories:
32 files changed, 1431 insertions(+), 58 deletions(-)

--
Generated by murpp 0.11.0