Re: [RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality

[Hannes Laimer <h.laimer@proxmox.com>]

This looks very good to me!
The cover-letter is very well written and is very helpful for giving
context, an overview and reasoning for the changes.

I also did some basic testing, nothing too extensive, but I didn't
notice anything off.

With my comment on path 06/16 addressed consider this

Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>


note: FRR does allow things like `4200000001:1000`, and that
wouldn't fit into the `u16`. FRR does accept[1] these 4-octet[2] ASs,
expanding the u16 to a u32 in `ExtCommunityRouteTarget` should be
enough. There shouldn't be much of a downside (I think)

[1]
https://github.com/FRRouting/frr/blob/3b419ef76b8c0dfe8eb47097955f673f462c05e3/bgpd/bgp_ecommunity.c#L807
[2] https://www.rfc-editor.org/rfc/rfc5668


On 2026-04-14 18:33, Stefan Hanreich wrote:
> This patch series is based on the route-maps / prefix-lists series [1].
> 
> ## Introduction
> 
> This patch series lifts several existing restrictions on the EVPN controller:
> 
> * Allow defining multiple EVPN controllers in a cluster (which is effectively
>   defining multiple EVPN peer-groups in the FRR configuration)
> * Allow restricting EVPN controllers to specific nodes
> * Allow overriding the type of BGP session for EVPN controllers
> 
> 
> ## Motivation
> 
> The functionality implemented in this patch series allows exit nodes to act
> essentially as a ASBR router for the whole cluster or, alternatively, certain
> PVE nodes as route-server. The core motivation behind this patch series is that
> enables the possibility to define a second EVPN controller that handles
> advertising routes to another AS, independent of the intra-AS peering. This is
> particularly useful with the route-maps / prefix-lists series that allows
> attaching route-maps to EVPN controllers. That way users can create route-maps
> for the EVPN controller that connects to another AS (filtering routes,
> manipulating next-hops, ..). This patch series enables the PVE stack to
> implement setups similar to the Inter-AS Option B/C (with full-mesh peering
> inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the
> VPNv4 AF) [2].
> 
> Due to the modular nature of all the components involved, it is possible to
> replace every component in a complex setup with a physical router / route
> reflector / ... - giving users flexibility in how they want to build their
> network. By providing virtual equivalents of every critical component, it is
> also possible for us to create and manage a DCI EVPN cluster completely via the
> Proxmox VE stack and its API. Users can still choose to use physical equivalents
> - with the Proxmox VE stack only handling announcing the routes for an EVPN
> zone.
> 
> By implementing and providing this functionality via the API, there is less need
> for users to run their own hand-crafted frr.conf[.local] files for implementing
> different types of EVPN setups or resort to running custom, external, FRR
> instances for mangling routes and then re-advertising them to an uplink. In
> addition, this API can be used internally or by other products (e.g. PDM) to
> setup more complex setups automatically, but afterwards still provide users with
> the ability to customize the pre-generated setups, since a lot of options are
> more explicit now.
> 
> 
> ## Future Work
> 
> Future work should additionally allow specific PVE nodes to act as route
> reflectors for the cluster. This is similar to how e.g. Calico implements route
> reflectors [3]. This would make it possible to have the whole route handling
> manageable via the Proxmox VE stack, without the scaling issues attached to
> full-mesh peering and without the nodes itself having to be in the data path.
> This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with
> the PVE exit nodes acting as RRs and either the PVE node or an external router
> acting as ASBR. Inter-AS peering between the Route Reflectors of different
> clusters in the Option C could then happen directly e.g. via the Wireguard
> fabric and/or exchanging the loopback IPs of the RRs via the ASBRs.
> 
> This patch series adds basic documentation to the newly introduced options. With
> the addition of this and other recent patch series, a lot of possibilities for
> configuring the EVPN stack are exposed to users, that can be leveraged to create
> sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage
> the new features and which setups are recommended by us, we will create
> documentation for example setups and. Since this depends not only on this patch
> series and the progress we make in the coming weeks, they will be added as a
> separate patch series - if everything goes as planned - depending on the state
> of the EVPN stack and the outstanding patch series, as well as the progress we
> make on PDM side.
> 
> 
> ## Discussion / Open questions for Reviewers
> 
> A primary controller still needs to be assigned to an EVPN zone, as that is
> required for generating the VTEP IPs as well as auto-generating the MTU for the
> VNet. This means that the controller responsible for announcing the EVPN routes
> *inside* the cluster needs to be selected, in order for the SDN stack to
> generate correct configurations. This could be a bit confusing to users if they
> want to utilize multiple controllers, but aren't sure which one to pick as their
> 'primary' controller. Simply allowing users to configure multiple EVPN
> controllers for a zone doesn't work - since we cannot know which controller to
> use for auto-generating VTEP IPs.
> 
> The 'legacy' mode might better be called 'Auto', since it automatically inherits
> the BGP session from a given BGP controller. This could also be used in
> conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP
> session established there, while still giving users the possibility to run iBGP
> sessions instead.
> 
> 
> ## Backwards Compatibility
> 
> Great care has been taken to ensure that existing EVPN controllers / zones and
> BGP controller still work the same as before, every existing test case still
> works the same without any adjustments. This is possible because existing SDN
> configs should just be a subset of the now possible EVPN configurations. The new
> capabilities can only be used with completely new configuration options that
> need to be set explicitly. Additionally, when generating the FRR configuration,
> the case of only one EVPN controller existing is special-cased and generating
> filtering of outgoing routes does not take place then.
> 
> Backwards Compatibility has been verified via the integration tests, new
> additional integration tests as well as the e2e tests.
> 
> 
> ## FRR issues
> 
> This series suffers from a bug in the FRR version that is currently contained in
> our repositories (10.4.1), but has been fixed in a subsequent release (10.6).
> When leaking routes into the default routing table (which happens when defining
> PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get
> assigned the wrong ASN (1). This breaks the route filtering implemented in this
> patch series when multiple EVPN controllers are configured. Existing EVPN setups
> are *not* affected by this, since this has been special cased and no route
> filtering takes place.
> 
> The bug should be addressed before merging this patch series, either by
> backporting the fix or upgrading the existing FRR version.
> 
> 
> ## Dependencies
> 
> proxmox-frr depends on proxmox-frr-templates
> proxmox-ve-config depends on proxmox-frr
> proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version
> pve-network depends on proxmox-perl-rs
> pve-manager depends on pve-network
> 
> 
> [1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/
> [2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10
> [3] www.tigera.io/blog/configuring-route-reflectors-in-calico/
> 
> 
> proxmox-ve-rs:
> 
> Stefan Hanreich (4):
>   frr: add local-as setting
>   frr: add support for extcommunity lists
>   frr-templates: render local-as setting
>   frr-templates: render community lists in templates
> 
>  .../templates/bgp_router.jinja                |   4 +
>  proxmox-frr-templates/templates/bgpd.jinja    |   6 +
>  proxmox-frr/src/ser/bgp.rs                    | 103 +++++++++++++++++-
>  proxmox-frr/src/ser/mod.rs                    |  23 +++-
>  proxmox-frr/src/ser/route_map.rs              |  43 ++++++--
>  5 files changed, 166 insertions(+), 13 deletions(-)
> 
> 
> pve-network:
> 
> Stefan Hanreich (9):
>   evpn controller: make nodes configurable
>   evpn controller: allow multiple evpn controllers in a cluster
>   evpn controller: add bgp-mode setting
>   evpn zone: add secondary-controllers and rt filtering
>   evpn controller: add ebgp-multihop setting
>   test: evpn: add test for ibgp + ebgp evpn controller
>   test: evpn: add legacy test
>   tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session
>   tests: test route filtering mechanism with multiple zones/controllers
> 
>  src/PVE/API2/Network/SDN/Controllers.pm       |  13 ++
>  src/PVE/API2/Network/SDN/Zones.pm             |  11 +
>  src/PVE/Network/SDN.pm                        |   1 +
>  src/PVE/Network/SDN/Controllers.pm            |  71 +++++++
>  src/PVE/Network/SDN/Controllers/BgpPlugin.pm  |  22 +-
>  src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++---
>  src/PVE/Network/SDN/Controllers/Plugin.pm     |  82 ++++++++
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       |  39 +++-
>  .../expected_controller_config                |  87 ++++++++
>  .../expected_sdn_interfaces                   |  41 ++++
>  .../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces   |  11 +
>  .../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config   |  59 ++++++
>  .../expected_controller_config                |  66 ++++++
>  .../expected_sdn_interfaces                   |  41 ++++
>  .../evpn_cluster_ibgp_uplink_ebgp/interfaces  |   7 +
>  .../evpn_cluster_ibgp_uplink_ebgp/sdn_config  |  51 +++++
>  .../expected_controller_config                |  78 ++++++++
>  .../expected_sdn_interfaces                   |  81 ++++++++
>  .../interfaces                                |   7 +
>  .../sdn_config                                |  67 +++++++
>  .../legacy_mode/expected_controller_config    |  61 ++++++
>  .../evpn/legacy_mode/expected_sdn_interfaces  |  41 ++++
>  src/test/zones/evpn/legacy_mode/interfaces    |   7 +
>  src/test/zones/evpn/legacy_mode/sdn_config    |  51 +++++
>  24 files changed, 1143 insertions(+), 40 deletions(-)
>  create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config
>  create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces
>  create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config
>  create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config
>  create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces
>  create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config
>  create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config
>  create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces
>  create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config
>  create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config
>  create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/legacy_mode/interfaces
>  create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config
> 
> 
> pve-manager:
> 
> Stefan Hanreich (1):
>   sdn: evpn: zone: controller: add new advanced fields
> 
>  www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++
>  www/manager6/sdn/zones/EvpnEdit.js       | 14 ++++++++-
>  2 files changed, 52 insertions(+), 1 deletion(-)
> 
> 
> pve-docs:
> 
> Stefan Hanreich (2):
>   sdn: evpn: document new zone / controller options
>   sdn: fix typo in bgp controller
> 
>  pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 70 insertions(+), 4 deletions(-)
> 
> 
> Summary over all repositories:
>   32 files changed, 1431 insertions(+), 58 deletions(-)
>