From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 4F8EB1FF136 for ; Mon, 04 May 2026 18:35:24 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 299F2C867; Mon, 4 May 2026 18:35:24 +0200 (CEST) Message-ID: <83466356-8ba5-47c0-9ac2-913ea7796274@proxmox.com> Date: Mon, 4 May 2026 18:35:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: superseded: [RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality To: pve-devel@lists.proxmox.com References: <20260414163315.419384-1-s.hanreich@proxmox.com> Content-Language: en-US From: Stefan Hanreich In-Reply-To: <20260414163315.419384-1-s.hanreich@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.845 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLACK 3 Contains an URL listed in the URIBL blacklist [sdn.pm] Message-ID-Hash: KMJYLX7I6UI3253FTO6UTWFPITQ7RAXV X-Message-ID-Hash: KMJYLX7I6UI3253FTO6UTWFPITQ7RAXV X-MailFrom: s.hanreich@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: https://lore.proxmox.com/pve-devel/20260504162501.425135-1-s.hanreich@proxmox.com/T/#t On 4/14/26 6:32 PM, Stefan Hanreich wrote: > This patch series is based on the route-maps / prefix-lists series [1]. > > ## Introduction > > This patch series lifts several existing restrictions on the EVPN controller: > > * Allow defining multiple EVPN controllers in a cluster (which is effectively > defining multiple EVPN peer-groups in the FRR configuration) > * Allow restricting EVPN controllers to specific nodes > * Allow overriding the type of BGP session for EVPN controllers > > > ## Motivation > > The functionality implemented in this patch series allows exit nodes to act > essentially as a ASBR router for the whole cluster or, alternatively, certain > PVE nodes as route-server. The core motivation behind this patch series is that > enables the possibility to define a second EVPN controller that handles > advertising routes to another AS, independent of the intra-AS peering. This is > particularly useful with the route-maps / prefix-lists series that allows > attaching route-maps to EVPN controllers. That way users can create route-maps > for the EVPN controller that connects to another AS (filtering routes, > manipulating next-hops, ..). This patch series enables the PVE stack to > implement setups similar to the Inter-AS Option B/C (with full-mesh peering > inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the > VPNv4 AF) [2]. > > Due to the modular nature of all the components involved, it is possible to > replace every component in a complex setup with a physical router / route > reflector / ... - giving users flexibility in how they want to build their > network. By providing virtual equivalents of every critical component, it is > also possible for us to create and manage a DCI EVPN cluster completely via the > Proxmox VE stack and its API. Users can still choose to use physical equivalents > - with the Proxmox VE stack only handling announcing the routes for an EVPN > zone. > > By implementing and providing this functionality via the API, there is less need > for users to run their own hand-crafted frr.conf[.local] files for implementing > different types of EVPN setups or resort to running custom, external, FRR > instances for mangling routes and then re-advertising them to an uplink. In > addition, this API can be used internally or by other products (e.g. PDM) to > setup more complex setups automatically, but afterwards still provide users with > the ability to customize the pre-generated setups, since a lot of options are > more explicit now. > > > ## Future Work > > Future work should additionally allow specific PVE nodes to act as route > reflectors for the cluster. This is similar to how e.g. Calico implements route > reflectors [3]. This would make it possible to have the whole route handling > manageable via the Proxmox VE stack, without the scaling issues attached to > full-mesh peering and without the nodes itself having to be in the data path. > This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with > the PVE exit nodes acting as RRs and either the PVE node or an external router > acting as ASBR. Inter-AS peering between the Route Reflectors of different > clusters in the Option C could then happen directly e.g. via the Wireguard > fabric and/or exchanging the loopback IPs of the RRs via the ASBRs. > > This patch series adds basic documentation to the newly introduced options. With > the addition of this and other recent patch series, a lot of possibilities for > configuring the EVPN stack are exposed to users, that can be leveraged to create > sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage > the new features and which setups are recommended by us, we will create > documentation for example setups and. Since this depends not only on this patch > series and the progress we make in the coming weeks, they will be added as a > separate patch series - if everything goes as planned - depending on the state > of the EVPN stack and the outstanding patch series, as well as the progress we > make on PDM side. > > > ## Discussion / Open questions for Reviewers > > A primary controller still needs to be assigned to an EVPN zone, as that is > required for generating the VTEP IPs as well as auto-generating the MTU for the > VNet. This means that the controller responsible for announcing the EVPN routes > *inside* the cluster needs to be selected, in order for the SDN stack to > generate correct configurations. This could be a bit confusing to users if they > want to utilize multiple controllers, but aren't sure which one to pick as their > 'primary' controller. Simply allowing users to configure multiple EVPN > controllers for a zone doesn't work - since we cannot know which controller to > use for auto-generating VTEP IPs. > > The 'legacy' mode might better be called 'Auto', since it automatically inherits > the BGP session from a given BGP controller. This could also be used in > conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP > session established there, while still giving users the possibility to run iBGP > sessions instead. > > > ## Backwards Compatibility > > Great care has been taken to ensure that existing EVPN controllers / zones and > BGP controller still work the same as before, every existing test case still > works the same without any adjustments. This is possible because existing SDN > configs should just be a subset of the now possible EVPN configurations. The new > capabilities can only be used with completely new configuration options that > need to be set explicitly. Additionally, when generating the FRR configuration, > the case of only one EVPN controller existing is special-cased and generating > filtering of outgoing routes does not take place then. > > Backwards Compatibility has been verified via the integration tests, new > additional integration tests as well as the e2e tests. > > > ## FRR issues > > This series suffers from a bug in the FRR version that is currently contained in > our repositories (10.4.1), but has been fixed in a subsequent release (10.6). > When leaking routes into the default routing table (which happens when defining > PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get > assigned the wrong ASN (1). This breaks the route filtering implemented in this > patch series when multiple EVPN controllers are configured. Existing EVPN setups > are *not* affected by this, since this has been special cased and no route > filtering takes place. > > The bug should be addressed before merging this patch series, either by > backporting the fix or upgrading the existing FRR version. > > > ## Dependencies > > proxmox-frr depends on proxmox-frr-templates > proxmox-ve-config depends on proxmox-frr > proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version > pve-network depends on proxmox-perl-rs > pve-manager depends on pve-network > > > [1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/ > [2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10 > [3] www.tigera.io/blog/configuring-route-reflectors-in-calico/ > > > proxmox-ve-rs: > > Stefan Hanreich (4): > frr: add local-as setting > frr: add support for extcommunity lists > frr-templates: render local-as setting > frr-templates: render community lists in templates > > .../templates/bgp_router.jinja | 4 + > proxmox-frr-templates/templates/bgpd.jinja | 6 + > proxmox-frr/src/ser/bgp.rs | 103 +++++++++++++++++- > proxmox-frr/src/ser/mod.rs | 23 +++- > proxmox-frr/src/ser/route_map.rs | 43 ++++++-- > 5 files changed, 166 insertions(+), 13 deletions(-) > > > pve-network: > > Stefan Hanreich (9): > evpn controller: make nodes configurable > evpn controller: allow multiple evpn controllers in a cluster > evpn controller: add bgp-mode setting > evpn zone: add secondary-controllers and rt filtering > evpn controller: add ebgp-multihop setting > test: evpn: add test for ibgp + ebgp evpn controller > test: evpn: add legacy test > tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session > tests: test route filtering mechanism with multiple zones/controllers > > src/PVE/API2/Network/SDN/Controllers.pm | 13 ++ > src/PVE/API2/Network/SDN/Zones.pm | 11 + > src/PVE/Network/SDN.pm | 1 + > src/PVE/Network/SDN/Controllers.pm | 71 +++++++ > src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 22 +- > src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++--- > src/PVE/Network/SDN/Controllers/Plugin.pm | 82 ++++++++ > src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 39 +++- > .../expected_controller_config | 87 ++++++++ > .../expected_sdn_interfaces | 41 ++++ > .../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces | 11 + > .../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config | 59 ++++++ > .../expected_controller_config | 66 ++++++ > .../expected_sdn_interfaces | 41 ++++ > .../evpn_cluster_ibgp_uplink_ebgp/interfaces | 7 + > .../evpn_cluster_ibgp_uplink_ebgp/sdn_config | 51 +++++ > .../expected_controller_config | 78 ++++++++ > .../expected_sdn_interfaces | 81 ++++++++ > .../interfaces | 7 + > .../sdn_config | 67 +++++++ > .../legacy_mode/expected_controller_config | 61 ++++++ > .../evpn/legacy_mode/expected_sdn_interfaces | 41 ++++ > src/test/zones/evpn/legacy_mode/interfaces | 7 + > src/test/zones/evpn/legacy_mode/sdn_config | 51 +++++ > 24 files changed, 1143 insertions(+), 40 deletions(-) > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config > create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/legacy_mode/interfaces > create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config > > > pve-manager: > > Stefan Hanreich (1): > sdn: evpn: zone: controller: add new advanced fields > > www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++ > www/manager6/sdn/zones/EvpnEdit.js | 14 ++++++++- > 2 files changed, 52 insertions(+), 1 deletion(-) > > > pve-docs: > > Stefan Hanreich (2): > sdn: evpn: document new zone / controller options > sdn: fix typo in bgp controller > > pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 70 insertions(+), 4 deletions(-) > > > Summary over all repositories: > 32 files changed, 1431 insertions(+), 58 deletions(-) >