From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 4A0E41FF13C for ; Thu, 16 Apr 2026 18:23:14 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C5F7EC35C; Thu, 16 Apr 2026 18:23:12 +0200 (CEST) Message-ID: <0fa5059a-4c0c-4f46-a351-025c10dd9e2a@proxmox.com> Date: Thu, 16 Apr 2026 18:22:35 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC docs/manager/network/proxmox-ve-rs 00/16] Extend EVPN controller functionality To: Stefan Hanreich , pve-devel@lists.proxmox.com References: <20260414163315.419384-1-s.hanreich@proxmox.com> Content-Language: en-US From: Hannes Laimer In-Reply-To: <20260414163315.419384-1-s.hanreich@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776356477048 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.082 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [sdn.pm,tigera.io,controllers.pm,bgpplugin.pm,bgp.rs,zones.pm,proxmox.com,evpnplugin.pm,plugin.pm,rfc-editor.org,mod.rs] Message-ID-Hash: NAUIAN7AQRAVY7RK25KX7LVTNFCKO4SM X-Message-ID-Hash: NAUIAN7AQRAVY7RK25KX7LVTNFCKO4SM X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This looks very good to me! The cover-letter is very well written and is very helpful for giving context, an overview and reasoning for the changes. I also did some basic testing, nothing too extensive, but I didn't notice anything off. With my comment on path 06/16 addressed consider this Reviewed-by: Hannes Laimer note: FRR does allow things like `4200000001:1000`, and that wouldn't fit into the `u16`. FRR does accept[1] these 4-octet[2] ASs, expanding the u16 to a u32 in `ExtCommunityRouteTarget` should be enough. There shouldn't be much of a downside (I think) [1] https://github.com/FRRouting/frr/blob/3b419ef76b8c0dfe8eb47097955f673f462c05e3/bgpd/bgp_ecommunity.c#L807 [2] https://www.rfc-editor.org/rfc/rfc5668 On 2026-04-14 18:33, Stefan Hanreich wrote: > This patch series is based on the route-maps / prefix-lists series [1]. > > ## Introduction > > This patch series lifts several existing restrictions on the EVPN controller: > > * Allow defining multiple EVPN controllers in a cluster (which is effectively > defining multiple EVPN peer-groups in the FRR configuration) > * Allow restricting EVPN controllers to specific nodes > * Allow overriding the type of BGP session for EVPN controllers > > > ## Motivation > > The functionality implemented in this patch series allows exit nodes to act > essentially as a ASBR router for the whole cluster or, alternatively, certain > PVE nodes as route-server. The core motivation behind this patch series is that > enables the possibility to define a second EVPN controller that handles > advertising routes to another AS, independent of the intra-AS peering. This is > particularly useful with the route-maps / prefix-lists series that allows > attaching route-maps to EVPN controllers. That way users can create route-maps > for the EVPN controller that connects to another AS (filtering routes, > manipulating next-hops, ..). This patch series enables the PVE stack to > implement setups similar to the Inter-AS Option B/C (with full-mesh peering > inside the AS instead of RRs) from RFC 4364 based on the EVPN AF (instead of the > VPNv4 AF) [2]. > > Due to the modular nature of all the components involved, it is possible to > replace every component in a complex setup with a physical router / route > reflector / ... - giving users flexibility in how they want to build their > network. By providing virtual equivalents of every critical component, it is > also possible for us to create and manage a DCI EVPN cluster completely via the > Proxmox VE stack and its API. Users can still choose to use physical equivalents > - with the Proxmox VE stack only handling announcing the routes for an EVPN > zone. > > By implementing and providing this functionality via the API, there is less need > for users to run their own hand-crafted frr.conf[.local] files for implementing > different types of EVPN setups or resort to running custom, external, FRR > instances for mangling routes and then re-advertising them to an uplink. In > addition, this API can be used internally or by other products (e.g. PDM) to > setup more complex setups automatically, but afterwards still provide users with > the ability to customize the pre-generated setups, since a lot of options are > more explicit now. > > > ## Future Work > > Future work should additionally allow specific PVE nodes to act as route > reflectors for the cluster. This is similar to how e.g. Calico implements route > reflectors [3]. This would make it possible to have the whole route handling > manageable via the Proxmox VE stack, without the scaling issues attached to > full-mesh peering and without the nodes itself having to be in the data path. > This would fully enable Inter-AS Option B & C as defined in RFC 4364 [2] with > the PVE exit nodes acting as RRs and either the PVE node or an external router > acting as ASBR. Inter-AS peering between the Route Reflectors of different > clusters in the Option C could then happen directly e.g. via the Wireguard > fabric and/or exchanging the loopback IPs of the RRs via the ASBRs. > > This patch series adds basic documentation to the newly introduced options. With > the addition of this and other recent patch series, a lot of possibilities for > configuring the EVPN stack are exposed to users, that can be leveraged to create > sophisticated {intra,inter}-AS setups. To give users an idea on how to leverage > the new features and which setups are recommended by us, we will create > documentation for example setups and. Since this depends not only on this patch > series and the progress we make in the coming weeks, they will be added as a > separate patch series - if everything goes as planned - depending on the state > of the EVPN stack and the outstanding patch series, as well as the progress we > make on PDM side. > > > ## Discussion / Open questions for Reviewers > > A primary controller still needs to be assigned to an EVPN zone, as that is > required for generating the VTEP IPs as well as auto-generating the MTU for the > VNet. This means that the controller responsible for announcing the EVPN routes > *inside* the cluster needs to be selected, in order for the SDN stack to > generate correct configurations. This could be a bit confusing to users if they > want to utilize multiple controllers, but aren't sure which one to pick as their > 'primary' controller. Simply allowing users to configure multiple EVPN > controllers for a zone doesn't work - since we cannot know which controller to > use for auto-generating VTEP IPs. > > The 'legacy' mode might better be called 'Auto', since it automatically inherits > the BGP session from a given BGP controller. This could also be used in > conjunction with the unnumbered BGP fabric - enabling easy re-use of the BGP > session established there, while still giving users the possibility to run iBGP > sessions instead. > > > ## Backwards Compatibility > > Great care has been taken to ensure that existing EVPN controllers / zones and > BGP controller still work the same as before, every existing test case still > works the same without any adjustments. This is possible because existing SDN > configs should just be a subset of the now possible EVPN configurations. The new > capabilities can only be used with completely new configuration options that > need to be set explicitly. Additionally, when generating the FRR configuration, > the case of only one EVPN controller existing is special-cased and generating > filtering of outgoing routes does not take place then. > > Backwards Compatibility has been verified via the integration tests, new > additional integration tests as well as the e2e tests. > > > ## FRR issues > > This series suffers from a bug in the FRR version that is currently contained in > our repositories (10.4.1), but has been fixed in a subsequent release (10.6). > When leaking routes into the default routing table (which happens when defining > PVE nodes as exit nodes), the Route Targets for EVPN routes sometimes get > assigned the wrong ASN (1). This breaks the route filtering implemented in this > patch series when multiple EVPN controllers are configured. Existing EVPN setups > are *not* affected by this, since this has been special cased and no route > filtering takes place. > > The bug should be addressed before merging this patch series, either by > backporting the fix or upgrading the existing FRR version. > > > ## Dependencies > > proxmox-frr depends on proxmox-frr-templates > proxmox-ve-config depends on proxmox-frr > proxmox-perl-rs needs to be bumped with the new proxmox-ve-config version > pve-network depends on proxmox-perl-rs > pve-manager depends on pve-network > > > [1] https://lore.proxmox.com/all/20260401143957.386809-1-s.hanreich@proxmox.com/ > [2] https://www.rfc-editor.org/rfc/rfc4364.html#section-10 > [3] www.tigera.io/blog/configuring-route-reflectors-in-calico/ > > > proxmox-ve-rs: > > Stefan Hanreich (4): > frr: add local-as setting > frr: add support for extcommunity lists > frr-templates: render local-as setting > frr-templates: render community lists in templates > > .../templates/bgp_router.jinja | 4 + > proxmox-frr-templates/templates/bgpd.jinja | 6 + > proxmox-frr/src/ser/bgp.rs | 103 +++++++++++++++++- > proxmox-frr/src/ser/mod.rs | 23 +++- > proxmox-frr/src/ser/route_map.rs | 43 ++++++-- > 5 files changed, 166 insertions(+), 13 deletions(-) > > > pve-network: > > Stefan Hanreich (9): > evpn controller: make nodes configurable > evpn controller: allow multiple evpn controllers in a cluster > evpn controller: add bgp-mode setting > evpn zone: add secondary-controllers and rt filtering > evpn controller: add ebgp-multihop setting > test: evpn: add test for ibgp + ebgp evpn controller > test: evpn: add legacy test > tests: evpn: force ibgp over ebgp bgp controller with ebgp wan session > tests: test route filtering mechanism with multiple zones/controllers > > src/PVE/API2/Network/SDN/Controllers.pm | 13 ++ > src/PVE/API2/Network/SDN/Zones.pm | 11 + > src/PVE/Network/SDN.pm | 1 + > src/PVE/Network/SDN/Controllers.pm | 71 +++++++ > src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 22 +- > src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 188 +++++++++++++++--- > src/PVE/Network/SDN/Controllers/Plugin.pm | 82 ++++++++ > src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 39 +++- > .../expected_controller_config | 87 ++++++++ > .../expected_sdn_interfaces | 41 ++++ > .../ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces | 11 + > .../ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config | 59 ++++++ > .../expected_controller_config | 66 ++++++ > .../expected_sdn_interfaces | 41 ++++ > .../evpn_cluster_ibgp_uplink_ebgp/interfaces | 7 + > .../evpn_cluster_ibgp_uplink_ebgp/sdn_config | 51 +++++ > .../expected_controller_config | 78 ++++++++ > .../expected_sdn_interfaces | 81 ++++++++ > .../interfaces | 7 + > .../sdn_config | 67 +++++++ > .../legacy_mode/expected_controller_config | 61 ++++++ > .../evpn/legacy_mode/expected_sdn_interfaces | 41 ++++ > src/test/zones/evpn/legacy_mode/interfaces | 7 + > src/test/zones/evpn/legacy_mode/sdn_config | 51 +++++ > 24 files changed, 1143 insertions(+), 40 deletions(-) > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/interfaces > create mode 100644 src/test/zones/evpn/ebgp_evpn_ibgp_wan_evpn_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/interfaces > create mode 100644 src/test/zones/evpn/evpn_cluster_ibgp_uplink_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_controller_config > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/interfaces > create mode 100644 src/test/zones/evpn/evpn_multiple_zones_cluster_ibgp_uplink_ebgp/sdn_config > create mode 100644 src/test/zones/evpn/legacy_mode/expected_controller_config > create mode 100644 src/test/zones/evpn/legacy_mode/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/legacy_mode/interfaces > create mode 100644 src/test/zones/evpn/legacy_mode/sdn_config > > > pve-manager: > > Stefan Hanreich (1): > sdn: evpn: zone: controller: add new advanced fields > > www/manager6/sdn/controllers/EvpnEdit.js | 39 ++++++++++++++++++++++++ > www/manager6/sdn/zones/EvpnEdit.js | 14 ++++++++- > 2 files changed, 52 insertions(+), 1 deletion(-) > > > pve-docs: > > Stefan Hanreich (2): > sdn: evpn: document new zone / controller options > sdn: fix typo in bgp controller > > pvesdn.adoc | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 70 insertions(+), 4 deletions(-) > > > Summary over all repositories: > 32 files changed, 1431 insertions(+), 58 deletions(-) >