From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id A0CD11FF165 for ; Thu, 17 Jul 2025 02:09:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 8910C1D6BC; Thu, 17 Jul 2025 02:10:24 +0200 (CEST) Message-ID: <4ce5111b-8112-4fcb-952a-6fc34acf8da4@proxmox.com> Date: Thu, 17 Jul 2025 02:10:16 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Proxmox VE development discussion , Gabriel Goller References: <20250716130837.585796-1-g.goller@proxmox.com> Content-Language: de-AT, en-US From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20250716130837.585796-1-g.goller@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.184 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: [pve-devel] applied-series: [PATCH access-control/cluster/docs/gui-tests/manager/network/proxmox{, -firewall, -ve-rs, -perl-rs, -widget-toolkit} v5 00/76] Add SDN Fabrics X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Am 16.07.25 um 15:07 schrieb Gabriel Goller: > Overview > ======== > > This series allows the user to easily use dynamic routing protocols such as > OpenFabric and OSPF in their clusters. It also integrates existing features, > such as Ceph with the new SDN fabrics feature to enable users simple > configuration of e.g. full-mesh Ceph clusters via the Web UI. > > This patch series adds the initial support for two routing protocols: > * OpenFabric > * OSPF > > In the future we plan on moving the existing IS-IS and BGP controllers into the > fabric structure. Christoph Heiss is also currently working on adding a new > Wireguard fabric, which can be combined with any other fabric types. This > feature allows layering different fabrics on top of each other, so adding > encryption to an existing fabric is as simple as just putting a Wireguard fabric > on top, or using a Wireguard fabric as the basis. > > Packages are available on sani: packages/sdn-fabrics-v4 > > > Implementation > ============== > > Every fabric consists of zero or more nodes, which themselves consist of zero or > more interfaces. Fabrics and nodes are modeled as different section config types > (which means two section types for each protocol), interfaces are an array > contained in a node section. > > For now, nodes in the fabric configuration always represent PVE nodes, but in > the future nodes could also represent external members of the fabric (e.g. in a > potential Wireguard fabric). An example use case for this would be securely > connecting PBS or PDM instances to the PVE cluster via Wireguard. > > Most of the functionality is implemented in Rust and exposed to the existing SDN > module via perlmod. This includes configuration reading / writing, FRR config > generation from the section config and API CRUD methods. Some functionality, > like digest matching and permission checking is still handled on the perl side, > due to the lack of facilities in Rust for that. > > > Configuration Format > -------------------- > > The whole configuration is now contained in just one configuration file > `/etc/pve/sdn/fabrics.cfg`. This makes handling the fabrics configuration easier > in many different areas: locking, digest calculation, validation. > > For every protocol there are two different section types (fabric and node). As > an example the two section types for OSPF are 'ospf_fabric' and 'ospf_node'. > > The ID of a fabric is a simple name, at most 8 alphanumeric characters since we > use it for generating network interfaces names with a prefix. This is analogous > to existing SDN entities, e.g. VNet. A node can only be uniquely identified by > its id (which is equivalent to the hostname of the node), as well as the > fabric_id. This is because a node can be part of multiple fabrics. > > An example how the configuration looks like for a full-mesh 3-node Openfabric > fabric called 'example': > > openfabric_fabric: example > csnp_interval 3 > hello_interval 3 > ip6_prefix 2001:db8::/64 > ip_prefix 192.0.2.0/24 > > openfabric_node: example_deadeye > interfaces name=eth1,ip=198.51.100.0/31 > interfaces name=eth2 > ip 192.0.2.1 > ip6 2001:db8::1 > > openfabric_node: example_pathfinder > interfaces name=eth1,ip=198.51.100.2/31 > interfaces name=eth2 > ip 192.0.2.2 > ip6 2001:db8::2 > > openfabric_node: example_raider > interfaces name=eth1,ip=198.51.100.4/31 > interfaces name=eth2 > ip 192.0.2.3 > ip6 2001:db8::3 > > We parse the configuration file flat in rust, and then afterwards split them > into Fabric / Node structs and store them hierarchically (fabric -> node) in a > dedicated FabricConfig struct. This struct provides the CRUD methods for > manipulating the FabricConfig safely as well as serializing the FabricConfig > back into its section config format. > > To prevent having to duplicate common properties for every protocol, we > introduced a generic (Fabric|Node)Section struct that contains all common > properties. Protocol-specific properties can be defined by the generic type > parameter T. This saves us from duplicating a lot of code (which was an initial > problem with the intermediate configuration) and conversions can be simplified > by providing a generic implementation that every protocol uses. > > This design also means that adding new protocols to the configuration is quite > straightforward: It is only required to add structs with the protocol-specific > properties in Rust and add them to the enums defining the Section Config. The > commit adding OSPF support shows how simple it is to add a new protocol. > > > Validation > ---------- > > The hierarchical nature of the configuration and the relationship between nodes > inside the fabrics requires validation of sections relative to other sections. > For this matter we introduced a new Validatable trait as well as a struct that > wraps valid configuration in a Valid struct. For more information on that see > the respective commit. > > > API & Permissions > ----------------- > > The whole API is contained in the /cluster/sdn/fabrics subfolder and contains > submodules for fabric / node. > > A quick overview of the methods provided by the API: > > GET /all - list fabrics & nodes > > GET /fabric - list all fabrics > POST /fabric - create a fabric > GET /fabric/{fabric_id} - get a single fabric > PUT /fabric/{fabric_id} - update a fabric > DELETE /fabric/{fabric_id} - delete a fabric > > GET /node - list all nodes (regardless of fabric) > > GET /node/{fabric_id} - list all nodes belonging to fabric {fabric_id} > POST /node/{fabric_id} - create a node in fabric {fabric_id} > GET /node/{fabric_id}/{node_id} - get a single node > PUT /node/{fabric_id}/{node_id} - update a single node > DELETE /node/{fabric_id}/{node_id} - delete a single node > > > FRR Configuration > ----------------- > > For the FRR-specific functionality we introduced a new proxmox-frr crate that > models the different entities in the FRR configuration format (routers, > interfaces, route-maps, ...) and provides serializers for those structs. For > more information see the respective FRR commits. When applying the SDN > configuration, perl calls into perlmod to utilize the proxmox-frr crate for > generating the FRR configuration of the fabrics. > > We also introduce a proxmox-sdn-types crate, where we extracted generic > fabric types (e.g., openfabric::HelloInterval), so we can reuse them across > multiple crates (proxmox-frr, proxmox-ve-config, ..). > > > UI > -- > > The UI allows users to easily create different types of fabrics. One can add > Nodes to the fabrics by selecting them from a dropdown which shows all the nodes > in the cluster. Additionally the user can then select the interfaces of the node > which should be added to the fabric. There are also protocol-specific options > such as "passive", "hello-interval" etc. available to select on the interface. > There are also options spanning whole fabrics: the "hello-interval" option on > openfabric for example, can be set on the fabric and will be applied to every > interface. > > We are also working on the integration of status reporting into the sidebar, > which also includes an integration into pvestatd. The plan is to show the status > for each node, which routes are learned, neighbor status and possibly topology > from the POV of each node. Since this patch series is already quite huge and the > sidebar integration is still a work in progress it is not included here. > > Integration with existing features > ---------------------------------- > > We also provide a UI for the Ceph, Migration Network, VXLAN zone and EVPN > controller integrations. Users can configure fabrics for those components simply > by selecting them from a dropdown, providing a streamlined experience and nice > integration with existing features. > > > Refactoring > =========== > > This patch series required some rework of existing functionality, mostly how SDN > generates the FRR configuration and writes /etc/network/interfaces. Prior the > FRR configuration was generated exclusively from the controllers, but fabrics > need to write it as well. Same goes for the interfaces file, which got written > by the Zone plugin, but Fabrics need to write this file as well. > > For this we moved the FRR and ifupdown config generation one level up to the SDN > module, which now calls into the respective child modules to generate the FRR / > ifupdown configuration. > > > Dependencies > ============ > This series relies on the FRR 10.2.2 backport series, since it fixes potential > issues with EVPN + Openfabric/OSPF: > > https://lore.proxmox.com/all/20250418112114.2747673-1-s.hanreich@proxmox.com/ > > > proxmox-frr depends on proxmox-network-types > proxmox-frr depends on proxmox-sdn-types > > proxmox-ve-config depends on proxmox-frr > proxmox-ve-config depends on proxmox-network-types > proxmox-ve-config depends on proxmox-sdn-types > proxmox-ve-config depends on proxmox-serde > proxmox-ve-config depends on proxmox-api-macro > > proxmox-firewall depends on proxmox-ve-config > > proxmox-perl-rs depends on proxmox-ve-config > proxmox-perl-rs depends on proxmox-frr > proxmox-perl-rs depends on proxmox-network-types > > pve-network depends on proxmox-perl-rs > pve-network depends on pve-cluster > pve-network depends on pve-access-control > > pve-docs depends on pve-gui-tests > > pve-manager depends on proxmox-widget-toolkit > pve-manager depends on pve-docs > pve-manager depends on pve-network > pve-manager depends on pve-access-control > > > pve-network commits 4-7 do not build independently, because it's one refactor > but split across multiple commits so it's easier to follow the steps during the > refactor. We could consider squashing those commits on applying, so each commit > still builds indepedently. > > Shoutout to Stefan for his great work on this patch series! > Talked with Hannes quickly before he left and in his opinion he saw no big blocker, so applied series, *huge* thanks to all involved, nice work and great details in the patch submission! While I'm certain things can still be polished anything, that's (more) true for basically everything else, and doing t will need actual user feedback anyway. The screenshots are still missing from the docs, I commented the thumbnail references out for now. It would be great if you could push a commit adding them to your staff repo. Tiny UX nits: For the UI it might be nice to add some emptyTexts in the input fields of the add/edit dialogue. Making those windows a bit wider wouldn't hurt either IMO. Some simple how-to's for the wiki for some real world use cases might be nice too, but maybe it's best to wait on some user feedback and get in other in progress stuff here. _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel