From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A59751FF0C0 for ; Wed, 21 Oct 2026 18:13:48 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 35246215CA; Wed, 21 Oct 2026 18:13:46 +0200 (CEST) Message-ID: <7d731bdf-dbc3-465e-b8ba-ce6531c8ee41@proxmox.com> Date: Wed, 21 Oct 2026 18:13:40 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC manager/network/proxmox{-ve-rs,-perl-rs} 00/15] SDN VRF support To: pve-devel@lists.proxmox.com References: <20260821140404.322081-1-g.goller@proxmox.com> Content-Language: en-US From: Stefan Hanreich In-Reply-To: <20260821140404.322081-1-g.goller@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.353 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: XCI4L77NOTLPFCLYOD77SAEMKNYPUBBM X-Message-ID-Hash: XCI4L77NOTLPFCLYOD77SAEMKNYPUBBM X-MailFrom: s.hanreich@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Ran this on a test cluster: * tested OSPF in a simple zone VRF * tested BGP in a simple zone VRF * tested DHCP in a simple zone with VRF * tried SNAT (see below) for a simple zone with custom VRF Found the following issues: * status reporting for OSPF interfaces/neighbors is broken if a VRF is assigned: error parsing ospf interfaces: missing field `interfaces` JSON seems to be nested an addtional level * UI for creating the simple zone seems ugly. I'd invert the logic, move the option to advanced and call it something snappier like 'dedicated VRF'. ------------------ SNAT doesn't currently work and I think there are two solutions that both have use cases: Either we SNAT the traffic via the default routing table or we add the possibility of moving a dedicated uplink into a VRF. The following PoC for a dedicated uplink worked on my setup: * add an uplink that is a separate IF (physical, vlan, whatever, ...) * enslave it to the VRF * define a default GW inside the VRF via that IF * enable forwarding for the bridge / uplink * create an SNAT rule accordingly with source / outgoing IF of the uplink The stack needs to resolve the source IP and outgoing interface for SNAT via the VRF routing table rather than via the default routing table (as is currently the case). We should also match on the input interface in that case, the bridge, as subnets are not unique anymore... This is nice, because then the zone stays completely isolated. Routes inside the VRF are reachable, routes from the default routing table aren't. This should be imo quite straight forward to implement and would be quite nice for a v3. The other alternative is utilizing the uplink from the default routing table and ct zones + fw marks (this is a simple poc!): Set ct original zone for outgoing packets from the vnet: chain prerouting { type filter hook prerouting priority raw; policy accept; iifname "simple" ct original zone set 67 } SNAT out via the default routing table (that'd require some other matcher than ip saddr if implemented): chain postrouting { type nat hook postrouting priority srcnat; policy accept; ip saddr 192.0.2.0/24 oif vmbr0 snat to 203.0.113.1 } Restore ct original zone as fwmark for incoming traffic: chain preroutingmangle { type filter hook prerouting priority mangle; policy accept; iifname "vmbr0" meta mark set ct original zone } Create an ip rule that jumps into the vrf for that fwmark: $ ip rule add fwmark 67 table 1001 priority 100 It's nice because everything can be SNATed out of the VRF via the *same* uplink - even with overlapping subnets the CT entries should never clash as they have a different zone. This allows reaching everything the default routing table can reach though. This is also an interesting option I've already explored for EVPN exit- node north/south traffic as it would allow us to NAT traffic from over- lapping subnets. EVPN has the additional problem that this would require a single (or a primary at least) exit-node or conntrack syncing. Would maybe keep that for a future series and implement it for both simple / evpn zones. We'd have to test it more imo and polish this PoC but that seems like a good follow-up to this series and a nice addition for EVPN. I'd see both options as complementary rather than either/or. Alternatively, we could also double-NAT from the VRFs to the default VRF via veths by assigning each VRF a CGNAT or link-local IP, that would skip the whole fwmark/ip rule thing, but seems clunky and breaks some protocols/applications. Just mentioning it here for completeness' sake. ------------------ So for the reasons above the following additions would be nice: * add possibility of managing static routes inside VRFs * set VRF of network interfaces We should also consider extending our status reporting for layer-3 zones to include routing tables now? But I'd see that as a follow-up. On 8/21/26 4:04 PM, Gabriel Goller wrote: > This RFC adds VRF support to SDN zones and fabrics. > > A simple zone can now use either the default VRF or a dedicated VRF. The > simple zone dialog has a checkbox for this and uses the default VRF by > default. If the checkbox is disabled, the zone creates a dedicated VRF named > `vrf_`. EVPN zones always create a dedicated VRF. BGP and OSPF > fabrics can select a zone and run inside its VRF. > > This is the second iteration. The first iteration added VRFs as a separate > SDN object with a distinct GUI subdirectory. This was scrapped because VRFs > are an implementation detail and should not be exposed on their own in the > GUI. Zones already provide a logical L3 building block and should map to VRFs > instead. > > Route leaking between VRFs partly depends on the following FRR change, so it > will come a bit later: https://github.com/FRRouting/frr/pull/22426 > > As always, thanks to Stefan Hanreich for his help. > > > proxmox-ve-rs: > > Gabriel Goller (5): > frr: add VRF-aware OSPF rendering > sdn: add zone references to OSPF and BGP fabrics > sdn: generate fabric routing configuration in zone VRFs > sdn: fix VRF route-map scoping and zone ID validation > tests: fabrics: add test for fabrics in VRFs > > proxmox-frr-templates/templates/fabricd.jinja | 2 +- > .../templates/interface.jinja | 6 +- > proxmox-frr-templates/templates/isisd.jinja | 2 +- > proxmox-frr-templates/templates/ospfd.jinja | 12 +- > .../templates/protocol_routemaps.jinja | 13 ++ > proxmox-frr/debian/control | 8 +- > proxmox-frr/src/ser/mod.rs | 8 + > proxmox-ve-config/src/sdn/fabric/frr.rs | 137 ++++++++---- > proxmox-ve-config/src/sdn/fabric/mod.rs | 30 ++- > .../src/sdn/fabric/section_config/fabric.rs | 26 ++- > .../sdn/fabric/section_config/protocol/bgp.rs | 11 +- > .../fabric/section_config/protocol/ospf.rs | 11 +- > .../tests/fabric/cfg/vrf/fabrics.cfg | 43 ++++ > .../tests/fabric/cfg/vrf/route-maps.cfg | 15 ++ > proxmox-ve-config/tests/fabric/helper.rs | 20 ++ > proxmox-ve-config/tests/fabric/main.rs | 207 +++++++++++++++++- > .../fabric/snapshots/fabric__vrf_pve.snap | 134 ++++++++++++ > 17 files changed, 627 insertions(+), 58 deletions(-) > create mode 100644 proxmox-ve-config/tests/fabric/cfg/vrf/fabrics.cfg > create mode 100644 proxmox-ve-config/tests/fabric/cfg/vrf/route-maps.cfg > create mode 100644 proxmox-ve-config/tests/fabric/snapshots/fabric__vrf_pve.snap > > > proxmox-perl-rs: > > Gabriel Goller (3): > pve-rs: fabrics: assign network interfaces to configured VRFs > pve-rs: fabrics: make per-fabric status queries VRF-aware > pve-rs: fabrics: include VRF routes in aggregate status > > pve-rs/src/bindings/sdn/fabrics.rs | 301 +++++++++++++++++++---------- > pve-rs/src/sdn/status.rs | 19 +- > 2 files changed, 207 insertions(+), 113 deletions(-) > > > pve-network: > > Gabriel Goller (5): > sdn: add optional VRFs for simple zones > sdn: allow fabrics to use simple zone VRFs > sdn: always generate EVPN zone VRFs > api: sdn: allow EVPN zones as fabric VRFs > api: sdn: disallow BGP fabrics in EVPN zone VRFs > > src/PVE/API2/Network/SDN/Fabrics/Fabric.pm | 50 ++++++++++++++++--- > .../API2/Network/SDN/Fabrics/FabricNode.pm | 18 +++++++ > src/PVE/API2/Network/SDN/Zones.pm | 33 ++++++++++++ > src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 2 +- > src/PVE/Network/SDN/Fabrics.pm | 29 ++++++++++- > src/PVE/Network/SDN/Zones.pm | 17 +++++-- > src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 35 +++++++------ > src/PVE/Network/SDN/Zones/Plugin.pm | 6 +++ > src/PVE/Network/SDN/Zones/SimplePlugin.pm | 20 +++++++- > .../evpn/vrf-empty/expected_sdn_interfaces | 6 +++ > src/test/zones/evpn/vrf-empty/interfaces | 2 + > src/test/zones/evpn/vrf-empty/sdn_config | 13 +++++ > .../simple/vrf-empty/expected_sdn_interfaces | 5 ++ > src/test/zones/simple/vrf-empty/interfaces | 2 + > src/test/zones/simple/vrf-empty/sdn_config | 12 +++++ > .../zones/simple/vrf/expected_sdn_interfaces | 12 +++++ > src/test/zones/simple/vrf/interfaces | 2 + > src/test/zones/simple/vrf/sdn_config | 17 +++++++ > 18 files changed, 253 insertions(+), 28 deletions(-) > create mode 100644 src/test/zones/evpn/vrf-empty/expected_sdn_interfaces > create mode 100644 src/test/zones/evpn/vrf-empty/interfaces > create mode 100644 src/test/zones/evpn/vrf-empty/sdn_config > create mode 100644 src/test/zones/simple/vrf-empty/expected_sdn_interfaces > create mode 100644 src/test/zones/simple/vrf-empty/interfaces > create mode 100644 src/test/zones/simple/vrf-empty/sdn_config > create mode 100644 src/test/zones/simple/vrf/expected_sdn_interfaces > create mode 100644 src/test/zones/simple/vrf/interfaces > create mode 100644 src/test/zones/simple/vrf/sdn_config > > > pve-manager: > > Gabriel Goller (2): > ui: sdn: add VRF zone selection for fabrics > ui: sdn: expose EVPN zones as fabric VRFs > > www/manager6/form/SDNZoneSelector.js | 16 +++++++++++++++- > www/manager6/sdn/fabrics/FabricEdit.js | 18 ++++++++++++++++++ > www/manager6/sdn/zones/SimpleEdit.js | 12 +++++++++++- > 3 files changed, 44 insertions(+), 2 deletions(-) > > > Summary over all repositories: > 40 files changed, 1131 insertions(+), 201 deletions(-) >