Re: [RFC manager/network/proxmox{-ve-rs,-perl-rs} 00/15] SDN VRF support

[Stefan Hanreich <s.hanreich@proxmox.com>]

Ran this on a test cluster:

* tested OSPF in a simple zone VRF
* tested BGP in a simple zone VRF
* tested DHCP in a simple zone with VRF
* tried SNAT (see below) for a simple zone with custom VRF

Found the following issues:

* status reporting for OSPF interfaces/neighbors is broken if a VRF is
assigned:

   error parsing ospf interfaces: missing field `interfaces`

JSON seems to be nested an addtional level

* UI for creating the simple zone seems ugly. I'd invert the logic, move
the option to advanced and call it something snappier like 'dedicated VRF'.

------------------

SNAT doesn't currently work and I think there are two solutions that both
have use cases: Either we SNAT the traffic via the default routing table
or we add the possibility of moving a dedicated uplink into a VRF.

The following PoC for a dedicated uplink worked on my setup:

* add an uplink that is a separate IF (physical, vlan, whatever, ...)
* enslave it to the VRF
* define a default GW inside the VRF via that IF
* enable forwarding for the bridge / uplink
* create an SNAT rule accordingly with source / outgoing IF of the uplink

The stack needs to resolve the source IP and outgoing interface for
SNAT via the VRF routing table rather than via the default routing table
(as is currently the case). We should also match on the input interface
in that case, the bridge, as subnets are not unique anymore...

This is nice, because then the zone stays completely isolated. Routes inside
the VRF are reachable, routes from the default routing table aren't. This
should be imo quite straight forward to implement and would be quite nice
for a v3.



The other alternative is utilizing the uplink from the default routing
table and ct zones + fw marks (this is a simple poc!):

Set ct original zone for outgoing packets from the vnet:

    chain prerouting {
        type filter hook prerouting priority raw; policy accept;
        iifname "simple" ct original zone set 67
    }

SNAT out via the default routing table (that'd require some other
matcher than ip saddr if implemented):

    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 192.0.2.0/24 oif vmbr0 snat to 203.0.113.1
    }

Restore ct original zone as fwmark for incoming traffic:

    chain preroutingmangle {
        type filter hook prerouting priority mangle; policy accept;
        iifname "vmbr0" meta mark set ct original zone
    }

Create an ip rule that jumps into the vrf for that fwmark:

    $ ip rule add fwmark 67 table 1001 priority 100

It's nice because everything can be SNATed out of the VRF via the *same*
uplink - even with overlapping subnets the CT entries should never clash
as they have a different zone. This allows reaching everything the default
routing table can reach though.

This is also an interesting option I've already explored for EVPN exit-
node north/south traffic as it would allow us to NAT traffic from over-
lapping subnets. EVPN has the additional problem that this would require
a single (or a primary at least) exit-node or conntrack syncing. Would
maybe keep that for a future series and implement it for both simple / 
evpn zones. We'd have to test it more imo and polish this PoC but that
seems like a good follow-up to this series and a nice addition for EVPN.

I'd see both options as complementary rather than either/or.



Alternatively, we could also double-NAT from the VRFs to the default
VRF via veths by assigning each VRF a CGNAT or link-local IP, that would
skip the whole fwmark/ip rule thing, but seems clunky and breaks some
protocols/applications. Just mentioning it here for completeness' sake.

------------------

So for the reasons above the following additions would be nice:
* add possibility of managing static routes inside VRFs
* set VRF of network interfaces

We should also consider extending our status reporting for layer-3 zones
to include routing tables now? But I'd see that as a follow-up.


On 8/21/26 4:04 PM, Gabriel Goller wrote:
> This RFC adds VRF support to SDN zones and fabrics.
> 
> A simple zone can now use either the default VRF or a dedicated VRF. The
> simple zone dialog has a checkbox for this and uses the default VRF by
> default. If the checkbox is disabled, the zone creates a dedicated VRF named
> `vrf_<zone-name>`. EVPN zones always create a dedicated VRF. BGP and OSPF
> fabrics can select a zone and run inside its VRF.
> 
> This is the second iteration. The first iteration added VRFs as a separate
> SDN object with a distinct GUI subdirectory. This was scrapped because VRFs
> are an implementation detail and should not be exposed on their own in the
> GUI. Zones already provide a logical L3 building block and should map to VRFs
> instead.
> 
> Route leaking between VRFs partly depends on the following FRR change, so it
> will come a bit later: https://github.com/FRRouting/frr/pull/22426
> 
> As always, thanks to Stefan Hanreich for his help.
> 
> 
> proxmox-ve-rs:
> 
> Gabriel Goller (5):
>   frr: add VRF-aware OSPF rendering
>   sdn: add zone references to OSPF and BGP fabrics
>   sdn: generate fabric routing configuration in zone VRFs
>   sdn: fix VRF route-map scoping and zone ID validation
>   tests: fabrics: add test for fabrics in VRFs
> 
>  proxmox-frr-templates/templates/fabricd.jinja |   2 +-
>  .../templates/interface.jinja                 |   6 +-
>  proxmox-frr-templates/templates/isisd.jinja   |   2 +-
>  proxmox-frr-templates/templates/ospfd.jinja   |  12 +-
>  .../templates/protocol_routemaps.jinja        |  13 ++
>  proxmox-frr/debian/control                    |   8 +-
>  proxmox-frr/src/ser/mod.rs                    |   8 +
>  proxmox-ve-config/src/sdn/fabric/frr.rs       | 137 ++++++++----
>  proxmox-ve-config/src/sdn/fabric/mod.rs       |  30 ++-
>  .../src/sdn/fabric/section_config/fabric.rs   |  26 ++-
>  .../sdn/fabric/section_config/protocol/bgp.rs |  11 +-
>  .../fabric/section_config/protocol/ospf.rs    |  11 +-
>  .../tests/fabric/cfg/vrf/fabrics.cfg          |  43 ++++
>  .../tests/fabric/cfg/vrf/route-maps.cfg       |  15 ++
>  proxmox-ve-config/tests/fabric/helper.rs      |  20 ++
>  proxmox-ve-config/tests/fabric/main.rs        | 207 +++++++++++++++++-
>  .../fabric/snapshots/fabric__vrf_pve.snap     | 134 ++++++++++++
>  17 files changed, 627 insertions(+), 58 deletions(-)
>  create mode 100644 proxmox-ve-config/tests/fabric/cfg/vrf/fabrics.cfg
>  create mode 100644 proxmox-ve-config/tests/fabric/cfg/vrf/route-maps.cfg
>  create mode 100644 proxmox-ve-config/tests/fabric/snapshots/fabric__vrf_pve.snap
> 
> 
> proxmox-perl-rs:
> 
> Gabriel Goller (3):
>   pve-rs: fabrics: assign network interfaces to configured VRFs
>   pve-rs: fabrics: make per-fabric status queries VRF-aware
>   pve-rs: fabrics: include VRF routes in aggregate status
> 
>  pve-rs/src/bindings/sdn/fabrics.rs | 301 +++++++++++++++++++----------
>  pve-rs/src/sdn/status.rs           |  19 +-
>  2 files changed, 207 insertions(+), 113 deletions(-)
> 
> 
> pve-network:
> 
> Gabriel Goller (5):
>   sdn: add optional VRFs for simple zones
>   sdn: allow fabrics to use simple zone VRFs
>   sdn: always generate EVPN zone VRFs
>   api: sdn: allow EVPN zones as fabric VRFs
>   api: sdn: disallow BGP fabrics in EVPN zone VRFs
> 
>  src/PVE/API2/Network/SDN/Fabrics/Fabric.pm    | 50 ++++++++++++++++---
>  .../API2/Network/SDN/Fabrics/FabricNode.pm    | 18 +++++++
>  src/PVE/API2/Network/SDN/Zones.pm             | 33 ++++++++++++
>  src/PVE/Network/SDN/Controllers/BgpPlugin.pm  |  2 +-
>  src/PVE/Network/SDN/Fabrics.pm                | 29 ++++++++++-
>  src/PVE/Network/SDN/Zones.pm                  | 17 +++++--
>  src/PVE/Network/SDN/Zones/EvpnPlugin.pm       | 35 +++++++------
>  src/PVE/Network/SDN/Zones/Plugin.pm           |  6 +++
>  src/PVE/Network/SDN/Zones/SimplePlugin.pm     | 20 +++++++-
>  .../evpn/vrf-empty/expected_sdn_interfaces    |  6 +++
>  src/test/zones/evpn/vrf-empty/interfaces      |  2 +
>  src/test/zones/evpn/vrf-empty/sdn_config      | 13 +++++
>  .../simple/vrf-empty/expected_sdn_interfaces  |  5 ++
>  src/test/zones/simple/vrf-empty/interfaces    |  2 +
>  src/test/zones/simple/vrf-empty/sdn_config    | 12 +++++
>  .../zones/simple/vrf/expected_sdn_interfaces  | 12 +++++
>  src/test/zones/simple/vrf/interfaces          |  2 +
>  src/test/zones/simple/vrf/sdn_config          | 17 +++++++
>  18 files changed, 253 insertions(+), 28 deletions(-)
>  create mode 100644 src/test/zones/evpn/vrf-empty/expected_sdn_interfaces
>  create mode 100644 src/test/zones/evpn/vrf-empty/interfaces
>  create mode 100644 src/test/zones/evpn/vrf-empty/sdn_config
>  create mode 100644 src/test/zones/simple/vrf-empty/expected_sdn_interfaces
>  create mode 100644 src/test/zones/simple/vrf-empty/interfaces
>  create mode 100644 src/test/zones/simple/vrf-empty/sdn_config
>  create mode 100644 src/test/zones/simple/vrf/expected_sdn_interfaces
>  create mode 100644 src/test/zones/simple/vrf/interfaces
>  create mode 100644 src/test/zones/simple/vrf/sdn_config
> 
> 
> pve-manager:
> 
> Gabriel Goller (2):
>   ui: sdn: add VRF zone selection for fabrics
>   ui: sdn: expose EVPN zones as fabric VRFs
> 
>  www/manager6/form/SDNZoneSelector.js   | 16 +++++++++++++++-
>  www/manager6/sdn/fabrics/FabricEdit.js | 18 ++++++++++++++++++
>  www/manager6/sdn/zones/SimpleEdit.js   | 12 +++++++++++-
>  3 files changed, 44 insertions(+), 2 deletions(-)
> 
> 
> Summary over all repositories:
>   40 files changed, 1131 insertions(+), 201 deletions(-)
>