[PVE-User] Inter VRF traffic

[PVE-User] Inter VRF traffic

[Cyrus]

Hello!,

I'm trying to make traffic work between VRFs passing through a an
external firewall (opnsense+frr) but traffic seems to be resolved
locally by the node, even though source/destination are on different
VRFs (and ultimately doesn't work):

root@pve-01:~/bin# ip route get 192.168.111.10
192.168.111.10 via 192.168.203.145 dev vrfbr_L01VPN01 src
192.168.203.212 uid 0
   cache

root@pve-01:~/bin# ip route get 192.168.111.10
192.168.111.10 dev ol111001 src 192.168.111.1 uid 0
   cache

root@pve-01:~/bin# ip addr show dev ol111001
191: ol111001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master vrf_L01VPN01 state UP group default qlen 1000
   link/ether bc:24:11:e6:34:58 brd ff:ff:ff:ff:ff:ff
   inet 192.168.111.1/25 scope global ol111001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fee6:3458/64 scope link
      valid_lft forever preferred_lft forever

root@pve-01:~/bin# ip addr show dev ol107001
63: ol107001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master vrf_SDCVPN01 state UP group default qlen 1000
   link/ether bc:24:11:a9:f9:46 brd ff:ff:ff:ff:ff:ff
   inet 192.168.107.1/27 scope global ol107001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fea9:f946/64 scope link
      valid_lft forever preferred_lft forever

"ip r" output: https://pastebin.com/Q9sF8uMv
"frr.conf.local" content: https://pastebin.com/KAqNqKB1
rendered "frr.conf": https://pastebin.com/gUpYnuc0
/etc/pve/sdn/*: https://pastebin.com/U7yjNe5N
"/etc/network/interfaces" for pve-01: https://pastebin.com/smEfYUJw

North/South traffic works anytime destination is an external network.

North/South traffic fails if destination is the host or a network in
another VRF and traffic should be forwarded via an external firewall.

Any hints?, is something missing?

Regards,

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Inter VRF traffic

[Schunke, Alexander]

[-- Attachment #1.1: Type: text/plain, Size: 2935 bytes --]

Hi Cyrus, 

from a first glance I would say this makes sense since from what I
understand the VMs are running on the same server and are thus connected to
the same bridge. Since the bridge knows both MAC addresses it forwards the
traffic between the VMs at layer 2. In other words, the packages never get
to layer 3 and thus never get routed upstream. 

What you could try is to enable "Isolate Ports" in your vnet config.[1] This
should set the `isolated` flag on each interface connected to the bridge,
forcing the kernel to route the packages. 

At least in theory. I am yet to migrate my setup to SDN to make use of that
future myself.

All the best, 
Alex.

[1]:
https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_config_vnet

-----Original Message-----
From: pve-user <pve-user-bounces@lists.proxmox.com> On Behalf Of Cyrus
Sent: 10 March 2025 21:39
To: pve-user@lists.proxmox.com
Subject: [PVE-User] Inter VRF traffic

Hello!,

I'm trying to make traffic work between VRFs passing through a an external
firewall (opnsense+frr) but traffic seems to be resolved locally by the
node, even though source/destination are on different VRFs (and ultimately
doesn't work):

root@pve-01:~/bin# ip route get 192.168.111.10
192.168.111.10 via 192.168.203.145 dev vrfbr_L01VPN01 src
192.168.203.212 uid 0
   cache

root@pve-01:~/bin# ip route get 192.168.111.10
192.168.111.10 dev ol111001 src 192.168.111.1 uid 0
   cache

root@pve-01:~/bin# ip addr show dev ol111001
191: ol111001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master vrf_L01VPN01 state UP group default qlen 1000
   link/ether bc:24:11:e6:34:58 brd ff:ff:ff:ff:ff:ff
   inet 192.168.111.1/25 scope global ol111001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fee6:3458/64 scope link
      valid_lft forever preferred_lft forever

root@pve-01:~/bin# ip addr show dev ol107001
63: ol107001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master vrf_SDCVPN01 state UP group default qlen 1000
   link/ether bc:24:11:a9:f9:46 brd ff:ff:ff:ff:ff:ff
   inet 192.168.107.1/27 scope global ol107001
      valid_lft forever preferred_lft forever
   inet6 fe80::be24:11ff:fea9:f946/64 scope link
      valid_lft forever preferred_lft forever

"ip r" output: https://pastebin.com/Q9sF8uMv "frr.conf.local" content:
https://pastebin.com/KAqNqKB1 rendered "frr.conf":
https://pastebin.com/gUpYnuc0
/etc/pve/sdn/*: https://pastebin.com/U7yjNe5N "/etc/network/interfaces" for
pve-01: https://pastebin.com/smEfYUJw

North/South traffic works anytime destination is an external network.

North/South traffic fails if destination is the host or a network in another
VRF and traffic should be forwarded via an external firewall.

Any hints?, is something missing?

Regards,

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Inter VRF traffic

[DERUMIER, Alexandre]

Hi,

>>I'm trying to make traffic work between VRFs passing through a an
>>external firewall (opnsense+frr) but traffic seems to be resolved
>>locally by the node, even though source/destination are on different
>>VRFs (and ultimately doesn't work):

as you have defined exit-nodes, they are leaking routes between the
main vrf && the evpn zone vrf. (to be able to route traffic between the
evpn network and the real network)


if you want to announce evpn subnets to your opensense, you can create
an extra bgp controller for each node, and add your opensense ip as
peer. it should be enough.









_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Inter VRF traffic

[Cyrus]

On Tue, Mar 11, 2025, 13:41 DERUMIER, Alexandre <
alexandre.derumier@groupe-cyllene.com> wrote:

> Hi,
>
> >>I'm trying to make traffic work between VRFs passing through a an
> >>external firewall (opnsense+frr) but traffic seems to be resolved
> >>locally by the node, even though source/destination are on different
> >>VRFs (and ultimately doesn't work):
>
> as you have defined exit-nodes, they are leaking routes between the
> main vrf && the evpn zone vrf. (to be able to route traffic between the
> evpn network and the real network)
>
>
> if you want to announce evpn subnets to your opensense, you can create
> an extra bgp controller for each node, and add your opensense ip as
> peer. it should be enough.
>

Hello!,

Now that you mention it.... Probably I don't need exit nodes to be defined.

In this specific usecase, I'm placing peering interfaces in the specific
VRFs and configuring 2 manual BGP instances towards the firewalls.

That might fix my current problem. Will try and report back!

Regards.

>
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Inter VRF traffic

[Cyrus]

On Tue, Mar 11, 2025, 14:44 Cyrus <cyruspy@gmail.com> wrote:

> On Tue, Mar 11, 2025, 13:41 DERUMIER, Alexandre <
> alexandre.derumier@groupe-cyllene.com> wrote:
>
>> Hi,
>>
>> >>I'm trying to make traffic work between VRFs passing through a an
>> >>external firewall (opnsense+frr) but traffic seems to be resolved
>> >>locally by the node, even though source/destination are on different
>> >>VRFs (and ultimately doesn't work):
>>
>> as you have defined exit-nodes, they are leaking routes between the
>> main vrf && the evpn zone vrf. (to be able to route traffic between the
>> evpn network and the real network)
>>
>>
>> if you want to announce evpn subnets to your opensense, you can create
>> an extra bgp controller for each node, and add your opensense ip as
>> peer. it should be enough.
>>
>
> Hello!,
>
> Now that you mention it.... Probably I don't need exit nodes to be defined.
>
> In this specific usecase, I'm placing peering interfaces in the specific
> VRFs and configuring 2 manual BGP instances towards the firewalls.
>
> That might fix my current problem. Will try and report back!
>
> Regards.
>

That was it!!!!, thanks for your comments.

I removed my exit nodes from the zone configuration and kept my manual BGP
peers (1 set per VRF) towards the external NE (not supporting EVPN).

Regards.

>
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user