Re: [pve-devel] [PATCH perl-rs v2] fix: sdn: fabrics: always add node-ip to all fabric interfaces

[Gabriel Goller <g.goller@proxmox.com>]

On 01.10.2025 17:00, Stefan Hanreich wrote:
>Have some additional comments I wanted to post on the ML so they don't
>get lost, but they're more food for though for further improvements of
>the whole SDN / Fabrics stack wrt VRFs.
>
>Tested the patch series and looked at the code - LGTM!
>
>Reviewed-by: Stefan Hanreich <s.hanreich@proxmox.com>
>Tested-by: Stefan Hanreich <s.hanreich@proxmox.com>

Thanks for the review!

>On 9/16/25 2:41 PM, Gabriel Goller wrote:
>> [snip]
>> In this scenario we ping from Node1 to Node3, which means that Node2
>> needs to forward our packet. When the packet arrives at Node2, we again
>> check the routing table, where we see the same entry as before, so
>> 10.0.1.3 is available at (e.g.) ens20 onlink with the src address
>> 10.0.1.2. This is fine, but we still need to do an ARP request to lookup
>> the mac address of the neighbor which is attached at ens20. So we take
>> the packet we get from Node1, which has a source address of 10.0.1.1, and
>> call arp_solicit on it to make an ARP request and find the neighbors mac
>> address. arp_solicit will take the source address of the packet
>> (10.0.1.1) and lookup to search it locally. This check fails because
>> 10.0.1.1 is not available locally (there is a direct route to it, but
>> it's not configured on any local interface (RTN_LOCAL)). arp_solicit
>> will thus [2] call inet_select_addr, which goes through all the ip
>> addresses on the current interface (there are none, because this
>> interface is unnumbered) and then iterate through all the other
>> interfaces on the host and select the first one with 'scope link'. This
>> ip will then be selected as the source address for the outgoing ARP
>> package. Now if we're lucky this is the dummy interface on our node and
>> we select the correct source address (10.0.1.2) -- but we could also be
>> unlucky and it selects a completely random address from another
>> interface e.g. 172.16.0.26. If we're unlucky arp_solicit will send out
>> the following ARP packet:
>>
>>     Request who-has 10.0.1.3 tell 172.16.0.26
>>
>> We will get a correct response but the response will end up on another
>> interface (because 172.16.0.26 is not on the same interface as
>> 10.0.1.2). This means we will send out these ARP requests repeatedly and
>> never get an answer, so the ping from Node1 to Node3 will get
>> "Destination host unreachable errors".
>
>Interesting, that the src IP address directive from the route is not
>considered at all:
>
>172.16.123.2 nhid 162 via 172.16.123.2 dev ens22 [..] src 172.16.123.1
>
>Didn't dig further into it, there's a good chance there's a good reason
>for that I just didn't see immediately.

Hmm yeah, I'll look into this. Currently the whole neighbor system is
quite generic and limited, I'll see if there is any way we can add
"hints" to the arp_solicit function when it's called from ip_forward.

>What I also found interesting while jumping down this rabbit hole is the
>following comment / code section in the inet_select_addr function [1]:
>
>/* For VRFs, the VRF device takes the place of the loopback device,
> * with addresses on it being preferred.  Note in such cases the
> * loopback device will be among the devices that fail the master_idx
> * equality check in the loop below.
> */
>
>So, in that case (iiuc) one could side-step that problem by
>compartmentalizing the fabric inside its own VRF (further reinforcing my
>belief in implementing VRF support sooner than later to avoid issues
>like this when running everything in one routing table, particularly
>multiple fabrics).
>
>fabricd has no VRF support atm though (could potentially run it via ip
>vrf exec, but that seems hacky) - OSPF and BGP do.

I agree, in theory this would be very nice, but hacking vrf support into
lots of stuff that doesn't have vrf support might be tricky. Another
example is wireguard, where vrf support is also kinda tricky.

I'll look into OpenFabric VRF support though, this shouldn't be too
tricky to implement as ISIS already supports it.

>[1]
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/devinet.c?h=v6.17#n1359
>
>> For more information check out the simplified version of the arp_solicit
>> function below:
>> [snip]


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel