SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

[Tobias Fiebig]

Moin,
i am currently configuring a proxmox cluster that is supposed to be
IPv6 only and attached to an RFC8950 based L3 fabric.

While doing so, I noticed that VXLAN zones cannot be configured on an
IPv6 only fabric; The reason for that is that

/usr/share/perl5/PVE/Network/SDN/Zones/VxlanPlugin.pm

Only checks for $current_node->{ip} in line 109.

Subsequently, the peers and ifaceip are also set for v4 only. Adding
logic for ip6 fixes this.

(dropping this to the devel list, as it is not really a question/users
point, and should be a straight forward fix.)

With best regards,
Tobias

-- 
My working day may not be your working day. Please do not feel obliged 
to reply to my email outside of your normal working hours.
-----------------------------------------------------------------
Univ.Prof. Dr.-Ing. Tobias Fiebig, E191-06 - Internet Infrastructures
TU Wien, Fakultät für Informatik, Treitl Str. 3, 1040 Wien
Room DE01 15 mobile phone: +31 616 80 98 99 mail: tobias@internet.wien

Re: SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

[Tobias Fiebig]

Moin,

oh, and what I also noticed (discussed in the forums already ~4y ago,
see:
https://forum.proxmox.com/threads/sdn-wrong-automatic-vxlan-configuration-to-use-it-over-wiregaurd.106401/
) is that vxlan over IPv6 needs vxlan-local-tunnelip to be set to the
local tunnel; Currently using explicit peer definitions to make this
work, where there (at least) that is also being skipped in
/etc/network/interfaces.d/sdn.

With best regards,
Tobias

On Tue, 2026-10-06 at 16:46 +0200, Tobias Fiebig wrote:
> Moin,
> i am currently configuring a proxmox cluster that is supposed to be
> IPv6 only and attached to an RFC8950 based L3 fabric.
> 
> While doing so, I noticed that VXLAN zones cannot be configured on an
> IPv6 only fabric; The reason for that is that
> 
> /usr/share/perl5/PVE/Network/SDN/Zones/VxlanPlugin.pm
> 
> Only checks for $current_node->{ip} in line 109.
> 
> Subsequently, the peers and ifaceip are also set for v4 only. Adding
> logic for ip6 fixes this.
> 
> (dropping this to the devel list, as it is not really a
> question/users
> point, and should be a straight forward fix.)
> 
> With best regards,
> Tobias

-- 
My working day may not be your working day. Please do not feel obliged 
to reply to my email outside of your normal working hours.
-----------------------------------------------------------------
Univ.Prof. Dr.-Ing. Tobias Fiebig, E191-06 - Internet Infrastructures
TU Wien, Fakultät für Informatik, Treitl Str. 3, 1040 Wien
Room DE01 15 mobile phone: +31 616 80 98 99 mail: tobias@internet.wien

Re: SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

[Hannes Laimer]

Hey Tobias,

yes, a patch for this specifically already exists [1], and with [2] also
v6 fabrics should work.

[1]
https://lore.proxmox.com/pve-devel/20260925134050.754528-1-h.laimer@proxmox.com/
[2]
https://lore.proxmox.com/pve-devel/20261007064553.164447-1-h.laimer@proxmox.com/T/#u

On 2026-10-06 17:20, Tobias Fiebig wrote:
> Moin,
> 
> oh, and what I also noticed (discussed in the forums already ~4y ago,
> see:
> https://forum.proxmox.com/threads/sdn-wrong-automatic-vxlan-configuration-to-use-it-over-wiregaurd.106401/
> ) is that vxlan over IPv6 needs vxlan-local-tunnelip to be set to the
> local tunnel; Currently using explicit peer definitions to make this
> work, where there (at least) that is also being skipped in
> /etc/network/interfaces.d/sdn.
> 
> With best regards,
> Tobias
> 
> On Tue, 2026-10-06 at 16:46 +0200, Tobias Fiebig wrote:
>> Moin,
>> i am currently configuring a proxmox cluster that is supposed to be
>> IPv6 only and attached to an RFC8950 based L3 fabric.
>>
>> While doing so, I noticed that VXLAN zones cannot be configured on an
>> IPv6 only fabric; The reason for that is that
>>
>> /usr/share/perl5/PVE/Network/SDN/Zones/VxlanPlugin.pm
>>
>> Only checks for $current_node->{ip} in line 109.
>>
>> Subsequently, the peers and ifaceip are also set for v4 only. Adding
>> logic for ip6 fixes this.
>>
>> (dropping this to the devel list, as it is not really a
>> question/users
>> point, and should be a straight forward fix.)
>>
>> With best regards,
>> Tobias
> 

Re: SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

[Tobias Fiebig]

Hello Hannes,


> yes, a patch for this specifically already exists [1], and with [2]
> also v6 fabrics should work.

Thanks, looking forward to it being merged.

In the context of the L3 fabric, i stumbled over two additional things.

1. BGP Pseudo interface MTU

If the L3 fabric is via links running on jumbos, and there is no
additional L2 network between the proxmox nodes, VM migration breaks as
soon as the BGP dummy interface for the loopback address in the fabric
is added on nodes, as that interface is created with an MTU of 1500 and
used as the default source address for contacting other nodes, i.e.,
while the other node originates packets for an MTU of, e.g., 9000,
those reach the initiating node, but cannot be forwarded to the (dummy)
interface where the selected source address resides. I am also not sure
why there are not PTB ICMPv6 packets originated that should fix this.
But on migration, copying over the memory state just hangs.

This can be fixed by either adjusting the MTU of the dummy interface to
match that of the actual links, or by creating a dedicated shared L2,
e.g., via another VXLAN, that is than addressed and explicitly
configured as the migration network.


2. Pure L3 Fabric + RFC8950 / IPv4 with IPv6 Nexthops

Beyond that, there is also the overall network concept. Not sure
whether this would be viable feature; But dumping it in here for
reference.

What I am essentially trying to build is a network (not only the PVE
components) that runs mostly IPv6 only, and has IPv4 as /32 routed to
GUA, handled via (i)BGP.

A description of the concept can be found here:
https://ripe88.ripe.net/presentations/15-ripe_88_v6_wg.pdf

The general advantages would be:
- No need for bridge devices
- No need for any L2 within the whole network; This also means that,
e.g., LACP is no longer needed on switches; Instead the L3 fabric is
BGP based (numbered or unnumbered); Also, with ECMP, traffic actually
balances etc.
- Most importantly: No ARP ;-)

Things I noticed that stand in the way there:

- Technically, nodes would have to be able to create a VM interface
that is not added to a bridge; Instead, it would just get DHCP/SLAAC
with a prefix selected from a configured covering prefix; The more
specific (automatically) selected for the VM would then just be
announced in the BGP fabric, and upon VM migration, the prefix would be
moved. The same could be done with v4, considering the VM OS supports
v6 NH for v4 default routes.
- There is no way to add communities via configurable routemaps in
proxmox atm; This makes it a bit more difficult to properly configure
filters when interacting with hosts speaking BGP outside of the proxmox
cluster; Same holds for import filters based on matching communities.
- The imported routes webinterface in the fabric view is a bit
overwhelmed when the nodes get a (v6) fulltable (questionable concept,
but... it can make sense ;-)) Pagination there would probably be
helpful (also, in general, for larger networks).

Happy to hear your thoughts; If this is the wrong place for discussing
this, please also let me know.

With best regards,
Tobias

-- 
My working day may not be your working day. Please do not feel obliged 
to reply to my email outside of your normal working hours.
-----------------------------------------------------------------
Univ.Prof. Dr.-Ing. Tobias Fiebig, E191-06 - Internet Infrastructures
TU Wien, Fakultät für Informatik, Treitl Str. 3, 1040 Wien
Room DE01 15 mobile phone: +31 616 80 98 99 mail: tobias@internet.wien

Re: SDN with BGP Fabric and IPv6 Only Underlay for VXLAN

[Hannes Laimer]

On 2026-10-07 11:46, Tobias Fiebig wrote:
> Hello Hannes,
> 
> 
>> yes, a patch for this specifically already exists [1], and with [2]
>> also v6 fabrics should work.
> 
> Thanks, looking forward to it being merged.
> 
> In the context of the L3 fabric, i stumbled over two additional things.
> 
> 1. BGP Pseudo interface MTU
> 
> If the L3 fabric is via links running on jumbos, and there is no
> additional L2 network between the proxmox nodes, VM migration breaks as
> soon as the BGP dummy interface for the loopback address in the fabric
> is added on nodes, as that interface is created with an MTU of 1500 and
> used as the default source address for contacting other nodes, i.e.,
> while the other node originates packets for an MTU of, e.g., 9000,
> those reach the initiating node, but cannot be forwarded to the (dummy)
> interface where the selected source address resides. I am also not sure
> why there are not PTB ICMPv6 packets originated that should fix this.
> But on migration, copying over the memory state just hangs.
> 

i think a switch somewhere along the way would not emit those..

> This can be fixed by either adjusting the MTU of the dummy interface to
> match that of the actual links, or by creating a dedicated shared L2,
> e.g., via another VXLAN, that is than addressed and explicitly
> configured as the migration network.
> 

hmm, the dummy doesn't transport any traffic, it just terminates it, MTU
should not matter there.. (did not test this)

does
`ping -M do -s 8952 {dummy ip}`
work between all hosts?

> 
> 2. Pure L3 Fabric + RFC8950 / IPv4 with IPv6 Nexthops
> 

yes, funnily enough I do have a poc laying around for basically this(L3
routed zone with /32 routes for each guest), but there are some things
that need to be addressed first. most importantly, we have to have a
stable/reliable way to know the ip a guest is using, and for that we
need IPAM to be more general and not only DHCP scoped(for which i also
have a poc flying around somewhere)

the main bottleneck is me getting around to work on this, and of course
also someone getting around to review it :) technically there isn't
really something blocking this for us

> Beyond that, there is also the overall network concept. Not sure
> whether this would be viable feature; But dumping it in here for
> reference.
> 
> What I am essentially trying to build is a network (not only the PVE
> components) that runs mostly IPv6 only, and has IPv4 as /32 routed to
> GUA, handled via (i)BGP.
> 
> A description of the concept can be found here:
> https://ripe88.ripe.net/presentations/15-ripe_88_v6_wg.pdf
> 
> The general advantages would be:
> - No need for bridge devices
> - No need for any L2 within the whole network; This also means that,
> e.g., LACP is no longer needed on switches; Instead the L3 fabric is
> BGP based (numbered or unnumbered); Also, with ECMP, traffic actually
> balances etc.
> - Most importantly: No ARP ;-)
> 
> Things I noticed that stand in the way there:
> 
> - Technically, nodes would have to be able to create a VM interface
> that is not added to a bridge; Instead, it would just get DHCP/SLAAC
> with a prefix selected from a configured covering prefix; The more
> specific (automatically) selected for the VM would then just be
> announced in the BGP fabric, and upon VM migration, the prefix would be
> moved. The same could be done with v4, considering the VM OS supports
> v6 NH for v4 default routes.

the main difference to what you propose is that it would be SRv6 based.
technically encapsulation would not be needed, but having it in place
would allow for some nice additions in the future, that would otherwise
be not really possible. also the local bridge would stay, but
technically that could also be dropped, just that currently vnet means
bridge basically, so im not sure about that..

> - There is no way to add communities via configurable routemaps in
> proxmox atm; This makes it a bit more difficult to properly configure
> filters when interacting with hosts speaking BGP outside of the proxmox
> cluster; Same holds for import filters based on matching communities.
> - The imported routes webinterface in the fabric view is a bit
> overwhelmed when the nodes get a (v6) fulltable (questionable concept,
> but... it can make sense ;-)) Pagination there would probably be
> helpful (also, in general, for larger networks).
> 

your last two points sound like a reasonable addition, and we track
things like this at [1] as an `enhancement`, you can just file them
there with your rationale and use-case

> Happy to hear your thoughts; If this is the wrong place for discussing
> this, please also let me know.
> 

it's fine, pve-user[2] might be more fitting, since pve-devel is mostly
for stuff that involves code

> With best regards,
> Tobias
> 

[1] https://bugzilla.proxmox.com/
[2] https://lists.proxmox.com/postorius/lists/pve-user.lists.proxmox.com/