[PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[Hannes Laimer]

With advertise-all-vni, which an EVPN controller sets even without an
EVPN zone, FRR treats every vxlan device on a node as an EVPN VNI.
That includes the devices of a VXLAN zone, for which the other nodes
running FRR become remote VTEPs.

When the controller is removed, FRR drops its VNIs and deletes the
flood entries of their remote VTEPs. The kernel identifies such an
entry by its address and not by who added it, so the zone's static
flood entries to those nodes are deleted. The zone can no longer
flood, so nothing resolves across nodes, and no reload repairs it,
since ifupdown2 re-applies the peer list only when the configured list
changed.

Re-append the entries after every FRR apply. The kernel treats an
existing remote as a no-op, so the entries are simply added where
missing.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/PVE/Network/SDN.pm       |  7 ++++++-
 src/PVE/Network/SDN/Zones.pm | 18 ++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
index 33a3cf3..42af00d 100644
--- a/src/PVE/Network/SDN.pm
+++ b/src/PVE/Network/SDN.pm
@@ -492,7 +492,12 @@ sub generate_frr_config {
     my $raw_config = PVE::Network::SDN::generate_frr_raw_config($running_config, $fabric_config);
     PVE::Network::SDN::Frr::write_raw_config($raw_config);
 
-    PVE::Network::SDN::Frr::apply($needs_restart) if $apply;
+    return if !$apply;
+
+    PVE::Network::SDN::Frr::apply($needs_restart);
+
+    # zebra removes a VXLAN zone's static flood entries with the VTEPs it withdraws
+    PVE::Network::SDN::Zones::restore_vxlan_flood_entries();
 }
 
 sub generate_dhcp_config {
diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
index f668303..2c3c69c 100644
--- a/src/PVE/Network/SDN/Zones.pm
+++ b/src/PVE/Network/SDN/Zones.pm
@@ -178,6 +178,24 @@ sub generate_etc_network_config {
     return $raw_network_config;
 }
 
+sub restore_vxlan_flood_entries {
+    my $raw_config = eval { PVE::Tools::file_get_contents($local_network_sdn_file) };
+    return if !defined($raw_config);
+
+    my $iface;
+    for my $line (split(/\n/, $raw_config)) {
+        if ($line =~ m/^iface (\S+)$/) {
+            $iface = $1;
+        } elsif ($line =~ m/^\s+vxlan_remoteip (\S+)$/) {
+            my $address = $1;
+            my $cmd =
+                ['bridge', 'fdb', 'append', '00:00:00:00:00:00', 'dev', $iface, 'dst', $address];
+            eval { run_command($cmd) };
+            warn "$iface: restoring the flood entry to $address failed: $@" if $@;
+        }
+    }
+}
+
 sub read_etc_network_config_version {
     my $versionstr = PVE::Tools::file_read_firstline($local_network_sdn_file);
 
-- 
2.47.3

Re: [PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[Gabriel Goller]

On 29.09.2026 11:28, Hannes Laimer wrote:
> With advertise-all-vni, which an EVPN controller sets even without an
> EVPN zone, FRR treats every vxlan device on a node as an EVPN VNI.
> That includes the devices of a VXLAN zone, for which the other nodes
> running FRR become remote VTEPs.
> 
> When the controller is removed, FRR drops its VNIs and deletes the
> flood entries of their remote VTEPs. The kernel identifies such an
> entry by its address and not by who added it, so the zone's static
> flood entries to those nodes are deleted. The zone can no longer
> flood, so nothing resolves across nodes, and no reload repairs it,
> since ifupdown2 re-applies the peer list only when the configured list
> changed.
> 
> Re-append the entries after every FRR apply. The kernel treats an
> existing remote as a no-op, so the entries are simply added where
> missing.
> 
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
>  src/PVE/Network/SDN.pm       |  7 ++++++-
>  src/PVE/Network/SDN/Zones.pm | 18 ++++++++++++++++++
>  2 files changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
> index 33a3cf3..42af00d 100644
> --- a/src/PVE/Network/SDN.pm
> +++ b/src/PVE/Network/SDN.pm
> @@ -492,7 +492,12 @@ sub generate_frr_config {
>      my $raw_config = PVE::Network::SDN::generate_frr_raw_config($running_config, $fabric_config);
>      PVE::Network::SDN::Frr::write_raw_config($raw_config);
>  
> -    PVE::Network::SDN::Frr::apply($needs_restart) if $apply;
> +    return if !$apply;
> +
> +    PVE::Network::SDN::Frr::apply($needs_restart);
> +
> +    # zebra removes a VXLAN zone's static flood entries with the VTEPs it withdraws
> +    PVE::Network::SDN::Zones::restore_vxlan_flood_entries();

Hmm so this is tricky, on reload bgpd tells zebra to remove these changes
and zebra enqueues this to the dplane, so this is asynchronous in many
ways. So we can't guarantee that the entries have been removed when running
restore_vxlan_flood_entries().

Not sure what we could do here.

Three options that came to my mind where:
1) sleep(5)
2) `bridge monitor fdb` before running the frr reload/restart and then check if the entries have been removed (kind of overkill)
3) force a frr restart (not 100% sure if this works)

>  }
>  
>  sub generate_dhcp_config {
> diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
> index f668303..2c3c69c 100644
> --- a/src/PVE/Network/SDN/Zones.pm
> +++ b/src/PVE/Network/SDN/Zones.pm
> @@ -178,6 +178,24 @@ sub generate_etc_network_config {
>      return $raw_network_config;
>  }
>  
> +sub restore_vxlan_flood_entries {
> +    my $raw_config = eval { PVE::Tools::file_get_contents($local_network_sdn_file) };
> +    return if !defined($raw_config);

Maybe an error here would be nice instead of a silent return. Although this
probably can't happen anyway...

> +
> +    my $iface;
> +    for my $line (split(/\n/, $raw_config)) {
> +        if ($line =~ m/^iface (\S+)$/) {
> +            $iface = $1;
> +        } elsif ($line =~ m/^\s+vxlan_remoteip (\S+)$/) {
> +            my $address = $1;
> +            my $cmd =
> +                ['bridge', 'fdb', 'append', '00:00:00:00:00:00', 'dev', $iface, 'dst', $address];
> +            eval { run_command($cmd) };
> +            warn "$iface: restoring the flood entry to $address failed: $@" if $@;
> +        }
> +    }
> +}
> +
>  sub read_etc_network_config_version {
>      my $versionstr = PVE::Tools::file_read_firstline($local_network_sdn_file);
>  
> -- 
> 2.47.3

Re: [PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[Hannes Laimer]

On 2026-10-05 14:31, Gabriel Goller wrote:
> On 29.09.2026 11:28, Hannes Laimer wrote:
>> With advertise-all-vni, which an EVPN controller sets even without an
>> EVPN zone, FRR treats every vxlan device on a node as an EVPN VNI.
>> That includes the devices of a VXLAN zone, for which the other nodes
>> running FRR become remote VTEPs.
>>
>> When the controller is removed, FRR drops its VNIs and deletes the
>> flood entries of their remote VTEPs. The kernel identifies such an
>> entry by its address and not by who added it, so the zone's static
>> flood entries to those nodes are deleted. The zone can no longer
>> flood, so nothing resolves across nodes, and no reload repairs it,
>> since ifupdown2 re-applies the peer list only when the configured list
>> changed.
>>
>> Re-append the entries after every FRR apply. The kernel treats an
>> existing remote as a no-op, so the entries are simply added where
>> missing.
>>
>> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
>> ---
>>  src/PVE/Network/SDN.pm       |  7 ++++++-
>>  src/PVE/Network/SDN/Zones.pm | 18 ++++++++++++++++++
>>  2 files changed, 24 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
>> index 33a3cf3..42af00d 100644
>> --- a/src/PVE/Network/SDN.pm
>> +++ b/src/PVE/Network/SDN.pm
>> @@ -492,7 +492,12 @@ sub generate_frr_config {
>>      my $raw_config = PVE::Network::SDN::generate_frr_raw_config($running_config, $fabric_config);
>>      PVE::Network::SDN::Frr::write_raw_config($raw_config);
>>  
>> -    PVE::Network::SDN::Frr::apply($needs_restart) if $apply;
>> +    return if !$apply;
>> +
>> +    PVE::Network::SDN::Frr::apply($needs_restart);
>> +
>> +    # zebra removes a VXLAN zone's static flood entries with the VTEPs it withdraws
>> +    PVE::Network::SDN::Zones::restore_vxlan_flood_entries();
> 
> Hmm so this is tricky, on reload bgpd tells zebra to remove these changes
> and zebra enqueues this to the dplane, so this is asynchronous in many
> ways. So we can't guarantee that the entries have been removed when running
> restore_vxlan_flood_entries().
> 

yes, this is unfortunately not something we can guarantee. restart
happens already at the end of FRR::apply, we could force a `reload` of
there is a removed evpn controller and an existing vxlan zone. I'm not
sure there is a difference for this here tough..


> Not sure what we could do here.
> 
> Three options that came to my mind where:
> 1) sleep(5)

not sure what kind of timeframes are realistic here, since we can't
guarantee anything anyway, something like 1s is probably enough. but in
my (limited) testing i also couldn't hit this without any sleep

> 2) `bridge monitor fdb` before running the frr reload/restart and then check if the entries have been removed (kind of overkill)

we still wouldn't know how long to wait..

> 3) force a frr restart (not 100% sure if this works)
> 

ideally we could just tell frr 'hands off these VNIs..', but we can't,
and given `advertise-all-vni` that might also be weird/wrong to begin
with


>>  }
>>  
>>  sub generate_dhcp_config {
>> diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
>> index f668303..2c3c69c 100644
>> --- a/src/PVE/Network/SDN/Zones.pm
>> +++ b/src/PVE/Network/SDN/Zones.pm
>> @@ -178,6 +178,24 @@ sub generate_etc_network_config {
>>      return $raw_network_config;
>>  }
>>  
>> +sub restore_vxlan_flood_entries {
>> +    my $raw_config = eval { PVE::Tools::file_get_contents($local_network_sdn_file) };
>> +    return if !defined($raw_config);
> 
> Maybe an error here would be nice instead of a silent return. Although this
> probably can't happen anyway...
> 

yeah, not really an error imho.. but a warning should be fine,
because the caller assumed there is a config file


thanks for taking a look! :)

>> +
>> +    my $iface;
>> +    for my $line (split(/\n/, $raw_config)) {
>> +        if ($line =~ m/^iface (\S+)$/) {
>> +            $iface = $1;
>> +        } elsif ($line =~ m/^\s+vxlan_remoteip (\S+)$/) {
>> +            my $address = $1;
>> +            my $cmd =
>> +                ['bridge', 'fdb', 'append', '00:00:00:00:00:00', 'dev', $iface, 'dst', $address];
>> +            eval { run_command($cmd) };
>> +            warn "$iface: restoring the flood entry to $address failed: $@" if $@;
>> +        }
>> +    }
>> +}
>> +
>>  sub read_etc_network_config_version {
>>      my $versionstr = PVE::Tools::file_read_firstline($local_network_sdn_file);
>>  
>> -- 
>> 2.47.3

Re: [PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[Gabriel Goller]

On 05.10.2026 15:01, Hannes Laimer wrote:
> On 2026-10-05 14:31, Gabriel Goller wrote:
> > On 29.09.2026 11:28, Hannes Laimer wrote:
> >> [snip]
> >> diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
> >> index 33a3cf3..42af00d 100644
> >> --- a/src/PVE/Network/SDN.pm
> >> +++ b/src/PVE/Network/SDN.pm
> >> @@ -492,7 +492,12 @@ sub generate_frr_config {
> >>      my $raw_config = PVE::Network::SDN::generate_frr_raw_config($running_config, $fabric_config);
> >>      PVE::Network::SDN::Frr::write_raw_config($raw_config);
> >>  
> >> -    PVE::Network::SDN::Frr::apply($needs_restart) if $apply;
> >> +    return if !$apply;
> >> +
> >> +    PVE::Network::SDN::Frr::apply($needs_restart);
> >> +
> >> +    # zebra removes a VXLAN zone's static flood entries with the VTEPs it withdraws
> >> +    PVE::Network::SDN::Zones::restore_vxlan_flood_entries();
> > 
> > Hmm so this is tricky, on reload bgpd tells zebra to remove these changes
> > and zebra enqueues this to the dplane, so this is asynchronous in many
> > ways. So we can't guarantee that the entries have been removed when running
> > restore_vxlan_flood_entries().
> > 
> 
> yes, this is unfortunately not something we can guarantee. restart
> happens already at the end of FRR::apply, we could force a `reload` of
> there is a removed evpn controller and an existing vxlan zone. I'm not
> sure there is a difference for this here tough..

Hmm the problem is with the reload though isn't it? A restart would simply kill
all the daemons and zebra on startup doesn't nuke the fdb entries right?

> > Not sure what we could do here.
> > 
> > Three options that came to my mind where:
> > 1) sleep(5)
> 
> not sure what kind of timeframes are realistic here, since we can't
> guarantee anything anyway, something like 1s is probably enough. but in
> my (limited) testing i also couldn't hit this without any sleep
> 
> > 2) `bridge monitor fdb` before running the frr reload/restart and then check if the entries have been removed (kind of overkill)
> 
> we still wouldn't know how long to wait..

True. At least we could shortcut it when they are gone.

> > 3) force a frr restart (not 100% sure if this works)
> 
> ideally we could just tell frr 'hands off these VNIs..', but we can't,
> and given `advertise-all-vni` that might also be weird/wrong to begin
> with

Yeah, don't think thats possible.
But IMO forcing a restart when advertise-all-vni is removed wouldn't even be so
bad. Of course that would break other connections the user has open.

> >>  }
> >>  
> >>  sub generate_dhcp_config {
> >> diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
> >> index f668303..2c3c69c 100644
> >> --- a/src/PVE/Network/SDN/Zones.pm
> >> +++ b/src/PVE/Network/SDN/Zones.pm
> >> @@ -178,6 +178,24 @@ sub generate_etc_network_config {
> >>      return $raw_network_config;
> >>  }
> >>  
> >> +sub restore_vxlan_flood_entries {
> >> +    my $raw_config = eval { PVE::Tools::file_get_contents($local_network_sdn_file) };
> >> +    return if !defined($raw_config);
> > 
> > Maybe an error here would be nice instead of a silent return. Although this
> > probably can't happen anyway...
> > 
> 
> yeah, not really an error imho.. but a warning should be fine,
> because the caller assumed there is a config file
> 
> 
> thanks for taking a look! :)
> 
> >> +
> >> [snip]

Re: [PATCH pve-network] sdn: zones: vxlan: restore the flood entries after an FRR reload

[Hannes Laimer]

On 2026-10-05 15:13, Gabriel Goller wrote:
> On 05.10.2026 15:01, Hannes Laimer wrote:
>> On 2026-10-05 14:31, Gabriel Goller wrote:
>>> On 29.09.2026 11:28, Hannes Laimer wrote:
>>>> [snip]
>>>> diff --git a/src/PVE/Network/SDN.pm b/src/PVE/Network/SDN.pm
>>>> index 33a3cf3..42af00d 100644
>>>> --- a/src/PVE/Network/SDN.pm
>>>> +++ b/src/PVE/Network/SDN.pm
>>>> @@ -492,7 +492,12 @@ sub generate_frr_config {
>>>>      my $raw_config = PVE::Network::SDN::generate_frr_raw_config($running_config, $fabric_config);
>>>>      PVE::Network::SDN::Frr::write_raw_config($raw_config);
>>>>  
>>>> -    PVE::Network::SDN::Frr::apply($needs_restart) if $apply;
>>>> +    return if !$apply;
>>>> +
>>>> +    PVE::Network::SDN::Frr::apply($needs_restart);
>>>> +
>>>> +    # zebra removes a VXLAN zone's static flood entries with the VTEPs it withdraws
>>>> +    PVE::Network::SDN::Zones::restore_vxlan_flood_entries();
>>>
>>> Hmm so this is tricky, on reload bgpd tells zebra to remove these changes
>>> and zebra enqueues this to the dplane, so this is asynchronous in many
>>> ways. So we can't guarantee that the entries have been removed when running
>>> restore_vxlan_flood_entries().
>>>
>>
>> yes, this is unfortunately not something we can guarantee. restart
>> happens already at the end of FRR::apply, we could force a `reload` of
>> there is a removed evpn controller and an existing vxlan zone. I'm not
>> sure there is a difference for this here tough..
> 
> Hmm the problem is with the reload though isn't it? A restart would simply kill
> all the daemons and zebra on startup doesn't nuke the fdb entries right?
> 

actually not sure, we did write the frr config already at that point,
but without the controller.. no, i guess, as you say it shouldn't

>>> Not sure what we could do here.
>>>
>>> Three options that came to my mind where:
>>> 1) sleep(5)
>>
>> not sure what kind of timeframes are realistic here, since we can't
>> guarantee anything anyway, something like 1s is probably enough. but in
>> my (limited) testing i also couldn't hit this without any sleep
>>
>>> 2) `bridge monitor fdb` before running the frr reload/restart and then check if the entries have been removed (kind of overkill)
>>
>> we still wouldn't know how long to wait..
> 
> True. At least we could shortcut it when they are gone.

yes, but as you said, a (or no) sleep is probably better here

> 
>>> 3) force a frr restart (not 100% sure if this works)
>>
>> ideally we could just tell frr 'hands off these VNIs..', but we can't,
>> and given `advertise-all-vni` that might also be weird/wrong to begin
>> with
> 
> Yeah, don't think thats possible.
> But IMO forcing a restart when advertise-all-vni is removed wouldn't even be so
> bad. Of course that would break other connections the user has open.

that would actually work i think.. could be seen as an unnecessary one
though, depending how easily the "remove after our add" would actually
be hit..

> 
>>>>  }
>>>>  
>>>>  sub generate_dhcp_config {
>>>> diff --git a/src/PVE/Network/SDN/Zones.pm b/src/PVE/Network/SDN/Zones.pm
>>>> index f668303..2c3c69c 100644
>>>> --- a/src/PVE/Network/SDN/Zones.pm
>>>> +++ b/src/PVE/Network/SDN/Zones.pm
>>>> @@ -178,6 +178,24 @@ sub generate_etc_network_config {
>>>>      return $raw_network_config;
>>>>  }
>>>>  
>>>> +sub restore_vxlan_flood_entries {
>>>> +    my $raw_config = eval { PVE::Tools::file_get_contents($local_network_sdn_file) };
>>>> +    return if !defined($raw_config);
>>>
>>> Maybe an error here would be nice instead of a silent return. Although this
>>> probably can't happen anyway...
>>>
>>
>> yeah, not really an error imho.. but a warning should be fine,
>> because the caller assumed there is a config file
>>
>>
>> thanks for taking a look! :)
>>
>>>> +
>>>> [snip]