[PVE-User] OSPF when migrating from v8 to v9

[PVE-User] OSPF when migrating from v8 to v9

[Julien OHAYON]

Hello,

We are running version 8.4.1 and using SDN with EVPN. Since we already had OSPF in our infrastructure, we forced its activation through the configuration files and added some local configs. A few examples:

in /etc/frr/daemons

ospfd=yes
and
in /etc/frr/frr.conf.local

router ospf
 redistribute connected
 redistribute static
 network X.X.X.X/24 area 0
 default-information originate
exit

However, with version 9, great news: it is now integrated. But for the migration, it’s not so simple. When we upgrade a node to v9, OSPF gets disabled in /etc/frr/daemons.

Is it possible to keep it enabled? If yes, in which configuration file?

Of course, afterwards it will be fine to modify the configuration via Proxmox, but what is the impact, especially if it doesn’t match the configuration we had manually applied?

Thanks in advance.

Julien OHAYON
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Stefan Hanreich]

On 10/7/25 8:17 AM, Julien OHAYON wrote:
> Hello,
> 
> We are running version 8.4.1 and using SDN with EVPN. Since we already had OSPF in our infrastructure, we forced its activation through the configuration files and added some local configs. A few examples:
> 
> in /etc/frr/daemons
> 
> ospfd=yes
> and
> in /etc/frr/frr.conf.local
> 
> router ospf
>  redistribute connected
>  redistribute static
>  network X.X.X.X/24 area 0
>  default-information originate
> exit
> 
> However, with version 9, great news: it is now integrated. But for the migration, it’s not so simple. When we upgrade a node to v9, OSPF gets disabled in /etc/frr/daemons.

We ship the /etc/frr/daemons file via our FRR package, so if it gets
modified manually you should get a prompt on upgrade that asks whether
the modified file should be kept or not.

You can check if this is what happened by checking the term.log (or the
subsequent term.log.N.gz) file in /var/log/apt.

> Is it possible to keep it enabled? If yes, in which configuration file?

It should be kept enabled when choosing to keep the existing
configuration file as described above.

> Of course, afterwards it will be fine to modify the configuration via Proxmox, but what is the impact, especially if it doesn’t match the configuration we had manually applied?

As long as the additional manual configuration for OSPF is kept in
/etc/frr/frr.conf.local, then it will get merged on applying the SDN
configuration the same as before.


Kind Regards
Stefan


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Julien OHAYON]

Thanks for your reply. Unfortunately, when we apply the SDN configuration, it modifies the daemons file and sets it to “no”, since we haven’t configured OSPF directly in Proxmox. That’s the file I would like to prevent from being modified.



Cheers

Julien

> Le 7 oct. 2025 à 15:52, Stefan Hanreich <s.hanreich@proxmox.com> a écrit :
> 
> On 10/7/25 8:17 AM, Julien OHAYON wrote:
>> Hello,
>> 
>> We are running version 8.4.1 and using SDN with EVPN. Since we already had OSPF in our infrastructure, we forced its activation through the configuration files and added some local configs. A few examples:
>> 
>> in /etc/frr/daemons
>> 
>> ospfd=yes
>> and
>> in /etc/frr/frr.conf.local
>> 
>> router ospf
>> redistribute connected
>> redistribute static
>> network X.X.X.X/24 area 0
>> default-information originate
>> exit
>> 
>> However, with version 9, great news: it is now integrated. But for the migration, it’s not so simple. When we upgrade a node to v9, OSPF gets disabled in /etc/frr/daemons.
> 
> We ship the /etc/frr/daemons file via our FRR package, so if it gets
> modified manually you should get a prompt on upgrade that asks whether
> the modified file should be kept or not.
> 
> You can check if this is what happened by checking the term.log (or the
> subsequent term.log.N.gz) file in /var/log/apt.
> 
>> Is it possible to keep it enabled? If yes, in which configuration file?
> 
> It should be kept enabled when choosing to keep the existing
> configuration file as described above.
> 
>> Of course, afterwards it will be fine to modify the configuration via Proxmox, but what is the impact, especially if it doesn’t match the configuration we had manually applied?
> 
> As long as the additional manual configuration for OSPF is kept in
> /etc/frr/frr.conf.local, then it will get merged on applying the SDN
> configuration the same as before.
> 
> 
> Kind Regards
> Stefan
> 
> 

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Stefan Hanreich]

On 10/7/25 5:58 PM, Julien OHAYON wrote:
> Thanks for your reply. Unfortunately, when we apply the SDN configuration, it modifies the daemons file and sets it to “no”, since we haven’t configured OSPF directly in Proxmox. That’s the file I would like to prevent from being modified.

Ah sorry, I was only thinking of the upgrade process itself. In your
particular scenario (having an EVPN zone configured as well) I'm afraid
there's currently no easy way of preventing the daemons file from
getting overwritten by the SDN module.

The defaults are set in the frr config generation module [1] and
currently not overridable. I guess we'd have to expose a way of setting
the default value for those daemons to 1 instead of 0.

Another way would be trying to detect if the status of the daemons needs
to change compared to the previous SDN configuration and only re-write
the daemons file in the case it really *needs* to change instead of
every time. But that is probably a bit more tricky.

Could you open a feature request on our Bugzilla?

[1]
https://git.proxmox.com/?p=pve-network.git;a=blob;f=src/PVE/Network/SDN/Frr.pm;h=b607b32c248db1147b458fbfe3f32a94297d2c53;hb=HEAD#l117


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Julien OHAYON]

Maybe I’m overcomplicating things and there’s actually a much simpler solution.

My concern is during the migration: if I configure OSPF on a node running v9, will this have an impact on the nodes still running v8?



I’m referring to the entire FRR part about OSPF.



Thanks


> Le 8 oct. 2025 à 11:34, Stefan Hanreich <s.hanreich@proxmox.com> a écrit :
> 
> On 10/7/25 5:58 PM, Julien OHAYON wrote:
>> Thanks for your reply. Unfortunately, when we apply the SDN configuration, it modifies the daemons file and sets it to “no”, since we haven’t configured OSPF directly in Proxmox. That’s the file I would like to prevent from being modified.
> 
> Ah sorry, I was only thinking of the upgrade process itself. In your
> particular scenario (having an EVPN zone configured as well) I'm afraid
> there's currently no easy way of preventing the daemons file from
> getting overwritten by the SDN module.
> 
> The defaults are set in the frr config generation module [1] and
> currently not overridable. I guess we'd have to expose a way of setting
> the default value for those daemons to 1 instead of 0.
> 
> Another way would be trying to detect if the status of the daemons needs
> to change compared to the previous SDN configuration and only re-write
> the daemons file in the case it really *needs* to change instead of
> every time. But that is probably a bit more tricky.
> 
> Could you open a feature request on our Bugzilla?
> 
> [1]
> https://git.proxmox.com/?p=pve-network.git;a=blob;f=src/PVE/Network/SDN/Frr.pm;h=b607b32c248db1147b458fbfe3f32a94297d2c53;hb=HEAD#l117
> 
> 

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Flavio Visentin via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 5135 bytes --]

From: Flavio Visentin <flavio-pve@zipman.it>
To: pve-user@lists.proxmox.com
Subject: Re: [PVE-User] OSPF when migrating from v8 to v9
Date: Wed, 8 Oct 2025 20:24:34 +0200
Message-ID: <0ae0296c-db0f-4924-a9ab-f517dc9230e9@zipman.it>

Hi,

On 10/7/25 17:58, Julien OHAYON wrote:
 > Thanks for your reply. Unfortunately, when we apply the SDN
 > configuration, it modifies the daemons file and sets it to “no”, since
 > we haven’t configured OSPF directly in Proxmox. That’s the file I
 > would like to prevent from being modified.


Did you try to make the file /etc/frr/daemons immutable during SDN enabling?

chattr +i /etc/frr/daemons

Maybe it fails with error, but giving a try doesn't cost anything.
Then you can configure the routing and finally remove the immutable 
attribute.

Flavio



[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Stefan Hanreich]

On 10/8/25 12:47 PM, Julien OHAYON wrote:
> My concern is during the migration: if I configure OSPF on a node running v9, will this have an impact on the nodes still running v8?

This one is a bit tricky to answer because of a change in the reload
endpoint between 8 and 9. The behavior depends on certain factors (which
node initiates the reload, how your SDN configuration looks like, ...).

This is due to the introduction of an additional API parameter that is
incompatible with PVE 8.x instances. For that reason, we recommend only
mixing major versions in a cluster while performing an upgrade, not for
prolonged periods of time. We do *not* recommend doing any configuration
changes while your cluster is on mixed major versions. Problems like
this can occur due to changes between major versions and we cannot
guarantee correct behavior while your cluster is running on mixed major
versions.


That being said, upgrading from 8 to 9 alone should not cause your
custom /etc/frr/daemons file to get overwritten - since the SDN stack
only writes to it when re-generating the configuration (= applying the
SDN configuration).

If you upgrade from an earlier FRR version (8), then you would get a
prompt that the daemons file changed, asking you to review you the
changes - like so:

Configuration file '/etc/frr/daemons'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** daemons (Y/I/N/O/D/Z) [default=N] ?

Upgrading from 10.2 (with ospfd=yes) does not cause a prompt to appear
and the daemons file stays untouched.


I've also verified this right now by upgrading a PVE 8 instance with
ospfd=yes in its daemon file. Upgrading itself did not overwrite the
directive in the daemons file - only reapplying the SDN configuration
does set it to ospfd=no.

So, as long as you do not re-apply the SDN configuration in any way,
your OSPF directive in the daemons file stays untouched. On PVE9, when
applying a SDN configuration that contains either a fabric or an EVPN
controller will overwrite the FRR configuration (including the daemons
file).


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Julien OHAYON]

Yes, indeed, during the upgrade nothing changes. Unfortunately, we will have to migrate over several days, and above all, there is still a huge unknown that we would have liked to be able to control: how the switch to the new configuration will happen.



What we would have liked (and I don’t think I’m the only one in this case) is to be able to apply the configuration on the new nodes running v9, without it affecting those still in v8. Because if I have to migrate all my nodes and then press the button afterwards thinking everything will work fine… No, that’s not ideal.



Unfortunately, with network interactions — and even more so interactions with routing protocols deeply embedded in the infrastructure — you want to have control over how the switchover will occur.



For now, I don’t see how to migrate to v9 without risking major downtime (a lab will not show the exact behavior we can expect in this type of SDN infrastructure).


> Le 9 oct. 2025 à 14:17, Stefan Hanreich <s.hanreich@proxmox.com> a écrit :
> 
> On 10/8/25 12:47 PM, Julien OHAYON wrote:
>> My concern is during the migration: if I configure OSPF on a node running v9, will this have an impact on the nodes still running v8?
> 
> This one is a bit tricky to answer because of a change in the reload
> endpoint between 8 and 9. The behavior depends on certain factors (which
> node initiates the reload, how your SDN configuration looks like, ...).
> 
> This is due to the introduction of an additional API parameter that is
> incompatible with PVE 8.x instances. For that reason, we recommend only
> mixing major versions in a cluster while performing an upgrade, not for
> prolonged periods of time. We do *not* recommend doing any configuration
> changes while your cluster is on mixed major versions. Problems like
> this can occur due to changes between major versions and we cannot
> guarantee correct behavior while your cluster is running on mixed major
> versions.
> 
> 
> That being said, upgrading from 8 to 9 alone should not cause your
> custom /etc/frr/daemons file to get overwritten - since the SDN stack
> only writes to it when re-generating the configuration (= applying the
> SDN configuration).
> 
> If you upgrade from an earlier FRR version (8), then you would get a
> prompt that the daemons file changed, asking you to review you the
> changes - like so:
> 
> Configuration file '/etc/frr/daemons'
> ==> Modified (by you or by a script) since installation.
> ==> Package distributor has shipped an updated version.
>   What would you like to do about it ?  Your options are:
>    Y or I  : install the package maintainer's version
>    N or O  : keep your currently-installed version
>      D     : show the differences between the versions
>      Z     : start a shell to examine the situation
> The default action is to keep your current version.
> *** daemons (Y/I/N/O/D/Z) [default=N] ?
> 
> Upgrading from 10.2 (with ospfd=yes) does not cause a prompt to appear
> and the daemons file stays untouched.
> 
> 
> I've also verified this right now by upgrading a PVE 8 instance with
> ospfd=yes in its daemon file. Upgrading itself did not overwrite the
> directive in the daemons file - only reapplying the SDN configuration
> does set it to ospfd=no.
> 
> So, as long as you do not re-apply the SDN configuration in any way,
> your OSPF directive in the daemons file stays untouched. On PVE9, when
> applying a SDN configuration that contains either a fabric or an EVPN
> controller will overwrite the FRR configuration (including the daemons
> file).
> 
> 

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Stefan Hanreich]

On 10/10/25 7:55 AM, Julien OHAYON wrote:
> Yes, indeed, during the upgrade nothing changes. Unfortunately, we will have to migrate over several days, and above all, there is still a huge unknown that we would have liked to be able to control: how the switch to the new configuration will happen.
> 
> 
> 
> What we would have liked (and I don’t think I’m the only one in this case) is to be able to apply the configuration on the new nodes running v9, without it affecting those still in v8. Because if I have to migrate all my nodes and then press the button afterwards thinking everything will work fine… No, that’s not ideal.
> 
> 
> 
> Unfortunately, with network interactions — and even more so interactions with routing protocols deeply embedded in the infrastructure — you want to have control over how the switchover will occur.
> 
> 
> 
> For now, I don’t see how to migrate to v9 without risking major downtime (a lab will not show the exact behavior we can expect in this type of SDN infrastructure).


Understandable, I took a closer look with a colleague today and we found
a way that should hopefully work for you.


What should work as a workaround is setting 'ospfd=yes' in
/etc/default/frr - which provides a way to override the /etc/frr/daemons
file. While that file (/etc/default/frr) is deprecated, it is still read
and respected by FRR and therefore provides a way of overriding the
value independent of our tooling. This prevents the OSPF daemon from
getting disabled on applying the SDN configuration, since our tooling
doesn't touch it.

After you are finished upgrading everything to 9, you can then test
migrating to the SDN fabrics by freeing up one node in your cluster
(migrate everything away, rename the frr.conf.local, ...) and define a
new fabric that only contains the node you want to use for testing.
Applying the SDN configuration then leaves the OSPF configuration for
all other nodes intact, while configuring OSPF using the Fabrics on the
one node that is part of the fabric. Alternatively, a virtualized
Proxmox VE instance could be used for testing the fabrics specifically.


If that test is successful, you can start migrating the other nodes over
one by one.

I tested this procedure on a cluster here locally and it worked
perfectly here. It would still be advisable to test this procedure on a
test cluster (can be virtualized) before you proceed on your production
cluster.


Kind Regards
Stefan


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] OSPF when migrating from v8 to v9

[Julien OHAYON]

Hi Stefan

Thank you for this workaround.

I’m testing this in the beginning of the week.

Thanks

> Le 10 oct. 2025 à 16:34, Stefan Hanreich <s.hanreich@proxmox.com> a écrit :
> 
> On 10/10/25 7:55 AM, Julien OHAYON wrote:
>> Yes, indeed, during the upgrade nothing changes. Unfortunately, we will have to migrate over several days, and above all, there is still a huge unknown that we would have liked to be able to control: how the switch to the new configuration will happen.
>> 
>> 
>> 
>> What we would have liked (and I don’t think I’m the only one in this case) is to be able to apply the configuration on the new nodes running v9, without it affecting those still in v8. Because if I have to migrate all my nodes and then press the button afterwards thinking everything will work fine… No, that’s not ideal.
>> 
>> 
>> 
>> Unfortunately, with network interactions — and even more so interactions with routing protocols deeply embedded in the infrastructure — you want to have control over how the switchover will occur.
>> 
>> 
>> 
>> For now, I don’t see how to migrate to v9 without risking major downtime (a lab will not show the exact behavior we can expect in this type of SDN infrastructure).
> 
> 
> Understandable, I took a closer look with a colleague today and we found
> a way that should hopefully work for you.
> 
> 
> What should work as a workaround is setting 'ospfd=yes' in
> /etc/default/frr - which provides a way to override the /etc/frr/daemons
> file. While that file (/etc/default/frr) is deprecated, it is still read
> and respected by FRR and therefore provides a way of overriding the
> value independent of our tooling. This prevents the OSPF daemon from
> getting disabled on applying the SDN configuration, since our tooling
> doesn't touch it.
> 
> After you are finished upgrading everything to 9, you can then test
> migrating to the SDN fabrics by freeing up one node in your cluster
> (migrate everything away, rename the frr.conf.local, ...) and define a
> new fabric that only contains the node you want to use for testing.
> Applying the SDN configuration then leaves the OSPF configuration for
> all other nodes intact, while configuring OSPF using the Fabrics on the
> one node that is part of the fabric. Alternatively, a virtualized
> Proxmox VE instance could be used for testing the fabrics specifically.
> 
> 
> If that test is successful, you can start migrating the other nodes over
> one by one.
> 
> I tested this procedure on a cluster here locally and it worked
> perfectly here. It would still be advisable to test this procedure on a
> test cluster (can be virtualized) before you proceed on your production
> cluster.
> 
> 
> Kind Regards
> Stefan

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user