[PATCH cluster/docs 0/3] create new corosync clusters with lower token coefficient

[PATCH cluster/docs 0/3] create new corosync clusters with lower token coefficient

[Friedrich Weber]

corosync calculates certain timeouts based on the cluster size. As a result,
after a node failure in a cluster with default corosync settings and more than
~30 nodes, the time until corosync reestablishes a membership can get close to
or exceed 50-60s. In an HA cluster, this can be too long because the watchdog
may start fencing nodes after ~60 seconds without a membership.

To avoid this, create new corosync clusters with smaller timeouts. See patch #2
for more details.

In a future patch series, we may also want to detect problematic settings at
certain points and notify the user accordingly (e.g. when adding a new node or
in `pvecm status`). This will also benefit users with existing clusters with
more than ~30 nodes.

pve-cluster:

Friedrich Weber (2):
  corosync: create config: allow setting token coefficient
  api: cluster config: create new clusters with lower token coefficient

 src/PVE/API2/ClusterConfig.pm | 9 +++++++++
 src/PVE/Corosync.pm           | 3 +++
 2 files changed, 12 insertions(+)


pve-docs:

Friedrich Weber (1):
  pvecm: config: document how to change the token coefficient

 pvecm.adoc | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)


Summary over all repositories:
  3 files changed, 29 insertions(+), 0 deletions(-)

-- 
Generated by git-murpp 0.8.1

[PATCH pve-cluster 1/2] corosync: create config: allow setting token coefficient

[Friedrich Weber]

By default, corosync calculates several timeouts based on the cluster
size, in particular the token timeout, which is a fixed base value
plus the number of nodes (minus 2) multipled by a factor called the
"token coefficient" (with a default of 650ms) [1].

New clusters should be created with a lower token coefficient. Thus,
allow to pass a custom token coefficient when creating the
configuration for a new corosync cluster.

[1] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#token_coefficient

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---
 src/PVE/Corosync.pm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/PVE/Corosync.pm b/src/PVE/Corosync.pm
index f5b10fa..aef0d31 100644
--- a/src/PVE/Corosync.pm
+++ b/src/PVE/Corosync.pm
@@ -274,6 +274,7 @@ sub create_conf {
     my $clustername = $param->{clustername};
     my $nodeid = $param->{nodeid} || 1;
     my $votes = $param->{votes} || 1;
+    my $token_coefficient = $param->{'token-coefficient'};
 
     my $local_ip_address = PVE::Cluster::remote_node_ip($nodename);
 
@@ -324,6 +325,8 @@ sub create_conf {
         $node->{"ring${lnum}_addr"} = $link->{address};
     }
 
+    $totem->{token_coefficient} = $token_coefficient if defined($token_coefficient);
+
     return { main => $conf };
 }
 
-- 
2.47.3

[PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Friedrich Weber]

corosync makes use of several timeouts, in particular the token and
consensus timeouts. The sum of these two timeouts yields the minimum
time a cluster needs to reestablish a membership after a token loss
due to a complete node failure.

By default, corosync sets the timeouts based on the cluster size [1]:

    token timeout = token + (#nodes - 2) * token_coefficient
    consensus timeout = 1.2 * token timeout

token defaults to 3000ms, token_coefficient defaults to 650ms.

With more than ~30 nodes in the default settings, the sum of token and
consensus timeouts gets close to or exceeds 50-60s. As a result, after
a token loss due to a complete node failure in an HA cluster, the
watchdog may fence nodes because it takes too long to reestablish a
new membership and quorum.

One way to avoid this is to lower the sum of the token and consensus
timeouts. The consensus timeout is intentionally slightly larger than
the token timeout [2], so the definition of the consensus timeout in
terms of the token timeout should be preserved. Since it does make
sense to define both timeouts in terms of the cluster size, the most
viable option to lower the timeouts appears to be to adjust the
token_coefficient. Experiments suggest that the default 650ms is
overly conservative considering the low-latency network requirements
postulated in the admin guide [3].

Hence, create new clusters with a default token coefficient of 125ms.
This keeps the sum of token and consensus timeouts well below 50s for
realistic cluster sizes. Users who prefer a larger token coefficient
can manually override the token coefficient when creating a cluster
via pvecm create. The token coefficient can also be changed for an
existing cluster, this will be documented separately.

Note that knet_ping_interval and knet_ping_timeout are derived from
the token timeout, hence, a lower token coefficient will result in
more frequent kronosnet pings and shorter ping timeouts.

With this change, newly created clusters will always set an explicit
token_coefficient in their corosync.conf.

[1] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#token_coefficient
[2] https://github.com/corosync/corosync/commit/b3e19b29058eafc3e808ded7f4c2440c3f957392
[3] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_requirements

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---
 src/PVE/API2/ClusterConfig.pm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/PVE/API2/ClusterConfig.pm b/src/PVE/API2/ClusterConfig.pm
index 1bc7bcf..8df257a 100644
--- a/src/PVE/API2/ClusterConfig.pm
+++ b/src/PVE/API2/ClusterConfig.pm
@@ -111,12 +111,21 @@ __PACKAGE__->register_method({
                 minimum => 1,
                 optional => 1,
             },
+            'token-coefficient' => {
+                type => 'integer',
+                description => "Token coefficient to set in the corosync configuration.",
+                default => 125,
+                minimum => 0,
+                optional => 1,
+            },
         }),
     },
     returns => { type => 'string' },
     code => sub {
         my ($param) = @_;
 
+        $param->{'token-coefficient'} //= 125;
+
         die "cluster config '$clusterconf' already exists\n" if -f $clusterconf;
 
         my $rpcenv = PVE::RPCEnvironment::get();
-- 
2.47.3

Re: [PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Maximiliano Sandoval]

Friedrich Weber <f.weber@proxmox.com> writes:

A comment below

> corosync makes use of several timeouts, in particular the token and
> consensus timeouts. The sum of these two timeouts yields the minimum
> time a cluster needs to reestablish a membership after a token loss
> due to a complete node failure.
>
> By default, corosync sets the timeouts based on the cluster size [1]:
>
>     token timeout = token + (#nodes - 2) * token_coefficient
>     consensus timeout = 1.2 * token timeout
>
> token defaults to 3000ms, token_coefficient defaults to 650ms.
>
> With more than ~30 nodes in the default settings, the sum of token and
> consensus timeouts gets close to or exceeds 50-60s. As a result, after
> a token loss due to a complete node failure in an HA cluster, the
> watchdog may fence nodes because it takes too long to reestablish a
> new membership and quorum.
>
> One way to avoid this is to lower the sum of the token and consensus
> timeouts. The consensus timeout is intentionally slightly larger than
> the token timeout [2], so the definition of the consensus timeout in
> terms of the token timeout should be preserved. Since it does make
> sense to define both timeouts in terms of the cluster size, the most
> viable option to lower the timeouts appears to be to adjust the
> token_coefficient. Experiments suggest that the default 650ms is
> overly conservative considering the low-latency network requirements
> postulated in the admin guide [3].
>
> Hence, create new clusters with a default token coefficient of 125ms.
> This keeps the sum of token and consensus timeouts well below 50s for
> realistic cluster sizes. Users who prefer a larger token coefficient
> can manually override the token coefficient when creating a cluster
> via pvecm create. The token coefficient can also be changed for an
> existing cluster, this will be documented separately.
>
> Note that knet_ping_interval and knet_ping_timeout are derived from
> the token timeout, hence, a lower token coefficient will result in
> more frequent kronosnet pings and shorter ping timeouts.
>
> With this change, newly created clusters will always set an explicit
> token_coefficient in their corosync.conf.
>
> [1] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#token_coefficient
> [2] https://github.com/corosync/corosync/commit/b3e19b29058eafc3e808ded7f4c2440c3f957392
> [3] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_requirements
>
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
>  src/PVE/API2/ClusterConfig.pm | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/src/PVE/API2/ClusterConfig.pm b/src/PVE/API2/ClusterConfig.pm
> index 1bc7bcf..8df257a 100644
> --- a/src/PVE/API2/ClusterConfig.pm
> +++ b/src/PVE/API2/ClusterConfig.pm
> @@ -111,12 +111,21 @@ __PACKAGE__->register_method({
>                  minimum => 1,
>                  optional => 1,
>              },
> +            'token-coefficient' => {
> +                type => 'integer',
> +                description => "Token coefficient to set in the corosync configuration.",
> +                default => 125,
> +                minimum => 0,

>From man 5 corosync.conf's token_coefficient documentation: "This value
can be set to 0 resulting in effective removal of this feature.". If we
want to expose setting this to 0 I would document that it has a special
meaning and what does this entail. I would personally feel more
comfortable setting `minimum => 1` for now instead.

> +                optional => 1,
> +            },
>          }),
>      },
>      returns => { type => 'string' },
>      code => sub {
>          my ($param) = @_;
>  
> +        $param->{'token-coefficient'} //= 125;
> +
>          die "cluster config '$clusterconf' already exists\n" if -f $clusterconf;
>  
>          my $rpcenv = PVE::RPCEnvironment::get();

-- 
Maximiliano

Re: [PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Thomas Lamprecht]

Am 16.02.26 um 17:00 schrieb Maximiliano Sandoval:
>> +            'token-coefficient' => {
>> +                type => 'integer',
>> +                description => "Token coefficient to set in the corosync configuration.",
>> +                default => 125,
>> +                minimum => 0,
>>From man 5 corosync.conf's token_coefficient documentation: "This value
> can be set to 0 resulting in effective removal of this feature.". If we
> want to expose setting this to 0 I would document that it has a special
> meaning and what does this entail. I would personally feel more
> comfortable setting `minimum => 1` for now instead.

At least a "see `man 5 corosync.conf` for details might be nice, adding some
extra hints here, like how it's roughly used and special values, could be
indeed nice too; some of that might be better off in the docs or the
verbose_descriptions property though.

But I'm not so sure about the actual value to the user of restricting this
here? I mean, if we ever would expose this in the UI in some advanced section
then one could show clear hints for such special/odd values and their potential
implications, for the CLI that's mostly the job of the docs and maybe an extra
informal "log" print, but forcing a user editing the corosync.conf manually in
case they want to try this, whyever that might be, seems to rather worsen UX not
improve it.

Re: [PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Maximiliano Sandoval]

Thomas Lamprecht <t.lamprecht@proxmox.com> writes:

> Am 16.02.26 um 17:00 schrieb Maximiliano Sandoval:
>>> +            'token-coefficient' => {
>>> +                type => 'integer',
>>> +                description => "Token coefficient to set in the corosync configuration.",
>>> +                default => 125,
>>> +                minimum => 0,
>>>From man 5 corosync.conf's token_coefficient documentation: "This value
>> can be set to 0 resulting in effective removal of this feature.". If we
>> want to expose setting this to 0 I would document that it has a special
>> meaning and what does this entail. I would personally feel more
>> comfortable setting `minimum => 1` for now instead.
>
> At least a "see `man 5 corosync.conf` for details might be nice, adding some
> extra hints here, like how it's roughly used and special values, could be
> indeed nice too; some of that might be better off in the docs or the
> verbose_descriptions property though.
>
> But I'm not so sure about the actual value to the user of restricting this
> here? I mean, if we ever would expose this in the UI in some advanced section
> then one could show clear hints for such special/odd values and their potential
> implications, for the CLI that's mostly the job of the docs and maybe an extra
> informal "log" print, but forcing a user editing the corosync.conf manually in
> case they want to try this, whyever that might be, seems to rather worsen UX not
> improve it.

>From corosync.conf(5) I wrongly got the feeling that `0` had some
special-casing going on, but it actually does not. The docs just say in
a somewhat verbose fashion that multiplying with zero generally results
in zero.

We discussed this off-list a bit and my suggestion in my other reply,
namely:

"Coefficient used to determine Corosync's token timeout. See the
corosync.conf(5) manual for more details."

is OK.

-- 
Maximiliano

Re: [PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Friedrich Weber]

Thanks for the review!

On 17/02/2026 13:44, Maximiliano Sandoval wrote:
> Thomas Lamprecht <t.lamprecht@proxmox.com> writes:
> 
>> Am 16.02.26 um 17:00 schrieb Maximiliano Sandoval:
>>>> +            'token-coefficient' => {
>>>> +                type => 'integer',
>>>> +                description => "Token coefficient to set in the corosync configuration.",
>>>> +                default => 125,
>>>> +                minimum => 0,
>>> >From man 5 corosync.conf's token_coefficient documentation: "This value
>>> can be set to 0 resulting in effective removal of this feature.". If we
>>> want to expose setting this to 0 I would document that it has a special
>>> meaning and what does this entail. I would personally feel more
>>> comfortable setting `minimum => 1` for now instead.
>>
>> At least a "see `man 5 corosync.conf` for details might be nice, adding some
>> extra hints here, like how it's roughly used and special values, could be
>> indeed nice too; some of that might be better off in the docs or the
>> verbose_descriptions property though.
>>
>> But I'm not so sure about the actual value to the user of restricting this
>> here? I mean, if we ever would expose this in the UI in some advanced section
>> then one could show clear hints for such special/odd values and their potential
>> implications, for the CLI that's mostly the job of the docs and maybe an extra
>> informal "log" print, but forcing a user editing the corosync.conf manually in
>> case they want to try this, whyever that might be, seems to rather worsen UX not
>> improve it.
> 
> From corosync.conf(5) I wrongly got the feeling that `0` had some
> special-casing going on, but it actually does not. The docs just say in
> a somewhat verbose fashion that multiplying with zero generally results
> in zero.
> 
> We discussed this off-list a bit and my suggestion in my other reply,
> namely:
> 
> "Coefficient used to determine Corosync's token timeout. See the
> corosync.conf(5) manual for more details."
> 
> is OK.

Yes, I agree my original description was not that fitting, I can send a
v2 with this updated description.

Re: [PATCH pve-cluster 2/2] api: cluster config: create new clusters with lower token coefficient

[Maximiliano Sandoval]

Friedrich Weber <f.weber@proxmox.com> writes:

> corosync makes use of several timeouts, in particular the token and
> consensus timeouts. The sum of these two timeouts yields the minimum
> time a cluster needs to reestablish a membership after a token loss
> due to a complete node failure.
>
> By default, corosync sets the timeouts based on the cluster size [1]:
>
>     token timeout = token + (#nodes - 2) * token_coefficient
>     consensus timeout = 1.2 * token timeout
>
> token defaults to 3000ms, token_coefficient defaults to 650ms.
>
> With more than ~30 nodes in the default settings, the sum of token and
> consensus timeouts gets close to or exceeds 50-60s. As a result, after
> a token loss due to a complete node failure in an HA cluster, the
> watchdog may fence nodes because it takes too long to reestablish a
> new membership and quorum.
>
> One way to avoid this is to lower the sum of the token and consensus
> timeouts. The consensus timeout is intentionally slightly larger than
> the token timeout [2], so the definition of the consensus timeout in
> terms of the token timeout should be preserved. Since it does make
> sense to define both timeouts in terms of the cluster size, the most
> viable option to lower the timeouts appears to be to adjust the
> token_coefficient. Experiments suggest that the default 650ms is
> overly conservative considering the low-latency network requirements
> postulated in the admin guide [3].
>
> Hence, create new clusters with a default token coefficient of 125ms.
> This keeps the sum of token and consensus timeouts well below 50s for
> realistic cluster sizes. Users who prefer a larger token coefficient
> can manually override the token coefficient when creating a cluster
> via pvecm create. The token coefficient can also be changed for an
> existing cluster, this will be documented separately.
>
> Note that knet_ping_interval and knet_ping_timeout are derived from
> the token timeout, hence, a lower token coefficient will result in
> more frequent kronosnet pings and shorter ping timeouts.
>
> With this change, newly created clusters will always set an explicit
> token_coefficient in their corosync.conf.
>
> [1] https://manpages.debian.org/trixie/corosync/corosync.conf.5.en.html#token_coefficient
> [2] https://github.com/corosync/corosync/commit/b3e19b29058eafc3e808ded7f4c2440c3f957392
> [3] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_network_requirements
>
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
>  src/PVE/API2/ClusterConfig.pm | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/src/PVE/API2/ClusterConfig.pm b/src/PVE/API2/ClusterConfig.pm
> index 1bc7bcf..8df257a 100644
> --- a/src/PVE/API2/ClusterConfig.pm
> +++ b/src/PVE/API2/ClusterConfig.pm
> @@ -111,12 +111,21 @@ __PACKAGE__->register_method({
>                  minimum => 1,
>                  optional => 1,
>              },
> +            'token-coefficient' => {
> +                type => 'integer',
> +                description => "Token coefficient to set in the corosync configuration.",

This description does not help understanding what it does, no more than
its name at least. It would perhaps be preferable to say something along
the lines of:

"Coefficient used to determine Corosync's token timeout. See the
corosync.conf(5) manual for more details."


> +                default => 125,
> +                minimum => 0,
> +                optional => 1,
> +            },
>          }),
>      },
>      returns => { type => 'string' },
>      code => sub {
>          my ($param) = @_;
>  
> +        $param->{'token-coefficient'} //= 125;
> +
>          die "cluster config '$clusterconf' already exists\n" if -f $clusterconf;
>  
>          my $rpcenv = PVE::RPCEnvironment::get();

-- 
Maximiliano

[PATCH pve-docs 1/1] pvecm: config: document how to change the token coefficient

[Friedrich Weber]

New clusters are created with a lower coefficient. Users with existing
clusters or users who prefer a different token coefficient may want to
change the token coefficient. Hence, add a note how to change the
token coefficient to the admin guide.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    - should this already include more details (in which cases changing
      the coefficient may be necessary?) in this first version?
    
    - currently, changing the token_coefficient in /etc/pve/corosync.conf
      without a corosync restart updates token and consensus timeouts, but
      knet_ping_timeout and knet_ping_interval are not updated, this is
      the reason why the patch suggests a corosync restart. I filed a bug
      for this [1] and there is a promising pull request [2] that appears
      to fix the issue. So the question is whether we want to backport [2]
      (when it's merged), and if yes, whether we want to remove the
      restart suggestion from the docs.
    
    [1] https://github.com/corosync/corosync/issues/813
    [2] https://github.com/corosync/corosync/pull/814

 pvecm.adoc | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/pvecm.adoc b/pvecm.adoc
index 0ed1bd2..9513a6c 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -1371,6 +1371,23 @@ systemctl restart corosync
 
 On errors, check the troubleshooting section below.
 
+Changing the token cofficient
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The token coefficient can be configured in the `totem` section in
+`/etc/pve/corosync.conf`. corosync uses the token coefficient to calculate
+several timeouts in relation to the cluster size.footnote:[
+`token_coefficient` in the corosync manual page
+https://manpages.debian.org/stable/corosync/corosync.conf.5.en.html#token_coefficient]
+
+If the token coefficient is not explicitly set, it defaults to 650 milliseconds.
+New clusters are created with a lower token coefficient of 125ms that is
+explicitly set in `/etc/pve/corosync.conf`.
+
+You can change the token coefficient of an existing cluster by
+xref:pvecm_edit_corosync_conf[editing corosync.conf]. For the change to fully
+take effect, you need to restart corosync on all nodes.
+
 Troubleshooting
 ~~~~~~~~~~~~~~~
 
-- 
2.47.3

Re: [PATCH pve-docs 1/1] pvecm: config: document how to change the token coefficient

[Friedrich Weber]

On 12/02/2026 12:59, Friedrich Weber wrote:
> New clusters are created with a lower coefficient. Users with existing
> clusters or users who prefer a different token coefficient may want to
> change the token coefficient. Hence, add a note how to change the
> token coefficient to the admin guide.
> 
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
> 
> Notes:
>     - should this already include more details (in which cases changing
>       the coefficient may be necessary?) in this first version?
>     
>     - currently, changing the token_coefficient in /etc/pve/corosync.conf
>       without a corosync restart updates token and consensus timeouts, but
>       knet_ping_timeout and knet_ping_interval are not updated, this is
>       the reason why the patch suggests a corosync restart. I filed a bug
>       for this [1] and there is a promising pull request [2] that appears
>       to fix the issue. So the question is whether we want to backport [2]
>       (when it's merged), and if yes, whether we want to remove the
>       restart suggestion from the docs.
>     
>     [1] https://github.com/corosync/corosync/issues/813
>     [2] https://github.com/corosync/corosync/pull/814

The PR was already merged, I sent a cherry-pick:
https://lore.proxmox.com/pve-devel/20260213101121.107073-1-f.weber@proxmox.com/T/