[pve-devel] [PATCH docs] pvecm, network: add section on corosync over bonds

[pve-devel] [PATCH docs] pvecm, network: add section on corosync over bonds

[Friedrich Weber]

Testing has shown that running corosync (only) over a bond can be
problematic in some failure scenarios and for certain bond modes. The
documentation only discourages bonds for corosync because corosync can
switch between available networks itself, but does not mention other
caveats when using bonds for corosync.

Hence, extend the documentation with recommendations and caveats
regarding bonds for corosync.

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    Aaron suggested we could expose the bond-lacp-rate in the GUI to
    make it easier to change the setting on the PVE side. I'd open a
    feature report for this.

 pve-network.adoc |  4 +++-
 pvecm.adoc       | 42 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/pve-network.adoc b/pve-network.adoc
index 2dec882..b361f97 100644
--- a/pve-network.adoc
+++ b/pve-network.adoc
@@ -495,7 +495,9 @@ use the active-backup mode.
 
 For the cluster network (Corosync) we recommend configuring it with multiple
 networks. Corosync does not need a bond for network redundancy as it can switch
-between networks by itself, if one becomes unusable.
+between networks by itself, if one becomes unusable. Some bond modes are known
+to be problematic for Corosync, see
+xref:pvecm_corosync_over_bonds[Corosync over Bonds].
 
 The following bond configuration can be used as distributed/shared
 storage network. The benefit would be that you get more speed and the
diff --git a/pvecm.adoc b/pvecm.adoc
index 312a26f..1045abb 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -90,15 +90,51 @@ another link on a different physical network. This enables Corosync to keep the
 cluster communication alive should the dedicated network be down.
 +
 NOTE: A single link backed by a bond is not enough to provide Corosync
-redundancy. When a bonded interface fails and Corosync cannot fall back to
-another link, it can lead to  asymmetric communication in the cluster, which in
-turn can lead to the cluster losing quorum.
+redundancy. See xref:pvecm_corosync_over_bonds[Corosync over Bonds].
 
 * The root password of a cluster node is required for adding nodes.
 
 * Online migration of virtual machines is only supported when nodes have CPUs
   from the same vendor. It might work otherwise, but this is never guaranteed.
 
+[[pvecm_corosync_over_bonds]]
+Corosync over Bonds
+~~~~~~~~~~~~~~~~~~~
+
+Using a xref:sysadmin_network_bond[bond] as the only Corosync link can be
+problematic in certain failure scenarios. If one of the bonded interfaces fails
+and stops transmitting packets, but its link state stays up, some bond modes
+may cause a state of asymmetric connectivity where cluster nodes can only
+communicate with different subsets of other nodes. In case of asymmetric
+connectivity, Corosync may not be able to form a stable quorum in the cluster.
+If this state persists and HA is enabled, nodes may fence themselves, even if
+their respective bond is still fully functioning. In the worst case, the whole
+cluster may fence itself.
+
+For this reason, our recommendations are as follows.
+
+* We recommend a dedicated physical NIC for the primary Corosync link. Bonds
+  can be used as additional links for increased redundancy.
+
+* We *advise against* using bond modes *balance-rr*, *balance-xor*,
+  *balance-tlb*, or *balance-alb* for Corosync traffic. As explained above,
+  they can cause asymmetric connectivity in certain failure scenarios.
+
+* *IEEE 802.3ad (LACP)*: This bond mode can cause asymmetric connectivity in
+  certain failure scenarios as explained above, but it can recover from this
+  state, as each side can stop using a bonded interface if it has not received
+  three LACPDUs in a row. However, with default settings, LACPDUs are only sent
+  every 30 seconds, yielding a failover time of 90 seconds. This is too long,
+  as nodes with HA resources will fence themselves already after roughly one
+  minute without a stable quorum. If LACP bonds are used for corosync traffic,
+  we recommend setting `bond-lacp-rate fast` *on the Proxmox VE node and the
+  switch*! Setting this option on one side requests the other side to send an
+  LACPDU every second, which reduces the failover time in the scenario above to
+  3 seconds.
+
+* Bond mode *active-backup* will not cause asymmetric connectivity in the
+  failure scenario described above, but the affected node may lose connection
+  to the cluster and, if HA is enabled, fence itself.
 
 Preparing Nodes
 ---------------
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs] pvecm, network: add section on corosync over bonds

[Daniel Herzig]

Thanks for documenting this!

I'd even go one step further and discourage the use of bonds in one of 
the sections of 'Cluster Network' in `pvecm.adoc` as well. Best with a 
link to the new 'Corosync over Bonds' section, with your decent 
explanation. That way it would be more difficult to miss for hasty 
readers (which would be a pity).

On 7/21/25 17:27, Friedrich Weber wrote:
> Testing has shown that running corosync (only) over a bond can be
> problematic in some failure scenarios and for certain bond modes. The
> documentation only discourages bonds for corosync because corosync can
> switch between available networks itself, but does not mention other
> caveats when using bonds for corosync.
>
> Hence, extend the documentation with recommendations and caveats
> regarding bonds for corosync.
>
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
>
> Notes:
>      Aaron suggested we could expose the bond-lacp-rate in the GUI to
>      make it easier to change the setting on the PVE side. I'd open a
>      feature report for this.
>
>   pve-network.adoc |  4 +++-
>   pvecm.adoc       | 42 +++++++++++++++++++++++++++++++++++++++---
>   2 files changed, 42 insertions(+), 4 deletions(-)
>
> diff --git a/pve-network.adoc b/pve-network.adoc
> index 2dec882..b361f97 100644
> --- a/pve-network.adoc
> +++ b/pve-network.adoc
> @@ -495,7 +495,9 @@ use the active-backup mode.
>   
>   For the cluster network (Corosync) we recommend configuring it with multiple
>   networks. Corosync does not need a bond for network redundancy as it can switch
> -between networks by itself, if one becomes unusable.
> +between networks by itself, if one becomes unusable. Some bond modes are known
> +to be problematic for Corosync, see
> +xref:pvecm_corosync_over_bonds[Corosync over Bonds].
>   
>   The following bond configuration can be used as distributed/shared
>   storage network. The benefit would be that you get more speed and the
> diff --git a/pvecm.adoc b/pvecm.adoc
> index 312a26f..1045abb 100644
> --- a/pvecm.adoc
> +++ b/pvecm.adoc
> @@ -90,15 +90,51 @@ another link on a different physical network. This enables Corosync to keep the
>   cluster communication alive should the dedicated network be down.
>   +
>   NOTE: A single link backed by a bond is not enough to provide Corosync
> -redundancy. When a bonded interface fails and Corosync cannot fall back to
> -another link, it can lead to  asymmetric communication in the cluster, which in
> -turn can lead to the cluster losing quorum.
> +redundancy. See xref:pvecm_corosync_over_bonds[Corosync over Bonds].
>   
>   * The root password of a cluster node is required for adding nodes.
>   
>   * Online migration of virtual machines is only supported when nodes have CPUs
>     from the same vendor. It might work otherwise, but this is never guaranteed.
>   
> +[[pvecm_corosync_over_bonds]]
> +Corosync over Bonds
> +~~~~~~~~~~~~~~~~~~~
> +
> +Using a xref:sysadmin_network_bond[bond] as the only Corosync link can be
> +problematic in certain failure scenarios. If one of the bonded interfaces fails
> +and stops transmitting packets, but its link state stays up, some bond modes
> +may cause a state of asymmetric connectivity where cluster nodes can only
> +communicate with different subsets of other nodes. In case of asymmetric
> +connectivity, Corosync may not be able to form a stable quorum in the cluster.
> +If this state persists and HA is enabled, nodes may fence themselves, even if
> +their respective bond is still fully functioning. In the worst case, the whole
> +cluster may fence itself.
> +
> +For this reason, our recommendations are as follows.
> +
> +* We recommend a dedicated physical NIC for the primary Corosync link. Bonds
> +  can be used as additional links for increased redundancy.
> +
> +* We *advise against* using bond modes *balance-rr*, *balance-xor*,
> +  *balance-tlb*, or *balance-alb* for Corosync traffic. As explained above,
> +  they can cause asymmetric connectivity in certain failure scenarios.
> +
> +* *IEEE 802.3ad (LACP)*: This bond mode can cause asymmetric connectivity in
> +  certain failure scenarios as explained above, but it can recover from this
> +  state, as each side can stop using a bonded interface if it has not received
> +  three LACPDUs in a row. However, with default settings, LACPDUs are only sent
> +  every 30 seconds, yielding a failover time of 90 seconds. This is too long,
> +  as nodes with HA resources will fence themselves already after roughly one
> +  minute without a stable quorum. If LACP bonds are used for corosync traffic,
> +  we recommend setting `bond-lacp-rate fast` *on the Proxmox VE node and the
> +  switch*! Setting this option on one side requests the other side to send an
> +  LACPDU every second, which reduces the failover time in the scenario above to
> +  3 seconds.
> +
> +* Bond mode *active-backup* will not cause asymmetric connectivity in the
> +  failure scenario described above, but the affected node may lose connection
> +  to the cluster and, if HA is enabled, fence itself.
>   
>   Preparing Nodes
>   ---------------


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs] pvecm, network: add section on corosync over bonds

[Friedrich Weber]

Thanks for taking a look at this!

On 24/07/2025 10:22, Daniel Herzig wrote:
> Thanks for documenting this!
> 
> I'd even go one step further and discourage the use of bonds in one of
> the sections of 'Cluster Network' in `pvecm.adoc` as well. Best with a
> link to the new 'Corosync over Bonds' section, with your decent
> explanation. That way it would be more difficult to miss for hasty
> readers (which would be a pity).

Actually, now that I think about it, maybe the "Corosync over Bonds"
section would be better placed as a subsection under "Cluster Networks",
seems a bit weird to have it under "Requirements". I'll try it out and
send a v2.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] superseded: [PATCH docs] pvecm, network: add section on corosync over bonds

[Friedrich Weber]

Superseded by:

https://lore.proxmox.com/pve-devel/20250725113922.99886-1-f.weber@proxmox.com/T/



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel