Re: [pve-devel] [PATCH docs] pvecm, network: add section on corosync over bonds

[Daniel Herzig <d.herzig@proxmox.com>]

Thanks for documenting this!

I'd even go one step further and discourage the use of bonds in one of 
the sections of 'Cluster Network' in `pvecm.adoc` as well. Best with a 
link to the new 'Corosync over Bonds' section, with your decent 
explanation. That way it would be more difficult to miss for hasty 
readers (which would be a pity).

On 7/21/25 17:27, Friedrich Weber wrote:
> Testing has shown that running corosync (only) over a bond can be
> problematic in some failure scenarios and for certain bond modes. The
> documentation only discourages bonds for corosync because corosync can
> switch between available networks itself, but does not mention other
> caveats when using bonds for corosync.
>
> Hence, extend the documentation with recommendations and caveats
> regarding bonds for corosync.
>
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---
>
> Notes:
>      Aaron suggested we could expose the bond-lacp-rate in the GUI to
>      make it easier to change the setting on the PVE side. I'd open a
>      feature report for this.
>
>   pve-network.adoc |  4 +++-
>   pvecm.adoc       | 42 +++++++++++++++++++++++++++++++++++++++---
>   2 files changed, 42 insertions(+), 4 deletions(-)
>
> diff --git a/pve-network.adoc b/pve-network.adoc
> index 2dec882..b361f97 100644
> --- a/pve-network.adoc
> +++ b/pve-network.adoc
> @@ -495,7 +495,9 @@ use the active-backup mode.
>   
>   For the cluster network (Corosync) we recommend configuring it with multiple
>   networks. Corosync does not need a bond for network redundancy as it can switch
> -between networks by itself, if one becomes unusable.
> +between networks by itself, if one becomes unusable. Some bond modes are known
> +to be problematic for Corosync, see
> +xref:pvecm_corosync_over_bonds[Corosync over Bonds].
>   
>   The following bond configuration can be used as distributed/shared
>   storage network. The benefit would be that you get more speed and the
> diff --git a/pvecm.adoc b/pvecm.adoc
> index 312a26f..1045abb 100644
> --- a/pvecm.adoc
> +++ b/pvecm.adoc
> @@ -90,15 +90,51 @@ another link on a different physical network. This enables Corosync to keep the
>   cluster communication alive should the dedicated network be down.
>   +
>   NOTE: A single link backed by a bond is not enough to provide Corosync
> -redundancy. When a bonded interface fails and Corosync cannot fall back to
> -another link, it can lead to  asymmetric communication in the cluster, which in
> -turn can lead to the cluster losing quorum.
> +redundancy. See xref:pvecm_corosync_over_bonds[Corosync over Bonds].
>   
>   * The root password of a cluster node is required for adding nodes.
>   
>   * Online migration of virtual machines is only supported when nodes have CPUs
>     from the same vendor. It might work otherwise, but this is never guaranteed.
>   
> +[[pvecm_corosync_over_bonds]]
> +Corosync over Bonds
> +~~~~~~~~~~~~~~~~~~~
> +
> +Using a xref:sysadmin_network_bond[bond] as the only Corosync link can be
> +problematic in certain failure scenarios. If one of the bonded interfaces fails
> +and stops transmitting packets, but its link state stays up, some bond modes
> +may cause a state of asymmetric connectivity where cluster nodes can only
> +communicate with different subsets of other nodes. In case of asymmetric
> +connectivity, Corosync may not be able to form a stable quorum in the cluster.
> +If this state persists and HA is enabled, nodes may fence themselves, even if
> +their respective bond is still fully functioning. In the worst case, the whole
> +cluster may fence itself.
> +
> +For this reason, our recommendations are as follows.
> +
> +* We recommend a dedicated physical NIC for the primary Corosync link. Bonds
> +  can be used as additional links for increased redundancy.
> +
> +* We *advise against* using bond modes *balance-rr*, *balance-xor*,
> +  *balance-tlb*, or *balance-alb* for Corosync traffic. As explained above,
> +  they can cause asymmetric connectivity in certain failure scenarios.
> +
> +* *IEEE 802.3ad (LACP)*: This bond mode can cause asymmetric connectivity in
> +  certain failure scenarios as explained above, but it can recover from this
> +  state, as each side can stop using a bonded interface if it has not received
> +  three LACPDUs in a row. However, with default settings, LACPDUs are only sent
> +  every 30 seconds, yielding a failover time of 90 seconds. This is too long,
> +  as nodes with HA resources will fence themselves already after roughly one
> +  minute without a stable quorum. If LACP bonds are used for corosync traffic,
> +  we recommend setting `bond-lacp-rate fast` *on the Proxmox VE node and the
> +  switch*! Setting this option on one side requests the other side to send an
> +  LACPDU every second, which reduces the failover time in the scenario above to
> +  3 seconds.
> +
> +* Bond mode *active-backup* will not cause asymmetric connectivity in the
> +  failure scenario described above, but the affected node may lose connection
> +  to the cluster and, if HA is enabled, fence itself.
>   
>   Preparing Nodes
>   ---------------


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel