[PVE-User] Experimenting with bond on a non-LACP switch...

[PVE-User] Experimenting with bond on a non-LACP switch...

[Marco Gaiarin]

I'm doing some experimentation on a switch that seems does not support LACP,
even thus claim that; is a Netgear GS724Tv2:

	https://www.downloads.netgear.com/files/GDC/GS724Tv2/enus_ds_gs724t.pdf

data sheet say:

	Port Trunking - Manual as per IEEE802.3ad Link Aggregation

and 'IEEE802.3ad Link Aggregation' is LACP, right?


Anyway, i'm experimenting a bit with other bonding mode, having (un)expected
results and troubles, but in:

	https://pve.proxmox.com/wiki/Network_Configuration#_linux_bond

i've stumble upon that sentence:

	If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported.

What exactly mean?! Thanks.

-- 
  Molti italiani sognavano di vedere Berlusconi in un cellulare,
  prima o poi...			(Stardust®, da i.n.n-a)

Re: [PVE-User] Experimenting with bond on a non-LACP switch...

[Laurent Dumont]

It's not made very clear from the documentation. I assume there are good
technical reasons why the cluster traffic would be impacted.

Afaik, proxmox leverages corosync which can leverage multicast for the
cluster checks. I don't think it can be badly impacted by LACP but
something to keep in mind.

There is this old thread with a similar discussion :

https://forum.proxmox.com/threads/cluster-lacp.90668/

On Sat, May 21, 2022 at 4:10 AM Marco Gaiarin <gaio@lilliput.linux.it>
wrote:

>
> I'm doing some experimentation on a switch that seems does not support
> LACP,
> even thus claim that; is a Netgear GS724Tv2:
>
>
> https://www.downloads.netgear.com/files/GDC/GS724Tv2/enus_ds_gs724t.pdf
>
> data sheet say:
>
>         Port Trunking - Manual as per IEEE802.3ad Link Aggregation
>
> and 'IEEE802.3ad Link Aggregation' is LACP, right?
>
>
> Anyway, i'm experimenting a bit with other bonding mode, having
> (un)expected
> results and troubles, but in:
>
>         https://pve.proxmox.com/wiki/Network_Configuration#_linux_bond
>
> i've stumble upon that sentence:
>
>         If you intend to run your cluster network on the bonding
> interfaces, then you have to use active-passive mode on the bonding
> interfaces, other modes are unsupported.
>
> What exactly mean?! Thanks.
>
> --
>   Molti italiani sognavano di vedere Berlusconi in un cellulare,
>   prima o poi...                        (Stardust®, da i.n.n-a)
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>

Re: [PVE-User] Experimenting with bond on a non-LACP switch...

[Wolf Noble]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]


Good Catch Marco!

I'd not seen that when I read through that page, but I just re-read it.... My read is that it can introduce ODD edge-case complications.

my synthesis of this information is  outlined here. I encourage anyone to correct my misunderstandings. 

network abstraction gets complicated QUICKLY.  Network gear vendors implement their support for the different bonding modes in subtly different ways.
Firewalls have their own quirks.

abstractions on top of abstractions on top of abstractions on top of abstractions on top of .... okay you get the point.


we want to avoid asymmetric pathing where possible, because stuff gets quirky and edge-casey quickly. the fewer explicitly supported virtual topologies, the fewer scenarios the engineering teams need to scrutinize the COMPLEX edge case behaviors of, resulting in a better experience for EVERYONE.... heres what I mean:

LACP: 
This is a pretty well known and consistently implemented aggregation mechanism. the behaviors of network interfaces and switching hardware that are involved are pretty consistent. 

This GENERALLY works fine. the only time I've seen it get a little wonky is LACP across switch chassis behavior can be odd..
                       

[-- Attachment #2: Type: text/plain, Size: 2938 bytes --]




When node 2 sends traffic destined for node 1 via lacp_b.1 ... it may traverse the trunked interconnect, it may not. (it depends)



active-backup:  

I talk, and listen on link0... if link 0 goes down, I switchover... I don't really pay attention to linkl1 otherwise.

active-passive:
  I talk on link0. I listen on link0 and link1.

The downside that I've seen here:


Arp caching can get wonky, and packets that SHOULD be directed to node0 link0 
        get directed to node0 link1... or  sometimes packets directed to node0.link0 have a destination mac address of the hwaddr of link1 and so get delivered to link1 ...  

There MAY be some oddities that manifest with this configuration. 

depending on (node scope configuration) sysctl settings, node0 could just ignore those packets, resulting in weird behavior

with the various balance algorithms nodes will see a different hardware addresses for each other, again, this isn't *USUALLY* a problem, but there are still some dragons that lurk within the trunking/bonding code... hardware checksumming can get whacky...  especially when VLANs get mixed in... 

My gut tells me that the main reason for this advise is that using LACP or active/backup provides sufficient durability while introducing as little edge-case wonky as possible, which generally speaking is a GoodThing™️ when it comes to intra-cluster-comms.


I could be wrong, so don't take this as gospel... if anyone has a better explanation, or can point out my flawed logic, by all means, chime in! :) 


Hope my understanding HELPS... if it doesn't, throw it away and ignore it ;)

🐺W



This message created and transmitted using 100% recycled electrons.

> On May 21, 2022, at 03:11, Marco Gaiarin <gaio@lilliput.linux.it> wrote:
> 
> 
> I'm doing some experimentation on a switch that seems does not support LACP,
> even thus claim that; is a Netgear GS724Tv2:
> 
>    https://www.downloads.netgear.com/files/GDC/GS724Tv2/enus_ds_gs724t.pdf
> 
> data sheet say:
> 
>    Port Trunking - Manual as per IEEE802.3ad Link Aggregation
> 
> and 'IEEE802.3ad Link Aggregation' is LACP, right?
> 
> 
> Anyway, i'm experimenting a bit with other bonding mode, having (un)expected
> results and troubles, but in:
> 
>    https://pve.proxmox.com/wiki/Network_Configuration#_linux_bond
> 
> i've stumble upon that sentence:
> 
>    If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported.
> 
> What exactly mean?! Thanks.
> 
> -- 
>  Molti italiani sognavano di vedere Berlusconi in un cellulare,
>  prima o poi...            (Stardust®, da i.n.n-a)
> 
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 

Re: [PVE-User] Experimenting with bond on a non-LACP switch...

[Marco Gaiarin]

Mandi! Laurent Dumont
  In chel di` si favelave...

> There is this old thread with a similar discussion :
> https://forum.proxmox.com/threads/cluster-lacp.90668/

Apart that i have other clusters where corosync run against LACP bonds
without roubles, i've a little simpler question:

>>         If you intend to run your cluster network on the bonding
>> interfaces, then you have to use active-passive mode on the bonding
>> interfaces, other modes are unsupported.

What is 'active-passive'?! Is the same of 'active-backup'? seems a
terminology inconsistency to me...

-- 
  Se non trovi nessuno vuol dire che siamo scappati alle sei-shell (bash,
  tcsh,csh...)							(Possi)

Re: [PVE-User] Experimenting with bond on a non-LACP switch...

[Wolf Noble]

I was condensing several (i emit traffic on one interface but listen on all bond members) into ‘active-passive’

this: 
https://help.ubuntu.com/community/UbuntuBonding

does a better job explaining the different modes… 

hope that helps?

🤡 ( sorry for the confusion )

W

[= The contents of this message have been written, read, processed, erased, sorted, sniffed, compressed, rewritten, misspelled, overcompensated, lost, found, and most importantly delivered entirely with recycled electrons =]

> On May 24, 2022, at 16:40, Marco Gaiarin <gaio@lilliput.linux.it> wrote:
> 
> Mandi! Laurent Dumont
>  In chel di` si favelave...
> 
>> There is this old thread with a similar discussion :
>> https://forum.proxmox.com/threads/cluster-lacp.90668/
> 
> Apart that i have other clusters where corosync run against LACP bonds
> without roubles, i've a little simpler question:
> 
>>>        If you intend to run your cluster network on the bonding
>>> interfaces, then you have to use active-passive mode on the bonding
>>> interfaces, other modes are unsupported.
> 
> What is 'active-passive'?! Is the same of 'active-backup'? seems a
> terminology inconsistency to me...
> 
> -- 
>  Se non trovi nessuno vuol dire che siamo scappati alle sei-shell (bash,
>  tcsh,csh...)                            (Possi)
> 
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 

Re: [PVE-User] Experimenting with bond on a non-LACP switch...

[Marco Gaiarin]

Mandi! Wolf Noble
  In chel di` si favelave...

> I was condensing several (i emit traffic on one interface but listen on all bond members) into ???active-passive???
> this: 
> https://help.ubuntu.com/community/UbuntuBonding
> does a better job explaining the different modes??? 
> hope that helps?

Mmmhhhh... seems to me roughly the same documentation of debian and PVE
wiki, and at least to me does not solve the 'confusion' about the
'active-passive' term...


Sorry...

-- 
  Si dice che se fai girare un cd di installazione di Windows al contrario
  nel lettore cd si sentono messaggi satanici. Questo è niente: fallo
  andare nel verso giusto e installa Windows!			(?)