* Re: [PVE-User] Experimenting with bond on a non-LACP switch...
2022-05-20 11:22 [PVE-User] Experimenting with bond on a non-LACP switch Marco Gaiarin
@ 2022-05-22 1:29 ` Laurent Dumont
2022-05-24 21:17 ` Marco Gaiarin
2022-05-22 4:12 ` Wolf Noble
1 sibling, 1 reply; 6+ messages in thread
From: Laurent Dumont @ 2022-05-22 1:29 UTC (permalink / raw)
To: Proxmox VE user list
It's not made very clear from the documentation. I assume there are good
technical reasons why the cluster traffic would be impacted.
Afaik, proxmox leverages corosync which can leverage multicast for the
cluster checks. I don't think it can be badly impacted by LACP but
something to keep in mind.
There is this old thread with a similar discussion :
https://forum.proxmox.com/threads/cluster-lacp.90668/
On Sat, May 21, 2022 at 4:10 AM Marco Gaiarin <gaio@lilliput.linux.it>
wrote:
>
> I'm doing some experimentation on a switch that seems does not support
> LACP,
> even thus claim that; is a Netgear GS724Tv2:
>
>
> https://www.downloads.netgear.com/files/GDC/GS724Tv2/enus_ds_gs724t.pdf
>
> data sheet say:
>
> Port Trunking - Manual as per IEEE802.3ad Link Aggregation
>
> and 'IEEE802.3ad Link Aggregation' is LACP, right?
>
>
> Anyway, i'm experimenting a bit with other bonding mode, having
> (un)expected
> results and troubles, but in:
>
> https://pve.proxmox.com/wiki/Network_Configuration#_linux_bond
>
> i've stumble upon that sentence:
>
> If you intend to run your cluster network on the bonding
> interfaces, then you have to use active-passive mode on the bonding
> interfaces, other modes are unsupported.
>
> What exactly mean?! Thanks.
>
> --
> Molti italiani sognavano di vedere Berlusconi in un cellulare,
> prima o poi... (Stardust®, da i.n.n-a)
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] Experimenting with bond on a non-LACP switch...
2022-05-20 11:22 [PVE-User] Experimenting with bond on a non-LACP switch Marco Gaiarin
2022-05-22 1:29 ` Laurent Dumont
@ 2022-05-22 4:12 ` Wolf Noble
1 sibling, 0 replies; 6+ messages in thread
From: Wolf Noble @ 2022-05-22 4:12 UTC (permalink / raw)
To: Proxmox VE user list
[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]
Good Catch Marco!
I'd not seen that when I read through that page, but I just re-read it.... My read is that it can introduce ODD edge-case complications.
my synthesis of this information is outlined here. I encourage anyone to correct my misunderstandings.
network abstraction gets complicated QUICKLY. Network gear vendors implement their support for the different bonding modes in subtly different ways.
Firewalls have their own quirks.
abstractions on top of abstractions on top of abstractions on top of abstractions on top of .... okay you get the point.
we want to avoid asymmetric pathing where possible, because stuff gets quirky and edge-casey quickly. the fewer explicitly supported virtual topologies, the fewer scenarios the engineering teams need to scrutinize the COMPLEX edge case behaviors of, resulting in a better experience for EVERYONE.... heres what I mean:
LACP:
This is a pretty well known and consistently implemented aggregation mechanism. the behaviors of network interfaces and switching hardware that are involved are pretty consistent.
This GENERALLY works fine. the only time I've seen it get a little wonky is LACP across switch chassis behavior can be odd..
[-- Attachment #2: Type: text/plain, Size: 2938 bytes --]
When node 2 sends traffic destined for node 1 via lacp_b.1 ... it may traverse the trunked interconnect, it may not. (it depends)
active-backup:
I talk, and listen on link0... if link 0 goes down, I switchover... I don't really pay attention to linkl1 otherwise.
active-passive:
I talk on link0. I listen on link0 and link1.
The downside that I've seen here:
Arp caching can get wonky, and packets that SHOULD be directed to node0 link0
get directed to node0 link1... or sometimes packets directed to node0.link0 have a destination mac address of the hwaddr of link1 and so get delivered to link1 ...
There MAY be some oddities that manifest with this configuration.
depending on (node scope configuration) sysctl settings, node0 could just ignore those packets, resulting in weird behavior
with the various balance algorithms nodes will see a different hardware addresses for each other, again, this isn't *USUALLY* a problem, but there are still some dragons that lurk within the trunking/bonding code... hardware checksumming can get whacky... especially when VLANs get mixed in...
My gut tells me that the main reason for this advise is that using LACP or active/backup provides sufficient durability while introducing as little edge-case wonky as possible, which generally speaking is a GoodThing™️ when it comes to intra-cluster-comms.
I could be wrong, so don't take this as gospel... if anyone has a better explanation, or can point out my flawed logic, by all means, chime in! :)
Hope my understanding HELPS... if it doesn't, throw it away and ignore it ;)
🐺W
This message created and transmitted using 100% recycled electrons.
> On May 21, 2022, at 03:11, Marco Gaiarin <gaio@lilliput.linux.it> wrote:
>
>
> I'm doing some experimentation on a switch that seems does not support LACP,
> even thus claim that; is a Netgear GS724Tv2:
>
> https://www.downloads.netgear.com/files/GDC/GS724Tv2/enus_ds_gs724t.pdf
>
> data sheet say:
>
> Port Trunking - Manual as per IEEE802.3ad Link Aggregation
>
> and 'IEEE802.3ad Link Aggregation' is LACP, right?
>
>
> Anyway, i'm experimenting a bit with other bonding mode, having (un)expected
> results and troubles, but in:
>
> https://pve.proxmox.com/wiki/Network_Configuration#_linux_bond
>
> i've stumble upon that sentence:
>
> If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported.
>
> What exactly mean?! Thanks.
>
> --
> Molti italiani sognavano di vedere Berlusconi in un cellulare,
> prima o poi... (Stardust®, da i.n.n-a)
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
^ permalink raw reply [flat|nested] 6+ messages in thread