From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1372162963 for ; Tue, 22 Feb 2022 13:03:14 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 08BE723CE3 for ; Tue, 22 Feb 2022 13:03:14 +0100 (CET) Received: from mail.merit.unu.edu (webmail.merit.unu.edu [192.87.143.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 4E7EB23CD7 for ; Tue, 22 Feb 2022 13:03:13 +0100 (CET) Received: from [192.87.143.100] (ws100.merit.unu.edu [192.87.143.100]) by mail.merit.unu.edu (Postfix) with ESMTPSA id A5C958197440D for ; Tue, 22 Feb 2022 13:03:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=merit.unu.edu; s=2021-1; t=1645531386; bh=Xy0FwZTnLnt1HzJuVTKPPf/SLvvSFOJ5+JJL9tuI9K4=; h=Date:To:From:Subject:From; b=EBpQnzTuBBK+TMIMRZ58JvNTN3ACLtKar05n24mqHlwPWlw3ZJeTToZnia07kswnU hrRKEjqnNhXgfCs9HHKhRASZhjQzWFHvjGT1noWBrkDTKZomqYKJJJCBszoAt27Fmz 7QXwCuItYZDPoVR4/9YXM58KB5/Es8Kku8sl/t8042/h/2LQWBRJhg5NS7+LiOaakF elftxWXqtAJGTPapcv5U83au5eqDjFsVoShAUSzkAEJysxkmD8Pp0Disopzv9/17tS XqKH8Z77gNcHqI3jFdxRKsa7h0e6PwzZVB+rmXIDno1JQebxKxRatQ34drKxixN64E lg6Xvv+mKmp/g== Message-ID: <639da1da-5e4c-7c13-03cb-ce716cd07954@merit.unu.edu> Date: Tue, 22 Feb 2022 13:03:06 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 To: pve-user@lists.proxmox.com Content-Language: en-US From: mj Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.508 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain RCVD_IN_MSPIKE_H2 -0.001 Average reputation (+2) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com, unu.edu] Subject: [PVE-User] 10G mesh to 40G DAC X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Feb 2022 12:03:14 -0000 Hi, I just wanted to share my experience of upgrading our cluster from 10G mesh networking (https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server) to arista 40G MLAG over DAC cables. I expected DAC over switches to have a slightly increased latency over direct (meshed) UTP RJ45 connections. Throughput would of course be much higher in the new config. Ceph (on SSDs) is used mainly for VM images, RBD. So, after having completed the migration yesterday, I can say that the VMs feel much quicker. Specially the VMs using databases (LAMP servers and a couple of management systems) are much more responsive now than they were before. Over all, we're very happy now. For the record, we're using dual arista DCS-7050QX-32S MLAG now. Purchased refurb for around 1000 euro each, including three years warranty. Just thought I'd share. :-) MJ