From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <leandro@tecnetmza.com.ar> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 46B8469434 for <pve-user@pve.proxmox.com>; Mon, 13 Sep 2021 13:33:11 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2EF79295E7 for <pve-user@pve.proxmox.com>; Mon, 13 Sep 2021 13:32:41 +0200 (CEST) Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 7B5BD295BF for <pve-user@pve.proxmox.com>; Mon, 13 Sep 2021 13:32:38 +0200 (CEST) Received: by mail-lj1-x232.google.com with SMTP id g14so16682791ljk.5 for <pve-user@pve.proxmox.com>; Mon, 13 Sep 2021 04:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tecnetmza-com-ar.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BKeehuiqJhYd8h3QMEinKJpQnpwER7IYYHRoB11wZOw=; b=SW4jERJ+0sjbweVJJF9LtlChHtQvt8xGyKNLfyQXPAHSeW0zWR4921wAfBikSiuzsq cTgmj1nIHxYkC0nY77mF3Jhb8rOXHSl6030a78EpZZLWk0ABBJxV1tLp9jVRfn1yBNjH dltogJRzP7UkrRFM3pxbKb2BhVaSFvWyYFyjBuXx+0OU0k2lxSBWllFRHMOj9iZmLMgP UsL75m4kxqdls57vh4+DICGa4Zb7v1iuOhr6IiBglY15Kcsrg1Ms+wPKHjC2gf715qeK 8B7IQE1JCcuVkvBUjWI0PcSNtASYVgarERcRYyV9oTvCqm4IuZI1vZHmF0liCEpOzCQM FrxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BKeehuiqJhYd8h3QMEinKJpQnpwER7IYYHRoB11wZOw=; b=76b6kVPmovYTtoenKUJKtN6hqnG6HuJAI1hzov/8np2OnGRzY/5lasgod06ACmyzgC zEH5gbRTK9TkJn3IBNvBsqhYYlGQmakj1XXEgnlNkwkpvQdRjf6g53kgWIkyCXMzOjha hzVQY9BEszHi/15mAWrUY6GyR4TLEtYo7z0oB1dHBq0dFMTUxJnsKZuaFlmNW/1z+W9R Gil5KofNYSBhY7VQqJTe87tRzZvAaJgzJbq5ajknsABV+CCURE1/KgpBhXi5yUf9/U0C G1GBPo3ccMIjg2/KtBOATUA4OU17Vku6hL69BUCnqieD3VKRKHIG19Pc84DMMdpBrkyQ xDSQ== X-Gm-Message-State: AOAM530PilSUQyJjXM68pUvZJW7MvFUcTVzy3MY+J08rhBBjVZdMGvLC vKVz8VYk2u9/tgrB7lm6BNJzGOMsZQ3KV1YK6EeGjdACw4em X-Google-Smtp-Source: ABdhPJxdpMBMvc/qWAS5GPYMFL56b9btkvMfmd70jwb/66WZuLV/6rVlYoJewQPglx1LjH3W8ZgDvON8E0TIbo7+PF4= X-Received: by 2002:a2e:8e8f:: with SMTP id z15mr9838030ljk.121.1631532752246; Mon, 13 Sep 2021 04:32:32 -0700 (PDT) MIME-Version: 1.0 References: <CALt2oz6WNOUjSoWHbmL1+E536YhZzc-FrAjYrsVQtEryWGMXjw@mail.gmail.com> <4D290E10-15B5-4816-8613-38438FC1F9CC@benappy.com> In-Reply-To: <4D290E10-15B5-4816-8613-38438FC1F9CC@benappy.com> From: Leandro Roggerone <leandro@tecnetmza.com.ar> Date: Mon, 13 Sep 2021 08:32:21 -0300 Message-ID: <CALt2oz7Ui2iSdwtNcU=JRF-+UxDk8Ev66w1Jk9WK7p=GbqoG4w@mail.gmail.com> To: Proxmox VE user list <pve-user@lists.proxmox.com> Cc: PVE User List <pve-user@pve.proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 1.056 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_SOFTFAIL 0.665 SPF: sender does not match SPF record (softfail) T_REMOTE_IMAGE 0.01 Message contains an external image Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] ceph X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Mon, 13 Sep 2021 11:33:11 -0000 hi guys , your responses were very useful. Lets suppose I have my 3 nodes running and forming a cluster. Please confirm: a -Can I add the ceph storage at any time ? b- All nodes should be running the same pve version ? c- All nodes should have 1 or more non used storages with no hardware raid to be included in the ceph ? Those storages (c) should be exactly same in capacity , speed , and so ... ? What can goes wrong if dont have 10 but 1 gbps ports ? Regards. Leandro <https://www.avast.com/sig-email?utm_medium=3Demail&utm_source=3Dlink&utm_c= ampaign=3Dsig-email&utm_content=3Dwebmail> Libre de virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=3Demail&utm_source=3Dlink&utm_c= ampaign=3Dsig-email&utm_content=3Dwebmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> El mi=C3=A9, 8 sept 2021 a las 19:21, ic (<lists@benappy.com>) escribi=C3= =B3: > Hi there, > > > On 8 Sep 2021, at 14:46, Leandro Roggerone <leandro@tecnetmza.com.ar> > wrote: > > > > I would like to know the goods that a ceph storage can bring to my > existing > > cluster. > > What is an easy / recommended way to implement it ? > > Wich hardware should I consider to use ? > > First, HW. > > Get two Cisco Nexus 3064PQ (they typically go for $600-700 for 48 10G > ports) and two Intel X520-DA2 per server. > > Hook up each port of the Intel cards to each of the Nexuses, getting a > full redundancy between network cards and switches. > > Add 4x40G DAC cables between the switches, setup 2 as VPC peer-links, 2 a= s > a simple L2 trunk (can provide more details as why if needed). > > Use ports 0 from both NICs for ceph, ports 1 for VM traffic. This way you > get 2x10 Gbps for Ceph only and 2x10 Gbps for everything else, and if you > loose one card or one switch, you still have 10 Gbps for each. > > The benefits? With default configuration, your data lives in 3 places. > Also, scale out. You know the expensive stuff, hyperconverged servers > (nutanix and such) ? You get that with this. > > The performance is wild, just moved my customers from a proxmox cluster > backed by a TrueNAS server (full flash, 4x10Gbps) to a 3 node cluster of > AMD EPYC nodes with Ceph on local SATA SSDs and the VMs started flying. > > Keep your old storage infrastructure, whatever that is, for backups with > PBS. > > YMMV > > Regards, ic > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >