* [PVE-User] ceph
@ 2021-09-08 12:46 Leandro Roggerone
2021-09-08 12:55 ` Gilberto Ferreira
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Leandro Roggerone @ 2021-09-08 12:46 UTC (permalink / raw)
To: PVE User List
Hi guys , I have a 2 nodes cluster working,
I will add a third node to the cluster.
I would like to know the goods that a ceph storage can bring to my existing
cluster.
What is an easy / recommended way to implement it ?
Wich hardware should I consider to use ?
#############
Currently im facing the upgrade from pve 6 to pve 7.
Having a ceph storage can make this process easier ?
Regards.
Leandro.,
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PVE-User] ceph
2021-09-08 12:46 [PVE-User] ceph Leandro Roggerone
@ 2021-09-08 12:55 ` Gilberto Ferreira
2021-09-08 20:07 ` Alex K
2021-09-08 22:11 ` ic
2 siblings, 0 replies; 11+ messages in thread
From: Gilberto Ferreira @ 2021-09-08 12:55 UTC (permalink / raw)
To: Proxmox VE user list; +Cc: PVE User List
>> Wich hardware should I consider to use ?
At least SSD Enterprise/DataCenter Class
10G network cards.
---
Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram
Em qua., 8 de set. de 2021 às 09:47, Leandro Roggerone <
leandro@tecnetmza.com.ar> escreveu:
> Hi guys , I have a 2 nodes cluster working,
> I will add a third node to the cluster.
> I would like to know the goods that a ceph storage can bring to my existing
> cluster.
> What is an easy / recommended way to implement it ?
> Wich hardware should I consider to use ?
>
> #############
>
> Currently im facing the upgrade from pve 6 to pve 7.
> Having a ceph storage can make this process easier ?
>
> Regards.
> Leandro.,
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PVE-User] ceph
2021-09-08 12:46 [PVE-User] ceph Leandro Roggerone
2021-09-08 12:55 ` Gilberto Ferreira
@ 2021-09-08 20:07 ` Alex K
2021-09-08 22:11 ` ic
2 siblings, 0 replies; 11+ messages in thread
From: Alex K @ 2021-09-08 20:07 UTC (permalink / raw)
To: Proxmox VE user list; +Cc: PVE User List
On Wed, Sep 8, 2021, 15:46 Leandro Roggerone <leandro@tecnetmza.com.ar>
wrote:
> Hi guys , I have a 2 nodes cluster working,
> I will add a third node to the cluster.
> I would like to know the goods that a ceph storage can bring to my existing
> cluster.
>
It brings high availability and Iive migration as any cluster aware shared
storage with the cost of having to take care ceph. One advantage also is
that you hopefully will not have downtime during maintenance.
What is an easy / recommended way to implement it ?
> Wich hardware should I consider to use ?
>
> #############
>
> Currently im facing the upgrade from pve 6 to pve 7.
> Having a ceph storage can make this process easier ?
>
> Regards.
> Leandro.,
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PVE-User] ceph
2021-09-08 12:46 [PVE-User] ceph Leandro Roggerone
2021-09-08 12:55 ` Gilberto Ferreira
2021-09-08 20:07 ` Alex K
@ 2021-09-08 22:11 ` ic
2021-09-13 11:32 ` Leandro Roggerone
2 siblings, 1 reply; 11+ messages in thread
From: ic @ 2021-09-08 22:11 UTC (permalink / raw)
To: Proxmox VE user list; +Cc: PVE User List
Hi there,
> On 8 Sep 2021, at 14:46, Leandro Roggerone <leandro@tecnetmza.com.ar> wrote:
>
> I would like to know the goods that a ceph storage can bring to my existing
> cluster.
> What is an easy / recommended way to implement it ?
> Wich hardware should I consider to use ?
First, HW.
Get two Cisco Nexus 3064PQ (they typically go for $600-700 for 48 10G ports) and two Intel X520-DA2 per server.
Hook up each port of the Intel cards to each of the Nexuses, getting a full redundancy between network cards and switches.
Add 4x40G DAC cables between the switches, setup 2 as VPC peer-links, 2 as a simple L2 trunk (can provide more details as why if needed).
Use ports 0 from both NICs for ceph, ports 1 for VM traffic. This way you get 2x10 Gbps for Ceph only and 2x10 Gbps for everything else, and if you loose one card or one switch, you still have 10 Gbps for each.
The benefits? With default configuration, your data lives in 3 places. Also, scale out. You know the expensive stuff, hyperconverged servers (nutanix and such) ? You get that with this.
The performance is wild, just moved my customers from a proxmox cluster backed by a TrueNAS server (full flash, 4x10Gbps) to a 3 node cluster of AMD EPYC nodes with Ceph on local SATA SSDs and the VMs started flying.
Keep your old storage infrastructure, whatever that is, for backups with PBS.
YMMV
Regards, ic
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PVE-User] ceph
2021-09-08 22:11 ` ic
@ 2021-09-13 11:32 ` Leandro Roggerone
0 siblings, 0 replies; 11+ messages in thread
From: Leandro Roggerone @ 2021-09-13 11:32 UTC (permalink / raw)
To: Proxmox VE user list; +Cc: PVE User List
hi guys , your responses were very useful.
Lets suppose I have my 3 nodes running and forming a cluster.
Please confirm:
a -Can I add the ceph storage at any time ?
b- All nodes should be running the same pve version ?
c- All nodes should have 1 or more non used storages with no hardware raid
to be included in the ceph ?
Those storages (c) should be exactly same in capacity , speed , and so ...
?
What can goes wrong if dont have 10 but 1 gbps ports ?
Regards.
Leandro
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Libre
de virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
El mié, 8 sept 2021 a las 19:21, ic (<lists@benappy.com>) escribió:
> Hi there,
>
> > On 8 Sep 2021, at 14:46, Leandro Roggerone <leandro@tecnetmza.com.ar>
> wrote:
> >
> > I would like to know the goods that a ceph storage can bring to my
> existing
> > cluster.
> > What is an easy / recommended way to implement it ?
> > Wich hardware should I consider to use ?
>
> First, HW.
>
> Get two Cisco Nexus 3064PQ (they typically go for $600-700 for 48 10G
> ports) and two Intel X520-DA2 per server.
>
> Hook up each port of the Intel cards to each of the Nexuses, getting a
> full redundancy between network cards and switches.
>
> Add 4x40G DAC cables between the switches, setup 2 as VPC peer-links, 2 as
> a simple L2 trunk (can provide more details as why if needed).
>
> Use ports 0 from both NICs for ceph, ports 1 for VM traffic. This way you
> get 2x10 Gbps for Ceph only and 2x10 Gbps for everything else, and if you
> loose one card or one switch, you still have 10 Gbps for each.
>
> The benefits? With default configuration, your data lives in 3 places.
> Also, scale out. You know the expensive stuff, hyperconverged servers
> (nutanix and such) ? You get that with this.
>
> The performance is wild, just moved my customers from a proxmox cluster
> backed by a TrueNAS server (full flash, 4x10Gbps) to a 3 node cluster of
> AMD EPYC nodes with Ceph on local SATA SSDs and the VMs started flying.
>
> Keep your old storage infrastructure, whatever that is, for backups with
> PBS.
>
> YMMV
>
> Regards, ic
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PVE-User] ceph
@ 2023-01-09 9:14 Piviul
[not found] ` <mailman.191.1673258105.458.pve-user@lists.proxmox.com>
0 siblings, 1 reply; 11+ messages in thread
From: Piviul @ 2023-01-09 9:14 UTC (permalink / raw)
To: pve-user
Hi all, during the CEPH installation I have dedicated a 10Gb network to
CEPH but I fear I wrong the monitors and managers IPs because they are
referring to the LAN IPs PVE nodes instead of using the IPs of the CEPH
nodes of the CEPH dedicated network. To solve this problem can I first
of all create the monitors/managers on the dedicated CEPH network and
then remove the olds monitors/managers having LAN IPs? This operation is
safely?
Piviul
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2023-01-11 15:52 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-08 12:46 [PVE-User] ceph Leandro Roggerone
2021-09-08 12:55 ` Gilberto Ferreira
2021-09-08 20:07 ` Alex K
2021-09-08 22:11 ` ic
2021-09-13 11:32 ` Leandro Roggerone
2023-01-09 9:14 Piviul
[not found] ` <mailman.191.1673258105.458.pve-user@lists.proxmox.com>
2023-01-09 11:47 ` Piviul
[not found] ` <mailman.203.1673265308.458.pve-user@lists.proxmox.com>
2023-01-10 7:23 ` Piviul
[not found] ` <mailman.215.1673337884.458.pve-user@lists.proxmox.com>
2023-01-10 13:29 ` Piviul
[not found] ` <mailman.232.1673430028.458.pve-user@lists.proxmox.com>
2023-01-11 11:19 ` Piviul
[not found] ` <mailman.235.1673444838.458.pve-user@lists.proxmox.com>
2023-01-11 15:51 ` Piviul
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox