From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6BA8269083 for ; Wed, 10 Mar 2021 11:48:18 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 56EE61BC2B for ; Wed, 10 Mar 2021 11:47:48 +0100 (CET) Received: from smtp2.ngi.it (smtp2.ngi.it [IPv6:2001:4c91::113]) by firstgate.proxmox.com (Proxmox) with ESMTP id 995951BC1F for ; Wed, 10 Mar 2021 11:47:46 +0100 (CET) Received: from tank.sv.lnf.it (88-147-114-200.static.eolo.it [88.147.114.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp2.ngi.it (Postfix) with ESMTPS id C345760E70 for ; Wed, 10 Mar 2021 11:47:33 +0100 (CET) Authentication-Results: smtp2.ngi.it; dkim=pass (1024-bit key; unprotected) header.d=sv.lnf.it header.i=@sv.lnf.it header.b="UZ7nqeQR"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sv.lnf.it; s=2017; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-ID: Subject:To:From:Date:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=rryK3di+GYyICIkaQjrsEwCfHpMqEZOqhUYCJsedYn4=; b=UZ7nqeQRRQWTbLLAgR6kYtSFZz qk7mOUzVqVeFETuR4gog5MtVO53gcL27f3YRsSHRLfZQ5Ld0vBX786bmANF2vOeyZ7nGDtyVCXCpB VyFssOIdS9r61L9NGpmojwUkw7ZMyqfW3I4BNaoIRr4yBuqfNY6KbDrKSbZhOO9gbhik=; Received: from [10.5.1.3] (helo=vdmsv1.ad.fvg.lnf.it) by tank.sv.lnf.it with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1lJwNJ-000Fup-6k for pve-user@pve.proxmox.com; Wed, 10 Mar 2021 11:47:33 +0100 Received: from [10.5.1.44] (helo=hermione.sv.lnf.it) by vdmsv1.ad.fvg.lnf.it with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1lJwNI-0000zi-2q for pve-user@pve.proxmox.com; Wed, 10 Mar 2021 11:47:32 +0100 Received: by hermione.sv.lnf.it (Postfix, from userid 1000) id EBE7AA0007F; Wed, 10 Mar 2021 11:47:31 +0100 (CET) Date: Wed, 10 Mar 2021 11:47:31 +0100 From: Marco Gaiarin To: pve-user@pve.proxmox.com Message-ID: <20210310104731.GH3397@sv.lnf.it> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit 'Organization: Associazione La Nostra Famiglia - Polo FVG' 'From: Marco Gaiarin ' X-SPAM-LEVEL: Spam detection results: 0 AWL 0.138 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain JMQ_SPF_NEUTRAL 0.5 SPF set to ?all SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record T_SPF_PERMERROR 0.01 SPF: test of record failed (permerror) URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [lnf.it, proxmox.com, lanostrafamiglia.it] Subject: [PVE-User] Three node Hyperconverged PVE+Ceph and failure domains... X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2021 10:48:18 -0000 One of the most interesting configuration of PVE is the three node, switchless (full mesh) configuration, depicted in some PVE docs, most notably: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09 But lurking 'ceph-user' mailing list, some weeks ago, lead to an interesting discussion about 'failure domains', and many user depicted the three node cluster as 'insecure'. The reasoning are about: a) 'min_size = 2' is a must if you need to keep your data safe; you can set 'min_size = 1', but clearly there's no scrub/checksumming, so no real guarantee against data corruption. b) but in a three node setup, with 'min_size = 2', if a node goes down, the cluster switch in 'readonly' at the very first subsequent failure, eg the cluster does not handle more then a failure. c) you can change the failure domain, eg: mon osd down out subtree limit = osd but in this way you have to guarantee (at worst case) room for the double of the space on a single node (eg, three node cluster with 2TB of space each, to guarantee the 'min_size = 2' you cannot use more then 1TB space on overral cluster; so, a 6TB total disk space for a 1TB usable space). I'm wrong? If not, the 3-node hyperconverged cluster is suitable only for testing? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)