From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1FE43D635 for ; Thu, 1 Dec 2022 08:45:14 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 00C53262A6 for ; Thu, 1 Dec 2022 08:45:13 +0100 (CET) Received: from nsmtp.uni-koblenz.de (nsmtp.uni-koblenz.de [141.26.64.14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 1 Dec 2022 08:45:13 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nsmtp.uni-koblenz.de (Postfix) with ESMTP id 72F8C1A02F0 for ; Thu, 1 Dec 2022 08:37:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uni-koblenz.de; h=content-transfer-encoding:content-type:content-type:subject :subject:from:from:content-language:user-agent:mime-version:date :date:message-id:received:received:received; s=mail; t= 1669880259; x=1671694660; bh=pCKU5ECVM+NqBdGOh6ncuGj7LHX1ELNZFL3 Mj7prNCg=; b=mcEDoDJ3rX6noVbTaaJxccK/iOOO45tb85fBx2N6MED6bQ/prMy lhxoVLMtbe0XUhQPe2gCzZG2dPYIsJPZtSwip9R7BxoM6dW3Br8IwCynO/Yre6Hz dGgmjoEtOcT3BK6ijr4/3eK5RTrAoFy/wDf6JQ8W0Rcrn6eJp0g950/g= X-Virus-Scanned: Debian amavisd-new at uni-koblenz.de Received: from nsmtp.uni-koblenz.de ([127.0.0.1]) by localhost (nsmtp.uni-koblenz.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CrlwPgDK5Z36 for ; Thu, 1 Dec 2022 08:37:39 +0100 (CET) Received: from mailproxy.uni-koblenz.de (mailproxy.uni-koblenz.de [141.26.64.230]) by nsmtp.uni-koblenz.de (Postfix) with ESMTPS for ; Thu, 1 Dec 2022 08:37:39 +0100 (CET) Received: from [IPV6:2001:4c80:50:127::506] (unknown [IPv6:2001:4c80:50:127::506]) by mailproxy.uni-koblenz.de (Postfix) with ESMTPSA id 629B21006E9 for ; Thu, 1 Dec 2022 08:37:36 +0100 (CET) Message-ID: Date: Thu, 1 Dec 2022 08:37:36 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Content-Language: de-DE To: Proxmox VE user list From: Rainer Krienke Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain JMQ_SPF_NEUTRAL 0.5 SPF set to ?all RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [uni-koblenz.de] Subject: [PVE-User] proxmox hyperconverged pg calculations in pve 7.2, ceph pacific X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2022 07:45:14 -0000 Hello, I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node has 8 4TB disks. pve and ceph are installed an running. Next I wanted to create some ceph-pools with each 512 pgs. Since I want to use erasure coding (5+3) when creating a pool one rbd pool for metadata and the data pool are created. I used pveceph pool this command: pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode off --pg_num 512 --pg_num_min 128 I was able to create two pools in this way but the third pveceph call threw this error: "got unexpected control message: TASK ERROR: error with 'osd pool create': mon_command failed - pg_num 512 size 8 would mean 22148 total pgs, which exceeds max 22000 (mon_max_pg_per_osd 250 * num_in_osds 88)" What I do not understand now are the calculations behind the scenes for the calculated total pg number of 22148. But how is this total number "22148" calculated? I already reduced the number of pgs for the metadata pool of each ec-pool and so I was able to create 4 pools in this way. But just for fun I now tried to create ec-pool number 5 and I see the message from above. Here are the pools created by now (scraped from ceph osd pool autoscale-status): Pool: Size: Bias: PG_NUM: rbd 4599 1.0 32 px-a-data 528.2G 1.0 512 px-a-metadata 838.1k 1.0 128 px-b-data 0 1.0 512 px-b-metadata 19 1.0 128 px-c-data 0 1.0 512 px-c-metadata 19 1.0 128 px-d-data 0 1.0 512 px-d-metadata 0 1.0 128 So the total number of pgs for all pools is currently 2592 which is far from 22148 pgs? Any ideas? Thanks Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312