[PVE-User] proxmox hyperconverged pg calculations in pve 7.2, ceph pacific

[Rainer Krienke <krienke@uni-koblenz.de>]

Hello,

I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node has 
8 4TB disks. pve and ceph are installed an running.

Next I wanted to create some ceph-pools with each 512 pgs. Since I want 
to use erasure coding (5+3) when creating a pool one rbd pool for 
metadata and the data pool are created. I used pveceph pool this command:

pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode 
off --pg_num 512 --pg_num_min 128

I was able to create two pools in this way but the third pveceph call 
threw this error:

"got unexpected control message: TASK ERROR: error with 'osd pool 
create': mon_command failed -  pg_num 512 size 8 would mean 22148 total 
pgs, which exceeds max 22000 (mon_max_pg_per_osd 250 * num_in_osds 88)"

What I do not understand now are the calculations behind the scenes for 
the calculated total pg number of 22148. But how is this total number 
"22148"  calculated?

I already reduced the number of pgs for the metadata pool of each 
ec-pool and so I was able to create 4 pools in this way. But just for 
fun I now tried to create ec-pool number 5 and I see the message from 
above.

Here are the pools created by now (scraped from ceph osd pool 
autoscale-status):
Pool:                Size:   Bias:  PG_NUM:
rbd                  4599    1.0      32
px-a-data          528.2G    1.0     512
px-a-metadata      838.1k    1.0     128
px-b-data              0     1.0     512
px-b-metadata         19     1.0     128
px-c-data              0     1.0     512
px-c-metadata         19     1.0     128
px-d-data              0     1.0     512
px-d-metadata          0     1.0     128

So the total number of pgs for all pools is currently 2592 which is far 
from 22148 pgs?

Any ideas?
Thanks Rainer
-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse  1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html,     Fax: +49261287 
1001312