[pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[aderumier]

Hi,

I was doing a training week with students,

and I see that the new ceph wizard to create pool don't allow to set
min_size=1 anymore.

It's currently displaying a warning "min_size <= size/2 can lead to
data loss, incomplete PGs or unfound objects", 

that's ok  ,  but It's also blocking the validation button.



Some users with small cluster/budgets want to do only size=2,

so with min_size=2, the cluster will go read only in case of any osd
down.

It could be great to allow at least min_size=1 when size=2 is used.


also,
Other setup like size=4, min_size=2, also display the warning, but
allow to validate the form.

I'm not sure this warning is correct in this case , as since octopus,
min_size
is auto compute when pool is created, and a simple

ceph osd pool create mypool 128 --size=4  , create a pool with
min_size=2 by default.

Re: [pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[Dominik Csapak]

On 6/4/21 04:47, aderumier@odiso.com wrote:
> Hi,
>


Hi,

> I was doing a training week with students,
> 
> and I see that the new ceph wizard to create pool don't allow to set
> min_size=1 anymore.
> 
> It's currently displaying a warning "min_size <= size/2 can lead to
> data loss, incomplete PGs or unfound objects",
> 
> that's ok  ,  but It's also blocking the validation button.
> 

yes, in our experience, setting min_size to 1 is always a bad idea
and most likely not what you want

what is possible though is to either create the pool on the cli,
or changing the min_size after creation to 1 (this is not blocked)

> 
> 
> Some users with small cluster/budgets want to do only size=2,
> 
> so with min_size=2, the cluster will go read only in case of any osd
> down.
> 
> It could be great to allow at least min_size=1 when size=2 is used.
> 

"great" but very dangerous

> 
> also,
> Other setup like size=4, min_size=2, also display the warning, but
> allow to validate the form.
> 
> I'm not sure this warning is correct in this case , as since octopus,
> min_size
> is auto compute when pool is created, and a simple
> 
> ceph osd pool create mypool 128 --size=4  , create a pool with
> min_size=2 by default.
> 
> 

the rationale behind this decision was (i think) because
if you have exactly 50% min_size of size (e.g. 4/2)
you can get inconsistent pgs, with no quorum as to
which pg is correct?
(though don't quote me on that)

so i think its always better to have > 50% min_size of size

Re: [pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[aderumier]

Le vendredi 04 juin 2021 à 15:23 +0200, Dominik Csapak a écrit :
> On 6/4/21 04:47, aderumier@odiso.com wrote:
> > Hi,
> > 
> 
> 
> Hi,
> 
> > I was doing a training week with students,
> > 
> > and I see that the new ceph wizard to create pool don't allow to
> > set
> > min_size=1 anymore.
> > 
> > It's currently displaying a warning "min_size <= size/2 can lead to
> > data loss, incomplete PGs or unfound objects",
> > 
> > that's ok  ,  but It's also blocking the validation button.
> > 
> 
> yes, in our experience, setting min_size to 1 is always a bad idea
> and most likely not what you want
> 
> what is possible though is to either create the pool on the cli,
> or changing the min_size after creation to 1 (this is not blocked)
> 
yes, Sute. I could be great to be able to change size/min_size from the
gui too.



> > 
> > 
> > Some users with small cluster/budgets want to do only size=2,
> > 
> > so with min_size=2, the cluster will go read only in case of any
> > osd
> > down.
> > 
> > It could be great to allow at least min_size=1 when size=2 is used.
> > 
> 
> "great" but very dangerous
> 
> > 
> > also,
> > Other setup like size=4, min_size=2, also display the warning, but
> > allow to validate the form.
> > 
> > I'm not sure this warning is correct in this case , as since
> > octopus,
> > min_size
> > is auto compute when pool is created, and a simple
> > 
> > ceph osd pool create mypool 128 --size=4  , create a pool with
> > min_size=2 by default.
> > 
> > 
> 
> the rationale behind this decision was (i think) because
> if you have exactly 50% min_size of size (e.g. 4/2)
> you can get inconsistent pgs, with no quorum as to
> which pg is correct?
> (though don't quote me on that)
> 
> so i think its always better to have > 50% min_size of size
> 
Well, afaik, they are no "quorum" on pg consistency for repair
currently.
if a pg is corrupt, ceph is simply copy data from a pg copy where
checksum is ok.
and if no checksum is available, it take a random copy. (maybe it need
a manual pg_repair in this case).
But They are not something like "theses 2 copies have the more majority
(quorum) of checksum.

(Maybe I'm wrong, but 1 or 2 year ago, Sage have confirmed this on the
ceph mailing)

Re: [pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[Dominik Csapak]

On 6/7/21 08:57, aderumier@odiso.com wrote:
> Le vendredi 04 juin 2021 à 15:23 +0200, Dominik Csapak a écrit :
>> On 6/4/21 04:47, aderumier@odiso.com <mailto:aderumier@odiso.com> wrote:
>>> Hi,
>>>
>>
>>
>> Hi,
>>
>>> I was doing a training week with students,
>>>
>>> and I see that the new ceph wizard to create pool don't allow to set
>>> min_size=1 anymore.
>>>
>>> It's currently displaying a warning "min_size <= size/2 can lead to
>>> data loss, incomplete PGs or unfound objects",
>>>
>>> that's ok  ,  but It's also blocking the validation button.
>>>
>>
>> yes, in our experience, setting min_size to 1 is always a bad idea
>> and most likely not what you want
>>
>> what is possible though is to either create the pool on the cli,
>> or changing the min_size after creation to 1 (this is not blocked)
>>
> yes, Sute. I could be great to be able to change size/min_size from the 
> gui too.
> 
> 

this should already possible in current versions, but as i said
not for pool creation, only afterwards

> 
>>>
>>>
>>> Some users with small cluster/budgets want to do only size=2,
>>>
>>> so with min_size=2, the cluster will go read only in case of any osd
>>> down.
>>>
>>> It could be great to allow at least min_size=1 when size=2 is used.
>>>
>>
>> "great" but very dangerous
>>
>>>
>>> also,
>>> Other setup like size=4, min_size=2, also display the warning, but
>>> allow to validate the form.
>>>
>>> I'm not sure this warning is correct in this case , as since octopus,
>>> min_size
>>> is auto compute when pool is created, and a simple
>>>
>>> ceph osd pool create mypool 128 --size=4  , create a pool with
>>> min_size=2 by default.
>>>
>>>
>>
>> the rationale behind this decision was (i think) because
>> if you have exactly 50% min_size of size (e.g. 4/2)
>> you can get inconsistent pgs, with no quorum as to
>> which pg is correct?
>> (though don't quote me on that)
>>
>> so i think its always better to have > 50% min_size of size
>>
> Well, afaik, they are no "quorum" on pg consistency for repair currently.
> if a pg is corrupt, ceph is simply copy data from a pg copy where 
> checksum is ok.
> and if no checksum is available, it take a random copy. (maybe it need a 
> manual pg_repair in this case).
> But They are not something like "theses 2 copies have the more majority 
> (quorum) of checksum.
> 
> (Maybe I'm wrong, but 1 or 2 year ago, Sage have confirmed this on the 
> ceph mailing)
> 
> 

i thought more about 'inconsistent' pgs, maybe i am wrong
but how does ceph cope with multiple 'valid' objects (all checksums are 
ok) but different content? (e.g, when during a write, theres a
power cut?) i assumed that there a 'majority' must be
established?

although i did not find any document to support that, and in [0]
it is only mentioned it will take the authoritative copy

i'll discuss this with my colleages, and check more sources to maybe
relax the '> 50%' rule a little for the warning

thanks :)

0: https://docs.ceph.com/en/latest/rados/operations/pg-repair

Re: [pve-devel] ceph create pool with min_size=1 not possible anymore with last gui wizard

[Maximilian Hill]

[-- Attachment #1: Type: text/plain, Size: 1634 bytes --]

Hi,

On Mon, Jun 07, 2021 at 09:14:13AM +0200, Dominik Csapak wrote:
> On 6/7/21 08:57, aderumier@odiso.com wrote:
> > Le vendredi 04 juin 2021 à 15:23 +0200, Dominik Csapak a écrit :
> > > On 6/4/21 04:47, aderumier@odiso.com <mailto:aderumier@odiso.com> wrote:
> > > > I was doing a training week with students,
> > > > 
> > > > and I see that the new ceph wizard to create pool don't allow to set
> > > > min_size=1 anymore.
> > > > 
> > > > It's currently displaying a warning "min_size <= size/2 can lead to
> > > > data loss, incomplete PGs or unfound objects",
> > > > 
> > > > that's ok  ,  but It's also blocking the validation button.
> > > > 
> > > 
> > > yes, in our experience, setting min_size to 1 is always a bad idea
> > > and most likely not what you want
> > > 
> > > what is possible though is to either create the pool on the cli,
> > > or changing the min_size after creation to 1 (this is not blocked)
> > > 
> > yes, Sute. I could be great to be able to change size/min_size from the
> > gui too.
> > 
> > 
> 
> this should already possible in current versions, but as i said
> not for pool creation, only afterwards
> 

In my opinion, setting min_size below 2 and size below 3 shouldn't be
possible at all. Both shouldn't be done unless you really know what you're
doing. At that point, you know enough about Ceph to just type that one
command.

That has nothing to do with an easy and safe Ceph setup and the
possibility in itself might lead to unseafe setups by users without
enough knowlege about Ceph.

Maybe a strong opinion, but --yes-i-really-mean-it


Max

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]