Re: [PVE-User] Fwd: [ceph-users] [IMPORTANT NOTICE] Potential data corruption in Pacific

Re: [PVE-User] Fwd: [ceph-users] [IMPORTANT NOTICE] Potential data corruption in Pacific

[Alain péan]

Le 28/10/2021 à 18:19, Eneko Lacunza via pve-user a écrit :
> Hi,
>
> Regarding this important issue reported for Ceph (just 
> cross-forwarding for those Ceph users not in ceph-user list):
>
> 1. Proxmox upgrade wiki page 
> https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific marks as optional 
> disabling the conversion with "ceph config set osd 
> bluestore_fsck_quick_fix_on_mount false", and urges to convert ASAP. 
> It is now advised not to perform the conversion until a fix is released.
>
> 2. We have upgraded 2 clusters, but both have 
> bluestore_fsck_quick_fix_on_mount set to false. No command setting 
> that flag in our server's history, so I don't think we set that (I 
> wasn't the admin performin the upgrade, I'll be able to check with her 
> tomorrow)
>
> 3. Is it possible that for ceph 16.2.6 
> bluestore_fsck_quick_fix_on_mount default value is false?
>
> I expect an emergency fix released soon but I guess checking won't 
> hurt anyone ;)

Hi Eneko,

Thanks for the warning. I saw the thread on ceph-users mailing list. 
Personnally, I upgraded my clusters from PVE 6.4 to 7.0, but not yet 
from Nautilus to Pacific. I'll wait a bit for this.

Also, I have one cluster (the one in production) that is still with 
filestore OSDs. I was wondering, considering ' 
bluestore_fsck_quick_fix_on_mount", if it was mandatory to have 
bluestore OSDs on Pacific.

Personnally, when I see that you have to manually destroy your OSDs one 
after the other, then recreate it as bluestore, I am a bit reluctant to 
do the upgrade...

Let's see !

Alain

-- 
Administrateur Système/Réseau
C2N (ex LPN) Centre de Nanosciences et Nanotechnologies (UMR 9001)
10 Boulevard Thomas Gobert (ex Avenue de la Vauve), 91120 Palaiseau
Tel : 01-70-27-06-88

Re: [PVE-User] Fwd: [ceph-users] [IMPORTANT NOTICE] Potential data corruption in Pacific

[Aaron Lauterer]

Thank you for bringing this to our attention.

I did a test on a freshly installed PVE 7 + Octopus cluster and there the `bluestore_fsck_quick_fix_on_mount` was set to true... That's why I updated the upgrade guide [0]. Checking the setting before upgrading and if true, setting it to false, should be okay IIUC.

Once we release a version with a fix we will update the guide again to reflect the new situation.

Thanks!
Aaron


[0] https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific

On 10/28/21 18:19, Eneko Lacunza wrote:
> Hi,
> 
> Regarding this important issue reported for Ceph (just cross-forwarding for those Ceph users not in ceph-user list):
> 
> 1. Proxmox upgrade wiki page https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific marks as optional disabling the conversion with "ceph config set osd bluestore_fsck_quick_fix_on_mount false", and urges to convert ASAP. It is now advised not to perform the conversion until a fix is released.
> 
> 2. We have upgraded 2 clusters, but both have bluestore_fsck_quick_fix_on_mount set to false. No command setting that flag in our server's history, so I don't think we set that (I wasn't the admin performin the upgrade, I'll be able to check with her tomorrow)
> 
> 3. Is it possible that for ceph 16.2.6 bluestore_fsck_quick_fix_on_mount default value is false?
> 
> I expect an emergency fix released soon but I guess checking won't hurt anyone ;)
> 
> Thanks
> Eneko
> 
> 
> 
>       EnekoLacunza
> 
> CTO | Zuzendari teknikoa
> 
> Binovo IT Human Project
> 
>      943 569 206 <tel:943 569 206>
> 
>      elacunza@binovo.es <mailto:elacunza@binovo.es>
> 
>      binovo.es <//binovo.es>
> 
>      Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun
> 
> 
> youtube <https://www.youtube.com/user/CANALBINOVO/>
>      linkedin <https://www.linkedin.com/company/37269706/>
>