[PVE-User] "nearfull" status in PVE Dashboard not consistent

[PVE-User] "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

Dear all,

I am currently in the process to add SSDs for DB/WAL to our "converged" 
3-node Ceph cluster. After having done so on two of three nodes, the PVE 
Ceph dashboard now reports "5 pool(s) nearfull":

      HEALTH_WARN: 5 pool(s) nearfull
      pool 'pve-pool1' is nearfull
      pool 'cephfs_data' is nearfull
      pool 'cephfs_metadata' is nearfull
      pool '.mgr' is nearfull
      pool '.rgw.root' is nearfull

(see also attached Ceph_dashboard_nearfull_warning.jpg). The storage in 
general is 73% full ("40.87 TiB of 55.67 TiB").

However when looking at the pool overview in PVE, the pools don't seem 
to be very full at all. Some of them are even reported as being 
completely empty (see the attached Ceph_pool_overview.jpg).

Please note: All Ceph manipulations have been done from the PVE UI, as 
we are not very experienced with the Ceph CLI.

We are running PVE 8.2.3 and Ceph runs on version 17.2.7.

Is this inconsistency normal or a problem? And if the latter, then (how) 
can it be fixed?

Cheers, Frank
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

It seems, the attachments got lost on their way. Here they are (again)
Frank

On 07.09.24 21:04, Frank Thommen wrote:
> Dear all,
> 
> I am currently in the process to add SSDs for DB/WAL to our "converged" 
> 3-node Ceph cluster. After having done so on two of three nodes, the PVE 
> Ceph dashboard now reports "5 pool(s) nearfull":
> 
>       HEALTH_WARN: 5 pool(s) nearfull
>       pool 'pve-pool1' is nearfull
>       pool 'cephfs_data' is nearfull
>       pool 'cephfs_metadata' is nearfull
>       pool '.mgr' is nearfull
>       pool '.rgw.root' is nearfull
> 
> (see also attached Ceph_dashboard_nearfull_warning.jpg). The storage in 
> general is 73% full ("40.87 TiB of 55.67 TiB").
> 
> However when looking at the pool overview in PVE, the pools don't seem 
> to be very full at all. Some of them are even reported as being 
> completely empty (see the attached Ceph_pool_overview.jpg).
> 
> Please note: All Ceph manipulations have been done from the PVE UI, as 
> we are not very experienced with the Ceph CLI.
> 
> We are running PVE 8.2.3 and Ceph runs on version 17.2.7.
> 
> Is this inconsistency normal or a problem? And if the latter, then (how) 
> can it be fixed?
> 
> Cheers, Frank
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

Mailman is making fun of me: First it does not accept the mail because 
of too big attachments, now that I reduced the size, it removes them 
completely :-(

Sorry, I digress... Please find the two images here:

   * https://pasteboard.co/CUPNjkTmyYV8.jpg 
(Ceph_dashboard_nearfull_warning.jpg)
   * https://pasteboard.co/34GBggOiUNII.jpg (Ceph_pool_overview.jpg)

HTH, Frank


On 07.09.24 21:07, Frank Thommen wrote:
> It seems, the attachments got lost on their way. Here they are (again)
> Frank
> 
> On 07.09.24 21:04, Frank Thommen wrote:
>> Dear all,
>>
>> I am currently in the process to add SSDs for DB/WAL to our 
>> "converged" 3-node Ceph cluster. After having done so on two of three 
>> nodes, the PVE Ceph dashboard now reports "5 pool(s) nearfull":
>>
>>       HEALTH_WARN: 5 pool(s) nearfull
>>       pool 'pve-pool1' is nearfull
>>       pool 'cephfs_data' is nearfull
>>       pool 'cephfs_metadata' is nearfull
>>       pool '.mgr' is nearfull
>>       pool '.rgw.root' is nearfull
>>
>> (see also attached Ceph_dashboard_nearfull_warning.jpg). The storage 
>> in general is 73% full ("40.87 TiB of 55.67 TiB").
>>
>> However when looking at the pool overview in PVE, the pools don't seem 
>> to be very full at all. Some of them are even reported as being 
>> completely empty (see the attached Ceph_pool_overview.jpg).
>>
>> Please note: All Ceph manipulations have been done from the PVE UI, as 
>> we are not very experienced with the Ceph CLI.
>>
>> We are running PVE 8.2.3 and Ceph runs on version 17.2.7.
>>
>> Is this inconsistency normal or a problem? And if the latter, then 
>> (how) can it be fixed?
>>
>> Cheers, Frank
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[David der Nederlanden | ITTY via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 15553 bytes --]

From: David der Nederlanden | ITTY <david@itty.nl>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: RE: [PVE-User] "nearfull" status in PVE Dashboard not consistent
Date: Sat, 7 Sep 2024 19:27:38 +0000
Message-ID: <AM8P193MB11393CE9AC8873699D5E5950B89F2@AM8P193MB1139.EURP193.PROD.OUTLOOK.COM>

Hi Frank,

Can you share your OSD layout too?

My first thought is that you added the SSD's as OSD which caused that OSD to get full, with a near full pool as a result.

You can get some insights with:
`ceph osd tree`

And if needed you can reweight the OSD's, but that would require a good OSD layout:
`ceph osd reweight-by-utilization`

Sources:
https://forum.proxmox.com/threads/ceph-pool-full.47810/ 
https://docs.ceph.com/en/reef/rados/operations/health-checks/#pool-near-full

Kind regards,
David der Nederlanden

-----Oorspronkelijk bericht-----
Van: pve-user <pve-user-bounces@lists.proxmox.com> Namens Frank Thommen
Verzonden: Saturday, September 7, 2024 21:15
Aan: pve-user@lists.proxmox.com
Onderwerp: Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

Mailman is making fun of me: First it does not accept the mail because of too big attachments, now that I reduced the size, it removes them completely :-(

Sorry, I digress... Please find the two images here:

   * https://pasteboard.co/CUPNjkTmyYV8.jpg
(Ceph_dashboard_nearfull_warning.jpg)
   * https://pasteboard.co/34GBggOiUNII.jpg (Ceph_pool_overview.jpg)

HTH, Frank


On 07.09.24 21:07, Frank Thommen wrote:
> It seems, the attachments got lost on their way. Here they are (again) 
> Frank
> 
> On 07.09.24 21:04, Frank Thommen wrote:
>> Dear all,
>>
>> I am currently in the process to add SSDs for DB/WAL to our 
>> "converged" 3-node Ceph cluster. After having done so on two of three 
>> nodes, the PVE Ceph dashboard now reports "5 pool(s) nearfull":
>>
>>       HEALTH_WARN: 5 pool(s) nearfull
>>       pool 'pve-pool1' is nearfull
>>       pool 'cephfs_data' is nearfull
>>       pool 'cephfs_metadata' is nearfull
>>       pool '.mgr' is nearfull
>>       pool '.rgw.root' is nearfull
>>
>> (see also attached Ceph_dashboard_nearfull_warning.jpg). The storage 
>> in general is 73% full ("40.87 TiB of 55.67 TiB").
>>
>> However when looking at the pool overview in PVE, the pools don't 
>> seem to be very full at all. Some of them are even reported as being 
>> completely empty (see the attached Ceph_pool_overview.jpg).
>>
>> Please note: All Ceph manipulations have been done from the PVE UI, 
>> as we are not very experienced with the Ceph CLI.
>>
>> We are running PVE 8.2.3 and Ceph runs on version 17.2.7.
>>
>> Is this inconsistency normal or a problem? And if the latter, then
>> (how) can it be fixed?
>>
>> Cheers, Frank
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[Peter Eisch via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 4497 bytes --]

From: Peter Eisch <peter@boku.net>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent
Date: Sat, 7 Sep 2024 14:49:39 -0500 (CDT)
Message-ID: <854037bc-bd1a-4dc7-91e7-73ccc68c73db@boku.net>

Also 'ceph osd df' would be useful to help.

peter

Sep 7, 2024 14:43:18 David der Nederlanden | ITTY via pve-user <pve-user@lists.proxmox.com>:

> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

Hi David,

I have deleted the OSDs one by one (sometimes two by two) and then 
recreated them, but this time with an SSD partition as DB device 
(previously the DB was on the OSD HDDs). So the SSDs should not have 
been added as OSDs.

`ceph osd tree` gives me
----------------------------------
$ ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME            STATUS  REWEIGHT  PRI-AFF
-1         55.66704  root default
-3         18.40823      host pve01
  0    hdd   3.68259          osd.0            up   1.00000  1.00000
  1    hdd   3.68259          osd.1            up   1.00000  1.00000
  2    hdd   3.68259          osd.2            up   1.00000  1.00000
  9    hdd   1.84380          osd.9            up   1.00000  1.00000
10    hdd   1.84380          osd.10           up   1.00000  1.00000
11    hdd   1.84380          osd.11           up   1.00000  1.00000
12    hdd   1.82909          osd.12           up   1.00000  1.00000
-5         19.06548      host pve02
  3    hdd   3.81450          osd.3            up   1.00000  1.00000
  4    hdd   3.81450          osd.4            up   1.00000  1.00000
  5    hdd   3.81450          osd.5            up   1.00000  1.00000
13    hdd   1.90720          osd.13           up   1.00000  1.00000
14    hdd   1.90720          osd.14           up   1.00000  1.00000
15    hdd   1.90720          osd.15           up   1.00000  1.00000
16    hdd   1.90039          osd.16           up   1.00000  1.00000
-7         18.19333      host pve03
  6    hdd   3.63869          osd.6            up   1.00000  1.00000
  7    hdd   3.63869          osd.7            up   1.00000  1.00000
  8    hdd   3.63869          osd.8            up   1.00000  1.00000
17    hdd   1.81929          osd.17           up   1.00000  1.00000
18    hdd   1.81929          osd.18           up   1.00000  1.00000
19    hdd   1.81940          osd.19           up   1.00000  1.00000
20    hdd   1.81929          osd.20           up   1.00000  1.00000
$
----------------------------------

Cheers, Frank



On 07.09.24 21:27, David der Nederlanden | ITTY via pve-user wrote:
> Hi Frank,
> 
> Can you share your OSD layout too?
> 
> My first thought is that you added the SSD's as OSD which caused that OSD to get full, with a near full pool as a result.
> 
> You can get some insights with:
> `ceph osd tree`
> 
> And if needed you can reweight the OSD's, but that would require a good OSD layout:
> `ceph osd reweight-by-utilization`
> 
> Sources:
> https://forum.proxmox.com/threads/ceph-pool-full.47810/  
> https://docs.ceph.com/en/reef/rados/operations/health-checks/#pool-near-full
> 
> Kind regards,
> David der Nederlanden


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

`ceph osd df` gives me:

----------------------------------
$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META 
AVAIL    %USE   VAR   PGS  STATUS
  0    hdd  3.68259   1.00000  3.7 TiB  3.0 TiB  2.9 TiB   38 MiB  8.9 
GiB  731 GiB  80.63  1.10  189      up
  1    hdd  3.68259   1.00000  3.7 TiB  2.5 TiB  2.5 TiB   34 MiB  7.7 
GiB  1.2 TiB  68.31  0.93  159      up
  2    hdd  3.68259   1.00000  3.7 TiB  3.0 TiB  2.9 TiB   32 MiB  7.6 
GiB  713 GiB  81.10  1.10  174      up
  9    hdd  1.84380   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   13 MiB  4.7 
GiB  652 GiB  65.49  0.89   71      up
10    hdd  1.84380   1.00000  1.8 TiB  1.3 TiB  1.3 TiB   16 MiB  4.8 
GiB  543 GiB  71.24  0.97   85      up
11    hdd  1.84380   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   13 MiB  4.7 
GiB  659 GiB  65.09  0.89   71      up
12    hdd  1.82909   1.00000  1.8 TiB  1.5 TiB  1.4 TiB   15 MiB  5.5 
GiB  380 GiB  79.70  1.09   84      up
  3    hdd  3.81450   1.00000  3.8 TiB  2.8 TiB  2.6 TiB   28 MiB  7.2 
GiB  1.0 TiB  73.65  1.00  158      up
  4    hdd  3.81450   1.00000  3.8 TiB  2.1 TiB  2.0 TiB   27 MiB  6.4 
GiB  1.7 TiB  56.06  0.76  137      up
  5    hdd  3.81450   1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5 
GiB  568 GiB  85.45  1.16  194      up
13    hdd  1.90720   1.00000  1.9 TiB  1.4 TiB  1.3 TiB   14 MiB  3.3 
GiB  543 GiB  72.22  0.98   78      up
14    hdd  1.90720   1.00000  1.9 TiB  1.6 TiB  1.5 TiB   19 MiB  5.1 
GiB  297 GiB  84.78  1.15   91      up
15    hdd  1.90720   1.00000  1.9 TiB  1.5 TiB  1.4 TiB   18 MiB  4.9 
GiB  444 GiB  77.28  1.05   82      up
16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB  5.4 
GiB  261 GiB  86.59  1.18   93      up
  6    hdd  3.63869   1.00000  3.6 TiB  2.6 TiB  2.6 TiB   35 MiB  5.9 
GiB  1.0 TiB  71.69  0.98  172      up
  7    hdd  3.63869   1.00000  3.6 TiB  2.8 TiB  2.8 TiB   30 MiB  6.3 
GiB  875 GiB  76.53  1.04  173      up
  8    hdd  3.63869   1.00000  3.6 TiB  2.3 TiB  2.3 TiB   29 MiB  5.4 
GiB  1.3 TiB  63.51  0.86  147      up
17    hdd  1.81929   1.00000  1.8 TiB  1.3 TiB  1.3 TiB   20 MiB  3.0 
GiB  486 GiB  73.93  1.01   92      up
18    hdd  1.81929   1.00000  1.8 TiB  1.5 TiB  1.5 TiB   17 MiB  3.7 
GiB  367 GiB  80.30  1.09   93      up
19    hdd  1.81940   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   17 MiB  3.6 
GiB  646 GiB  65.35  0.89   78      up
20    hdd  1.81929   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   15 MiB  2.7 
GiB  631 GiB  66.16  0.90   78      up
                        TOTAL   56 TiB   41 TiB   40 TiB  483 MiB  115 
GiB   15 TiB  73.42
MIN/MAX VAR: 0.76/1.18  STDDEV: 8.03
$
----------------------------------

Frank


On 07.09.24 21:49, Peter Eisch via pve-user wrote:
> Also 'ceph osd df' would be useful to help.
> 
> peter


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] "nearfull" status in PVE Dashboard not consistent

[Bryan Fields]

On 9/7/24 3:14 PM, Frank Thommen wrote:
> Sorry, I digress... Please find the two images here:
> 
>    * https://pasteboard.co/CUPNjkTmyYV8.jpg 
> (Ceph_dashboard_nearfull_warning.jpg)
>    * https://pasteboard.co/34GBggOiUNII.jpg (Ceph_pool_overview.jpg)
> 
> HTH, Frank

The links are broken, "502 Bad Gateway, nginx/1.1.19"

-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Eneko Lacunza via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 6018 bytes --]

From: Eneko Lacunza <elacunza@binovo.es>
To: pve-user@lists.proxmox.com
Subject: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
Date: Mon, 9 Sep 2024 12:36:59 +0200
Message-ID: <e78689f7-58e2-4a4f-8a3e-e65f627e8936@binovo.es>

Hi Frank,

El 8/9/24 a las 14:17, Frank Thommen escribió:
>  5 hdd  3.81450   1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5 
> GiB  568 GiB  85.45  1.16  194      up
> 16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB 5.4 
> GiB  261 GiB  86.59  1.18   93      up

Those OSD are nearfull, if you have the default 0.85 value for that. You 
can try to lower a bit their weight.

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/



[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

pasteboard.co seems to be out-of-order at the time. Are there good 
alternatives? It would really be easier, if I could just send 
attachments to the list :-/

On 09.09.24 02:46, Bryan Fields wrote:
> On 9/7/24 3:14 PM, Frank Thommen wrote:
>> Sorry, I digress... Please find the two images here:
>>
>>     * https://pasteboard.co/CUPNjkTmyYV8.jpg
>> (Ceph_dashboard_nearfull_warning.jpg)
>>     * https://pasteboard.co/34GBggOiUNII.jpg (Ceph_pool_overview.jpg)
>>
>> HTH, Frank
> 
> The links are broken, "502 Bad Gateway, nginx/1.1.19"
> 

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

Yes, but Ceph also reports five(!) nearfull pools, but the pool overview 
doesn't reflect that. I'd love to show this with CLI, but I could not 
find an appropriate command so far. Hopefully, inline images work:

 From the Ceph overview dashboard:

And from the pool overview

Two pools are quite full (80% and 82%), but far from nearfull, and the 
other tree are basically empty.

Frank


On 09.09.24 12:36, Eneko Lacunza via pve-user wrote:
> Hi Frank,
>
> El 8/9/24 a las 14:17, Frank Thommen escribió:
>>  5 hdd  3.81450 1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5 GiB  
>> 568 GiB 85.45  1.16  194      up
>> 16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB 5.4 
>> GiB  261 GiB  86.59  1.18   93      up
>
> Those OSD are nearfull, if you have the default 0.85 value for that. 
> You can try to lower a bit their weight.
>
> Cheers
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project 
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[David der Nederlanden | ITTY via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 15591 bytes --]

From: David der Nederlanden | ITTY <david@itty.nl>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: RE: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
Date: Tue, 10 Sep 2024 18:31:51 +0000
Message-ID: <AM8P193MB1139BBAFA85B6FA5CFFADE5AB89A2@AM8P193MB1139.EURP193.PROD.OUTLOOK.COM>

Hi Frank,

The images didn't work :)

Pool and osd nearfull are closely related, when OSD's get full your pool also gets nearfull as Ceph needs to be able to follow the crush rules,
which it can't if one of the OSD's gets full, hence it warns when it gets nearfull.

I see that you're mixing OSD sizes, deleting and recreating the OSD's one by one caused this, as the OSD's got new weights you should be OK when you reweight them.
You can do this by hand or using reweight-by-utilization, what you prefer.

Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max rule should never be above 80%, as this gives you a nearfull pool when it starts backfilling when you lose one node, or even a full pool worst case, rendering the pool read only.

Kind regards,
David



-----Oorspronkelijk bericht-----
Van: pve-user <pve-user-bounces@lists.proxmox.com> Namens Frank Thommen
Verzonden: Tuesday, September 10, 2024 14:02
Aan: pve-user@lists.proxmox.com
Onderwerp: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

Yes, but Ceph also reports five(!) nearfull pools, but the pool overview doesn't reflect that. I'd love to show this with CLI, but I could not find an appropriate command so far. Hopefully, inline images work:

 From the Ceph overview dashboard:

And from the pool overview

Two pools are quite full (80% and 82%), but far from nearfull, and the other tree are basically empty.

Frank


On 09.09.24 12:36, Eneko Lacunza via pve-user wrote:
> Hi Frank,
>
> El 8/9/24 a las 14:17, Frank Thommen escribió:
>>  5 hdd  3.81450 1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5 GiB
>> 568 GiB 85.45  1.16  194      up
>> 16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB 5.4 
>> GiB  261 GiB  86.59  1.18   93      up
>
> Those OSD are nearfull, if you have the default 0.85 value for that. 
> You can try to lower a bit their weight.
>
> Cheers
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

ok, here are - finally - the screenshots:

   * from the Ceph overview dashboard: https://postimg.cc/fVJFstqW
   * from the pool overview: https://postimg.cc/ykf52XPr

Frank


On 10.09.24 14:02, Frank Thommen wrote:
> Yes, but Ceph also reports five(!) nearfull pools, but the pool overview 
> doesn't reflect that. I'd love to show this with CLI, but I could not 
> find an appropriate command so far. Hopefully, inline images work:
> 
>  From the Ceph overview dashboard:
> 
> And from the pool overview
> 
> Two pools are quite full (80% and 82%), but far from nearfull, and the 
> other tree are basically empty.
> 
> Frank
> 
> 
> On 09.09.24 12:36, Eneko Lacunza via pve-user wrote:
>> Hi Frank,
>>
>> El 8/9/24 a las 14:17, Frank Thommen escribió:
>>>  5 hdd  3.81450 1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5 GiB 
>>> 568 GiB 85.45  1.16  194      up
>>> 16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB 5.4 
>>> GiB  261 GiB  86.59  1.18   93      up
>>
>> Those OSD are nearfull, if you have the default 0.85 value for that. 
>> You can try to lower a bit their weight.
>>
>> Cheers
>>
>> Eneko Lacunza
>> Zuzendari teknikoa | Director técnico
>> Binovo IT Human Project 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

The OSDs are of different size, because we have 4 TB and 2 TB disks in 
the systems.

We might give the reweight a try.



On 10.09.24 20:31, David der Nederlanden | ITTY via pve-user wrote:
> Hi Frank,
> 
> The images didn't work 🙂
> 
> Pool and osd nearfull are closely related, when OSD's get full your pool
>   also gets nearfull as Ceph needs to be able to follow the crush rules,
> which it can't if one of the OSD's gets full, hence it warns when it
> gets nearfull.
> 
> I see that you're mixing OSD sizes, deleting and recreating the OSD's
> one by one caused this, as the OSD's got new weights you should be OK
> when you reweight them.
> You can do this by hand or using reweight-by-utilization, what you
> prefer.
> 
> Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max
> rule should never be above 80%, as this gives you a nearfull pool when
> it starts backfilling when you lose one node, or even a full pool worst
> case, rendering the pool read only.
> 
> Kind regards,
> David

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Daniel Oliver]

The built-in Ceph balancer only balances based on PG numbers, which can vary wildly in size for several reasons.

I ended up disabling the built-in balancer and switching to https://github.com/TheJJ/ceph-balancer, which we now run daily with the following parameters:
placementoptimizer.py balance --ignore-ideal-pgcounts=all --osdused=delta --osdfrom fullest

This keeps things nicely balanced from a fullness perspective, with the most important bit being ignore-ideal-pgcounts, as it allows balancing decisions outside of what the built-in balancer would decide.

From: pve-user <pve-user-bounces@lists.proxmox.com> on behalf of Frank Thommen <f.thommen@dkfz-heidelberg.de>
Date: Wednesday, 11 September 2024 at 12:00
To: pve-user@lists.proxmox.com <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
The OSDs are of different size, because we have 4 TB and 2 TB disks in
the systems.

We might give the reweight a try.



On 10.09.24 20:31, David der Nederlanden | ITTY via pve-user wrote:
> Hi Frank,
>
> The images didn't work 🙂
>
> Pool and osd nearfull are closely related, when OSD's get full your pool
>   also gets nearfull as Ceph needs to be able to follow the crush rules,
> which it can't if one of the OSD's gets full, hence it warns when it
> gets nearfull.
>
> I see that you're mixing OSD sizes, deleting and recreating the OSD's
> one by one caused this, as the OSD's got new weights you should be OK
> when you reweight them.
> You can do this by hand or using reweight-by-utilization, what you
> prefer.
>
> Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max
> rule should never be above 80%, as this gives you a nearfull pool when
> it starts backfilling when you lose one node, or even a full pool worst
> case, rendering the pool read only.
>
> Kind regards,
> David

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

-- 
Unless expressly stated otherwise, the contents of this e-mail represent 
only the views of the sender as expressed only to the intended recipient. 
It does not commit ripjar limited to any course of action or legal 
responsibility. This e-mail and the information it contains may be 
privileged and/or confidential. It is for the intended addressee(s) only. 
The unauthorised use, disclosure or copying of this e-mail, or any 
information it contains, is prohibited and could, in certain circumstances, 
be a criminal offence. If you are not an intended recipient, please notify 
admin@ripjar.com <mailto:admin@ripjar.com> immediately.


RIPJAR LIMITED - 
Registered in England & Wales - Company No: 8217339 - Registered Address: 
Suite 404, Eagle Tower, Montpellier Drive, Cheltenham, Gloucestershire GL50 
1TA.
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

I'm not sure, if that is related, but looking at the activity LEDs of 
the DB/WAL SSD devices (two SSDs in RAID 1 each) in each host, the 
device in one host is basically permanently active (LEDs constantly 
flickering w/o pause), while on the other host, the device seems almost 
completely inactive (one blink every few seconds). The third host has no 
SSD DB device yet. To me that looks, as one Ceph node is extremely 
active, while the other isn't. That also looks like an imbalance to me.

Frank


On 11.09.24 13:00, Frank Thommen wrote:
> The OSDs are of different size, because we have 4 TB and 2 TB disks in 
> the systems.
> 
> We might give the reweight a try.
> 
> 
> 
> On 10.09.24 20:31, David der Nederlanden | ITTY via pve-user wrote:
>> Hi Frank,
>>
>> The images didn't work 🙂
>>
>> Pool and osd nearfull are closely related, when OSD's get full your pool
>>   also gets nearfull as Ceph needs to be able to follow the crush rules,
>> which it can't if one of the OSD's gets full, hence it warns when it
>> gets nearfull.
>>
>> I see that you're mixing OSD sizes, deleting and recreating the OSD's
>> one by one caused this, as the OSD's got new weights you should be OK
>> when you reweight them.
>> You can do this by hand or using reweight-by-utilization, what you
>> prefer.
>>
>> Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max
>> rule should never be above 80%, as this gives you a nearfull pool when
>> it starts backfilling when you lose one node, or even a full pool worst
>> case, rendering the pool read only.
>>
>> Kind regards,
>> David
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

I will have a look. However, not having real working experience with 
Ceph, using an external balancer requires a "leap of faith" from my side :-)


On 11.09.24 13:52, Daniel Oliver wrote:
> The built-in Ceph balancer only balances based on PG numbers, which can vary wildly in size for several reasons.
> 
> I ended up disabling the built-in balancer and switching to https://github.com/TheJJ/ceph-balancer, which we now run daily with the following parameters:
> placementoptimizer.py balance --ignore-ideal-pgcounts=all --osdused=delta --osdfrom fullest
> 
> This keeps things nicely balanced from a fullness perspective, with the most important bit being ignore-ideal-pgcounts, as it allows balancing decisions outside of what the built-in balancer would decide.
> 
> From: pve-user <pve-user-bounces@lists.proxmox.com> on behalf of Frank Thommen <f.thommen@dkfz-heidelberg.de>
> Date: Wednesday, 11 September 2024 at 12:00
> To: pve-user@lists.proxmox.com <pve-user@lists.proxmox.com>
> Subject: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
> The OSDs are of different size, because we have 4 TB and 2 TB disks in
> the systems.
> 
> We might give the reweight a try.
> 
> 
> 
> On 10.09.24 20:31, David der Nederlanden | ITTY via pve-user wrote:
>> Hi Frank,
>>
>> The images didn't work 🙂
>>
>> Pool and osd nearfull are closely related, when OSD's get full your pool
>>    also gets nearfull as Ceph needs to be able to follow the crush rules,
>> which it can't if one of the OSD's gets full, hence it warns when it
>> gets nearfull.
>>
>> I see that you're mixing OSD sizes, deleting and recreating the OSD's
>> one by one caused this, as the OSD's got new weights you should be OK
>> when you reweight them.
>> You can do this by hand or using reweight-by-utilization, what you
>> prefer.
>>
>> Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max
>> rule should never be above 80%, as this gives you a nearfull pool when
>> it starts backfilling when you lose one node, or even a full pool worst
>> case, rendering the pool read only.
>>
>> Kind regards,
>> David
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[David der Nederlanden | ITTY via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 12152 bytes --]

From: David der Nederlanden | ITTY <david@itty.nl>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: RE: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
Date: Tue, 10 Sep 2024 18:32:19 +0000
Message-ID: <AM8P193MB1139009B3EB1F017922B2610B89A2@AM8P193MB1139.EURP193.PROD.OUTLOOK.COM>

In hindsight, after reading some more about this, I don't think this is relevant.

-----Oorspronkelijk bericht-----
Van: pve-user <pve-user-bounces@lists.proxmox.com> Namens Frank Thommen
Verzonden: Tuesday, September 10, 2024 14:13
Aan: pve-user@lists.proxmox.com
Onderwerp: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

How/where can I see that? All commands that I found so far, only print out size and/or usage of the OSDs themselves. Also with LVM commands I only managed to see the size of the DB LVs, but not the current usage.


On 09.09.24 23:01, David der Nederlanden | ITTY via pve-user wrote:
> Thanks, the used procedure seems solid.
>
> In addition, how full are your DB/WAL drives?
>
> Kind regards,
> David

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Frank Thommen]

How/where can I see that? All commands that I found so far, only print 
out size and/or usage of the OSDs themselves. Also with LVM commands I 
only managed to see the size of the DB LVs, but not the current usage.


On 09.09.24 23:01, David der Nederlanden | ITTY via pve-user wrote:
> Thanks, the used procedure seems solid.
> 
> In addition, how full are your DB/WAL drives?
> 
> Kind regards,
> David

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[David der Nederlanden | ITTY via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 15583 bytes --]

From: David der Nederlanden | ITTY <david@itty.nl>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
Date: Mon, 9 Sep 2024 21:01:49 +0000
Message-ID: <c94117a4-2c9e-458b-91ee-04f2f856866e@email.android.com>

Thanks, the used procedure seems solid.

In addition, how full are your DB/WAL drives?

Kind regards,
David

On 8 Sept 2024 14:18, Frank Thommen <f.thommen@dkfz-heidelberg.de> wrote:
`ceph osd df` gives me:

----------------------------------
$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META
AVAIL    %USE   VAR   PGS  STATUS
  0    hdd  3.68259   1.00000  3.7 TiB  3.0 TiB  2.9 TiB   38 MiB  8.9
GiB  731 GiB  80.63  1.10  189      up
  1    hdd  3.68259   1.00000  3.7 TiB  2.5 TiB  2.5 TiB   34 MiB  7.7
GiB  1.2 TiB  68.31  0.93  159      up
  2    hdd  3.68259   1.00000  3.7 TiB  3.0 TiB  2.9 TiB   32 MiB  7.6
GiB  713 GiB  81.10  1.10  174      up
  9    hdd  1.84380   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   13 MiB  4.7
GiB  652 GiB  65.49  0.89   71      up
10    hdd  1.84380   1.00000  1.8 TiB  1.3 TiB  1.3 TiB   16 MiB  4.8
GiB  543 GiB  71.24  0.97   85      up
11    hdd  1.84380   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   13 MiB  4.7
GiB  659 GiB  65.09  0.89   71      up
12    hdd  1.82909   1.00000  1.8 TiB  1.5 TiB  1.4 TiB   15 MiB  5.5
GiB  380 GiB  79.70  1.09   84      up
  3    hdd  3.81450   1.00000  3.8 TiB  2.8 TiB  2.6 TiB   28 MiB  7.2
GiB  1.0 TiB  73.65  1.00  158      up
  4    hdd  3.81450   1.00000  3.8 TiB  2.1 TiB  2.0 TiB   27 MiB  6.4
GiB  1.7 TiB  56.06  0.76  137      up
  5    hdd  3.81450   1.00000  3.8 TiB  3.3 TiB  3.1 TiB   37 MiB  8.5
GiB  568 GiB  85.45  1.16  194      up
13    hdd  1.90720   1.00000  1.9 TiB  1.4 TiB  1.3 TiB   14 MiB  3.3
GiB  543 GiB  72.22  0.98   78      up
14    hdd  1.90720   1.00000  1.9 TiB  1.6 TiB  1.5 TiB   19 MiB  5.1
GiB  297 GiB  84.78  1.15   91      up
15    hdd  1.90720   1.00000  1.9 TiB  1.5 TiB  1.4 TiB   18 MiB  4.9
GiB  444 GiB  77.28  1.05   82      up
16    hdd  1.90039   1.00000  1.9 TiB  1.6 TiB  1.6 TiB   19 MiB  5.4
GiB  261 GiB  86.59  1.18   93      up
  6    hdd  3.63869   1.00000  3.6 TiB  2.6 TiB  2.6 TiB   35 MiB  5.9
GiB  1.0 TiB  71.69  0.98  172      up
  7    hdd  3.63869   1.00000  3.6 TiB  2.8 TiB  2.8 TiB   30 MiB  6.3
GiB  875 GiB  76.53  1.04  173      up
  8    hdd  3.63869   1.00000  3.6 TiB  2.3 TiB  2.3 TiB   29 MiB  5.4
GiB  1.3 TiB  63.51  0.86  147      up
17    hdd  1.81929   1.00000  1.8 TiB  1.3 TiB  1.3 TiB   20 MiB  3.0
GiB  486 GiB  73.93  1.01   92      up
18    hdd  1.81929   1.00000  1.8 TiB  1.5 TiB  1.5 TiB   17 MiB  3.7
GiB  367 GiB  80.30  1.09   93      up
19    hdd  1.81940   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   17 MiB  3.6
GiB  646 GiB  65.35  0.89   78      up
20    hdd  1.81929   1.00000  1.8 TiB  1.2 TiB  1.2 TiB   15 MiB  2.7
GiB  631 GiB  66.16  0.90   78      up
                        TOTAL   56 TiB   41 TiB   40 TiB  483 MiB  115
GiB   15 TiB  73.42
MIN/MAX VAR: 0.76/1.18  STDDEV: 8.03
$
----------------------------------

Frank


On 07.09.24 21:49, Peter Eisch via pve-user wrote:
> Also 'ceph osd df' would be useful to help.
>
> peter


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user



[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user