Re: [pve-devel] [PATCH docs] pveceph: update OSD memory considerations

[Alwin Antreich via pve-devel <pve-devel@lists.proxmox.com>]

[-- Attachment #1: Type: message/rfc822, Size: 7123 bytes --]

From: Alwin Antreich <alwin@antreich.com>
To: pve-devel@lists.proxmox.com
Subject: Re: [PATCH docs] pveceph: update OSD memory considerations
Date: Sat, 20 Sep 2025 20:21:15 +0200
Message-ID: <18C016FD-4966-40C9-8E78-F343A68D928A@antreich.com>

On 19 September 2025 14:00:18 CEST, Aaron Lauterer <a.lauterer@proxmox.com> wrote:
>thanks for the patch! see inline for comments
>
>On  2025-09-18  18:45, Alwin Antreich wrote:
>> Since bluestore, OSDs adhere to the osd_memory_target and the
>> recommended amount of memory was increased.
>> 
>> See: https://docs.ceph.com/en/reef/start/hardware-recommendations/#ram
>> 
>> Signed-off-by: Alwin Antreich <alwin@antreich.com>
>> ---
>>   pveceph.adoc | 16 ++++++++--------
>>   1 file changed, 8 insertions(+), 8 deletions(-)
>> 
>> diff --git a/pveceph.adoc b/pveceph.adoc
>> index 17efa4d..a2d71e7 100644
>> --- a/pveceph.adoc
>> +++ b/pveceph.adoc
>> @@ -131,14 +131,14 @@ carefully planned out and monitored. In addition to the predicted memory usage
>>   of virtual machines and containers, you must also account for having enough
>>   memory available for Ceph to provide excellent and stable performance.
>>   -As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
>> -by an OSD. While the usage might be less under normal conditions, it will use
>> -most during critical operations like recovery, re-balancing or backfilling.
>> -That means that you should avoid maxing out your available memory already on
>> -normal operation, but rather leave some headroom to cope with outages.
>> -
>> -The OSD service itself will use additional memory. The Ceph BlueStore backend of
>> -the daemon requires by default **3-5 GiB of memory** (adjustable).
>> +While usage may be less under normal conditions, it will consume more memory
>> +during critical operations, such as recovery, rebalancing, or backfilling. That
>> +means you should avoid maxing out your available memory already on regular
>> +operation, but rather leave some headroom to cope with outages.
>> +
>> +The current recommendation is to configure at least **8 GiB of memory per OSD
>> +daemon** for good performance. The OSD daemon requires, by default, 4 GiB of
>> +memory.
>
>given how the current latest Ceph docs phrase it [0], I am not sure here. They sound like the default osd_memory_target of 4G is okay, but that they might use more in recovery situations and one should calculate with ~8G.
>
>So unless I understand that wrong, maybe we could phrase it more like the following?
>===
>The current recommendation is to calculate with at least 8 GiB of memory per OSD daemon to give it enough memory if needed. By default, the OSD daemon is set to use up to 4 GiB of memory in normal scenarios.
>===
>
>If I understand it wrong and users should change the osd_memory_target to 8 GiB, we should document how, or maybe even try to make it configurable in the GUI/API/pveceph...

I didn't want to clutter the cluster sizing text with configuration details.

The OSD daemon will adhere to the osd_memory_target , as it isn't a limit, the OSD may overshoot by 10-20%, as buffers (probably other things) aren't accounted for. Unless auto tuning is enabled, the memory_target should be adjusted to 8GiB. The experience we gathered also shows that 8GiB is worth it, especially when the cluster is degraded.

See inline



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel