From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 8B4B01FF186 for ; Sat, 20 Sep 2025 20:21:10 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6BB3F27472; Sat, 20 Sep 2025 20:21:25 +0200 (CEST) Date: Sat, 20 Sep 2025 20:21:15 +0200 To: pve-devel@lists.proxmox.com In-Reply-To: References: <20250918164549.3018879-1-alwin@antreich.com> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Alwin Antreich via pve-devel Precedence: list Cc: Alwin Antreich X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] [PATCH docs] pveceph: update OSD memory considerations Content-Type: multipart/mixed; boundary="===============1040939840173454783==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============1040939840173454783== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 953EAD300B for ; Sat, 20 Sep 2025 20:21:23 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6A9A12744E for ; Sat, 20 Sep 2025 20:21:23 +0200 (CEST) Received: from mx.antreich.com (mx.antreich.com [173.249.42.230]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Sat, 20 Sep 2025 20:21:22 +0200 (CEST) Received: from mx.antreich.com (localhost [127.0.0.1]) by mx.antreich.com (Proxmox) with ESMTP id A4F2B764EA1; Sat, 20 Sep 2025 20:21:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=antreich.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :from:from:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=2025; bh=3uDRN+24SeFFfwXbgYLm eZT0HQPOX3die7UcnNaxjgA=; b=hBRGp58az7OXeVa1hMKM6nhZAmzQIBXGNJxo PG/NbZfeDDecUhktO9IYPEVvyVcxDznox1cktvmaWmtOYYP2VuqW6JeYrhZ7xOgU 6dz53hhtObk3Ev4WSjURmPXQJp0NpogW29Vso7GqyUXWEHYI9E0PnQuJcOFbyq5F s5u3bjaaLxfr1i97+FiuAyvIFbhX504lp7tajiXK45/tgBZfVIc+ZS6jwv26A93w wW5U5zA3EWavOwNEhMPBhXXgXd5TOSq9ItuYTRw2Ka1usJtbacrc2Dii2o2ag1rW 9FfVg1kSmXBVRmUjuo/nxSa98snatQCWUDkwF580BBp5sc8aQw== Date: Sat, 20 Sep 2025 20:21:15 +0200 From: Alwin Antreich To: pve-devel@lists.proxmox.com Subject: Re: [PATCH docs] pveceph: update OSD memory considerations In-Reply-To: References: <20250918164549.3018879-1-alwin@antreich.com> Message-ID: <18C016FD-4966-40C9-8E78-F343A68D928A@antreich.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.091 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [ceph.com,antreich.com] On 19 September 2025 14:00:18 CEST, Aaron Lauterer wrote: >thanks for the patch! see inline for comments > >On 2025-09-18 18:45, Alwin Antreich wrote: >> Since bluestore, OSDs adhere to the osd_memory_target and the >> recommended amount of memory was increased=2E >>=20 >> See: https://docs=2Eceph=2Ecom/en/reef/start/hardware-recommendations/#= ram >>=20 >> Signed-off-by: Alwin Antreich >> --- >> pveceph=2Eadoc | 16 ++++++++-------- >> 1 file changed, 8 insertions(+), 8 deletions(-) >>=20 >> diff --git a/pveceph=2Eadoc b/pveceph=2Eadoc >> index 17efa4d=2E=2Ea2d71e7 100644 >> --- a/pveceph=2Eadoc >> +++ b/pveceph=2Eadoc >> @@ -131,14 +131,14 @@ carefully planned out and monitored=2E In additio= n to the predicted memory usage >> of virtual machines and containers, you must also account for having = enough >> memory available for Ceph to provide excellent and stable performance= =2E >> -As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** w= ill be used >> -by an OSD=2E While the usage might be less under normal conditions, it= will use >> -most during critical operations like recovery, re-balancing or backfil= ling=2E >> -That means that you should avoid maxing out your available memory alre= ady on >> -normal operation, but rather leave some headroom to cope with outages= =2E >> - >> -The OSD service itself will use additional memory=2E The Ceph BlueStor= e backend of >> -the daemon requires by default **3-5 GiB of memory** (adjustable)=2E >> +While usage may be less under normal conditions, it will consume more = memory >> +during critical operations, such as recovery, rebalancing, or backfill= ing=2E That >> +means you should avoid maxing out your available memory already on reg= ular >> +operation, but rather leave some headroom to cope with outages=2E >> + >> +The current recommendation is to configure at least **8 GiB of memory = per OSD >> +daemon** for good performance=2E The OSD daemon requires, by default, = 4 GiB of >> +memory=2E > >given how the current latest Ceph docs phrase it [0], I am not sure here= =2E They sound like the default osd_memory_target of 4G is okay, but that t= hey might use more in recovery situations and one should calculate with ~8G= =2E > >So unless I understand that wrong, maybe we could phrase it more like the= following? >=3D=3D=3D >The current recommendation is to calculate with at least 8 GiB of memory = per OSD daemon to give it enough memory if needed=2E By default, the OSD da= emon is set to use up to 4 GiB of memory in normal scenarios=2E >=3D=3D=3D > >If I understand it wrong and users should change the osd_memory_target to= 8 GiB, we should document how, or maybe even try to make it configurable i= n the GUI/API/pveceph=2E=2E=2E I didn't want to clutter the cluster sizing text with configuration detail= s=2E The OSD daemon will adhere to the osd_memory_target , as it isn't a limit,= the OSD may overshoot by 10-20%, as buffers (probably other things) aren't= accounted for=2E Unless auto tuning is enabled, the memory_target should b= e adjusted to 8GiB=2E The experience we gathered also shows that 8GiB is wo= rth it, especially when the cluster is degraded=2E See inline --===============1040939840173454783== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============1040939840173454783==--