From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-user-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 9FF5A1FF161 for <inbox@lore.proxmox.com>; Sat, 12 Apr 2025 19:44:15 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 531EF3099; Sat, 12 Apr 2025 19:44:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mattcorallo.com; s=1744478462; h=Subject:From:To:From:Subject:To:Cc:Cc: Reply-To:In-Reply-To:References; bh=+bjQ8UNs+x3ooo3khj83DOBc4BplunCG0jI6N/1gits=; b=UK3yU8BQEoUh/kO2rUZrxdv3OS r4BlfxSmdGCyt6FHgHLVvUh5rMGWRCzNaPOviiCz6vlE2mElsp2CEjIm6NtnnvPShI7tzOsfBZG7y T2RTg2MayAStFkcMUauXjU3seWqRd2B0R8k9tkUegsO2Nw4SUB20g4Gb2OA0YdETUKUJJEmT6KNwJ 9F6ueGtX90Fzbt6b77eqFzGFni/FMUg2Cgu+XWh7t6yKXbptLsjEhfLAHABcBeRXsTGrpIG5gcgW5 YVzE97P2y1bsOuR1fwSVpfbM2nus8hvweUGk4gIchkqf4aMOzVLREsL8k4kLdurFpHmjqNq+j7fRW 6+F+3gfQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=clients.mail.as397444.net; s=1744478465; h=Subject:From:To:From:Subject:To: Cc:Cc:Reply-To:In-Reply-To:References; bh=+bjQ8UNs+x3ooo3khj83DOBc4BplunCG0jI6N/1gits=; b=QAKnGeyU1QH1yRhneI3bd9A0qH +BwwfeRWdLCgjf1pK9SX5IzsIfqiR3YxNev0eltIKETp5TEvhGAcZt0BHI+KRNAZEoPD2EH6rdvA9 Q0n/Cp0WxlSsnp7uDrroPZnSS+4qs7/zpBlAB/syffJ4CHwoJtmZcYZfFk7sTPjWQmC9Dgfu/UJQY O0eARu/+kuOxawFdESe/afTuTH4E9e58IJXPhQnDYC+9hfLNirkj0Z7DaSv53aKKayQXeDP1Mgh+g tYYUhXa1mpb88i8Ut9Gup+/ZBK5vhmJ1MxdaiMX0mKgHF32UXfzlOmOSzcFCswqtyI76Xz9qpRfFq UbByY2yw==; X-DKIM-Note: Keys used to sign are likely public at X-DKIM-Note: https://as397444.net/dkim/mattcorallo.com and X-DKIM-Note: https://as397444.net/dkim/clients.mail.as397444.net X-DKIM-Note: For more info, see https://as397444.net/dkim/ Message-ID: <4567346d-2a49-4989-9080-21a632113df3@mattcorallo.com> Date: Sat, 12 Apr 2025 13:43:49 -0400 MIME-Version: 1.0 Content-Language: en-US To: pve-user@lists.proxmox.com From: Matt Corallo <lpxdfsfs@mattcorallo.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.256 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FROM_LOCAL_NOVOWEL 0.5 From: localpart has series of non-vowel letters HK_RANDOM_ENVFROM 0.001 Envelope sender username looks random HK_RANDOM_FROM 1 From username looks random RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [mattcorallo.com, proxmox.com, as397444.net] Subject: [PVE-User] Memory Hotplug Woes X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox VE user list <pve-user@lists.proxmox.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: pve-user-bounces@lists.proxmox.com Sender: "pve-user" <pve-user-bounces@lists.proxmox.com> The Memory Hot Unplug section on the wiki at https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)#Memory_Hot_Unplug is somewhat outdated but subtly dangerous. It still refers to the now-removed CONFIG_MOVABLE_NODE, but its now available at least on Debian stable as of two years ago. But, more importantly, just blindly setting movable_node leads to system instability if the VM has a nontrivial amount of memory. eg I had a stock debian VM running postgresql with 16GB RAM with "memhp_default_state=online movable_node" added to the kernel command line and the result was that the OOM killer got invoked regularly when postgres pulled lots of disk into filesystem caches all at once for large queries, despite there being a GB or two of available memory. After chatting with the #mm folks on OFTC they pointed out that, yes, this is generally expected behavior because the result will be zero memory in the Normal zone, forcing all kernel allocations aside from pagecache into the bottom 1GiB of RAM which can easily run out and lead to OOM kills. In fact, movable_node's documentation even says "This means that the memory of such nodes will be usable only for movable allocations which rules out almost all kernel allocations. Use with caution!" The hotplug guide at https://www.kernel.org/doc/html/latest/admin-guide/mm/memory-hotplug.html suggests a better option as memory_hotplug.online_policy=auto-movable which keeps a sought ratio between movable and normal zones, fixing the issue, but sadly proxmox doesn't handle it with the automagic hotplug. Sadly, linux (at least 6.12.21) doesn't pick the last dimms to make movable when using auto-movable, but rather (sometimes?) picks ones in the middle of the range (eg the VM I'm looking at is making memory zones 32-67/183 movable, with the rest Normal/DMA32). If I go in the qemu monitor and device_del DIMMs in the lower range they get removed fine, however. It seems like the wiki should be updated to mention the drawbacks of `movable_node` and ideally the auto-hotunplug logic should try more than just the highest dimm and `online_policy` mentioned in the wiki. Thanks, Matt _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user