From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D127B9A20 for ; Sun, 27 Aug 2023 01:15:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B11CD11D2A for ; Sun, 27 Aug 2023 01:15:44 +0200 (CEST) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on20620.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e8b::620]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Sun, 27 Aug 2023 01:15:42 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MVkV2Z1CDSF2KYayzfIsjtc5sXDKCOfVRY52LOuYSjAUd81HQuCJ7Knlvz9WaSEuyROw62UzzfX9/RmCzFligMlMq/JCXXTHLdT0quTlF8ZAdRc+aeThO4zrTpBCc1VuzBPzWCfeRU435wITjBNje7D8h8U5hmT2dEqEWUhnmQS40HjlLiiy1sZGSau4ZcP7W9pN/ZpAxxxe+71ZraiCLz+DLaCjajqHboXyVa7fWRWxtBPIP+vaga5vJnggPiHaY6FxYb1TinAuFJFFWxi7uJEhDhzT9VGeZ0xSGbqcKzvJvHfUYWLYLD+H2YsJVh2ggNnoJ94peqoJP/JA60Wk9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KreUR3xlgHPgvZctK60jazOUzYTF8YlqkoyADSblRsk=; b=kqRsr7lwaNMQQVoH1C3FENaOmSu5pQUgRO+WhiUg/uqsJoJrsTi4YEfFUHMv1+YHDU+Y5hLdWHwSy/Wb6xpJdLsXd02C2oIpNcVRadh06JOWCW2MU8i0d1C51mNe++2ngXhkLoLxcnBbWaMm4YAxEr58OliEngsJGXf11CiLhHrNXQ7QWNqP4siYz02rRmomm1eLhg0+Z724Ujhu5AX0gxtYECfp7o0pF3f233y3hfWmuiiWPSmjLq5S4Bm/I8Ni+BH7jCKtKGFaEPLvpDIJOI5FK/jXPtCfPvh0jc5UP0FI/Wh5pkPy53Sj9cQSoYa4BzachfwyaugdG2wICtgEew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=caltech.edu; dmarc=pass action=none header.from=caltech.edu; dkim=pass header.d=caltech.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=caltech.edu; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KreUR3xlgHPgvZctK60jazOUzYTF8YlqkoyADSblRsk=; b=ozaB3WYIlOEk4qHaqCDtv8SrTmBooCAHlnbgq3s1j7kQ6qsBr3uONJsndWNNv3K4h7GVTmVEXJwls+JZ8yS6IOjSW1hod9YNCbG6HHr084ZfsQPK4Dbr+KBW1/NAE4ydV8D5A0kdFZ7JeT28E5VKA4aRkjfNGBJ0kzyzcGSDQuplfME+MpVKHUrLXJtvKJN+7ZmRH/ePPdnTzBLtz0D7EezppPmdDQK1TVMNgh6OdfcdyEz1Hy3HVHIGAQ6XAYnMFw7yaAAIOAiCLzw6H9k/3xIMG4QTBjJ0meyYjTBhziGlaIzV0U6+SPcNXxMjF6VzRMQhHxjKA9pFMWoELHjraA== Received: from SJ0PR03MB6303.namprd03.prod.outlook.com (2603:10b6:a03:3bc::22) by PH0PR03MB6446.namprd03.prod.outlook.com (2603:10b6:510:bb::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.34; Sat, 26 Aug 2023 22:41:46 +0000 Received: from SJ0PR03MB6303.namprd03.prod.outlook.com ([fe80::8310:3804:7a09:33ca]) by SJ0PR03MB6303.namprd03.prod.outlook.com ([fe80::8310:3804:7a09:33ca%4]) with mapi id 15.20.6699.034; Sat, 26 Aug 2023 22:41:45 +0000 From: "Anderson, Stuart B." To: "pve-user@lists.proxmox.com" Thread-Topic: Hotplug Memory and default Linux kernel parameters Thread-Index: AQHZ2G5+j5zi368jxUyCC0eI2ymAJA== Date: Sat, 26 Aug 2023 22:41:45 +0000 Message-ID: <222509B0-CEB7-436E-82AA-D16AA78B633C@caltech.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3731.700.6) authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=caltech.edu; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SJ0PR03MB6303:EE_|PH0PR03MB6446:EE_ x-ms-office365-filtering-correlation-id: f047b3d4-ce46-4b68-3fb3-08dba685a091 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ib2HJHwQNH/QEUPr5qOfKSZom5cUAdMQED9DX5omaQanlJ8kvws3Zb7IlPrIc7a+4hbnUKQNC0kNB8kwAtYF0IqZV9jTdpNbZa3VXz7XGj5MEK9QCY8tYnVSN+Z5YB/KGo5PStxdjtuXSyR2mTGgICSktoD9oa6NXL1vGUHk2SLYiDPwiiAJC51iM2FFScVYVjxuMbDGxrP6ZaYsq40qKQuPEkaaSB3BPTptxpryUpaqFyEwzGJp21FrgMdf2Zgons++1VoMOCSRaV7pVQ/SS4AcBl6yWlZ/1avYSjpcAt71/jdpuwo6CZxZK7p6Dw/CAWuWBjrKl26ZtI6i6co8n2GEOmenyeHNjDedt5rX93isxyiLTiv7jAKFpRtsk96tt7UBLUKMF0nrtT5PI7J7qHKDDpxuXNIZ850lmDTeibk+RDoGj3woJ6UwiapFdWFLjm1VBoufgsrAS3SOwstJHPL1Emx5Oz4V007uiqi4uGRkHevT5a4AOi2ooRLEuPaX3pMx6odLAwJ3ZfpnvahhmxfDzNgbo6W+4s3s/O9UghAIhwFSCkFUyqCh0YYxfPdt6G4pMhWNtgjFQ16TAxLWaFeLJUWRke83qyDt+HyPqI9kJBlOrBEMaX5mlSypORX9KWIiBJ3K7rVdWmGqKOCavQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR03MB6303.namprd03.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(346002)(136003)(39850400004)(396003)(376002)(186009)(1800799009)(451199024)(71200400001)(6506007)(6486002)(2616005)(75432002)(6512007)(86362001)(5660300002)(33656002)(2906002)(38100700002)(8676002)(8936002)(76116006)(66446008)(66476007)(64756008)(66556008)(36756003)(41300700001)(66946007)(316002)(91956017)(786003)(6916009)(38070700005)(122000001)(478600001)(83380400001)(26005)(45980500001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?B2UA4YAJhick/JfZt3QsZaaN723HdEv/kbr0+OuQHJCUuXVTuCl35yGBm57n?= =?us-ascii?Q?uq1ZOFCl0T3W+if4YZQQEoaZviLAOvaDW37wLWQGaSik9OUuOMjBMANWaBBC?= =?us-ascii?Q?isOYlObOMdqwHJ7SVT70+IPZZGfunJeKqbVcz9+dFsIPcJkxF5tj1CHSvcTw?= =?us-ascii?Q?ZHygsaSOZabgOSIF/1zYHpnf2thDksDZaMvyluI9t6NvKFW3eZ2Pf3K5X+lH?= =?us-ascii?Q?j8pMXErmGNvDXxD2uxurcpdaOCsbpAdfuPfnygN9F8ItL8as4yzMruCXW77t?= =?us-ascii?Q?3X59G1Z6xLtjDOmd+kftY2f6mvBqhjf6nZOvam5vl/i0+Q8G3OnvPNWWVfNd?= =?us-ascii?Q?xQxXc5/AzUOWUfqDSCXBspQVRakTaooLtka/VDtsuiTGoDgaXHeR+73b5+ja?= =?us-ascii?Q?KnBjc3+GjglXTF8QgMZHiSH/YsWLDqaUA7XSEzjB/LfuLdrb3qFLZQH4ieVd?= =?us-ascii?Q?G4Xq3V/m/lL7xKzV4J+G657urLDYfG7LsJEBxzBsGDI16dd0Rl/jLO5wc+xR?= =?us-ascii?Q?baDxlu6bts+8hKXGOnTjUs36OsgTwPX4/KOVsxDyBfgy0kwamuGKfpSwZx8o?= =?us-ascii?Q?a1YFe34Y2QSJKqH1g+GPUjEeC+/hhXcPJXgIkOeNBXMhYpjAK+1OS/V4Sgtj?= =?us-ascii?Q?D0TQtqM5y6fw+4z6khsqUYlE0TQ720fRD05NkH+XdhbE4CgvqcgSmVH81+kc?= =?us-ascii?Q?K4Xn4C29kFMYlqUmEHkLGxNSltAI3pmym6FW+K8wFtCfs9HQnngQcoW+vSC5?= =?us-ascii?Q?L1d8AT36zU9GKv7bVVZ+BttcjkVPQZZdmO0y1XKl4GjOM7/jAoydWFyMC1ZK?= =?us-ascii?Q?yP34z5jEIq0lh3nKMY+pUiMhOxwjhTbuWikWQSAJ30X5TtOWYacSG+VWCeqb?= =?us-ascii?Q?cU+Tm2wGnPcUHS0PDhVTk1JBzvclBn7pvh3gjw/5LraUZCt4qnemNdtGPRFU?= =?us-ascii?Q?pOR6AfB7mTi05ZlI8uJTivIWAXkw6nGoC5qn2tg+0bZLfX9Doau5s8FYspU2?= =?us-ascii?Q?lT2Gb+lplmnSwROi65pSBSXBj+TdbIb8mfMPqHLFYn+PqcOGviXosjo7cvuo?= =?us-ascii?Q?H8fNBCfVU4xRy1x5C1GyRdWWv80JeyKGMyOnlMVBN/zsr9WGFrpSeQBvty78?= =?us-ascii?Q?4JcuVAtSeOf385QXulDadPExwEQvzp0QSYg69beavvQwfCyYYJnFGjZlqMpg?= =?us-ascii?Q?Wd3LAyjLc085QMjxV1BAQrsLSA39IzSfP5gfp6Argyg1deNnnojf99Z8qqF7?= =?us-ascii?Q?yRPgr8s4MQ4jKTKkkodzjakt30rJqXI8/O2H5Hi2Lt0VztPerOW8T8G3GBjm?= =?us-ascii?Q?JVv4qvZoi7iRex4Z+It2a3gMm9GSJ6juv0FRvB5MDcQRLkTc4dEu/2Odm1PE?= =?us-ascii?Q?aLS8MayLa5AR66Cs4OOo+YtL51FZwkjVFE0iGsN13OCaEFpfrn/Xgo703M/l?= =?us-ascii?Q?Oi679qFYMT4+RsUNx8C0EhEJq9hVDJW0/YIswSmTDxYD3X5wUcpPE6hANw4q?= =?us-ascii?Q?zuerk76+UvupaC9SWcOEO1nLfS6BZKcyAKLyde+zO+2oylX6ORdERLNLnD7A?= =?us-ascii?Q?9SIbyPwGuy4Qi4l83rOtaa0JQDyXEXMCIO4W8ZJp?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: caltech.edu X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR03MB6303.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f047b3d4-ce46-4b68-3fb3-08dba685a091 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Aug 2023 22:41:45.7653 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: fd5be9d9-7b72-4df9-830e-b1f9cc5b44bd X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 9n3Z8R5a/MP3wwz95J0c3gqGagNXFd+Ss4oj7aiRuIXyjZMiA4poyicB1LkAsg9K X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR03MB6446 X-SPAM-LEVEL: Spam detection results: 0 ARC_SIGNED 0.001 Message has a ARC signature ARC_VALID 0.001 Message has a valid ARC signature BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy JMQ_SPF_NEUTRAL 0.5 SPF set to ?all SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [caltech.edu, qemu.org] Subject: [PVE-User] Hotplug Memory and default Linux kernel parameters X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Aug 2023 23:15:44 -0000 Enabling PVE Hotplug Memory for a Linux Guest (tested with PVE 7/8 and EL8/= 9) results in default kernel parameters that are orders of magnitude smalle= r than without Hotplug. It appears that the Kernel is mistakenly setting de= faults as if the guest has only 1GB of memory. Does anyone know how to get = the same kernel defaults with Hotplug Memory enabled as for disabled? Is th= is a bug in PVE, QEMU, or the way Linux queries QEMU? For example, a PVE7/EL8 VM with 32GB of Hotplug Memory has a very small val= ue of Max processes: [root@ldas-pcdev4 ~]# grep processes /proc/$(pgrep systemd-logind)/limits Max processes 2654 2654 process= es=20 compared to disabling Hotplug Memory: [root@condor-f1 ~]# grep processes /proc/$(pgrep systemd-logind)/limits Max processes 127390 127390 process= es=20 Presumably this is due to the following memory layout as seen by the kernel= , # # With Hotplug Memory: 1 bank with a 1GB DIMM # [root@ldas-pcdev4 ~]# lsmem RANGE SIZE STATE REMOVABLE BLOCK 0x0000000000000000-0x000000003fffffff 1G online yes 0-7 0x0000010000000000-0x00000107bfffffff 31G online yes 8192-8439 Memory block size: 128M Total online memory: 32G Total offline memory: 0B [root@ldas-pcdev4 ~]# lshw -class memory *-firmware description: BIOS vendor: SeaBIOS physical id: 0 version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org date: 04/01/2014 size: 96KiB *-memory description: System Memory physical id: 1000 size: 32GiB capabilities: ecc configuration: errordetection=3Dmulti-bit-ecc *-bank description: DIMM RAM vendor: QEMU physical id: 0 slot: DIMM 0 size: 1GiB # # Without Hotplug Memory: 2 banks of of 16GB DIMM # [root@condor-f1 ~]# lsmem RANGE SIZE STATE REMOVABLE BLOCK 0x0000000000000000-0x00000000bfffffff 3G online yes 0-23 0x0000000100000000-0x000000083fffffff 29G online yes 32-263 Memory block size: 128M Total online memory: 32G Total offline memory: 0B [root@condor-f1 ~]# lshw -class memory *-firmware description: BIOS vendor: SeaBIOS physical id: 0 version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org date: 04/01/2014 size: 96KiB *-memory description: System Memory physical id: 1000 size: 32GiB capabilities: ecc configuration: errordetection=3Dmulti-bit-ecc *-bank:0 description: DIMM RAM vendor: QEMU physical id: 0 slot: DIMM 0 size: 16GiB *-bank:1 description: DIMM RAM vendor: QEMU physical id: 1 slot: DIMM 1 size: 16GiB P.S. Unfornately, this isn't fixed with PVE8 (with a newer QEMU) or updatin= g to a newer EL9 kernel. Here is PVE8/EL9 VM with 233GB of Hotplug Memory s= howing the same problematic small value:=20 [root@pcdev15 ~]# cat /etc/redhat-release=20 Rocky Linux release 9.2 (Blue Onyx) [root@pcdev15 ~]# grep processes /proc/$(pgrep systemd-logind)/limits Max processes 2659 2659 process= es -- Stuart Anderson sba@caltech.edu