* [PVE-User] Hotplug Memory and default Linux kernel parameters
@ 2023-08-26 22:41 Anderson, Stuart B.
2023-08-29 8:14 ` DERUMIER, Alexandre
0 siblings, 1 reply; 2+ messages in thread
From: Anderson, Stuart B. @ 2023-08-26 22:41 UTC (permalink / raw)
To: pve-user
Enabling PVE Hotplug Memory for a Linux Guest (tested with PVE 7/8 and EL8/9) results in default kernel parameters that are orders of magnitude smaller than without Hotplug. It appears that the Kernel is mistakenly setting defaults as if the guest has only 1GB of memory. Does anyone know how to get the same kernel defaults with Hotplug Memory enabled as for disabled? Is this a bug in PVE, QEMU, or the way Linux queries QEMU?
For example, a PVE7/EL8 VM with 32GB of Hotplug Memory has a very small value of Max processes:
[root@ldas-pcdev4 ~]# grep processes /proc/$(pgrep systemd-logind)/limits
Max processes 2654 2654 processes
compared to disabling Hotplug Memory:
[root@condor-f1 ~]# grep processes /proc/$(pgrep systemd-logind)/limits
Max processes 127390 127390 processes
Presumably this is due to the following memory layout as seen by the kernel,
#
# With Hotplug Memory: 1 bank with a 1GB DIMM
#
[root@ldas-pcdev4 ~]# lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000003fffffff 1G online yes 0-7
0x0000010000000000-0x00000107bfffffff 31G online yes 8192-8439
Memory block size: 128M
Total online memory: 32G
Total offline memory: 0B
[root@ldas-pcdev4 ~]# lshw -class memory
*-firmware description: BIOS
vendor: SeaBIOS
physical id: 0
version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org
date: 04/01/2014
size: 96KiB
*-memory
description: System Memory
physical id: 1000
size: 32GiB
capabilities: ecc
configuration: errordetection=multi-bit-ecc
*-bank
description: DIMM RAM
vendor: QEMU
physical id: 0
slot: DIMM 0
size: 1GiB
#
# Without Hotplug Memory: 2 banks of of 16GB DIMM
#
[root@condor-f1 ~]# lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x00000000bfffffff 3G online yes 0-23
0x0000000100000000-0x000000083fffffff 29G online yes 32-263
Memory block size: 128M
Total online memory: 32G
Total offline memory: 0B
[root@condor-f1 ~]# lshw -class memory
*-firmware description: BIOS
vendor: SeaBIOS
physical id: 0
version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org
date: 04/01/2014
size: 96KiB
*-memory
description: System Memory
physical id: 1000
size: 32GiB
capabilities: ecc
configuration: errordetection=multi-bit-ecc
*-bank:0
description: DIMM RAM
vendor: QEMU
physical id: 0
slot: DIMM 0
size: 16GiB
*-bank:1
description: DIMM RAM
vendor: QEMU
physical id: 1
slot: DIMM 1
size: 16GiB
P.S. Unfornately, this isn't fixed with PVE8 (with a newer QEMU) or updating to a newer EL9 kernel. Here is PVE8/EL9 VM with 233GB of Hotplug Memory showing the same problematic small value:
[root@pcdev15 ~]# cat /etc/redhat-release
Rocky Linux release 9.2 (Blue Onyx)
[root@pcdev15 ~]# grep processes /proc/$(pgrep systemd-logind)/limits
Max processes 2659 2659 processes
--
Stuart Anderson
sba@caltech.edu
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PVE-User] Hotplug Memory and default Linux kernel parameters
2023-08-26 22:41 [PVE-User] Hotplug Memory and default Linux kernel parameters Anderson, Stuart B.
@ 2023-08-29 8:14 ` DERUMIER, Alexandre
0 siblings, 0 replies; 2+ messages in thread
From: DERUMIER, Alexandre @ 2023-08-29 8:14 UTC (permalink / raw)
To: pve-user
Hi,
see
https://pve.proxmox.com/wiki/Hotplug_(qemu_disk,nic,cpu,memory)#Memory_Hotplug
2 possiblity:
add "memhp_default_state=online" to grub
or
add udev script
/lib/udev/rules.d/80-hotplug-cpu.rules
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0",
ATTR{online}="1"
When hotplug is enabled, only 1GB of "static" memory is enabled (to
boot the kernel,...).
Then the other memory modules, are hotpluggable, and by default are
offline.
Le samedi 26 août 2023 à 22:41 +0000, Anderson, Stuart B. a écrit :
> Enabling PVE Hotplug Memory for a Linux Guest (tested with PVE 7/8
> and EL8/9) results in default kernel parameters that are orders of
> magnitude smaller than without Hotplug. It appears that the Kernel is
> mistakenly setting defaults as if the guest has only 1GB of memory.
> Does anyone know how to get the same kernel defaults with Hotplug
> Memory enabled as for disabled? Is this a bug in PVE, QEMU, or the
> way Linux queries QEMU?
>
>
> For example, a PVE7/EL8 VM with 32GB of Hotplug Memory has a very
> small value of Max processes:
>
> [root@ldas-pcdev4 ~]# grep processes /proc/$(pgrep systemd-
> logind)/limits
> Max processes 2654 2654
> processes
>
> compared to disabling Hotplug Memory:
>
> [root@condor-f1 ~]# grep processes /proc/$(pgrep systemd-
> logind)/limits
> Max processes 127390 127390
> processes
>
>
> Presumably this is due to the following memory layout as seen by the
> kernel,
>
> #
> # With Hotplug Memory: 1 bank with a 1GB DIMM
> #
> [root@ldas-pcdev4 ~]# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x0000000000000000-0x000000003fffffff 1G online yes 0-7
> 0x0000010000000000-0x00000107bfffffff 31G online yes 8192-8439
>
> Memory block size: 128M
> Total online memory: 32G
> Total offline memory: 0B
>
> [root@ldas-pcdev4 ~]# lshw -class memory
> *-firmware description: BIOS
> vendor: SeaBIOS
> physical id: 0
> version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org
> date: 04/01/2014
> size: 96KiB
> *-memory
> description: System Memory
> physical id: 1000
> size: 32GiB
> capabilities: ecc
> configuration: errordetection=multi-bit-ecc
> *-bank
> description: DIMM RAM
> vendor: QEMU
> physical id: 0
> slot: DIMM 0
> size: 1GiB
>
>
> #
> # Without Hotplug Memory: 2 banks of of 16GB DIMM
> #
> [root@condor-f1 ~]# lsmem
> RANGE SIZE STATE REMOVABLE BLOCK
> 0x0000000000000000-0x00000000bfffffff 3G online yes 0-23
> 0x0000000100000000-0x000000083fffffff 29G online yes 32-263
>
> Memory block size: 128M
> Total online memory: 32G
> Total offline memory: 0B
>
> [root@condor-f1 ~]# lshw -class memory
> *-firmware description: BIOS
> vendor: SeaBIOS
> physical id: 0
> version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org
> date: 04/01/2014
> size: 96KiB
> *-memory
> description: System Memory
> physical id: 1000
> size: 32GiB
> capabilities: ecc
> configuration: errordetection=multi-bit-ecc
> *-bank:0
> description: DIMM RAM
> vendor: QEMU
> physical id: 0
> slot: DIMM 0
> size: 16GiB
> *-bank:1
> description: DIMM RAM
> vendor: QEMU
> physical id: 1
> slot: DIMM 1
> size: 16GiB
>
>
>
> P.S. Unfornately, this isn't fixed with PVE8 (with a newer QEMU) or
> updating to a newer EL9 kernel. Here is PVE8/EL9 VM with 233GB of
> Hotplug Memory showing the same problematic small value:
>
> [root@pcdev15 ~]# cat /etc/redhat-release
> Rocky Linux release 9.2 (Blue Onyx)
>
> [root@pcdev15 ~]# grep processes /proc/$(pgrep systemd-logind)/limits
> Max processes 2659 2659
> processes
>
>
> --
> Stuart Anderson
> sba@caltech.edu
>
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2023-08-29 8:20 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-08-26 22:41 [PVE-User] Hotplug Memory and default Linux kernel parameters Anderson, Stuart B.
2023-08-29 8:14 ` DERUMIER, Alexandre
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox