From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 5F3A91FF0E4 for ; Tue, 14 Jul 2026 14:32:01 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7D206213E9; Tue, 14 Jul 2026 14:32:00 +0200 (CEST) From: Maximiliano Sandoval To: Fabian =?utf-8?Q?Gr=C3=BCnbichler?= Subject: Re: [RFC PATCH qemu-server] memory: deallocate hugepages on cleanup In-Reply-To: <1783955382.5ljpcnns4p.astroid@yuna.none> ("Fabian =?utf-8?Q?Gr=C3=BCnbichler=22's?= message of "Mon, 13 Jul 2026 17:34:17 +0200") References: <20260710125119.422202-1-m.sandoval@proxmox.com> <1783955382.5ljpcnns4p.astroid@yuna.none> User-Agent: mu4e 1.12.9; emacs 30.1 Date: Tue, 14 Jul 2026 14:31:54 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1784032298037 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.331 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FNUGJOPMCQ4KSNCRWQALNUDD5LXT7Q66 X-Message-ID-Hash: FNUGJOPMCQ4KSNCRWQALNUDD5LXT7Q66 X-MailFrom: m.sandoval@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Fabian Gr=C3=BCnbichler writes: > On July 10, 2026 2:51 pm, Maximiliano Sandoval wrote: >> Right before VM startup the hugepages_allocate call would increase the n= umber of >> persistent hugepages (at >> /sys/kernel/mm/hugepages/hugepages-{size}kB/nr_hugepages) based on the g= uest and >> host numa topologies, then the VM is started, and immediately after the >> hugepages_pre_deallocate would set the nr_hugepages back to the original= value. >>=20 >>>>From the HugeTLB Pages documentation [1]: > > this part here should not be quoted I do not see this part quoted on my end, I am holding it wrong? >>=20 >> Caveat: Shrinking the persistent huge page pool via ``nr_hugepages`` suc= h that >> it becomes less than the number of huge pages in use will convert the ba= lance >> of the in-use huge pages to surplus huge pages. This will occur even if >> the number of surplus pages would exceed the overcommit value. As long = as >> this condition holds--that is, until ``nr_hugepages+nr_overcommit_hugepa= ges`` is >> increased sufficiently, or the surplus huge pages go out of use and are = freed-- >> no more surplus huge pages will be allowed to be allocated. > > but this part here should be. but also note that you need to indent such > quotes otherwise their "quotedness" is lost when applying the patch. =F0=9F=91=8C >> In a setup where nr_overcommit_hugepages is 0, if the KVM process frees = its >> hugepages, the number of surplus hugepages can potentially be above >> `nr_overcommit_hugepages` and no new surplus will be allowed to be alloc= ated by >> the process. >>=20 >> This can be reproduced as follows: >>=20 >> On a setup with 1 numa node, create a Linux VM, then configure it as fol= lows: >>=20 >> $ qm set --numa 1 --hugepages 2 --balloon 1 >>=20 >> If one then starts the VM, the nr_hugepages would decrease monotonically= and the >> VM would get stuck during its boot process. > > is this only an issue with 2M hugepages, or was this just your example > reproducer? could you provide the full VM config? I have reproduced this on a VM running directly on the host but also when running nested. It is reproducible with 2M hugepages. I did not put too much energy into reproducing it with 1G, but booting a Linux VM works with 1G pages and when assigning the VM either 4G, 10G or 40G of memory would result in a constant nr_hugepages. The top level VM would be a plain Proxmox VE 9 VM on a ZFS storage: ``` boot: order=3Dscsi0;net0 cores: 8 cpu: host memory: 12288 meta: creation-qemu=3D11.0.0,ctime=3D1781180270 net0: virtio=3DDA:AA:AA:1D:97:A6,bridge=3Dguests,firewall=3D1 ostype: l26 scsi0: local-zfs:base-156-disk-0/vm-123-disk-0,discard=3Don,iothread=3D1,si= ze=3D32G scsihw: virtio-scsi-single smbios1: uuid=3Dd5142704-8f67-4380-a74c-37af54c269d6 sockets: 1 vmgenid: 10bb4503-5fde-4280-8cd8-03e689e1265a ``` while the VM config where it was reproduced is ``` boot: order=3Dscsi0;ide2;net0 cores: 4 cpu: x86-64-v2-AES hugepages: 2 ide2: local:iso/proxmox-ve_9.2-1.iso,media=3Dcdrom,size=3D1666190K memory: 4096 meta: creation-qemu=3D11.0.0,ctime=3D1783584825 net0: virtio=3DBC:24:11:99:73:3A,bridge=3Dvmbr0,firewall=3D1 numa: 1 ostype: l26 scsi0: local-lvm:vm-100-disk-0,iothread=3D1,size=3D32G scsihw: virtio-scsi-single smbios1: uuid=3Db935e0ec-24d4-4c5c-8de7-d4d56496067f sockets: 1 vmgenid: 86b1df9c-59ff-472a-a3e1-648199fb02ea ``` >> We address this by deferring the de-allocation until qm cleanup. >>=20 >> [1] https://www.kernel.org/doc/html/latest/admin-guide/mm/hugetlbpage.ht= ml >>=20 >> Signed-off-by: Maximiliano Sandoval >> --- >> src/PVE/CLI/qm.pm | 8 ++++++++ >> src/PVE/QemuServer.pm | 3 --- >> 2 files changed, 8 insertions(+), 3 deletions(-) >>=20 >> diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm >> index b903c1f1..8a88fb48 100755 >> --- a/src/PVE/CLI/qm.pm >> +++ b/src/PVE/CLI/qm.pm >> @@ -1137,6 +1137,14 @@ __PACKAGE__->register_method({ >> die "skipping cleanup - 'rollback' lock is present\n" >> if $conf->{lock} && $conf->{lock} eq 'rollback'; >>=20=20 >> + my $hotplug_features =3D >> + PVE::QemuServer::parse_hotplug_features(defined($co= nf->{hotplug}) ? $conf->{hotplug} : '1'); >> + my $hugepages_topology =3D >> + PVE::QemuServer::Memory::hugepages_topology($conf, = $hotplug_features->{memory}); >> + >> + PVE::QemuServer::Memory::hugepages_pre_deallocate($huge= pages_topology) >> + if !$conf->{keephugepages}; >> + > > the same codepath exists in the memory hotplug code.. Forgot to mention this, yes this would be addressed in the future. > but besides that, if we move it here, doesn't this still have the same > issue if two VMs are started and the first one is stopped? Indeed, I just noticed that the deallocate would need to be taught to deallocate exactly what was allocated for this to work. >> if (!$clean) { >> # we have to cleanup the tap devices after a crash >>=20=20 >> diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm >> index 59a37098..23d5aa9f 100644 >> --- a/src/PVE/QemuServer.pm >> +++ b/src/PVE/QemuServer.pm >> @@ -5840,9 +5840,6 @@ sub vm_start_nolock { >> if !$conf->{keephugepages}; >> die $err; >> } >> - >> - PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepage= s_topology) >> - if !$conf->{keephugepages}; >> }; >> eval { PVE::QemuServer::Memory::hugepages_update_locked($code);= }; >>=20=20 >> --=20 >> 2.47.3 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 --=20 Maximiliano