From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 2B44D1FF0E1 for ; Mon, 13 Jul 2026 17:34:58 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B4684213A2; Mon, 13 Jul 2026 17:34:56 +0200 (CEST) Date: Mon, 13 Jul 2026 17:34:17 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [RFC PATCH qemu-server] memory: deallocate hugepages on cleanup To: Maximiliano Sandoval , pve-devel@lists.proxmox.com References: <20260710125119.422202-1-m.sandoval@proxmox.com> In-Reply-To: <20260710125119.422202-1-m.sandoval@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1783955382.5ljpcnns4p.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783956844987 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.275 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RSA5E4E6NFPMVTFTB4GSB2BWMA4MIAWB X-Message-ID-Hash: RSA5E4E6NFPMVTFTB4GSB2BWMA4MIAWB X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On July 10, 2026 2:51 pm, Maximiliano Sandoval wrote: > Right before VM startup the hugepages_allocate call would increase the nu= mber of > persistent hugepages (at > /sys/kernel/mm/hugepages/hugepages-{size}kB/nr_hugepages) based on the gu= est and > host numa topologies, then the VM is started, and immediately after the > hugepages_pre_deallocate would set the nr_hugepages back to the original = value. >=20 >>>From the HugeTLB Pages documentation [1]: this part here should not be quoted >=20 > Caveat: Shrinking the persistent huge page pool via ``nr_hugepages`` such= that > it becomes less than the number of huge pages in use will convert the bal= ance > of the in-use huge pages to surplus huge pages. This will occur even if > the number of surplus pages would exceed the overcommit value. As long a= s > this condition holds--that is, until ``nr_hugepages+nr_overcommit_hugepag= es`` is > increased sufficiently, or the surplus huge pages go out of use and are f= reed-- > no more surplus huge pages will be allowed to be allocated. but this part here should be. but also note that you need to indent such quotes otherwise their "quotedness" is lost when applying the patch. > In a setup where nr_overcommit_hugepages is 0, if the KVM process frees i= ts > hugepages, the number of surplus hugepages can potentially be above > `nr_overcommit_hugepages` and no new surplus will be allowed to be alloca= ted by > the process. >=20 > This can be reproduced as follows: >=20 > On a setup with 1 numa node, create a Linux VM, then configure it as foll= ows: >=20 > $ qm set --numa 1 --hugepages 2 --balloon 1 >=20 > If one then starts the VM, the nr_hugepages would decrease monotonically = and the > VM would get stuck during its boot process. is this only an issue with 2M hugepages, or was this just your example reproducer? could you provide the full VM config? >=20 > We address this by deferring the de-allocation until qm cleanup. >=20 > [1] https://www.kernel.org/doc/html/latest/admin-guide/mm/hugetlbpage.htm= l >=20 > Signed-off-by: Maximiliano Sandoval > --- > src/PVE/CLI/qm.pm | 8 ++++++++ > src/PVE/QemuServer.pm | 3 --- > 2 files changed, 8 insertions(+), 3 deletions(-) >=20 > diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm > index b903c1f1..8a88fb48 100755 > --- a/src/PVE/CLI/qm.pm > +++ b/src/PVE/CLI/qm.pm > @@ -1137,6 +1137,14 @@ __PACKAGE__->register_method({ > die "skipping cleanup - 'rollback' lock is present\n" > if $conf->{lock} && $conf->{lock} eq 'rollback'; > =20 > + my $hotplug_features =3D > + PVE::QemuServer::parse_hotplug_features(defined($con= f->{hotplug}) ? $conf->{hotplug} : '1'); > + my $hugepages_topology =3D > + PVE::QemuServer::Memory::hugepages_topology($conf, $= hotplug_features->{memory}); > + > + PVE::QemuServer::Memory::hugepages_pre_deallocate($hugep= ages_topology) > + if !$conf->{keephugepages}; > + the same codepath exists in the memory hotplug code.. but besides that, if we move it here, doesn't this still have the same issue if two VMs are started and the first one is stopped? > if (!$clean) { > # we have to cleanup the tap devices after a crash > =20 > diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm > index 59a37098..23d5aa9f 100644 > --- a/src/PVE/QemuServer.pm > +++ b/src/PVE/QemuServer.pm > @@ -5840,9 +5840,6 @@ sub vm_start_nolock { > if !$conf->{keephugepages}; > die $err; > } > - > - PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages= _topology) > - if !$conf->{keephugepages}; > }; > eval { PVE::QemuServer::Memory::hugepages_update_locked($code); = }; > =20 > --=20 > 2.47.3 >=20 >=20 >=20 >=20 >=20 >=20