Re: [RFC PATCH qemu-server] memory: deallocate hugepages on cleanup

[Maximiliano Sandoval <m.sandoval@proxmox.com>]

Fabian Grünbichler <f.gruenbichler@proxmox.com> writes:

> On July 10, 2026 2:51 pm, Maximiliano Sandoval wrote:
>> Right before VM startup the hugepages_allocate call would increase the number of
>> persistent hugepages (at
>> /sys/kernel/mm/hugepages/hugepages-{size}kB/nr_hugepages) based on the guest and
>> host numa topologies, then the VM is started, and immediately after the
>> hugepages_pre_deallocate would set the nr_hugepages back to the original value.
>> 
>>>From the HugeTLB Pages documentation [1]:
>
> this part here should not be quoted

I do not see this part quoted on my end, I am holding it wrong?

>> 
>> Caveat: Shrinking the persistent huge page pool via ``nr_hugepages`` such that
>> it becomes less than the number of huge pages in use will convert the balance
>> of the in-use huge pages to surplus huge pages.  This will occur even if
>> the number of surplus pages would exceed the overcommit value.  As long as
>> this condition holds--that is, until ``nr_hugepages+nr_overcommit_hugepages`` is
>> increased sufficiently, or the surplus huge pages go out of use and are freed--
>> no more surplus huge pages will be allowed to be allocated.
>
> but this part here should be. but also note that you need to indent such
> quotes otherwise their "quotedness" is lost when applying the patch.

👌

>> In a setup where nr_overcommit_hugepages is 0, if the KVM process frees its
>> hugepages, the number of surplus hugepages can potentially be above
>> `nr_overcommit_hugepages` and no new surplus will be allowed to be allocated by
>> the process.
>> 
>> This can be reproduced as follows:
>> 
>> On a setup with 1 numa node, create a Linux VM, then configure it as follows:
>> 
>> $ qm set <vmid> --numa 1 --hugepages 2 --balloon 1
>> 
>> If one then starts the VM, the nr_hugepages would decrease monotonically and the
>> VM would get stuck during its boot process.
>
> is this only an issue with 2M hugepages, or was this just your example
> reproducer? could you provide the full VM config?

I have reproduced this on a VM running directly on the host but also
when running nested.

It is reproducible with 2M hugepages. I did not put too much energy into
reproducing it with 1G, but booting a Linux VM works with 1G pages and
when assigning the VM either 4G, 10G or 40G of memory would result in a
constant nr_hugepages.

The top level VM would be a plain Proxmox VE 9 VM on a ZFS storage:

```
boot: order=scsi0;net0
cores: 8
cpu: host
memory: 12288
meta: creation-qemu=11.0.0,ctime=1781180270
net0: virtio=DA:AA:AA:1D:97:A6,bridge=guests,firewall=1
ostype: l26
scsi0: local-zfs:base-156-disk-0/vm-123-disk-0,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=d5142704-8f67-4380-a74c-37af54c269d6
sockets: 1
vmgenid: 10bb4503-5fde-4280-8cd8-03e689e1265a
```

while the VM config where it was reproduced is

```
boot: order=scsi0;ide2;net0
cores: 4
cpu: x86-64-v2-AES
hugepages: 2
ide2: local:iso/proxmox-ve_9.2-1.iso,media=cdrom,size=1666190K
memory: 4096
meta: creation-qemu=11.0.0,ctime=1783584825
net0: virtio=BC:24:11:99:73:3A,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=b935e0ec-24d4-4c5c-8de7-d4d56496067f
sockets: 1
vmgenid: 86b1df9c-59ff-472a-a3e1-648199fb02ea
```



>> We address this by deferring the de-allocation until qm cleanup.
>> 
>> [1] https://www.kernel.org/doc/html/latest/admin-guide/mm/hugetlbpage.html
>> 
>> Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
>> ---
>>  src/PVE/CLI/qm.pm     | 8 ++++++++
>>  src/PVE/QemuServer.pm | 3 ---
>>  2 files changed, 8 insertions(+), 3 deletions(-)
>> 
>> diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm
>> index b903c1f1..8a88fb48 100755
>> --- a/src/PVE/CLI/qm.pm
>> +++ b/src/PVE/CLI/qm.pm
>> @@ -1137,6 +1137,14 @@ __PACKAGE__->register_method({
>>                  die "skipping cleanup - 'rollback' lock is present\n"
>>                      if $conf->{lock} && $conf->{lock} eq 'rollback';
>>  
>> +                my $hotplug_features =
>> +                    PVE::QemuServer::parse_hotplug_features(defined($conf->{hotplug}) ? $conf->{hotplug} : '1');
>> +                my $hugepages_topology =
>> +                    PVE::QemuServer::Memory::hugepages_topology($conf, $hotplug_features->{memory});
>> +
>> +                PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology)
>> +                    if !$conf->{keephugepages};
>> +
>
> the same codepath exists in the memory hotplug code..

Forgot to mention this, yes this would be addressed in the future.

> but besides that, if we move it here, doesn't this still have the same
> issue if two VMs are started and the first one is stopped?

Indeed, I just noticed that the deallocate would need to be taught to
deallocate exactly what was allocated for this to work.

>>                  if (!$clean) {
>>                      # we have to cleanup the tap devices after a crash
>>  
>> diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
>> index 59a37098..23d5aa9f 100644
>> --- a/src/PVE/QemuServer.pm
>> +++ b/src/PVE/QemuServer.pm
>> @@ -5840,9 +5840,6 @@ sub vm_start_nolock {
>>                      if !$conf->{keephugepages};
>>                  die $err;
>>              }
>> -
>> -            PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology)
>> -                if !$conf->{keephugepages};
>>          };
>>          eval { PVE::QemuServer::Memory::hugepages_update_locked($code); };
>>  
>> -- 
>> 2.47.3
>> 
>> 
>> 
>> 
>> 
>> 

-- 
Maximiliano