* Re: [pve-devel] [PATCH qemu-server] fix #2570: add 'keephugepages' config
[not found] <20200212133228.8442-1-s.reiter@proxmox.com>
@ 2020-08-20 7:59 ` Stefan Reiter
2020-08-20 8:17 ` Thomas Lamprecht
1 sibling, 0 replies; 3+ messages in thread
From: Stefan Reiter @ 2020-08-20 7:59 UTC (permalink / raw)
To: pve-devel
Ping? This is an old one, but the bug report is still active.
Would need a rebase if the approach is deemed ok.
On 2/12/20 2:32 PM, Stefan Reiter wrote:
> We already keep hugepages if they are created with the kernel
> commandline (hugepagesz=x hugepages=y), but some setups (specifically
> hugepages across multiple NUMA nodes) cannot be configured that way.
> Since we always clear these hugepages at VM shutdown, rebooting a VM
> that uses them might not work, since the requested count might not be
> available anymore by the time we want to use them (also, we would then
> no longer allocate them correctly on the NUMA nodes).
>
> Add a 'keephugepages' parameter to skip cleanup and simply leave them
> untouched.
>
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
>
> I tried adding it as a 'keep' sub-parameter first (i.e.
> 'hugepages: 1024,keep=1' etc.), but it turns out that we hardcode
> $config->{hugepages} to be a numeric string in a *lot* of different places, so I
> opted for this variant instead. Open for suggestions ofc.
>
> PVE/QemuServer.pm | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 23176dd..4741707 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -384,6 +384,14 @@ EODESC
> description => "Enable/disable hugepages memory.",
> enum => [qw(any 2 1024)],
> },
> + keephugepages => {
> + optional => 1,
> + type => 'boolean',
> + default => 0,
> + description => "Use together with hugepages. If enabled, hugepages will"
> + . " not be deleted after VM shutdown and can be used for"
> + . " subsequent starts.",
> + },
> vcpus => {
> optional => 1,
> type => 'integer',
> @@ -5441,11 +5449,13 @@ sub vm_start {
>
> eval { $run_qemu->() };
> if (my $err = $@) {
> - PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology);
> + PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology)
> + if !$conf->{keephugepages};
> die $err;
> }
>
> - PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology);
> + PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology)
> + if !$conf->{keephugepages};
> };
> eval { PVE::QemuServer::Memory::hugepages_update_locked($code); };
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [pve-devel] [PATCH qemu-server] fix #2570: add 'keephugepages' config
[not found] <20200212133228.8442-1-s.reiter@proxmox.com>
2020-08-20 7:59 ` [pve-devel] [PATCH qemu-server] fix #2570: add 'keephugepages' config Stefan Reiter
@ 2020-08-20 8:17 ` Thomas Lamprecht
2020-08-20 8:20 ` Stefan Reiter
1 sibling, 1 reply; 3+ messages in thread
From: Thomas Lamprecht @ 2020-08-20 8:17 UTC (permalink / raw)
To: PVE development discussion, Stefan Reiter
On 12.02.20 14:32, Stefan Reiter wrote:
> We already keep hugepages if they are created with the kernel
> commandline (hugepagesz=x hugepages=y), but some setups (specifically
> hugepages across multiple NUMA nodes) cannot be configured that way.
> Since we always clear these hugepages at VM shutdown, rebooting a VM
> that uses them might not work, since the requested count might not be
> available anymore by the time we want to use them (also, we would then
> no longer allocate them correctly on the NUMA nodes).
>
but they also will be kept on normal shutdown now.
Why not do this transparent, without extra config, solely on API reboot call?
> Add a 'keephugepages' parameter to skip cleanup and simply leave them
> untouched.
>
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
>
> I tried adding it as a 'keep' sub-parameter first (i.e.
> 'hugepages: 1024,keep=1' etc.), but it turns out that we hardcode
> $config->{hugepages} to be a numeric string in a *lot* of different places, so I
> opted for this variant instead. Open for suggestions ofc.
>
> PVE/QemuServer.pm | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 23176dd..4741707 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -384,6 +384,14 @@ EODESC
> description => "Enable/disable hugepages memory.",
> enum => [qw(any 2 1024)],
> },
> + keephugepages => {
> + optional => 1,
> + type => 'boolean',
> + default => 0,
> + description => "Use together with hugepages. If enabled, hugepages will"
> + . " not be deleted after VM shutdown and can be used for"
> + . " subsequent starts.",
> + },
> vcpus => {
> optional => 1,
> type => 'integer',
> @@ -5441,11 +5449,13 @@ sub vm_start {
>
> eval { $run_qemu->() };
> if (my $err = $@) {
> - PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology);
> + PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology)
> + if !$conf->{keephugepages};
> die $err;
> }
>
> - PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology);
> + PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology)
> + if !$conf->{keephugepages};
> };
> eval { PVE::QemuServer::Memory::hugepages_update_locked($code); };
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [pve-devel] [PATCH qemu-server] fix #2570: add 'keephugepages' config
2020-08-20 8:17 ` Thomas Lamprecht
@ 2020-08-20 8:20 ` Stefan Reiter
0 siblings, 0 replies; 3+ messages in thread
From: Stefan Reiter @ 2020-08-20 8:20 UTC (permalink / raw)
To: Thomas Lamprecht, PVE development discussion
On 8/20/20 10:17 AM, Thomas Lamprecht wrote:
> On 12.02.20 14:32, Stefan Reiter wrote:
>> We already keep hugepages if they are created with the kernel
>> commandline (hugepagesz=x hugepages=y), but some setups (specifically
>> hugepages across multiple NUMA nodes) cannot be configured that way.
>> Since we always clear these hugepages at VM shutdown, rebooting a VM
>> that uses them might not work, since the requested count might not be
>> available anymore by the time we want to use them (also, we would then
>> no longer allocate them correctly on the NUMA nodes).
>>
>
> but they also will be kept on normal shutdown now.
> Why not do this transparent, without extra config, solely on API reboot call?
>
That's the intent, though the reboot API call not releasing them seems
like an additional fix.
From the bug report:
"""
Sometimes it's useful to allocate hugepages at boot time to prevent too
much continuous memory pages being used (eg. cached), so when we start
the VM we can sure it has enough hugepages.
Currently hugepages are freed when VM stops, it's possible that when we
need start the VM again, not enough hugepages can be allocated (which is
my case).
It would be great to have an option to disable hugepage free when VM stop.
"""
It really is an edge case though...
>> Add a 'keephugepages' parameter to skip cleanup and simply leave them
>> untouched.
>>
>> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
>> ---
>>
>> I tried adding it as a 'keep' sub-parameter first (i.e.
>> 'hugepages: 1024,keep=1' etc.), but it turns out that we hardcode
>> $config->{hugepages} to be a numeric string in a *lot* of different places, so I
>> opted for this variant instead. Open for suggestions ofc.
>>
>> PVE/QemuServer.pm | 14 ++++++++++++--
>> 1 file changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
>> index 23176dd..4741707 100644
>> --- a/PVE/QemuServer.pm
>> +++ b/PVE/QemuServer.pm
>> @@ -384,6 +384,14 @@ EODESC
>> description => "Enable/disable hugepages memory.",
>> enum => [qw(any 2 1024)],
>> },
>> + keephugepages => {
>> + optional => 1,
>> + type => 'boolean',
>> + default => 0,
>> + description => "Use together with hugepages. If enabled, hugepages will"
>> + . " not be deleted after VM shutdown and can be used for"
>> + . " subsequent starts.",
>> + },
>> vcpus => {
>> optional => 1,
>> type => 'integer',
>> @@ -5441,11 +5449,13 @@ sub vm_start {
>>
>> eval { $run_qemu->() };
>> if (my $err = $@) {
>> - PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology);
>> + PVE::QemuServer::Memory::hugepages_reset($hugepages_host_topology)
>> + if !$conf->{keephugepages};
>> die $err;
>> }
>>
>> - PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology);
>> + PVE::QemuServer::Memory::hugepages_pre_deallocate($hugepages_topology)
>> + if !$conf->{keephugepages};
>> };
>> eval { PVE::QemuServer::Memory::hugepages_update_locked($code); };
>>
>>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-08-20 8:21 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20200212133228.8442-1-s.reiter@proxmox.com>
2020-08-20 7:59 ` [pve-devel] [PATCH qemu-server] fix #2570: add 'keephugepages' config Stefan Reiter
2020-08-20 8:17 ` Thomas Lamprecht
2020-08-20 8:20 ` Stefan Reiter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox