[pve-devel] [PATCH qemu-server] vm start: set minimum timeout of 300s if using PCI passthrough

[pve-devel] [PATCH qemu-server] vm start: set minimum timeout of 300s if using PCI passthrough

[Friedrich Weber]

The default VM startup timeout is max(30, VM memory in GiB) seconds.
Multiple reports in the forum [0] [1] and the bug tracker [2] suggest
this is too short when using PCI passthrough with a large amount of VM
memory, since QEMU needs to map the whole memory during startup (see
comment #2 in [2]). As a result, VM startup fails with "got timeout".

To work around this, ensure that the startup timeout is at least 300s
in case the VM config contains at least one `hostpci[n]` option.

[0]: https://forum.proxmox.com/threads/83765/post-552071
[1]: https://forum.proxmox.com/threads/126398/post-552807
[2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
---

Notes:
    An alternative workaround is offered by an unapplied patch series [3]
    of bug #3502 [2] that makes it possible to set VM-specific timeouts
    (also in the GUI). Users could use this option to manually set a
    higher timeout for VMs that use PCI passthrough. However, it is not
    immediately obvious that a higher timeout is necessary. Since the
    problem seems to come up somewhat frequently, I think it makes sense
    to have the heuristic choose a higher timeout by default.
    
    [2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502
    [3]: https://lists.proxmox.com/pipermail/pve-devel/2023-January/055352.html

 PVE/QemuServer/Helpers.pm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/PVE/QemuServer/Helpers.pm b/PVE/QemuServer/Helpers.pm
index e91f906..11897f4 100644
--- a/PVE/QemuServer/Helpers.pm
+++ b/PVE/QemuServer/Helpers.pm
@@ -160,6 +160,11 @@ sub config_aware_timeout {
 	$timeout = 150;
     }
 
+    # Users reported the default timeout being too short when using PCI passthrough
+    if (grep(/^hostpci[0-9]+$/, keys %$config) && $timeout < 300) {
+	$timeout = 300;
+    }
+
     return $timeout;
 }
 
-- 
2.30.2

Re: [pve-devel] [PATCH qemu-server] vm start: set minimum timeout of 300s if using PCI passthrough

[Fiona Ebner]

Am 03.05.23 um 15:37 schrieb Friedrich Weber:
> The default VM startup timeout is max(30, VM memory in GiB) seconds.
> Multiple reports in the forum [0] [1] and the bug tracker [2] suggest
> this is too short when using PCI passthrough with a large amount of VM
> memory, since QEMU needs to map the whole memory during startup (see
> comment #2 in [2]). As a result, VM startup fails with "got timeout".
> 
> To work around this, ensure that the startup timeout is at least 300s
> in case the VM config contains at least one `hostpci[n]` option.
> 
> [0]: https://forum.proxmox.com/threads/83765/post-552071
> [1]: https://forum.proxmox.com/threads/126398/post-552807
> [2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502
> 
> Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
> ---

Would it make sense to instead add a constant multiplier to the memory
timeout heuristic in presence of PCI passthrough? The user says 65 GiB
takes about 3 min 30 s, so assuming it's more or less linear, the 5 min
from this patch would not be enough for more than ~130 GiB of memory.

> 
> Notes:
>     An alternative workaround is offered by an unapplied patch series [3]
>     of bug #3502 [2] that makes it possible to set VM-specific timeouts
>     (also in the GUI). Users could use this option to manually set a
>     higher timeout for VMs that use PCI passthrough. However, it is not
>     immediately obvious that a higher timeout is necessary. Since the
>     problem seems to come up somewhat frequently, I think it makes sense
>     to have the heuristic choose a higher timeout by default.
>     
>     [2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502
>     [3]: https://lists.proxmox.com/pipermail/pve-devel/2023-January/055352.html

Yes, I think having both the better heuristic and the configurable
timeout makes sense. Since Daniel left, do you want to have another look
at the series/pick it up?

Re: [pve-devel] [PATCH qemu-server] vm start: set minimum timeout of 300s if using PCI passthrough

[Friedrich Weber]

On 21/08/2023 10:33, Fiona Ebner wrote:
> Would it make sense to instead add a constant multiplier to the memory
> timeout heuristic in presence of PCI passthrough? The user says 65 GiB
> takes about 3 min 30 s, so assuming it's more or less linear, the 5 min
> from this patch would not be enough for more than ~130 GiB of memory.

You're right, a heuristic makes more sense here than a constant
multiplier. I'll give it a try in the next version.

>> Notes:
>>     An alternative workaround is offered by an unapplied patch series [3]
>>     of bug #3502 [2] that makes it possible to set VM-specific timeouts
>>     (also in the GUI). Users could use this option to manually set a
>>     higher timeout for VMs that use PCI passthrough. However, it is not
>>     immediately obvious that a higher timeout is necessary. Since the
>>     problem seems to come up somewhat frequently, I think it makes sense
>>     to have the heuristic choose a higher timeout by default.
>>     
>>     [2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502
>>     [3]: https://lists.proxmox.com/pipermail/pve-devel/2023-January/055352.html
> 
> Yes, I think having both the better heuristic and the configurable
> timeout makes sense. Since Daniel left, do you want to have another look
> at the series/pick it up?

Sure! When I'm back from vacation, I'll send another version of this
patch series and also take a look at Daniel's old patch series. I'll
probably send them separately though, as they are somewhat independent.

Re: [pve-devel] [PATCH qemu-server] vm start: set minimum timeout of 300s if using PCI passthrough

[Fiona Ebner]

Am 08.09.23 um 13:40 schrieb Friedrich Weber:
> On 21/08/2023 10:33, Fiona Ebner wrote:
>> Would it make sense to instead add a constant multiplier to the memory
>> timeout heuristic in presence of PCI passthrough? The user says 65 GiB
>> takes about 3 min 30 s, so assuming it's more or less linear, the 5 min
>> from this patch would not be enough for more than ~130 GiB of memory.
> 
> You're right, a heuristic makes more sense here than a constant
> multiplier. I'll give it a try in the next version.
> 

I think a constant multiplier in combination with the current logic is
fine, just not a fixed constant overall ;)