* [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
@ 2020-10-06 15:59 Daniel Berteaud
2020-10-06 16:23 ` Daniel Berteaud
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Berteaud @ 2020-10-06 15:59 UTC (permalink / raw)
To: pve-user
Hi there.
I've been migrating a few VM from an NFS storage to a ZFS over iSCSI one (using LIO backend) on a fully updated PVE6 cluster. Everything went fine while live-moving VM disks. There's also no problem starting VM from the CLI, eg
qm start 105
But, starting the same VM from the web interface fails with the unhelpful message :
TASK ERROR: start failed: QEMU exited with code -1
Seems there're some more checks somewhere which are failing when running through the web interface. How could I debug this ? I usually just start with qm or manually the qemu command (using the qm showcmd output), but as both are working here ...
Regards,
Daniel
--
[ https://www.firewall-services.com/ ]
Daniel Berteaud
FIREWALL-SERVICES SAS, La sécurité des réseaux
Société de Services en Logiciels Libres
Tél : +33.5 56 64 15 32
Matrix: @dani:fws.fr
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
2020-10-06 15:59 [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI Daniel Berteaud
@ 2020-10-06 16:23 ` Daniel Berteaud
2020-10-07 6:20 ` Dominik Csapak
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Berteaud @ 2020-10-06 16:23 UTC (permalink / raw)
To: Proxmox VE user list
----- Le 6 Oct 20, à 17:59, Daniel Berteaud daniel@firewall-services.com a écrit :
> Hi there.
>
> I've been migrating a few VM from an NFS storage to a ZFS over iSCSI one (using
> LIO backend) on a fully updated PVE6 cluster. Everything went fine while
> live-moving VM disks. There's also no problem starting VM from the CLI, eg
>
> qm start 105
>
> But, starting the same VM from the web interface fails with the unhelpful
> message :
>
> TASK ERROR: start failed: QEMU exited with code -1
>
> Seems there're some more checks somewhere which are failing when running through
> the web interface. How could I debug this ? I usually just start with qm or
> manually the qemu command (using the qm showcmd output), but as both are
> working here ...
>
Some more info :
* This cluster was using a ZFS over iSCSI storage a few months ago, without issue
* A VM with ZFS over iSCSI disk started with "qm start" can be migrated from the web interface (so, the start phase on the new node is not affected when done in a migration context)
* The problem goes away as soon as I replace the ZFS over iSCSI storage with a image on an NFS server (no other change)
Regards,
Daniel
--
[ https://www.firewall-services.com/ ]
Daniel Berteaud
FIREWALL-SERVICES SAS, La sécurité des réseaux
Société de Services en Logiciels Libres
Tél : +33.5 56 64 15 32
Matrix: @dani:fws.fr
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
2020-10-06 16:23 ` Daniel Berteaud
@ 2020-10-07 6:20 ` Dominik Csapak
2020-10-07 7:55 ` Daniel Berteaud
0 siblings, 1 reply; 6+ messages in thread
From: Dominik Csapak @ 2020-10-07 6:20 UTC (permalink / raw)
To: pve-user
Hi,
On 10/6/20 6:23 PM, Daniel Berteaud wrote:
> ----- Le 6 Oct 20, à 17:59, Daniel Berteaud daniel@firewall-services.com a écrit :
>
>> Hi there.
>>
>> I've been migrating a few VM from an NFS storage to a ZFS over iSCSI one (using
>> LIO backend) on a fully updated PVE6 cluster. Everything went fine while
>> live-moving VM disks. There's also no problem starting VM from the CLI, eg
>>
>> qm start 105
>>
>> But, starting the same VM from the web interface fails with the unhelpful
>> message :
>>
>> TASK ERROR: start failed: QEMU exited with code -1
>>
>> Seems there're some more checks somewhere which are failing when running through
>> the web interface. How could I debug this ? I usually just start with qm or
>> manually the qemu command (using the qm showcmd output), but as both are
>> working here ...
>>
>
> Some more info :
>
> * This cluster was using a ZFS over iSCSI storage a few months ago, without issue
> * A VM with ZFS over iSCSI disk started with "qm start" can be migrated from the web interface (so, the start phase on the new node is not affected when done in a migration context)
> * The problem goes away as soon as I replace the ZFS over iSCSI storage with a image on an NFS server (no other change)
is there anything else in the task log of the start? ( you can obtain
that by dobuble clicking the task)
also maybe there are some hints in the pve log during that time?
basically 'qm start ID' does the same think as the API so
there should be no difference
can you post your vm config? (qm config ID)
Regards,
Dominik
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
2020-10-07 6:20 ` Dominik Csapak
@ 2020-10-07 7:55 ` Daniel Berteaud
2020-10-07 12:55 ` Daniel Berteaud
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Berteaud @ 2020-10-07 7:55 UTC (permalink / raw)
To: Proxmox VE user list
----- Le 7 Oct 20, à 8:20, Dominik Csapak d.csapak@proxmox.com a écrit :
>
> is there anything else in the task log of the start? ( you can obtain
> that by dobuble clicking the task)
Nop, nothing else
>
> also maybe there are some hints in the pve log during that time?
Nothing relevant. Syslog just show :
Oct 06 16:15:00 pvo1 pvedaemon[15956]: start failed: QEMU exited with code -1
Oct 06 16:15:00 pvo1 pvedaemon[14869]: <dani@pam> end task UPID:pvo1:00003E54:029AB217:5F7C7BE3:qmstart:149:dani@pam: start failed: QEMU exited with code -1
And the task log in /var/log/pve/tasks just
TASK ERROR: start failed: QEMU exited with code -1
>
> basically 'qm start ID' does the same think as the API so
> there should be no difference
I agree, that's why I do not understand what's going on
>
> can you post your vm config? (qm config ID)
Here's one exmple :
agent: 1
audio0: device=ich9-intel-hda,driver=spice
balloon: 3072
bootdisk: scsi0
cores: 4
cpu: Skylake-Client
cpuunits: 512
ide0: none,media=cdrom
memory: 4096
name: win10-test.fws.fr
net0: virtio=32:8B:E2:AD:90:03,bridge=tests
numa: 0
ostype: win10
scsi0: iscsi-zfs-zol1-prd:vm-115-disk-0,discard=on,size=33G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=fb7d5364-6cfa-46f7-8866-4dedb3427605
sockets: 1
usb0: spice
vga: qxl
(changing other settings, like discard, CPU model, SPICE vs VNC etc has no effect)
And my storage :
zfs: iscsi-zfs-zol1-prd
blocksize 16k
iscsiprovider LIO
pool pool/pve/prod
portal 10.29.255.252
target iqn.2019-10.fr.fws.pve:prod
content images
lio_tpg tpg1
nowritecache 1
sparse 1
Cheers,
Daniel
--
[ https://www.firewall-services.com/ ]
Daniel Berteaud
FIREWALL-SERVICES SAS, La sécurité des réseaux
Société de Services en Logiciels Libres
Tél : +33.5 56 64 15 32
Matrix: @dani:fws.fr
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
2020-10-07 7:55 ` Daniel Berteaud
@ 2020-10-07 12:55 ` Daniel Berteaud
2020-10-09 15:18 ` Stoiko Ivanov
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Berteaud @ 2020-10-07 12:55 UTC (permalink / raw)
To: Proxmox VE user list
----- Le 7 Oct 20, à 9:55, Daniel Berteaud daniel@firewall-services.com a écrit :
Looks like I'm not the only one with this issue. And it's not affecting only ZFS over iSCSI. RAW iSCSI seems to be affected too, see https://forum.proxmox.com/threads/after-upgrading-from-5-x-to-6-x-iscsi-vms-can-only-be-started-via-commandline.77090/
Regards,
Daniel
--
[ https://www.firewall-services.com/ ]
Daniel Berteaud
FIREWALL-SERVICES SAS, La sécurité des réseaux
Société de Services en Logiciels Libres
Tél : +33.5 56 64 15 32
Matrix: @dani:fws.fr
[ https://www.firewall-services.com/ | https://www.firewall-services.com ]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI
2020-10-07 12:55 ` Daniel Berteaud
@ 2020-10-09 15:18 ` Stoiko Ivanov
0 siblings, 0 replies; 6+ messages in thread
From: Stoiko Ivanov @ 2020-10-09 15:18 UTC (permalink / raw)
To: Daniel Berteaud; +Cc: Proxmox VE user list
Hi,
On Wed, 7 Oct 2020 14:55:32 +0200 (CEST)
Daniel Berteaud <daniel@firewall-services.com> wrote:
> ----- Le 7 Oct 20, à 9:55, Daniel Berteaud daniel@firewall-services.com a écrit :
>
> Looks like I'm not the only one with this issue. And it's not affecting only ZFS over iSCSI. RAW iSCSI seems to be affected too, see https://forum.proxmox.com/threads/after-upgrading-from-5-x-to-6-x-iscsi-vms-can-only-be-started-via-commandline.77090/
just to keep both channels (forum and pve-user list) updated - I sent a
patch for the issue to the pve-devel mailinglist [0] (would be grateful
for testing and feedback)
>
> Regards,
> Daniel
>
Best Regards,
stoiko
[0] https://lists.proxmox.com/pipermail/pve-devel/2020-October/045516.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-10-09 15:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-06 15:59 [PVE-User] QEMU failing to start from the web with ZFS Over iSCSI Daniel Berteaud
2020-10-06 16:23 ` Daniel Berteaud
2020-10-07 6:20 ` Dominik Csapak
2020-10-07 7:55 ` Daniel Berteaud
2020-10-07 12:55 ` Daniel Berteaud
2020-10-09 15:18 ` Stoiko Ivanov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox