Re: [PVE-User] LVM over ISCSI issue

Re: [PVE-User] LVM over ISCSI issue

[Yannis Milios]

The fact that it was successful at least once with the discard option
disabled is a progress in my opinion. Did you ever have a successful clone
job with discard enabled ?

Also, the error seems to be different in each case...

with discard enabled:
qemu-img: error while writing at byte 1990201344:* Input/output error*

with discard disabled: qemu-img: error while writing at byte
2147483136: *Device
or resource busy*

Can you check also in dmesg for any clues during each case ?
Also, when was the last time this was working without issues ? any changes
both on PVE and NAS side after that ?

Y.


On Wed, 12 Aug 2020 at 09:12, Amin Vakil via pve-user <
pve-user@lists.proxmox.com> wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil <info@aminvakil.com>
> To: pve-user@lists.proxmox.com
> Cc:
> Bcc:
> Date: Wed, 12 Aug 2020 12:42:10 +0430
> Subject: [PVE-User] LVM over ISCSI issue
> No difference, totally random.
>
> Also here are the logs of clones in case I'm missing something.
>
> Error, discard enabled: http://ix.io/2tXB
> OK, discard disabled: http://ix.io/2tXC
> Error, discard disabled: http://ix.io/2tXD
>
> > Just out of curiosity, can you temporarily disable discard support on
> > the
> > target VM disk to see if that will make a difference?
>
>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil via pve-user <pve-user@lists.proxmox.com>
> To: pve-user@lists.proxmox.com
> Cc: Amin Vakil <info@aminvakil.com>
> Bcc:
> Date: Wed, 12 Aug 2020 12:42:10 +0430
> Subject: [PVE-User] LVM over ISCSI issue
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] LVM over ISCSI issue

[Yannis Milios]

On Wed, 12 Aug 2020 at 14:05, Amin Vakil via pve-user <
pve-user@lists.proxmox.com> wrote:

>
> There isn't any useful log on qnap NAS, but most of times its RAM is
> full and it doesn't have any cache. Maybe qnap prevents writing when
> it's fully used and maybe proxmox thinks it has complete access over
> /dev/sdb and therefore when it sees it cannot write gives an
> input/output error.
>

Indeed, that would make sense...


> Isn't there a tweak in proxmox to retry writing on iscsi until it succeeds?
>

I cannot comment on that, as I'm not using iscsi on pve. You can check in
pve documentation or wiki, otherwise on open-iscsi documentation.

Y.

Re: [PVE-User] LVM over ISCSI issue

[Yannis Milios]

On Wed, 12 Aug 2020 at 12:12, Amin Vakil via pve-user <
pve-user@lists.proxmox.com> wrote:

> Do you have any reason leading you to believe enabling discard is the
> problem?
>

I was mainly driven by search results on the internet based on
"0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0" as keywords.

>> kernel: sd 1:0:0:0: [sdb] tag#74 Add. Sense: Logical unit communication
failure

This also suggests that the (iscsi?) communication between the initiator
and the target was lost for some reason.
Can you check on iscsi target (NAS) logs for any clues at the same time?
If possible, you could try separating iscsi network from nfs network or
even rolling back to a previous pve kernel to see if that will help.

Y.

Re: [PVE-User] LVM over ISCSI issue

[Yannis Milios]

Just out of curiosity, can you temporarily disable discard support on the
target VM disk to see if that will make a difference?


On Tue, 11 Aug 2020 at 11:52, Amin Vakil via pve-user <
pve-user@lists.proxmox.com> wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil <info@aminvakil.com>
> To: pve-user@lists.proxmox.com
> Cc:
> Bcc:
> Date: Tue, 11 Aug 2020 15:22:02 +0430
> Subject: [PVE-User] LVM over ISCSI issue
> Sorry, I just realized I've send my previous email with wrong subject.
>
> > Looks like you are trying to clone an existing VM stored on the NFS
> server
> > to the iSCSI target. Problem is that both the NFS server and iSCSI target
> > are served by the same machine, so my best guess is that the I/O on the
> > storage server is getting saturated. Have you tried creating a new VM on
> > iSCSI instead ? Can you also try setting bandwidth limits at "Datacenter
> ->
> > Options -> Bandwidth Limits" and then retry cloning operation ?
> >
>
> I cloned to nfs to make sure it's ok, it doesn't make any difference
> when cloning from local or nfs to iscsi. the problem is the same, here
> is the log cloning from local to iscsi: http://ix.io/2tRX
>
> Also sometimes it errors in the middle of cloning process:
> http://ix.io/2tRV
>
> Creating a vm and adding a disk on iscsi to an existing vm is fine.
>
> I have set bandwidth limits and it didn't make any difference.
>
> >>> kernel: blk_update_request: I/O error, dev sdb, sector 1762240503 op
> > 0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0 prio class 0
> >
> > Is this on the storage server or the PVE host ? What does /dev/sdb
> > represent in this case ? if it's a hard drive, it looks like it's having
> > some hw issues ? Better to test with smartctl if that's the case.
>
> The storage server is a qnap NAS, and it's health and smart is
> constantly monitored. Of course faulty hardware can never be ruled out
> completely, but I doubt that would be the case, because qnap smart
> status and hard health is ok and none of the running VMs on iscsi are
> facing any issue.
>
> fdisk -l:
> ...
> Disk /dev/sdb: 3.9 TiB, 4294967296000 bytes, 8388608000 sectors
> Disk model: iSCSI Storage
> ...
>
> iscsiadm -m session -P 2:
> http://ix.io/2tRY (ip address and username have been changed, everything
> else are unchanged)
>
> Although I have cloned to iscsi now, but it succeed after trying many
> times, the error seems to be random.
>
> Also if it helps someone on ##proxmox IRC told me this:
> it basically fails at the sparse scsi command
> WRITE_ZEROES
> that is _usually_ an array issue
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil via pve-user <pve-user@lists.proxmox.com>
> To: pve-user@lists.proxmox.com
> Cc: Amin Vakil <info@aminvakil.com>
> Bcc:
> Date: Tue, 11 Aug 2020 15:22:02 +0430
> Subject: [PVE-User] LVM over ISCSI issue
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
-- 
Sent from Gmail Mobile

Re: [PVE-User] LVM over ISCSI issue

[Yannis Milios]

>> qemu-img: error while writing at byte 14827520: Input/output error
>> TASK ERROR: storage migration failed: copy failed: command
'/usr/bin/qemu-img convert -p -n -f raw -O raw
/mnt/pve/nas02nfs/images/2014/vm-2014-disk-0.raw
/dev/vg_nas02/vm-2014-disk-0' failed: exit code 1

Looks like you are trying to clone an existing VM stored on the NFS server
to the iSCSI target. Problem is that both the NFS server and iSCSI target
are served by the same machine, so my best guess is that the I/O on the
storage server is getting saturated. Have you tried creating a new VM on
iSCSI instead ? Can you also try setting bandwidth limits at "Datacenter ->
Options -> Bandwidth Limits" and then retry cloning operation ?

>>kernel: blk_update_request: I/O error, dev sdb, sector 1762240503 op
0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0 prio class 0

Is this on the storage server or the PVE host ? What does /dev/sdb
represent in this case ? if it's a hard drive, it looks like it's having
some hw issues ? Better to test with smartctl if that's the case.

Y.




On Tue, 11 Aug 2020 at 08:59, Amin Vakil via pve-user <
pve-user@lists.proxmox.com> wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil <info@aminvakil.com>
> To: pve-user@lists.proxmox.com
> Cc:
> Bcc:
> Date: Tue, 11 Aug 2020 12:20:20 +0430
> Subject: LVM over ISCSI issue
> Hello
> I'm facing issues with lvm over iscsi sometimes.
> There is a nas and I have created a nfs directory and an iscsi and I've
> attached them both to my proxmoxs.
> NFS is fine and data will be copied on it.
> but iscsi faces issue sometimes and I can't migrate vm's disk to it
> Although it's up and running, there are vms which disks are already on
> iscsi and works without problem
> error: http://ix.io/2tRq
> Also in journalctl I'm seeing this error: http://ix.io/2tRr
> As I've mentioned there are vms which are already running on iscsi, I've
> even shut them down, disable iscsi, enable iscsi and start them again
> and they're ok.
> Also `iscsiadm -m session -P 2` shows me that the proxmox is logged in.
> Any idea?
>
>
>
>
> ---------- Forwarded message ----------
> From: Amin Vakil via pve-user <pve-user@lists.proxmox.com>
> To: pve-user@lists.proxmox.com
> Cc: Amin Vakil <info@aminvakil.com>
> Bcc:
> Date: Tue, 11 Aug 2020 12:20:20 +0430
> Subject: [PVE-User] LVM over ISCSI issue
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>