From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 41BE2600DE for ; Tue, 11 Aug 2020 18:20:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 24C73A0B6 for ; Tue, 11 Aug 2020 18:20:14 +0200 (CEST) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 15BB6A09E for ; Tue, 11 Aug 2020 18:20:12 +0200 (CEST) Received: by mail-lj1-x22b.google.com with SMTP id v12so14162931ljc.10 for ; Tue, 11 Aug 2020 09:20:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=QefB52LeHDfQHUGlt/pHhzZLO+s02KIK31Ei9zb5hfw=; b=Ceqf9/YiKyCiOYib5l6vSRZGL0AUs0KfXp9fJoWrzWA+5c+zkUYzN9V0anU+LxQodA MYzBCrf9s3c2KGLrtjcVpQO+qWXVYb8+BXc71Y9/tBCb68nNHuiz8kgzMxTchdKV8Ja6 AwHix7H3RxikCUy2eCNzmu5z2+DfzPVDrLej1WBYVm49e6kd6xthN0hqOFshsYE177Dj aC1NUq4AcwpMk4jjkfkcMaZyuEQoZ15OUWef/BPa3cjFztewnVCKeFOOPdGkK3g/Yel9 q5BUXEipsehOfGKqh+0eBRhTGIITTWS+DOPGSyQzgES3CpCSanAozVhk4QwpxGuPPVzY 5IYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=QefB52LeHDfQHUGlt/pHhzZLO+s02KIK31Ei9zb5hfw=; b=tqIbGFQFOytiPsWZKW3kuN/O+XvDvFgKXjUJqC1T4ZORCPJ3OiIOf59qVolKk3TPbd sVwmYa52f9HnOWHr0fygIlKr3Cub3HZhQgwNwFzHe4sWMTIaZLff2DlWqIWt/bIccaVs RxffhtlhrQwCjG5/tNlSi/ajh1BwGYKEO+0aryv3S+ayIDU0MvgrxMSNWM4VTKK8JkUU m+cnNZZsuMoWy8662Grj+ezCgZtBf+1wTlp1OWC07MiBtuWoL5eVl3p20bY2ofOu4wds LKhyA+B+rQmsqCfcLQDRK+3TzNXQAMVb/uTXAN87MfGPQTVTcxrQpPYdNRRAls4AiZAU A1KQ== X-Gm-Message-State: AOAM532zlgN5g5firIX/nkotmofs2DEwvS/UdSSoNdfFVfeqjTchyg3v IxQBsFRyTVjHN9FV3/k0oqzdXlYr/RuLUwQeR2D0A8nd X-Google-Smtp-Source: ABdhPJybPgE6bCwudKG6M30QwPJZqsDgee1GrsngjetAl9Z0pbG1eJNsJ0dN+BMZ7SOJODotSFrlVURDinz0Fk6jLug= X-Received: by 2002:a2e:9b95:: with SMTP id z21mr3475829lji.83.1597162804749; Tue, 11 Aug 2020 09:20:04 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yannis Milios Date: Tue, 11 Aug 2020 17:19:53 +0100 Message-ID: To: Proxmox VE user list X-SPAM-LEVEL: Spam detection results: 0 AWL -0.000 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com, ix.io] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] LVM over ISCSI issue X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2020 16:20:44 -0000 Just out of curiosity, can you temporarily disable discard support on the target VM disk to see if that will make a difference? On Tue, 11 Aug 2020 at 11:52, Amin Vakil via pve-user < pve-user@lists.proxmox.com> wrote: > > > > ---------- Forwarded message ---------- > From: Amin Vakil > To: pve-user@lists.proxmox.com > Cc: > Bcc: > Date: Tue, 11 Aug 2020 15:22:02 +0430 > Subject: [PVE-User] LVM over ISCSI issue > Sorry, I just realized I've send my previous email with wrong subject. > > > Looks like you are trying to clone an existing VM stored on the NFS > server > > to the iSCSI target. Problem is that both the NFS server and iSCSI target > > are served by the same machine, so my best guess is that the I/O on the > > storage server is getting saturated. Have you tried creating a new VM on > > iSCSI instead ? Can you also try setting bandwidth limits at "Datacenter > -> > > Options -> Bandwidth Limits" and then retry cloning operation ? > > > > I cloned to nfs to make sure it's ok, it doesn't make any difference > when cloning from local or nfs to iscsi. the problem is the same, here > is the log cloning from local to iscsi: http://ix.io/2tRX > > Also sometimes it errors in the middle of cloning process: > http://ix.io/2tRV > > Creating a vm and adding a disk on iscsi to an existing vm is fine. > > I have set bandwidth limits and it didn't make any difference. > > >>> kernel: blk_update_request: I/O error, dev sdb, sector 1762240503 op > > 0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0 prio class 0 > > > > Is this on the storage server or the PVE host ? What does /dev/sdb > > represent in this case ? if it's a hard drive, it looks like it's having > > some hw issues ? Better to test with smartctl if that's the case. > > The storage server is a qnap NAS, and it's health and smart is > constantly monitored. Of course faulty hardware can never be ruled out > completely, but I doubt that would be the case, because qnap smart > status and hard health is ok and none of the running VMs on iscsi are > facing any issue. > > fdisk -l: > ... > Disk /dev/sdb: 3.9 TiB, 4294967296000 bytes, 8388608000 sectors > Disk model: iSCSI Storage > ... > > iscsiadm -m session -P 2: > http://ix.io/2tRY (ip address and username have been changed, everything > else are unchanged) > > Although I have cloned to iscsi now, but it succeed after trying many > times, the error seems to be random. > > Also if it helps someone on ##proxmox IRC told me this: > it basically fails at the sparse scsi command > WRITE_ZEROES > that is _usually_ an array issue > > > > > > > ---------- Forwarded message ---------- > From: Amin Vakil via pve-user > To: pve-user@lists.proxmox.com > Cc: Amin Vakil > Bcc: > Date: Tue, 11 Aug 2020 15:22:02 +0430 > Subject: [PVE-User] LVM over ISCSI issue > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile