From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <a.lauterer@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id E064B615F5
 for <pve-user@lists.proxmox.com>; Mon, 14 Sep 2020 09:07:50 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id D2570DA2F
 for <pve-user@lists.proxmox.com>; Mon, 14 Sep 2020 09:07:50 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [212.186.127.180])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id C8C34DA20
 for <pve-user@lists.proxmox.com>; Mon, 14 Sep 2020 09:07:49 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8B16444864;
 Mon, 14 Sep 2020 09:07:49 +0200 (CEST)
To: Proxmox VE user list <pve-user@lists.proxmox.com>,
 Gianni Milo <gianni.milo22@gmail.com>
References: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl>
 <CACzVk9WshXO9QE8Dj9iorNUEAeHiJway+gnKFcuwg12cKm0HDA@mail.gmail.com>
From: Aaron Lauterer <a.lauterer@proxmox.com>
Message-ID: <4c25b97a-507c-5ff9-effd-c04d39e848dc@proxmox.com>
Date: Mon, 14 Sep 2020 09:07:48 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.2.2
MIME-Version: 1.0
In-Reply-To: <CACzVk9WshXO9QE8Dj9iorNUEAeHiJway+gnKFcuwg12cKm0HDA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.424 Adjusted score from AWL reputation of From: address
 KAM_ASCII_DIVIDERS        0.8 Spam that uses ascii formatting tricks
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 NICE_REPLY_A           -0.001 Looks like a legit reply (A)
 RCVD_IN_DNSWL_MED        -2.3 Sender listed at https://www.dnswl.org/,
 medium trust
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, 
 <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, 
 <mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Mon, 14 Sep 2020 07:07:50 -0000

Alternatively you can try to install the machine in UEFI mode. In UEFI mode with root ZFS the installer will set up systemd-boot instead of grub. This alleviates the problem with grubs limited ZFS compatibility.

See the docs for more infos: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot

HTH,
Aaron

On 9/14/20 12:16 AM, Gianni Milo wrote:
> GRUB does not support all zfs features, so it's quite common for it to fail
> to recognise the rpool during boot if it has a feature which is
> incompatible with it. In your case, I believe that is the "encyption"
> feature.
> Do you recall the issue starting after enabling encryption on rpool/data
> dataset? If so, you may have to rebuild the pool and leave rpool/data
> unencrypted.
>   Note that even though you enabled encryption only on rpool/data, the
> feature will take effect at the pool level, hence the GRUB issue you
> are facing.
> Because of this issue, people have started using a separate boot pool
> (bpool), with limited zfs features enabled and a different data pool
> (rpool) for the OS and the data. I believe that the PVE installer should be
> modified to follow this approach (if it hasn't done it already) to overcome
> similar issues.
> 
> Gianni
> 
> 
> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg <sleemburg@it-functions.nl>
> wrote:
> 
>> Hi All,
>>
>>
>>
>> I have a proxmox system at netcup that was clean installed about two
>>
>> weeks ago.
>>
>>
>>
>> It is full zfs, so root on zfs. I am migrating from one netcup system
>>
>> with less storage to this system.
>>
>>
>>
>> This system is using the pve-no-subscription repository, but after
>>
>> migration I will move the subscription from the 'old' system to this
>> system.
>>
>>
>>
>> The rpool/data dataset is zfs encrypted.
>>
>>
>>
>> Today I did zfs send/recv from the 'old' system to this system to the
>>
>> rpool/data dataset.
>>
>>
>>
>> After that I did an apt update and noticed there where updates available.
>>
>>
>>
>> After the upgrade and the mandatory reboot, the system does not come up
>>
>> anymore. It is stuck in grub rescue.
>>
>>
>>
>> Grub mentions that it has a 'unknown filesystem'.
>>
>>
>>
>> Has anyone else experienced this same situation?
>>
>>
>>
>> If so and you could recover, what was the reason and fix?
>>
>>
>>
>> I am still researching what is causing this. If I boot a .iso then I can
>>
>> import the pool and see all datasets and subvolumes.
>>
>>
>>
>> So, it seems that the zpool itself and the datasets are ok, it is just
>>
>> that grub is unable to recognize them for some reason.
>>
>>
>>
>> I have read about other situations like this where large_dnode seemed to
>>
>> be the cause. I noticed that on the zpool large_dnode is enabled. And it
>>
>> is a creation only setting. It cannot be changed to disabled afterwards.
>>
>>
>>
>> This must have been done by the Proxmox installer. As booting is about
>>
>> the root filesystem, I guess the zfs send / recv to the rpool/data
>>
>> dataset would have nothing to do with it, but I could be wrong.
>>
>>
>>
>> I will continue my research tomorrow evening after some other
>>
>> obligations, but if anyone has an idea, please share it. If it is just
>>
>> how to get more debugging info out of the zfs module in grub.
>>
>>
>>
>> Because 'unknown filesystem', with the zfs module loaded is kind of not
>>
>> helping enough..
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Stephan
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> pve-user mailing list
>>
>> pve-user@lists.proxmox.com
>>
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>>
>>
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 
>