From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E064B615F5 for ; Mon, 14 Sep 2020 09:07:50 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D2570DA2F for ; Mon, 14 Sep 2020 09:07:50 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id C8C34DA20 for ; Mon, 14 Sep 2020 09:07:49 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8B16444864; Mon, 14 Sep 2020 09:07:49 +0200 (CEST) To: Proxmox VE user list , Gianni Milo References: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl> From: Aaron Lauterer Message-ID: <4c25b97a-507c-5ff9-effd-c04d39e848dc@proxmox.com> Date: Mon, 14 Sep 2020 09:07:48 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.2.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.424 Adjusted score from AWL reputation of From: address KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2020 07:07:50 -0000 Alternatively you can try to install the machine in UEFI mode. In UEFI mode with root ZFS the installer will set up systemd-boot instead of grub. This alleviates the problem with grubs limited ZFS compatibility. See the docs for more infos: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot HTH, Aaron On 9/14/20 12:16 AM, Gianni Milo wrote: > GRUB does not support all zfs features, so it's quite common for it to fail > to recognise the rpool during boot if it has a feature which is > incompatible with it. In your case, I believe that is the "encyption" > feature. > Do you recall the issue starting after enabling encryption on rpool/data > dataset? If so, you may have to rebuild the pool and leave rpool/data > unencrypted. > Note that even though you enabled encryption only on rpool/data, the > feature will take effect at the pool level, hence the GRUB issue you > are facing. > Because of this issue, people have started using a separate boot pool > (bpool), with limited zfs features enabled and a different data pool > (rpool) for the OS and the data. I believe that the PVE installer should be > modified to follow this approach (if it hasn't done it already) to overcome > similar issues. > > Gianni > > > On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg > wrote: > >> Hi All, >> >> >> >> I have a proxmox system at netcup that was clean installed about two >> >> weeks ago. >> >> >> >> It is full zfs, so root on zfs. I am migrating from one netcup system >> >> with less storage to this system. >> >> >> >> This system is using the pve-no-subscription repository, but after >> >> migration I will move the subscription from the 'old' system to this >> system. >> >> >> >> The rpool/data dataset is zfs encrypted. >> >> >> >> Today I did zfs send/recv from the 'old' system to this system to the >> >> rpool/data dataset. >> >> >> >> After that I did an apt update and noticed there where updates available. >> >> >> >> After the upgrade and the mandatory reboot, the system does not come up >> >> anymore. It is stuck in grub rescue. >> >> >> >> Grub mentions that it has a 'unknown filesystem'. >> >> >> >> Has anyone else experienced this same situation? >> >> >> >> If so and you could recover, what was the reason and fix? >> >> >> >> I am still researching what is causing this. If I boot a .iso then I can >> >> import the pool and see all datasets and subvolumes. >> >> >> >> So, it seems that the zpool itself and the datasets are ok, it is just >> >> that grub is unable to recognize them for some reason. >> >> >> >> I have read about other situations like this where large_dnode seemed to >> >> be the cause. I noticed that on the zpool large_dnode is enabled. And it >> >> is a creation only setting. It cannot be changed to disabled afterwards. >> >> >> >> This must have been done by the Proxmox installer. As booting is about >> >> the root filesystem, I guess the zfs send / recv to the rpool/data >> >> dataset would have nothing to do with it, but I could be wrong. >> >> >> >> I will continue my research tomorrow evening after some other >> >> obligations, but if anyone has an idea, please share it. If it is just >> >> how to get more debugging info out of the zfs module in grub. >> >> >> >> Because 'unknown filesystem', with the zfs module loaded is kind of not >> >> helping enough.. >> >> >> >> Best regards, >> >> >> >> Stephan >> >> >> >> >> >> _______________________________________________ >> >> pve-user mailing list >> >> pve-user@lists.proxmox.com >> >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >