Re: [PVE-User] zfs root after last update grub unknown filesystem

[Stephan Leemburg <sleemburg@it-functions.nl>]

On 14-09-2020 11:29, Gianni Milo wrote:
> UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool
> for booting the system (it uses a separate VFAT partition for the
> bootloader).

Yes, that would be good, but the netcup system does not 'support' UEFI.

So, I guess I will take the pain and reinstall the system with a 
traditional lvm as root and a luks encrypted container with the zpool in it.

luks provides more than one keyslot. LVM will do ok for root and works 
at least with grub.

I could access the rpool by booting into the debug installer. So I could 
make a tar of the rpool/ROOT/pve-1 filesystem. That will make a recovery 
more easy, without loosing too much work.

BTW Also the 'rescue mode' of the Proxmox installer barks that there is 
'no rpool'. Must be related.

> However, this method does not solve another problem which is the redundancy
> of the bootloader, as the UEFI partition can be installed only on a single
> (boot) drive. If that drive fails, then one must rebuild the UEFI partition
> on a different or on the replacement drive.
>
> On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user <pve-user@lists.proxmox.com>
> wrote:
>
>>
>>
>> ---------- Forwarded message ----------
>> From: Arjen <leesteken@protonmail.ch>
>> To: Proxmox VE user list <pve-user@lists.proxmox.com>
>> Cc:
>> Bcc:
>> Date: Mon, 14 Sep 2020 08:04:03 +0000
>> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Monday, September 14, 2020 9:07 AM, Aaron Lauterer <
>> a.lauterer@proxmox.com> wrote:
>>
>>> Alternatively you can try to install the machine in UEFI mode. In UEFI
>> mode with root ZFS the installer will set up systemd-boot instead of grub.
>> This alleviates the problem with grubs limited ZFS compatibility.
>>> See the docs for more infos:
>> https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot
>>> HTH,
>>> Aaron
>> I had the same problem with a zpool feature that could not be turned off.
>> I used this work-around for a while in the hope that the bug would be fixed
>> (GRUB patch existed for 5 years but it was never tested/reviewed):
>> Put a new Proxmox VE installation on USB and copy /boot from the hard
>> drives to the USB installation.
>> With that, it could boot from USB, after which it would pickup on the
>> rpool on the hard drives (probably by lucky chance).
>> Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not
>> exists when I encountered the issue) fixed it.
>>
>> I would suggest taking the pain now, and reinstall using UEFI, to prevent
>> such issues in the future.
>>
>>> On 9/14/20 12:16 AM, Gianni Milo wrote:
>>>
>>>> GRUB does not support all zfs features, so it's quite common for it to
>> fail
>>>> to recognise the rpool during boot if it has a feature which is
>>>> incompatible with it. In your case, I believe that is the "encyption"
>>>> feature.
>>>> Do you recall the issue starting after enabling encryption on
>> rpool/data
>>>> dataset? If so, you may have to rebuild the pool and leave rpool/data
>>>> unencrypted.
>>>> Note that even though you enabled encryption only on rpool/data, the
>>>> feature will take effect at the pool level, hence the GRUB issue you
>>>> are facing.
>>>> Because of this issue, people have started using a separate boot pool
>>>> (bpool), with limited zfs features enabled and a different data pool
>>>> (rpool) for the OS and the data. I believe that the PVE installer
>> should be
>>>> modified to follow this approach (if it hasn't done it already) to
>> overcome
>>>> similar issues.
>>>> Gianni
>>>> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg
>> sleemburg@it-functions.nl
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>> I have a proxmox system at netcup that was clean installed about two
>>>>> weeks ago.
>>>>> It is full zfs, so root on zfs. I am migrating from one netcup system
>>>>> with less storage to this system.
>>>>> This system is using the pve-no-subscription repository, but after
>>>>> migration I will move the subscription from the 'old' system to this
>>>>> system.
>>>>> The rpool/data dataset is zfs encrypted.
>>>>> Today I did zfs send/recv from the 'old' system to this system to the
>>>>> rpool/data dataset.
>>>>> After that I did an apt update and noticed there where updates
>> available.
>>>>> After the upgrade and the mandatory reboot, the system does not come
>> up
>>>>> anymore. It is stuck in grub rescue.
>>>>> Grub mentions that it has a 'unknown filesystem'.
>>>>> Has anyone else experienced this same situation?
>>>>> If so and you could recover, what was the reason and fix?
>>>>> I am still researching what is causing this. If I boot a .iso then I
>> can
>>>>> import the pool and see all datasets and subvolumes.
>>>>> So, it seems that the zpool itself and the datasets are ok, it is
>> just
>>>>> that grub is unable to recognize them for some reason.
>>>>> I have read about other situations like this where large_dnode
>> seemed to
>>>>> be the cause. I noticed that on the zpool large_dnode is enabled.
>> And it
>>>>> is a creation only setting. It cannot be changed to disabled
>> afterwards.
>>>>> This must have been done by the Proxmox installer. As booting is
>> about
>>>>> the root filesystem, I guess the zfs send / recv to the rpool/data
>>>>> dataset would have nothing to do with it, but I could be wrong.
>>>>> I will continue my research tomorrow evening after some other
>>>>> obligations, but if anyone has an idea, please share it. If it is
>> just
>>>>> how to get more debugging info out of the zfs module in grub.
>>>>> Because 'unknown filesystem', with the zfs module loaded is kind of
>> not
>>>>> helping enough..
>>>>> Best regards,
>>>>> Stephan
>>>>>
>>>>> pve-user mailing list
>>>>> pve-user@lists.proxmox.com
>>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>> pve-user mailing list
>>>> pve-user@lists.proxmox.com
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> pve-user mailing list
>>> pve-user@lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Arjen via pve-user <pve-user@lists.proxmox.com>
>> To: Proxmox VE user list <pve-user@lists.proxmox.com>
>> Cc: Arjen <leesteken@protonmail.ch>
>> Bcc:
>> Date: Mon, 14 Sep 2020 08:04:03 +0000
>> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user