* [PVE-User] zfs root after last update grub unknown filesystem @ 2020-09-13 21:11 Stephan Leemburg 2020-09-13 22:16 ` Gianni Milo 0 siblings, 1 reply; 7+ messages in thread From: Stephan Leemburg @ 2020-09-13 21:11 UTC (permalink / raw) To: Proxmox VE user list Hi All, I have a proxmox system at netcup that was clean installed about two weeks ago. It is full zfs, so root on zfs. I am migrating from one netcup system with less storage to this system. This system is using the pve-no-subscription repository, but after migration I will move the subscription from the 'old' system to this system. The rpool/data dataset is zfs encrypted. Today I did zfs send/recv from the 'old' system to this system to the rpool/data dataset. After that I did an apt update and noticed there where updates available. After the upgrade and the mandatory reboot, the system does not come up anymore. It is stuck in grub rescue. Grub mentions that it has a 'unknown filesystem'. Has anyone else experienced this same situation? If so and you could recover, what was the reason and fix? I am still researching what is causing this. If I boot a .iso then I can import the pool and see all datasets and subvolumes. So, it seems that the zpool itself and the datasets are ok, it is just that grub is unable to recognize them for some reason. I have read about other situations like this where large_dnode seemed to be the cause. I noticed that on the zpool large_dnode is enabled. And it is a creation only setting. It cannot be changed to disabled afterwards. This must have been done by the Proxmox installer. As booting is about the root filesystem, I guess the zfs send / recv to the rpool/data dataset would have nothing to do with it, but I could be wrong. I will continue my research tomorrow evening after some other obligations, but if anyone has an idea, please share it. If it is just how to get more debugging info out of the zfs module in grub. Because 'unknown filesystem', with the zfs module loaded is kind of not helping enough.. Best regards, Stephan ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PVE-User] zfs root after last update grub unknown filesystem 2020-09-13 21:11 [PVE-User] zfs root after last update grub unknown filesystem Stephan Leemburg @ 2020-09-13 22:16 ` Gianni Milo 2020-09-14 7:07 ` Aaron Lauterer 0 siblings, 1 reply; 7+ messages in thread From: Gianni Milo @ 2020-09-13 22:16 UTC (permalink / raw) To: Proxmox VE user list GRUB does not support all zfs features, so it's quite common for it to fail to recognise the rpool during boot if it has a feature which is incompatible with it. In your case, I believe that is the "encyption" feature. Do you recall the issue starting after enabling encryption on rpool/data dataset? If so, you may have to rebuild the pool and leave rpool/data unencrypted. Note that even though you enabled encryption only on rpool/data, the feature will take effect at the pool level, hence the GRUB issue you are facing. Because of this issue, people have started using a separate boot pool (bpool), with limited zfs features enabled and a different data pool (rpool) for the OS and the data. I believe that the PVE installer should be modified to follow this approach (if it hasn't done it already) to overcome similar issues. Gianni On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg <sleemburg@it-functions.nl> wrote: > Hi All, > > > > I have a proxmox system at netcup that was clean installed about two > > weeks ago. > > > > It is full zfs, so root on zfs. I am migrating from one netcup system > > with less storage to this system. > > > > This system is using the pve-no-subscription repository, but after > > migration I will move the subscription from the 'old' system to this > system. > > > > The rpool/data dataset is zfs encrypted. > > > > Today I did zfs send/recv from the 'old' system to this system to the > > rpool/data dataset. > > > > After that I did an apt update and noticed there where updates available. > > > > After the upgrade and the mandatory reboot, the system does not come up > > anymore. It is stuck in grub rescue. > > > > Grub mentions that it has a 'unknown filesystem'. > > > > Has anyone else experienced this same situation? > > > > If so and you could recover, what was the reason and fix? > > > > I am still researching what is causing this. If I boot a .iso then I can > > import the pool and see all datasets and subvolumes. > > > > So, it seems that the zpool itself and the datasets are ok, it is just > > that grub is unable to recognize them for some reason. > > > > I have read about other situations like this where large_dnode seemed to > > be the cause. I noticed that on the zpool large_dnode is enabled. And it > > is a creation only setting. It cannot be changed to disabled afterwards. > > > > This must have been done by the Proxmox installer. As booting is about > > the root filesystem, I guess the zfs send / recv to the rpool/data > > dataset would have nothing to do with it, but I could be wrong. > > > > I will continue my research tomorrow evening after some other > > obligations, but if anyone has an idea, please share it. If it is just > > how to get more debugging info out of the zfs module in grub. > > > > Because 'unknown filesystem', with the zfs module loaded is kind of not > > helping enough.. > > > > Best regards, > > > > Stephan > > > > > > _______________________________________________ > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PVE-User] zfs root after last update grub unknown filesystem 2020-09-13 22:16 ` Gianni Milo @ 2020-09-14 7:07 ` Aaron Lauterer [not found] ` <mailman.74.1600071257.452.pve-user@lists.proxmox.com> 0 siblings, 1 reply; 7+ messages in thread From: Aaron Lauterer @ 2020-09-14 7:07 UTC (permalink / raw) To: Proxmox VE user list, Gianni Milo Alternatively you can try to install the machine in UEFI mode. In UEFI mode with root ZFS the installer will set up systemd-boot instead of grub. This alleviates the problem with grubs limited ZFS compatibility. See the docs for more infos: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot HTH, Aaron On 9/14/20 12:16 AM, Gianni Milo wrote: > GRUB does not support all zfs features, so it's quite common for it to fail > to recognise the rpool during boot if it has a feature which is > incompatible with it. In your case, I believe that is the "encyption" > feature. > Do you recall the issue starting after enabling encryption on rpool/data > dataset? If so, you may have to rebuild the pool and leave rpool/data > unencrypted. > Note that even though you enabled encryption only on rpool/data, the > feature will take effect at the pool level, hence the GRUB issue you > are facing. > Because of this issue, people have started using a separate boot pool > (bpool), with limited zfs features enabled and a different data pool > (rpool) for the OS and the data. I believe that the PVE installer should be > modified to follow this approach (if it hasn't done it already) to overcome > similar issues. > > Gianni > > > On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg <sleemburg@it-functions.nl> > wrote: > >> Hi All, >> >> >> >> I have a proxmox system at netcup that was clean installed about two >> >> weeks ago. >> >> >> >> It is full zfs, so root on zfs. I am migrating from one netcup system >> >> with less storage to this system. >> >> >> >> This system is using the pve-no-subscription repository, but after >> >> migration I will move the subscription from the 'old' system to this >> system. >> >> >> >> The rpool/data dataset is zfs encrypted. >> >> >> >> Today I did zfs send/recv from the 'old' system to this system to the >> >> rpool/data dataset. >> >> >> >> After that I did an apt update and noticed there where updates available. >> >> >> >> After the upgrade and the mandatory reboot, the system does not come up >> >> anymore. It is stuck in grub rescue. >> >> >> >> Grub mentions that it has a 'unknown filesystem'. >> >> >> >> Has anyone else experienced this same situation? >> >> >> >> If so and you could recover, what was the reason and fix? >> >> >> >> I am still researching what is causing this. If I boot a .iso then I can >> >> import the pool and see all datasets and subvolumes. >> >> >> >> So, it seems that the zpool itself and the datasets are ok, it is just >> >> that grub is unable to recognize them for some reason. >> >> >> >> I have read about other situations like this where large_dnode seemed to >> >> be the cause. I noticed that on the zpool large_dnode is enabled. And it >> >> is a creation only setting. It cannot be changed to disabled afterwards. >> >> >> >> This must have been done by the Proxmox installer. As booting is about >> >> the root filesystem, I guess the zfs send / recv to the rpool/data >> >> dataset would have nothing to do with it, but I could be wrong. >> >> >> >> I will continue my research tomorrow evening after some other >> >> obligations, but if anyone has an idea, please share it. If it is just >> >> how to get more debugging info out of the zfs module in grub. >> >> >> >> Because 'unknown filesystem', with the zfs module loaded is kind of not >> >> helping enough.. >> >> >> >> Best regards, >> >> >> >> Stephan >> >> >> >> >> >> _______________________________________________ >> >> pve-user mailing list >> >> pve-user@lists.proxmox.com >> >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <mailman.74.1600071257.452.pve-user@lists.proxmox.com>]
* Re: [PVE-User] zfs root after last update grub unknown filesystem [not found] ` <mailman.74.1600071257.452.pve-user@lists.proxmox.com> @ 2020-09-14 9:29 ` Gianni Milo 2020-09-14 9:34 ` Aaron Lauterer 2020-09-15 6:02 ` Stephan Leemburg 0 siblings, 2 replies; 7+ messages in thread From: Gianni Milo @ 2020-09-14 9:29 UTC (permalink / raw) To: Proxmox VE user list UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool for booting the system (it uses a separate VFAT partition for the bootloader). However, this method does not solve another problem which is the redundancy of the bootloader, as the UEFI partition can be installed only on a single (boot) drive. If that drive fails, then one must rebuild the UEFI partition on a different or on the replacement drive. On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user <pve-user@lists.proxmox.com> wrote: > > > > ---------- Forwarded message ---------- > From: Arjen <leesteken@protonmail.ch> > To: Proxmox VE user list <pve-user@lists.proxmox.com> > Cc: > Bcc: > Date: Mon, 14 Sep 2020 08:04:03 +0000 > Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < > a.lauterer@proxmox.com> wrote: > > > Alternatively you can try to install the machine in UEFI mode. In UEFI > mode with root ZFS the installer will set up systemd-boot instead of grub. > This alleviates the problem with grubs limited ZFS compatibility. > > > > See the docs for more infos: > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot > > > > HTH, > > Aaron > > I had the same problem with a zpool feature that could not be turned off. > I used this work-around for a while in the hope that the bug would be fixed > (GRUB patch existed for 5 years but it was never tested/reviewed): > Put a new Proxmox VE installation on USB and copy /boot from the hard > drives to the USB installation. > With that, it could boot from USB, after which it would pickup on the > rpool on the hard drives (probably by lucky chance). > Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not > exists when I encountered the issue) fixed it. > > I would suggest taking the pain now, and reinstall using UEFI, to prevent > such issues in the future. > > > On 9/14/20 12:16 AM, Gianni Milo wrote: > > > > > GRUB does not support all zfs features, so it's quite common for it to > fail > > > to recognise the rpool during boot if it has a feature which is > > > incompatible with it. In your case, I believe that is the "encyption" > > > feature. > > > Do you recall the issue starting after enabling encryption on > rpool/data > > > dataset? If so, you may have to rebuild the pool and leave rpool/data > > > unencrypted. > > > Note that even though you enabled encryption only on rpool/data, the > > > feature will take effect at the pool level, hence the GRUB issue you > > > are facing. > > > Because of this issue, people have started using a separate boot pool > > > (bpool), with limited zfs features enabled and a different data pool > > > (rpool) for the OS and the data. I believe that the PVE installer > should be > > > modified to follow this approach (if it hasn't done it already) to > overcome > > > similar issues. > > > Gianni > > > On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg > sleemburg@it-functions.nl > > > wrote: > > > > > > > Hi All, > > > > I have a proxmox system at netcup that was clean installed about two > > > > weeks ago. > > > > It is full zfs, so root on zfs. I am migrating from one netcup system > > > > with less storage to this system. > > > > This system is using the pve-no-subscription repository, but after > > > > migration I will move the subscription from the 'old' system to this > > > > system. > > > > The rpool/data dataset is zfs encrypted. > > > > Today I did zfs send/recv from the 'old' system to this system to the > > > > rpool/data dataset. > > > > After that I did an apt update and noticed there where updates > available. > > > > After the upgrade and the mandatory reboot, the system does not come > up > > > > anymore. It is stuck in grub rescue. > > > > Grub mentions that it has a 'unknown filesystem'. > > > > Has anyone else experienced this same situation? > > > > If so and you could recover, what was the reason and fix? > > > > I am still researching what is causing this. If I boot a .iso then I > can > > > > import the pool and see all datasets and subvolumes. > > > > So, it seems that the zpool itself and the datasets are ok, it is > just > > > > that grub is unable to recognize them for some reason. > > > > I have read about other situations like this where large_dnode > seemed to > > > > be the cause. I noticed that on the zpool large_dnode is enabled. > And it > > > > is a creation only setting. It cannot be changed to disabled > afterwards. > > > > This must have been done by the Proxmox installer. As booting is > about > > > > the root filesystem, I guess the zfs send / recv to the rpool/data > > > > dataset would have nothing to do with it, but I could be wrong. > > > > I will continue my research tomorrow evening after some other > > > > obligations, but if anyone has an idea, please share it. If it is > just > > > > how to get more debugging info out of the zfs module in grub. > > > > Because 'unknown filesystem', with the zfs module loaded is kind of > not > > > > helping enough.. > > > > Best regards, > > > > Stephan > > > > > > > > pve-user mailing list > > > > pve-user@lists.proxmox.com > > > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > pve-user mailing list > > > pve-user@lists.proxmox.com > > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > ---------- Forwarded message ---------- > From: Arjen via pve-user <pve-user@lists.proxmox.com> > To: Proxmox VE user list <pve-user@lists.proxmox.com> > Cc: Arjen <leesteken@protonmail.ch> > Bcc: > Date: Mon, 14 Sep 2020 08:04:03 +0000 > Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PVE-User] zfs root after last update grub unknown filesystem 2020-09-14 9:29 ` Gianni Milo @ 2020-09-14 9:34 ` Aaron Lauterer 2020-09-14 10:09 ` Gianni Milo 2020-09-15 6:02 ` Stephan Leemburg 1 sibling, 1 reply; 7+ messages in thread From: Aaron Lauterer @ 2020-09-14 9:34 UTC (permalink / raw) To: pve-user On 9/14/20 11:29 AM, Gianni Milo wrote: > UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool > for booting the system (it uses a separate VFAT partition for the > bootloader). > However, this method does not solve another problem which is the redundancy > of the bootloader, as the UEFI partition can be installed only on a single > (boot) drive. If that drive fails, then one must rebuild the UEFI partition > on a different or on the replacement drive. Have you read the documentation [0]? Especially chapter 3.11.4? We added and the pve-efiboot-tool and hooked it into the update-initramfs to keep the EFI partition up to date on each disk used for the root pool so that if one disk fails, you can still boot from one of the remaining ones. The documentation on how to replace a failed boot drive takes into account on how to get the bootloader set up as well. [0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot > > On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user <pve-user@lists.proxmox.com> > wrote: > >> >> >> >> ---------- Forwarded message ---------- >> From: Arjen <leesteken@protonmail.ch> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < >> a.lauterer@proxmox.com> wrote: >> >>> Alternatively you can try to install the machine in UEFI mode. In UEFI >> mode with root ZFS the installer will set up systemd-boot instead of grub. >> This alleviates the problem with grubs limited ZFS compatibility. >>> >>> See the docs for more infos: >> https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot >>> >>> HTH, >>> Aaron >> >> I had the same problem with a zpool feature that could not be turned off. >> I used this work-around for a while in the hope that the bug would be fixed >> (GRUB patch existed for 5 years but it was never tested/reviewed): >> Put a new Proxmox VE installation on USB and copy /boot from the hard >> drives to the USB installation. >> With that, it could boot from USB, after which it would pickup on the >> rpool on the hard drives (probably by lucky chance). >> Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not >> exists when I encountered the issue) fixed it. >> >> I would suggest taking the pain now, and reinstall using UEFI, to prevent >> such issues in the future. >> >>> On 9/14/20 12:16 AM, Gianni Milo wrote: >>> >>>> GRUB does not support all zfs features, so it's quite common for it to >> fail >>>> to recognise the rpool during boot if it has a feature which is >>>> incompatible with it. In your case, I believe that is the "encyption" >>>> feature. >>>> Do you recall the issue starting after enabling encryption on >> rpool/data >>>> dataset? If so, you may have to rebuild the pool and leave rpool/data >>>> unencrypted. >>>> Note that even though you enabled encryption only on rpool/data, the >>>> feature will take effect at the pool level, hence the GRUB issue you >>>> are facing. >>>> Because of this issue, people have started using a separate boot pool >>>> (bpool), with limited zfs features enabled and a different data pool >>>> (rpool) for the OS and the data. I believe that the PVE installer >> should be >>>> modified to follow this approach (if it hasn't done it already) to >> overcome >>>> similar issues. >>>> Gianni >>>> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg >> sleemburg@it-functions.nl >>>> wrote: >>>> >>>>> Hi All, >>>>> I have a proxmox system at netcup that was clean installed about two >>>>> weeks ago. >>>>> It is full zfs, so root on zfs. I am migrating from one netcup system >>>>> with less storage to this system. >>>>> This system is using the pve-no-subscription repository, but after >>>>> migration I will move the subscription from the 'old' system to this >>>>> system. >>>>> The rpool/data dataset is zfs encrypted. >>>>> Today I did zfs send/recv from the 'old' system to this system to the >>>>> rpool/data dataset. >>>>> After that I did an apt update and noticed there where updates >> available. >>>>> After the upgrade and the mandatory reboot, the system does not come >> up >>>>> anymore. It is stuck in grub rescue. >>>>> Grub mentions that it has a 'unknown filesystem'. >>>>> Has anyone else experienced this same situation? >>>>> If so and you could recover, what was the reason and fix? >>>>> I am still researching what is causing this. If I boot a .iso then I >> can >>>>> import the pool and see all datasets and subvolumes. >>>>> So, it seems that the zpool itself and the datasets are ok, it is >> just >>>>> that grub is unable to recognize them for some reason. >>>>> I have read about other situations like this where large_dnode >> seemed to >>>>> be the cause. I noticed that on the zpool large_dnode is enabled. >> And it >>>>> is a creation only setting. It cannot be changed to disabled >> afterwards. >>>>> This must have been done by the Proxmox installer. As booting is >> about >>>>> the root filesystem, I guess the zfs send / recv to the rpool/data >>>>> dataset would have nothing to do with it, but I could be wrong. >>>>> I will continue my research tomorrow evening after some other >>>>> obligations, but if anyone has an idea, please share it. If it is >> just >>>>> how to get more debugging info out of the zfs module in grub. >>>>> Because 'unknown filesystem', with the zfs module loaded is kind of >> not >>>>> helping enough.. >>>>> Best regards, >>>>> Stephan >>>>> >>>>> pve-user mailing list >>>>> pve-user@lists.proxmox.com >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>>> pve-user mailing list >>>> pve-user@lists.proxmox.com >>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >>> pve-user mailing list >>> pve-user@lists.proxmox.com >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> >> >> >> ---------- Forwarded message ---------- >> From: Arjen via pve-user <pve-user@lists.proxmox.com> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: Arjen <leesteken@protonmail.ch> >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PVE-User] zfs root after last update grub unknown filesystem 2020-09-14 9:34 ` Aaron Lauterer @ 2020-09-14 10:09 ` Gianni Milo 0 siblings, 0 replies; 7+ messages in thread From: Gianni Milo @ 2020-09-14 10:09 UTC (permalink / raw) To: Proxmox VE user list I was not aware of that tool as I'm not using UEFI boot mode at the moment. It looks like UEFI mode is the way to go from now on for root on zfs setups... Thanks, G. On Mon, 14 Sep 2020 at 10:34, Aaron Lauterer <a.lauterer@proxmox.com> wrote: > > > On 9/14/20 11:29 AM, Gianni Milo wrote: > > UEFI boot mode is indeed a good option, as it does not rely on the ZFS > pool > > for booting the system (it uses a separate VFAT partition for the > > bootloader). > > However, this method does not solve another problem which is the > redundancy > > of the bootloader, as the UEFI partition can be installed only on a > single > > (boot) drive. If that drive fails, then one must rebuild the UEFI > partition > > on a different or on the replacement drive. > > Have you read the documentation [0]? Especially chapter 3.11.4? > > We added and the pve-efiboot-tool and hooked it into the update-initramfs > to keep the EFI partition up to date on each disk used for the root pool so > that if one disk fails, you can still boot from one of the remaining ones. > The documentation on how to replace a failed boot drive takes into account > on how to get the bootloader set up as well. > > [0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot > > > > > On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user < > pve-user@lists.proxmox.com> > > wrote: > > > >> > >> > >> > >> ---------- Forwarded message ---------- > >> From: Arjen <leesteken@protonmail.ch> > >> To: Proxmox VE user list <pve-user@lists.proxmox.com> > >> Cc: > >> Bcc: > >> Date: Mon, 14 Sep 2020 08:04:03 +0000 > >> Subject: Re: [PVE-User] zfs root after last update grub unknown > filesystem > >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >> On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < > >> a.lauterer@proxmox.com> wrote: > >> > >>> Alternatively you can try to install the machine in UEFI mode. In UEFI > >> mode with root ZFS the installer will set up systemd-boot instead of > grub. > >> This alleviates the problem with grubs limited ZFS compatibility. > >>> > >>> See the docs for more infos: > >> https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot > >>> > >>> HTH, > >>> Aaron > >> > >> I had the same problem with a zpool feature that could not be turned > off. > >> I used this work-around for a while in the hope that the bug would be > fixed > >> (GRUB patch existed for 5 years but it was never tested/reviewed): > >> Put a new Proxmox VE installation on USB and copy /boot from the hard > >> drives to the USB installation. > >> With that, it could boot from USB, after which it would pickup on the > >> rpool on the hard drives (probably by lucky chance). > >> Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not > >> exists when I encountered the issue) fixed it. > >> > >> I would suggest taking the pain now, and reinstall using UEFI, to > prevent > >> such issues in the future. > >> > >>> On 9/14/20 12:16 AM, Gianni Milo wrote: > >>> > >>>> GRUB does not support all zfs features, so it's quite common for it to > >> fail > >>>> to recognise the rpool during boot if it has a feature which is > >>>> incompatible with it. In your case, I believe that is the "encyption" > >>>> feature. > >>>> Do you recall the issue starting after enabling encryption on > >> rpool/data > >>>> dataset? If so, you may have to rebuild the pool and leave rpool/data > >>>> unencrypted. > >>>> Note that even though you enabled encryption only on rpool/data, the > >>>> feature will take effect at the pool level, hence the GRUB issue you > >>>> are facing. > >>>> Because of this issue, people have started using a separate boot pool > >>>> (bpool), with limited zfs features enabled and a different data pool > >>>> (rpool) for the OS and the data. I believe that the PVE installer > >> should be > >>>> modified to follow this approach (if it hasn't done it already) to > >> overcome > >>>> similar issues. > >>>> Gianni > >>>> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg > >> sleemburg@it-functions.nl > >>>> wrote: > >>>> > >>>>> Hi All, > >>>>> I have a proxmox system at netcup that was clean installed about two > >>>>> weeks ago. > >>>>> It is full zfs, so root on zfs. I am migrating from one netcup system > >>>>> with less storage to this system. > >>>>> This system is using the pve-no-subscription repository, but after > >>>>> migration I will move the subscription from the 'old' system to this > >>>>> system. > >>>>> The rpool/data dataset is zfs encrypted. > >>>>> Today I did zfs send/recv from the 'old' system to this system to the > >>>>> rpool/data dataset. > >>>>> After that I did an apt update and noticed there where updates > >> available. > >>>>> After the upgrade and the mandatory reboot, the system does not come > >> up > >>>>> anymore. It is stuck in grub rescue. > >>>>> Grub mentions that it has a 'unknown filesystem'. > >>>>> Has anyone else experienced this same situation? > >>>>> If so and you could recover, what was the reason and fix? > >>>>> I am still researching what is causing this. If I boot a .iso then I > >> can > >>>>> import the pool and see all datasets and subvolumes. > >>>>> So, it seems that the zpool itself and the datasets are ok, it is > >> just > >>>>> that grub is unable to recognize them for some reason. > >>>>> I have read about other situations like this where large_dnode > >> seemed to > >>>>> be the cause. I noticed that on the zpool large_dnode is enabled. > >> And it > >>>>> is a creation only setting. It cannot be changed to disabled > >> afterwards. > >>>>> This must have been done by the Proxmox installer. As booting is > >> about > >>>>> the root filesystem, I guess the zfs send / recv to the rpool/data > >>>>> dataset would have nothing to do with it, but I could be wrong. > >>>>> I will continue my research tomorrow evening after some other > >>>>> obligations, but if anyone has an idea, please share it. If it is > >> just > >>>>> how to get more debugging info out of the zfs module in grub. > >>>>> Because 'unknown filesystem', with the zfs module loaded is kind of > >> not > >>>>> helping enough.. > >>>>> Best regards, > >>>>> Stephan > >>>>> > >>>>> pve-user mailing list > >>>>> pve-user@lists.proxmox.com > >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>>> > >>>> pve-user mailing list > >>>> pve-user@lists.proxmox.com > >>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >>> pve-user mailing list > >>> pve-user@lists.proxmox.com > >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > >> > >> > >> > >> > >> > >> ---------- Forwarded message ---------- > >> From: Arjen via pve-user <pve-user@lists.proxmox.com> > >> To: Proxmox VE user list <pve-user@lists.proxmox.com> > >> Cc: Arjen <leesteken@protonmail.ch> > >> Bcc: > >> Date: Mon, 14 Sep 2020 08:04:03 +0000 > >> Subject: Re: [PVE-User] zfs root after last update grub unknown > filesystem > >> _______________________________________________ > >> pve-user mailing list > >> pve-user@lists.proxmox.com > >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > _______________________________________________ > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PVE-User] zfs root after last update grub unknown filesystem 2020-09-14 9:29 ` Gianni Milo 2020-09-14 9:34 ` Aaron Lauterer @ 2020-09-15 6:02 ` Stephan Leemburg 1 sibling, 0 replies; 7+ messages in thread From: Stephan Leemburg @ 2020-09-15 6:02 UTC (permalink / raw) To: pve-user On 14-09-2020 11:29, Gianni Milo wrote: > UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool > for booting the system (it uses a separate VFAT partition for the > bootloader). Yes, that would be good, but the netcup system does not 'support' UEFI. So, I guess I will take the pain and reinstall the system with a traditional lvm as root and a luks encrypted container with the zpool in it. luks provides more than one keyslot. LVM will do ok for root and works at least with grub. I could access the rpool by booting into the debug installer. So I could make a tar of the rpool/ROOT/pve-1 filesystem. That will make a recovery more easy, without loosing too much work. BTW Also the 'rescue mode' of the Proxmox installer barks that there is 'no rpool'. Must be related. > However, this method does not solve another problem which is the redundancy > of the bootloader, as the UEFI partition can be installed only on a single > (boot) drive. If that drive fails, then one must rebuild the UEFI partition > on a different or on the replacement drive. > > On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user <pve-user@lists.proxmox.com> > wrote: > >> >> >> ---------- Forwarded message ---------- >> From: Arjen <leesteken@protonmail.ch> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < >> a.lauterer@proxmox.com> wrote: >> >>> Alternatively you can try to install the machine in UEFI mode. In UEFI >> mode with root ZFS the installer will set up systemd-boot instead of grub. >> This alleviates the problem with grubs limited ZFS compatibility. >>> See the docs for more infos: >> https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot >>> HTH, >>> Aaron >> I had the same problem with a zpool feature that could not be turned off. >> I used this work-around for a while in the hope that the bug would be fixed >> (GRUB patch existed for 5 years but it was never tested/reviewed): >> Put a new Proxmox VE installation on USB and copy /boot from the hard >> drives to the USB installation. >> With that, it could boot from USB, after which it would pickup on the >> rpool on the hard drives (probably by lucky chance). >> Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not >> exists when I encountered the issue) fixed it. >> >> I would suggest taking the pain now, and reinstall using UEFI, to prevent >> such issues in the future. >> >>> On 9/14/20 12:16 AM, Gianni Milo wrote: >>> >>>> GRUB does not support all zfs features, so it's quite common for it to >> fail >>>> to recognise the rpool during boot if it has a feature which is >>>> incompatible with it. In your case, I believe that is the "encyption" >>>> feature. >>>> Do you recall the issue starting after enabling encryption on >> rpool/data >>>> dataset? If so, you may have to rebuild the pool and leave rpool/data >>>> unencrypted. >>>> Note that even though you enabled encryption only on rpool/data, the >>>> feature will take effect at the pool level, hence the GRUB issue you >>>> are facing. >>>> Because of this issue, people have started using a separate boot pool >>>> (bpool), with limited zfs features enabled and a different data pool >>>> (rpool) for the OS and the data. I believe that the PVE installer >> should be >>>> modified to follow this approach (if it hasn't done it already) to >> overcome >>>> similar issues. >>>> Gianni >>>> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg >> sleemburg@it-functions.nl >>>> wrote: >>>> >>>>> Hi All, >>>>> I have a proxmox system at netcup that was clean installed about two >>>>> weeks ago. >>>>> It is full zfs, so root on zfs. I am migrating from one netcup system >>>>> with less storage to this system. >>>>> This system is using the pve-no-subscription repository, but after >>>>> migration I will move the subscription from the 'old' system to this >>>>> system. >>>>> The rpool/data dataset is zfs encrypted. >>>>> Today I did zfs send/recv from the 'old' system to this system to the >>>>> rpool/data dataset. >>>>> After that I did an apt update and noticed there where updates >> available. >>>>> After the upgrade and the mandatory reboot, the system does not come >> up >>>>> anymore. It is stuck in grub rescue. >>>>> Grub mentions that it has a 'unknown filesystem'. >>>>> Has anyone else experienced this same situation? >>>>> If so and you could recover, what was the reason and fix? >>>>> I am still researching what is causing this. If I boot a .iso then I >> can >>>>> import the pool and see all datasets and subvolumes. >>>>> So, it seems that the zpool itself and the datasets are ok, it is >> just >>>>> that grub is unable to recognize them for some reason. >>>>> I have read about other situations like this where large_dnode >> seemed to >>>>> be the cause. I noticed that on the zpool large_dnode is enabled. >> And it >>>>> is a creation only setting. It cannot be changed to disabled >> afterwards. >>>>> This must have been done by the Proxmox installer. As booting is >> about >>>>> the root filesystem, I guess the zfs send / recv to the rpool/data >>>>> dataset would have nothing to do with it, but I could be wrong. >>>>> I will continue my research tomorrow evening after some other >>>>> obligations, but if anyone has an idea, please share it. If it is >> just >>>>> how to get more debugging info out of the zfs module in grub. >>>>> Because 'unknown filesystem', with the zfs module loaded is kind of >> not >>>>> helping enough.. >>>>> Best regards, >>>>> Stephan >>>>> >>>>> pve-user mailing list >>>>> pve-user@lists.proxmox.com >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> pve-user mailing list >>>> pve-user@lists.proxmox.com >>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> pve-user mailing list >>> pve-user@lists.proxmox.com >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> >> >> ---------- Forwarded message ---------- >> From: Arjen via pve-user <pve-user@lists.proxmox.com> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: Arjen <leesteken@protonmail.ch> >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-09-15 6:03 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-13 21:11 [PVE-User] zfs root after last update grub unknown filesystem Stephan Leemburg
2020-09-13 22:16 ` Gianni Milo
2020-09-14 7:07 ` Aaron Lauterer
[not found] ` <mailman.74.1600071257.452.pve-user@lists.proxmox.com>
2020-09-14 9:29 ` Gianni Milo
2020-09-14 9:34 ` Aaron Lauterer
2020-09-14 10:09 ` Gianni Milo
2020-09-15 6:02 ` Stephan Leemburg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox