From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <sleemburg@it-functions.nl> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C525E6196C for <pve-user@lists.proxmox.com>; Tue, 15 Sep 2020 08:03:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id AF9A816901 for <pve-user@lists.proxmox.com>; Tue, 15 Sep 2020 08:03:07 +0200 (CEST) Received: from mx0.it-functions.nl (mx0.it-functions.nl [178.32.167.210]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 8FD07168F5 for <pve-user@lists.proxmox.com>; Tue, 15 Sep 2020 08:03:05 +0200 (CEST) Received: from [217.100.26.194] (helo=daruma-old.hachimitsu.nl) by mx0.it-functions.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from <sleemburg@it-functions.nl>) id 1kI43P-0007Vb-0p for pve-user@lists.proxmox.com; Tue, 15 Sep 2020 08:02:59 +0200 Received: from [192.168.254.32] by daruma-old.hachimitsu.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from <sleemburg@it-functions.nl>) id 1kI43M-000580-Hx for pve-user@lists.proxmox.com; Tue, 15 Sep 2020 08:02:56 +0200 To: pve-user@lists.proxmox.com References: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl> <CACzVk9WshXO9QE8Dj9iorNUEAeHiJway+gnKFcuwg12cKm0HDA@mail.gmail.com> <4c25b97a-507c-5ff9-effd-c04d39e848dc@proxmox.com> <mailman.74.1600071257.452.pve-user@lists.proxmox.com> <CACzVk9VxYKJkhoLmUcKa5P4=HFJ3H6vfUgeAt-y7OHTkLRM6GQ@mail.gmail.com> From: Stephan Leemburg <sleemburg@it-functions.nl> Organization: IT Functions Message-ID: <b0b76d70-64c0-af9f-363c-e73e665686f0@it-functions.nl> Date: Tue, 15 Sep 2020 08:02:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <CACzVk9VxYKJkhoLmUcKa5P4=HFJ3H6vfUgeAt-y7OHTkLRM6GQ@mail.gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Scan-Signature: 6ba56dcfdcfb58dc83f9584da45161e4 X-GeoIP: NL X-Virus-Scanned: by clamav-new X-Scan-Signature: 6071fef9992c5ac79f451c33ea78e251 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.941 Adjusted score from AWL reputation of From: address KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Tue, 15 Sep 2020 06:03:37 -0000 On 14-09-2020 11:29, Gianni Milo wrote: > UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool > for booting the system (it uses a separate VFAT partition for the > bootloader). Yes, that would be good, but the netcup system does not 'support' UEFI. So, I guess I will take the pain and reinstall the system with a traditional lvm as root and a luks encrypted container with the zpool in it. luks provides more than one keyslot. LVM will do ok for root and works at least with grub. I could access the rpool by booting into the debug installer. So I could make a tar of the rpool/ROOT/pve-1 filesystem. That will make a recovery more easy, without loosing too much work. BTW Also the 'rescue mode' of the Proxmox installer barks that there is 'no rpool'. Must be related. > However, this method does not solve another problem which is the redundancy > of the bootloader, as the UEFI partition can be installed only on a single > (boot) drive. If that drive fails, then one must rebuild the UEFI partition > on a different or on the replacement drive. > > On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user <pve-user@lists.proxmox.com> > wrote: > >> >> >> ---------- Forwarded message ---------- >> From: Arjen <leesteken@protonmail.ch> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < >> a.lauterer@proxmox.com> wrote: >> >>> Alternatively you can try to install the machine in UEFI mode. In UEFI >> mode with root ZFS the installer will set up systemd-boot instead of grub. >> This alleviates the problem with grubs limited ZFS compatibility. >>> See the docs for more infos: >> https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot >>> HTH, >>> Aaron >> I had the same problem with a zpool feature that could not be turned off. >> I used this work-around for a while in the hope that the bug would be fixed >> (GRUB patch existed for 5 years but it was never tested/reviewed): >> Put a new Proxmox VE installation on USB and copy /boot from the hard >> drives to the USB installation. >> With that, it could boot from USB, after which it would pickup on the >> rpool on the hard drives (probably by lucky chance). >> Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not >> exists when I encountered the issue) fixed it. >> >> I would suggest taking the pain now, and reinstall using UEFI, to prevent >> such issues in the future. >> >>> On 9/14/20 12:16 AM, Gianni Milo wrote: >>> >>>> GRUB does not support all zfs features, so it's quite common for it to >> fail >>>> to recognise the rpool during boot if it has a feature which is >>>> incompatible with it. In your case, I believe that is the "encyption" >>>> feature. >>>> Do you recall the issue starting after enabling encryption on >> rpool/data >>>> dataset? If so, you may have to rebuild the pool and leave rpool/data >>>> unencrypted. >>>> Note that even though you enabled encryption only on rpool/data, the >>>> feature will take effect at the pool level, hence the GRUB issue you >>>> are facing. >>>> Because of this issue, people have started using a separate boot pool >>>> (bpool), with limited zfs features enabled and a different data pool >>>> (rpool) for the OS and the data. I believe that the PVE installer >> should be >>>> modified to follow this approach (if it hasn't done it already) to >> overcome >>>> similar issues. >>>> Gianni >>>> On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg >> sleemburg@it-functions.nl >>>> wrote: >>>> >>>>> Hi All, >>>>> I have a proxmox system at netcup that was clean installed about two >>>>> weeks ago. >>>>> It is full zfs, so root on zfs. I am migrating from one netcup system >>>>> with less storage to this system. >>>>> This system is using the pve-no-subscription repository, but after >>>>> migration I will move the subscription from the 'old' system to this >>>>> system. >>>>> The rpool/data dataset is zfs encrypted. >>>>> Today I did zfs send/recv from the 'old' system to this system to the >>>>> rpool/data dataset. >>>>> After that I did an apt update and noticed there where updates >> available. >>>>> After the upgrade and the mandatory reboot, the system does not come >> up >>>>> anymore. It is stuck in grub rescue. >>>>> Grub mentions that it has a 'unknown filesystem'. >>>>> Has anyone else experienced this same situation? >>>>> If so and you could recover, what was the reason and fix? >>>>> I am still researching what is causing this. If I boot a .iso then I >> can >>>>> import the pool and see all datasets and subvolumes. >>>>> So, it seems that the zpool itself and the datasets are ok, it is >> just >>>>> that grub is unable to recognize them for some reason. >>>>> I have read about other situations like this where large_dnode >> seemed to >>>>> be the cause. I noticed that on the zpool large_dnode is enabled. >> And it >>>>> is a creation only setting. It cannot be changed to disabled >> afterwards. >>>>> This must have been done by the Proxmox installer. As booting is >> about >>>>> the root filesystem, I guess the zfs send / recv to the rpool/data >>>>> dataset would have nothing to do with it, but I could be wrong. >>>>> I will continue my research tomorrow evening after some other >>>>> obligations, but if anyone has an idea, please share it. If it is >> just >>>>> how to get more debugging info out of the zfs module in grub. >>>>> Because 'unknown filesystem', with the zfs module loaded is kind of >> not >>>>> helping enough.. >>>>> Best regards, >>>>> Stephan >>>>> >>>>> pve-user mailing list >>>>> pve-user@lists.proxmox.com >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> pve-user mailing list >>>> pve-user@lists.proxmox.com >>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> pve-user mailing list >>> pve-user@lists.proxmox.com >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> >> >> ---------- Forwarded message ---------- >> From: Arjen via pve-user <pve-user@lists.proxmox.com> >> To: Proxmox VE user list <pve-user@lists.proxmox.com> >> Cc: Arjen <leesteken@protonmail.ch> >> Bcc: >> Date: Mon, 14 Sep 2020 08:04:03 +0000 >> Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user