From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 53B68616EA for ; Mon, 14 Sep 2020 11:29:55 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 44500F3A4 for ; Mon, 14 Sep 2020 11:29:25 +0200 (CEST) Received: from mail-yb1-xb33.google.com (mail-yb1-xb33.google.com [IPv6:2607:f8b0:4864:20::b33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 53008F38F for ; Mon, 14 Sep 2020 11:29:23 +0200 (CEST) Received: by mail-yb1-xb33.google.com with SMTP id h206so11324366ybc.11 for ; Mon, 14 Sep 2020 02:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=dBWMcAgQ2JUyFHHbTUoxbDCCm1CgFyMy9WNP53gMBa4=; b=D//HRHZYbslBs43VlpOX/PoO6sGZnEbrtXY0g8s8XwyinmQn0Z/aUifce7WKl+p+o7 nrjNDv2NNY4DeJHMV+Qh0hwFoImtXStRAhIxT+sIEU05uJf3wraCx9v7Lqco0kLCeugP j8RrGN1X8Rr45VIeKcKINv1l8bHtVxCasJYyAQLLOCeL9do1n8TjM8V8m7NeqsNPkHtO bbwwsD+HTNlPkr2tOSyFJCk3TWEZVmEI3B4BdwvAkto9yQcxRL/jW6U1qEedOHF+o/mx qG7bcnHOkuWIC0ipgMRpKiMVAFzKzxgHTsoUCOyBE1Py6iAH3m9zn349EHn2FfcnLXqS 2hkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=dBWMcAgQ2JUyFHHbTUoxbDCCm1CgFyMy9WNP53gMBa4=; b=NfftuDGWIToa1cxcyNyEmLABvLv9Hv2DkoWq11RRkKDcO8+E7jTEYyWIGSfeyVrG4t +IcPI5YHA4tB/AFuzzW4ZRkNDdC5Qq72gehpQj8rrLIgevNG/JjW8VGfZo/9pljCClaC kMufBhhwzoNNLN1gF2x0sImsaY03p/JcOkBs1hGYOihO6MVppzXeC/r51O01dGvqp2wN uSkUGs6/fOFQ4uAhcnuYsDMBJRyBMoy1BaJ6FQBLSytRH6wd5gnLvbHoWqsXzPcffx0j k1R10QQud+5d4SjGPaSxc9ZIuFWVuJ4w5s/kb1ACnQpU6PZX2JNi/dHekk6++QNOQ+EC f4wQ== X-Gm-Message-State: AOAM5317TaNwH0gxq0MMv5QOhJ1CbGtaPPqk6gxJ5ZNreNjZ4g4K7Jqk RWtleY7rKLRbyiSfx4SuqAFSRcv8Ruej5WYOUCgumuWS X-Google-Smtp-Source: ABdhPJxFIDZ58DVnZ7M1HkVVBPJ7Tj328TYTY/1vE1ntTn7u4jFXLWgiPFJX4b7jHf9jELhPSurQ/7SHohSQ6Sfd8lc= X-Received: by 2002:a25:cf54:: with SMTP id f81mr20372747ybg.227.1600075755447; Mon, 14 Sep 2020 02:29:15 -0700 (PDT) MIME-Version: 1.0 References: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl> <4c25b97a-507c-5ff9-effd-c04d39e848dc@proxmox.com> In-Reply-To: From: Gianni Milo Date: Mon, 14 Sep 2020 10:29:03 +0100 Message-ID: To: Proxmox VE user list X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2020 09:29:55 -0000 UEFI boot mode is indeed a good option, as it does not rely on the ZFS pool for booting the system (it uses a separate VFAT partition for the bootloader). However, this method does not solve another problem which is the redundancy of the bootloader, as the UEFI partition can be installed only on a single (boot) drive. If that drive fails, then one must rebuild the UEFI partition on a different or on the replacement drive. On Mon, 14 Sep 2020 at 09:14, Arjen via pve-user wrote: > > > > ---------- Forwarded message ---------- > From: Arjen > To: Proxmox VE user list > Cc: > Bcc: > Date: Mon, 14 Sep 2020 08:04:03 +0000 > Subject: Re: [PVE-User] zfs root after last update grub unknown filesyste= m > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original = Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 > On Monday, September 14, 2020 9:07 AM, Aaron Lauterer < > a.lauterer@proxmox.com> wrote: > > > Alternatively you can try to install the machine in UEFI mode. In UEFI > mode with root ZFS the installer will set up systemd-boot instead of grub= . > This alleviates the problem with grubs limited ZFS compatibility. > > > > See the docs for more infos: > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot > > > > HTH, > > Aaron > > I had the same problem with a zpool feature that could not be turned off. > I used this work-around for a while in the hope that the bug would be fix= ed > (GRUB patch existed for 5 years but it was never tested/reviewed): > Put a new Proxmox VE installation on USB and copy /boot from the hard > drives to the USB installation. > With that, it could boot from USB, after which it would pickup on the > rpool on the hard drives (probably by lucky chance). > Eventually, reinstalling with UEFI+GPT+ESP+systemd-boot (which did not > exists when I encountered the issue) fixed it. > > I would suggest taking the pain now, and reinstall using UEFI, to prevent > such issues in the future. > > > On 9/14/20 12:16 AM, Gianni Milo wrote: > > > > > GRUB does not support all zfs features, so it's quite common for it t= o > fail > > > to recognise the rpool during boot if it has a feature which is > > > incompatible with it. In your case, I believe that is the "encyption" > > > feature. > > > Do you recall the issue starting after enabling encryption on > rpool/data > > > dataset? If so, you may have to rebuild the pool and leave rpool/data > > > unencrypted. > > > Note that even though you enabled encryption only on rpool/data, the > > > feature will take effect at the pool level, hence the GRUB issue you > > > are facing. > > > Because of this issue, people have started using a separate boot pool > > > (bpool), with limited zfs features enabled and a different data pool > > > (rpool) for the OS and the data. I believe that the PVE installer > should be > > > modified to follow this approach (if it hasn't done it already) to > overcome > > > similar issues. > > > Gianni > > > On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg > sleemburg@it-functions.nl > > > wrote: > > > > > > > Hi All, > > > > I have a proxmox system at netcup that was clean installed about tw= o > > > > weeks ago. > > > > It is full zfs, so root on zfs. I am migrating from one netcup syst= em > > > > with less storage to this system. > > > > This system is using the pve-no-subscription repository, but after > > > > migration I will move the subscription from the 'old' system to thi= s > > > > system. > > > > The rpool/data dataset is zfs encrypted. > > > > Today I did zfs send/recv from the 'old' system to this system to t= he > > > > rpool/data dataset. > > > > After that I did an apt update and noticed there where updates > available. > > > > After the upgrade and the mandatory reboot, the system does not com= e > up > > > > anymore. It is stuck in grub rescue. > > > > Grub mentions that it has a 'unknown filesystem'. > > > > Has anyone else experienced this same situation? > > > > If so and you could recover, what was the reason and fix? > > > > I am still researching what is causing this. If I boot a .iso then = I > can > > > > import the pool and see all datasets and subvolumes. > > > > So, it seems that the zpool itself and the datasets are ok, it is > just > > > > that grub is unable to recognize them for some reason. > > > > I have read about other situations like this where large_dnode > seemed to > > > > be the cause. I noticed that on the zpool large_dnode is enabled. > And it > > > > is a creation only setting. It cannot be changed to disabled > afterwards. > > > > This must have been done by the Proxmox installer. As booting is > about > > > > the root filesystem, I guess the zfs send / recv to the rpool/data > > > > dataset would have nothing to do with it, but I could be wrong. > > > > I will continue my research tomorrow evening after some other > > > > obligations, but if anyone has an idea, please share it. If it is > just > > > > how to get more debugging info out of the zfs module in grub. > > > > Because 'unknown filesystem', with the zfs module loaded is kind of > not > > > > helping enough.. > > > > Best regards, > > > > Stephan > > > > > > > > pve-user mailing list > > > > pve-user@lists.proxmox.com > > > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > pve-user mailing list > > > pve-user@lists.proxmox.com > > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > ---------- Forwarded message ---------- > From: Arjen via pve-user > To: Proxmox VE user list > Cc: Arjen > Bcc: > Date: Mon, 14 Sep 2020 08:04:03 +0000 > Subject: Re: [PVE-User] zfs root after last update grub unknown filesyste= m > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >