From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A75AD61567 for ; Mon, 14 Sep 2020 00:16:33 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 93AF2AF62 for ; Mon, 14 Sep 2020 00:16:33 +0200 (CEST) Received: from mail-yb1-xb29.google.com (mail-yb1-xb29.google.com [IPv6:2607:f8b0:4864:20::b29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 3D2A4AF4D for ; Mon, 14 Sep 2020 00:16:32 +0200 (CEST) Received: by mail-yb1-xb29.google.com with SMTP id v60so4572173ybi.10 for ; Sun, 13 Sep 2020 15:16:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=82gq2JWoidDFElwf1+4I0RfOPAGC536iXXEAZFHfZmA=; b=u5U07rJyQuNRhjjmEfPiPIDjjcF/bKuTG9LPKEDU3oPq0qPd3dMVZYmCZ+LhQ6FAVD uvgVUUtoDt8xrYcUmhCDoqi1ZUuP0XcIZE62Ia79Vmj+PL4IgafivS2fIMCEsxJtjQ+u JBRzoeN6Cd2+GUA18K+DHLTvkM9ZNDgqqHeTOLeRhy/W7O9HquCrGfAxzN1OUlsaW7Kx iNYnQpO5CaN6gheX51nOmi4V0OhiFfxMgPOgy5fTGHiE4plvpWSBJUc0STTs6hlIyUUl 6lsEaQaZuIq1CK/Fn4ncV3+bLqmP0e04DIGKDVhMPu3JwTpZLjMfAulR4MXMVg0qkz7v WPOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=82gq2JWoidDFElwf1+4I0RfOPAGC536iXXEAZFHfZmA=; b=cWDjGnD4E+eXvsWH2BKDAE171tZxwzY8lyRcfZhOy1nMa+GidUpcPzv2aggr8mF9f8 UQz1u0auWQhZz7KzMhr44FbsChydQMA5pXxFoCCtXPpZ7BAL5BBW3Vzo0/72jSmoWOM4 p+M4AFCXK7O4dcyRcFdnLxVQyK66xbKHV8q2ocGWMAvC2QAuS8LUr8FotWGe5soApp1a uu1sQ9GE1ggt66qKezzcVlFXvd+9eTJ7pyDE/jl9vKdfkYM1GgEc3qBQljcADyc4GFrB YNZL9XmjvuLh5IMSgJvLB/0A24RCzeU6wZczMCZFMmKX0upVwDH4dnQmgdMB050f216A F9kw== X-Gm-Message-State: AOAM5318NoemsG2/RP+HFxnLNnILXUaCpclGSeT5NYxzU4ynqeeAdDBY vbx3VPh+OfAqfHda8n08VDqMDkpkTpCAI7GaO5S1n6Ve X-Google-Smtp-Source: ABdhPJziLNmkSlLrdVpxyqKaa4qBTobMB4326EAuBVULGR/QaMUZ9R0gAILfb1J61oi0AgwiZhp21c00Tvbi0bwROso= X-Received: by 2002:a5b:3d2:: with SMTP id t18mr15276566ybp.333.1600035384491; Sun, 13 Sep 2020 15:16:24 -0700 (PDT) MIME-Version: 1.0 References: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl> In-Reply-To: <35ff9a9f-60ba-5588-b5f7-d5c9b7483c84@it-functions.nl> From: Gianni Milo Date: Sun, 13 Sep 2020 23:16:13 +0100 Message-ID: To: Proxmox VE user list X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] zfs root after last update grub unknown filesystem X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Sep 2020 22:16:33 -0000 GRUB does not support all zfs features, so it's quite common for it to fail to recognise the rpool during boot if it has a feature which is incompatible with it. In your case, I believe that is the "encyption" feature. Do you recall the issue starting after enabling encryption on rpool/data dataset? If so, you may have to rebuild the pool and leave rpool/data unencrypted. Note that even though you enabled encryption only on rpool/data, the feature will take effect at the pool level, hence the GRUB issue you are facing. Because of this issue, people have started using a separate boot pool (bpool), with limited zfs features enabled and a different data pool (rpool) for the OS and the data. I believe that the PVE installer should be modified to follow this approach (if it hasn't done it already) to overcome similar issues. Gianni On Sun, 13 Sep 2020 at 22:29, Stephan Leemburg wrote: > Hi All, > > > > I have a proxmox system at netcup that was clean installed about two > > weeks ago. > > > > It is full zfs, so root on zfs. I am migrating from one netcup system > > with less storage to this system. > > > > This system is using the pve-no-subscription repository, but after > > migration I will move the subscription from the 'old' system to this > system. > > > > The rpool/data dataset is zfs encrypted. > > > > Today I did zfs send/recv from the 'old' system to this system to the > > rpool/data dataset. > > > > After that I did an apt update and noticed there where updates available. > > > > After the upgrade and the mandatory reboot, the system does not come up > > anymore. It is stuck in grub rescue. > > > > Grub mentions that it has a 'unknown filesystem'. > > > > Has anyone else experienced this same situation? > > > > If so and you could recover, what was the reason and fix? > > > > I am still researching what is causing this. If I boot a .iso then I can > > import the pool and see all datasets and subvolumes. > > > > So, it seems that the zpool itself and the datasets are ok, it is just > > that grub is unable to recognize them for some reason. > > > > I have read about other situations like this where large_dnode seemed to > > be the cause. I noticed that on the zpool large_dnode is enabled. And it > > is a creation only setting. It cannot be changed to disabled afterwards. > > > > This must have been done by the Proxmox installer. As booting is about > > the root filesystem, I guess the zfs send / recv to the rpool/data > > dataset would have nothing to do with it, but I could be wrong. > > > > I will continue my research tomorrow evening after some other > > obligations, but if anyone has an idea, please share it. If it is just > > how to get more debugging info out of the zfs module in grub. > > > > Because 'unknown filesystem', with the zfs module loaded is kind of not > > helping enough.. > > > > Best regards, > > > > Stephan > > > > > > _______________________________________________ > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > >