[pve-devel] [RFC pve-kernel-meta 0/2] boot ZFS on legacy BIOS systems from vfat

[pve-devel] [RFC pve-kernel-meta 0/2] boot ZFS on legacy BIOS systems from vfat

[Stoiko Ivanov]

This patchset has been long overdue, and complements the solution to booting
ZFS on UEFI systems using systemd-boot.

With the upgrade of ZFS 2.0.0 (and it's support for ZSTD compression), quite
a few users found out that their systems were still booted with legacy bios
boot and were consequently rendered unbootable with enabling zstd
compression on (a dataset on rpool).

The solution is inspired by our community-forum, especially @avw, and seems
rather lightweight (patch 2/2 is best viewed with '-w').
My first approach was to generate a working grub-config ourselves, but
given that grub has a few years of handling special cases - bind-mounting
the ESP on /boot and running 'update-grub' seems like a less painful way.

* patch 1/2 renames pve-efiboot-tool to proxmox-boot-tool (which seems more
appropriate by now)
* patch 2/2 adds support for installing grub appropriately on the ESPs
  and running the kernel sync-logic in a way that update-grub feels fine
  with

Sending as RFC, because this is a proof-of-concept and missing quite a few
things.

What works:
* installing this version on a root ZFS RAID-Z@ PVE (based on an old pre 6.2
  install)
* reformatting all 4 ESPs `proxmox-boot-tool format /dev/sda2 --force)
* initializing them
* rebooting into 5.4.106 and zfs 2.0.0
* upgrading the pool, setting compression=zstd, writing a file, rebooting
  (successfully)
* rebooting into an old 5.3 kernel - and getting greeted by busy-box instead
  of grub-rescue

What's missing (at least):
* support in the installer
* the renaming is not quite through (the kernel-hooks are still containing
  pve/efi in their name)
* testing the removal part of the kernel-sync


Stoiko Ivanov (2):
  proxmox-boot-tool: rename from pve-efiboot-tool
  proxmox-boot-tool: handle legacy boot zfs installs

 Makefile                                    |  2 +-
 bin/Makefile                                |  2 +-
 bin/{pve-efiboot-tool => proxmox-boot-tool} | 21 ++++--
 debian/pve-kernel-helper.install            |  2 +-
 debian/pve-kernel-helper.links              |  1 +
 {efiboot => proxmox-boot}/Makefile          |  0
 {efiboot => proxmox-boot}/functions         |  0
 {efiboot => proxmox-boot}/pve-auto-removal  |  0
 {efiboot => proxmox-boot}/pve-efiboot-sync  |  2 +-
 {efiboot => proxmox-boot}/zz-pve-efiboot    | 81 +++++++++++++++------
 10 files changed, 75 insertions(+), 36 deletions(-)
 rename bin/{pve-efiboot-tool => proxmox-boot-tool} (94%)
 create mode 100644 debian/pve-kernel-helper.links
 rename {efiboot => proxmox-boot}/Makefile (100%)
 rename {efiboot => proxmox-boot}/functions (100%)
 rename {efiboot => proxmox-boot}/pve-auto-removal (100%)
 rename {efiboot => proxmox-boot}/pve-efiboot-sync (84%)
 rename {efiboot => proxmox-boot}/zz-pve-efiboot (69%)

-- 
2.20.1

[pve-devel] [RFC pve-kernel-meta 1/2] proxmox-boot-tool: rename from pve-efiboot-tool

[Stoiko Ivanov]

We will be using the mechanics also for ZFS systems booting with BIOS
legacy boot, and the tool is used also in PMG and PBS.

A symlink is kept in place for compatibility reasons

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 Makefile                                    | 2 +-
 bin/Makefile                                | 2 +-
 bin/{pve-efiboot-tool => proxmox-boot-tool} | 0
 debian/pve-kernel-helper.install            | 2 +-
 debian/pve-kernel-helper.links              | 1 +
 {efiboot => proxmox-boot}/Makefile          | 0
 {efiboot => proxmox-boot}/functions         | 0
 {efiboot => proxmox-boot}/pve-auto-removal  | 0
 {efiboot => proxmox-boot}/pve-efiboot-sync  | 2 +-
 {efiboot => proxmox-boot}/zz-pve-efiboot    | 0
 10 files changed, 5 insertions(+), 4 deletions(-)
 rename bin/{pve-efiboot-tool => proxmox-boot-tool} (100%)
 create mode 100644 debian/pve-kernel-helper.links
 rename {efiboot => proxmox-boot}/Makefile (100%)
 rename {efiboot => proxmox-boot}/functions (100%)
 rename {efiboot => proxmox-boot}/pve-auto-removal (100%)
 rename {efiboot => proxmox-boot}/pve-efiboot-sync (84%)
 rename {efiboot => proxmox-boot}/zz-pve-efiboot (100%)

diff --git a/Makefile b/Makefile
index 0b62b3e..90d5989 100644
--- a/Makefile
+++ b/Makefile
@@ -13,7 +13,7 @@ BUILD_DIR=build
 
 DEBS=${KERNEL_DEB} ${HEADERS_DEB} ${HELPER_DEB}
 
-SUBDIRS = efiboot bin
+SUBDIRS = proxmox-boot bin
 
 .PHONY: all
 all: ${SUBDIRS}
diff --git a/bin/Makefile b/bin/Makefile
index 058c86f..b78fa42 100644
--- a/bin/Makefile
+++ b/bin/Makefile
@@ -5,7 +5,7 @@ all:
 
 install:
 	install -d ${SBINDIR}
-	install -m 0755 pve-efiboot-tool ${SBINDIR}/
+	install -m 0755 proxmox-boot-tool ${SBINDIR}/
 
 .PHONY: clean distclean
 distclean:
diff --git a/bin/pve-efiboot-tool b/bin/proxmox-boot-tool
similarity index 100%
rename from bin/pve-efiboot-tool
rename to bin/proxmox-boot-tool
diff --git a/debian/pve-kernel-helper.install b/debian/pve-kernel-helper.install
index 6f7f713..82a9672 100644
--- a/debian/pve-kernel-helper.install
+++ b/debian/pve-kernel-helper.install
@@ -1,5 +1,5 @@
 etc/kernel/postinst.d/*
 etc/kernel/postrm.d/*
 etc/initramfs/post-update.d/pve-efiboot-sync
-usr/sbin/pve-efiboot-tool
+usr/sbin/proxmox-boot-tool
 usr/share/pve-kernel-helper/scripts/functions
diff --git a/debian/pve-kernel-helper.links b/debian/pve-kernel-helper.links
new file mode 100644
index 0000000..70bf372
--- /dev/null
+++ b/debian/pve-kernel-helper.links
@@ -0,0 +1 @@
+/usr/sbin/proxmox-boot-tool /usr/sbin/pve-efiboot-tool
diff --git a/efiboot/Makefile b/proxmox-boot/Makefile
similarity index 100%
rename from efiboot/Makefile
rename to proxmox-boot/Makefile
diff --git a/efiboot/functions b/proxmox-boot/functions
similarity index 100%
rename from efiboot/functions
rename to proxmox-boot/functions
diff --git a/efiboot/pve-auto-removal b/proxmox-boot/pve-auto-removal
similarity index 100%
rename from efiboot/pve-auto-removal
rename to proxmox-boot/pve-auto-removal
diff --git a/efiboot/pve-efiboot-sync b/proxmox-boot/pve-efiboot-sync
similarity index 84%
rename from efiboot/pve-efiboot-sync
rename to proxmox-boot/pve-efiboot-sync
index c3ccf8e..21adc85 100644
--- a/efiboot/pve-efiboot-sync
+++ b/proxmox-boot/pve-efiboot-sync
@@ -7,5 +7,5 @@ set -e
 # this variable will be set to 1 and we do nothing, since our pve-kernel
 # hooks will update the ESPs all at once anyway.
 if [ -z "$INITRAMFS_TOOLS_KERNEL_HOOK" ]; then
-	/usr/sbin/pve-efiboot-tool refresh --hook 'zz-pve-efiboot'
+	/usr/sbin/proxmox-boot-tool refresh --hook 'zz-pve-efiboot'
 fi
diff --git a/efiboot/zz-pve-efiboot b/proxmox-boot/zz-pve-efiboot
similarity index 100%
rename from efiboot/zz-pve-efiboot
rename to proxmox-boot/zz-pve-efiboot
-- 
2.20.1

[pve-devel] [RFC pve-kernel-meta 2/2] proxmox-boot-tool: handle legacy boot zfs installs

[Stoiko Ivanov]

This patch adds support for booting non-uefi/legacy/bios-boot ZFS
installs, by using proxmox-boot-tool to copy the kernels to the ESP
and then generate a fitting grub config for booting from the vfat ESP:

* grub is installed onto the ESP and the MBR points to the ESP
* after copying/deleting the kernels proxmox-boot-tool bindmounts the
  ESP on /boot (inside the new mount namespace)
* grub-update then manages to generate a fitting config.

Some paths/sanity-checks needed adaptation (to differentiate between
EFI boot and not (based on the existence of /sys/firmware/efi)

The approach is inspired by @avw in our community-forum [0].

[0] https://forum.proxmox.com/threads/zfs-error-no-such-device-error-unknown-filesystem-entering-rescue-mode.75122/post-374799

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
best viewed with `git show -w`

 bin/proxmox-boot-tool       | 21 ++++++----
 proxmox-boot/zz-pve-efiboot | 81 ++++++++++++++++++++++++++-----------
 2 files changed, 70 insertions(+), 32 deletions(-)

diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
index f57a752..dd23231 100755
--- a/bin/proxmox-boot-tool
+++ b/bin/proxmox-boot-tool
@@ -150,14 +150,19 @@ init() {
 	echo "Mounting '$part' on '$esp_mp'."
 	mount -t vfat "$part" "$esp_mp"
 
-	echo "Installing systemd-boot.."
-	mkdir -p "$esp_mp/$PMX_ESP_DIR"
-	bootctl --path "$esp_mp" install
-
-	echo "Configuring systemd-boot.."
-	echo "timeout 3" > "$esp_mp/$PMX_LOADER_CONF.tmp"
-	echo "default proxmox-*" >> "$esp_mp/$PMX_LOADER_CONF.tmp"
-	mv "$esp_mp/$PMX_LOADER_CONF.tmp" "$esp_mp/$PMX_LOADER_CONF"
+	if [ -d /sys/firmware/efi ]; then
+		echo "Installing systemd-boot.."
+		mkdir -p "$esp_mp/$PMX_ESP_DIR"
+		bootctl --path "$esp_mp" install
+
+		echo "Configuring systemd-boot.."
+		echo "timeout 3" > "$esp_mp/$PMX_LOADER_CONF.tmp"
+		echo "default proxmox-*" >> "$esp_mp/$PMX_LOADER_CONF.tmp"
+		mv "$esp_mp/$PMX_LOADER_CONF.tmp" "$esp_mp/$PMX_LOADER_CONF"
+	else
+		echo "Installing grub i386-pc target.."
+		grub-install --boot-directory $esp_mp --target i386-pc "/dev/$PKNAME"
+	fi
 	echo "Unmounting '$part'."
 	umount "$part"
 
diff --git a/proxmox-boot/zz-pve-efiboot b/proxmox-boot/zz-pve-efiboot
index 1c4ad73..1ce89f7 100755
--- a/proxmox-boot/zz-pve-efiboot
+++ b/proxmox-boot/zz-pve-efiboot
@@ -76,18 +76,30 @@ update_esp_func() {
 		{ warn "creation of mountpoint ${mountpoint} failed - skipping"; return; }
 	mount "${path}" "${mountpoint}" || \
 		{ warn "mount of ${path} failed - skipping"; return; }
-	if [ ! -f "${mountpoint}/$PMX_LOADER_CONF" ]; then
-		warn "${path} contains no loader.conf - skipping"
-		return
-	fi
-	if [ ! -d "${mountpoint}/$PMX_ESP_DIR" ]; then
-		warn "${path}/$PMX_ESP_DIR does not exist- skipping"
+	if [ -d /sys/firmware/efi ]; then
+		if [ ! -f "${mountpoint}/$PMX_LOADER_CONF" ]; then
+			warn "${path} contains no loader.conf - skipping"
+			return
+		fi
+		if [ ! -d "${mountpoint}/$PMX_ESP_DIR" ]; then
+			warn "${path}/$PMX_ESP_DIR does not exist- skipping"
+			return
+		fi
+	elif [ ! -d "${mountpoint}/grub" ]; then
+		warn "${path} contains no grub directory - skipping"
 		return
 	fi
-
 	warn "Copying and configuring kernels on ${path}"
 	copy_and_config_kernels "${mountpoint}"
-	remove_old_kernels "${mountpoint}"
+	if [ -d /sys/firmware/efi ]; then
+		remove_old_kernels_efi "${mountpoint}"
+	else
+		remove_old_kernels_legacy "${mountpoint}"
+		mount --bind "${mountpoint}" "/boot"
+		update-grub
+		umount /boot
+
+	fi
 
 	umount "${mountpoint}" || \
 		{ warn "umount of ${path} failed - failure"; exit 0; }
@@ -113,26 +125,33 @@ copy_and_config_kernels() {
 			continue
 		fi
 
-		warn "	Copying kernel and creating boot-entry for ${kver}"
-		KERNEL_ESP_DIR="${PMX_ESP_DIR}/${kver}"
-		KERNEL_LIVE_DIR="${esp}/${KERNEL_ESP_DIR}"
-		mkdir -p "${KERNEL_LIVE_DIR}"
-		cp -u --preserve=timestamps "${linux_image}" "${KERNEL_LIVE_DIR}/"
-		cp -u --preserve=timestamps "${initrd}" "${KERNEL_LIVE_DIR}/"
-
-		# create loader entry
-		cat > "${esp}/loader/entries/proxmox-${kver}.conf" <<- EOF
-			title    ${LOADER_TITLE}
-			version  ${kver}
-			options   ${CMDLINE}
-			linux    /${KERNEL_ESP_DIR}/vmlinuz-${kver}
-			initrd   /${KERNEL_ESP_DIR}/initrd.img-${kver}
-		EOF
+		if [ -d /sys/firmware/efi ]; then
+
+			warn "	Copying kernel and creating boot-entry for ${kver}"
+			KERNEL_ESP_DIR="${PMX_ESP_DIR}/${kver}"
+			KERNEL_LIVE_DIR="${esp}/${KERNEL_ESP_DIR}"
+			mkdir -p "${KERNEL_LIVE_DIR}"
+			cp -u --preserve=timestamps "${linux_image}" "${KERNEL_LIVE_DIR}/"
+			cp -u --preserve=timestamps "${initrd}" "${KERNEL_LIVE_DIR}/"
+
+			# create loader entry
+			cat > "${esp}/loader/entries/proxmox-${kver}.conf" <<- EOF
+				title    ${LOADER_TITLE}
+				version  ${kver}
+				options   ${CMDLINE}
+				linux    /${KERNEL_ESP_DIR}/vmlinuz-${kver}
+				initrd   /${KERNEL_ESP_DIR}/initrd.img-${kver}
+			EOF
+		else
+			warn "	Copying kernel ${kver}"
+			cp -u --preserve=timestamps "${linux_image}" "${esp}/"
+			cp -u --preserve=timestamps "${initrd}" "${esp}/"
+		fi
 	done
 
 }
 
-remove_old_kernels() {
+remove_old_kernels_efi() {
 	esp="$1"
 
 	for kerneldir in "${esp}/${PMX_ESP_DIR}"/*; do
@@ -151,6 +170,20 @@ remove_old_kernels() {
 
 }
 
+remove_old_kernels_legacy() {
+	esp="$1"
+
+	for kernel in "${esp}/"vmlinuz-*; do
+		kver="$(echo "${kernel}" | sed -r "s#^${esp}/vmlinuz-(.+)\$#\\1#")"
+
+		echo "${BOOT_KVERS}" | grep -q "${kver}" && continue;
+		warn "	Removing old version ${kver}"
+		rm -rf "vmlinux-${kerneldir}"
+		rm -rf "initrd.img-${kerneldir}"
+	done
+
+}
+
 set -- $DEB_MAINT_PARAMS
 mode="${1#\'}"
 mode="${mode%\'}"
-- 
2.20.1