[pmg-devel] [PATCH pmg-docs v2] administration, installation: add chapter "Firmware Updates"

[pmg-devel] [PATCH pmg-docs v2] administration, installation: add chapter "Firmware Updates"

[Alexander Zeidler]

and "Debian Firmware Repository", with mutual linking. Largely
identical to PVE docs, except for:
- remove mentions of ensuring a safe cluster node reboot
- adapt internal links for this doc structure

Firmware updates are important, their existence should not be checked
only when there are already noticeable problems.

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* place chapter "Firmware Updates" under chapter "Administration" instead of "Installation", as it is partly a recurring manual task
* make commit message verbose

v1: https://lists.proxmox.com/pipermail/pmg-devel/2023-November/002570.html


 pmg-administration.adoc | 200 ++++++++++++++++++++++++++++++++++++++++
 pmg-installation.adoc   |  17 ++++
 2 files changed, 217 insertions(+)

diff --git a/pmg-administration.adoc b/pmg-administration.adoc
index 25af4b1..95a3749 100644
--- a/pmg-administration.adoc
+++ b/pmg-administration.adoc
@@ -42,6 +42,7 @@ systemctl status postfix
 -----
 
 
+[[pmg_updates]]
 Updates
 ~~~~~~~
 
@@ -238,3 +239,202 @@ You can view the complete headers and filter by sender or receiver of
 queued emails.
 
 Here, you can also flush or delete each deferred email independently.
+
+
+[[pmg_firmware_updates]]
+Firmware Updates
+----------------
+Firmware updates from this chapter should be applied when running {pmg} or
+Debian on a bare-metal server. Whether configuring firmware updates is
+appropriate within a virtualized environment, e.g. when using device
+pass-through, depends strongly on your setup and is therefore out of scope.
+
+In addition to regular software updates, firmware updates are also important for
+reliable and secure operation.
+
+When obtaining and applying firmware updates, a combination of available options
+is recommended to get them as early as possible or at all.
+
+The term firmware is usually divided linguistically into microcode (for CPUs)
+and firmware (for other devices).
+
+
+[[pmg_firmware_persistent]]
+Persistent Firmware
+~~~~~~~~~~~~~~~~~~~
+This section is suitable for all devices. Updated microcode, which is usually
+included in a BIOS/UEFI update, is stored on the motherboard, whereas other
+firmware is stored on the respective device. This persistent method is
+especially important for the CPU, as it enables the earliest possible regular
+loading of the updated microcode at boot time.
+
+CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
+device configuration could be reset. Please follow the vendor's instructions
+carefully and back up the current configuration.
+
+Please check with your vendor which update methods are available.
+
+* Convenient update methods for servers can include Dell's Lifecycle Manager or
+Service Packs from HPE.
+
+* Sometimes there are Linux utilities available as well. Examples are
+https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
+ConnectX or
+https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
+for Broadcom network cards.
+
+* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
+a https://fwupd.org/lvfs/vendors/[vendor] and
+https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
+requirement for this is that the system was manufactured after 2014, is booted
+via UEFI and the easiest way is to mount the EFI partition from which you boot
+(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
+'fwupd'.
+
+TIP: If the update instructions require a host reboot, please do not forget
+about it.
+
+
+[[pmg_firmware_runtime_files]]
+Runtime Firmware Files
+~~~~~~~~~~~~~~~~~~~~~~
+This method stores firmware on the {pmg} operating system and will pass it to a
+device if its xref:pmg_firmware_persistent[persisted firmware] is less recent.
+It is supported by devices such as network and graphics cards, but not by those
+that rely on persisted firmware such as the motherboard and hard disks.
+
+In {pmg} the package `pve-firmware` is already installed by default. Therefore,
+with the normal xref:pmg_updates[system updates (APT)], included firmware of
+common hardware is automatically kept up to date.
+
+An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists,
+but is not configured by default.
+
+If you try to install an additional firmware package but it conflicts, APT will
+abort the installation. Perhaps the particular firmware can be obtained in
+another way.
+
+
+[[pmg_firmware_cpu]]
+CPU Microcode Updates
+~~~~~~~~~~~~~~~~~~~~~
+Microcode updates are intended to fix found security vulnerabilities and other
+serious CPU bugs. While the CPU performance can be affected, a patched microcode
+is usually still more performant than an unpatched microcode where the kernel
+itself has to do mitigations. Depending on the CPU type, it is possible that
+performance results of the flawed factory state can no longer be achieved
+without knowingly running the CPU in an unsafe state.
+
+To get an overview of present CPU vulnerabilities and their mitigations, run
+`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg}
+host is xref:pmg_updates[up to date], its version not
+xref:faq-support-table[end of life], and has at least been rebooted since the
+last kernel update.
+
+Besides the recommended microcode update via
+xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an
+independent method via *Early OS Microcode Updates*. It is convenient to use and
+also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
+updates. Regardless of the method in use, a reboot is always needed to apply a
+microcode update.
+
+
+Set up Early OS Microcode Updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+To set up microcode updates that are applied early on boot by the Linux kernel,
+you need to:
+
+. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository]
+. Get the latest available packages: `apt update` (or use the web interface,
+  under Administration -> Updates)
+. Install the CPU-vendor specific microcode package:
+  - For Intel CPUs:  `apt install intel-microcode`
+  - For AMD CPUs:  `apt install amd64-microcode`
+. Reboot the {pmg} host
+
+Any future microcode update will also require a reboot to be loaded.
+
+
+Microcode Version
+^^^^^^^^^^^^^^^^^
+To get the current running microcode revision for comparison or debugging
+purposes:
+
+----
+# grep microcode /proc/cpuinfo | uniq
+microcode	: 0xf0
+----
+
+A microcode package has updates for many different CPUs. But updates
+specifically for your CPU might not come often. So, just looking at the date on
+the package won't tell you when the company actually released an update for your
+specific CPU.
+
+If you've installed a new microcode package and rebooted your {pmg} host, and
+this new microcode is newer than both, the version baked into the CPU and the
+one from the motherboard's firmware, you'll see a message in the system log
+saying "microcode updated early".
+
+----
+# dmesg | grep microcode
+[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
+[    0.896580] microcode: Microcode Update Driver: v2.2.
+----
+
+
+[[pmg_firmware_troubleshooting]]
+Troubleshooting
+^^^^^^^^^^^^^^^
+For debugging purposes, the set up Early OS Microcode Update applied regularly
+at system boot can be temporarily disabled as follows:
+
+. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
+. At the desired {pmg} boot entry press `E`
+. Go to the line which starts with `linux` and append separated by a space
+*`dis_ucode_ldr`*
+. Press `CTRL-X` to boot this time without an Early OS Microcode Update
+
+If a problem related to a recent microcode update is suspected, a package
+downgrade should be considered instead of package removal
+(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
+xref:pmg_firmware_persistent[persisted] microcode might be loaded, even
+though a more recent one would run without problems.
+
+A downgrade is possible if an earlier microcode package version is
+available in the Debian repository, as shown in this example:
+
+----
+# apt list -a intel-microcode
+Listing... Done
+intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
+intel-microcode/stable 3.20230512.1 amd64
+----
+----
+# apt install intel-microcode=3.202305*
+...
+Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
+...
+dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
+...
+intel-microcode: microcode will be updated at next boot
+...
+----
+
+To apply an older microcode potentially included in the microcode package for
+your CPU type, reboot now.
+
+[TIP]
+====
+It makes sense to hold the downgraded package for a while and try more recent
+versions again at a later time. Even if the package version is the same in the
+future, system updates may have fixed the experienced problem in the meantime.
+----
+# apt-mark hold intel-microcode
+intel-microcode set on hold.
+----
+----
+# apt-mark unhold intel-microcode
+# apt update
+# apt upgrade
+----
+====
diff --git a/pmg-installation.adoc b/pmg-installation.adoc
index 1a0bb59..a77c155 100644
--- a/pmg-installation.adoc
+++ b/pmg-installation.adoc
@@ -456,3 +456,20 @@ Following this, you can install the required packages with:
 apt update
 apt install libclamunrar p7zip-rar
 ----
+
+
+[[pmg_debian_firmware_repo]]
+Debian Firmware Repository
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Starting with Debian Bookworm ({pmg} 8) non-free firmware (as defined by
+https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
+newly created Debian repository component `non-free-firmware`.
+
+Enable this repository if you want to set up
+xref:pmg_firmware_cpu[Early OS Microcode Updates] or need additional
+xref:pmg_firmware_runtime_files[Runtime Firmware Files] not already included in
+the pre-installed package `pve-firmware`.
+
+To be able to install packages from this component, run
+`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
+`.debian.org` repository line and run `apt update`.
-- 
2.39.2

[pmg-devel] applied: [PATCH pmg-docs v2] administration, installation: add chapter "Firmware Updates"

[Stoiko Ivanov]

On Wed, 24 Jan 2024 16:01:48 +0100
Thanks for the update!
applied the patch - having consistent docs across our products goes a long
way

one minor nit inline - if you happen to touch this again for another
reason:

Alexander Zeidler <a.zeidler@proxmox.com> wrote:

> and "Debian Firmware Repository", with mutual linking. Largely
> identical to PVE docs, except for:
> - remove mentions of ensuring a safe cluster node reboot
> - adapt internal links for this doc structure
> 
> Firmware updates are important, their existence should not be checked
> only when there are already noticeable problems.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * place chapter "Firmware Updates" under chapter "Administration" instead of "Installation", as it is partly a recurring manual task
> * make commit message verbose
> 
> v1: https://lists.proxmox.com/pipermail/pmg-devel/2023-November/002570.html
> 
> 
>  pmg-administration.adoc | 200 ++++++++++++++++++++++++++++++++++++++++
>  pmg-installation.adoc   |  17 ++++
>  2 files changed, 217 insertions(+)
> 
>.. snip..
> +[[pmg_firmware_troubleshooting]]
> +Troubleshooting
> +^^^^^^^^^^^^^^^
> +For debugging purposes, the set up Early OS Microcode Update applied regularly
> +at system boot can be temporarily disabled as follows:
> +
> +. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
> +. At the desired {pmg} boot entry press `E`
For the time being we have not only grub as a boot-loader, but also
systemd-boot - while both use `E` as edit-key - we might want to link to
the section where this is described (for pve:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline
for pmg - I'll try to add it in the coming days)

> ..snip..