[pve-devel] [PATCH docs] sysadmin: revise firmware chapter, add firmware repo section

[Alexander Zeidler <a.zeidler@proxmox.com>]

Chapter "Firmware Updates":
* improve the structure and clarity of information provided
* mention which update methods are when available/recommended
* add information about the already pre-installed pve-firmware package
* emphasise the importance of CPU microcode updates, how to interpret
  versions and how to recover a possibly unbootable system
* move info about non-free-firmware repo to "Package Repositories"

Chapter "Package Repositories":
* add new section "Debian Firmware Repository"

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Depends on "[PATCH docs] ha-manager, software updates: add useful BlockIds"
(https://lists.proxmox.com/pipermail/pve-devel/2023-October/059710.html)

 firmware-updates.adoc  | 233 ++++++++++++++++++++++++++++-------------
 pve-package-repos.adoc |  18 ++++
 2 files changed, 179 insertions(+), 72 deletions(-)

diff --git a/firmware-updates.adoc b/firmware-updates.adoc
index 3b3064e..7e25a50 100644
--- a/firmware-updates.adoc
+++ b/firmware-updates.adoc
@@ -4,103 +4,192 @@ Firmware Updates
 ifdef::wiki[]
 :pve-toplevel:
 endif::wiki[]
-
 Firmware updates from this chapter should be applied when running {pve} on a
 bare-metal server. Whether configuring firmware updates is appropriate within
 guests, e.g. when using device pass-through, depends strongly on your setup and
 is therefore out of scope.
 
-Regular firmware updates for devices are just as important for proper operation
-as regular software updates. There are several ways to obtain and apply those
-updates. The methods listed in this chapter can also be combined to minimize the
-chance of missing an important update.
+In addition to regular software updates, firmware updates are also important
+for reliable and secure operation.
+
+When obtaining and applying firmware updates, a combination of available options
+is recommended to get them as early as possible or at all.
 
-TIP: When a firmware was updated, a system reboot is the safest way to apply the
-new version.
+The term firmware is usually divided linguistically into microcode (for CPUs)
+and firmware (for other devices).
 
 
 [[sysadmin_firmware_persistent]]
 Persistent Firmware
 ~~~~~~~~~~~~~~~~~~~
-The following methods write the new firmware permanently to the respective
-device. The firmware therefore remains up to date regardless of the booted
-operating system.
-
-TIP: When using a user space application or 'fwupd', the hardware must usually
-have been manufactured after 2014, the system must have been booted with UEFI
-and the EFI partition manually mounted.
+This section is suitable for all devices. Updated microcode, which is usually
+included in a BIOS/UEFI update, is stored on the motherboard, whereas other
+firmware is stored on the respective device. This persistent method is
+especially important for the CPU, as it enables the earliest possible regular
+loading of the updated microcode at boot time.
 
-CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be
-prepared to reconfigure them afterwards.
+CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
+device configuration could be reset. Please follow the vendor's instructions
+carefully and back up the current configuration.
 
+Please check with your vendor which update methods are available.
 
-[[sysadmin_firmware_persistent_vendor_specific]]
-Vendor-specific
-^^^^^^^^^^^^^^^
-Firmware updates are usually available from the vendor directly. Please check
-with your vendor what options are available.
+* Convenient update methods for servers can include Dell's Lifecycle Manager or
+Service Packs from HPE.
 
-Depending on the platform and vendor, there are convenient methods available.
-For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
-Sometimes there are Linux utilities available as well. Examples are
+* Sometimes there are Linux utilities available as well. Examples are
 https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
 ConnectX or
 https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
 for Broadcom network cards.
 
+* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
+a https://fwupd.org/lvfs/vendors/[vendor] and
+https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
+requirement for this is that the system was manufactured after 2014, is booted
+via UEFI and the easiest way is to mount the EFI partition from which you boot
+(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
+'fwupd'.
 
-[[sysadmin_firmware_persistent_lvfs_fwupd]]
-Linux Vendor Firmware Service (LVFS) via fwupd
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-On https://fwupd.org['LVFS'], vendors can make their firmware updates available
-in a standardized way to a wide range of Linux hosts. Here is the growing list
-of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently
-supported https://fwupd.org/lvfs/devices/[devices].
-
-To use 'fwupd', manually mount your
-https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition]
-(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update
-firmware with the following commands:
-----
-# fwupdmgr refresh
-# fwupdmgr get-updates
-# fwupdmgr update
-# reboot
-----
+TIP: If the update instructions require a host reboot, make sure that it can be
+done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
 
 
 [[sysadmin_firmware_runtime_files]]
 Runtime Firmware Files
 ~~~~~~~~~~~~~~~~~~~~~~
-The following methods keep the firmware files available at the {pve} host and do
-not persist it on the device itself. Whenever a device is initialized, usually
-during the boot process, the corresponding firmware is loaded into the RAM of
-the respective device. These methods do not provide and can not update firmware
-that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
+This method stores firmware on the {pve} operating system and will pass it to a
+device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
+recent. It is supported by devices such as network and graphics cards, but not
+by those that rely on persisted firmware such as the motherboard and hard disks.
 
 In {pve} the package `pve-firmware` is already installed by default. Therefore,
-with the normal system updates (APT), the included firmware of common hardware
-is automatically kept up to date. Be aware that CPU microcode updates are
-located in a separate Debian repository component, which is not configured by
-default.
-
-
-[[sysadmin_firmware_runtime_files_debian_repo]]
-Debian Firmware Repository
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
-https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
-newly created Debian repository component `non-free-firmware`. It contains
-firmware for CPUs (called microcode) as well as other firmware. In the past,
-CPUs repeatedly had security vulnerabilities beside other issues. Using this
-update method (additional) to apply microcode updates is convenient, safe and
-fast.
-
-To be able to install microcode updates or other firmware from the
-`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append
-`non-free-firmware` to the end of each of the three Debian repository lines and
-run `apt-get update`.
-
-To keep the CPU microcode up to date, depending on the vendor, install the
-package `intel-microcode` or `amd64-microcode` and reboot your {pve} host
-afterwards.
+with the normal xref:system_software_updates[system updates (APT)], included
+firmware of common hardware is automatically kept up to date.
+
+An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
+exists, but is not configured by default.
+
+If you try to install an additional firmware package but it conflicts, APT will
+abort the installation. Perhaps the particular firmware can be obtained in
+another way.
+
+
+[[sysadmin_firmware_cpu]]
+CPU Microcode Updates
+~~~~~~~~~~~~~~~~~~~~~
+Microcode updates are intended to fix found security vulnerabilities and other
+serious CPU bugs. While the CPU performance can be affected, a patched microcode
+is usually still more performant than an unpatched microcode where the kernel
+itself has to do mitigations. Depending on the CPU type, it is possible that
+performance results of the flawed factory state can no longer be achieved
+without knowingly running the CPU in an unsafe state.
+
+To get an overview of present CPU vulnerabilities and their mitigations, run
+`lscpu`. Current real-world known vulnerabilities can only show up if the
+{pve} host is xref:system_software_updates[up to date], its version not
+xref:faq-support-table[end of life], and has at least been rebooted since the
+last kernel update.
+
+Besides the recommended microcode update via
+xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
+an independent method via *Early OS Microcode Updates*. It is convenient to use
+and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
+updates. Regardless of the method in use, a reboot is always needed to apply a
+microcode update.
+
+
+Set up Early OS Microcode Updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+After enabling the
+xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run
+`apt install <intel-microcode|amd64-microcode>` and reboot the {pve} host
+xref:ha_manager_node_maintenance[safely] afterwards.
+
+Infrequent future microcode updates also require a reboot to be loaded.
+
+
+Microcode Version
+^^^^^^^^^^^^^^^^^
+To get the current running microcode revision for comparison or debugging
+purposes:
+
+----
+# grep microcode /proc/cpuinfo | uniq
+microcode	: 0xf0
+----
+
+Since a microcode package contains microcode for different CPU types, an updated
+microcode for your CPU will only be included occasionally. Therefore, date
+information from the package can also not be matched to a specific release date
+from the vendor.
+
+If the microcode package is installed, the system has been rebooted, and the
+microcode included in the package is more recent than that on the motherboard
+and CPU, the message "microcode updated early" appears in the log:
+
+----
+# dmesg | grep microcode
+[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
+[    0.896580] microcode: Microcode Update Driver: v2.2.
+----
+
+
+[[sysadmin_firmware_troubleshooting]]
+Troubleshooting
+^^^^^^^^^^^^^^^
+For debugging purposes, the set up Early OS Microcode Update applied regularly
+at system boot can be temporarily disabled as follows:
+
+1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
+2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
+3. at the desired {pve} boot entry press `E`
+4. go to the line which starts with `linux` and append separated by a space
+*`dis_ucode_ldr`*
+5. press `CTRL-X` to boot this time without an Early OS Microcode Update
+
+If a problem related to a recent microcode update is suspected, a package
+downgrade should be considered instead of package removal
+(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
+xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
+though a more recent one would run without problems.
+
+A downgrade is possible if an earlier microcode package version is
+available in the Debian repository, as shown in this example:
+
+----
+# apt list -a intel-microcode
+Listing... Done
+intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
+intel-microcode/stable 3.20230512.1 amd64
+----
+----
+# apt install intel-microcode=3.202305*
+...
+Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
+...
+dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
+...
+intel-microcode: microcode will be updated at next boot
+...
+----
+
+Make sure (again) that the host can be rebooted
+xref:ha_manager_node_maintenance[safely]. To apply an older microcode
+potentially included in the microcode package for your CPU type, reboot now.
+
+[TIP]
+====
+It makes sense to hold the downgraded package for a while and try more recent
+versions again at a later time. Even if the package version is the same in the
+future, system updates may have fixed the experienced problem in the meantime.
+----
+# apt-mark hold intel-microcode
+intel-microcode set on hold.
+----
+----
+# apt-mark unhold intel-microcode
+# apt update
+# apt upgrade
+----
+====
diff --git a/pve-package-repos.adoc b/pve-package-repos.adoc
index deda092..89d041e 100644
--- a/pve-package-repos.adoc
+++ b/pve-package-repos.adoc
@@ -208,6 +208,24 @@ See the respective
 https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details.
 
 
+[[sysadmin_debian_firmware_repo]]
+Debian Firmware Repository
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
+https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
+newly created Debian repository component `non-free-firmware`.
+
+Enable this repository if you want to set up
+xref:sysadmin_firmware_cpu[Early OS Microcode Updates] or need additional
+xref:sysadmin_firmware_runtime_files[Runtime Firmware Files] not already
+included in the pre-installed package `pve-firmware`.
+
+To be able to install packages from this component, run
+`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
+`.debian.org` repository line and run `apt update`.
+
+
 [[repos_secure_apt]]
 
 SecureApt
-- 
2.39.2