[pve-devel] [PATCH docs] sysadmin: revise firmware chapter, add firmware repo section

[pve-devel] [PATCH docs] sysadmin: revise firmware chapter, add firmware repo section

[Alexander Zeidler]

Chapter "Firmware Updates":
* improve the structure and clarity of information provided
* mention which update methods are when available/recommended
* add information about the already pre-installed pve-firmware package
* emphasise the importance of CPU microcode updates, how to interpret
  versions and how to recover a possibly unbootable system
* move info about non-free-firmware repo to "Package Repositories"

Chapter "Package Repositories":
* add new section "Debian Firmware Repository"

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Depends on "[PATCH docs] ha-manager, software updates: add useful BlockIds"
(https://lists.proxmox.com/pipermail/pve-devel/2023-October/059710.html)

 firmware-updates.adoc  | 233 ++++++++++++++++++++++++++++-------------
 pve-package-repos.adoc |  18 ++++
 2 files changed, 179 insertions(+), 72 deletions(-)

diff --git a/firmware-updates.adoc b/firmware-updates.adoc
index 3b3064e..7e25a50 100644
--- a/firmware-updates.adoc
+++ b/firmware-updates.adoc
@@ -4,103 +4,192 @@ Firmware Updates
 ifdef::wiki[]
 :pve-toplevel:
 endif::wiki[]
-
 Firmware updates from this chapter should be applied when running {pve} on a
 bare-metal server. Whether configuring firmware updates is appropriate within
 guests, e.g. when using device pass-through, depends strongly on your setup and
 is therefore out of scope.
 
-Regular firmware updates for devices are just as important for proper operation
-as regular software updates. There are several ways to obtain and apply those
-updates. The methods listed in this chapter can also be combined to minimize the
-chance of missing an important update.
+In addition to regular software updates, firmware updates are also important
+for reliable and secure operation.
+
+When obtaining and applying firmware updates, a combination of available options
+is recommended to get them as early as possible or at all.
 
-TIP: When a firmware was updated, a system reboot is the safest way to apply the
-new version.
+The term firmware is usually divided linguistically into microcode (for CPUs)
+and firmware (for other devices).
 
 
 [[sysadmin_firmware_persistent]]
 Persistent Firmware
 ~~~~~~~~~~~~~~~~~~~
-The following methods write the new firmware permanently to the respective
-device. The firmware therefore remains up to date regardless of the booted
-operating system.
-
-TIP: When using a user space application or 'fwupd', the hardware must usually
-have been manufactured after 2014, the system must have been booted with UEFI
-and the EFI partition manually mounted.
+This section is suitable for all devices. Updated microcode, which is usually
+included in a BIOS/UEFI update, is stored on the motherboard, whereas other
+firmware is stored on the respective device. This persistent method is
+especially important for the CPU, as it enables the earliest possible regular
+loading of the updated microcode at boot time.
 
-CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be
-prepared to reconfigure them afterwards.
+CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
+device configuration could be reset. Please follow the vendor's instructions
+carefully and back up the current configuration.
 
+Please check with your vendor which update methods are available.
 
-[[sysadmin_firmware_persistent_vendor_specific]]
-Vendor-specific
-^^^^^^^^^^^^^^^
-Firmware updates are usually available from the vendor directly. Please check
-with your vendor what options are available.
+* Convenient update methods for servers can include Dell's Lifecycle Manager or
+Service Packs from HPE.
 
-Depending on the platform and vendor, there are convenient methods available.
-For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
-Sometimes there are Linux utilities available as well. Examples are
+* Sometimes there are Linux utilities available as well. Examples are
 https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
 ConnectX or
 https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
 for Broadcom network cards.
 
+* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
+a https://fwupd.org/lvfs/vendors/[vendor] and
+https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
+requirement for this is that the system was manufactured after 2014, is booted
+via UEFI and the easiest way is to mount the EFI partition from which you boot
+(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
+'fwupd'.
 
-[[sysadmin_firmware_persistent_lvfs_fwupd]]
-Linux Vendor Firmware Service (LVFS) via fwupd
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-On https://fwupd.org['LVFS'], vendors can make their firmware updates available
-in a standardized way to a wide range of Linux hosts. Here is the growing list
-of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently
-supported https://fwupd.org/lvfs/devices/[devices].
-
-To use 'fwupd', manually mount your
-https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition]
-(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update
-firmware with the following commands:
-----
-# fwupdmgr refresh
-# fwupdmgr get-updates
-# fwupdmgr update
-# reboot
-----
+TIP: If the update instructions require a host reboot, make sure that it can be
+done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
 
 
 [[sysadmin_firmware_runtime_files]]
 Runtime Firmware Files
 ~~~~~~~~~~~~~~~~~~~~~~
-The following methods keep the firmware files available at the {pve} host and do
-not persist it on the device itself. Whenever a device is initialized, usually
-during the boot process, the corresponding firmware is loaded into the RAM of
-the respective device. These methods do not provide and can not update firmware
-that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
+This method stores firmware on the {pve} operating system and will pass it to a
+device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
+recent. It is supported by devices such as network and graphics cards, but not
+by those that rely on persisted firmware such as the motherboard and hard disks.
 
 In {pve} the package `pve-firmware` is already installed by default. Therefore,
-with the normal system updates (APT), the included firmware of common hardware
-is automatically kept up to date. Be aware that CPU microcode updates are
-located in a separate Debian repository component, which is not configured by
-default.
-
-
-[[sysadmin_firmware_runtime_files_debian_repo]]
-Debian Firmware Repository
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
-https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
-newly created Debian repository component `non-free-firmware`. It contains
-firmware for CPUs (called microcode) as well as other firmware. In the past,
-CPUs repeatedly had security vulnerabilities beside other issues. Using this
-update method (additional) to apply microcode updates is convenient, safe and
-fast.
-
-To be able to install microcode updates or other firmware from the
-`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append
-`non-free-firmware` to the end of each of the three Debian repository lines and
-run `apt-get update`.
-
-To keep the CPU microcode up to date, depending on the vendor, install the
-package `intel-microcode` or `amd64-microcode` and reboot your {pve} host
-afterwards.
+with the normal xref:system_software_updates[system updates (APT)], included
+firmware of common hardware is automatically kept up to date.
+
+An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
+exists, but is not configured by default.
+
+If you try to install an additional firmware package but it conflicts, APT will
+abort the installation. Perhaps the particular firmware can be obtained in
+another way.
+
+
+[[sysadmin_firmware_cpu]]
+CPU Microcode Updates
+~~~~~~~~~~~~~~~~~~~~~
+Microcode updates are intended to fix found security vulnerabilities and other
+serious CPU bugs. While the CPU performance can be affected, a patched microcode
+is usually still more performant than an unpatched microcode where the kernel
+itself has to do mitigations. Depending on the CPU type, it is possible that
+performance results of the flawed factory state can no longer be achieved
+without knowingly running the CPU in an unsafe state.
+
+To get an overview of present CPU vulnerabilities and their mitigations, run
+`lscpu`. Current real-world known vulnerabilities can only show up if the
+{pve} host is xref:system_software_updates[up to date], its version not
+xref:faq-support-table[end of life], and has at least been rebooted since the
+last kernel update.
+
+Besides the recommended microcode update via
+xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
+an independent method via *Early OS Microcode Updates*. It is convenient to use
+and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
+updates. Regardless of the method in use, a reboot is always needed to apply a
+microcode update.
+
+
+Set up Early OS Microcode Updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+After enabling the
+xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run
+`apt install <intel-microcode|amd64-microcode>` and reboot the {pve} host
+xref:ha_manager_node_maintenance[safely] afterwards.
+
+Infrequent future microcode updates also require a reboot to be loaded.
+
+
+Microcode Version
+^^^^^^^^^^^^^^^^^
+To get the current running microcode revision for comparison or debugging
+purposes:
+
+----
+# grep microcode /proc/cpuinfo | uniq
+microcode	: 0xf0
+----
+
+Since a microcode package contains microcode for different CPU types, an updated
+microcode for your CPU will only be included occasionally. Therefore, date
+information from the package can also not be matched to a specific release date
+from the vendor.
+
+If the microcode package is installed, the system has been rebooted, and the
+microcode included in the package is more recent than that on the motherboard
+and CPU, the message "microcode updated early" appears in the log:
+
+----
+# dmesg | grep microcode
+[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
+[    0.896580] microcode: Microcode Update Driver: v2.2.
+----
+
+
+[[sysadmin_firmware_troubleshooting]]
+Troubleshooting
+^^^^^^^^^^^^^^^
+For debugging purposes, the set up Early OS Microcode Update applied regularly
+at system boot can be temporarily disabled as follows:
+
+1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
+2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
+3. at the desired {pve} boot entry press `E`
+4. go to the line which starts with `linux` and append separated by a space
+*`dis_ucode_ldr`*
+5. press `CTRL-X` to boot this time without an Early OS Microcode Update
+
+If a problem related to a recent microcode update is suspected, a package
+downgrade should be considered instead of package removal
+(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
+xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
+though a more recent one would run without problems.
+
+A downgrade is possible if an earlier microcode package version is
+available in the Debian repository, as shown in this example:
+
+----
+# apt list -a intel-microcode
+Listing... Done
+intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
+intel-microcode/stable 3.20230512.1 amd64
+----
+----
+# apt install intel-microcode=3.202305*
+...
+Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
+...
+dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
+...
+intel-microcode: microcode will be updated at next boot
+...
+----
+
+Make sure (again) that the host can be rebooted
+xref:ha_manager_node_maintenance[safely]. To apply an older microcode
+potentially included in the microcode package for your CPU type, reboot now.
+
+[TIP]
+====
+It makes sense to hold the downgraded package for a while and try more recent
+versions again at a later time. Even if the package version is the same in the
+future, system updates may have fixed the experienced problem in the meantime.
+----
+# apt-mark hold intel-microcode
+intel-microcode set on hold.
+----
+----
+# apt-mark unhold intel-microcode
+# apt update
+# apt upgrade
+----
+====
diff --git a/pve-package-repos.adoc b/pve-package-repos.adoc
index deda092..89d041e 100644
--- a/pve-package-repos.adoc
+++ b/pve-package-repos.adoc
@@ -208,6 +208,24 @@ See the respective
 https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details.
 
 
+[[sysadmin_debian_firmware_repo]]
+Debian Firmware Repository
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
+https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
+newly created Debian repository component `non-free-firmware`.
+
+Enable this repository if you want to set up
+xref:sysadmin_firmware_cpu[Early OS Microcode Updates] or need additional
+xref:sysadmin_firmware_runtime_files[Runtime Firmware Files] not already
+included in the pre-installed package `pve-firmware`.
+
+To be able to install packages from this component, run
+`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
+`.debian.org` repository line and run `apt update`.
+
+
 [[repos_secure_apt]]
 
 SecureApt
-- 
2.39.2

[pve-devel] applied: [PATCH docs] sysadmin: revise firmware chapter, add firmware repo section

[Thomas Lamprecht]

Am 30/10/2023 um 13:10 schrieb Alexander Zeidler:
> Chapter "Firmware Updates":
> * improve the structure and clarity of information provided
> * mention which update methods are when available/recommended
> * add information about the already pre-installed pve-firmware package
> * emphasise the importance of CPU microcode updates, how to interpret
>   versions and how to recover a possibly unbootable system
> * move info about non-free-firmware repo to "Package Repositories"
> 
> Chapter "Package Repositories":
> * add new section "Debian Firmware Repository"
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Depends on "[PATCH docs] ha-manager, software updates: add useful BlockIds"
> (https://lists.proxmox.com/pipermail/pve-devel/2023-October/059710.html)
> 
>  firmware-updates.adoc  | 233 ++++++++++++++++++++++++++++-------------
>  pve-package-repos.adoc |  18 ++++
>  2 files changed, 179 insertions(+), 72 deletions(-)
> 
>

applied, thanks!

but I restructured the "set up early os µcode updates" section to a actionable
list, and dropped the subtle link to the HA maintenance mode there, such links
are rather odd and for users it's not really clear what they want to tell them
here, especially as it's only specific to HA for now (and how is the maintenance
mode safe, but a normal reboot isn't?)

If, you'd need to add a chapter/section about (safe) host reboot, that describes
how things work, what one might want to look out for to ensure guest uptime,
and in the future also things like full cluster poweroff (can be very tricky,
e.g., with ceph and/or HA).

I also reworded the part about detecting the version, and that not every update
will necessarily include a update for one's CPU - feel free to follow-up if
something got lost.