Re: [pmg-devel] [PATCH pmg-docs 2/2] installation: add section 'Firmware Updates' & repository

[Stoiko Ivanov <s.ivanov@proxmox.com>]

Content-wise this looks very good - thanks!

I just wondered if the 'Installation' chapter is the appropriate location
for this - I'd rather look for it in the 'Administration' chapter.

While the same could arguably be said about the p7zip-rar and libclamunrar
installation above - this is just 1 paragraph as opposed to the quite
detailed documentation in this patch

Don't feel too strongly about this - so could go in as is as well - but
adding it as separate 6.5 under Administration might have some merit

What do you think?


On Fri, Nov 03, 2023 at 02:54:56PM +0100, Alexander Zeidler wrote:
> Firmware updates are important, their existence should not be checked
> only when there are already noticeable problems.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Information provided with this patch is largely identical to that in the
> "Firmware Updates" chapter from PVE. A few minor changes have been made
> to make it suitable for PMG.
> 
> Since firmware/microcode has little to do with PMG's configuration, but
> more with "Installation", I have added a chapter 3.6 after 3.5 "Package
> Repositories"
> 
> 
>  pmg-administration.adoc |   1 +
>  pmg-installation.adoc   | 216 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 217 insertions(+)
> 
> diff --git a/pmg-administration.adoc b/pmg-administration.adoc
> index 05f4589..760f88a 100644
> --- a/pmg-administration.adoc
> +++ b/pmg-administration.adoc
> @@ -42,6 +42,7 @@ systemctl status postfix
>  -----
>  
>  
> +[[pmg_updates]]
>  Updates
>  ~~~~~~~
>  
> diff --git a/pmg-installation.adoc b/pmg-installation.adoc
> index 1a0bb59..ca6e759 100644
> --- a/pmg-installation.adoc
> +++ b/pmg-installation.adoc
> @@ -456,3 +456,219 @@ Following this, you can install the required packages with:
>  apt update
>  apt install libclamunrar p7zip-rar
>  ----
> +
> +
> +[[pmg_debian_firmware_repo]]
> +Debian Firmware Repository
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +Starting with Debian Bookworm ({pmg} 8) non-free firmware (as defined by
> +https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
> +newly created Debian repository component `non-free-firmware`.
> +
> +Enable this repository if you want to set up
> +xref:pmg_firmware_cpu[Early OS Microcode Updates] or need additional
> +xref:pmg_firmware_runtime_files[Runtime Firmware Files] not already included in
> +the pre-installed package `pve-firmware`.
> +
> +To be able to install packages from this component, run
> +`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each
> +`.debian.org` repository line and run `apt update`.
> +
> +
> +[[pmg_firmware_updates]]
> +Firmware Updates
> +----------------
> +Firmware updates from this chapter should be applied when running {pmg} or
> +Debian on a bare-metal server. Whether configuring firmware updates is
> +appropriate within a virtualized environment, e.g. when using device
> +pass-through, depends strongly on your setup and is therefore out of scope.
> +
> +In addition to regular software updates, firmware updates are also important for
> +reliable and secure operation.
> +
> +When obtaining and applying firmware updates, a combination of available options
> +is recommended to get them as early as possible or at all.
> +
> +The term firmware is usually divided linguistically into microcode (for CPUs)
> +and firmware (for other devices).
> +
> +
> +[[pmg_firmware_persistent]]
> +Persistent Firmware
> +~~~~~~~~~~~~~~~~~~~
> +This section is suitable for all devices. Updated microcode, which is usually
> +included in a BIOS/UEFI update, is stored on the motherboard, whereas other
> +firmware is stored on the respective device. This persistent method is
> +especially important for the CPU, as it enables the earliest possible regular
> +loading of the updated microcode at boot time.
> +
> +CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
> +device configuration could be reset. Please follow the vendor's instructions
> +carefully and back up the current configuration.
> +
> +Please check with your vendor which update methods are available.
> +
> +* Convenient update methods for servers can include Dell's Lifecycle Manager or
> +Service Packs from HPE.
> +
> +* Sometimes there are Linux utilities available as well. Examples are
> +https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
> +ConnectX or
> +https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
> +for Broadcom network cards.
> +
> +* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
> +a https://fwupd.org/lvfs/vendors/[vendor] and
> +https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
> +requirement for this is that the system was manufactured after 2014, is booted
> +via UEFI and the easiest way is to mount the EFI partition from which you boot
> +(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
> +'fwupd'.
> +
> +TIP: If the update instructions require a host reboot, please do not forget
> +about it.
> +
> +
> +[[pmg_firmware_runtime_files]]
> +Runtime Firmware Files
> +~~~~~~~~~~~~~~~~~~~~~~
> +This method stores firmware on the {pmg} operating system and will pass it to a
> +device if its xref:pmg_firmware_persistent[persisted firmware] is less recent.
> +It is supported by devices such as network and graphics cards, but not by those
> +that rely on persisted firmware such as the motherboard and hard disks.
> +
> +In {pmg} the package `pve-firmware` is already installed by default. Therefore,
> +with the normal xref:pmg_updates[system updates (APT)], included firmware of
> +common hardware is automatically kept up to date.
> +
> +An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists,
> +but is not configured by default.
> +
> +If you try to install an additional firmware package but it conflicts, APT will
> +abort the installation. Perhaps the particular firmware can be obtained in
> +another way.
> +
> +
> +[[pmg_firmware_cpu]]
> +CPU Microcode Updates
> +~~~~~~~~~~~~~~~~~~~~~
> +Microcode updates are intended to fix found security vulnerabilities and other
> +serious CPU bugs. While the CPU performance can be affected, a patched microcode
> +is usually still more performant than an unpatched microcode where the kernel
> +itself has to do mitigations. Depending on the CPU type, it is possible that
> +performance results of the flawed factory state can no longer be achieved
> +without knowingly running the CPU in an unsafe state.
> +
> +To get an overview of present CPU vulnerabilities and their mitigations, run
> +`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg}
> +host is xref:pmg_updates[up to date], its version not
> +xref:faq-support-table[end of life], and has at least been rebooted since the
> +last kernel update.
> +
> +Besides the recommended microcode update via
> +xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an
> +independent method via *Early OS Microcode Updates*. It is convenient to use and
> +also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
> +updates. Regardless of the method in use, a reboot is always needed to apply a
> +microcode update.
> +
> +
> +Set up Early OS Microcode Updates
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +To set up microcode updates that are applied early on boot by the Linux kernel,
> +you need to:
> +
> +. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository]
> +. Get the latest available packages: `apt update` (or use the web interface,
> +  under Administration -> Updates)
> +. Install the CPU-vendor specific microcode package:
> +  - For Intel CPUs:  `apt install intel-microcode`
> +  - For AMD CPUs:  `apt install amd64-microcode`
> +. Reboot the {pmg} host
> +
> +Any future microcode update will also require a reboot to be loaded.
> +
> +
> +Microcode Version
> +^^^^^^^^^^^^^^^^^
> +To get the current running microcode revision for comparison or debugging
> +purposes:
> +
> +----
> +# grep microcode /proc/cpuinfo | uniq
> +microcode	: 0xf0
> +----
> +
> +A microcode package has updates for many different CPUs. But updates
> +specifically for your CPU might not come often. So, just looking at the date on
> +the package won't tell you when the company actually released an update for your
> +specific CPU.
> +
> +If you've installed a new microcode package and rebooted your {pmg} host, and
> +this new microcode is newer than both, the version baked into the CPU and the
> +one from the motherboard's firmware, you'll see a message in the system log
> +saying "microcode updated early".
> +
> +----
> +# dmesg | grep microcode
> +[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
> +[    0.896580] microcode: Microcode Update Driver: v2.2.
> +----
> +
> +
> +[[pmg_firmware_troubleshooting]]
> +Troubleshooting
> +^^^^^^^^^^^^^^^
> +For debugging purposes, the set up Early OS Microcode Update applied regularly
> +at system boot can be temporarily disabled as follows:
> +
> +. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
> +. At the desired {pmg} boot entry press `E`
> +. Go to the line which starts with `linux` and append separated by a space
> +*`dis_ucode_ldr`*
> +. Press `CTRL-X` to boot this time without an Early OS Microcode Update
> +
> +If a problem related to a recent microcode update is suspected, a package
> +downgrade should be considered instead of package removal
> +(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
> +xref:pmg_firmware_persistent[persisted] microcode might be loaded, even
> +though a more recent one would run without problems.
> +
> +A downgrade is possible if an earlier microcode package version is
> +available in the Debian repository, as shown in this example:
> +
> +----
> +# apt list -a intel-microcode
> +Listing... Done
> +intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
> +intel-microcode/stable 3.20230512.1 amd64
> +----
> +----
> +# apt install intel-microcode=3.202305*
> +...
> +Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
> +...
> +dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
> +...
> +intel-microcode: microcode will be updated at next boot
> +...
> +----
> +
> +To apply an older microcode potentially included in the microcode package for
> +your CPU type, reboot now.
> +
> +[TIP]
> +====
> +It makes sense to hold the downgraded package for a while and try more recent
> +versions again at a later time. Even if the package version is the same in the
> +future, system updates may have fixed the experienced problem in the meantime.
> +----
> +# apt-mark hold intel-microcode
> +intel-microcode set on hold.
> +----
> +----
> +# apt-mark unhold intel-microcode
> +# apt update
> +# apt upgrade
> +----
> +====
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pmg-devel mailing list
> pmg-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel
> 
>