From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id AB14B9E115 for ; Mon, 30 Oct 2023 13:10:26 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 8A77CEC6A for ; Mon, 30 Oct 2023 13:10:26 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 30 Oct 2023 13:10:23 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DF9DF4273B for ; Mon, 30 Oct 2023 13:10:22 +0100 (CET) From: Alexander Zeidler To: pve-devel@lists.proxmox.com Date: Mon, 30 Oct 2023 13:10:10 +0100 Message-Id: <20231030121010.155191-1-a.zeidler@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH docs] sysadmin: revise firmware chapter, add firmware repo section X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2023 12:10:26 -0000 Chapter "Firmware Updates": * improve the structure and clarity of information provided * mention which update methods are when available/recommended * add information about the already pre-installed pve-firmware package * emphasise the importance of CPU microcode updates, how to interpret versions and how to recover a possibly unbootable system * move info about non-free-firmware repo to "Package Repositories" Chapter "Package Repositories": * add new section "Debian Firmware Repository" Signed-off-by: Alexander Zeidler --- Depends on "[PATCH docs] ha-manager, software updates: add useful BlockIds" (https://lists.proxmox.com/pipermail/pve-devel/2023-October/059710.html) firmware-updates.adoc | 233 ++++++++++++++++++++++++++++------------- pve-package-repos.adoc | 18 ++++ 2 files changed, 179 insertions(+), 72 deletions(-) diff --git a/firmware-updates.adoc b/firmware-updates.adoc index 3b3064e..7e25a50 100644 --- a/firmware-updates.adoc +++ b/firmware-updates.adoc @@ -4,103 +4,192 @@ Firmware Updates ifdef::wiki[] :pve-toplevel: endif::wiki[] - Firmware updates from this chapter should be applied when running {pve} on a bare-metal server. Whether configuring firmware updates is appropriate within guests, e.g. when using device pass-through, depends strongly on your setup and is therefore out of scope. -Regular firmware updates for devices are just as important for proper operation -as regular software updates. There are several ways to obtain and apply those -updates. The methods listed in this chapter can also be combined to minimize the -chance of missing an important update. +In addition to regular software updates, firmware updates are also important +for reliable and secure operation. + +When obtaining and applying firmware updates, a combination of available options +is recommended to get them as early as possible or at all. -TIP: When a firmware was updated, a system reboot is the safest way to apply the -new version. +The term firmware is usually divided linguistically into microcode (for CPUs) +and firmware (for other devices). [[sysadmin_firmware_persistent]] Persistent Firmware ~~~~~~~~~~~~~~~~~~~ -The following methods write the new firmware permanently to the respective -device. The firmware therefore remains up to date regardless of the booted -operating system. - -TIP: When using a user space application or 'fwupd', the hardware must usually -have been manufactured after 2014, the system must have been booted with UEFI -and the EFI partition manually mounted. +This section is suitable for all devices. Updated microcode, which is usually +included in a BIOS/UEFI update, is stored on the motherboard, whereas other +firmware is stored on the respective device. This persistent method is +especially important for the CPU, as it enables the earliest possible regular +loading of the updated microcode at boot time. -CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be -prepared to reconfigure them afterwards. +CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the +device configuration could be reset. Please follow the vendor's instructions +carefully and back up the current configuration. +Please check with your vendor which update methods are available. -[[sysadmin_firmware_persistent_vendor_specific]] -Vendor-specific -^^^^^^^^^^^^^^^ -Firmware updates are usually available from the vendor directly. Please check -with your vendor what options are available. +* Convenient update methods for servers can include Dell's Lifecycle Manager or +Service Packs from HPE. -Depending on the platform and vendor, there are convenient methods available. -For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE. -Sometimes there are Linux utilities available as well. Examples are +* Sometimes there are Linux utilities available as well. Examples are https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA ConnectX or https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] for Broadcom network cards. +* https://fwupd.org[LVFS] could also be an option if there is a cooperation with +a https://fwupd.org/lvfs/vendors/[vendor] and +https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical +requirement for this is that the system was manufactured after 2014, is booted +via UEFI and the easiest way is to mount the EFI partition from which you boot +(`mount /dev/disk/by-partuuid/ /boot/efi`) before installing +'fwupd'. -[[sysadmin_firmware_persistent_lvfs_fwupd]] -Linux Vendor Firmware Service (LVFS) via fwupd -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -On https://fwupd.org['LVFS'], vendors can make their firmware updates available -in a standardized way to a wide range of Linux hosts. Here is the growing list -of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently -supported https://fwupd.org/lvfs/devices/[devices]. - -To use 'fwupd', manually mount your -https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition] -(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update -firmware with the following commands: ----- -# fwupdmgr refresh -# fwupdmgr get-updates -# fwupdmgr update -# reboot ----- +TIP: If the update instructions require a host reboot, make sure that it can be +done safely. See also xref:ha_manager_node_maintenance[Node Maintenance]. [[sysadmin_firmware_runtime_files]] Runtime Firmware Files ~~~~~~~~~~~~~~~~~~~~~~ -The following methods keep the firmware files available at the {pve} host and do -not persist it on the device itself. Whenever a device is initialized, usually -during the boot process, the corresponding firmware is loaded into the RAM of -the respective device. These methods do not provide and can not update firmware -that is used in the very early boot process (e.g. BIOS/UEFI, hard disks). +This method stores firmware on the {pve} operating system and will pass it to a +device if its xref:sysadmin_firmware_persistent[persisted firmware] is less +recent. It is supported by devices such as network and graphics cards, but not +by those that rely on persisted firmware such as the motherboard and hard disks. In {pve} the package `pve-firmware` is already installed by default. Therefore, -with the normal system updates (APT), the included firmware of common hardware -is automatically kept up to date. Be aware that CPU microcode updates are -located in a separate Debian repository component, which is not configured by -default. - - -[[sysadmin_firmware_runtime_files_debian_repo]] -Debian Firmware Repository -^^^^^^^^^^^^^^^^^^^^^^^^^^ -Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by -https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the -newly created Debian repository component `non-free-firmware`. It contains -firmware for CPUs (called microcode) as well as other firmware. In the past, -CPUs repeatedly had security vulnerabilities beside other issues. Using this -update method (additional) to apply microcode updates is convenient, safe and -fast. - -To be able to install microcode updates or other firmware from the -`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append -`non-free-firmware` to the end of each of the three Debian repository lines and -run `apt-get update`. - -To keep the CPU microcode up to date, depending on the vendor, install the -package `intel-microcode` or `amd64-microcode` and reboot your {pve} host -afterwards. +with the normal xref:system_software_updates[system updates (APT)], included +firmware of common hardware is automatically kept up to date. + +An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] +exists, but is not configured by default. + +If you try to install an additional firmware package but it conflicts, APT will +abort the installation. Perhaps the particular firmware can be obtained in +another way. + + +[[sysadmin_firmware_cpu]] +CPU Microcode Updates +~~~~~~~~~~~~~~~~~~~~~ +Microcode updates are intended to fix found security vulnerabilities and other +serious CPU bugs. While the CPU performance can be affected, a patched microcode +is usually still more performant than an unpatched microcode where the kernel +itself has to do mitigations. Depending on the CPU type, it is possible that +performance results of the flawed factory state can no longer be achieved +without knowingly running the CPU in an unsafe state. + +To get an overview of present CPU vulnerabilities and their mitigations, run +`lscpu`. Current real-world known vulnerabilities can only show up if the +{pve} host is xref:system_software_updates[up to date], its version not +xref:faq-support-table[end of life], and has at least been rebooted since the +last kernel update. + +Besides the recommended microcode update via +xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also +an independent method via *Early OS Microcode Updates*. It is convenient to use +and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI +updates. Regardless of the method in use, a reboot is always needed to apply a +microcode update. + + +Set up Early OS Microcode Updates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +After enabling the +xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run +`apt install ` and reboot the {pve} host +xref:ha_manager_node_maintenance[safely] afterwards. + +Infrequent future microcode updates also require a reboot to be loaded. + + +Microcode Version +^^^^^^^^^^^^^^^^^ +To get the current running microcode revision for comparison or debugging +purposes: + +---- +# grep microcode /proc/cpuinfo | uniq +microcode : 0xf0 +---- + +Since a microcode package contains microcode for different CPU types, an updated +microcode for your CPU will only be included occasionally. Therefore, date +information from the package can also not be matched to a specific release date +from the vendor. + +If the microcode package is installed, the system has been rebooted, and the +microcode included in the package is more recent than that on the motherboard +and CPU, the message "microcode updated early" appears in the log: + +---- +# dmesg | grep microcode +[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 +[ 0.896580] microcode: Microcode Update Driver: v2.2. +---- + + +[[sysadmin_firmware_troubleshooting]] +Troubleshooting +^^^^^^^^^^^^^^^ +For debugging purposes, the set up Early OS Microcode Update applied regularly +at system boot can be temporarily disabled as follows: + +1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely] +2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) +3. at the desired {pve} boot entry press `E` +4. go to the line which starts with `linux` and append separated by a space +*`dis_ucode_ldr`* +5. press `CTRL-X` to boot this time without an Early OS Microcode Update + +If a problem related to a recent microcode update is suspected, a package +downgrade should be considered instead of package removal +(`apt purge `). Otherwise, a too old +xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even +though a more recent one would run without problems. + +A downgrade is possible if an earlier microcode package version is +available in the Debian repository, as shown in this example: + +---- +# apt list -a intel-microcode +Listing... Done +intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] +intel-microcode/stable 3.20230512.1 amd64 +---- +---- +# apt install intel-microcode=3.202305* +... +Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' +... +dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 +... +intel-microcode: microcode will be updated at next boot +... +---- + +Make sure (again) that the host can be rebooted +xref:ha_manager_node_maintenance[safely]. To apply an older microcode +potentially included in the microcode package for your CPU type, reboot now. + +[TIP] +==== +It makes sense to hold the downgraded package for a while and try more recent +versions again at a later time. Even if the package version is the same in the +future, system updates may have fixed the experienced problem in the meantime. +---- +# apt-mark hold intel-microcode +intel-microcode set on hold. +---- +---- +# apt-mark unhold intel-microcode +# apt update +# apt upgrade +---- +==== diff --git a/pve-package-repos.adoc b/pve-package-repos.adoc index deda092..89d041e 100644 --- a/pve-package-repos.adoc +++ b/pve-package-repos.adoc @@ -208,6 +208,24 @@ See the respective https://pve.proxmox.com/wiki/Category:Ceph_Upgrade[upgrade guide] for details. +[[sysadmin_debian_firmware_repo]] +Debian Firmware Repository +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by +https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the +newly created Debian repository component `non-free-firmware`. + +Enable this repository if you want to set up +xref:sysadmin_firmware_cpu[Early OS Microcode Updates] or need additional +xref:sysadmin_firmware_runtime_files[Runtime Firmware Files] not already +included in the pre-installed package `pve-firmware`. + +To be able to install packages from this component, run +`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each +`.debian.org` repository line and run `apt update`. + + [[repos_secure_apt]] SecureApt -- 2.39.2