[pve-devel] [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[pve-devel] [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Alexander Zeidler]

Firmware updates are important, their existence should not be checked
only when there are already noticeable problems.

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Changes since v1:
  * apply Aaron's comment (section Vendor-specific)


 firmware-updates.adoc | 106 ++++++++++++++++++++++++++++++++++++++++++
 local-lvm.adoc        |   4 +-
 qm.adoc               |  15 +++---
 sysadmin.adoc         |   4 ++
 4 files changed, 120 insertions(+), 9 deletions(-)
 create mode 100644 firmware-updates.adoc

diff --git a/firmware-updates.adoc b/firmware-updates.adoc
new file mode 100644
index 0000000..3b3064e
--- /dev/null
+++ b/firmware-updates.adoc
@@ -0,0 +1,106 @@
+[[chapter_firmware_updates]]
+Firmware Updates
+----------------
+ifdef::wiki[]
+:pve-toplevel:
+endif::wiki[]
+
+Firmware updates from this chapter should be applied when running {pve} on a
+bare-metal server. Whether configuring firmware updates is appropriate within
+guests, e.g. when using device pass-through, depends strongly on your setup and
+is therefore out of scope.
+
+Regular firmware updates for devices are just as important for proper operation
+as regular software updates. There are several ways to obtain and apply those
+updates. The methods listed in this chapter can also be combined to minimize the
+chance of missing an important update.
+
+TIP: When a firmware was updated, a system reboot is the safest way to apply the
+new version.
+
+
+[[sysadmin_firmware_persistent]]
+Persistent Firmware
+~~~~~~~~~~~~~~~~~~~
+The following methods write the new firmware permanently to the respective
+device. The firmware therefore remains up to date regardless of the booted
+operating system.
+
+TIP: When using a user space application or 'fwupd', the hardware must usually
+have been manufactured after 2014, the system must have been booted with UEFI
+and the EFI partition manually mounted.
+
+CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be
+prepared to reconfigure them afterwards.
+
+
+[[sysadmin_firmware_persistent_vendor_specific]]
+Vendor-specific
+^^^^^^^^^^^^^^^
+Firmware updates are usually available from the vendor directly. Please check
+with your vendor what options are available.
+
+Depending on the platform and vendor, there are convenient methods available.
+For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
+Sometimes there are Linux utilities available as well. Examples are
+https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
+ConnectX or
+https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
+for Broadcom network cards.
+
+
+[[sysadmin_firmware_persistent_lvfs_fwupd]]
+Linux Vendor Firmware Service (LVFS) via fwupd
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+On https://fwupd.org['LVFS'], vendors can make their firmware updates available
+in a standardized way to a wide range of Linux hosts. Here is the growing list
+of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently
+supported https://fwupd.org/lvfs/devices/[devices].
+
+To use 'fwupd', manually mount your
+https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition]
+(ESP) you booted from on `/boot/`. After installing the package 'fwupd', update
+firmware with the following commands:
+----
+# fwupdmgr refresh
+# fwupdmgr get-updates
+# fwupdmgr update
+# reboot
+----
+
+
+[[sysadmin_firmware_runtime_files]]
+Runtime Firmware Files
+~~~~~~~~~~~~~~~~~~~~~~
+The following methods keep the firmware files available at the {pve} host and do
+not persist it on the device itself. Whenever a device is initialized, usually
+during the boot process, the corresponding firmware is loaded into the RAM of
+the respective device. These methods do not provide and can not update firmware
+that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
+
+In {pve} the package `pve-firmware` is already installed by default. Therefore,
+with the normal system updates (APT), the included firmware of common hardware
+is automatically kept up to date. Be aware that CPU microcode updates are
+located in a separate Debian repository component, which is not configured by
+default.
+
+
+[[sysadmin_firmware_runtime_files_debian_repo]]
+Debian Firmware Repository
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by
+https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the
+newly created Debian repository component `non-free-firmware`. It contains
+firmware for CPUs (called microcode) as well as other firmware. In the past,
+CPUs repeatedly had security vulnerabilities beside other issues. Using this
+update method (additional) to apply microcode updates is convenient, safe and
+fast.
+
+To be able to install microcode updates or other firmware from the
+`non-free-firmware` component, edit the file `/etc/apt/sources.list`, append
+`non-free-firmware` to the end of each of the three Debian repository lines and
+run `apt-get update`.
+
+To keep the CPU microcode up to date, depending on the vendor, install the
+package `intel-microcode` or `amd64-microcode` and reboot your {pve} host
+afterwards.
diff --git a/local-lvm.adoc b/local-lvm.adoc
index 8092bb8..b6009c8 100644
--- a/local-lvm.adoc
+++ b/local-lvm.adoc
@@ -64,7 +64,9 @@ Bootloader
 
 We install two boot loaders by default. The first partition contains
 the standard GRUB boot loader. The second partition is an **E**FI **S**ystem
-**P**artition (ESP), which makes it possible to boot on EFI systems.
+**P**artition (ESP), which makes it possible to boot on EFI systems and to
+apply xref:sysadmin_firmware_persistent[persistent firmware updates] from the
+user space.
 
 
 Creating a Volume Group
diff --git a/qm.adoc b/qm.adoc
index b3c3034..c4f1024 100644
--- a/qm.adoc
+++ b/qm.adoc
@@ -352,9 +352,9 @@ CPU Type
 
 QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon
 processors. Each new processor generation adds new features, like hardware
-assisted 3d rendering, random number generation, memory protection, etc.. Also,
-a current generation can be upgraded through microcode update with bug or
-security fixes.
+assisted 3d rendering, random number generation, memory protection, etc. Also,
+a current generation can be upgraded through
+xref:chapter_firmware_updates[microcode update] with bug or security fixes.
 
 Usually you should select for your VM a processor type which closely matches the
 CPU of the host system, as it means that the host CPU features (also called _CPU
@@ -460,10 +460,9 @@ editing the CPU options in the WebUI, or by setting the 'flags' property of the
 'cpu' option in the VM configuration file.
 
 For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
-so-called ``microcode update'' footnote:[You can use `intel-microcode' /
-`amd-microcode' from Debian non-free if your vendor does not provide such an
-update. Note that not all affected CPUs can be updated to support spec-ctrl.]
-for your CPU.
+so-called ``microcode update'' for your CPU, see
+xref:chapter_firmware_updates[chapter Firmware Updates]. Note that not all
+affected CPUs can be updated to support spec-ctrl.
 
 
 To check if the {pve} host is vulnerable, execute the following command as root:
@@ -472,7 +471,7 @@ To check if the {pve} host is vulnerable, execute the following command as root:
 for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done
 ----
 
-A community script is also available to detect is the host is still vulnerable.
+A community script is also available to detect if the host is still vulnerable.
 footnote:[spectre-meltdown-checker https://meltdown.ovh/]
 
 Intel processors
diff --git a/sysadmin.adoc b/sysadmin.adoc
index d3de40a..dd43f73 100644
--- a/sysadmin.adoc
+++ b/sysadmin.adoc
@@ -32,6 +32,8 @@ See Also
 
 * link:/wiki/System_Software_Updates[System Software Updates]
 
+* link:/wiki/Firmware_Updates[Firmware Updates]
+
 * link:/wiki/Host_Bootloader[Host Bootloader]
 
 * link:/wiki/Time_Synchronization[Time Synchronization]
@@ -59,6 +61,8 @@ include::pve-package-repos.adoc[]
 
 include::system-software-updates.adoc[]
 
+include::firmware-updates.adoc[]
+
 include::pve-network.adoc[]
 
 include::system-timesync.adoc[]
-- 
2.39.2

Re: [pve-devel] [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Alexander Zeidler]

On Fri, 2023-07-28 at 15:21 +0200, Alexander Zeidler wrote:
> Firmware updates are important, their existence should not be checked
> only when there are already noticeable problems.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>

Ping. Still applies on master.

Would be of interest, see e.g.:

[PVE-User] EPYC ZEN gen1 migration success after installing amd64
microcode (FYI)
https://lists.proxmox.com/pipermail/pve-user/2023-September/017223.html

Re: [pve-devel] [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Dominik Csapak]

personally i would be cautious to include such deep links to vendor sites in our
documentation, because they can be outdated fast and it becomes a catch-up game....

No strong feelings though if the others are fine with it.
Besides that, consider this

Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>

Re: [pve-devel] [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Thomas Lamprecht]

Am 20/09/2023 um 11:59 schrieb Dominik Csapak:
> personally i would be cautious to include such deep links to vendor sites in our
> documentation, because they can be outdated fast and it becomes a catch-up game....
> 
> No strong feelings though if the others are fine with it.

I normally make web.archive scrape it, and sometimes even use that link then
(otherwise it's just to have the option to switch to that link if even still
relevant when the original breaks)

https://web.archive.org/

Can you please do so and use those links instead and send a v2 with Dominik's
R-b tag already included?

[pve-devel] applied: [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Thomas Lamprecht]

Am 28/07/2023 um 15:21 schrieb Alexander Zeidler:
> Firmware updates are important, their existence should not be checked
> only when there are already noticeable problems.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Changes since v1:
>   * apply Aaron's comment (section Vendor-specific)
> 
> 
>  firmware-updates.adoc | 106 ++++++++++++++++++++++++++++++++++++++++++
>  local-lvm.adoc        |   4 +-
>  qm.adoc               |  15 +++---
>  sysadmin.adoc         |   4 ++
>  4 files changed, 120 insertions(+), 9 deletions(-)
>  create mode 100644 firmware-updates.adoc
> 
>

applied, thanks!

w.r.t. to the links to vendor sites: Alexander argued that it has some
merits to link directly, not some archived version, so that users get an
up-to-date info, which seems valid to me.
I mentioned that we could go for providing both, just like often done for
external references in, e.g., Wikipedia - for example:
"https://example.com/real-url (https://archive.org/archived-url[archive])"

But, let's keep it as is for now, Alexander agreed on submitting those
URLs still to web.archive.org so that we can at least check in the future
to what content we linked here.

Oh, and this made me think that it would be nice to have a test for
checking all linked URLs, i.e., make a request and see if that
immediately return HTTP OK 200 code, otherwise output the url as
possibly broken or in need of update. This could be a simple make
target, albeit not sure what tooling to use for scan for urls and
query them, some simple perl script might be enough.

Re: [pve-devel] applied: [PATCH v2 docs] sysadmin: add section 'Firmware Updates' and references

[Thomas Lamprecht]

Am 21/09/2023 um 15:17 schrieb Thomas Lamprecht:
> Oh, and this made me think that it would be nice to have a test for
> checking all linked URLs, i.e., make a request and see if that
> immediately return HTTP OK 200 code, otherwise output the url as
> possibly broken or in need of update. This could be a simple make
> target, albeit not sure what tooling to use for scan for urls and
> query them, some simple perl script might be enough.

Fabian pointed me to mlc [0], and with that we already found two dead
links (now fixed), and also that both ceph.com and ceph.io are broken if
a user-agent doesn't sends a "Accept-Language" header [1], while not even
providing multiple languages – can't make this up..

[0]: https://crates.io/crates/mlc
[1]: https://tracker.ceph.com/issues/51364