From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2C0BFC2481 for ; Mon, 22 Jan 2024 18:53:16 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 045F21E55A for ; Mon, 22 Jan 2024 18:53:16 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 22 Jan 2024 18:53:14 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 3BBFD46216 for ; Mon, 22 Jan 2024 18:53:14 +0100 (CET) Date: Mon, 22 Jan 2024 18:53:11 +0100 From: Stoiko Ivanov To: Alexander Zeidler Cc: pmg-devel@lists.proxmox.com Message-ID: References: <20231103135456.120601-1-a.zeidler@proxmox.com> <20231103135456.120601-2-a.zeidler@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231103135456.120601-2-a.zeidler@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.313 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pmg-devel] [PATCH pmg-docs 2/2] installation: add section 'Firmware Updates' & repository X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2024 17:53:16 -0000 Content-wise this looks very good - thanks! I just wondered if the 'Installation' chapter is the appropriate location for this - I'd rather look for it in the 'Administration' chapter. While the same could arguably be said about the p7zip-rar and libclamunrar installation above - this is just 1 paragraph as opposed to the quite detailed documentation in this patch Don't feel too strongly about this - so could go in as is as well - but adding it as separate 6.5 under Administration might have some merit What do you think? On Fri, Nov 03, 2023 at 02:54:56PM +0100, Alexander Zeidler wrote: > Firmware updates are important, their existence should not be checked > only when there are already noticeable problems. > > Signed-off-by: Alexander Zeidler > --- > Information provided with this patch is largely identical to that in the > "Firmware Updates" chapter from PVE. A few minor changes have been made > to make it suitable for PMG. > > Since firmware/microcode has little to do with PMG's configuration, but > more with "Installation", I have added a chapter 3.6 after 3.5 "Package > Repositories" > > > pmg-administration.adoc | 1 + > pmg-installation.adoc | 216 ++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 217 insertions(+) > > diff --git a/pmg-administration.adoc b/pmg-administration.adoc > index 05f4589..760f88a 100644 > --- a/pmg-administration.adoc > +++ b/pmg-administration.adoc > @@ -42,6 +42,7 @@ systemctl status postfix > ----- > > > +[[pmg_updates]] > Updates > ~~~~~~~ > > diff --git a/pmg-installation.adoc b/pmg-installation.adoc > index 1a0bb59..ca6e759 100644 > --- a/pmg-installation.adoc > +++ b/pmg-installation.adoc > @@ -456,3 +456,219 @@ Following this, you can install the required packages with: > apt update > apt install libclamunrar p7zip-rar > ---- > + > + > +[[pmg_debian_firmware_repo]] > +Debian Firmware Repository > +~~~~~~~~~~~~~~~~~~~~~~~~~ > +Starting with Debian Bookworm ({pmg} 8) non-free firmware (as defined by > +https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the > +newly created Debian repository component `non-free-firmware`. > + > +Enable this repository if you want to set up > +xref:pmg_firmware_cpu[Early OS Microcode Updates] or need additional > +xref:pmg_firmware_runtime_files[Runtime Firmware Files] not already included in > +the pre-installed package `pve-firmware`. > + > +To be able to install packages from this component, run > +`editor /etc/apt/sources.list`, append `non-free-firmware` to the end of each > +`.debian.org` repository line and run `apt update`. > + > + > +[[pmg_firmware_updates]] > +Firmware Updates > +---------------- > +Firmware updates from this chapter should be applied when running {pmg} or > +Debian on a bare-metal server. Whether configuring firmware updates is > +appropriate within a virtualized environment, e.g. when using device > +pass-through, depends strongly on your setup and is therefore out of scope. > + > +In addition to regular software updates, firmware updates are also important for > +reliable and secure operation. > + > +When obtaining and applying firmware updates, a combination of available options > +is recommended to get them as early as possible or at all. > + > +The term firmware is usually divided linguistically into microcode (for CPUs) > +and firmware (for other devices). > + > + > +[[pmg_firmware_persistent]] > +Persistent Firmware > +~~~~~~~~~~~~~~~~~~~ > +This section is suitable for all devices. Updated microcode, which is usually > +included in a BIOS/UEFI update, is stored on the motherboard, whereas other > +firmware is stored on the respective device. This persistent method is > +especially important for the CPU, as it enables the earliest possible regular > +loading of the updated microcode at boot time. > + > +CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the > +device configuration could be reset. Please follow the vendor's instructions > +carefully and back up the current configuration. > + > +Please check with your vendor which update methods are available. > + > +* Convenient update methods for servers can include Dell's Lifecycle Manager or > +Service Packs from HPE. > + > +* Sometimes there are Linux utilities available as well. Examples are > +https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA > +ConnectX or > +https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] > +for Broadcom network cards. > + > +* https://fwupd.org[LVFS] could also be an option if there is a cooperation with > +a https://fwupd.org/lvfs/vendors/[vendor] and > +https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical > +requirement for this is that the system was manufactured after 2014, is booted > +via UEFI and the easiest way is to mount the EFI partition from which you boot > +(`mount /dev/disk/by-partuuid/ /boot/efi`) before installing > +'fwupd'. > + > +TIP: If the update instructions require a host reboot, please do not forget > +about it. > + > + > +[[pmg_firmware_runtime_files]] > +Runtime Firmware Files > +~~~~~~~~~~~~~~~~~~~~~~ > +This method stores firmware on the {pmg} operating system and will pass it to a > +device if its xref:pmg_firmware_persistent[persisted firmware] is less recent. > +It is supported by devices such as network and graphics cards, but not by those > +that rely on persisted firmware such as the motherboard and hard disks. > + > +In {pmg} the package `pve-firmware` is already installed by default. Therefore, > +with the normal xref:pmg_updates[system updates (APT)], included firmware of > +common hardware is automatically kept up to date. > + > +An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists, > +but is not configured by default. > + > +If you try to install an additional firmware package but it conflicts, APT will > +abort the installation. Perhaps the particular firmware can be obtained in > +another way. > + > + > +[[pmg_firmware_cpu]] > +CPU Microcode Updates > +~~~~~~~~~~~~~~~~~~~~~ > +Microcode updates are intended to fix found security vulnerabilities and other > +serious CPU bugs. While the CPU performance can be affected, a patched microcode > +is usually still more performant than an unpatched microcode where the kernel > +itself has to do mitigations. Depending on the CPU type, it is possible that > +performance results of the flawed factory state can no longer be achieved > +without knowingly running the CPU in an unsafe state. > + > +To get an overview of present CPU vulnerabilities and their mitigations, run > +`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg} > +host is xref:pmg_updates[up to date], its version not > +xref:faq-support-table[end of life], and has at least been rebooted since the > +last kernel update. > + > +Besides the recommended microcode update via > +xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an > +independent method via *Early OS Microcode Updates*. It is convenient to use and > +also quite helpful when the motherboard vendor no longer provides BIOS/UEFI > +updates. Regardless of the method in use, a reboot is always needed to apply a > +microcode update. > + > + > +Set up Early OS Microcode Updates > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > +To set up microcode updates that are applied early on boot by the Linux kernel, > +you need to: > + > +. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository] > +. Get the latest available packages: `apt update` (or use the web interface, > + under Administration -> Updates) > +. Install the CPU-vendor specific microcode package: > + - For Intel CPUs: `apt install intel-microcode` > + - For AMD CPUs: `apt install amd64-microcode` > +. Reboot the {pmg} host > + > +Any future microcode update will also require a reboot to be loaded. > + > + > +Microcode Version > +^^^^^^^^^^^^^^^^^ > +To get the current running microcode revision for comparison or debugging > +purposes: > + > +---- > +# grep microcode /proc/cpuinfo | uniq > +microcode : 0xf0 > +---- > + > +A microcode package has updates for many different CPUs. But updates > +specifically for your CPU might not come often. So, just looking at the date on > +the package won't tell you when the company actually released an update for your > +specific CPU. > + > +If you've installed a new microcode package and rebooted your {pmg} host, and > +this new microcode is newer than both, the version baked into the CPU and the > +one from the motherboard's firmware, you'll see a message in the system log > +saying "microcode updated early". > + > +---- > +# dmesg | grep microcode > +[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 > +[ 0.896580] microcode: Microcode Update Driver: v2.2. > +---- > + > + > +[[pmg_firmware_troubleshooting]] > +Troubleshooting > +^^^^^^^^^^^^^^^ > +For debugging purposes, the set up Early OS Microcode Update applied regularly > +at system boot can be temporarily disabled as follows: > + > +. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) > +. At the desired {pmg} boot entry press `E` > +. Go to the line which starts with `linux` and append separated by a space > +*`dis_ucode_ldr`* > +. Press `CTRL-X` to boot this time without an Early OS Microcode Update > + > +If a problem related to a recent microcode update is suspected, a package > +downgrade should be considered instead of package removal > +(`apt purge `). Otherwise, a too old > +xref:pmg_firmware_persistent[persisted] microcode might be loaded, even > +though a more recent one would run without problems. > + > +A downgrade is possible if an earlier microcode package version is > +available in the Debian repository, as shown in this example: > + > +---- > +# apt list -a intel-microcode > +Listing... Done > +intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] > +intel-microcode/stable 3.20230512.1 amd64 > +---- > +---- > +# apt install intel-microcode=3.202305* > +... > +Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' > +... > +dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 > +... > +intel-microcode: microcode will be updated at next boot > +... > +---- > + > +To apply an older microcode potentially included in the microcode package for > +your CPU type, reboot now. > + > +[TIP] > +==== > +It makes sense to hold the downgraded package for a while and try more recent > +versions again at a later time. Even if the package version is the same in the > +future, system updates may have fixed the experienced problem in the meantime. > +---- > +# apt-mark hold intel-microcode > +intel-microcode set on hold. > +---- > +---- > +# apt-mark unhold intel-microcode > +# apt update > +# apt upgrade > +---- > +==== > -- > 2.39.2 > > > > _______________________________________________ > pmg-devel mailing list > pmg-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > >