[pve-devel] [PATCH v2 pve-docs] qemu: add cpu models documentation

[pve-devel] [PATCH v2 pve-docs] qemu: add cpu models documentation

[Alexandre Derumier]

add doc for differents cpu models including
new x86-64-vX models

changelog v2: x86-64-v2 is compatible with >= opteron_g3

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 qm.adoc | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 138 insertions(+), 7 deletions(-)

diff --git a/qm.adoc b/qm.adoc
index 2f9afe2..33b1986 100644
--- a/qm.adoc
+++ b/qm.adoc
@@ -353,6 +353,9 @@ CPU Type
 QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon
 processors. Each new processor generation adds new features, like hardware
 assisted 3d rendering, random number generation, memory protection, etc ...
+Also, a current generation can be upgraded through microcode update with bugs
+or security fixes.
+
 Usually you should select for your VM a processor type which closely matches the
 CPU of the host system, as it means that the host CPU features (also called _CPU
 flags_ ) will be available in your VMs. If you want an exact match, you can set
@@ -360,17 +363,145 @@ the CPU type to *host* in which case the VM will have exactly the same CPU flags
 as your host system.
 
 This has a downside though. If you want to do a live migration of VMs between
-different hosts, your VM might end up on a new system with a different CPU type.
+different hosts, your VM might end up on a new system with a different CPU type
+or a different microcode.
 If the CPU flags passed to the guest are missing, the qemu process will stop. To
-remedy this QEMU has also its own CPU type *kvm64*, that {pve} uses by defaults.
-kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set,
-but is guaranteed to work everywhere.
+remedy this QEMU has also its own virtual CPU types, that {pve} uses by defaults.
+
+Default is x86-64-v2-AES, compatible with Intel >= Westmere and Amd >= Opteron_G4
+
+In short:
 
-In short, if you care about live migration and moving VMs between nodes, leave
-the kvm64 default. If you don’t care about live migration or have a homogeneous
-cluster where all nodes have the same CPU, set the CPU type to host, as in
+If you don’t care about live migration or have a homogeneous cluster where 
+all nodes have the same CPU and same microcode version, set the CPU type to host, as in
 theory this will give your guests maximum performance.
 
+if you care about live migration and security, and you have only Intel CPU or only AMD CPU,
+choose the lowest generation cpu model of your cluster.
+
+if you care about live migration without security, or have mixed intel/amd cluster,
+choose the lowest compatible virtual qemu type.
+
+NOTE: Intel <> AMD migrations have no guarantee to work
+
+
+Intel CPU Types since 2007
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors[Intel Processors]
+
+* 'Nahelem' : https://fr.wikipedia.org/wiki/Nehalem[1th generation of the Intel Core Processor]
++
+* 'Nahelem-IBRS (v2)' : add spectre (+spec-ctrl)
++
+* 'Westmere' : https://en.wikipedia.org/wiki/Westmere_(microarchitecture)[1th generation of the Intel Core Processor (Xeon E7-)]
++
+* 'Westmere-IBRS (v2)' : add spectre (+spec-ctrl)
++
+* 'SandyBridge' : https://fr.wikipedia.org/wiki/Sandy_Bridge[2th generation of the Intel Core Processor]
++
+* 'SandyBridge-IBRS (v2)' : add spectre v1 protection (+spec-ctrl)
++
+* 'IvyBridge' : https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)[3th generation of the Intel Core Processor]
++
+* 'IvyBridge-IBRS (v2)': add spectre v1 protection (+spec-ctrl)
++
+* 'Haswell' : https://fr.wikipedia.org/wiki/Haswell_(microarchitecture)[4th generation of the Intel Core Processor]
++
+* 'Haswell-noTSX (v2)' : disable TSX (-hle,-rtm)
++
+* 'Haswell-IBRS (v3)' : readd TSX, add spectre (+hle,+rtm, +spec-ctrl)
++
+* 'Harwell-noTSX-IBRS (v4)' : disable TSX (-hle,-rtm)
++
+* 'Broadwell': https://en.wikipedia.org/wiki/Broadwell_(microarchitecture)[5th generation of the Intel Core Processor]
++
+* 'Skylake': https://en.wikipedia.org/wiki/Skylake_(microarchitecture)[1st generation Xeon Scalable server processors]
++
+* 'Skylake-IBRS (v2)' : add +spec-ctrl,-clflushopt
++
+* 'Skylake-noTSX-IBRS (v3)' : disable TSX (-hle, -rtm)
++
+* 'Skylake-v4': add EPT switching (+vmx-eptp-switching)
++
+* 'Cascadelake': https://en.wikipedia.org/wiki/Cascade_Lake_(microprocessor)[2nd generation Xeon scalable processor]
++
+* 'Cascadelake-v2' : add arch_capabilities msr (+arch-capabilities,+rdctl-no,+ibrs-all,+skip-l1dfl-vmentry,+mds-no)
++
+* 'Cascadelake-v3' : disable TSX (-hle, -rtm)
++
+* 'Cascadelake-v4' : add EPT switching (+vmx-eptp-switching)
++
+* 'Cascadelake-v5' : add XSAVES (+xsaves,+vmx-xsaves)
++
+* 'CooperLake' : https://en.wikipedia.org/wiki/Cooper_Lake_(microprocessor)[3rd generation Xeon scalable processors for 4 & 8 sockets servers]
++
+* 'CooperLake-v2' : add XSAVES (+xsaves,+vmx-xsaves)
++
+* 'IceLake': https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)[3rd generation Xeon Scalable server processors]
++
+* 'Icelake-v2' : disable TSX(-hle,-rtm)
++
+* 'Icelake-v3' : add arch_capabilities msr (+arch-capabilities, +rdctl-no, +ibrs-all, +skip-l1dfl-vmentry,+mds-no,+pschange-mc-no,+taa-no)
++
+* 'Icelake-v4' : add missing flags (+sha-ni,+avx512ifma,+rdpid,+fsrm,+vmx-rdseed-exit,+vmx-pml,+vmx-eptp-switching)
++
+* 'Icelake-v5' : add XSAVES (+xsaves,+vmx-xsaves)
++
+* 'Icelake-v6' : add "5-level EPT" (+vmx-page-walk-5)
++
+* 'Sapphire Rapids' : https://en.wikipedia.org/wiki/Sapphire_Rapids[4th generation Xeon Scalable server processors]
+
+AMD CPU Types since 2007
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+https://en.wikipedia.org/wiki/List_of_AMD_processors[AMD Processors]
+
+* 'Opteron_G3' : https://en.wikipedia.org/wiki/AMD_10h[K10]
++
+* 'Opteron_G4' : https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)[Bulldozer]
++
+* 'Opteron_G5' :  https://en.wikipedia.org/wiki/Piledriver_(microarchitecture)[Piledriver]
++
+* 'EPYC' : https://en.wikipedia.org/wiki/Zen_(first_generation)[1st Generation of Zen Processors]
++
+* 'EPYC-IBPB (v2)' : add spectre v1 protection (+ibpb)
++
+* 'EPYC-v3' : add missing flags (+perfctr-core,+clzero,+xsaveerptr,+xsaves)
++
+* 'EPYC-Rome' : https://en.wikipedia.org/wiki/Zen_2[2nd Generation of Zen Processors]
++
+* 'EPYC-Rome-v2' : add spectre v2,v4 protection (+ibrs,+amd-ssbd)
++
+* 'EPYC-Milan' : https://en.wikipedia.org/wiki/Zen_3[3th Generation of Zen Processors]
++
+* 'EPYC-Milan-v2' : add missing flags (+vaes,+vpclmulqdq,+stibp-always-on,+amd-psfd,+no-nested-data-bp,+lfence-always-serializing,+null-sel-clr-base
+
+Qemu CPU Types
+^^^^^^^^^^^^^^
+
+Qemu also provide virtual cpu types, compatible with both intel/amd.
+
+NOTE: To keep best compatibility, no security flag for spectre/meltdown/... exist in qemu virtual types, so you need to do it manually
+
+Historically, Proxmox had the kvm64 cpu model, with only pentium4 cpu flags enabled, so performance was not great for some workload.
+
+In the summer of 2020, AMD, Intel, Red Hat, and SUSE collaborated to define three x86-64 microarchitecture levels on top of the x86-64 baseline, 
+with modern flags enabled. https://gitlab.com/x86-psABIs/x86-64-ABI[x86-64-ABI specs]
+
+Some newer distro like Centos9 are now built with x86-64-v2 flags as minimum requirement !
+
+
+* 'kvm64 (v1)' : Compatible >=pentium4 , >= phenom
++
+* 'x86-64-v2' : Compatible >= Nehalem, >= Opteron_G3. add cx16,lahf-lm,popcnt,pni,sse4.1,sse4.2,ssse3
++
+* 'x86-64-v2-AES' : Compatible >= Westmere, >= Opteron_G4 : add aes
++
+* 'x86-64-v3' : Compatible >= Broadwell, >= Epyc : add +avx,+avx2,+bmi1,+bmi2,+f16c,+fma,+movbe,xsave
++
+* 'x86-64-v4' : Compatible >= Skylake , >= EPYC-Genoa(V4) : add +avx512f, +avx512bw, +avx512cd,+avx512dq,+avx512vl
+
 Custom CPU Types
 ^^^^^^^^^^^^^^^^
 
-- 
2.39.2

Re: [pve-devel] [PATCH v2 pve-docs] qemu: add cpu models documentation

[Fiona Ebner]

Am 19.06.23 um 10:38 schrieb Alexandre Derumier:
> add doc for differents cpu models including
> new x86-64-vX models
> 
> changelog v2: x86-64-v2 is compatible with >= opteron_g3
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  qm.adoc | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 138 insertions(+), 7 deletions(-)
> 
> diff --git a/qm.adoc b/qm.adoc
> index 2f9afe2..33b1986 100644
> --- a/qm.adoc
> +++ b/qm.adoc
> @@ -353,6 +353,9 @@ CPU Type
>  QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon
>  processors. Each new processor generation adds new features, like hardware
>  assisted 3d rendering, random number generation, memory protection, etc ...
> +Also, a current generation can be upgraded through microcode update with bugs

s/bugs/bug/

> +or security fixes.
> +
>  Usually you should select for your VM a processor type which closely matches the
>  CPU of the host system, as it means that the host CPU features (also called _CPU
>  flags_ ) will be available in your VMs. If you want an exact match, you can set
> @@ -360,17 +363,145 @@ the CPU type to *host* in which case the VM will have exactly the same CPU flags
>  as your host system.
>  
>  This has a downside though. If you want to do a live migration of VMs between
> -different hosts, your VM might end up on a new system with a different CPU type.
> +different hosts, your VM might end up on a new system with a different CPU type
> +or a different microcode.

I'd use one of:
or different microcode.
or a different microcode version.

>  If the CPU flags passed to the guest are missing, the qemu process will stop. To
> -remedy this QEMU has also its own CPU type *kvm64*, that {pve} uses by defaults.
> -kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set,
> -but is guaranteed to work everywhere.
> +remedy this QEMU has also its own virtual CPU types, that {pve} uses by defaults.

Not your mistake, but s/defaults/default

> +
> +Default is x86-64-v2-AES, compatible with Intel >= Westmere and Amd >= Opteron_G4

The UI default is x86-64-v2-AES, which requires a host CPU starting from
Westmere for Intel or at least a fourth generation Opteron for AMD. If
your CPU doesn't support AES, use x86-64-v2 instead. If you have an
otherwise incompatible CPU, use a CPU type more closely matching your
hardware or qemu64 instead.

> +
> +In short:
>  
> -In short, if you care about live migration and moving VMs between nodes, leave
> -the kvm64 default. If you don’t care about live migration or have a homogeneous
> -cluster where all nodes have the same CPU, set the CPU type to host, as in
> +If you don’t care about live migration or have a homogeneous cluster where 
> +all nodes have the same CPU and same microcode version, set the CPU type to host, as in
>  theory this will give your guests maximum performance.
>  
> +if you care about live migration and security, and you have only Intel CPU or only AMD CPU,
> +choose the lowest generation cpu model of your cluster.

s/if/If/

> +
> +if you care about live migration without security, or have mixed intel/amd cluster,

s/if/If/
Intel/AMD

> +choose the lowest compatible virtual qemu type.

QEMU CPU type

> +
> +NOTE: Intel <> AMD migrations have no guarantee to work
> +
> +
> +Intel CPU Types since 2007
> +^^^^^^^^^^^^^^^^^^^^^^^^^^

To be more precise, it's the Intel CPU Types in QEMU

> +
> +https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors[Intel Processors]
> +
> +* 'Nahelem' : https://fr.wikipedia.org/wiki/Nehalem[1th generation of the Intel Core Processor]

This and two other links go to the French Wikipedia ;)

> ++
> +* 'Nahelem-IBRS (v2)' : add spectre (+spec-ctrl)
> ++
> +* 'Westmere' : https://en.wikipedia.org/wiki/Westmere_(microarchitecture)[1th generation of the Intel Core Processor (Xeon E7-)]
> ++
> +* 'Westmere-IBRS (v2)' : add spectre (+spec-ctrl)
> ++
> +* 'SandyBridge' : https://fr.wikipedia.org/wiki/Sandy_Bridge[2th generation of the Intel Core Processor]

here

> ++
> +* 'SandyBridge-IBRS (v2)' : add spectre v1 protection (+spec-ctrl)
> ++
> +* 'IvyBridge' : https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)[3th generation of the Intel Core Processor]
> ++
> +* 'IvyBridge-IBRS (v2)': add spectre v1 protection (+spec-ctrl)
> ++
> +* 'Haswell' : https://fr.wikipedia.org/wiki/Haswell_(microarchitecture)[4th generation of the Intel Core Processor]

and here

> ++
> +* 'Haswell-noTSX (v2)' : disable TSX (-hle,-rtm)
> ++
> +* 'Haswell-IBRS (v3)' : readd TSX, add spectre (+hle,+rtm, +spec-ctrl)
> ++
> +* 'Harwell-noTSX-IBRS (v4)' : disable TSX (-hle,-rtm)
> ++
> +* 'Broadwell': https://en.wikipedia.org/wiki/Broadwell_(microarchitecture)[5th generation of the Intel Core Processor]
> ++
> +* 'Skylake': https://en.wikipedia.org/wiki/Skylake_(microarchitecture)[1st generation Xeon Scalable server processors]
> ++
> +* 'Skylake-IBRS (v2)' : add +spec-ctrl,-clflushopt
> ++
> +* 'Skylake-noTSX-IBRS (v3)' : disable TSX (-hle, -rtm)
> ++
> +* 'Skylake-v4': add EPT switching (+vmx-eptp-switching)
> ++
> +* 'Cascadelake': https://en.wikipedia.org/wiki/Cascade_Lake_(microprocessor)[2nd generation Xeon scalable processor]
> ++
> +* 'Cascadelake-v2' : add arch_capabilities msr (+arch-capabilities,+rdctl-no,+ibrs-all,+skip-l1dfl-vmentry,+mds-no)
> ++
> +* 'Cascadelake-v3' : disable TSX (-hle, -rtm)
> ++
> +* 'Cascadelake-v4' : add EPT switching (+vmx-eptp-switching)
> ++
> +* 'Cascadelake-v5' : add XSAVES (+xsaves,+vmx-xsaves)
> ++
> +* 'CooperLake' : https://en.wikipedia.org/wiki/Cooper_Lake_(microprocessor)[3rd generation Xeon scalable processors for 4 & 8 sockets servers]
> ++
> +* 'CooperLake-v2' : add XSAVES (+xsaves,+vmx-xsaves)
> ++
> +* 'IceLake': https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)[3rd generation Xeon Scalable server processors]

If we want to stick with the QEMU model names, please use lowercase l
for the above three.

> ++
> +* 'Icelake-v2' : disable TSX(-hle,-rtm)
> ++
> +* 'Icelake-v3' : add arch_capabilities msr (+arch-capabilities, +rdctl-no, +ibrs-all, +skip-l1dfl-vmentry,+mds-no,+pschange-mc-no,+taa-no)
> ++
> +* 'Icelake-v4' : add missing flags (+sha-ni,+avx512ifma,+rdpid,+fsrm,+vmx-rdseed-exit,+vmx-pml,+vmx-eptp-switching)
> ++
> +* 'Icelake-v5' : add XSAVES (+xsaves,+vmx-xsaves)
> ++
> +* 'Icelake-v6' : add "5-level EPT" (+vmx-page-walk-5)
> ++
> +* 'Sapphire Rapids' : https://en.wikipedia.org/wiki/Sapphire_Rapids[4th generation Xeon Scalable server processors]
> +
> +AMD CPU Types since 2007
> +^^^^^^^^^^^^^^^^^^^^^^^^

To be more precise, it's the AMD CPU Types in QEMU.

> +https://en.wikipedia.org/wiki/List_of_AMD_processors[AMD Processors]
> +
> +* 'Opteron_G3' : https://en.wikipedia.org/wiki/AMD_10h[K10]
> ++
> +* 'Opteron_G4' : https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)[Bulldozer]
> ++
> +* 'Opteron_G5' :  https://en.wikipedia.org/wiki/Piledriver_(microarchitecture)[Piledriver]
> ++
> +* 'EPYC' : https://en.wikipedia.org/wiki/Zen_(first_generation)[1st Generation of Zen Processors]
> ++
> +* 'EPYC-IBPB (v2)' : add spectre v1 protection (+ibpb)
> ++
> +* 'EPYC-v3' : add missing flags (+perfctr-core,+clzero,+xsaveerptr,+xsaves)
> ++
> +* 'EPYC-Rome' : https://en.wikipedia.org/wiki/Zen_2[2nd Generation of Zen Processors]
> ++
> +* 'EPYC-Rome-v2' : add spectre v2,v4 protection (+ibrs,+amd-ssbd)
> ++
> +* 'EPYC-Milan' : https://en.wikipedia.org/wiki/Zen_3[3th Generation of Zen Processors]
> ++
> +* 'EPYC-Milan-v2' : add missing flags (+vaes,+vpclmulqdq,+stibp-always-on,+amd-psfd,+no-nested-data-bp,+lfence-always-serializing,+null-sel-clr-base
> +
> +Qemu CPU Types
> +^^^^^^^^^^^^^^

QEMU should always be capitalized like that

> +
> +Qemu also provide virtual cpu types, compatible with both intel/amd.

CPU should be capitalized, same for Intel/AMD

> +
> +NOTE: To keep best compatibility, no security flag for spectre/meltdown/... exist in qemu virtual types, so you need to do it manually

I'm not sure what you mean here. What do you need to do manually?

> +
> +Historically, Proxmox had the kvm64 cpu model, with only pentium4 cpu flags enabled, so performance was not great for some workload.

Please use {pve} instead of Proxmox

> +
> +In the summer of 2020, AMD, Intel, Red Hat, and SUSE collaborated to define three x86-64 microarchitecture levels on top of the x86-64 baseline, 
> +with modern flags enabled. https://gitlab.com/x86-psABIs/x86-64-ABI[x86-64-ABI specs]

Most of the lines above are too long, please keep the 80 character limit.

> +
> +Some newer distro like Centos9 are now built with x86-64-v2 flags as minimum requirement !

s/distro/distributions/
Maybe turn it into a note?

> +
> +* 'kvm64 (v1)' : Compatible >=pentium4 , >= phenom
> ++
> +* 'x86-64-v2' : Compatible >= Nehalem, >= Opteron_G3. add cx16,lahf-lm,popcnt,pni,sse4.1,sse4.2,ssse3
> ++
> +* 'x86-64-v2-AES' : Compatible >= Westmere, >= Opteron_G4 : add aes
> ++
> +* 'x86-64-v3' : Compatible >= Broadwell, >= Epyc : add +avx,+avx2,+bmi1,+bmi2,+f16c,+fma,+movbe,xsave
> ++
> +* 'x86-64-v4' : Compatible >= Skylake , >= EPYC-Genoa(V4) : add +avx512f, +avx512bw, +avx512cd,+avx512dq,+avx512vl
> +

g/Compatible/Compatible with/
g/add/Added CPU flags:/
Maybe also put Intel CPU >= Nehalem, AMD CPU >= Opteron_G3? Feels
slightly more readable to me.

>  Custom CPU Types
>  ^^^^^^^^^^^^^^^^
>  

Re: [pve-devel] [PATCH v2 pve-docs] qemu: add cpu models documentation

[Fiona Ebner]

Am 19.06.23 um 12:03 schrieb Fiona Ebner:
> Am 19.06.23 um 10:38 schrieb Alexandre Derumier:
>> add doc for differents cpu models including
>> new x86-64-vX models
>>
>> changelog v2: x86-64-v2 is compatible with >= opteron_g3
>>

I'm applying this with follow-ups for my suggestions, so no need to send
a v3

[pve-devel] applied: [PATCH v2 pve-docs] qemu: add cpu models documentation

[Fiona Ebner]

Am 19.06.23 um 10:38 schrieb Alexandre Derumier:
> add doc for differents cpu models including
> new x86-64-vX models
> 
> changelog v2: x86-64-v2 is compatible with >= opteron_g3
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>

applied, thanks! Made some follow-ups for typo/style/consistency fixes
and moved the list of AMD and Intel CPU types to the end of the chapter
to avoid cluttering the CPU types section.