From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 641E7A2756 for ; Mon, 19 Jun 2023 12:03:47 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 464B027BA1 for ; Mon, 19 Jun 2023 12:03:17 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 19 Jun 2023 12:03:16 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0072F468BF; Mon, 19 Jun 2023 12:03:16 +0200 (CEST) Message-ID: Date: Mon, 19 Jun 2023 12:03:14 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Content-Language: en-US To: Proxmox VE development discussion , Alexandre Derumier References: <20230619083834.202496-1-aderumier@odiso.com> From: Fiona Ebner In-Reply-To: <20230619083834.202496-1-aderumier@odiso.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.002 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.09 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] [PATCH v2 pve-docs] qemu: add cpu models documentation X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jun 2023 10:03:47 -0000 Am 19.06.23 um 10:38 schrieb Alexandre Derumier: > add doc for differents cpu models including > new x86-64-vX models > > changelog v2: x86-64-v2 is compatible with >= opteron_g3 > > Signed-off-by: Alexandre Derumier > --- > qm.adoc | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 138 insertions(+), 7 deletions(-) > > diff --git a/qm.adoc b/qm.adoc > index 2f9afe2..33b1986 100644 > --- a/qm.adoc > +++ b/qm.adoc > @@ -353,6 +353,9 @@ CPU Type > QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon > processors. Each new processor generation adds new features, like hardware > assisted 3d rendering, random number generation, memory protection, etc ... > +Also, a current generation can be upgraded through microcode update with bugs s/bugs/bug/ > +or security fixes. > + > Usually you should select for your VM a processor type which closely matches the > CPU of the host system, as it means that the host CPU features (also called _CPU > flags_ ) will be available in your VMs. If you want an exact match, you can set > @@ -360,17 +363,145 @@ the CPU type to *host* in which case the VM will have exactly the same CPU flags > as your host system. > > This has a downside though. If you want to do a live migration of VMs between > -different hosts, your VM might end up on a new system with a different CPU type. > +different hosts, your VM might end up on a new system with a different CPU type > +or a different microcode. I'd use one of: or different microcode. or a different microcode version. > If the CPU flags passed to the guest are missing, the qemu process will stop. To > -remedy this QEMU has also its own CPU type *kvm64*, that {pve} uses by defaults. > -kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set, > -but is guaranteed to work everywhere. > +remedy this QEMU has also its own virtual CPU types, that {pve} uses by defaults. Not your mistake, but s/defaults/default > + > +Default is x86-64-v2-AES, compatible with Intel >= Westmere and Amd >= Opteron_G4 The UI default is x86-64-v2-AES, which requires a host CPU starting from Westmere for Intel or at least a fourth generation Opteron for AMD. If your CPU doesn't support AES, use x86-64-v2 instead. If you have an otherwise incompatible CPU, use a CPU type more closely matching your hardware or qemu64 instead. > + > +In short: > > -In short, if you care about live migration and moving VMs between nodes, leave > -the kvm64 default. If you don’t care about live migration or have a homogeneous > -cluster where all nodes have the same CPU, set the CPU type to host, as in > +If you don’t care about live migration or have a homogeneous cluster where > +all nodes have the same CPU and same microcode version, set the CPU type to host, as in > theory this will give your guests maximum performance. > > +if you care about live migration and security, and you have only Intel CPU or only AMD CPU, > +choose the lowest generation cpu model of your cluster. s/if/If/ > + > +if you care about live migration without security, or have mixed intel/amd cluster, s/if/If/ Intel/AMD > +choose the lowest compatible virtual qemu type. QEMU CPU type > + > +NOTE: Intel <> AMD migrations have no guarantee to work > + > + > +Intel CPU Types since 2007 > +^^^^^^^^^^^^^^^^^^^^^^^^^^ To be more precise, it's the Intel CPU Types in QEMU > + > +https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors[Intel Processors] > + > +* 'Nahelem' : https://fr.wikipedia.org/wiki/Nehalem[1th generation of the Intel Core Processor] This and two other links go to the French Wikipedia ;) > ++ > +* 'Nahelem-IBRS (v2)' : add spectre (+spec-ctrl) > ++ > +* 'Westmere' : https://en.wikipedia.org/wiki/Westmere_(microarchitecture)[1th generation of the Intel Core Processor (Xeon E7-)] > ++ > +* 'Westmere-IBRS (v2)' : add spectre (+spec-ctrl) > ++ > +* 'SandyBridge' : https://fr.wikipedia.org/wiki/Sandy_Bridge[2th generation of the Intel Core Processor] here > ++ > +* 'SandyBridge-IBRS (v2)' : add spectre v1 protection (+spec-ctrl) > ++ > +* 'IvyBridge' : https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)[3th generation of the Intel Core Processor] > ++ > +* 'IvyBridge-IBRS (v2)': add spectre v1 protection (+spec-ctrl) > ++ > +* 'Haswell' : https://fr.wikipedia.org/wiki/Haswell_(microarchitecture)[4th generation of the Intel Core Processor] and here > ++ > +* 'Haswell-noTSX (v2)' : disable TSX (-hle,-rtm) > ++ > +* 'Haswell-IBRS (v3)' : readd TSX, add spectre (+hle,+rtm, +spec-ctrl) > ++ > +* 'Harwell-noTSX-IBRS (v4)' : disable TSX (-hle,-rtm) > ++ > +* 'Broadwell': https://en.wikipedia.org/wiki/Broadwell_(microarchitecture)[5th generation of the Intel Core Processor] > ++ > +* 'Skylake': https://en.wikipedia.org/wiki/Skylake_(microarchitecture)[1st generation Xeon Scalable server processors] > ++ > +* 'Skylake-IBRS (v2)' : add +spec-ctrl,-clflushopt > ++ > +* 'Skylake-noTSX-IBRS (v3)' : disable TSX (-hle, -rtm) > ++ > +* 'Skylake-v4': add EPT switching (+vmx-eptp-switching) > ++ > +* 'Cascadelake': https://en.wikipedia.org/wiki/Cascade_Lake_(microprocessor)[2nd generation Xeon scalable processor] > ++ > +* 'Cascadelake-v2' : add arch_capabilities msr (+arch-capabilities,+rdctl-no,+ibrs-all,+skip-l1dfl-vmentry,+mds-no) > ++ > +* 'Cascadelake-v3' : disable TSX (-hle, -rtm) > ++ > +* 'Cascadelake-v4' : add EPT switching (+vmx-eptp-switching) > ++ > +* 'Cascadelake-v5' : add XSAVES (+xsaves,+vmx-xsaves) > ++ > +* 'CooperLake' : https://en.wikipedia.org/wiki/Cooper_Lake_(microprocessor)[3rd generation Xeon scalable processors for 4 & 8 sockets servers] > ++ > +* 'CooperLake-v2' : add XSAVES (+xsaves,+vmx-xsaves) > ++ > +* 'IceLake': https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)[3rd generation Xeon Scalable server processors] If we want to stick with the QEMU model names, please use lowercase l for the above three. > ++ > +* 'Icelake-v2' : disable TSX(-hle,-rtm) > ++ > +* 'Icelake-v3' : add arch_capabilities msr (+arch-capabilities, +rdctl-no, +ibrs-all, +skip-l1dfl-vmentry,+mds-no,+pschange-mc-no,+taa-no) > ++ > +* 'Icelake-v4' : add missing flags (+sha-ni,+avx512ifma,+rdpid,+fsrm,+vmx-rdseed-exit,+vmx-pml,+vmx-eptp-switching) > ++ > +* 'Icelake-v5' : add XSAVES (+xsaves,+vmx-xsaves) > ++ > +* 'Icelake-v6' : add "5-level EPT" (+vmx-page-walk-5) > ++ > +* 'Sapphire Rapids' : https://en.wikipedia.org/wiki/Sapphire_Rapids[4th generation Xeon Scalable server processors] > + > +AMD CPU Types since 2007 > +^^^^^^^^^^^^^^^^^^^^^^^^ To be more precise, it's the AMD CPU Types in QEMU. > +https://en.wikipedia.org/wiki/List_of_AMD_processors[AMD Processors] > + > +* 'Opteron_G3' : https://en.wikipedia.org/wiki/AMD_10h[K10] > ++ > +* 'Opteron_G4' : https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)[Bulldozer] > ++ > +* 'Opteron_G5' : https://en.wikipedia.org/wiki/Piledriver_(microarchitecture)[Piledriver] > ++ > +* 'EPYC' : https://en.wikipedia.org/wiki/Zen_(first_generation)[1st Generation of Zen Processors] > ++ > +* 'EPYC-IBPB (v2)' : add spectre v1 protection (+ibpb) > ++ > +* 'EPYC-v3' : add missing flags (+perfctr-core,+clzero,+xsaveerptr,+xsaves) > ++ > +* 'EPYC-Rome' : https://en.wikipedia.org/wiki/Zen_2[2nd Generation of Zen Processors] > ++ > +* 'EPYC-Rome-v2' : add spectre v2,v4 protection (+ibrs,+amd-ssbd) > ++ > +* 'EPYC-Milan' : https://en.wikipedia.org/wiki/Zen_3[3th Generation of Zen Processors] > ++ > +* 'EPYC-Milan-v2' : add missing flags (+vaes,+vpclmulqdq,+stibp-always-on,+amd-psfd,+no-nested-data-bp,+lfence-always-serializing,+null-sel-clr-base > + > +Qemu CPU Types > +^^^^^^^^^^^^^^ QEMU should always be capitalized like that > + > +Qemu also provide virtual cpu types, compatible with both intel/amd. CPU should be capitalized, same for Intel/AMD > + > +NOTE: To keep best compatibility, no security flag for spectre/meltdown/... exist in qemu virtual types, so you need to do it manually I'm not sure what you mean here. What do you need to do manually? > + > +Historically, Proxmox had the kvm64 cpu model, with only pentium4 cpu flags enabled, so performance was not great for some workload. Please use {pve} instead of Proxmox > + > +In the summer of 2020, AMD, Intel, Red Hat, and SUSE collaborated to define three x86-64 microarchitecture levels on top of the x86-64 baseline, > +with modern flags enabled. https://gitlab.com/x86-psABIs/x86-64-ABI[x86-64-ABI specs] Most of the lines above are too long, please keep the 80 character limit. > + > +Some newer distro like Centos9 are now built with x86-64-v2 flags as minimum requirement ! s/distro/distributions/ Maybe turn it into a note? > + > +* 'kvm64 (v1)' : Compatible >=pentium4 , >= phenom > ++ > +* 'x86-64-v2' : Compatible >= Nehalem, >= Opteron_G3. add cx16,lahf-lm,popcnt,pni,sse4.1,sse4.2,ssse3 > ++ > +* 'x86-64-v2-AES' : Compatible >= Westmere, >= Opteron_G4 : add aes > ++ > +* 'x86-64-v3' : Compatible >= Broadwell, >= Epyc : add +avx,+avx2,+bmi1,+bmi2,+f16c,+fma,+movbe,xsave > ++ > +* 'x86-64-v4' : Compatible >= Skylake , >= EPYC-Genoa(V4) : add +avx512f, +avx512bw, +avx512cd,+avx512dq,+avx512vl > + g/Compatible/Compatible with/ g/add/Added CPU flags:/ Maybe also put Intel CPU >= Nehalem, AMD CPU >= Opteron_G3? Feels slightly more readable to me. > Custom CPU Types > ^^^^^^^^^^^^^^^^ >