[pve-devel] [PATCH-SERIES v4 qemu-server/manager] add and set x86-64-v2-AES as default model for new vms

[pve-devel] [PATCH-SERIES v4 qemu-server/manager] add and set x86-64-v2-AES as default model for new vms

[Alexandre Derumier]

Hi,

we used kvm64 as default cpumodel since the begin of proxmox. (basically, it's like a pentium4 cpu flags).

New distros like rhel9 are compiled to use more modern cpu flags.
(and windows already use new flags since year, and we already add some extra cpu flags)

"
In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
three microarchitecture levels on top of the historical x86-64
baseline:

  * x86-64-v1: original x86_64 baseline instruction set
  * x86-64-v2: vector instructions up to Streaming SIMD
               Extensions 4.2 (SSE4.2)  and Supplemental
               Streaming SIMD Extensions 3 (SSSE3), the
               POPCNT instruction, and CMPXCHG16B
  * x86-64-v3: vector instructions up to AVX2, MOVBE,
               and additional bit-manipulation instructions.
  * x86-64-v4: vector instructions from some of the
               AVX-512 variants.

"


This patch series add new models inspired from a patch was found on qemu mailing, but never appplied

https://lore.kernel.org/all/20210526144038.278899-1-berrange@redhat.com/T/


In addition to theses model, I have enabled aes too.
I think it's really important, because a lot of users use default values and have
bad performance with ssl and other crypto stuffs.


This was discussed on the qemu mailing

"
Crypto accelerator caveats
==========================

Similarly I'm not a huge fan of leaving out the "aes"
instruction for accelerated crypto, as missing "aes" is
also one of the key factors in making qemu64 a bad choice.

If we include 'aes' in x86-64-v2, then we loose support
for Nehalem hosts.

If we include 'aes' in x86-64-v3 then we further loose
support for Dhyana hosts (an EPYC derived CPU).
"


Nahelemn is a 2008 cpu, so I think it's ok, we are in 2013 ;)
(and user can disable aes flag in gui too)

Dhyana is a chinese fork of epyc, so we don't support the vendor


I still think than enable aes by default is really the more easy,
but for x86-64-v2, just do 1 model without aes (for nehalem), and a
model with aes.

Like this, users don't need to play manually with flags.

This patch series add new models, and set x86-64-v2-AES model as default in pve-manager
wizard only.
(to not break current vm, where kvm64 is the default when cputype is not defined
in configuration)

Here the new builtin models:


x86-64-v1 : not implemented, as it's basicaly qemu64|kvm64 -vme,-cx16 for compat Opteron_G1 from 2004
            so will use it as qemu64|kvm64 is higher are not working on opteron_g1 anyway

x86-64-v2 : Derived from qemu, +popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Nehalem
min amd : Opteron_G3

x86-64-v2-AES : Derived from qemu, +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Westmere
min amd : Opteron_G3

x86-64-v3 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe

min intel: Haswell
min amd : EPYC_v1

x86-64-v4 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512vl

min intel: Skylake
min amd : EPYC_v4





changelog v4:
 - remove patches for best cpu detection (maybe do a standalone tool later)
 - use qemu64 as base model and add extra flags
 - add x64-64-v2-AES (default)
 - remove x64-64-v1
 - add x64-64-v4
 - fix fiona comments

qemu-server :

Alexandre Derumier (1):
  cpuconfig: add new x86-64-vX models

 PVE/QemuServer/CPUConfig.pm | 48 +++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 5 deletions(-)

pve-manager:

Alexandre Derumier (1):
  qemu: processor : set x86-64-v2-AES as default cputype for create
    wizard

 www/manager6/qemu/OSDefaults.js    |  1 +
 www/manager6/qemu/ProcessorEdit.js | 13 +++++++++++++
 2 files changed, 14 insertions(+)


-- 
2.30.2

[pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[Alexandre Derumier]

https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg01592.html
"
In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
three microarchitecture levels on top of the historical x86-64
baseline:

  * x86-64:    original x86_64 baseline instruction set
  * x86-64-v2: vector instructions up to Streaming SIMD
               Extensions 4.2 (SSE4.2)  and Supplemental
               Streaming SIMD Extensions 3 (SSSE3), the
               POPCNT instruction, and CMPXCHG16B
  * x86-64-v3: vector instructions up to AVX2, MOVBE,
               and additional bit-manipulation instructions.
  * x86-64-v4: vector instructions from some of the
               AVX-512 variants.
"

This patch add new builtin model derivated from qemu64 model,
to be compatible between intel/amd.

x86-64-v1 : I'm skipping it, as it's basicaly qemu64|kvm64 -vme,-cx16 for compat Opteron_G1 from 2004
            so will use it as qemu64|kvm64 is higher are not working on opteron_g1 anyway

x86-64-v2 : Derived from qemu, +popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Nehalem
min amd : Opteron_G3

x86-64-v2-AES : Derived from qemu, +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Westmere
min amd : Opteron_G3

x86-64-v3 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe

min intel: Haswell
min amd : EPYC_v1

x86-64-v4 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512vl

min intel: Skylake
min amd : EPYC_v4

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer/CPUConfig.pm | 48 +++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 5 deletions(-)

diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
index fb0861b..ac532fb 100644
--- a/PVE/QemuServer/CPUConfig.pm
+++ b/PVE/QemuServer/CPUConfig.pm
@@ -31,6 +31,25 @@ sub load_custom_model_conf {
     return cfs_read_file($default_filename);
 }
 
+my $builtin_models = {
+    'x86-64-v2' => {
+	'reported-model' => 'qemu64',
+	flags => "+popcnt;+pni;+sse4.1;+sse4.2;+ssse3",
+    },
+    'x86-64-v2-AES' => {
+	'reported-model' => 'qemu64',
+	flags => "+aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3",
+    },
+    'x86-64-v3' => {
+	'reported-model' => 'qemu64',
+	flags => "+aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe",
+    },
+    'x86-64-v4' => {
+	'reported-model' => 'qemu64',
+	flags => "+aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512vl",
+    },
+};
+
 my $depreacated_cpu_map = {
     # there never was such a client CPU, so map it to the server one for backward compat
     'Icelake-Client' => 'Icelake-Server',
@@ -212,7 +231,7 @@ sub validate_vm_cpu_conf {
 	get_custom_model($cputype);
     } else {
 	die "Built-in cputype '$cputype' is not defined (missing 'custom-' prefix?)\n"
-	    if !defined($cpu_vendor_list->{$cputype});
+	    if !defined($cpu_vendor_list->{$cputype}) && !defined($builtin_models->{$cputype});
     }
 
     # in a VM-specific config, certain properties are limited/forbidden
@@ -302,6 +321,17 @@ sub get_cpu_models {
 	};
     }
 
+    for my $model (keys %{$builtin_models}) {
+	my $reported_model = $builtin_models->{$model}->{'reported-model'};
+	$reported_model //= $cpu_fmt->{'reported-model'}->{default};
+	my $vendor = $cpu_vendor_list->{$reported_model};
+	push @$models, {
+	    name => $model,
+	    custom => 0,
+	    vendor => $vendor,
+	};
+    }
+
     return $models if !$include_custom;
 
     my $conf = load_custom_model_conf();
@@ -359,7 +389,9 @@ sub print_cpu_device {
 	    or die "Cannot parse cpu description: $cputype\n";
 	$cpu = $cpuconf->{cputype};
 
-	if (is_custom_model($cpu)) {
+	if (my $model = $builtin_models->{$cpu}) {
+	    $cpu = $model->{'reported-model'} // $cpu_fmt->{'reported-model'}->{default};
+	} elsif (is_custom_model($cputype)) {
 	    my $custom_cpu = get_custom_model($cpu);
 
 	    $cpu = $custom_cpu->{'reported-model'} // $cpu_fmt->{'reported-model'}->{default};
@@ -468,14 +500,17 @@ sub get_cpu_options {
 
     my $cpu = {};
     my $custom_cpu;
+    my $builtin_cpu;
     my $hv_vendor_id;
     if (my $cpu_prop_str = $conf->{cpu}) {
 	$cpu = PVE::JSONSchema::parse_property_string('pve-vm-cpu-conf', $cpu_prop_str)
 	    or die "Cannot parse cpu description: $cpu_prop_str\n";
 
 	$cputype = $cpu->{cputype};
-
-	if (is_custom_model($cputype)) {
+	if (my $model = $builtin_models->{$cputype}) {
+	    $cputype = $model->{'reported-model'} // $cpu_fmt->{'reported-model'}->{default};
+	    $builtin_cpu->{flags} = $model->{'flags'};
+	} elsif (is_custom_model($cputype)) {
 	    $custom_cpu = get_custom_model($cputype);
 
 	    $cputype = $custom_cpu->{'reported-model'} // $cpu_fmt->{'reported-model'}->{default};
@@ -504,6 +539,9 @@ sub get_cpu_options {
 	)
 	: undef;
 
+    my $builtin_cputype_flags = parse_cpuflag_list(
+	$cpu_flag_any_re, "set by builtin CPU model", $builtin_cpu->{flags});
+
     my $custom_cputype_flags = parse_cpuflag_list(
 	$cpu_flag_any_re, "set by custom CPU model", $custom_cpu->{flags});
 
@@ -534,7 +572,7 @@ sub get_cpu_options {
 
     # will be resolved in parameter order
     $cpu_str .= resolve_cpu_flags(
-	$pve_flags, $hv_flags, $custom_cputype_flags, $vm_flags, $pve_forced_flags);
+	$pve_flags, $hv_flags, $builtin_cputype_flags, $custom_cputype_flags, $vm_flags, $pve_forced_flags);
 
     my $phys_bits = '';
     foreach my $conf ($custom_cpu, $cpu) {
-- 
2.30.2

[pve-devel] [PATCH v2 pve-manager 1/1] qemu: processor : set x86-64-v2-AES as default cputype for create wizard

[Alexandre Derumier]

---
 www/manager6/qemu/OSDefaults.js    |  1 +
 www/manager6/qemu/ProcessorEdit.js | 13 +++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/www/manager6/qemu/OSDefaults.js b/www/manager6/qemu/OSDefaults.js
index 5e588a58..58bc76ff 100644
--- a/www/manager6/qemu/OSDefaults.js
+++ b/www/manager6/qemu/OSDefaults.js
@@ -43,6 +43,7 @@ Ext.define('PVE.qemu.OSDefaults', {
 		    virtio: 1,
 	    },
 	    scsihw: 'virtio-scsi-single',
+	    cputype: 'x86-64-v2-AES',
 	};
 
        // virtio-net is in kernel since 2.6.25
diff --git a/www/manager6/qemu/ProcessorEdit.js b/www/manager6/qemu/ProcessorEdit.js
index b845ff66..727d1679 100644
--- a/www/manager6/qemu/ProcessorEdit.js
+++ b/www/manager6/qemu/ProcessorEdit.js
@@ -29,6 +29,19 @@ Ext.define('PVE.qemu.ProcessorInputPanel', {
 
 	    viewModel.set('userIsRoot', Proxmox.UserName === 'root@pam');
 	},
+	control: {
+	    '#': {
+		afterrender: 'setCputype',
+	    },
+	},
+	setCputype: function() {
+	    let me = this;
+	    let view = me.getView();
+	    let cputype = view.down('CPUModelSelector[name=cputype]');
+	    if (view.insideWizard) {
+		cputype.setValue(PVE.qemu.OSDefaults.generic.cputype);
+	    }
+        },
     },
 
     onGetValues: function(values) {
-- 
2.30.2

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[Fiona Ebner]

Am 02.06.23 um 12:05 schrieb Alexandre Derumier:
> https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg01592.html
> "
> In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
> three microarchitecture levels on top of the historical x86-64
> baseline:
> 
>   * x86-64:    original x86_64 baseline instruction set
>   * x86-64-v2: vector instructions up to Streaming SIMD
>                Extensions 4.2 (SSE4.2)  and Supplemental
>                Streaming SIMD Extensions 3 (SSSE3), the
>                POPCNT instruction, and CMPXCHG16B
>   * x86-64-v3: vector instructions up to AVX2, MOVBE,
>                and additional bit-manipulation instructions.
>   * x86-64-v4: vector instructions from some of the
>                AVX-512 variants.
> "

Can we also link to https://gitlab.com/x86-psABIs/x86-64-ABI/ because
table 3.1 in the PDF contains a bit more precise information?

I used the following to test for some instructions, but feel free to
tell me something better ;)

> int main()
> {
>     __asm__ __volatile__
>     (
>         "xgetbv"
>     );
>     return 0;
> }


I also found
https://gist.github.com/rindeal/81198b1cf8f55c356743
and after adding #include <stdbool.h> to the cpuid-dump2.c it seemed to
work.

> 
> This patch add new builtin model derivated from qemu64 model,
> to be compatible between intel/amd.
> 
> x86-64-v1 : I'm skipping it, as it's basicaly qemu64|kvm64 -vme,-cx16 for compat Opteron_G1 from 2004
>             so will use it as qemu64|kvm64 is higher are not working on opteron_g1 anyway
> 
> x86-64-v2 : Derived from qemu, +popcnt;+pni;+sse4.1;+sse4.2;+ssse3
> 
> min intel: Nehalem
> min amd : Opteron_G3
> 
> x86-64-v2-AES : Derived from qemu, +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3

Could you describe how you chose/tested the flags?

From the table in the PDF, most flags are clear (with pni apparently
being SSE3). There are also CMPXCHG16B and LAHF-SAHF CPU
features/instructions, but they are already part of qemu64 AFAICT. At
least the example instructions cmpxchg16b (needs an operand) and lahf
seem to work with my little test program and the cpuid-dump2 program
also outputs -mcx16 -msahf indicating support.

> min intel: Westmere
> min amd : Opteron_G3
> 
> x86-64-v3 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe

Again comparing with the table in the PDF all flags are clear (with abm
adding the LZCNT feature/instruction).

But isn't the OSXSAVE feature missing? At least if I try with my little
test program above I get "illegal hardware instruction" for xgetbv (that
is the example instruction for the OSXSAVE CPU Feature mentioned in the
PDF) and the has_osxsave variable in the cpuid-dump2 program is also false.

> 
> min intel: Haswell
> min amd : EPYC_v1
> 
> x86-64-v4 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512vl
> 

Those match up exactly with the table in the PDF, so should be fine
(except the OSXSAVE feature will also be missing as v4 extends v3).

> min intel: Skylake
> min amd : EPYC_v4
> 

(...)

> @@ -302,6 +321,17 @@ sub get_cpu_models {
>  	};
>      }
>  
> +    for my $model (keys %{$builtin_models}) {
> +	my $reported_model = $builtin_models->{$model}->{'reported-model'};
> +	$reported_model //= $cpu_fmt->{'reported-model'}->{default};

Nit: we could just avoid this and similar fallbacks in the rest of the
patch, because we know that 'reported-model' is set for the built-in
models. But it's not a big deal and in a language like Perl we don't
have compile-time checks for it, so I'm fine with either way. If you
drop these fallbacks, please add a comment above $builtin_models stating
that 'reported-model' has to be set.

Re: [pve-devel] [PATCH v2 pve-manager 1/1] qemu: processor : set x86-64-v2-AES as default cputype for create wizard

[Fiona Ebner]

Am 02.06.23 um 12:05 schrieb Alexandre Derumier:
> ---

Missing your Signed-off-by

>  www/manager6/qemu/OSDefaults.js    |  1 +
>  www/manager6/qemu/ProcessorEdit.js | 13 +++++++++++++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/www/manager6/qemu/OSDefaults.js b/www/manager6/qemu/OSDefaults.js
> index 5e588a58..58bc76ff 100644
> --- a/www/manager6/qemu/OSDefaults.js
> +++ b/www/manager6/qemu/OSDefaults.js
> @@ -43,6 +43,7 @@ Ext.define('PVE.qemu.OSDefaults', {
>  		    virtio: 1,
>  	    },
>  	    scsihw: 'virtio-scsi-single',
> +	    cputype: 'x86-64-v2-AES',
>  	};
>  
>         // virtio-net is in kernel since 2.6.25
> diff --git a/www/manager6/qemu/ProcessorEdit.js b/www/manager6/qemu/ProcessorEdit.js
> index b845ff66..727d1679 100644
> --- a/www/manager6/qemu/ProcessorEdit.js
> +++ b/www/manager6/qemu/ProcessorEdit.js
> @@ -29,6 +29,19 @@ Ext.define('PVE.qemu.ProcessorInputPanel', {
>  
>  	    viewModel.set('userIsRoot', Proxmox.UserName === 'root@pam');
>  	},
> +	control: {
> +	    '#': {
> +		afterrender: 'setCputype',
> +	    },
> +	},
> +	setCputype: function() {
> +	    let me = this;
> +	    let view = me.getView();
> +	    let cputype = view.down('CPUModelSelector[name=cputype]');
> +	    if (view.insideWizard) {
> +		cputype.setValue(PVE.qemu.OSDefaults.generic.cputype);

You always use the generic one, but if we ever override cputype for a
more specific OSDefaults definition, it'll be wrong.

To avoid that, you can use getDefaults() like
www/manager6/qemu/NetworkEdit.js does:

>                     let ostype = me.vmconfig.ostype;
>                     let defaults = PVE.qemu.OSDefaults.getDefaults(ostype);
>                     let data = {
>                         model: defaults.networkCard,
>                     };
> 
>                     ipanel.setNetwork(me.confid, data);

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[DERUMIER, Alexandre]

Le mardi 06 juin 2023 à 14:09 +0200, Fiona Ebner a écrit :
> Am 02.06.23 um 12:05 schrieb Alexandre Derumier:
> > 
> > "
> > In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
> > three microarchitecture levels on top of the historical x86-64
> > baseline:
> > 
> >   * x86-64:    original x86_64 baseline instruction set
> >   * x86-64-v2: vector instructions up to Streaming SIMD
> >                Extensions 4.2 (SSE4.2)  and Supplemental
> >                Streaming SIMD Extensions 3 (SSSE3), the
> >                POPCNT instruction, and CMPXCHG16B
> >   * x86-64-v3: vector instructions up to AVX2, MOVBE,
> >                and additional bit-manipulation instructions.
> >   * x86-64-v4: vector instructions from some of the
> >                AVX-512 variants.
> > "
> 
> Can we also link to  because
> table 3.1 in the PDF contains a bit more precise information?
> 
> I used the following to test for some instructions, but feel free to
> tell me something better ;)
> 
I'm a pretty poor C developper, but that's seem right!
(To be honest, I didn't have even thinked to test instructions one by
one  )

> > int main()
> > {
> >     __asm__ __volatile__
> >     (
> >         "xgetbv"
> >     );
> >     return 0;
> > }
> 
> 
> I also found
> and after adding #include <stdbool.h> to the cpuid-dump2.c it seemed
> to
> work.
> 
> > 
> > This patch add new builtin model derivated from qemu64 model,
> > to be compatible between intel/amd.
> > 
> > x86-64-v1 : I'm skipping it, as it's basicaly qemu64|kvm64 -vme,-
> > cx16 for compat Opteron_G1 from 2004
> >             so will use it as qemu64|kvm64 is higher are not
> > working on opteron_g1 anyway
> > 
> > x86-64-v2 : Derived from qemu, +popcnt;+pni;+sse4.1;+sse4.2;+ssse3
> > 
> > min intel: Nehalem
> > min amd : Opteron_G3
> > 
> > x86-64-v2-AES : Derived from qemu,
> > +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3
> 
> Could you describe how you chose/tested the flags?

from the qemu doc generator
https://github.com/qemu/qemu/commit/4e2f5f3a9db06e87a73eb60a7cc9754fc13596ee


+# Mandatory CPUID features for each microarch ABI level
> +levels = [
> +    [ # x86-64 baseline
> +        "cmov",
> +        "cx8",
> +        "fpu",
> +        "fxsr",
> +        "mmx",
> +        "syscall",
> +        "sse",
> +        "sse2",
> +    ],
> +    [ # x86-64-v2
> +        "cx16",
> +        "lahf-lm",
> +        "popcnt",
> +        "pni",
> +        "sse4.1",
> +        "sse4.2",
> +        "ssse3",
> +    ],
> +    [ # x86-64-v3
> +        "avx",
> +        "avx2",
> +        "bmi1",
> +        "bmi2",
> +        "f16c",
> +        "fma",
> +        "abm",
> +        "movbe",
> +    ],
> +    [ # x86-64-v4
> +        "avx512f",
> +        "avx512bw",
> +        "avx512cd",
> +        "avx512dq",
> +        "avx512vl",
> +    ],
> +]
> 
> From the table in the PDF, most flags are clear (with pni apparently
> being SSE3). There are also CMPXCHG16B and LAHF-SAHF CPU
> features/instructions, but they are already part of qemu64 AFAICT. At
> least the example instructions cmpxchg16b (needs an operand) and lahf
> seem to work with my little test program and the cpuid-dump2 program
> also outputs -mcx16 -msahf indicating support.

yes, cx16 && lahf are already included in qemu64. (I have used qemu64
as it's more recent,and qemu don't recommend to use kvm64)


> 
> > min intel: Westmere
> > min amd : Opteron_G3
> > 
> > x86-64-v3 : Derived from qemu64
> > +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f1
> > 6c;+fma;+abm;+movbe
> 
> Again comparing with the table in the PDF all flags are clear (with
> abm
> adding the LZCNT feature/instruction).
> 
> But isn't the OSXSAVE feature missing? At least if I try with my
> little
> test program above I get "illegal hardware instruction" for xgetbv
> (that
> is the example instruction for the OSXSAVE CPU Feature mentioned in
> the
> PDF) and the has_osxsave variable in the cpuid-dump2 program is also
> false.

AFAIK, it's has been removed from qemu some years ago. (and I don't see
reference in other qemu models)

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1825195
https://listman.redhat.com/archives/libvir-list/2019-May/msg00274.html
> 
> > 
> > min intel: Haswell
> > min amd : EPYC_v1
> > 
> > x86-64-v4 : Derived from qemu64
> > +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f1
> > 6c;+fma;+abm;+movbe;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512v
> > l
> > 
> 
> Those match up exactly with the table in the PDF, so should be fine
> (except the OSXSAVE feature will also be missing as v4 extends v3).
> 
> > min intel: Skylake
> > min amd : EPYC_v4
> > 
> 
> (...)
> 
> > @@ -302,6 +321,17 @@ sub get_cpu_models {
> >         };
> >      }
> >  
> > +    for my $model (keys %{$builtin_models}) {
> > +       my $reported_model = $builtin_models->{$model}->{'reported-
> > model'};
> > +       $reported_model //= $cpu_fmt->{'reported-model'}-
> > >{default};
> 
> Nit: we could just avoid this and similar fallbacks in the rest of
> the
> patch, because we know that 'reported-model' is set for the built-in
> models. But it's not a big deal and in a language like Perl we don't
> have compile-time checks for it, so I'm fine with either way. If you
> drop these fallbacks, please add a comment above $builtin_models
> stating
> that 'reported-model' has to be set.
> 

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[Fiona Ebner]

Am 06.06.23 um 15:36 schrieb DERUMIER, Alexandre:
> Le mardi 06 juin 2023 à 14:09 +0200, Fiona Ebner a écrit :
>> Am 02.06.23 um 12:05 schrieb Alexandre Derumier:
>>>
>>> "
>>> In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
>>> three microarchitecture levels on top of the historical x86-64
>>> baseline:
>>>
>>>   * x86-64:    original x86_64 baseline instruction set
>>>   * x86-64-v2: vector instructions up to Streaming SIMD
>>>                Extensions 4.2 (SSE4.2)  and Supplemental
>>>                Streaming SIMD Extensions 3 (SSSE3), the
>>>                POPCNT instruction, and CMPXCHG16B
>>>   * x86-64-v3: vector instructions up to AVX2, MOVBE,
>>>                and additional bit-manipulation instructions.
>>>   * x86-64-v4: vector instructions from some of the
>>>                AVX-512 variants.
>>> "
>>
>> Can we also link to  because
>> table 3.1 in the PDF contains a bit more precise information?
>>
>> I used the following to test for some instructions, but feel free to
>> tell me something better ;)
>>
> I'm a pretty poor C developper, but that's seem right!
> (To be honest, I didn't have even thinked to test instructions one by
> one  )
> 

I just tested the example instruction from the PDF for the features that
did not obviously correspond to a certain flag.

>> Could you describe how you chose/tested the flags?
> 
> from the qemu doc generator
> https://github.com/qemu/qemu/commit/4e2f5f3a9db06e87a73eb60a7cc9754fc13596ee
> 
> 
> +# Mandatory CPUID features for each microarch ABI level
>> +levels = [
>> +    [ # x86-64 baseline
>> +        "cmov",
>> +        "cx8",
>> +        "fpu",
>> +        "fxsr",
>> +        "mmx",
>> +        "syscall",
>> +        "sse",
>> +        "sse2",
>> +    ],
>> +    [ # x86-64-v2
>> +        "cx16",
>> +        "lahf-lm",
>> +        "popcnt",
>> +        "pni",
>> +        "sse4.1",
>> +        "sse4.2",
>> +        "ssse3",
>> +    ],
>> +    [ # x86-64-v3
>> +        "avx",
>> +        "avx2",
>> +        "bmi1",
>> +        "bmi2",
>> +        "f16c",
>> +        "fma",
>> +        "abm",
>> +        "movbe",
>> +    ],
>> +    [ # x86-64-v4
>> +        "avx512f",
>> +        "avx512bw",
>> +        "avx512cd",
>> +        "avx512dq",
>> +        "avx512vl",
>> +    ],
>> +]
>>

Okay, great!

>>> x86-64-v3 : Derived from qemu64
>>> +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f1
>>> 6c;+fma;+abm;+movbe
>>
>> Again comparing with the table in the PDF all flags are clear (with
>> abm
>> adding the LZCNT feature/instruction).
>>
>> But isn't the OSXSAVE feature missing? At least if I try with my
>> little
>> test program above I get "illegal hardware instruction" for xgetbv
>> (that
>> is the example instruction for the OSXSAVE CPU Feature mentioned in
>> the
>> PDF) and the has_osxsave variable in the cpuid-dump2 program is also
>> false.
> 
> AFAIK, it's has been removed from qemu some years ago. (and I don't see
> reference in other qemu models)
> 
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1825195
> https://listman.redhat.com/archives/libvir-list/2019-May/msg00274.html

The commit removing it mentions that it is a dynamic feature

> commit f1a23522b03a569f13aad49294bb4c4b1a9500c7
> Author: Eduardo Habkost <ehabkost@redhat.com>
> Date:   Mon Jun 11 17:38:55 2018 -0300
> 
>     i386: Remove osxsave CPUID flag name
>     
>     OSXAVE is not a static feature flag: it changes dynamically at
>     runtime depending on CR4, and it was never configurable: KVM
>     never returned OSXSAVE on GET_SUPPORTED_CPUID, and it is not
>     included in TCG_EXT_FEATURES.
>     
>     Remove OSXSAVE from the feature name array so users don't try to
>     configure it manually.

which according to
https://www.gnu.org/software/libc/manual/html_node/X86.html needs to be
enabled by the OS:

> OSXSAVE – The OS has set CR4.OSXSAVE[bit 18] to enable XSETBV/XGETBV instructions to access XCR0 and to support processor extended state management using XSAVE/XRSTOR.
But my Debian 11 guest apparently doesn't do it when using x86-64-v3.
Then I get "illegal hardware instruction" for xgetbv. If I use CPU type
host on the other hand, I don't get that error.

So I checked in the kernel sources and found that apparently depends on
the xsave feature. In arch/x86/kernel/fpu/xstate.c:

> void fpu__init_cpu_xstate(void) 
> {   
>     if (!boot_cpu_has(X86_FEATURE_XSAVE) || !fpu_kernel_cfg.max_features)
>         return;
> 
>     cr4_set_bits(X86_CR4_OSXSAVE);

and sure enough, adding +xsave to the flags for x86-64-v3 makes the
"illegal hardware instruction" error for xgetbv go away.

So while the QEMU docs don't mention the xsave flag, it seems necessary
to be in-line with the ABI definition?

In the table "CPUID EAX=1: Feature Information in EDX and ECX" in
https://en.wikipedia.org/wiki/CPUID#EAX=1:_Processor_Info_and_Feature_Bits

bit 26 of ECX is xsave which does list the XGETBV instruction
bit 27 of ECX is osxsave which just says "XSAVE enabled by OS"

So again, sounds like the xsave feature is required for the osxsave
feature to even make sense.

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[DERUMIER, Alexandre]

> 
> So again, sounds like the xsave feature is required for the osxsave
> feature to even make sense.
> 
The problem is that xsaves is not supported on all intel/amd cpu, 

older model don't seem to support xsaves too  (intel only since
sandybridge , amd since epyc v1)
https://github.com/qemu/qemu/blob/master/target/i386/cpu.c#L3050



but also newer model depending of firmware revision

"target/i386: EPYC-Rome model without XSAVES"
https://github.com/qemu/qemu/commit/fb00aa61267c8b9c57a2d1a1fa1e336d02e3bcd1

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[DERUMIER, Alexandre]

> 
> but also newer model depending of firmware revision
> 
> "target/i386: EPYC-Rome model without XSAVES"
> https://github.com/qemu/qemu/commit/fb00aa61267c8b9c57a2d1a1fa1e336d02e3bcd1
> 
> 
> 
seem that linux kernel have disable it recently for zen1/2

https://www.spinics.net/lists/linux-tip-commits/msg61863.html

+	/*
+	 * Work around Erratum 1386.  The XSAVES instruction
malfunctions in
+	 * certain circumstances on Zen1/2 uarch, and not all parts
have had
+	 * updated microcode at the time of writing (March 2023).
+	 *
+	 * Affected parts all have no supervisor XSAVE states, meaning
that
+	 * the XSAVEC instruction (which works fine) is equivalent.
+	 */
+	clear_cpu_cap(c, X86_FEATURE_XSAVES);
 }
 

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[Fiona Ebner]

Am 07.06.23 um 12:57 schrieb DERUMIER, Alexandre:
>>
>> So again, sounds like the xsave feature is required for the osxsave
>> feature to even make sense.
>>
> The problem is that xsaves is not supported on all intel/amd cpu, 
> 
> older model don't seem to support xsaves too  (intel only since
> sandybridge , amd since epyc v1)
> https://github.com/qemu/qemu/blob/master/target/i386/cpu.c#L3050
> 
> 
> 
> but also newer model depending of firmware revision
> 
> "target/i386: EPYC-Rome model without XSAVES"
> https://github.com/qemu/qemu/commit/fb00aa61267c8b9c57a2d1a1fa1e336d02e3bcd1
> 

AFAIU, xsaves with the additional s is just an optional sub-feature of
xsave:
https://en.wikipedia.org/wiki/CPUID#EAX%3D0Dh%3A_XSAVE_features_and_state-components

Note that QEMU has both, a xsave and an xsaves CPU flag.

If I run the cpuid-dump2.c script in my VM with

-cpu
qemu64,+abm,+aes,+avx,+avx2,+bmi1,+bmi2,enforce,+f16c,+fma,+kvm_pv_eoi,+kvm_pv_unhalt,+movbe,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,xsave

I get

cpu=bdver4 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a
-mcx16 -msahf -mmovbe -maes -mno-sha -mno-pclmul -mpopcnt -mabm -mno-lwp
-mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2
-msse4.1 -mlzcnt -mno-rtm -mno-hle -mno-rdrnd -mf16c -mno-fsgsbase
-mno-rdseed -mno-prfchw -mno-adx -mfxsr

here -> -mxsave

-mno-xsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf
-mno-prefetchwt1 -mno-clflushopt -mno-xsavec

here -> -mno-xsaves

-mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma
-mno-avx512vbmi -mno-clwb -mno-pcommit

So there is -mxsave, but also -mno-xsaves.

But couldn't find from which models onward xsave is supported from a
quick search.

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[DERUMIER, Alexandre]

Le mercredi 07 juin 2023 à 13:48 +0200, Fiona Ebner a écrit :
> Am 07.06.23 um 12:57 schrieb DERUMIER, Alexandre:
> > > 
> > > So again, sounds like the xsave feature is required for the
> > > osxsave
> > > feature to even make sense.
> > > 
> > The problem is that xsaves is not supported on all intel/amd cpu, 
> > 
> > older model don't seem to support xsaves too  (intel only since
> > sandybridge , amd since epyc v1)
> > https://antiphishing.cetsi.fr/proxy/v3?i=U2pXU09ocHltdTEydGM2aWb2SGhafebCBTjK3jebDSU&r=SjA3d003VWxKRk1kazNaeRJgzukDmhQdY5g-DacBRkZ4pgKdvLOyt2Z87havu-ae7CZLNw-FYpOPxDnH4AVQTw&f=Rld2eGhGQ3psZjlOWGwxQ0mBfBn7v4P9N5oaI0-k6dG5NatpgnMGXGhApgyzFe2H&u=https%3A//github.com/qemu/qemu/blob/master/target/i386/cpu.c%23L3050&k=1XpP
> > 
> > 
> > 
> > but also newer model depending of firmware revision
> > 
> > "target/i386: EPYC-Rome model without XSAVES"
> > https://antiphishing.cetsi.fr/proxy/v3?i=U2pXU09ocHltdTEydGM2aWb2SGhafebCBTjK3jebDSU&r=SjA3d003VWxKRk1kazNaeRJgzukDmhQdY5g-DacBRkZ4pgKdvLOyt2Z87havu-ae7CZLNw-FYpOPxDnH4AVQTw&f=Rld2eGhGQ3psZjlOWGwxQ0mBfBn7v4P9N5oaI0-k6dG5NatpgnMGXGhApgyzFe2H&u=https%3A//github.com/qemu/qemu/commit/fb00aa61267c8b9c57a2d1a1fa1e336d02e3bcd1&k=1XpP
> > 
> 
> AFAIU, xsaves with the additional s is just an optional sub-feature
> of
> xsave:
> https://antiphishing.cetsi.fr/proxy/v3?i=U2pXU09ocHltdTEydGM2aWb2SGhafebCBTjK3jebDSU&r=SjA3d003VWxKRk1kazNaeRJgzukDmhQdY5g-DacBRkZ4pgKdvLOyt2Z87havu-ae7CZLNw-FYpOPxDnH4AVQTw&f=Rld2eGhGQ3psZjlOWGwxQ0mBfBn7v4P9N5oaI0-k6dG5NatpgnMGXGhApgyzFe2H&u=https%3A//en.wikipedia.org/wiki/CPUID%23EAX%253D0Dh%253A_XSAVE_features_and_state-components&k=1XpP
> 
> Note that QEMU has both, a xsave and an xsaves CPU flag.
> 
> If I run the cpuid-dump2.c script in my VM with
> 
> -cpu
> qemu64,+abm,+aes,+avx,+avx2,+bmi1,+bmi2,enforce,+f16c,+fma,+kvm_pv_eo
> i,+kvm_pv_unhalt,+movbe,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,xsave
> 
> I get
> 
> cpu=bdver4 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a
> -mcx16 -msahf -mmovbe -maes -mno-sha -mno-pclmul -mpopcnt -mabm -mno-
> lwp
> -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2
> -msse4.1 -mlzcnt -mno-rtm -mno-hle -mno-rdrnd -mf16c -mno-fsgsbase
> -mno-rdseed -mno-prfchw -mno-adx -mfxsr
> 
> here -> -mxsave
> 
> -mno-xsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf
> -mno-prefetchwt1 -mno-clflushopt -mno-xsavec
> 
> here -> -mno-xsaves
> 
> -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma
> -mno-avx512vbmi -mno-clwb -mno-pcommit
> 
> So there is -mxsave, but also -mno-xsaves.
> 
oh ok, got it ! 

> But couldn't find from which models onward xsave is supported from a
> quick search.
> 
you could find it here:
https://github.com/qemu/qemu/blob/master/target/i386/cpu.c

CPUID_EXT_XSAVE

>= intel sandybridge

>= amd Opteron_G4


so for -v3 & -v4  models, it's not a problem.

Re: [pve-devel] [PATCH v4 qemu-server 1/1] cpuconfig: add new x86-64-vX models

[DERUMIER, Alexandre]

> 
> > But couldn't find from which models onward xsave is supported from
> > a
> > quick search.
> > 
> you could find it here:
> https://github.com/qemu/qemu/blob/master/target/i386/cpu.c
> 
> CPUID_EXT_XSAVE
> 
> > = intel sandybridge
> 
> > = amd Opteron_G4
> 
> 
> so for -v3 & -v4  models, it's not a problem.
> 
> 
> 
> 
more references in gcc

i386: Add support for x86-64-v2, x86-64-v3, x86-64-v4 levels for x86-64
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=324bec558e95584e8c1997575ae9d75978af59f1

with flags for

v2:

https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=324bec558e95584e8c1997575ae9d75978af59f1#patch5

v3:
https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=324bec558e95584e8c1997575ae9d75978af59f1#patch8

v4:

https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=324bec558e95584e8c1997575ae9d75978af59f1#patch9


also the list of supported extensions by cpumodel is available here:

https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html