[pve-devel] [PATCH qemu-server] QEMU AMD SEV enable

[pve-devel] [PATCH qemu-server] QEMU AMD SEV enable

[Markus Frank]

This Patch is for enabling AMD SEV (Secure Encrypted Virtualization) support in
QEMU and for supporting other memory encryption technologies like INTEL MKTME
(Multi-key Total Memory Encryption) and AMD-SNP in the future.

Config-Example:
memory_encryption: type=sev,cbitpos=47,policy=0x0005,reduced-phys-bits=1

"reduced-phys-bios" and "cbitpos" are system specific and can be read out with
QMP. If not set by the user, a dummy-vm gets started to read QMP for these
variables out and save them to config. Afterwards, the dummy-vm gets stopped.

For a more detailed Explanation plus Requirements & Limitations
see my coherent pve-docs patch and the qemu documentation.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
I could not test SEV-ES because I get a similar error to
https://www.mail-archive.com/devel@edk2.groups.io/msg38521.html
But I still get the same error on master and mentioned patched versions.
On some older versions I just get "kvm: SEV-ES reset address is zero
kvm: failed to locate and/or save reset vector"

Maybe I will report my error to the edk2 project or find a way to fix
it, when I know more about it.

I also could not test SEV-SNP, because I do not have a EPYC 7003 to test
on and there is also no support in the Linux Kernel yet. SEV-SNP support
should be in Linux 5.19:
https://www.phoronix.com/scan.php?page=news_item&px=AMD-SEV-SNP-Arrives-Linux-5.19
patched kernel-fork: https://github.com/AMDESE/linux/tree/sev-snp-5.18-rc3

 PVE/QemuServer.pm | 133 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index e9aa248..abc21d4 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -174,6 +174,55 @@ my $agent_fmt = {
     },
 };
 
+my $memory_encryption_fmt = {
+    type => {
+	type => 'string',
+	default_key => 1,
+	description => "Memory Encryption Type:"
+	    ."for AMD SEV -> 'memory_encryption: type=sev'"
+	    ."for AMD SEV-SNP -> 'memory_encryption: type=sev-snp'"
+	    ."for AMD SEV-ES -> use 'sev' and change policy to between 0x4 and 0x7"
+	    ."(Bit-2 has to be set 1 (LSB 0 bit numbering))"
+	    ."(sev requires edk2-ovmf & on guest: up-to-date kernel + sev support &"
+	    ."on host: add kernel-parameters 'mem_encrypt=on kvm_amd.sev=1')"
+	    ."see https://github.com/AMDESE/AMDSEV",
+	format_description => "qemu-memory-encryption-type",
+	#pattern => '(sev|sev-snp|mktme)',
+	pattern => '(sev|sev-snp)',
+	default => "",
+	maxLength => 10,
+    },
+    'reduced-phys-bits' => {
+	description => "Number of bits the physical address space is reduced by. System dependent",
+	type => 'integer',
+	default => 1,
+	optional => 1,
+	minium => 0,
+	maximum => 100,
+    },
+    cbitpos => {
+	description => "C-bit: marks if a memory page is protected. System dependent",
+	type => 'integer',
+	default => 47,
+	optional => 1,
+	minium => 0,
+	maximum => 100,
+    },
+    policy => {
+	description => "SEV Guest Policy"
+	    ."see Capter 3:"
+	    ."https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf"
+	    ."& https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html",
+	format_description => "qemu-memory-encryption-policy",
+	type => 'string',
+	default => '0x0000',
+	pattern => '0[xX][0-9a-fA-F]{1,4}',
+	optional => 1,
+	maxLength => 6,
+    },
+};
+PVE::JSONSchema::register_format('pve-qemu-memory-encryption-fmt', $memory_encryption_fmt);
+
 my $vga_fmt = {
     type => {
 	description => "Select the VGA type.",
@@ -348,6 +397,12 @@ my $confdesc = {
 	minimum => 16,
 	default => 512,
     },
+    memory_encryption => {
+	description => "Memory Encryption",
+	optional => 1,
+	format => 'pve-qemu-memory-encryption-fmt',
+	type => 'string',
+    },
     balloon => {
 	optional => 1,
 	type => 'integer',
@@ -2107,6 +2162,16 @@ sub parse_guest_agent {
     return $res;
 }
 
+sub parse_memory_encryption {
+    my ($value) = @_;
+
+    return if !$value;
+
+    my $res = eval { parse_property_string($memory_encryption_fmt, $value) };
+    warn $@ if $@;
+    return $res;
+}
+
 sub get_qga_key {
     my ($conf, $key) = @_;
     return undef if !defined($conf->{agent});
@@ -4079,6 +4144,74 @@ sub config_to_command {
     }
     push @$machineFlags, "type=${machine_type_min}";
 
+    # Memory Encryption configuration
+    my $memory_encryption = parse_memory_encryption($conf->{'memory_encryption'});
+
+    # Die if bios is not ovmf
+    if (
+	$conf->{'memory_encryption'}
+	&& $memory_encryption->{type} eq 'sev'
+	&& !$conf->{bios} eq 'ovmf'
+    ) {
+	die "SEV needs ovmf";
+    }
+
+    # AMD SEV
+    if ($conf->{'memory_encryption'} && $memory_encryption->{type} =~ /(sev|sev-snp)/) {
+	# Get reduced-phys-bits & cbitpos from QMP, if not set
+	if (
+	    !$memory_encryption->{'reduced-phys-bits'}
+	    || !$memory_encryption->{cbitpos}
+	) {
+	    my $fakevmid = -1;
+	    my $qemu_cmd = get_command_for_arch($arch);
+	    my $pidfile = PVE::QemuServer::Helpers::pidfile_name($fakevmid);
+	    my $default_machine = $default_machines->{$arch};
+	    my $cmd = [
+		$qemu_cmd,
+		'-machine', $default_machine,
+		'-display', 'none',
+		'-chardev', "socket,id=qmp,path=/var/run/qemu-server/$fakevmid.qmp,server=on,wait=off",
+		'-mon', 'chardev=qmp,mode=control',
+		'-pidfile', $pidfile,
+		'-S', '-daemonize'
+	    ];
+	    if (!$kvm) {
+		push @$cmd, '-accel', 'tcg';
+	    }
+	    my $rc = run_command($cmd, noerr => 1, quiet => 0);
+	    die "QEMU flag querying VM exited with code " . $rc if $rc;
+	    my $res = mon_cmd($fakevmid, 'query-sev-capabilities');
+	    $memory_encryption->{'reduced-phys-bits'} = $res->{'reduced-phys-bits'};
+	    $memory_encryption->{cbitpos} = $res->{cbitpos};
+	    $conf->{'memory_encryption'} = PVE::JSONSchema::print_property_string(
+		$memory_encryption,
+		$memory_encryption_fmt
+	    );
+	    PVE::QemuConfig->write_config($vmid, $conf);
+	    vm_stop(undef, $fakevmid, 1, 1, 10, 0, 1);
+	}
+
+	# qemu-Example: -object 'sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1';
+	# see https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html
+	my $memobjcmd = "";
+	if ($memory_encryption->{type} eq 'sev-snp') {
+	    # https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel
+	    $memobjcmd = $memobjcmd.'sev-snp-guest,';
+	} else {
+	    $memobjcmd = $memobjcmd.'sev-guest,';
+	}
+	$memobjcmd = $memobjcmd . 'id=sev0,cbitpos='.$memory_encryption->{cbitpos}
+	    .',reduced-phys-bits='.$memory_encryption->{'reduced-phys-bits'};
+	if ($memory_encryption->{type} eq 'sev' && $memory_encryption->{policy}) {
+	    $memobjcmd = $memobjcmd.',policy='.$memory_encryption->{policy}
+	}
+	push @$devices, '-object' , $memobjcmd;
+	# old qemu-Example: -machine 'type=q35+pve0,memory-encryption=sev0'
+	# https://fossies.org/linux/qemu/docs/system/confidential-guest-support.rst
+	push @$machineFlags, 'confidential-guest-support=sev0';
+    }
+
     push @$cmd, @$devices;
     push @$cmd, '-rtc', join(',', @$rtcFlags) if scalar(@$rtcFlags);
     push @$cmd, '-machine', join(',', @$machineFlags) if scalar(@$machineFlags);
-- 
2.30.2

[pve-devel] [PATCH pve-docs] added Memory Encryption documentation

[Markus Frank]

added AMD SEV documentation for "[PATCH qemu-server] QEMU AMD SEV
enable"

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 qm.adoc | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/qm.adoc b/qm.adoc
index e666d7d..027d0a1 100644
--- a/qm.adoc
+++ b/qm.adoc
@@ -583,6 +583,65 @@ systems.
 When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB
 of RAM available to the host.
 
+[[qm_memory_encryption]]
+Memory Encryption
+~~~~~~~~~~~~~~~~~
+
+AMD SEV
+^^^^^^^
+
+Memory Encryption using AES-128 Encryption and the AMD Secure Processor.
+See https://developer.amd.com/sev/[AMD SEV]
+
+Requirements:
+
+* AMD EPYC/Ryzen PRO CPU
+* configured SEV BIOS Settings on Host Machine
+* add Kernel Parameters: "mem_encrypt=on kvm_amd.sev=1"
+
+Example Configuration:
+
+----
+# qm set <vmid> -memory_encryption type=sev,cbitpos=47,policy=0x0005,reduced-phys-bits=1
+----
+
+"type" defines the encryption technology ("type=" is not necessary): sev, sev-snp, mktme
+
+"reduced-phys-bios", "cbitpos" and "policy" correspond to the variables with the
+same name in qemu.
+
+"reduced-phys-bios" and "cbitpos" are system specific and can be read out
+with QMP. If not set, qm starts a dummy-vm to read QMP
+for these variables out and saves them to config.
+
+"policy" can be calculated with
+https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf[AMD SEV API Specification Chapter 3]
+
+To use SEV-ES (CPU register encryption) the "policy" should be set
+somewhere between 0x4 and 0x7 or 0xC and 0xF, etc.
+(Bit-2 has to be set 1 (LSB 0 bit numbering))
+
+Limitations:
+
+* Memory usage on host is always wrong and around 82% Usage
+* Snapshots do not work
+* edk2-OVMF required
+* Recommendable: VirtIO RNG for more entropy (VMs sometimes will not
+boot without)
+
+Links:
+
+* https://github.com/AMDESE/AMDSEV
+* https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html
+* https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf
+
+AMD SEV-SNP
+^^^^^^^^^^^
+
+* SEV-SNP support is not in the Linux Kernel yet and needs EPYC 7003 "Milan"
+processors.
+* SEV-SNP should be in Kernel 5.19: https://www.phoronix.com/scan.php?page=news_item&px=AMD-SEV-SNP-Arrives-Linux-5.19
+* patched Kernel: https://github.com/AMDESE/linux/tree/sev-snp-5.18-rc3
 
 [[qm_network_device]]
 Network Device
-- 
2.30.2

Re: [pve-devel] [PATCH qemu-server] QEMU AMD SEV enable

[Daniel Tschlatscher]

The SEV memory encryption works without problems.
Parameters for 'cbitpos' and 'reduced-phys-bits' were successfully and
automatically applied, when not supplied in the config.
I could run up to 15 SEV-enabled VMs concurrently, which is in line with
the expected maximum capabilities of a 1st gen Epyc CPU.
Memory settings in the range of 512MiB to 56GiB all worked for me
without problems.

Tested on an AMD Epyc 7351P with a SuperMicro H11SSL-i Motherboard (Bios
Revision 1.3)


I tried to get SEV-ES running but even though my CPU signals support for
it, starting VMs with the feature enabled always yields following error:

> kvm: sev_kvm_init: failed to initialize ret=-25 fw_error=0 ''
> kvm: failed to initialize kvm: Operation not permitted
> kvm: falling back to tcg
> kvm: ../softmmu/vl.c:2738: qemu_machine_creation_done: Assertion
`machine->cgs->ready' failed.

I suspect that this feature is simply not supported by my
motherboard/BIOS yet, but did not find the actual cause for this problem.


I did not test SEV-SNP as it seems to be a 3rd gen Epyc CPU feature.


Tested-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>


Some more comments inline in the code, all of which are non-breaking though:

On 6/9/22 13:14, Markus Frank wrote:
> This Patch is for enabling AMD SEV (Secure Encrypted Virtualization) support in
> QEMU and for supporting other memory encryption technologies like INTEL MKTME
> (Multi-key Total Memory Encryption) and AMD-SNP in the future.
> 
> Config-Example:
> memory_encryption: type=sev,cbitpos=47,policy=0x0005,reduced-phys-bits=1
> 
> "reduced-phys-bios" and "cbitpos" are system specific and can be read out with
> QMP. If not set by the user, a dummy-vm gets started to read QMP for these
> variables out and save them to config. Afterwards, the dummy-vm gets stopped.
> 
> For a more detailed Explanation plus Requirements & Limitations
> see my coherent pve-docs patch and the qemu documentation.
> 
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
> I could not test SEV-ES because I get a similar error to
> https://www.mail-archive.com/devel@edk2.groups.io/msg38521.html
> But I still get the same error on master and mentioned patched versions.
> On some older versions I just get "kvm: SEV-ES reset address is zero
> kvm: failed to locate and/or save reset vector"
> 
> Maybe I will report my error to the edk2 project or find a way to fix
> it, when I know more about it.
> 
> I also could not test SEV-SNP, because I do not have a EPYC 7003 to test
> on and there is also no support in the Linux Kernel yet. SEV-SNP support
> should be in Linux 5.19:
> https://www.phoronix.com/scan.php?page=news_item&px=AMD-SEV-SNP-Arrives-Linux-5.19
> patched kernel-fork: https://github.com/AMDESE/linux/tree/sev-snp-5.18-rc3
> 
>  PVE/QemuServer.pm | 133 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 133 insertions(+)
> 
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index e9aa248..abc21d4 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -174,6 +174,55 @@ my $agent_fmt = {
>      },
>  };
>  
> +my $memory_encryption_fmt = {
> +    type => {
> +	type => 'string',
> +	default_key => 1,
> +	description => "Memory Encryption Type:"
> +	    ."for AMD SEV -> 'memory_encryption: type=sev'"
> +	    ."for AMD SEV-SNP -> 'memory_encryption: type=sev-snp'"
> +	    ."for AMD SEV-ES -> use 'sev' and change policy to between 0x4 and 0x7"
> +	    ."(Bit-2 has to be set 1 (LSB 0 bit numbering))"
> +	    ."(sev requires edk2-ovmf & on guest: up-to-date kernel + sev support &"
> +	    ."on host: add kernel-parameters 'mem_encrypt=on kvm_amd.sev=1')"
> +	    ."see https://github.com/AMDESE/AMDSEV",
> +	format_description => "qemu-memory-encryption-type",
> +	#pattern => '(sev|sev-snp|mktme)',
> +	pattern => '(sev|sev-snp)',
> +	default => "",
> +	maxLength => 10,
> +    },
> +    'reduced-phys-bits' => {
> +	description => "Number of bits the physical address space is reduced by. System dependent",
> +	type => 'integer',
> +	default => 1,
> +	optional => 1,
> +	minium => 0,
Says "minium" instead of "minimum"
> +	maximum => 100,
> +    },
> +    cbitpos => {
> +	description => "C-bit: marks if a memory page is protected. System dependent",
> +	type => 'integer',
> +	default => 47,
> +	optional => 1,
> +	minium => 0,
Same as above
> +	maximum => 100,
> +    },
> +    policy => {
> +	description => "SEV Guest Policy"
> +	    ."see Capter 3:"
> +	    ."https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf"
> +	    ."& https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html",
> +	format_description => "qemu-memory-encryption-policy",
> +	type => 'string',
> +	default => '0x0000',
> +	pattern => '0[xX][0-9a-fA-F]{1,4}',
> +	optional => 1,
> +	maxLength => 6,
> +    },
> +};
> +PVE::JSONSchema::register_format('pve-qemu-memory-encryption-fmt', $memory_encryption_fmt);
> +
>  my $vga_fmt = {
>      type => {
>  	description => "Select the VGA type.",
> @@ -348,6 +397,12 @@ my $confdesc = {
>  	minimum => 16,
>  	default => 512,
>      },
> +    memory_encryption => {
> +	description => "Memory Encryption",
> +	optional => 1,
> +	format => 'pve-qemu-memory-encryption-fmt',
> +	type => 'string',
> +    },
>      balloon => {
>  	optional => 1,
>  	type => 'integer',
> @@ -2107,6 +2162,16 @@ sub parse_guest_agent {
>      return $res;
>  }
>  
> +sub parse_memory_encryption {
> +    my ($value) = @_;
> +
> +    return if !$value;
> +
> +    my $res = eval { parse_property_string($memory_encryption_fmt, $value) };
> +    warn $@ if $@;
> +    return $res;
> +}
> +
>  sub get_qga_key {
>      my ($conf, $key) = @_;
>      return undef if !defined($conf->{agent});
> @@ -4079,6 +4144,74 @@ sub config_to_command {
>      }
>      push @$machineFlags, "type=${machine_type_min}";
>  
> +    # Memory Encryption configuration
> +    my $memory_encryption = parse_memory_encryption($conf->{'memory_encryption'});
> +
> +    # Die if bios is not ovmf
> +    if (
> +	$conf->{'memory_encryption'}
> +	&& $memory_encryption->{type} eq 'sev'
> +	&& !$conf->{bios} eq 'ovmf'
!$conf->{bios} eq 'ovmf' will never yield true (did you forget the
parenthesis?) and die with the error
This creates a problem where the VM status changes to running but
noVNC will forever show "Guest has not initialized display yet" when run
with anything other than OVMF
> +    ) {
> +	die "SEV needs ovmf";
Nit: \n at the end here to suppress the "at .../QemuServer.pm line xxxx"
message
> +    }
> +
> +    # AMD SEV
> +    if ($conf->{'memory_encryption'} && $memory_encryption->{type} =~ /(sev|sev-snp)/) {
> +	# Get reduced-phys-bits & cbitpos from QMP, if not set
> +	if (
> +	    !$memory_encryption->{'reduced-phys-bits'}
> +	    || !$memory_encryption->{cbitpos}
> +	) {
> +	    my $fakevmid = -1;
> +	    my $qemu_cmd = get_command_for_arch($arch);
> +	    my $pidfile = PVE::QemuServer::Helpers::pidfile_name($fakevmid);
> +	    my $default_machine = $default_machines->{$arch};
> +	    my $cmd = [
> +		$qemu_cmd,
> +		'-machine', $default_machine,
> +		'-display', 'none',
> +		'-chardev', "socket,id=qmp,path=/var/run/qemu-server/$fakevmid.qmp,server=on,wait=off",
> +		'-mon', 'chardev=qmp,mode=control',
> +		'-pidfile', $pidfile,
> +		'-S', '-daemonize'
> +	    ];
> +	    if (!$kvm) {
> +		push @$cmd, '-accel', 'tcg';
> +	    }
> +	    my $rc = run_command($cmd, noerr => 1, quiet => 0);
> +	    die "QEMU flag querying VM exited with code " . $rc if $rc;
> +	    my $res = mon_cmd($fakevmid, 'query-sev-capabilities');
> +	    $memory_encryption->{'reduced-phys-bits'} = $res->{'reduced-phys-bits'};
> +	    $memory_encryption->{cbitpos} = $res->{cbitpos};
> +	    $conf->{'memory_encryption'} = PVE::JSONSchema::print_property_string(
> +		$memory_encryption,
> +		$memory_encryption_fmt
> +	    );
> +	    PVE::QemuConfig->write_config($vmid, $conf);
> +	    vm_stop(undef, $fakevmid, 1, 1, 10, 0, 1);
> +	}
> +
> +	# qemu-Example: -object 'sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1';
> +	# see https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html
> +	my $memobjcmd = "";
> +	if ($memory_encryption->{type} eq 'sev-snp') {
> +	    # https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel
> +	    $memobjcmd = $memobjcmd.'sev-snp-guest,';
> +	} else {
> +	    $memobjcmd = $memobjcmd.'sev-guest,';
> +	}
> +	$memobjcmd = $memobjcmd . 'id=sev0,cbitpos='.$memory_encryption->{cbitpos}
> +	    .',reduced-phys-bits='.$memory_encryption->{'reduced-phys-bits'};
> +	if ($memory_encryption->{type} eq 'sev' && $memory_encryption->{policy}) {
> +	    $memobjcmd = $memobjcmd.',policy='.$memory_encryption->{policy}
> +	}
> +	push @$devices, '-object' , $memobjcmd;
> +	# old qemu-Example: -machine 'type=q35+pve0,memory-encryption=sev0'
> +	# https://fossies.org/linux/qemu/docs/system/confidential-guest-support.rst
> +	push @$machineFlags, 'confidential-guest-support=sev0';
> +    }
> +
>      push @$cmd, @$devices;
>      push @$cmd, '-rtc', join(',', @$rtcFlags) if scalar(@$rtcFlags);
>      push @$cmd, '-machine', join(',', @$machineFlags) if scalar(@$machineFlags);

Re: [pve-devel] [PATCH qemu-server] QEMU AMD SEV enable

[Thomas Lamprecht]

Am 02/09/2022 um 13:24 schrieb Daniel Tschlatscher:
> Tested-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
> 

thx!

> 
> Some more comments inline in the code, all of which are non-breaking though:

@Markus, Daniels comment seem all reasonable and not only cosmetic (e.g.,
the typo in the schema for "minimum") can you please send a v2 with those
addressed, maybe also include a new docs patch with the respective comments
there either addressed, if sensible - thx!