[pve-devel] [PATCH qemu-server v3] QEMU AMD SEV enable

[pve-devel] [PATCH qemu-server v3] QEMU AMD SEV enable

[Markus Frank]

This Patch is for enabling AMD SEV (Secure Encrypted
Virtualization) support in QEMU

VM-Config-Examples:
amd_sev: type=std,nodbg=1,noks=1
amd_sev: es,nodbg=1,kernel-hashes=1

Node-Config-Example (gets generated automatically):
amd_sev: cbitpos=47,reduced-phys-bios=1

kernel-hashes, reduced-phys-bios & cbitpos correspond to the varibles
with the same name in qemu.

kernel-hashes=1 adds kernel-hashes to enable measured linux kernel
launch since it is per default off for backward compatibility.

reduced-phys-bios and cbitpos are system specific and can be read out
with QMP. If not set by the user, a dummy-vm gets started to read QMP
for these variables out and save them to the node config.
Afterwards the dummy-vm gets stopped.

type=std stands for standard sev to differentiate it from sev-es (es)
or sev-snp (snp) when support is upstream.

Qemu's sev-guest policy gets calculated with the parameters nodbg & noks
These parameters correspond to policy-bits 0 & 1.
If type=es than policy-bit 2 gets set to 1 to activate SEV-ES.
Policy bit 3 (nosend) is always set to 1, because migration
features for sev are not upstream yet and are attackable.

see coherent doc patch

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
I still could not get SEV-ES to work.
After a firmware update I got the same error like Daniel in his testing:
kvm: ../softmmu/vl.c:2568: qemu_machine_creation_done: Assertion `machine->cgs->ready' failed.

v3:
* moved parameters to node config
* created get_sev_parameters_from_node function
* added policy calculation

v2:
* spelling of minimum
* !$conf->{bios} eq 'ovmf' changed to $conf->{bios} ne 'ovmf'

 PVE/API2/Qemu.pm  |   9 +++
 PVE/QemuServer.pm | 140 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 149 insertions(+)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index badfc37..82b53d0 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -4358,6 +4358,10 @@ __PACKAGE__->register_method({
 	# test if VM exists
 	my $conf = PVE::QemuConfig->load_config($vmid);
 
+	my $amd_sev_conf = PVE::QemuServer::parse_amd_sev($conf->{amd_sev});
+	die "AMD SEV does not support migration\n"
+	    if ($amd_sev_conf->{type} eq 'std');
+
 	# try to detect errors early
 
 	PVE::QemuConfig->check_lock($conf);
@@ -4909,6 +4913,11 @@ __PACKAGE__->register_method({
 	die "unable to use snapshot name 'pending' (reserved name)\n"
 	    if lc($snapname) eq 'pending';
 
+	my $conf = PVE::QemuConfig->load_config($vmid);
+	my $amd_sev_conf = PVE::QemuServer::parse_amd_sev($conf->{amd_sev});
+	die "AMD SEV does not support snapshots\n"
+	    if ($amd_sev_conf->{type} eq 'std');
+
 	my $realcmd = sub {
 	    PVE::Cluster::log_msg('info', $authuser, "snapshot VM $vmid: $snapname");
 	    PVE::QemuConfig->snapshot_create($vmid, $snapname, $param->{vmstate},
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index a52a883..9e1267d 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -56,6 +56,7 @@ use PVE::QemuServer::Memory;
 use PVE::QemuServer::Monitor qw(mon_cmd);
 use PVE::QemuServer::PCI qw(print_pci_addr print_pcie_addr print_pcie_root_port parse_hostpci);
 use PVE::QemuServer::USB qw(parse_usb_device);
+use PVE::NodeConfig;
 
 my $have_sdn;
 eval {
@@ -170,6 +171,59 @@ my $agent_fmt = {
     },
 };
 
+my $sev_fmt = {
+    type => {
+	description => "Enable standard SEV with type='std' or enable SEV-ES"
+	." with the 'es' option.",
+	type => 'string',
+	default_key => 1,
+	format_description => "qemu-sev-type",
+	enum => ['std', 'es'],
+	maxLength => 3,
+    },
+    nodbg => {
+	description => "Sets policy bit 0 to 1 to disallow debugging of guest",
+	type => 'boolean',
+	format_description => "qemu-sev-nodbg",
+	default => 0,
+	optional => 1,
+    },
+    noks => {
+	description => "Sets policy bit 1 to 1 to disallow key sharing with other guests",
+	type => 'boolean',
+	format_description => "qemu-sev-noks",
+	default => 0,
+	optional => 1,
+    },
+    "kernel-hashes" => {
+	description => "Add kernel hashes to guest firmware for measured linux kernel launch",
+	type => 'boolean',
+	format_description => "qemu-sev-kernel-hashes",
+	default => 0,
+	optional => 1,
+    },
+};
+PVE::JSONSchema::register_format('pve-qemu-sev-fmt', $sev_fmt);
+
+my $sev_node_fmt = {
+    cbitpos => {
+	description => "C-bit: marks if a memory page is protected. System dependent",
+	type => 'integer',
+	default => 47,
+	optional => 1,
+	minimum => 0,
+	maximum => 100,
+    },
+    'reduced-phys-bits' => {
+	description => "Number of bits the physical address space is reduced by. System dependent",
+	type => 'integer',
+	default => 1,
+	optional => 1,
+	minimum => 0,
+	maximum => 100,
+    },
+};
+
 my $vga_fmt = {
     type => {
 	description => "Select the VGA type.",
@@ -346,6 +400,12 @@ my $confdesc = {
 	minimum => 16,
 	default => 512,
     },
+    amd_sev => {
+	description => "Secure Encrypted Virtualization (SEV) features by AMD CPUs",
+	optional => 1,
+	format => 'pve-qemu-sev-fmt',
+	type => 'string',
+    },
     balloon => {
 	optional => 1,
 	type => 'integer',
@@ -2141,6 +2201,15 @@ sub parse_guest_agent {
     return $res;
 }
 
+sub parse_amd_sev {
+    my ($value) = @_;
+
+    return if !$value;
+
+    my $res = parse_property_string($sev_fmt, $value);
+    return $res;
+}
+
 sub get_qga_key {
     my ($conf, $key) = @_;
     return undef if !defined($conf->{agent});
@@ -4132,6 +4201,40 @@ sub config_to_command {
     }
     push @$machineFlags, "type=${machine_type_min}";
 
+    my $amd_sev_conf = parse_amd_sev($conf->{amd_sev});
+
+    if (
+	$amd_sev_conf->{'type'}
+	&& ($amd_sev_conf->{type} eq 'std' || $amd_sev_conf->{type} eq 'es')
+	&& $conf->{bios}
+	&& $conf->{bios} ne 'ovmf'
+    ) {
+	die "For using SEV you need to change your guest bios to ovmf.\n";
+    }
+
+    if ($amd_sev_conf->{'type'} && $amd_sev_conf->{type} eq 'es' && $kvm) {
+	die "SEV-ES does not work with kvm. Disable kvm to use tcg.\n";
+    }
+
+    if (
+	$amd_sev_conf->{'type'}
+	&& ($amd_sev_conf->{type} eq 'std' || $amd_sev_conf->{type} eq 'es')
+    ) {
+	my $node_config = get_sev_parameters_from_node($nodename, $arch);
+	my $memobjcmd = 'sev-guest,id=sev0,cbitpos='.$node_config->{cbitpos}
+	    .',reduced-phys-bits='.$node_config->{'reduced-phys-bits'};
+	my $policy = 0b0;
+	$policy += 0b1 if ($amd_sev_conf->{nodbg});
+	$policy += 0b10 if ($amd_sev_conf->{noks});
+	$policy += 0b100 if ($amd_sev_conf->{type} eq 'es');
+	# disable migration with bit 3 nosend to prevent amd-sev-migration-attack
+	$policy += 0b1000;
+	$memobjcmd .= ',policy='.sprintf("%#x", $policy);
+	$memobjcmd .= ',kernel-hashes=on' if ($amd_sev_conf->{'kernel-hashes'});
+	push @$devices, '-object' , $memobjcmd;
+	push @$machineFlags, 'confidential-guest-support=sev0';
+    }
+
     push @$cmd, @$devices;
     push @$cmd, '-rtc', join(',', @$rtcFlags) if scalar(@$rtcFlags);
     push @$cmd, '-machine', join(',', @$machineFlags) if scalar(@$machineFlags);
@@ -4175,6 +4278,43 @@ sub check_rng_source {
     }
 }
 
+sub get_sev_parameters_from_node {
+    my ($nodename, $arch) = @_;
+    # Get reduced-phys-bits & cbitpos from QMP, if not set
+    my $node_config = PVE::NodeConfig::load_config($nodename);
+    my $sev_node_config;
+    if ($node_config->{amd_sev}) {
+	$sev_node_config = parse_property_string($sev_node_fmt, $node_config->{amd_sev});
+    }
+    if (
+	!$sev_node_config->{'reduced-phys-bits'}
+	|| !$sev_node_config->{cbitpos}
+    ) {
+	my $fakevmid = -1;
+	my $qemu_cmd = get_command_for_arch($arch);
+	my $pidfile = PVE::QemuServer::Helpers::pidfile_name($fakevmid);
+	my $default_machine = $default_machines->{$arch};
+	my $cmd = [
+	    $qemu_cmd,
+	    '-machine', $default_machine,
+	    '-display', 'none',
+	    '-chardev', "socket,id=qmp,path=/var/run/qemu-server/$fakevmid.qmp,server=on,wait=off",
+	    '-mon', 'chardev=qmp,mode=control',
+	    '-pidfile', $pidfile,
+	    '-S', '-daemonize'
+	];
+	my $rc = run_command($cmd, noerr => 1, quiet => 0);
+	die "QEMU flag querying VM exited with code " . $rc . "\n" if $rc;
+	my $res = mon_cmd($fakevmid, 'query-sev-capabilities');
+	vm_stop(undef, $fakevmid, 1, 1, 10, 0, 1);
+	$sev_node_config->{'reduced-phys-bits'} = $res->{'reduced-phys-bits'};
+	$sev_node_config->{cbitpos} = $res->{cbitpos};
+	$node_config->{amd_sev} = PVE::JSONSchema::print_property_string($sev_node_config, $sev_node_fmt);
+	PVE::NodeConfig::write_config($nodename, $node_config);
+    }
+    return $sev_node_config;
+}
+
 sub spice_port {
     my ($vmid) = @_;
 
-- 
2.30.2

[pve-devel] [PATCH docs v3] added Memory Encryption documentation

[Markus Frank]

added AMD SEV documentation for "[PATCH qemu-server] QEMU AMD SEV
enable"

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
v3:
* added more information
* removed some grammar errors

v2:
* added more details for host & guests
* moved things from Limitations to Requirements
* changed order of text

 qm.adoc | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)

diff --git a/qm.adoc b/qm.adoc
index e7d0c07..6f79289 100644
--- a/qm.adoc
+++ b/qm.adoc
@@ -598,6 +598,124 @@ systems.
 When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB
 of RAM available to the host.
 
+[[qm_memory_encryption]]
+Memory Encryption
+~~~~~~~~~~~~~~~~~
+
+[[qm_memory_encryption_sev]]
+AMD SEV
+^^^^^^^
+
+SEV (Secure Encrypted Virtualization) enables Memory Encryption per VM using
+AES-128 Encryption and the AMD Secure Processor.
+
+SEV-ES (Secure Encrypted Virtualization-Encrypted State) in addition encrypts
+all CPU register contents when a VM stops running, to prevent leakage of
+information to the hypervisor.
+
+*Host Requirements:*
+
+* AMD EPYC/Ryzen PRO
+* SEV-ES is only supported on AMD EPYC 7xx2 and newer
+* configured SEV BIOS settings on Host Machine
+* add "kvm_amd.sev=1" to kernel parameters if not enabled by default
+* add "mem_encrypt=on" to kernel parameters if you want to encrypt memory on the
+host (SME)
+see https://www.kernel.org/doc/Documentation/x86/amd-memory-encryption.txt
+* maybe increase SWIOTLB see https://github.com/AMDESE/AMDSEV#faq-4
+
+To check if SEV is enabled on the host search for `sev` in dmesg
+and print out the SEV kernel parameter of kvm_amd:
+
+----
+# dmesg | grep -i sev
+[...] ccp 0000:45:00.1: sev enabled
+[...] ccp 0000:45:00.1: SEV API: <buildversion>
+[...] SEV supported: <number> ASIDs
+[...] SEV-ES supported: <number> ASIDs
+# cat /sys/module/kvm_amd/parameters/sev
+Y
+----
+
+Node Configuration (/etc/pve/nodes/mona/config):
+
+----
+amd_sev: cbitpos=47,reduced-phys-bits=1
+----
+
+*reduced-phys-bios* and *cbitpos* correspond to the variables with the
+same name in qemu. They are system specific and can be read out
+with QMP. If not set, qm starts a dummy-vm to read QMP
+for these variables out and saves them to config.
+
+*Guest Requirements:*
+
+* edk2-OVMF
+* advisable to use Q35
+* The guest operating system must contain SEV-support.
+* If there are problems while booting (stops at blank/splash screen)
+try to add virtio-rng.
+
+*Limitations:*
+
+* Because the memory is encrypted the memory usage on host is always wrong.
+* Operations that involve saving or restoring memory like snapshots
+& live migration do not work yet or are attackable.
+https://github.com/PSPReverse/amd-sev-migration-attack
+* PCI passthrough is not supported.
+* Qemu & AMD-SEV documentation is very limited.
+* Nested virtualization and kvm is not supported under SEV-ES.
+
+Example Configuration:
+
+----
+# qm set <vmid> -amd_sev type=std,nodbg=1,noks=1,kernel-hashes=1
+----
+
+*type* defines the encryption technology ("type=" is not necessary).
+Available options: std, es
+
+The Qemu *policy* parameter gets calculated with the *nodbg* and *noks*
+parameters.
+These parameters correspond to policy-bit 0 and 1.
+If *type* is *es* the policy-bit 2 is set to 1 so that SEV-ES is enabled.
+Policy-bit 3 (nosend) is always set to 1 to prevent migration-attacks.
+For more information on how to calculate the policy see:
+https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf[AMD SEV API Specification Chapter 3]
+
+The *kernel-hashes* is per default off for backward compatibility with older OVMF images
+and guests that do not measure the kernel/initrd.
+See https://lists.gnu.org/archive/html/qemu-devel/2021-11/msg02598.html
+
+*Check if SEV is working on the guest*
+
+Method 1 - dmesg:
+
+Output should look like this.
+
+----
+# dmesg | grep -i sev
+AMD Memory Encryption Features active: SEV
+----
+
+Method 2 - MSR 0xc0010131 (MSR_AMD64_SEV):
+
+Output should be 1.
+
+----
+# apt install msr-tools
+# modprobe msr
+# rdmsr -a 0xc0010131
+1
+----
+
+Links:
+
+* https://developer.amd.com/sev/
+* https://github.com/AMDESE/AMDSEV
+* https://www.qemu.org/docs/master/system/i386/amd-memory-encryption.html
+* https://www.amd.com/system/files/TechDocs/55766_SEV-KM_API_Specification.pdf
+* https://documentation.suse.com/sles/15-SP1/html/SLES-amd-sev/index.html
 
 [[qm_network_device]]
 Network Device
-- 
2.30.2

Re: [pve-devel] [PATCH qemu-server v3] QEMU AMD SEV enable

[Thomas Lamprecht]

Am 09/12/2022 um 15:25 schrieb Markus Frank:
> This Patch is for enabling AMD SEV (Secure Encrypted
> Virtualization) support in QEMU
> 
> VM-Config-Examples:
> amd_sev: type=std,nodbg=1,noks=1
> amd_sev: es,nodbg=1,kernel-hashes=1
> 
> Node-Config-Example (gets generated automatically):
> amd_sev: cbitpos=47,reduced-phys-bios=1
> 
> kernel-hashes, reduced-phys-bios & cbitpos correspond to the varibles
> with the same name in qemu.
> 
> kernel-hashes=1 adds kernel-hashes to enable measured linux kernel
> launch since it is per default off for backward compatibility.
> 
> reduced-phys-bios and cbitpos are system specific and can be read out
> with QMP. If not set by the user, a dummy-vm gets started to read QMP
> for these variables out and save them to the node config.
> Afterwards the dummy-vm gets stopped.
> 
> type=std stands for standard sev to differentiate it from sev-es (es)
> or sev-snp (snp) when support is upstream.
> 
> Qemu's sev-guest policy gets calculated with the parameters nodbg & noks
> These parameters correspond to policy-bits 0 & 1.
> If type=es than policy-bit 2 gets set to 1 to activate SEV-ES.
> Policy bit 3 (nosend) is always set to 1, because migration
> features for sev are not upstream yet and are attackable.
> 
> see coherent doc patch
> 
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
> I still could not get SEV-ES to work.
> After a firmware update I got the same error like Daniel in his testing:
> kvm: ../softmmu/vl.c:2568: qemu_machine_creation_done: Assertion `machine->cgs->ready' failed.
> 


This was one of the main turn-offs for me, but maybe the situation change
here w.r.t newer HW, kernel and QEMU support.

Can you please re-test this rather soonish? E.g. with kernel 6.5 and 6.8,
also trying a newer QEMU like Fiona's 8.2 build and our newer AMD based
HW would be good to check out.