From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 7C3631FF0C1 for ; Wed, 26 Aug 2026 10:19:22 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E81A5214F1; Wed, 26 Aug 2026 10:19:15 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QtNSG8p4ikr/a0cyL5u9gJLSG2xg+mjydXtWfvCyQVGPXlrk6Ra/wUX+7ECF6vSC91IifwKYxIyq+R0N/tT31MxJT1JqUHdCcFHitEJ0heqDscjcbhPikus7Z4ye/9VC/969UP1wAlmCkBy1YWqevp88uEZZYhvTQDaHTaO8azRU6mxMZ6aBsxw5/mgIrazT5Br0EZ02DZjAM0gC/kL7NoOMpGBd1X5kywJPIp8ojMPaEAs1Ds6tSW35a+sobbwLIYgGA1mZWLoPwy15qvvpXI+qJYldKUhTD0R4zDnvU+tP5BnIV4yJOXp+kXXuNEUjp3mZ1g5gTl/ClybCW2m95A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mHFKBs4pwm6sF71ZXbT2JBP2a5Z/6vmSaKjfgHqMs10=; b=lHTu1GZUzFjpdVzbudqODDn6cg14wS74SG6QWWuOCpRH/glPU1p8QVoiAKV6oWAexWqaEctIFUI0S2zZ+uSqOOhxN8o6kiKtZiX6YzRUIpD0ILLWbNT8+R4R+0DgFPRNUmQqQJdqgdyy+NafTGPdMFIVON54MYtyDgE+eVLAwy8sUYfFPIU+XPsPaz76Vbmlzkl242RPXKCZOE2CJK1t1wU+rNFHJ/wzNW9Nim6LQKhxyBfGCL4mKhMf0CshnBkLbg90hUilkxVlPJwdmtVEKw/Z1pR7irLn0qDjsZFEe8Apa7N7utYEVe/mnY4h7Y/cI7y+m5L0X5q8VuHc5Ie+4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=groupe-cyllene.com; dmarc=pass action=none header.from=groupe-cyllene.com; dkim=pass header.d=groupe-cyllene.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=groupecyllene.onmicrosoft.com; s=selector2-groupecyllene-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mHFKBs4pwm6sF71ZXbT2JBP2a5Z/6vmSaKjfgHqMs10=; b=AgTmyv14dNqVbnOiSb7DxxdF5Vy8jzy9tX2vksLTuXx457lKZLJh3Qia7QYbS33aZhA+aOhD0YQIFlHUOEFlxCZRtt+8R5i7IEoZci4SzrOQvBcKVm5nD+Ngp6tuBRE5Y5V0IR6gwUPb5xD4W3CQcNcZSg+dvquGCeTGOxShYB0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=groupe-cyllene.com; From: Alexandre Derumier To: pve-devel@lists.proxmox.com Subject: [RFC v2 qemu-server 03/13] add kyber display Date: Wed, 26 Aug 2026 09:43:35 +0200 Message-ID: <20260826074347.1256659-4-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260826074347.1256659-1-alexandre.derumier@groupe-cyllene.com> References: <20260826074347.1256659-1-alexandre.derumier@groupe-cyllene.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: PA7P264CA0229.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:372::8) To PR1P264MB3696.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:143::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PR1P264MB3696:EE_|PR0P264MB2257:EE_ X-MS-Office365-Filtering-Correlation-Id: 348f5487-6416-46e3-45d3-08df0345d43f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|23010399003|52116014|1800799024|366016|38350700014|6133799003|3023799007|56012099006|10067099003|5023799004|20052099010|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: QGoypj1SzkM7YtCS+hqxCcNENkCabz1e68JXndSf//B9VeFMvbn3Xjd1i4yJhcVYm2ZSN431snCHva4SH6GxjBDqftn3C9QIMYRcQPoZarKUOQWbKdf63O6e6BMPJHoqsUzcCdXYbBIaWpM+fsRHQ8u5NmF+9dEwcBXexIEgj5rSbD9EIESfAuVl5kQWt50q4wmzjZ/nRz4itXB9S8O9E1gCZvuPo6elXLNCL43yh36WxOeSr8iSQ7xDYg4kwP06yXP9chLqDdz28Pnx4nd1fgNPCIZWGFnJ2pMGZdpfkWf6+VryA9O0onvdWrN//vFxuTggLKYW1fkmCnoImhHz7sDmrCcfFD/NYR0Asn0JxWIPBSAQQdHdSlLUoEwcL4lW72U7TjHkjLyH+7CDSiDtXcbbWBeBNHd0maZyQH6ZecqnOEOSKfwgKkR5x6hISA8DCWLc1lofwd5WzVbYzxO/AxsnJuatqncz+u3bvPea8PnHVhU5R3VG+3y1IQARBBijk8Bo7gVeh2SxJQMDqxslhKlhrG2T/L+KBe7iA1/Wmr1CUFAsiPBS/e09YTZWpXjwIJdCkiDTYWCQhHP3ZAPzX1zmZtokAjKnsylR79uSJMlevUyoQtYNRupywBLSwiLqnOF3ucn4iiTQowS2P8Gndqsy+cJnnMURFJmIe1CgDnf8i3oNblHaFo9eQTUUxzxNym1mzxSJhDSUYc7ppxMhbHD20SE0ohzdqbJTNbGBsc8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PR1P264MB3696.FRAP264.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(376014)(23010399003)(52116014)(1800799024)(366016)(38350700014)(6133799003)(3023799007)(56012099006)(10067099003)(5023799004)(20052099010)(22082099003)(18002099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uXXGaysvzrsqyyBFwr9mhmfhSw8edIJsVxDx5DN2URMpGkz8cBXwDWlqa7sC?= =?us-ascii?Q?V92TFIqQDIwTrp9sOCeBaq9SK/ZoSPPENNKLUeGXcIaOdm+Wn/mWqUjy+sFz?= =?us-ascii?Q?UGQ7/2KzRIviYClvxg+uUFYpucCrQhJZ+vZ/bXTm9KZ+zyYu0ENSKkcj/9Uw?= =?us-ascii?Q?/wS1aRkC6v6br1qc5YthhnAyqmsR5PsvpjmMQ9rVIBIsjpFgIUjdihtZmq3y?= =?us-ascii?Q?gCYctCHiHH+2H1dGtO1G0gbPIRm+BishvGhUtnsKkq4WzuDh2VGDFKlytwaC?= =?us-ascii?Q?jElmSsSn9LmfgEY7BYwlP/rGe4C7ZvpNdUdbpNJ0f3xcITkl9EiwRSRWStY7?= =?us-ascii?Q?+bsoPnbrcSPPb+pt2Z0TIIdiH6SD4xKnkT6S1fOS0a1lI4fiRFFSFmgx5HBq?= =?us-ascii?Q?A/X8tJ1xawFCeEF/Ft3hxAf2MqWhYu1cv0EeuOzb8tYerL+68m5n/UtrIkhD?= =?us-ascii?Q?8EgY3i0nLK2nR694Svn4olBcYYkJGYSHTq/nYHES1J9qSxehBFvzyOOop1+R?= =?us-ascii?Q?9XqfxsShq/STBQrGMLaqLKpTyFFgPxUj3cUdRR46he9jp/BNxXqyLMEK4lJQ?= =?us-ascii?Q?QnsY4IjTwpi1NO1auHlN946ns1Mm4JsLZeAlr3vi8iEn3jSHRKg1NmUYt3Ki?= =?us-ascii?Q?KHrtO+Bomk737zkE0TfKlTLi8Ugo4616Msht0PEOB7c+qjDZlMdjR7ZeaFl+?= =?us-ascii?Q?F5d1WKVPRdmVk0KA3ExV5WCKxQtAq6P/BeYiLN9xoZq5HRKzRcI/UmkSp+QV?= =?us-ascii?Q?tA36xWz0TqMYZZD5svHTskU5r/JoqlQEEQFWBw406jy3FkwOwZXJEhjWYaKD?= =?us-ascii?Q?hb/5N+jB69EmaUtuClISc2SsmmbHNe8Uv2AVVU3YtvOLmKE04KNJNKNaTF4y?= =?us-ascii?Q?hXbVbEFlcEL+UXVnA/dnj3YpYo0G5KOiYizhAlWwXr35Vh4Dain0HiZlK0BH?= =?us-ascii?Q?Hu4/Ch2cJi++YQ+AHP9Kra2WJbmOVLPwdejKw3k/ZM+Y0kXyqgMFzKd5NtMQ?= =?us-ascii?Q?3HyVTeLQy3Sk509viL8DXSfrXDL3msR/FCqyjYHrEM2vSyXfXSDC7WPuguqI?= =?us-ascii?Q?df6PC79RzXY+qWEKW9PkmuNj9mScq0kiQd1bJBQlcSEWp0T6FQsdY6W4YibL?= =?us-ascii?Q?ersL+bpnofGAbWzWPaT7hwERZTUeBrwZtIug8MSyn068gWy1z1IpGvDUNDfB?= =?us-ascii?Q?ufabe+fsOoaEbIur9Joz6q04JNLJ3dyH0FgbC71LFFs20oPsAqW30W2Kw8fd?= =?us-ascii?Q?gdW9AvlMqjfFntB2YccxVu54q2s2SPK7aAwstDq33sHDyiQa0/Es6xOQxhs7?= =?us-ascii?Q?k1eWAPkcoFWxup/3bu66UDRuu70kqgmlwNK9uaKHkhXCveKAw21g59ySxsCg?= =?us-ascii?Q?2nFB5JJYVOTti/gf683pdhYtNMD+/NsA4Yy9unPG9eDNJFCGmOi3zsCgNuOg?= =?us-ascii?Q?APqMlVhp1X58mOFY1wJghgiEMTNcG+TXttmOh7f2AtnO0Rzt0wY+4krcJczz?= =?us-ascii?Q?KTR3jxiTcPWxUdWDCVb9lx/RJGr618pttSYYIO2r9FRT+rzQPpgyg8RkiP6D?= =?us-ascii?Q?aNlzR9tsdDG7bmVlGecLnQyk24qIUQYWO+LWQtz7GQH9mWQeXAu6ENmX0Sm8?= =?us-ascii?Q?wkFNh6Y9JEk3/wagBi3AbW/7UN6t4OF2q9mM1Mjvuhu2FxF6GYHUg09tGxQh?= =?us-ascii?Q?tgYJX3tYIUj69jG47Btqri4qehsVeBUkteS0Xaf8qe6ooZC0P+pVeG8aU1m4?= =?us-ascii?Q?YmTexxHrcWn4q7Z5xVWduNOwFtmoVmtPJrYcy5X+GAwAMdJ4nwAU?= X-OriginatorOrg: groupe-cyllene.com X-MS-Exchange-CrossTenant-Network-Message-Id: 348f5487-6416-46e3-45d3-08df0345d43f X-MS-Exchange-CrossTenant-AuthSource: PR1P264MB3696.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 07:44:15.9536 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: ee11ccf7-112c-4284-848b-f229745e715b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uJ/3VUC7nqo2yXlg9mUWZn+jntBGqaQtGE+4dOULlRkFYusWKD9i83qLZySLo1zYOJJm+bedBCUsvBL716rICZp1D0DcHeZTd9BJgJqFuPXobHVFKuUELG9MQxxAPo0x X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR0P264MB2257 X-SPAM-LEVEL: Spam detection results: 1 AWL -1.286 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_PASS -0.1 DMARC pass policy KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLACK 3 Contains an URL listed in the URIBL blacklist [agent.pm] Message-ID-Hash: 7NYZXTYRXDS7KD3GNTBT4WG2ELMGB64O X-Message-ID-Hash: 7NYZXTYRXDS7KD3GNTBT4WG2ELMGB64O X-MailFrom: Alexandre.DERUMIER@groupe-cyllene.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: kyber maps to virtio-vga, with a controller started per VM on demand by the kyberproxy API call. kyber-gl maps to virtio-vga-gl instead: virgl renders inside QEMU and exports a dmabuf, so the encoder imports it rather than reading pixels back. The vhost-user-gpu helper does the same job out of process, but deadlocks intermittently on QEMU 11.0.2 - the main thread blocks on a vhost-user reply while holding the BQL, so the guest freezes rather than just the stream. Signed-off-by: Alexandre Derumier --- src/PVE/API2/Qemu.pm | 97 +++++++++++++++++++ src/PVE/QemuServer.pm | 53 ++++++++++- src/PVE/QemuServer/Kyber.pm | 148 +++++++++++++++++++++++++++++ src/PVE/QemuServer/Makefile | 2 + src/test/cfg2cmd/kyber-gl.conf | 3 + src/test/cfg2cmd/kyber-gl.conf.cmd | 27 ++++++ src/test/cfg2cmd/kyber.conf | 3 + src/test/cfg2cmd/kyber.conf.cmd | 27 ++++++ src/usr/Makefile | 1 + src/usr/pve-qemu-kyber@.service | 25 +++++ 10 files changed, 382 insertions(+), 4 deletions(-) create mode 100644 src/PVE/QemuServer/Kyber.pm create mode 100644 src/test/cfg2cmd/kyber-gl.conf create mode 100644 src/test/cfg2cmd/kyber-gl.conf.cmd create mode 100644 src/test/cfg2cmd/kyber.conf create mode 100644 src/test/cfg2cmd/kyber.conf.cmd create mode 100644 src/usr/pve-qemu-kyber@.service diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm index 71247ee..378e103 100644 --- a/src/PVE/API2/Qemu.pm +++ b/src/PVE/API2/Qemu.pm @@ -12,6 +12,7 @@ use IPC::Open3; use JSON; use URI::Escape; use Socket qw(SOCK_STREAM); +use Time::HiRes qw(usleep); use PVE::APIClient::LWP; use PVE::CGroup; @@ -35,6 +36,7 @@ use PVE::QemuServer::Cloudinit; use PVE::QemuServer::CPUConfig; use PVE::QemuServer::Drive qw(checked_volume_format checked_parse_volname); use PVE::QemuServer::Helpers; +use PVE::QemuServer::Kyber; use PVE::QemuServer::ImportDisk; use PVE::QemuServer::Monitor qw(mon_cmd vm_qmp_peer); use PVE::QemuServer::Machine; @@ -3329,6 +3331,95 @@ __PACKAGE__->register_method({ }, }); +__PACKAGE__->register_method({ + name => 'kyberproxy', + path => '{vmid}/kyberproxy', + method => 'POST', + protected => 1, + proxyto => 'node', + permissions => { + check => ['perm', '/vms/{vmid}', ['VM.Console']], + }, + description => "Start a Kyber console controller for the VM and return how to reach it.", + parameters => { + additionalProperties => 0, + properties => { + node => get_standard_option('pve-node'), + vmid => get_standard_option('pve-vmid'), + }, + }, + returns => { + additionalProperties => 0, + properties => { + user => { type => 'string' }, + ticket => { + type => 'string', + description => "Short-lived token authenticating this user to the controller.", + }, + }, + }, + code => sub { + my ($param) = @_; + + my $rpcenv = PVE::RPCEnvironment::get(); + my $authuser = $rpcenv->get_user(); + + my $vmid = $param->{vmid}; + my $node = $param->{node}; + + my $conf = PVE::QemuConfig->load_config($vmid, $node); + + my $vga = PVE::QemuServer::parse_vga($conf->{vga} // ''); + my $vga_type = $vga->{type} // ''; + die "VM $vmid is not configured for the Kyber console" + . " - set its display to 'kyber' or 'kyber-gl' and restart it\n" + if $vga_type !~ /^kyber(?:-gl)?$/; + + die "VM $vmid is not running\n" if !PVE::QemuServer::Helpers::vm_running_locally($vmid); + + my $socket = PVE::QemuServer::Helpers::dbus_socket($vmid); + die "VM $vmid has no D-Bus display socket at $socket" + . " - it was started before its display was set to 'kyber'," + . " so it needs a restart\n" + if !-S $socket; + + # Join a controller that is already streaming: it shares one capture between + # clients, and restarting to install a new secret would cut the first off. + my ($secret, $port) = PVE::QemuServer::Kyber::running_secret($vmid); + + if (!$secret) { + my $family = PVE::Tools::get_host_address_family($node); + $port = PVE::QemuServer::Kyber::next_port($family); + + # Fresh per controller: with none running there is nothing to cut off. + $secret = PVE::QemuServer::Kyber::generate_secret(); + + # Only 'vnc' adds the vdagent chardev the guest needs to share a clipboard. + my $clipboard = ($vga->{clipboard} // '') eq 'vnc'; + + my $dmabuf = $vga_type eq 'kyber-gl'; + + PVE::QemuServer::Kyber::write_env($vmid, $secret, $port, $clipboard, $dmabuf); + PVE::QemuServer::Kyber::restart_controller($vmid); + } + + my $ticket = PVE::QemuServer::Kyber::assemble_ticket($secret, $authuser); + + # Listening within ~30ms, so waiting here saves the client a retry loop. + my $kybersocket = PVE::QemuServer::Kyber::socket_file($vmid); + for (my $waited = 0; $waited < 5; $waited += 0.05) { + last if -S $kybersocket; + usleep(50_000); + } + die "Kyber console controller for VM $vmid did not start\n" if !-S $kybersocket; + + return { + user => $authuser, + ticket => $ticket, + }; + }, +}); + __PACKAGE__->register_method({ name => 'spiceproxy', path => '{vmid}/spiceproxy', @@ -3452,6 +3543,11 @@ __PACKAGE__->register_method({ type => 'boolean', optional => 1, }, + kyber => { + description => "QEMU VGA configuration supports the Kyber console.", + type => 'boolean', + optional => 1, + }, agent => { description => "QEMU Guest Agent is enabled in config.", type => 'boolean', @@ -3482,6 +3578,7 @@ __PACKAGE__->register_method({ my $spice = defined($vga->{type}) && $vga->{type} =~ /^virtio/; $spice ||= PVE::QemuServer::vga_conf_has_spice($conf->{vga}); $status->{spice} = 1 if $spice; + $status->{kyber} = 1 if ($vga->{type} // '') eq 'kyber'; $status->{clipboard} = $vga->{clipboard}; } $status->{agent} = 1 if PVE::QemuServer::Agent::get_qga_key($conf, 'enabled'); diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm index 2f43faa..cadc8fe 100644 --- a/src/PVE/QemuServer.pm +++ b/src/PVE/QemuServer.pm @@ -80,6 +80,7 @@ use PVE::QemuServer::Drive qw( storage_allows_io_uring_default ); use PVE::QemuServer::DriveDevice qw(print_drivedevice_full scsihw_infos); +use PVE::QemuServer::Kyber; use PVE::QemuServer::Machine; use PVE::QemuServer::Memory qw(get_current_memory); use PVE::QemuServer::MetaInfo; @@ -98,6 +99,7 @@ use PVE::QemuServer::StateFile; use PVE::QemuServer::USB; use PVE::QemuServer::Virtiofs qw(max_virtiofs start_all_virtiofsd); use PVE::QemuServer::VolumeChain; +use PVE::QemuServer::DBusDisplay; use PVE::QemuServer::DBusVMState; my $have_ha_config; @@ -168,7 +170,7 @@ my $vga_fmt = { optional => 1, default_key => 1, enum => [ - qw(cirrus qxl qxl2 qxl3 qxl4 none serial0 serial1 serial2 serial3 std virtio virtio-gl vmware) + qw(cirrus kyber kyber-gl qxl qxl2 qxl3 qxl4 none serial0 serial1 serial2 serial3 std virtio virtio-gl vmware) ], }, memory => { @@ -213,10 +215,12 @@ my $audio_fmt = { }, driver => { type => 'string', - enum => ['spice', 'none'], + enum => ['spice', 'dbus', 'none'], default => 'spice', optional => 1, - description => "Driver backend for the audio device.", + description => "Driver backend for the audio device." + . " 'dbus' exposes it on the VM's D-Bus display, which is what the" + . " Kyber and RDP consoles read.", }, }; @@ -1481,6 +1485,10 @@ my $vga_map = { 'vmware' => 'vmware-svga', 'virtio' => 'virtio-vga', 'virtio-gl' => 'virtio-vga-gl', + # A display transport, not a card, so it picks one: virtio-vga rather than a GL + # variant, both of which currently break QEMU. + 'kyber' => 'virtio-vga', + 'kyber-gl' => 'virtio-vga-gl', }; # QEMU builds only the non-VGA variants of the virtio GPU for aarch64 @@ -1488,6 +1496,8 @@ my $vga_map_aarch64 = { $vga_map->%*, 'virtio' => 'virtio-gpu', 'virtio-gl' => 'virtio-gpu-gl', + 'kyber' => 'virtio-gpu', + 'kyber-gl' => 'virtio-gpu-gl', }; my sub map_vga_model { @@ -2839,7 +2849,14 @@ sub audio_devs { die "unknown audio device '$audio->{dev}', implement me!"; } - push @$devs, '-audiodev', "$audio->{backend},id=$audio->{backend_id}"; + my $backend = "$audio->{backend},id=$audio->{backend_id}"; + + # Pinned for the D-Bus backend: what reads it is an Opus encoder and libopus takes + # 48kHz only. QEMU already resamples, so this costs nothing new. + $backend .= ',out.frequency=48000,out.channels=2,out.format=s16' + if $audio->{backend} eq 'dbus'; + + push @$devs, '-audiodev', $backend; return $devs; } @@ -3408,6 +3425,20 @@ sub config_to_command { push @$cmd, '-display', 'egl-headless,gl=core' if $vga->{type} eq 'virtio-gl'; # VIRGL + if ($vga->{type} =~ /^(?:kyber|kyber-gl)$/) { + my $dbus = PVE::QemuServer::Helpers::dbus_socket($vmid); + my $display = "dbus,addr=unix:path=$dbus"; + $display .= ",gl=on" if $vga->{type} eq 'kyber-gl'; + + # The display exports org.qemu.Display1.Audio only when told which audiodev to + # read, and nothing else can consume a dbus audiodev. + my $audio = conf_has_audio($conf); + $display .= ",audiodev=$audio->{backend_id}" + if $audio && $audio->{backend} eq 'dbus'; + + push @$cmd, '-display', $display; + } + my $socket = PVE::QemuServer::Helpers::vnc_socket($vmid); push @$cmd, '-vnc', "unix:$socket,password=on"; } else { @@ -5812,6 +5843,11 @@ sub vm_start_nolock { my $virtiofs_sockets = start_all_virtiofsd($conf, $vmid); + # QEMU connects to the D-Bus address, so the bus has to be listening first. + my $dbus_vga = parse_vga($conf->{vga} // ''); + PVE::QemuServer::DBusDisplay::start($vmid) + if ($dbus_vga->{type} // '') =~ /^(?:kyber|kyber-gl)$/; + my $tpmpid; if ((my $tpm = $conf->{tpmstate0}) && !PVE::QemuConfig->is_template($conf)) { # start the TPM emulator so QEMU can connect on start @@ -6196,6 +6232,15 @@ sub vm_stop_cleanup { my ($storecfg, $vmid, $conf, $keepActive, $apply_pending_changes, $noerr, $skip_hookscript) = @_; + # Before the cleanup flag is consulted, deliberately: the bus is forked into the + # VM's systemd scope, and a survivor keeps that cgroup from emptying, so the next + # start fails with "timeout waiting on systemd". + eval { + PVE::QemuServer::Kyber::stop_controller($vmid); + PVE::QemuServer::DBusDisplay::stop($vmid); + }; + warn $@ if $@; + my $can_use_cleanup_flag = PVE::QemuServer::RunState::can_use_cleanup_flag(); if ($can_use_cleanup_flag) { return if !PVE::QemuServer::RunState::cleanup_flag_exists($vmid); diff --git a/src/PVE/QemuServer/Kyber.pm b/src/PVE/QemuServer/Kyber.pm new file mode 100644 index 0000000..a240e5a --- /dev/null +++ b/src/PVE/QemuServer/Kyber.pm @@ -0,0 +1,148 @@ +package PVE::QemuServer::Kyber; + +# Per-VM Kyber console controller: one kycontroller per VM, spawned on demand and +# reaped when it exits. + +use strict; +use warnings; + +use Digest::SHA qw(hmac_sha256); +use JSON; +use Crypt::OpenSSL::Random; +use IO::Socket::UNIX; +use Socket qw(SOCK_STREAM); +use MIME::Base64 qw(encode_base64url); + +use PVE::Tools qw(file_set_contents); +use PVE::QemuServer::DBusDisplay; +use PVE::QemuServer::Helpers; + +# Whether anything is listening, rather than whether a file is in the way. +# Both servers quit on their own when the VM's D-Bus display goes, and leave +# their socket behind when they do, so a plain -S reports a server that is not +# there - and the console then fails with ECONNREFUSED one hop further on. +my sub socket_answers { + my ($path) = @_; + + return 0 if !-S $path; + + my $sock = IO::Socket::UNIX->new(Type => SOCK_STREAM, Peer => $path); + return 0 if !$sock; + + close($sock); + return 1; +} + +sub socket_file { + my ($vmid) = @_; + return "$PVE::QemuServer::Helpers::var_run_tmpdir/$vmid.kyber.sock"; +} + +# Carries the signing key, and only that: /proc//cmdline is world-readable, +# so it cannot go on the command line. systemd passes it as KYBER_JWT_KEY. +sub env_file { + my ($vmid) = @_; + return "$PVE::QemuServer::Helpers::var_run_tmpdir/$vmid.kyber.env"; +} + +sub next_port { + my ($family) = @_; + return PVE::Tools::next_unused_port(63000, 63099, $family, '127.0.0.1'); +} + +sub generate_secret { + my $bytes = Crypt::OpenSSL::Random::random_bytes(32) + or die "unable to generate a random secret\n"; + return unpack('H*', $bytes); +} + +# An HS256 token the controller will accept. The secret is regenerated whenever a +# controller starts and never leaves the node, so it is useless against other VMs. +sub assemble_ticket { + my ($secret, $username, $ttl) = @_; + + # As long as a VNC ticket, and for the same reason: the client re-presents it to + # renew its session. It is worth little alone - the controller is on loopback. + $ttl //= 3600; + my $now = time(); + + my $header = encode_base64url(encode_json({ alg => 'HS256', typ => 'JWT' })); + # The controller requires aud=kyber (auth/jwt.rs) and rejects a token without it + # as malformed, which reads like a signing failure. + my $claims = encode_base64url( + encode_json({ + aud => 'kyber', + sub => $username, + iat => $now, + exp => $now + $ttl, + }), + ); + + my $signature = encode_base64url(hmac_sha256("$header.$claims", $secret)); + + return "$header.$claims.$signature"; +} + +# The secret a running controller is verifying against, or undef when there is +# none. Kept in a root-only env file so a second console can join instead of +# restarting the controller and cutting the first viewer off. +sub running_secret { + my ($vmid) = @_; + + return undef if !socket_answers(socket_file($vmid)); + + my $env = eval { PVE::Tools::file_get_contents(env_file($vmid)) }; + return undef if !defined($env); + + my ($secret) = $env =~ m/^KYBER_JWT_KEY=(\S+)$/m; + my ($port) = $env =~ m/^KYBER_DATAPLANE_PORT=(\d+)$/m; + return undef if !$secret || !$port; + + return ($secret, $port); +} + +sub write_env { + my ($vmid, $secret, $dataplane_port, $clipboard, $dmabuf) = @_; + + my $clipboard_env = $clipboard ? 1 : 0; + my $dmabuf_env = $dmabuf ? 1 : 0; + + my $env = <<"EOF"; +KYBER_JWT_KEY=$secret +KYBER_DATAPLANE_PORT=$dataplane_port +KQS_CLIPBOARD=$clipboard_env +KQS_DMABUF=$dmabuf_env +EOF + + my $path = env_file($vmid); + file_set_contents($path, $env, 0600); + + return $path; +} + +sub restart_controller { + my ($vmid) = @_; + + PVE::Tools::run_command( + ['systemctl', 'restart', "pve-qemu-kyber\@$vmid"], + errmsg => "failed to start Kyber console controller for VM $vmid", + ); + + return; +} + +sub stop_controller { + my ($vmid) = @_; + + eval { + PVE::Tools::run_command(['systemctl', 'stop', "pve-qemu-kyber\@$vmid"]); + }; + warn $@ if $@; + + unlink env_file($vmid); + unlink socket_file($vmid); + + return; +} + +1; diff --git a/src/PVE/QemuServer/Makefile b/src/PVE/QemuServer/Makefile index 060fac2..061d61f 100644 --- a/src/PVE/QemuServer/Makefile +++ b/src/PVE/QemuServer/Makefile @@ -10,11 +10,13 @@ SOURCES=Agent.pm \ Cloudinit.pm \ CPUConfig.pm \ CPUFlags.pm \ + DBusDisplay.pm \ DBusVMState.pm \ Drive.pm \ DriveDevice.pm \ Helpers.pm \ ImportDisk.pm \ + Kyber.pm \ Machine.pm \ Memory.pm \ MetaInfo.pm \ diff --git a/src/test/cfg2cmd/kyber-gl.conf b/src/test/cfg2cmd/kyber-gl.conf new file mode 100644 index 0000000..5150835 --- /dev/null +++ b/src/test/cfg2cmd/kyber-gl.conf @@ -0,0 +1,3 @@ +# TEST: Kyber console on the GL display +memory: 2048 +vga: kyber-gl diff --git a/src/test/cfg2cmd/kyber-gl.conf.cmd b/src/test/cfg2cmd/kyber-gl.conf.cmd new file mode 100644 index 0000000..ad0e217 --- /dev/null +++ b/src/test/cfg2cmd/kyber-gl.conf.cmd @@ -0,0 +1,27 @@ +/usr/bin/kvm +-id 8006 +-name vm8006 +-no-shutdown +-chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' +-mon 'chardev=qmp,mode=control' +-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' +-mon 'chardev=qmp-event,mode=control' +-pidfile /var/run/qemu-server/8006.pid +-daemonize +-smp '1,sockets=1,cores=1,maxcpus=1' +-nodefaults +-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' +-display 'dbus,addr=unix:path=/var/run/qemu-server/8006.dbusdisplay,gl=on' +-vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' +-cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep +-m 2048 +-global 'PIIX4_PM.disable_s3=1' +-global 'PIIX4_PM.disable_s4=1' +-device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' +-device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' +-device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' +-device 'usb-tablet,id=tablet,bus=uhci.0,port=1' +-device 'virtio-vga-gl,id=vga,bus=pci.0,addr=0x2' +-device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' +-iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' +-machine 'type=pc+pve0' \ No newline at end of file diff --git a/src/test/cfg2cmd/kyber.conf b/src/test/cfg2cmd/kyber.conf new file mode 100644 index 0000000..31000dd --- /dev/null +++ b/src/test/cfg2cmd/kyber.conf @@ -0,0 +1,3 @@ +# TEST: Kyber console display +memory: 2048 +vga: kyber diff --git a/src/test/cfg2cmd/kyber.conf.cmd b/src/test/cfg2cmd/kyber.conf.cmd new file mode 100644 index 0000000..dfb2e99 --- /dev/null +++ b/src/test/cfg2cmd/kyber.conf.cmd @@ -0,0 +1,27 @@ +/usr/bin/kvm +-id 8006 +-name vm8006 +-no-shutdown +-chardev 'socket,id=qmp,path=/var/run/qemu-server/8006.qmp,server=on,wait=off' +-mon 'chardev=qmp,mode=control' +-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' +-mon 'chardev=qmp-event,mode=control' +-pidfile /var/run/qemu-server/8006.pid +-daemonize +-smp '1,sockets=1,cores=1,maxcpus=1' +-nodefaults +-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' +-display 'dbus,addr=unix:path=/var/run/qemu-server/8006.dbusdisplay' +-vnc 'unix:/var/run/qemu-server/8006.vnc,password=on' +-cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep +-m 2048 +-global 'PIIX4_PM.disable_s3=1' +-global 'PIIX4_PM.disable_s4=1' +-device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' +-device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' +-device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' +-device 'usb-tablet,id=tablet,bus=uhci.0,port=1' +-device 'virtio-vga,id=vga,bus=pci.0,addr=0x2' +-device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' +-iscsi 'initiator-name=iqn.1993-08.org.debian:01:aabbccddeeff' +-machine 'type=pc+pve0' \ No newline at end of file diff --git a/src/usr/Makefile b/src/usr/Makefile index 1365544..58dbb1d 100644 --- a/src/usr/Makefile +++ b/src/usr/Makefile @@ -22,6 +22,7 @@ install: pve-usb.cfg pve-q35.cfg pve-q35-4.0.cfg bootsplash.jpg modules-load.con install -D -m 0755 dbus-vmstate $(LIBEXECDIR)/dbus-vmstate install -d $(LIBSYSTEMDDIR) install -D -m 0644 pve-dbus-vmstate@.service $(LIBSYSTEMDDIR)/system/pve-dbus-vmstate@.service + install -D -m 0644 pve-qemu-kyber@.service $(LIBSYSTEMDDIR)/system/pve-qemu-kyber@.service install -d $(DBUSDIR) install -D -m 0644 org.qemu.VMState1.conf $(DBUSDIR)/system.d/org.qemu.VMState1.conf diff --git a/src/usr/pve-qemu-kyber@.service b/src/usr/pve-qemu-kyber@.service new file mode 100644 index 0000000..46e394e --- /dev/null +++ b/src/usr/pve-qemu-kyber@.service @@ -0,0 +1,25 @@ +[Unit] +Description=PVE Kyber Console Controller (VM %i) +# Tie it to the VM's scope: it goes away with the VM. +PartOf=%i.scope +After=%i.scope + +[Service] +Slice=qemu.slice +Type=simple +# The adapters find their VM's QEMU here rather than on a session bus. +Environment=KQS_DBUS_ADDR=unix:path=/var/run/qemu-server/%i.dbusdisplay +# Carries KYBER_JWT_KEY: /proc//cmdline is readable by every user. +EnvironmentFile=/var/run/qemu-server/%i.kyber.env +# No configuration file: one line is all this needs. The control plane is a unix +# socket; the data plane needs a UDP port, and --dataplane-addr keeps it off +# every other interface - upstream binds the wildcard. +ExecStart=/usr/bin/kycontroller \ + --listen-socket /var/run/qemu-server/%i.kyber.sock \ + --dataplane-addr 127.0.0.1 \ + --tls-cert /etc/pve/local/pve-ssl.pem \ + --tls-key /etc/pve/local/pve-ssl.key \ + --no-basic-auth \ + --no-oidc-auth \ + --no-tray +Restart=no -- 2.55.0