* [RFC] WebRTC console for VMs via QEMU's D-Bus display
@ 2026-10-01 15:10 Dmitry R
2026-10-07 14:10 ` SPAM: " Dmitry R
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry R @ 2026-10-01 15:10 UTC (permalink / raw)
To: pve-devel
Hi all,
I would like to ask whether a hardware-encoded WebRTC console for VMs
would be welcome upstream, and if so, in which shape. It runs today as
an add-on for PVE 9.2, and I have prepared the qemu-server and
pve-manager parts as patch series against current master (tested on
PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion on
the design before sending them.
Feature request: bug #8106
https://bugzilla.proxmox.com/show_bug.cgi?id=8106
Motivation
----------
noVNC is the right console for firmware, installers and text consoles.
For time spent inside a running desktop it transfers changed rectangles
instead of video, has no audio, and a VirGL (virtio-gl) guest reaches it
through egl-headless readback. Measured with one guest image and the
same workloads (method and raw numbers in the repository below):
WebRTC, virtio-gl noVNC, std VGA
720p video, pictures/s 30 (source rate) 7.4
scrolling, pictures/s 36 (52 on NVENC) 12.0
animated UI, pictures/s 38 (45 on NVENC) 11.0
bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
single pointer event 88 ms 52 ms
console open -> 1st picture 3.4 s 1.0 s
The last two rows are where noVNC stays better: a single isolated
event is one small rectangle for VNC, but a full encode/decode round
trip for video (the 88 ms were measured with software libx264 on a
CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
streaming the animated scene.
What QEMU already provides
--------------------------
QEMU's D-Bus display (-display dbus) exports the guest scanout -- the
GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard, mouse,
multi-touch, clipboard and audio. With p2p=yes there is no bus daemon:
a client is attached per connection through QMP (getfd + add_client),
which PVE::QMPClient can already do (it passes the fd for getfd).
Proposed series
---------------
1. qemu-server: 'rendernode' for the VGA property
vga: virtio-gl,rendernode=/dev/dri/renderD129
so a virtio-gl guest can be placed on a chosen GPU instead of the
first render node (fixes #4771). Small and useful on its own.
2. qemu-server: opt-in 'dbus' for the VGA property
vga: virtio-gl,dbus=1
adds '-display dbus,p2p=yes' (gl=on with the render node for
virtio-gl, gl=off for the other types). One constraint: with GL,
QEMU refuses VNC and SPICE next to it ("Display vnc is incompatible
with the GL context"), so a virtio-gl VM with dbus=1 has neither and
offers the WebRTC console instead; non-GL types keep VNC as well.
The clipboard reaches the guest through qemu-vdagent (as for the
VNC clipboard) and the guest's spice-vdagent.
Audio is not part of the series yet: the add-on plays it through a
bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
with p2p=yes our listener got no audio from QEMU 11.0 -- only the
Init/SetEnabled calls made while RegisterOutListener runs, nothing
afterwards. A 'dbus' audio0 driver is ready once that is resolved.
3. qemu-server: API call to attach a console
POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
takes the browser's SDP offer, creates a socketpair, attaches one
end to QEMU (getfd + add_client) and hands the other end with the
offer to a local media service over a root-owned unix socket
(SCM_RIGHTS); returns the SDP answer. Authentication and ACL stay
exactly those of vncproxy/termproxy -- no extra port, no own auth.
The status API reports 'dbus' and 'vnc' so the UI knows which
consoles a VM offers.
4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
in the console button and the VM's Console panel, offered when the
VM's display has dbus=1; the display editor gets the two options.
Like noVNC and xterm.js, the viewer itself is a page from a separate
package that pveproxy serves ('?console=kvm&webrtc=1').
5. A separate package with the media service: capture from the D-Bus
display (the virtio-gl DMA-BUF scanout is imported through EGL and
read back; a zero-copy path to the encoder is future work), encode
(NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
browser decodes it),
Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a Python/aiortc
WebRTC bridge; I realise Python/aiortc is unusual in your stack.
Questions
---------
- Is this something you would consider upstream at all?
- Property naming: 'rendernode' and 'dbus' inside 'vga', or a separate
display property?
- Would you rather keep the media service out of tree (users install
it, the API call returns an error when it is absent), or should it
eventually live in a Proxmox-maintained package, and would you then
want it in Rust?
- Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
Not part of this proposal: the prototype also has a graphical console
for LXC containers (a headless compositor in the container with its
own login screen, one console per tty like the terminal console). It
needs software inside the guest, so I plan to keep it as a separate
add-on package.
Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
9.2.11 plus the series): a Kubuntu 26.04 guest with
'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
through webrtcproxy, in a popup and in the VM's Console panel, for a
user holding only PVEVMUser on the VM (403 without VM.Console).
Code and measurements: https://github.com/rdmitry0911/qsm-rd
(qsm-direct/, docs/COMPARISON.md for the numbers; the prepared series
are in qsm-direct/upstream/patches/).
Transparency: I developed the prototype with the help of an AI coding
assistant (Claude). I have reviewed and tested the code and will sign
the Harmony CLA; please tell me if that is a problem for you.
Thanks,
Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* SPAM: RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
2026-10-01 15:10 [RFC] WebRTC console for VMs via QEMU's D-Bus display Dmitry R
@ 2026-10-07 14:10 ` Dmitry R
2026-10-14 13:12 ` DERUMIER, Alexandre
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry R @ 2026-10-07 14:10 UTC (permalink / raw)
To: pve-devel
Hi all, hi Alexandre,
only after sending this RFC I noticed that Alexandre posted "add rdp &&
kyber consoles for qemu over D-Bus display" in August [1]. Both proposals
need the same foundation in qemu-server -- QEMU's D-Bus display -- and
differ in what consumes it. Rather than two competing ways to enable it, I
would suggest one shared base with the consoles on top. What the two series
have in common:
*
qemu-server enables '-display dbus' for a VM, and a separate package
attaches to it and streams the guest to the browser (RDP and Kyber daemons
there, a WebRTC media service here);
*
with GL, QEMU refuses VNC and SPICE next to the D-Bus display, so
both have to deal with a VM that has no VNC console. Where they differ, and
why I think Alexandre's base is the better one:
*
His series runs a private dbus-daemon per VM; mine uses p2p=yes and
attaches every console with getfd + add_client.
*
With p2p=yes, each add_client replaces the previous connection, so
only one consumer can be attached at a time. On a bus, RDP, Kyber and WebRTC
can attach to the same VM side by side.
*
With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
AudioOutListener gets the Init/SetEnabled calls made while
RegisterOutListener runs, then nothing (the same with a plain GDBus client,
so it does not look like a problem of my listener). On a private bus, which
my add-on uses today, both playback and the microphone (AudioInListener)
work.
A possible split:
1.
qemu-server: the per-VM D-Bus display (Alexandre's DBusDisplay.pm),
the option that enables it, 'rendernode' for virtio-gl (bug #4771) and a
'dbus' driver for audio0.
2.
One package per console, each attaching as a client on that bus: RDP
and Kyber (Alexandre), WebRTC (mine, bug #8106), with their API calls and UI
entries in qemu-server and pve-manager.
I am happy to rebase the WebRTC part onto Alexandre's DBusDisplay.pm and
drop my p2p attach and the separate 'dbus' VGA flag. Alexandre wrote that
the DMA-BUF path is untested on NVIDIA, so I tried it on an RTX 3080 (driver
595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a virtio-vga-gl guest on a
private per-VM bus; kyber-qemu-server and txproto built from the v2 patch
against Debian trixie's FFmpeg 7.1, encoding to a file):
- --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24 with the
NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI frames
context fails on the first frame -- "Failed to create surface: 14 (the
requested RT Format is not supported)" -- and nothing is
encoded. With --encoder h264_nvenc it ends the same way, since the path goes
through VAAPI whatever the encoder.
- The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-driver, which
only decodes (VAEntrypointVLD): no encoding, no video processing, no RGB
surfaces. The mmap fallback would not help either, as a block-linear buffer
does not read as linear memory.
- Without --dmabuf, the listener received no scanout at all from the GL
display within 15 s (libx264 and h264_nvenc alike).
- Control on the same VM: my worker imports the same DMA-BUF with EGL
(eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier), reads it back
with glReadPixels and encodes it with NVENC -- that works. So with a
virtio-gl guest on NVIDIA, the Kyber console shows no picture today.
A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
kyber-qemu-server (358 lines, mostly one new file, LGPL like dmabuf.c). It
still requests DMA-BUF scanouts, imports each buffer with EGL (GBM platform,
surfaceless context, the modifier passed), reads it back with glReadPixels
and feeds your existing software path, so any --encoder works -- h264_nvenc
on NVIDIA. A buffer is imported once and read only after QEMU reports
damage. Same setup, a terminal running 'top -d 0.1' for 20 s: 1226 frames
encoded with h264_nvenc,
0 dropped, the picture correct; about 24% of one CPU core (36% with
libx264). The --dmabuf path is unchanged; kycontroller could choose
'--dmabuf-readback --encoder h264_nvenc' when the render node belongs to the
nvidia driver.
To be fair about the trade-off: the readback copies every changed frame into
system memory, so on AMD and Intel your zero-copy VAAPI path is likely the
more efficient one. The best of both would be zero-copy where VAAPI can
import and encode, and the EGL readback everywhere else -- the same split
would serve the WebRTC console.
Maintainers: would one shared base series like this work for you, and is
there a preferred name and place for the option that enables the D-Bus
display?
[1]
https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-alexandre.derumi
er@groupe-cyllene.com/
[2]
https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-direct/upstream/kyber-nv
idia/0001-kyber-qemu-server-read-DMA-BUF-scanouts-back-through.patch
Thanks,
Dmitry
_____
From: Dmitry R [mailto:rdmitry0911@gmail.com]
Sent: Thursday, October 01, 2026 6:11 PM
To: pve-devel@lists.proxmox.com
Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
Hi all,
I would like to ask whether a hardware-encoded WebRTC console for VMs
would be welcome upstream, and if so, in which shape. It runs today as
an add-on for PVE 9.2, and I have prepared the qemu-server and
pve-manager parts as patch series against current master (tested on
PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion on
the design before sending them.
Feature request: bug #8106
https://bugzilla.proxmox.com/show_bug.cgi?id=8106
Motivation
----------
noVNC is the right console for firmware, installers and text consoles.
For time spent inside a running desktop it transfers changed rectangles
instead of video, has no audio, and a VirGL (virtio-gl) guest reaches it
through egl-headless readback. Measured with one guest image and the
same workloads (method and raw numbers in the repository below):
WebRTC, virtio-gl noVNC, std VGA
720p video, pictures/s 30 (source rate) 7.4
scrolling, pictures/s 36 (52 on NVENC) 12.0
animated UI, pictures/s 38 (45 on NVENC) 11.0
bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
single pointer event 88 ms 52 ms
console open -> 1st picture 3.4 s 1.0 s
The last two rows are where noVNC stays better: a single isolated
event is one small rectangle for VNC, but a full encode/decode round
trip for video (the 88 ms were measured with software libx264 on a
CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
streaming the animated scene.
What QEMU already provides
--------------------------
QEMU's D-Bus display (-display dbus) exports the guest scanout -- the
GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard, mouse,
multi-touch, clipboard and audio. With p2p=yes there is no bus daemon:
a client is attached per connection through QMP (getfd + add_client),
which PVE::QMPClient can already do (it passes the fd for getfd).
Proposed series
---------------
1. qemu-server: 'rendernode' for the VGA property
vga: virtio-gl,rendernode=/dev/dri/renderD129
so a virtio-gl guest can be placed on a chosen GPU instead of the
first render node (fixes #4771). Small and useful on its own.
2. qemu-server: opt-in 'dbus' for the VGA property
vga: virtio-gl,dbus=1
adds '-display dbus,p2p=yes' (gl=on with the render node for
virtio-gl, gl=off for the other types). One constraint: with GL,
QEMU refuses VNC and SPICE next to it ("Display vnc is incompatible
with the GL context"), so a virtio-gl VM with dbus=1 has neither and
offers the WebRTC console instead; non-GL types keep VNC as well.
The clipboard reaches the guest through qemu-vdagent (as for the
VNC clipboard) and the guest's spice-vdagent.
Audio is not part of the series yet: the add-on plays it through a
bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
with p2p=yes our listener got no audio from QEMU 11.0 -- only the
Init/SetEnabled calls made while RegisterOutListener runs, nothing
afterwards. A 'dbus' audio0 driver is ready once that is resolved.
3. qemu-server: API call to attach a console
POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
takes the browser's SDP offer, creates a socketpair, attaches one
end to QEMU (getfd + add_client) and hands the other end with the
offer to a local media service over a root-owned unix socket
(SCM_RIGHTS); returns the SDP answer. Authentication and ACL stay
exactly those of vncproxy/termproxy -- no extra port, no own auth.
The status API reports 'dbus' and 'vnc' so the UI knows which
consoles a VM offers.
4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
in the console button and the VM's Console panel, offered when the
VM's display has dbus=1; the display editor gets the two options.
Like noVNC and xterm.js, the viewer itself is a page from a separate
package that pveproxy serves ('?console=kvm&webrtc=1').
5. A separate package with the media service: capture from the D-Bus
display (the virtio-gl DMA-BUF scanout is imported through EGL and
read back; a zero-copy path to the encoder is future work), encode
(NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
browser decodes it),
Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a Python/aiortc
WebRTC bridge; I realise Python/aiortc is unusual in your stack.
Questions
---------
- Is this something you would consider upstream at all?
- Property naming: 'rendernode' and 'dbus' inside 'vga', or a separate
display property?
- Would you rather keep the media service out of tree (users install
it, the API call returns an error when it is absent), or should it
eventually live in a Proxmox-maintained package, and would you then
want it in Rust?
- Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
Not part of this proposal: the prototype also has a graphical console
for LXC containers (a headless compositor in the container with its
own login screen, one console per tty like the terminal console). It
needs software inside the guest, so I plan to keep it as a separate
add-on package.
Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
9.2.11 plus the series): a Kubuntu 26.04 guest with
'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
through webrtcproxy, in a popup and in the VM's Console panel, for a
user holding only PVEVMUser on the VM (403 without VM.Console).
Code and measurements: https://github.com/rdmitry0911/qsm-rd
(qsm-direct/, docs/COMPARISON.md for the numbers; the prepared series
are in qsm-direct/upstream/patches/).
Transparency: I developed the prototype with the help of an AI coding
assistant (Claude). I have reviewed and tested the code and will sign
the Harmony CLA; please tell me if that is a problem for you.
Thanks,
Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
2026-10-07 14:10 ` SPAM: " Dmitry R
@ 2026-10-14 13:12 ` DERUMIER, Alexandre
2026-10-14 14:19 ` SPAM: " Dmitry R
0 siblings, 1 reply; 7+ messages in thread
From: DERUMIER, Alexandre @ 2026-10-14 13:12 UTC (permalink / raw)
To: pve-devel@lists.proxmox.com, rdmitry0911@gmail.com
Hi Dmitry,
Thanks for testing on nvidia, I didn't have hardware to test it.
Feel free to rework my patches, I was waiting from proxmox devs review
&& comments before doing more work.
I was waiting for the merge of this patch too:
[PATCH qemu-server/pve-manager/docs v4 0/1] add virtio-vga-gl OpenGL
4.6 & Vulkan (venus) support
https://lore.proxmox.com/all/20260831094605.114396-1-m.frank@proxmox.com/
to rebase my serie.
I'll be pretty busy at work until next month, so I'll don't have too
much time to work on it.
but I'm glad that somebody is interested by me patch serie :)
Best Regards,
Alexandre
Le mercredi 07 octobre 2026 à 17:10 +0300, Dmitry R a écrit :
>
> Hi all, hi Alexandre,
>
> only after sending this RFC I noticed that Alexandre posted "add rdp
> && kyber consoles for qemu over D-Bus display" in August [1]. Both
> proposals need the same foundation in qemu-server -- QEMU's D-Bus
> display -- and differ in what consumes it. Rather than two competing
> ways to enable it, I would suggest one shared base with the consoles
> on top. What the two series have in common:
> * qemu-server enables '-display dbus' for a VM, and a separate
> package attaches to it and streams the guest to the browser (RDP and
> Kyber daemons there, a WebRTC media service here);
> * with GL, QEMU refuses VNC and SPICE next to the D-Bus display, so
> both have to deal with a VM that has no VNC console. Where they
> differ, and why I think Alexandre's base is the better one:
> * His series runs a private dbus-daemon per VM; mine uses p2p=yes
> and attaches every console with getfd + add_client.
> * With p2p=yes, each add_client replaces the previous connection,
> so only one consumer can be attached at a time. On a bus, RDP, Kyber
> and WebRTC can attach to the same VM side by side.
> * With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
> AudioOutListener gets the Init/SetEnabled calls made while
> RegisterOutListener runs, then nothing (the same with a plain GDBus
> client, so it does not look like a problem of my listener). On a
> private bus, which my add-on uses today, both playback and the
> microphone (AudioInListener) work.
> A possible split:
> 1. qemu-server: the per-VM D-Bus display (Alexandre's
> DBusDisplay.pm), the option that enables it, 'rendernode' for virtio-
> gl (bug #4771) and a 'dbus' driver for audio0.
> 2. One package per console, each attaching as a client on that
> bus: RDP and Kyber (Alexandre), WebRTC (mine, bug #8106), with their
> API calls and UI entries in qemu-server and pve-manager.
>
> I am happy to rebase the WebRTC part onto Alexandre's DBusDisplay.pm
> and drop my p2p attach and the separate 'dbus' VGA flag. Alexandre
> wrote that the DMA-BUF path is untested on NVIDIA, so I tried it on
> an RTX 3080 (driver 595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a
> virtio-vga-gl guest on a private per-VM bus; kyber-qemu-server and
> txproto built from the v2 patch against Debian trixie's FFmpeg 7.1,
> encoding to a file):
>
> - --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24 with
> the NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI
> frames context fails on the first frame -- "Failed to create surface:
> 14 (the requested RT Format is not supported)" -- and nothing is
> encoded. With --encoder h264_nvenc it ends the same way, since the
> path goes through VAAPI whatever the encoder.
> - The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-
> driver, which only decodes (VAEntrypointVLD): no encoding, no video
> processing, no RGB surfaces. The mmap fallback would not help either,
> as a block-linear buffer does not read as linear memory.
> - Without --dmabuf, the listener received no scanout at all from the
> GL display within 15 s (libx264 and h264_nvenc alike).
> - Control on the same VM: my worker imports the same DMA-BUF with EGL
> (eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier), reads
> it back with glReadPixels and encodes it with NVENC -- that works. So
> with a virtio-gl guest on NVIDIA, the Kyber console shows no picture
> today.
>
> A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
> kyber-qemu-server (358 lines, mostly one new file, LGPL like
> dmabuf.c). It still requests DMA-BUF scanouts, imports each buffer
> with EGL (GBM platform, surfaceless context, the modifier passed),
> reads it back with glReadPixels and feeds your existing software
> path, so any --encoder works -- h264_nvenc on NVIDIA. A buffer is
> imported once and read only after QEMU reports damage. Same setup, a
> terminal running 'top -d 0.1' for 20 s: 1226 frames encoded with
> h264_nvenc,
> 0 dropped, the picture correct; about 24% of one CPU core (36% with
> libx264). The --dmabuf path is unchanged; kycontroller could choose
> '--dmabuf-readback --encoder h264_nvenc' when the render node belongs
> to the nvidia driver.
>
> To be fair about the trade-off: the readback copies every changed
> frame into system memory, so on AMD and Intel your zero-copy VAAPI
> path is likely the more efficient one. The best of both would be
> zero-copy where VAAPI can import and encode, and the EGL readback
> everywhere else -- the same split would serve the WebRTC console.
>
> Maintainers: would one shared base series like this work for you, and
> is there a preferred name and place for the option that enables the
> D-Bus display?
> [1] https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-
> alexandre.derumier@groupe-cyllene.com/
> [2] https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-
> direct/upstream/kyber-nvidia/0001-kyber-qemu-server-read-DMA-BUF-
> scanouts-back-through.patch
>
> Thanks,
> Dmitry
>
>
> From: Dmitry R [mailto:rdmitry0911@gmail.com]
> Sent: Thursday, October 01, 2026 6:11 PM
> To: pve-devel@lists.proxmox.com
> Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi all,
>
> I would like to ask whether a hardware-encoded WebRTC console for VMs
> would be welcome upstream, and if so, in which shape. It runs today
> as
> an add-on for PVE 9.2, and I have prepared the qemu-server and
> pve-manager parts as patch series against current master (tested on
> PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion on
> the design before sending them.
>
> Feature request: bug #8106
> https://bugzilla.proxmox.com/show_bug.cgi?id=8106
>
> Motivation
> ----------
> noVNC is the right console for firmware, installers and text
> consoles.
> For time spent inside a running desktop it transfers changed
> rectangles
> instead of video, has no audio, and a VirGL (virtio-gl) guest reaches
> it
> through egl-headless readback. Measured with one guest image and the
> same workloads (method and raw numbers in the repository below):
>
> WebRTC, virtio-gl noVNC, std VGA
> 720p video, pictures/s 30 (source rate) 7.4
> scrolling, pictures/s 36 (52 on NVENC) 12.0
> animated UI, pictures/s 38 (45 on NVENC) 11.0
> bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
> single pointer event 88 ms 52 ms
> console open -> 1st picture 3.4 s 1.0 s
>
> The last two rows are where noVNC stays better: a single isolated
> event is one small rectangle for VNC, but a full encode/decode round
> trip for video (the 88 ms were measured with software libx264 on a
> CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
> streaming the animated scene.
>
> What QEMU already provides
> --------------------------
> QEMU's D-Bus display (-display dbus) exports the guest scanout -- the
> GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard,
> mouse,
> multi-touch, clipboard and audio. With p2p=yes there is no bus
> daemon:
> a client is attached per connection through QMP (getfd + add_client),
> which PVE::QMPClient can already do (it passes the fd for getfd).
>
> Proposed series
> ---------------
> 1. qemu-server: 'rendernode' for the VGA property
> vga: virtio-gl,rendernode=/dev/dri/renderD129
> so a virtio-gl guest can be placed on a chosen GPU instead of the
> first render node (fixes #4771). Small and useful on its own.
>
> 2. qemu-server: opt-in 'dbus' for the VGA property
> vga: virtio-gl,dbus=1
> adds '-display dbus,p2p=yes' (gl=on with the render node for
> virtio-gl, gl=off for the other types). One constraint: with GL,
> QEMU refuses VNC and SPICE next to it ("Display vnc is
> incompatible
> with the GL context"), so a virtio-gl VM with dbus=1 has neither
> and
> offers the WebRTC console instead; non-GL types keep VNC as well.
> The clipboard reaches the guest through qemu-vdagent (as for the
> VNC clipboard) and the guest's spice-vdagent.
> Audio is not part of the series yet: the add-on plays it through a
> bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
> with p2p=yes our listener got no audio from QEMU 11.0 -- only the
> Init/SetEnabled calls made while RegisterOutListener runs, nothing
> afterwards. A 'dbus' audio0 driver is ready once that is
> resolved.
>
> 3. qemu-server: API call to attach a console
> POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
> takes the browser's SDP offer, creates a socketpair, attaches one
> end to QEMU (getfd + add_client) and hands the other end with the
> offer to a local media service over a root-owned unix socket
> (SCM_RIGHTS); returns the SDP answer. Authentication and ACL stay
> exactly those of vncproxy/termproxy -- no extra port, no own auth.
> The status API reports 'dbus' and 'vnc' so the UI knows which
> consoles a VM offers.
>
> 4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
> in the console button and the VM's Console panel, offered when the
> VM's display has dbus=1; the display editor gets the two options.
> Like noVNC and xterm.js, the viewer itself is a page from a
> separate
> package that pveproxy serves ('?console=kvm&webrtc=1').
>
> 5. A separate package with the media service: capture from the D-Bus
> display (the virtio-gl DMA-BUF scanout is imported through EGL and
> read back; a zero-copy path to the encoder is future work), encode
> (NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
> browser decodes it),
> Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a Python/aiortc
> WebRTC bridge; I realise Python/aiortc is unusual in your stack.
>
> Questions
> ---------
> - Is this something you would consider upstream at all?
> - Property naming: 'rendernode' and 'dbus' inside 'vga', or a
> separate
> display property?
> - Would you rather keep the media service out of tree (users install
> it, the API call returns an error when it is absent), or should it
> eventually live in a Proxmox-maintained package, and would you then
> want it in Rust?
> - Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
>
> Not part of this proposal: the prototype also has a graphical console
> for LXC containers (a headless compositor in the container with its
> own login screen, one console per tty like the terminal console). It
> needs software inside the guest, so I plan to keep it as a separate
> add-on package.
>
> Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
> 9.2.11 plus the series): a Kubuntu 26.04 guest with
> 'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
> through webrtcproxy, in a popup and in the VM's Console panel, for a
> user holding only PVEVMUser on the VM (403 without VM.Console).
>
> Code and measurements: https://github.com/rdmitry0911/qsm-rd
> (qsm-direct/, docs/COMPARISON.md for the numbers; the prepared series
> are in qsm-direct/upstream/patches/).
>
> Transparency: I developed the prototype with the help of an AI coding
> assistant (Claude). I have reviewed and tested the code and will
> sign
> the Harmony CLA; please tell me if that is a problem for you.
>
> Thanks,
> Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* SPAM: RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
2026-10-14 13:12 ` DERUMIER, Alexandre
@ 2026-10-14 14:19 ` Dmitry R
0 siblings, 0 replies; 7+ messages in thread
From: Dmitry R @ 2026-10-14 14:19 UTC (permalink / raw)
To: 'DERUMIER, Alexandre', pve-devel
Hi Alexandre,
Thanks for your reply, and for making your patches available. I'm glad
we can build on the same foundation.
I'll work from your v2 series, first separating the shared qemu-server
D-Bus display support from the console-specific parts. Then I'll rebase
the WebRTC console onto that base, using the per-VM bus from your
DBusDisplay.pm. I'll keep the media service in a separate package and
preserve your authorship on the original patches.
I'll also include the NVIDIA EGL readback fix I linked in my previous
mail, keeping the existing VAAPI path available. My testing of that fix
was limited to the capture/encoding adapter with output to a file; I
still need to check it through the full Kyber console in the browser.
I'll take the OpenGL 4.6 / Venus series you mentioned into account when
rebasing. The shared base will still need feedback from the Proxmox
developers on the configuration and integration.
No problem about your availability. I'll send the revised patches to
the list with you in Cc, so you can have a look when you have time.
Thanks,
Dmitry
-----Original Message-----
From: DERUMIER, Alexandre [mailto:alexandre.derumier@groupe-cyllene.com]
Sent: Wednesday, October 14, 2026 4:12 PM
To: pve-devel@lists.proxmox.com; rdmitry0911@gmail.com
Subject: Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
Hi Dmitry,
Thanks for testing on nvidia, I didn't have hardware to test it.
Feel free to rework my patches, I was waiting from proxmox devs review &&
comments before doing more work.
I was waiting for the merge of this patch too:
[PATCH qemu-server/pve-manager/docs v4 0/1] add virtio-vga-gl OpenGL
4.6 & Vulkan (venus) support
https://lore.proxmox.com/all/20260831094605.114396-1-m.frank@proxmox.com/
to rebase my serie.
I'll be pretty busy at work until next month, so I'll don't have too much
time to work on it.
but I'm glad that somebody is interested by me patch serie :)
Best Regards,
Alexandre
Le mercredi 07 octobre 2026 à 17:10 +0300, Dmitry R a écrit :
>
> Hi all, hi Alexandre,
>
> only after sending this RFC I noticed that Alexandre posted "add rdp
> && kyber consoles for qemu over D-Bus display" in August [1]. Both
> proposals need the same foundation in qemu-server -- QEMU's D-Bus
> display -- and differ in what consumes it. Rather than two competing
> ways to enable it, I would suggest one shared base with the consoles
> on top. What the two series have in common:
> * qemu-server enables '-display dbus' for a VM, and a separate
> package attaches to it and streams the guest to the browser (RDP and
> Kyber daemons there, a WebRTC media service here);
> * with GL, QEMU refuses VNC and SPICE next to the D-Bus display, so
> both have to deal with a VM that has no VNC console. Where they
> differ, and why I think Alexandre's base is the better one:
> * His series runs a private dbus-daemon per VM; mine uses p2p=yes
> and attaches every console with getfd + add_client.
> * With p2p=yes, each add_client replaces the previous connection, so
> only one consumer can be attached at a time. On a bus, RDP, Kyber and
> WebRTC can attach to the same VM side by side.
> * With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
> AudioOutListener gets the Init/SetEnabled calls made while
> RegisterOutListener runs, then nothing (the same with a plain GDBus
> client, so it does not look like a problem of my listener). On a
> private bus, which my add-on uses today, both playback and the
> microphone (AudioInListener) work.
> A possible split:
> 1. qemu-server: the per-VM D-Bus display (Alexandre's
> DBusDisplay.pm), the option that enables it, 'rendernode' for virtio-
> gl (bug #4771) and a 'dbus' driver for audio0.
> 2. One package per console, each attaching as a client on that
> bus: RDP and Kyber (Alexandre), WebRTC (mine, bug #8106), with their
> API calls and UI entries in qemu-server and pve-manager.
>
> I am happy to rebase the WebRTC part onto Alexandre's DBusDisplay.pm
> and drop my p2p attach and the separate 'dbus' VGA flag. Alexandre
> wrote that the DMA-BUF path is untested on NVIDIA, so I tried it on an
> RTX 3080 (driver 595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a
> virtio-vga-gl guest on a private per-VM bus; kyber-qemu-server and
> txproto built from the v2 patch against Debian trixie's FFmpeg 7.1,
> encoding to a file):
>
> - --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24 with
> the NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI
> frames context fails on the first frame -- "Failed to create surface:
> 14 (the requested RT Format is not supported)" -- and nothing is
> encoded. With --encoder h264_nvenc it ends the same way, since the
> path goes through VAAPI whatever the encoder.
> - The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-
> driver, which only decodes (VAEntrypointVLD): no encoding, no video
> processing, no RGB surfaces. The mmap fallback would not help either,
> as a block-linear buffer does not read as linear memory.
> - Without --dmabuf, the listener received no scanout at all from the
> GL display within 15 s (libx264 and h264_nvenc alike).
> - Control on the same VM: my worker imports the same DMA-BUF with EGL
> (eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier), reads it
> back with glReadPixels and encodes it with NVENC -- that works. So
> with a virtio-gl guest on NVIDIA, the Kyber console shows no picture
> today.
>
> A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
> kyber-qemu-server (358 lines, mostly one new file, LGPL like
> dmabuf.c). It still requests DMA-BUF scanouts, imports each buffer
> with EGL (GBM platform, surfaceless context, the modifier passed),
> reads it back with glReadPixels and feeds your existing software path,
> so any --encoder works -- h264_nvenc on NVIDIA. A buffer is imported
> once and read only after QEMU reports damage. Same setup, a terminal
> running 'top -d 0.1' for 20 s: 1226 frames encoded with h264_nvenc, 0
> dropped, the picture correct; about 24% of one CPU core (36% with
> libx264). The --dmabuf path is unchanged; kycontroller could choose
> '--dmabuf-readback --encoder h264_nvenc' when the render node belongs
> to the nvidia driver.
>
> To be fair about the trade-off: the readback copies every changed
> frame into system memory, so on AMD and Intel your zero-copy VAAPI
> path is likely the more efficient one. The best of both would be
> zero-copy where VAAPI can import and encode, and the EGL readback
> everywhere else -- the same split would serve the WebRTC console.
>
> Maintainers: would one shared base series like this work for you, and
> is there a preferred name and place for the option that enables the
> D-Bus display?
> [1] https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-
> alexandre.derumier@groupe-cyllene.com/
> [2] https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-
> direct/upstream/kyber-nvidia/0001-kyber-qemu-server-read-DMA-BUF-
> scanouts-back-through.patch
>
> Thanks,
> Dmitry
>
>
> From: Dmitry R [mailto:rdmitry0911@gmail.com]
> Sent: Thursday, October 01, 2026 6:11 PM
> To: pve-devel@lists.proxmox.com
> Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi all,
>
> I would like to ask whether a hardware-encoded WebRTC console for VMs
> would be welcome upstream, and if so, in which shape. It runs today
> as an add-on for PVE 9.2, and I have prepared the qemu-server and
> pve-manager parts as patch series against current master (tested on
> PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion on
> the design before sending them.
>
> Feature request: bug #8106
> https://bugzilla.proxmox.com/show_bug.cgi?id=8106
>
> Motivation
> ----------
> noVNC is the right console for firmware, installers and text consoles.
> For time spent inside a running desktop it transfers changed
> rectangles instead of video, has no audio, and a VirGL (virtio-gl)
> guest reaches it through egl-headless readback. Measured with one
> guest image and the same workloads (method and raw numbers in the
> repository below):
>
> WebRTC, virtio-gl noVNC, std VGA
> 720p video, pictures/s 30 (source rate) 7.4
> scrolling, pictures/s 36 (52 on NVENC) 12.0
> animated UI, pictures/s 38 (45 on NVENC) 11.0
> bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
> single pointer event 88 ms 52 ms
> console open -> 1st picture 3.4 s 1.0 s
>
> The last two rows are where noVNC stays better: a single isolated
> event is one small rectangle for VNC, but a full encode/decode round
> trip for video (the 88 ms were measured with software libx264 on a
> CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
> streaming the animated scene.
>
> What QEMU already provides
> --------------------------
> QEMU's D-Bus display (-display dbus) exports the guest scanout -- the
> GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard, mouse,
> multi-touch, clipboard and audio. With p2p=yes there is no bus
> daemon:
> a client is attached per connection through QMP (getfd + add_client),
> which PVE::QMPClient can already do (it passes the fd for getfd).
>
> Proposed series
> ---------------
> 1. qemu-server: 'rendernode' for the VGA property
> vga: virtio-gl,rendernode=/dev/dri/renderD129
> so a virtio-gl guest can be placed on a chosen GPU instead of the
> first render node (fixes #4771). Small and useful on its own.
>
> 2. qemu-server: opt-in 'dbus' for the VGA property
> vga: virtio-gl,dbus=1
> adds '-display dbus,p2p=yes' (gl=on with the render node for
> virtio-gl, gl=off for the other types). One constraint: with GL,
> QEMU refuses VNC and SPICE next to it ("Display vnc is incompatible
> with the GL context"), so a virtio-gl VM with dbus=1 has neither
> and
> offers the WebRTC console instead; non-GL types keep VNC as well.
> The clipboard reaches the guest through qemu-vdagent (as for the
> VNC clipboard) and the guest's spice-vdagent.
> Audio is not part of the series yet: the add-on plays it through a
> bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
> with p2p=yes our listener got no audio from QEMU 11.0 -- only the
> Init/SetEnabled calls made while RegisterOutListener runs, nothing
> afterwards. A 'dbus' audio0 driver is ready once that is resolved.
>
> 3. qemu-server: API call to attach a console
> POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
> takes the browser's SDP offer, creates a socketpair, attaches one
> end to QEMU (getfd + add_client) and hands the other end with the
> offer to a local media service over a root-owned unix socket
> (SCM_RIGHTS); returns the SDP answer. Authentication and ACL stay
> exactly those of vncproxy/termproxy -- no extra port, no own auth.
> The status API reports 'dbus' and 'vnc' so the UI knows which
> consoles a VM offers.
>
> 4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
> in the console button and the VM's Console panel, offered when the
> VM's display has dbus=1; the display editor gets the two options.
> Like noVNC and xterm.js, the viewer itself is a page from a
> separate
> package that pveproxy serves ('?console=kvm&webrtc=1').
>
> 5. A separate package with the media service: capture from the D-Bus
> display (the virtio-gl DMA-BUF scanout is imported through EGL and
> read back; a zero-copy path to the encoder is future work), encode
> (NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
> browser decodes it),
> Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a Python/aiortc
> WebRTC bridge; I realise Python/aiortc is unusual in your stack.
>
> Questions
> ---------
> - Is this something you would consider upstream at all?
> - Property naming: 'rendernode' and 'dbus' inside 'vga', or a separate
> display property?
> - Would you rather keep the media service out of tree (users install
> it, the API call returns an error when it is absent), or should it
> eventually live in a Proxmox-maintained package, and would you then
> want it in Rust?
> - Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
>
> Not part of this proposal: the prototype also has a graphical console
> for LXC containers (a headless compositor in the container with its
> own login screen, one console per tty like the terminal console). It
> needs software inside the guest, so I plan to keep it as a separate
> add-on package.
>
> Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
> 9.2.11 plus the series): a Kubuntu 26.04 guest with
> 'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
> through webrtcproxy, in a popup and in the VM's Console panel, for a
> user holding only PVEVMUser on the VM (403 without VM.Console).
>
> Code and measurements: https://github.com/rdmitry0911/qsm-rd
> (qsm-direct/, docs/COMPARISON.md for the numbers; the prepared series
> are in qsm-direct/upstream/patches/).
>
> Transparency: I developed the prototype with the help of an AI coding
> assistant (Claude). I have reviewed and tested the code and will sign
> the Harmony CLA; please tell me if that is a problem for you.
>
> Thanks,
> Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
@ 2026-10-15 7:53 Dmitry R
2026-10-15 9:52 ` DERUMIER, Alexandre
0 siblings, 1 reply; 7+ messages in thread
From: Dmitry R @ 2026-10-15 7:53 UTC (permalink / raw)
To: 'DERUMIER, Alexandre', pve-devel
Hi Alexandre, hi all,
Following up on my reply yesterday: I've completed the QSM port to the
shared per-VM D-Bus foundation and uploaded the source, patches and a
matching preview build:
https://github.com/rdmitry0911/qsm-rd/releases/tag/v0.4.0-qsm-direct-2026101
5-shared9
The original D-Bus patch retains Alexandre's authorship and sign-off.
qemu-server owns the bus and its lifetime; QSM attaches as a consumer. The
proposed opt-in is dbus-display: 1, with GPU type/rendernode in vga and
driver=dbus for audio0. CPU displays can keep VNC; GL displays cannot
provide VNC/SPICE alongside the D-Bus display.
The common display foundation, WebRTC API and pve-manager UI are separate
series. A small pve-guest-common module now handles the local console
service exchange for both VM and LXC APIs. Native signalling, status and
media settings use PVE's existing HTTPS origin on8006, with VM.Console for
offers/reads and VM.Config.Options for settings writes. There is no separate
HTTPS port; WebRTC media still uses UDP.
I checked Firefox video and an ordinary Reconnect in the embedded VM
console, and LXC video/settings, with the old8007 service stopped. Anonymous
reads and writes without a CSRF token are rejected. The Firefox mDNS
negotiation and reconnect issues found during VPN testing are fixed. The
earlier shared-base lab checks also covered CPU/GL display, audio,
microphone, clipboard and a second independent display consumer.
The graphical LXC sessions and provisioning remain in a separate add-on.
Their API patches can be reviewed independently of the QEMU foundation.
There are builds against pve-container 6.1.13 and 6.1.14, so the newer
container package does not need to be downgraded.
Markus Frank's Venus configuration adaptation and the NVIDIA readback patch
remain separate optional work. Runtime Venus and full Kyber/RDP browser
integration have not been qualified by these tests. The source bases,
package hashes, test logs and migration instructions are here:
https://github.com/rdmitry0911/qsm-rd/tree/main/qsm-direct/upstream
I'd appreciate feedback on the VM-wide option, bus lifetime, audio
configuration and the API/service split. The patches are still a proposal; I
have not treated the new configuration as an agreed upstream interface.
Thanks,
Dmitry
-----Original Message-----
From: Dmitry R [mailto:rdmitry0911@gmail.com]
Sent: Wednesday, October 14, 2026 5:20 PM
To: 'DERUMIER, Alexandre'; 'pve-devel@lists.proxmox.com'
Subject: RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
Hi Alexandre,
Thanks for your reply, and for making your patches available. I'm glad we
can build on the same foundation.
I'll work from your v2 series, first separating the shared qemu-server D-Bus
display support from the console-specific parts. Then I'll rebase the WebRTC
console onto that base, using the per-VM bus from your DBusDisplay.pm. I'll
keep the media service in a separate package and preserve your authorship on
the original patches.
I'll also include the NVIDIA EGL readback fix I linked in my previous mail,
keeping the existing VAAPI path available. My testing of that fix was
limited to the capture/encoding adapter with output to a file; I still need
to check it through the full Kyber console in the browser.
I'll take the OpenGL 4.6 / Venus series you mentioned into account when
rebasing. The shared base will still need feedback from the Proxmox
developers on the configuration and integration.
No problem about your availability. I'll send the revised patches to the
list with you in Cc, so you can have a look when you have time.
Thanks,
Dmitry
-----Original Message-----
From: DERUMIER, Alexandre [mailto:alexandre.derumier@groupe-cyllene.com]
Sent: Wednesday, October 14, 2026 4:12 PM
To: pve-devel@lists.proxmox.com; rdmitry0911@gmail.com
Subject: Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
Hi Dmitry,
Thanks for testing on nvidia, I didn't have hardware to test it.
Feel free to rework my patches, I was waiting from proxmox devs review &&
comments before doing more work.
I was waiting for the merge of this patch too:
[PATCH qemu-server/pve-manager/docs v4 0/1] add virtio-vga-gl OpenGL
4.6 & Vulkan (venus) support
https://lore.proxmox.com/all/20260831094605.114396-1-m.frank@proxmox.com/
to rebase my serie.
I'll be pretty busy at work until next month, so I'll don't have too much
time to work on it.
but I'm glad that somebody is interested by me patch serie :)
Best Regards,
Alexandre
Le mercredi 07 octobre 2026 à 17:10 +0300, Dmitry R a écrit :
>
> Hi all, hi Alexandre,
>
> only after sending this RFC I noticed that Alexandre posted "add rdp
> && kyber consoles for qemu over D-Bus display" in August [1]. Both
> proposals need the same foundation in qemu-server -- QEMU's D-Bus
> display -- and differ in what consumes it. Rather than two competing
> ways to enable it, I would suggest one shared base with the consoles
> on top. What the two series have in common:
> * qemu-server enables '-display dbus' for a VM, and a separate
> package attaches to it and streams the guest to the browser (RDP and
> Kyber daemons there, a WebRTC media service here);
> * with GL, QEMU refuses VNC and SPICE next to the D-Bus display, so
> both have to deal with a VM that has no VNC console. Where they
> differ, and why I think Alexandre's base is the better one:
> * His series runs a private dbus-daemon per VM; mine uses p2p=yes
> and attaches every console with getfd + add_client.
> * With p2p=yes, each add_client replaces the previous connection, so
> only one consumer can be attached at a time. On a bus, RDP, Kyber and
> WebRTC can attach to the same VM side by side.
> * With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
> AudioOutListener gets the Init/SetEnabled calls made while
> RegisterOutListener runs, then nothing (the same with a plain GDBus
> client, so it does not look like a problem of my listener). On a
> private bus, which my add-on uses today, both playback and the
> microphone (AudioInListener) work.
> A possible split:
> 1. qemu-server: the per-VM D-Bus display (Alexandre's
> DBusDisplay.pm), the option that enables it, 'rendernode' for virtio-
> gl (bug #4771) and a 'dbus' driver for audio0.
> 2. One package per console, each attaching as a client on that
> bus: RDP and Kyber (Alexandre), WebRTC (mine, bug #8106), with their
> API calls and UI entries in qemu-server and pve-manager.
>
> I am happy to rebase the WebRTC part onto Alexandre's DBusDisplay.pm
> and drop my p2p attach and the separate 'dbus' VGA flag. Alexandre
> wrote that the DMA-BUF path is untested on NVIDIA, so I tried it on an
> RTX 3080 (driver 595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a
> virtio-vga-gl guest on a private per-VM bus; kyber-qemu-server and
> txproto built from the v2 patch against Debian trixie's FFmpeg 7.1,
> encoding to a file):
>
> - --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24 with
> the NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI
> frames context fails on the first frame -- "Failed to create surface:
> 14 (the requested RT Format is not supported)" -- and nothing is
> encoded. With --encoder h264_nvenc it ends the same way, since the
> path goes through VAAPI whatever the encoder.
> - The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-
> driver, which only decodes (VAEntrypointVLD): no encoding, no video
> processing, no RGB surfaces. The mmap fallback would not help either,
> as a block-linear buffer does not read as linear memory.
> - Without --dmabuf, the listener received no scanout at all from the
> GL display within 15 s (libx264 and h264_nvenc alike).
> - Control on the same VM: my worker imports the same DMA-BUF with EGL
> (eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier), reads it
> back with glReadPixels and encodes it with NVENC -- that works. So
> with a virtio-gl guest on NVIDIA, the Kyber console shows no picture
> today.
>
> A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
> kyber-qemu-server (358 lines, mostly one new file, LGPL like
> dmabuf.c). It still requests DMA-BUF scanouts, imports each buffer
> with EGL (GBM platform, surfaceless context, the modifier passed),
> reads it back with glReadPixels and feeds your existing software path,
> so any --encoder works -- h264_nvenc on NVIDIA. A buffer is imported
> once and read only after QEMU reports damage. Same setup, a terminal
> running 'top -d 0.1' for 20 s: 1226 frames encoded with h264_nvenc, 0
> dropped, the picture correct; about 24% of one CPU core (36% with
> libx264). The --dmabuf path is unchanged; kycontroller could choose
> '--dmabuf-readback --encoder h264_nvenc' when the render node belongs
> to the nvidia driver.
>
> To be fair about the trade-off: the readback copies every changed
> frame into system memory, so on AMD and Intel your zero-copy VAAPI
> path is likely the more efficient one. The best of both would be
> zero-copy where VAAPI can import and encode, and the EGL readback
> everywhere else -- the same split would serve the WebRTC console.
>
> Maintainers: would one shared base series like this work for you, and
> is there a preferred name and place for the option that enables the
> D-Bus display?
> [1] https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-
> alexandre.derumier@groupe-cyllene.com/
> [2] https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-
> direct/upstream/kyber-nvidia/0001-kyber-qemu-server-read-DMA-BUF-
> scanouts-back-through.patch
>
> Thanks,
> Dmitry
>
>
> From: Dmitry R [mailto:rdmitry0911@gmail.com]
> Sent: Thursday, October 01, 2026 6:11 PM
> To: pve-devel@lists.proxmox.com
> Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi all,
>
> I would like to ask whether a hardware-encoded WebRTC console for VMs
> would be welcome upstream, and if so, in which shape. It runs today
> as an add-on for PVE 9.2, and I have prepared the qemu-server and
> pve-manager parts as patch series against current master (tested on
> PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion on
> the design before sending them.
>
> Feature request: bug #8106
> https://bugzilla.proxmox.com/show_bug.cgi?id=8106
>
> Motivation
> ----------
> noVNC is the right console for firmware, installers and text consoles.
> For time spent inside a running desktop it transfers changed
> rectangles instead of video, has no audio, and a VirGL (virtio-gl)
> guest reaches it through egl-headless readback. Measured with one
> guest image and the same workloads (method and raw numbers in the
> repository below):
>
> WebRTC, virtio-gl noVNC, std VGA
> 720p video, pictures/s 30 (source rate) 7.4
> scrolling, pictures/s 36 (52 on NVENC) 12.0
> animated UI, pictures/s 38 (45 on NVENC) 11.0
> bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
> single pointer event 88 ms 52 ms
> console open -> 1st picture 3.4 s 1.0 s
>
> The last two rows are where noVNC stays better: a single isolated
> event is one small rectangle for VNC, but a full encode/decode round
> trip for video (the 88 ms were measured with software libx264 on a
> CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
> streaming the animated scene.
>
> What QEMU already provides
> --------------------------
> QEMU's D-Bus display (-display dbus) exports the guest scanout -- the
> GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard, mouse,
> multi-touch, clipboard and audio. With p2p=yes there is no bus
> daemon:
> a client is attached per connection through QMP (getfd + add_client),
> which PVE::QMPClient can already do (it passes the fd for getfd).
>
> Proposed series
> ---------------
> 1. qemu-server: 'rendernode' for the VGA property
> vga: virtio-gl,rendernode=/dev/dri/renderD129
> so a virtio-gl guest can be placed on a chosen GPU instead of the
> first render node (fixes #4771). Small and useful on its own.
>
> 2. qemu-server: opt-in 'dbus' for the VGA property
> vga: virtio-gl,dbus=1
> adds '-display dbus,p2p=yes' (gl=on with the render node for
> virtio-gl, gl=off for the other types). One constraint: with GL,
> QEMU refuses VNC and SPICE next to it ("Display vnc is incompatible
> with the GL context"), so a virtio-gl VM with dbus=1 has neither
> and
> offers the WebRTC console instead; non-GL types keep VNC as well.
> The clipboard reaches the guest through qemu-vdagent (as for the
> VNC clipboard) and the guest's spice-vdagent.
> Audio is not part of the series yet: the add-on plays it through a
> bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
> with p2p=yes our listener got no audio from QEMU 11.0 -- only the
> Init/SetEnabled calls made while RegisterOutListener runs, nothing
> afterwards. A 'dbus' audio0 driver is ready once that is resolved.
>
> 3. qemu-server: API call to attach a console
> POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
> takes the browser's SDP offer, creates a socketpair, attaches one
> end to QEMU (getfd + add_client) and hands the other end with the
> offer to a local media service over a root-owned unix socket
> (SCM_RIGHTS); returns the SDP answer. Authentication and ACL stay
> exactly those of vncproxy/termproxy -- no extra port, no own auth.
> The status API reports 'dbus' and 'vnc' so the UI knows which
> consoles a VM offers.
>
> 4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
> in the console button and the VM's Console panel, offered when the
> VM's display has dbus=1; the display editor gets the two options.
> Like noVNC and xterm.js, the viewer itself is a page from a
> separate
> package that pveproxy serves ('?console=kvm&webrtc=1').
>
> 5. A separate package with the media service: capture from the D-Bus
> display (the virtio-gl DMA-BUF scanout is imported through EGL and
> read back; a zero-copy path to the encoder is future work), encode
> (NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
> browser decodes it),
> Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a Python/aiortc
> WebRTC bridge; I realise Python/aiortc is unusual in your stack.
>
> Questions
> ---------
> - Is this something you would consider upstream at all?
> - Property naming: 'rendernode' and 'dbus' inside 'vga', or a separate
> display property?
> - Would you rather keep the media service out of tree (users install
> it, the API call returns an error when it is absent), or should it
> eventually live in a Proxmox-maintained package, and would you then
> want it in Rust?
> - Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
>
> Not part of this proposal: the prototype also has a graphical console
> for LXC containers (a headless compositor in the container with its
> own login screen, one console per tty like the terminal console). It
> needs software inside the guest, so I plan to keep it as a separate
> add-on package.
>
> Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
> 9.2.11 plus the series): a Kubuntu 26.04 guest with
> 'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
> through webrtcproxy, in a popup and in the VM's Console panel, for a
> user holding only PVEVMUser on the VM (403 without VM.Console).
>
> Code and measurements: https://github.com/rdmitry0911/qsm-rd
> (qsm-direct/, docs/COMPARISON.md for the numbers; the prepared series
> are in qsm-direct/upstream/patches/).
>
> Transparency: I developed the prototype with the help of an AI coding
> assistant (Claude). I have reviewed and tested the code and will sign
> the Harmony CLA; please tell me if that is a problem for you.
>
> Thanks,
> Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
2026-10-15 7:53 Dmitry R
@ 2026-10-15 9:52 ` DERUMIER, Alexandre
2026-10-16 6:58 ` Dmitry R
0 siblings, 1 reply; 7+ messages in thread
From: DERUMIER, Alexandre @ 2026-10-15 9:52 UTC (permalink / raw)
To: pve-devel@lists.proxmox.com, rdmitry0911@gmail.com
Hi Dmitry,
I wonder if it could be possible to put your qsm-pve-direct daemon
behind pveproxy , like I do for my rdp daemon ? (it's just forwarding)
This could avoid to expose another port.
(I really don't known if it's possible).
Also, be carefull, 8007 is also used by PBS. (and some user sometimes
install PBS on same host than PVE)
Btw, how do you have benched and retrieve metrics for latency ?
Le jeudi 15 octobre 2026 à 10:53 +0300, Dmitry R a écrit :
> Hi Alexandre, hi all,
>
> Following up on my reply yesterday: I've completed the QSM port to
> the
> shared per-VM D-Bus foundation and uploaded the source, patches and a
> matching preview build:
>
> https://github.com/rdmitry0911/qsm-rd/releases/tag/v0.4.0-qsm-direct-2026101
> 5-shared9
>
> The original D-Bus patch retains Alexandre's authorship and sign-off.
> qemu-server owns the bus and its lifetime; QSM attaches as a
> consumer. The
> proposed opt-in is dbus-display: 1, with GPU type/rendernode in vga
> and
> driver=dbus for audio0. CPU displays can keep VNC; GL displays cannot
> provide VNC/SPICE alongside the D-Bus display.
>
> The common display foundation, WebRTC API and pve-manager UI are
> separate
> series. A small pve-guest-common module now handles the local console
> service exchange for both VM and LXC APIs. Native signalling, status
> and
> media settings use PVE's existing HTTPS origin on8006, with
> VM.Console for
> offers/reads and VM.Config.Options for settings writes. There is no
> separate
> HTTPS port; WebRTC media still uses UDP.
>
> I checked Firefox video and an ordinary Reconnect in the embedded VM
> console, and LXC video/settings, with the old8007 service stopped.
> Anonymous
> reads and writes without a CSRF token are rejected. The Firefox mDNS
> negotiation and reconnect issues found during VPN testing are fixed.
> The
> earlier shared-base lab checks also covered CPU/GL display, audio,
> microphone, clipboard and a second independent display consumer.
>
> The graphical LXC sessions and provisioning remain in a separate add-
> on.
> Their API patches can be reviewed independently of the QEMU
> foundation.
> There are builds against pve-container 6.1.13 and 6.1.14, so the
> newer
> container package does not need to be downgraded.
>
> Markus Frank's Venus configuration adaptation and the NVIDIA readback
> patch
> remain separate optional work. Runtime Venus and full Kyber/RDP
> browser
> integration have not been qualified by these tests. The source bases,
> package hashes, test logs and migration instructions are here:
>
> https://github.com/rdmitry0911/qsm-rd/tree/main/qsm-direct/upstream
>
> I'd appreciate feedback on the VM-wide option, bus lifetime, audio
> configuration and the API/service split. The patches are still a
> proposal; I
> have not treated the new configuration as an agreed upstream
> interface.
>
> Thanks,
> Dmitry
>
> -----Original Message-----
> From: Dmitry R [mailto:rdmitry0911@gmail.com]
> Sent: Wednesday, October 14, 2026 5:20 PM
> To: 'DERUMIER, Alexandre'; 'pve-devel@lists.proxmox.com'
> Subject: RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi Alexandre,
>
> Thanks for your reply, and for making your patches available. I'm
> glad we
> can build on the same foundation.
>
> I'll work from your v2 series, first separating the shared qemu-
> server D-Bus
> display support from the console-specific parts. Then I'll rebase the
> WebRTC
> console onto that base, using the per-VM bus from your
> DBusDisplay.pm. I'll
> keep the media service in a separate package and preserve your
> authorship on
> the original patches.
>
> I'll also include the NVIDIA EGL readback fix I linked in my previous
> mail,
> keeping the existing VAAPI path available. My testing of that fix was
> limited to the capture/encoding adapter with output to a file; I
> still need
> to check it through the full Kyber console in the browser.
>
> I'll take the OpenGL 4.6 / Venus series you mentioned into account
> when
> rebasing. The shared base will still need feedback from the Proxmox
> developers on the configuration and integration.
>
> No problem about your availability. I'll send the revised patches to
> the
> list with you in Cc, so you can have a look when you have time.
>
> Thanks,
> Dmitry
>
> -----Original Message-----
> From: DERUMIER, Alexandre
> [mailto:alexandre.derumier@groupe-cyllene.com]
> Sent: Wednesday, October 14, 2026 4:12 PM
> To: pve-devel@lists.proxmox.com; rdmitry0911@gmail.com
> Subject: Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi Dmitry,
>
> Thanks for testing on nvidia, I didn't have hardware to test it.
>
> Feel free to rework my patches, I was waiting from proxmox devs
> review &&
> comments before doing more work.
>
> I was waiting for the merge of this patch too:
>
> [PATCH qemu-server/pve-manager/docs v4 0/1] add virtio-vga-gl OpenGL
> 4.6 & Vulkan (venus) support
> https://lore.proxmox.com/all/20260831094605.114396-1-m.frank@proxmox.com/
>
> to rebase my serie.
>
> I'll be pretty busy at work until next month, so I'll don't have too
> much
> time to work on it.
>
> but I'm glad that somebody is interested by me patch serie :)
>
> Best Regards,
>
> Alexandre
>
>
>
> Le mercredi 07 octobre 2026 à 17:10 +0300, Dmitry R a écrit :
> >
> > Hi all, hi Alexandre,
> >
> > only after sending this RFC I noticed that Alexandre posted "add
> > rdp
> > && kyber consoles for qemu over D-Bus display" in August [1]. Both
> > proposals need the same foundation in qemu-server -- QEMU's D-Bus
> > display -- and differ in what consumes it. Rather than two
> > competing
> > ways to enable it, I would suggest one shared base with the
> > consoles
> > on top. What the two series have in common:
> > * qemu-server enables '-display dbus' for a VM, and a separate
> > package attaches to it and streams the guest to the browser (RDP
> > and
> > Kyber daemons there, a WebRTC media service here);
> > * with GL, QEMU refuses VNC and SPICE next to the D-Bus display,
> > so
> > both have to deal with a VM that has no VNC console. Where they
> > differ, and why I think Alexandre's base is the better one:
> > * His series runs a private dbus-daemon per VM; mine uses p2p=yes
> > and attaches every console with getfd + add_client.
> > * With p2p=yes, each add_client replaces the previous connection,
> > so
> > only one consumer can be attached at a time. On a bus, RDP, Kyber
> > and
> > WebRTC can attach to the same VM side by side.
> > * With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
> > AudioOutListener gets the Init/SetEnabled calls made while
> > RegisterOutListener runs, then nothing (the same with a plain GDBus
> > client, so it does not look like a problem of my listener). On a
> > private bus, which my add-on uses today, both playback and the
> > microphone (AudioInListener) work.
> > A possible split:
> > 1. qemu-server: the per-VM D-Bus display (Alexandre's
> > DBusDisplay.pm), the option that enables it, 'rendernode' for
> > virtio-
> > gl (bug #4771) and a 'dbus' driver for audio0.
> > 2. One package per console, each attaching as a client on that
> > bus: RDP and Kyber (Alexandre), WebRTC (mine, bug #8106), with
> > their
> > API calls and UI entries in qemu-server and pve-manager.
> >
> > I am happy to rebase the WebRTC part onto Alexandre's
> > DBusDisplay.pm
> > and drop my p2p attach and the separate 'dbus' VGA flag. Alexandre
> > wrote that the DMA-BUF path is untested on NVIDIA, so I tried it on
> > an
> > RTX 3080 (driver 595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a
> > virtio-vga-gl guest on a private per-VM bus; kyber-qemu-server and
> > txproto built from the v2 patch against Debian trixie's FFmpeg 7.1,
> > encoding to a file):
> >
> > - --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24
> > with
> > the NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI
> > frames context fails on the first frame -- "Failed to create
> > surface:
> > 14 (the requested RT Format is not supported)" -- and nothing is
> > encoded. With --encoder h264_nvenc it ends the same way, since the
> > path goes through VAAPI whatever the encoder.
> > - The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-
> > driver, which only decodes (VAEntrypointVLD): no encoding, no video
> > processing, no RGB surfaces. The mmap fallback would not help
> > either,
> > as a block-linear buffer does not read as linear memory.
> > - Without --dmabuf, the listener received no scanout at all from
> > the
> > GL display within 15 s (libx264 and h264_nvenc alike).
> > - Control on the same VM: my worker imports the same DMA-BUF with
> > EGL
> > (eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier), reads
> > it
> > back with glReadPixels and encodes it with NVENC -- that works. So
> > with a virtio-gl guest on NVIDIA, the Kyber console shows no
> > picture
> > today.
> >
> > A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
> > kyber-qemu-server (358 lines, mostly one new file, LGPL like
> > dmabuf.c). It still requests DMA-BUF scanouts, imports each buffer
> > with EGL (GBM platform, surfaceless context, the modifier passed),
> > reads it back with glReadPixels and feeds your existing software
> > path,
> > so any --encoder works -- h264_nvenc on NVIDIA. A buffer is
> > imported
> > once and read only after QEMU reports damage. Same setup, a
> > terminal
> > running 'top -d 0.1' for 20 s: 1226 frames encoded with h264_nvenc,
> > 0
> > dropped, the picture correct; about 24% of one CPU core (36% with
> > libx264). The --dmabuf path is unchanged; kycontroller could choose
> > '--dmabuf-readback --encoder h264_nvenc' when the render node
> > belongs
> > to the nvidia driver.
> >
> > To be fair about the trade-off: the readback copies every changed
> > frame into system memory, so on AMD and Intel your zero-copy VAAPI
> > path is likely the more efficient one. The best of both would be
> > zero-copy where VAAPI can import and encode, and the EGL readback
> > everywhere else -- the same split would serve the WebRTC console.
> >
> > Maintainers: would one shared base series like this work for you,
> > and
> > is there a preferred name and place for the option that enables the
> > D-Bus display?
> > [1] https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-
> > alexandre.derumier@groupe-cyllene.com/
> > [2] https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-
> > direct/upstream/kyber-nvidia/0001-kyber-qemu-server-read-DMA-BUF-
> > scanouts-back-through.patch
> >
> > Thanks,
> > Dmitry
> >
> >
> > From: Dmitry R [mailto:rdmitry0911@gmail.com]
> > Sent: Thursday, October 01, 2026 6:11 PM
> > To: pve-devel@lists.proxmox.com
> > Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
> >
> > Hi all,
> >
> > I would like to ask whether a hardware-encoded WebRTC console for
> > VMs
> > would be welcome upstream, and if so, in which shape. It runs
> > today
> > as an add-on for PVE 9.2, and I have prepared the qemu-server and
> > pve-manager parts as patch series against current master (tested on
> > PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion
> > on
> > the design before sending them.
> >
> > Feature request: bug #8106
> > https://bugzilla.proxmox.com/show_bug.cgi?id=8106
> >
> > Motivation
> > ----------
> > noVNC is the right console for firmware, installers and text
> > consoles.
> > For time spent inside a running desktop it transfers changed
> > rectangles instead of video, has no audio, and a VirGL (virtio-gl)
> > guest reaches it through egl-headless readback. Measured with one
> > guest image and the same workloads (method and raw numbers in the
> > repository below):
> >
> > WebRTC, virtio-gl noVNC, std VGA
> > 720p video, pictures/s 30 (source rate) 7.4
> > scrolling, pictures/s 36 (52 on NVENC) 12.0
> > animated UI, pictures/s 38 (45 on NVENC) 11.0
> > bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
> > single pointer event 88 ms 52 ms
> > console open -> 1st picture 3.4 s 1.0 s
> >
> > The last two rows are where noVNC stays better: a single isolated
> > event is one small rectangle for VNC, but a full encode/decode
> > round
> > trip for video (the 88 ms were measured with software libx264 on a
> > CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
> > streaming the animated scene.
> >
> > What QEMU already provides
> > --------------------------
> > QEMU's D-Bus display (-display dbus) exports the guest scanout --
> > the
> > GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard,
> > mouse,
> > multi-touch, clipboard and audio. With p2p=yes there is no bus
> > daemon:
> > a client is attached per connection through QMP (getfd +
> > add_client),
> > which PVE::QMPClient can already do (it passes the fd for getfd).
> >
> > Proposed series
> > ---------------
> > 1. qemu-server: 'rendernode' for the VGA property
> > vga: virtio-gl,rendernode=/dev/dri/renderD129
> > so a virtio-gl guest can be placed on a chosen GPU instead of
> > the
> > first render node (fixes #4771). Small and useful on its own.
> >
> > 2. qemu-server: opt-in 'dbus' for the VGA property
> > vga: virtio-gl,dbus=1
> > adds '-display dbus,p2p=yes' (gl=on with the render node for
> > virtio-gl, gl=off for the other types). One constraint: with
> > GL,
> > QEMU refuses VNC and SPICE next to it ("Display vnc is
> > incompatible
> > with the GL context"), so a virtio-gl VM with dbus=1 has neither
> > and
> > offers the WebRTC console instead; non-GL types keep VNC as
> > well.
> > The clipboard reaches the guest through qemu-vdagent (as for the
> > VNC clipboard) and the guest's spice-vdagent.
> > Audio is not part of the series yet: the add-on plays it through
> > a
> > bus-mode D-Bus display (a 'dbus' audiodev named in -display),
> > but
> > with p2p=yes our listener got no audio from QEMU 11.0 -- only
> > the
> > Init/SetEnabled calls made while RegisterOutListener runs,
> > nothing
> > afterwards. A 'dbus' audio0 driver is ready once that is
> > resolved.
> >
> > 3. qemu-server: API call to attach a console
> > POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
> > takes the browser's SDP offer, creates a socketpair, attaches
> > one
> > end to QEMU (getfd + add_client) and hands the other end with
> > the
> > offer to a local media service over a root-owned unix socket
> > (SCM_RIGHTS); returns the SDP answer. Authentication and ACL
> > stay
> > exactly those of vncproxy/termproxy -- no extra port, no own
> > auth.
> > The status API reports 'dbus' and 'vnc' so the UI knows which
> > consoles a VM offers.
> >
> > 4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
> > in the console button and the VM's Console panel, offered when
> > the
> > VM's display has dbus=1; the display editor gets the two
> > options.
> > Like noVNC and xterm.js, the viewer itself is a page from a
> > separate
> > package that pveproxy serves ('?console=kvm&webrtc=1').
> >
> > 5. A separate package with the media service: capture from the D-
> > Bus
> > display (the virtio-gl DMA-BUF scanout is imported through EGL
> > and
> > read back; a zero-copy path to the encoder is future work),
> > encode
> > (NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
> > browser decodes it),
> > Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a
> > Python/aiortc
> > WebRTC bridge; I realise Python/aiortc is unusual in your stack.
> >
> > Questions
> > ---------
> > - Is this something you would consider upstream at all?
> > - Property naming: 'rendernode' and 'dbus' inside 'vga', or a
> > separate
> > display property?
> > - Would you rather keep the media service out of tree (users
> > install
> > it, the API call returns an error when it is absent), or should
> > it
> > eventually live in a Proxmox-maintained package, and would you
> > then
> > want it in Rust?
> > - Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
> >
> > Not part of this proposal: the prototype also has a graphical
> > console
> > for LXC containers (a headless compositor in the container with its
> > own login screen, one console per tty like the terminal console).
> > It
> > needs software inside the guest, so I plan to keep it as a separate
> > add-on package.
> >
> > Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
> > 9.2.11 plus the series): a Kubuntu 26.04 guest with
> > 'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
> > through webrtcproxy, in a popup and in the VM's Console panel, for
> > a
> > user holding only PVEVMUser on the VM (403 without VM.Console).
> >
> > Code and measurements: https://github.com/rdmitry0911/qsm-rd
> > (qsm-direct/, docs/COMPARISON.md for the numbers; the prepared
> > series
> > are in qsm-direct/upstream/patches/).
> >
> > Transparency: I developed the prototype with the help of an AI
> > coding
> > assistant (Claude). I have reviewed and tested the code and will
> > sign
> > the Harmony CLA; please tell me if that is a problem for you.
> >
> > Thanks,
> > Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
2026-10-15 9:52 ` DERUMIER, Alexandre
@ 2026-10-16 6:58 ` Dmitry R
0 siblings, 0 replies; 7+ messages in thread
From: Dmitry R @ 2026-10-16 6:58 UTC (permalink / raw)
To: 'DERUMIER, Alexandre', pve-devel
Hi Alexandre,
Thanks for looking at it.
Yes, the native version now goes through pveproxy on 8006. The browser
uses the PVE API for signalling, status and media settings; the API handlers
check the PVE permissions and forward requests to the media daemon over a
local Unix socket. This works for both VMs and LXC. WebRTC media still uses
UDP between the browser and the node.
Thanks for pointing out the PBS conflict. Port 8007 belongs to the older
standalone add-on. The native path does not need it, and the compatibility
service can be disabled once the old consumers are migrated. I checked VM
and LXC consoles with that service stopped, so a native installation can
leave 8007 available for PBS.
For latency, I used a small test page inside the guest and a Playwright
script observing the console in the browser. It records performance.now()
when the console receives the mousemove event, then detects the test page's
colored hover popup in the returned image. Each detection poll redraws the
current video or noVNC canvas before reading its pixels. The poll runs at
the browser's animation cadence, and both timestamps use the same browser
clock. The popup is cleared before each repetition.
On the CPU-only nested lab, the median of five hover runs on the same
Standard VGA VM was 88 ms through QSM with libx264 and 52 ms through noVNC.
So noVNC was faster for that isolated pointer event. The smoother
video/scrolling results are separate measurements of delivered pictures and
bandwidth, rather than evidence of lower interaction latency.
This measures browser input-event to detected guest response, including
guest repaint and browser scheduling. It does not measure physical
mouse-to-monitor latency. These figures are from the September setup; I have
not repeated this latency comparison on the shared-platform revision or with
NVENC.
Method, environment and results
(https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-direct/docs/COMPARISON.
md#interaction-latency-2026-09-08)
Measurement script
(https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-direct/lab/proxmox9/mea
sure-interaction.cjs)
Thanks,
Dmitry
-----Original Message-----
From: DERUMIER, Alexandre [mailto:alexandre.derumier@groupe-cyllene.com]
Sent: Thursday, October 15, 2026 12:52 PM
To: pve-devel@lists.proxmox.com; rdmitry0911@gmail.com
Subject: Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
Hi Dmitry,
I wonder if it could be possible to put your qsm-pve-direct daemon behind
pveproxy , like I do for my rdp daemon ? (it's just forwarding)
This could avoid to expose another port.
(I really don't known if it's possible).
Also, be carefull, 8007 is also used by PBS. (and some user sometimes
install PBS on same host than PVE)
Btw, how do you have benched and retrieve metrics for latency ?
Le jeudi 15 octobre 2026 à 10:53 +0300, Dmitry R a écrit :
> Hi Alexandre, hi all,
>
> Following up on my reply yesterday: I've completed the QSM port to the
> shared per-VM D-Bus foundation and uploaded the source, patches and a
> matching preview build:
>
> https://github.com/rdmitry0911/qsm-rd/releases/tag/v0.4.0-qsm-direct-2
> 026101
> 5-shared9
>
> The original D-Bus patch retains Alexandre's authorship and sign-off.
> qemu-server owns the bus and its lifetime; QSM attaches as a consumer.
> The proposed opt-in is dbus-display: 1, with GPU type/rendernode in
> vga and driver=dbus for audio0. CPU displays can keep VNC; GL displays
> cannot provide VNC/SPICE alongside the D-Bus display.
>
> The common display foundation, WebRTC API and pve-manager UI are
> separate series. A small pve-guest-common module now handles the local
> console service exchange for both VM and LXC APIs. Native signalling,
> status and media settings use PVE's existing HTTPS origin on8006, with
> VM.Console for offers/reads and VM.Config.Options for settings writes.
> There is no separate HTTPS port; WebRTC media still uses UDP.
>
> I checked Firefox video and an ordinary Reconnect in the embedded VM
> console, and LXC video/settings, with the old8007 service stopped.
> Anonymous
> reads and writes without a CSRF token are rejected. The Firefox mDNS
> negotiation and reconnect issues found during VPN testing are fixed.
> The
> earlier shared-base lab checks also covered CPU/GL display, audio,
> microphone, clipboard and a second independent display consumer.
>
> The graphical LXC sessions and provisioning remain in a separate add-
> on.
> Their API patches can be reviewed independently of the QEMU
> foundation.
> There are builds against pve-container 6.1.13 and 6.1.14, so the newer
> container package does not need to be downgraded.
>
> Markus Frank's Venus configuration adaptation and the NVIDIA readback
> patch remain separate optional work. Runtime Venus and full Kyber/RDP
> browser integration have not been qualified by these tests. The source
> bases, package hashes, test logs and migration instructions are here:
>
> https://github.com/rdmitry0911/qsm-rd/tree/main/qsm-direct/upstream
>
> I'd appreciate feedback on the VM-wide option, bus lifetime, audio
> configuration and the API/service split. The patches are still a
> proposal; I have not treated the new configuration as an agreed
> upstream interface.
>
> Thanks,
> Dmitry
>
> -----Original Message-----
> From: Dmitry R [mailto:rdmitry0911@gmail.com]
> Sent: Wednesday, October 14, 2026 5:20 PM
> To: 'DERUMIER, Alexandre'; 'pve-devel@lists.proxmox.com'
> Subject: RE: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi Alexandre,
>
> Thanks for your reply, and for making your patches available. I'm glad
> we can build on the same foundation.
>
> I'll work from your v2 series, first separating the shared qemu-
> server D-Bus display support from the console-specific parts. Then
> I'll rebase the WebRTC console onto that base, using the per-VM bus
> from your DBusDisplay.pm. I'll keep the media service in a separate
> package and preserve your authorship on the original patches.
>
> I'll also include the NVIDIA EGL readback fix I linked in my previous
> mail, keeping the existing VAAPI path available. My testing of that
> fix was limited to the capture/encoding adapter with output to a file;
> I still need to check it through the full Kyber console in the
> browser.
>
> I'll take the OpenGL 4.6 / Venus series you mentioned into account
> when rebasing. The shared base will still need feedback from the
> Proxmox developers on the configuration and integration.
>
> No problem about your availability. I'll send the revised patches to
> the list with you in Cc, so you can have a look when you have time.
>
> Thanks,
> Dmitry
>
> -----Original Message-----
> From: DERUMIER, Alexandre
> [mailto:alexandre.derumier@groupe-cyllene.com]
> Sent: Wednesday, October 14, 2026 4:12 PM
> To: pve-devel@lists.proxmox.com; rdmitry0911@gmail.com
> Subject: Re: [RFC] WebRTC console for VMs via QEMU's D-Bus display
>
> Hi Dmitry,
>
> Thanks for testing on nvidia, I didn't have hardware to test it.
>
> Feel free to rework my patches, I was waiting from proxmox devs review
> && comments before doing more work.
>
> I was waiting for the merge of this patch too:
>
> [PATCH qemu-server/pve-manager/docs v4 0/1] add virtio-vga-gl OpenGL
> 4.6 & Vulkan (venus) support
> https://lore.proxmox.com/all/20260831094605.114396-1-m.frank@proxmox.c
> om/
>
> to rebase my serie.
>
> I'll be pretty busy at work until next month, so I'll don't have too
> much time to work on it.
>
> but I'm glad that somebody is interested by me patch serie :)
>
> Best Regards,
>
> Alexandre
>
>
>
> Le mercredi 07 octobre 2026 à 17:10 +0300, Dmitry R a écrit :
> >
> > Hi all, hi Alexandre,
> >
> > only after sending this RFC I noticed that Alexandre posted "add rdp
> > && kyber consoles for qemu over D-Bus display" in August [1]. Both
> > proposals need the same foundation in qemu-server -- QEMU's D-Bus
> > display -- and differ in what consumes it. Rather than two competing
> > ways to enable it, I would suggest one shared base with the consoles
> > on top. What the two series have in common:
> > * qemu-server enables '-display dbus' for a VM, and a separate
> > package attaches to it and streams the guest to the browser (RDP and
> > Kyber daemons there, a WebRTC media service here);
> > * with GL, QEMU refuses VNC and SPICE next to the D-Bus display,
> > so both have to deal with a VM that has no VNC console. Where they
> > differ, and why I think Alexandre's base is the better one:
> > * His series runs a private dbus-daemon per VM; mine uses p2p=yes
> > and attaches every console with getfd + add_client.
> > * With p2p=yes, each add_client replaces the previous connection,
> > so only one consumer can be attached at a time. On a bus, RDP, Kyber
> > and WebRTC can attach to the same VM side by side.
> > * With p2p=yes, QEMU 11.0 did not deliver audio in my tests: the
> > AudioOutListener gets the Init/SetEnabled calls made while
> > RegisterOutListener runs, then nothing (the same with a plain GDBus
> > client, so it does not look like a problem of my listener). On a
> > private bus, which my add-on uses today, both playback and the
> > microphone (AudioInListener) work.
> > A possible split:
> > 1. qemu-server: the per-VM D-Bus display (Alexandre's
> > DBusDisplay.pm), the option that enables it, 'rendernode' for
> > virtio-
> > gl (bug #4771) and a 'dbus' driver for audio0.
> > 2. One package per console, each attaching as a client on that
> > bus: RDP and Kyber (Alexandre), WebRTC (mine, bug #8106), with their
> > API calls and UI entries in qemu-server and pve-manager.
> >
> > I am happy to rebase the WebRTC part onto Alexandre's DBusDisplay.pm
> > and drop my p2p attach and the separate 'dbus' VGA flag. Alexandre
> > wrote that the DMA-BUF path is untested on NVIDIA, so I tried it on
> > an RTX 3080 (driver 595.91, nvidia-vaapi-driver 0.0.16, QEMU 11.0, a
> > virtio-vga-gl guest on a private per-VM bus; kyber-qemu-server and
> > txproto built from the v2 patch against Debian trixie's FFmpeg 7.1,
> > encoding to a file):
> >
> > - --dmabuf (h264_vaapi): QEMU hands over the scanout (XB24/AB24 with
> > the NVIDIA block-linear modifier 0x300000000e08014), but the VAAPI
> > frames context fails on the first frame -- "Failed to create
> > surface:
> > 14 (the requested RT Format is not supported)" -- and nothing is
> > encoded. With --encoder h264_nvenc it ends the same way, since the
> > path goes through VAAPI whatever the encoder.
> > - The cause is the driver: on NVIDIA, VAAPI means nvidia-vaapi-
> > driver, which only decodes (VAEntrypointVLD): no encoding, no video
> > processing, no RGB surfaces. The mmap fallback would not help
> > either, as a block-linear buffer does not read as linear memory.
> > - Without --dmabuf, the listener received no scanout at all from the
> > GL display within 15 s (libx264 and h264_nvenc alike).
> > - Control on the same VM: my worker imports the same DMA-BUF with
> > EGL (eglCreateImageKHR, EGL_LINUX_DMA_BUF_EXT with the modifier),
> > reads it back with glReadPixels and encodes it with NVENC -- that
> > works. So with a virtio-gl guest on NVIDIA, the Kyber console shows
> > no picture today.
> >
> > A fix, tested [2]: a patch on top of v2 adds --dmabuf-readback to
> > kyber-qemu-server (358 lines, mostly one new file, LGPL like
> > dmabuf.c). It still requests DMA-BUF scanouts, imports each buffer
> > with EGL (GBM platform, surfaceless context, the modifier passed),
> > reads it back with glReadPixels and feeds your existing software
> > path, so any --encoder works -- h264_nvenc on NVIDIA. A buffer is
> > imported once and read only after QEMU reports damage. Same setup, a
> > terminal running 'top -d 0.1' for 20 s: 1226 frames encoded with
> > h264_nvenc, 0 dropped, the picture correct; about 24% of one CPU
> > core (36% with libx264). The --dmabuf path is unchanged;
> > kycontroller could choose '--dmabuf-readback --encoder h264_nvenc'
> > when the render node belongs to the nvidia driver.
> >
> > To be fair about the trade-off: the readback copies every changed
> > frame into system memory, so on AMD and Intel your zero-copy VAAPI
> > path is likely the more efficient one. The best of both would be
> > zero-copy where VAAPI can import and encode, and the EGL readback
> > everywhere else -- the same split would serve the WebRTC console.
> >
> > Maintainers: would one shared base series like this work for you,
> > and is there a preferred name and place for the option that enables
> > the D-Bus display?
> > [1] https://lore.proxmox.com/pve-devel/20260826074347.1256659-1-
> > alexandre.derumier@groupe-cyllene.com/
> > [2] https://github.com/rdmitry0911/qsm-rd/blob/main/qsm-
> > direct/upstream/kyber-nvidia/0001-kyber-qemu-server-read-DMA-BUF-
> > scanouts-back-through.patch
> >
> > Thanks,
> > Dmitry
> >
> >
> > From: Dmitry R [mailto:rdmitry0911@gmail.com]
> > Sent: Thursday, October 01, 2026 6:11 PM
> > To: pve-devel@lists.proxmox.com
> > Subject: [RFC] WebRTC console for VMs via QEMU's D-Bus display
> >
> > Hi all,
> >
> > I would like to ask whether a hardware-encoded WebRTC console for
> > VMs would be welcome upstream, and if so, in which shape. It runs
> > today as an add-on for PVE 9.2, and I have prepared the qemu-server
> > and pve-manager parts as patch series against current master (tested
> > on PVE 9.2 with an RTX 3080, see below) -- but I'd like your opinion
> > on the design before sending them.
> >
> > Feature request: bug #8106
> > https://bugzilla.proxmox.com/show_bug.cgi?id=8106
> >
> > Motivation
> > ----------
> > noVNC is the right console for firmware, installers and text
> > consoles.
> > For time spent inside a running desktop it transfers changed
> > rectangles instead of video, has no audio, and a VirGL (virtio-gl)
> > guest reaches it through egl-headless readback. Measured with one
> > guest image and the same workloads (method and raw numbers in the
> > repository below):
> >
> > WebRTC, virtio-gl noVNC, std VGA
> > 720p video, pictures/s 30 (source rate) 7.4
> > scrolling, pictures/s 36 (52 on NVENC) 12.0
> > animated UI, pictures/s 38 (45 on NVENC) 11.0
> > bandwidth, 720p video 18.8 Mbit/s 120 Mbit/s
> > single pointer event 88 ms 52 ms
> > console open -> 1st picture 3.4 s 1.0 s
> >
> > The last two rows are where noVNC stays better: a single isolated
> > event is one small rectangle for VNC, but a full encode/decode round
> > trip for video (the 88 ms were measured with software libx264 on a
> > CPU-only node). With NVENC (RTX 3080) the encoder used 4% while
> > streaming the animated scene.
> >
> > What QEMU already provides
> > --------------------------
> > QEMU's D-Bus display (-display dbus) exports the guest scanout --
> > the GL/DMA-BUF scanout of virtio-gl included -- and takes keyboard,
> > mouse, multi-touch, clipboard and audio. With p2p=yes there is no
> > bus
> > daemon:
> > a client is attached per connection through QMP (getfd +
> > add_client), which PVE::QMPClient can already do (it passes the fd
> > for getfd).
> >
> > Proposed series
> > ---------------
> > 1. qemu-server: 'rendernode' for the VGA property
> > vga: virtio-gl,rendernode=/dev/dri/renderD129
> > so a virtio-gl guest can be placed on a chosen GPU instead of the
> > first render node (fixes #4771). Small and useful on its own.
> >
> > 2. qemu-server: opt-in 'dbus' for the VGA property
> > vga: virtio-gl,dbus=1
> > adds '-display dbus,p2p=yes' (gl=on with the render node for
> > virtio-gl, gl=off for the other types). One constraint: with GL,
> > QEMU refuses VNC and SPICE next to it ("Display vnc is
> > incompatible
> > with the GL context"), so a virtio-gl VM with dbus=1 has neither
> > and
> > offers the WebRTC console instead; non-GL types keep VNC as well.
> > The clipboard reaches the guest through qemu-vdagent (as for the
> > VNC clipboard) and the guest's spice-vdagent.
> > Audio is not part of the series yet: the add-on plays it through
> > a
> > bus-mode D-Bus display (a 'dbus' audiodev named in -display), but
> > with p2p=yes our listener got no audio from QEMU 11.0 -- only the
> > Init/SetEnabled calls made while RegisterOutListener runs,
> > nothing
> > afterwards. A 'dbus' audio0 driver is ready once that is
> > resolved.
> >
> > 3. qemu-server: API call to attach a console
> > POST /nodes/{node}/qemu/{vmid}/webrtcproxy (VM.Console)
> > takes the browser's SDP offer, creates a socketpair, attaches one
> > end to QEMU (getfd + add_client) and hands the other end with the
> > offer to a local media service over a root-owned unix socket
> > (SCM_RIGHTS); returns the SDP answer. Authentication and ACL
> > stay
> > exactly those of vncproxy/termproxy -- no extra port, no own
> > auth.
> > The status API reports 'dbus' and 'vnc' so the UI knows which
> > consoles a VM offers.
> >
> > 4. pve-manager: WebRTC console type next to noVNC/SPICE/xterm.js
> > in the console button and the VM's Console panel, offered when
> > the
> > VM's display has dbus=1; the display editor gets the two options.
> > Like noVNC and xterm.js, the viewer itself is a page from a
> > separate
> > package that pveproxy serves ('?console=kvm&webrtc=1').
> >
> > 5. A separate package with the media service: capture from the D-
> > Bus
> > display (the virtio-gl DMA-BUF scanout is imported through EGL
> > and
> > read back; a zero-copy path to the encoder is future work),
> > encode
> > (NVENC, VA-API, QSV; libx264 fallback; H.264, HEVC where the
> > browser decodes it),
> > Opus audio, WebRTC. Today it is C++ (FFmpeg) plus a
> > Python/aiortc
> > WebRTC bridge; I realise Python/aiortc is unusual in your stack.
> >
> > Questions
> > ---------
> > - Is this something you would consider upstream at all?
> > - Property naming: 'rendernode' and 'dbus' inside 'vga', or a
> > separate
> > display property?
> > - Would you rather keep the media service out of tree (users install
> > it, the API call returns an error when it is absent), or should it
> > eventually live in a Proxmox-maintained package, and would you
> > then
> > want it in Rust?
> > - Is 'webrtcproxy' next to 'vncproxy'/'termproxy' the right place?
> >
> > Not part of this proposal: the prototype also has a graphical
> > console for LXC containers (a headless compositor in the container
> > with its own login screen, one console per tty like the terminal
> > console).
> > It
> > needs software inside the guest, so I plan to keep it as a separate
> > add-on package.
> >
> > Tested end to end on PVE 9.2 (qemu-server 9.2.7 and pve-manager
> > 9.2.11 plus the series): a Kubuntu 26.04 guest with
> > 'vga: virtio-gl,dbus=1' streams 1280x800 at 60 fps (NVENC H.264)
> > through webrtcproxy, in a popup and in the VM's Console panel, for a
> > user holding only PVEVMUser on the VM (403 without VM.Console).
> >
> > Code and measurements: https://github.com/rdmitry0911/qsm-rd
> > (qsm-direct/, docs/COMPARISON.md for the numbers; the prepared
> > series are in qsm-direct/upstream/patches/).
> >
> > Transparency: I developed the prototype with the help of an AI
> > coding assistant (Claude). I have reviewed and tested the code and
> > will sign the Harmony CLA; please tell me if that is a problem for
> > you.
> >
> > Thanks,
> > Dmitry
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-10-16 6:56 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-01 15:10 [RFC] WebRTC console for VMs via QEMU's D-Bus display Dmitry R
2026-10-07 14:10 ` SPAM: " Dmitry R
2026-10-14 13:12 ` DERUMIER, Alexandre
2026-10-14 14:19 ` SPAM: " Dmitry R
-- strict thread matches above, loose matches on Subject: below --
2026-10-15 7:53 Dmitry R
2026-10-15 9:52 ` DERUMIER, Alexandre
2026-10-16 6:58 ` Dmitry R
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox