[PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Elias Huhsovitz]

The current Problem
-------------------
Console session processes can outlive their container when it stops or
is destroyed. The cleanup currently runs in pvestatd every 10 seconds.
It calls vmstatus() and kills stale processes by PID.

The killing of the process can result in a race condition:
If the console gets closed by something else, just before the kill
command is executed, a different process can reuse the PID and we kill
a completely unrelated process.

Also checking this every 10s is too much unnecessary overhead IMO.

My understanding of a "better" architecture
-------------------------------------------
IMO console deletion should be tied to the state of the underlying
container. Also I was unable to find a "convenient" time/event for a
cleanup job to run.

Proposed implementation
-----------------------
Wrap console commands in transient systemd scopes and bind them to the
container service via the PartOf= property. When the container stops,
systemd stops the scope, terminating the console session and the dtach
session. This allows removing the pvestatd periodic cleanup entirely.

Summary of Changes
------------------

pve-manager:

Elias Huhsovitz (1):
  fix #8093: pvestatd: remove cleanup of stale lxc consoles

 PVE/Service/pvestatd.pm | 18 ------------------
 1 file changed, 18 deletions(-)


pve-container:

Elias Huhsovitz (1):
  fix #8093: console: bind console sessions to container lifetime via
    systemd scopes

 src/PVE/API2/LXC.pm |  6 +++---
 src/PVE/CLI/pct.pm  | 10 +++++++---
 src/PVE/LXC.pm      | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+), 6 deletions(-)


Summary over all repositories:
  4 files changed, 51 insertions(+), 24 deletions(-)

-- 
Generated by murpp 0.12.0

[PATCH manager 1/2] fix #8093: pvestatd: remove cleanup of stale lxc consoles

[Elias Huhsovitz]

Responsibility is moved to pve-container.

Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
 PVE/Service/pvestatd.pm | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 34f9da71..cec66e00 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -444,20 +444,6 @@ sub update_qemu_status {
     PVE::PullMetric::update($pull_txn, 'qemu', $vmstatus, $ctime);
 }
 
-sub remove_stale_lxc_consoles {
-
-    my $vmstatus = PVE::LXC::vmstatus();
-    my $pidhash = PVE::LXC::find_lxc_console_pids();
-
-    foreach my $vmid (keys %$pidhash) {
-        next if defined($vmstatus->{$vmid});
-        syslog('info', "remove stale lxc-console for CT $vmid");
-        foreach my $pid (@{ $pidhash->{$vmid} }) {
-            kill(9, $pid);
-        }
-    }
-}
-
 my $rebalance_error_count = {};
 
 my $NO_REBALANCE;
@@ -848,10 +834,6 @@ sub update_status {
     $err = $@;
     syslog('err', "storage status update error: $err") if $err;
 
-    eval { remove_stale_lxc_consoles(); };
-    $err = $@;
-    syslog('err', "lxc console cleanup error: $err") if $err;
-
     eval { rotate_authkeys(); };
     $err = $@;
     syslog('err', "authkey rotation error: $err") if $err;
-- 
2.47.3

[PATCH container 2/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Elias Huhsovitz]

Console processes can outlive their useful session when a container
stops or disappears while a console is attached. The previous cleanup
ran in pvestatd, which polled every 10s: forked lxc-info for every
running container, and killed orphaned processes by their PID.

Manage console lifetime in the container lifecycle instead. Add
wrap_in_console_scope() to start console commands in a systemd scope.
Bind each scope with PartOf= to the respective pve-container service.
Use the wrapper for pct console/enter/exec, and the API console
endpoints.

When the container service stops, systemd stops the related console
scopes.

Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
 src/PVE/API2/LXC.pm |  6 +++---
 src/PVE/CLI/pct.pm  | 10 +++++++---
 src/PVE/LXC.pm      | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
index 5f94d5a..3fd2bf8 100644
--- a/src/PVE/API2/LXC.pm
+++ b/src/PVE/API2/LXC.pm
@@ -1015,7 +1015,7 @@ __PACKAGE__->register_method({
         my $ticket = PVE::AccessControl::assemble_vnc_ticket($authuser, $authpath, $port);
 
         my $conf = PVE::LXC::Config->load_config($vmid, $node);
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf, -1);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf, -1);
 
         my $shcmd = [
             '/usr/bin/dtach',
@@ -1146,7 +1146,7 @@ __PACKAGE__->register_method({
         my $ticket = PVE::AccessControl::assemble_vnc_ticket($authuser, $authpath, $port);
 
         my $conf = PVE::LXC::Config->load_config($vmid, $node);
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf, -1);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf, -1);
 
         my $shcmd = [
             '/usr/bin/dtach',
@@ -1288,7 +1288,7 @@ __PACKAGE__->register_method({
 
         die "CT $vmid not running\n" if !PVE::LXC::check_running($vmid);
 
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf);
 
         my $shcmd = [
             '/usr/bin/dtach',
diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm
index ceb0bea..11e85bc 100755
--- a/src/PVE/CLI/pct.pm
+++ b/src/PVE/CLI/pct.pm
@@ -153,7 +153,7 @@ __PACKAGE__->register_method({
         # test if container exists on this node
         my $conf = PVE::LXC::Config->load_config($param->{vmid});
 
-        my $cmd = PVE::LXC::get_console_command($param->{vmid}, $conf, $param->{escape});
+        my $cmd = PVE::LXC::get_console_command_scoped($param->{vmid}, $conf, $param->{escape});
         exec(@$cmd);
     },
 });
@@ -207,7 +207,9 @@ __PACKAGE__->register_method({
 
         my @lxc_attach_cmd = ('lxc-attach', '-n', $vmid);
         push @lxc_attach_cmd, $keep_env ? '--keep-env' : '--clear-env';
-        exec(@lxc_attach_cmd);
+
+        my $scoped_cmd = PVE::LXC::wrap_in_console_scope($vmid, \@lxc_attach_cmd);
+        exec(@$scoped_cmd);
     },
 });
 
@@ -250,7 +252,9 @@ __PACKAGE__->register_method({
         my @lxc_attach_cmd = ('lxc-attach', '-n', $vmid);
         push @lxc_attach_cmd, $keep_env ? '--keep-env' : '--clear-env';
         push @lxc_attach_cmd, '--', @{ $param->{'extra-args'} };
-        exec(@lxc_attach_cmd);
+
+        my $scoped_cmd = PVE::LXC::wrap_in_console_scope($vmid, \@lxc_attach_cmd);
+        exec(@$scoped_cmd);
     },
 });
 
diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index dee073d..14bc6e4 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -954,6 +954,39 @@ sub verify_searchdomain_list {
     return join(' ', @list);
 }
 
+=head2 wrap_in_console_scope
+
+Wraps a command array in a transient systemd scope and binds it to the
+container's main systemd service. This ensures the console process is
+automatically terminated by systemd when the container stops, preventing
+orphaned processes without requiring manual cleanup scripts.
+
+=cut
+
+sub wrap_in_console_scope {
+    my ($vmid, $cmd) = @_;
+
+    my $session_id = "$$-" . time();
+    my $unit_name = "pve-lxc-console-$vmid-$session_id";
+
+    my $service = "pve-container\@$vmid.service";
+    # service name appends "-debug" when container is started in debug mode
+    my $debug_service = "pve-container-debug\@$vmid.service";
+
+    my @scope_cmd = (
+        'systemd-run',
+        '--scope',
+        '--unit', $unit_name,
+        '--description', "PVE LXC Console for $vmid",
+        '--property', "PartOf=$service $debug_service",
+        '--quiet',
+    );
+
+    push @scope_cmd, @$cmd;
+
+    return \@scope_cmd;
+}
+
 sub get_console_command {
     my ($vmid, $conf, $escapechar) = @_;
 
@@ -978,6 +1011,14 @@ sub get_console_command {
     return $cmd;
 }
 
+sub get_console_command_scoped {
+    my ($vmid, $conf, $escapechar) = @_;
+
+    my $cmd = get_console_command($vmid, $conf, $escapechar);
+
+    return wrap_in_console_scope($vmid, $cmd)
+}
+
 sub get_primary_ips {
     my ($conf) = @_;
 
-- 
2.47.3

Re: [PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Filip Schauer]

On 07/10/2026 10:37, Elias Huhsovitz wrote:
> The current Problem
> -------------------
> Console session processes can outlive their container when it stops or
> is destroyed. The cleanup currently runs in pvestatd every 10 seconds.
> It calls vmstatus() and kills stale processes by PID.
> 
> The killing of the process can result in a race condition:
> If the console gets closed by something else, just before the kill
> command is executed, a different process can reuse the PID and we kill
> a completely unrelated process.
> 
> Also checking this every 10s is too much unnecessary overhead IMO.
> 
> My understanding of a "better" architecture
> -------------------------------------------
> IMO console deletion should be tied to the state of the underlying
> container. Also I was unable to find a "convenient" time/event for a
> cleanup job to run.
> 
> Proposed implementation
> -----------------------
> Wrap console commands in transient systemd scopes and bind them to the
> container service via the PartOf= property. When the container stops,
> systemd stops the scope, terminating the console session and the dtach
> session. This allows removing the pvestatd periodic cleanup entirely.

Getting away from the polling in pvestatd would be nice.
However, I don't think `PartOf=` can be relied on here.

As a test I created a scope running `sleep 1000` with
`PartOf=pve-container@103.service`. Stopping the container left the
scope and its process running. It seems like `lxc-stop` ends the service
without a systemd stop job, and `PartOf=` only propagates explicit stop
jobs. The same goes for a `poweroff` inside the container. As far as I
can tell, the scope would only be stopped by `systemctl stop` or
`systemctl restart`. So most stop paths are not covered.

But do we even need to kill the lxc-console process in the first place?
lxc-console already exits automatically when the container stops. Or am
I missing something? Is there any case where the process lingers?

Re: [PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Elias Huhsovitz]

On Wed Oct 7, 2026 at 3:38 PM CEST, Filip Schauer wrote:
> On 07/10/2026 10:37, Elias Huhsovitz wrote:
>> The current Problem
>> -------------------
>> Console session processes can outlive their container when it stops or
>> is destroyed. The cleanup currently runs in pvestatd every 10 seconds.
>> It calls vmstatus() and kills stale processes by PID.
>> 
>> The killing of the process can result in a race condition:
>> If the console gets closed by something else, just before the kill
>> command is executed, a different process can reuse the PID and we kill
>> a completely unrelated process.
>> 
>> Also checking this every 10s is too much unnecessary overhead IMO.
>> 
>> My understanding of a "better" architecture
>> -------------------------------------------
>> IMO console deletion should be tied to the state of the underlying
>> container. Also I was unable to find a "convenient" time/event for a
>> cleanup job to run.
>> 
>> Proposed implementation
>> -----------------------
>> Wrap console commands in transient systemd scopes and bind them to the
>> container service via the PartOf= property. When the container stops,
>> systemd stops the scope, terminating the console session and the dtach
>> session. This allows removing the pvestatd periodic cleanup entirely.
>
> Getting away from the polling in pvestatd would be nice.
> However, I don't think `PartOf=` can be relied on here.
>
> As a test I created a scope running `sleep 1000` with
> `PartOf=pve-container@103.service`. Stopping the container left the
> scope and its process running. It seems like `lxc-stop` ends the service
> without a systemd stop job, and `PartOf=` only propagates explicit stop
> jobs. The same goes for a `poweroff` inside the container. As far as I
> can tell, the scope would only be stopped by `systemctl stop` or
> `systemctl restart`. So most stop paths are not covered.

Thats interesting. I was not able to re-produce this behaviour when
calling poweroff from inside the container

Did you create the console using `pct`, or via the API 
(e.g. `/nodes/{node}/lxc/{vmid}/vncproxy)`.

But nevertheless, you are correct. My assumptions about the `PartOf=`
were wrong.

>
> But do we even need to kill the lxc-console process in the first place?
> lxc-console already exits automatically when the container stops. Or am
> I missing something? Is there any case where the process lingers?

There are a few scenarios where lxc-console remains:

1. rm-rf <id>.conf 

while the process is running. Killing the container via lxc-stop <id>
leaves the console running.

2. pid=$(pgrep -f "lxc-console.*<id>")
kill -STOP $pid
lxc-stop <id>

also results in a remaining console.

In my testing yesterday, calling 

pct detory <id> --force
while the container is still running, resulted in an orphaned console,
which prompted me to create this patch in the first place

But I am unable to re-produce this currently. So probably the issue was
something else...

Bottom Line
-----------
The scenarios that leave an orphaned are not standard lifecycle events.

What about:

Moving the `remove_stale_lxc_consoles` logic (with some
improvements) to pve-container.

Introduce a new pct command `pct console cleanup` (name TBD), so that
administrators can cleanup the orphaned consoles in the edge cases
where an lxc-console might get left behind.

I would love to hear your thoughts on this matter!