[PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Elias Huhsovitz]

The current Problem
-------------------
Console session processes can outlive their container when it stops or
is destroyed. The cleanup currently runs in pvestatd every 10 seconds.
It calls vmstatus() and kills stale processes by PID.

The killing of the process can result in a race condition:
If the console gets closed by something else, just before the kill
command is executed, a different process can reuse the PID and we kill
a completely unrelated process.

Also checking this every 10s is too much unnecessary overhead IMO.

My understanding of a "better" architecture
-------------------------------------------
IMO console deletion should be tied to the state of the underlying
container. Also I was unable to find a "convenient" time/event for a
cleanup job to run.

Proposed implementation
-----------------------
Wrap console commands in transient systemd scopes and bind them to the
container service via the PartOf= property. When the container stops,
systemd stops the scope, terminating the console session and the dtach
session. This allows removing the pvestatd periodic cleanup entirely.

Summary of Changes
------------------

pve-manager:

Elias Huhsovitz (1):
  fix #8093: pvestatd: remove cleanup of stale lxc consoles

 PVE/Service/pvestatd.pm | 18 ------------------
 1 file changed, 18 deletions(-)


pve-container:

Elias Huhsovitz (1):
  fix #8093: console: bind console sessions to container lifetime via
    systemd scopes

 src/PVE/API2/LXC.pm |  6 +++---
 src/PVE/CLI/pct.pm  | 10 +++++++---
 src/PVE/LXC.pm      | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+), 6 deletions(-)


Summary over all repositories:
  4 files changed, 51 insertions(+), 24 deletions(-)

-- 
Generated by murpp 0.12.0

[PATCH manager 1/2] fix #8093: pvestatd: remove cleanup of stale lxc consoles

[Elias Huhsovitz]

Responsibility is moved to pve-container.

Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
 PVE/Service/pvestatd.pm | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 34f9da71..cec66e00 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -444,20 +444,6 @@ sub update_qemu_status {
     PVE::PullMetric::update($pull_txn, 'qemu', $vmstatus, $ctime);
 }
 
-sub remove_stale_lxc_consoles {
-
-    my $vmstatus = PVE::LXC::vmstatus();
-    my $pidhash = PVE::LXC::find_lxc_console_pids();
-
-    foreach my $vmid (keys %$pidhash) {
-        next if defined($vmstatus->{$vmid});
-        syslog('info', "remove stale lxc-console for CT $vmid");
-        foreach my $pid (@{ $pidhash->{$vmid} }) {
-            kill(9, $pid);
-        }
-    }
-}
-
 my $rebalance_error_count = {};
 
 my $NO_REBALANCE;
@@ -848,10 +834,6 @@ sub update_status {
     $err = $@;
     syslog('err', "storage status update error: $err") if $err;
 
-    eval { remove_stale_lxc_consoles(); };
-    $err = $@;
-    syslog('err', "lxc console cleanup error: $err") if $err;
-
     eval { rotate_authkeys(); };
     $err = $@;
     syslog('err', "authkey rotation error: $err") if $err;
-- 
2.47.3

[PATCH container 2/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Elias Huhsovitz]

Console processes can outlive their useful session when a container
stops or disappears while a console is attached. The previous cleanup
ran in pvestatd, which polled every 10s: forked lxc-info for every
running container, and killed orphaned processes by their PID.

Manage console lifetime in the container lifecycle instead. Add
wrap_in_console_scope() to start console commands in a systemd scope.
Bind each scope with PartOf= to the respective pve-container service.
Use the wrapper for pct console/enter/exec, and the API console
endpoints.

When the container service stops, systemd stops the related console
scopes.

Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
 src/PVE/API2/LXC.pm |  6 +++---
 src/PVE/CLI/pct.pm  | 10 +++++++---
 src/PVE/LXC.pm      | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
index 5f94d5a..3fd2bf8 100644
--- a/src/PVE/API2/LXC.pm
+++ b/src/PVE/API2/LXC.pm
@@ -1015,7 +1015,7 @@ __PACKAGE__->register_method({
         my $ticket = PVE::AccessControl::assemble_vnc_ticket($authuser, $authpath, $port);
 
         my $conf = PVE::LXC::Config->load_config($vmid, $node);
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf, -1);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf, -1);
 
         my $shcmd = [
             '/usr/bin/dtach',
@@ -1146,7 +1146,7 @@ __PACKAGE__->register_method({
         my $ticket = PVE::AccessControl::assemble_vnc_ticket($authuser, $authpath, $port);
 
         my $conf = PVE::LXC::Config->load_config($vmid, $node);
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf, -1);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf, -1);
 
         my $shcmd = [
             '/usr/bin/dtach',
@@ -1288,7 +1288,7 @@ __PACKAGE__->register_method({
 
         die "CT $vmid not running\n" if !PVE::LXC::check_running($vmid);
 
-        my $concmd = PVE::LXC::get_console_command($vmid, $conf);
+        my $concmd = PVE::LXC::get_console_command_scoped($vmid, $conf);
 
         my $shcmd = [
             '/usr/bin/dtach',
diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm
index ceb0bea..11e85bc 100755
--- a/src/PVE/CLI/pct.pm
+++ b/src/PVE/CLI/pct.pm
@@ -153,7 +153,7 @@ __PACKAGE__->register_method({
         # test if container exists on this node
         my $conf = PVE::LXC::Config->load_config($param->{vmid});
 
-        my $cmd = PVE::LXC::get_console_command($param->{vmid}, $conf, $param->{escape});
+        my $cmd = PVE::LXC::get_console_command_scoped($param->{vmid}, $conf, $param->{escape});
         exec(@$cmd);
     },
 });
@@ -207,7 +207,9 @@ __PACKAGE__->register_method({
 
         my @lxc_attach_cmd = ('lxc-attach', '-n', $vmid);
         push @lxc_attach_cmd, $keep_env ? '--keep-env' : '--clear-env';
-        exec(@lxc_attach_cmd);
+
+        my $scoped_cmd = PVE::LXC::wrap_in_console_scope($vmid, \@lxc_attach_cmd);
+        exec(@$scoped_cmd);
     },
 });
 
@@ -250,7 +252,9 @@ __PACKAGE__->register_method({
         my @lxc_attach_cmd = ('lxc-attach', '-n', $vmid);
         push @lxc_attach_cmd, $keep_env ? '--keep-env' : '--clear-env';
         push @lxc_attach_cmd, '--', @{ $param->{'extra-args'} };
-        exec(@lxc_attach_cmd);
+
+        my $scoped_cmd = PVE::LXC::wrap_in_console_scope($vmid, \@lxc_attach_cmd);
+        exec(@$scoped_cmd);
     },
 });
 
diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index dee073d..14bc6e4 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -954,6 +954,39 @@ sub verify_searchdomain_list {
     return join(' ', @list);
 }
 
+=head2 wrap_in_console_scope
+
+Wraps a command array in a transient systemd scope and binds it to the
+container's main systemd service. This ensures the console process is
+automatically terminated by systemd when the container stops, preventing
+orphaned processes without requiring manual cleanup scripts.
+
+=cut
+
+sub wrap_in_console_scope {
+    my ($vmid, $cmd) = @_;
+
+    my $session_id = "$$-" . time();
+    my $unit_name = "pve-lxc-console-$vmid-$session_id";
+
+    my $service = "pve-container\@$vmid.service";
+    # service name appends "-debug" when container is started in debug mode
+    my $debug_service = "pve-container-debug\@$vmid.service";
+
+    my @scope_cmd = (
+        'systemd-run',
+        '--scope',
+        '--unit', $unit_name,
+        '--description', "PVE LXC Console for $vmid",
+        '--property', "PartOf=$service $debug_service",
+        '--quiet',
+    );
+
+    push @scope_cmd, @$cmd;
+
+    return \@scope_cmd;
+}
+
 sub get_console_command {
     my ($vmid, $conf, $escapechar) = @_;
 
@@ -978,6 +1011,14 @@ sub get_console_command {
     return $cmd;
 }
 
+sub get_console_command_scoped {
+    my ($vmid, $conf, $escapechar) = @_;
+
+    my $cmd = get_console_command($vmid, $conf, $escapechar);
+
+    return wrap_in_console_scope($vmid, $cmd)
+}
+
 sub get_primary_ips {
     my ($conf) = @_;
 
-- 
2.47.3

Re: [PATCH container/manager 0/2] fix #8093: console: bind console sessions to container lifetime via systemd scopes

[Filip Schauer]

On 07/10/2026 10:37, Elias Huhsovitz wrote:
> The current Problem
> -------------------
> Console session processes can outlive their container when it stops or
> is destroyed. The cleanup currently runs in pvestatd every 10 seconds.
> It calls vmstatus() and kills stale processes by PID.
> 
> The killing of the process can result in a race condition:
> If the console gets closed by something else, just before the kill
> command is executed, a different process can reuse the PID and we kill
> a completely unrelated process.
> 
> Also checking this every 10s is too much unnecessary overhead IMO.
> 
> My understanding of a "better" architecture
> -------------------------------------------
> IMO console deletion should be tied to the state of the underlying
> container. Also I was unable to find a "convenient" time/event for a
> cleanup job to run.
> 
> Proposed implementation
> -----------------------
> Wrap console commands in transient systemd scopes and bind them to the
> container service via the PartOf= property. When the container stops,
> systemd stops the scope, terminating the console session and the dtach
> session. This allows removing the pvestatd periodic cleanup entirely.

Getting away from the polling in pvestatd would be nice.
However, I don't think `PartOf=` can be relied on here.

As a test I created a scope running `sleep 1000` with
`PartOf=pve-container@103.service`. Stopping the container left the
scope and its process running. It seems like `lxc-stop` ends the service
without a systemd stop job, and `PartOf=` only propagates explicit stop
jobs. The same goes for a `poweroff` inside the container. As far as I
can tell, the scope would only be stopped by `systemctl stop` or
`systemctl restart`. So most stop paths are not covered.

But do we even need to kill the lxc-console process in the first place?
lxc-console already exits automatically when the container stops. Or am
I missing something? Is there any case where the process lingers?