[PATCH manager v2] fix #8093: pvestatd: remove stale lxc console reaper

[PATCH manager v2] fix #8093: pvestatd: remove stale lxc console reaper

[Elias Huhsovitz]

remove_stale_lxc_consoles() calls PVE::LXC::vmstatus() every cycle,
which runs `lxc-info` once per running container. Since
update_lxc_status() already calls vmstatus() in the same cycle, this
doubles the overhead of running `lxc-info`. Thus, this overhead is
scaling linearly with the number of containers.

Under normal operation, lxc-console and dtach exit automatically when
the container stops. The reaper only fires in broken system states, such
as deleting the container config file while the container is still
running. A continuous polling reaper is not an appropriate mechanism for
these situations.

Remove the reaper entirely.

Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
---
This patch removes the reaper entirely. No replacement mechanism is
added. If a console process becomes orphaned due to manual system
interference, the administrator can identify and kill it directly.

A type of "replacement" machenanism that bridges this gap is
currently being discussed [1], based on patch [2].

Alternatives considered
-----------------------
* Wrapping console commands in transient systemd scopes bound via
  PartOf= to the container service.
  Verdict: rejected because lxc-stop ends the
  service without a systemd stop job, so PartOf= does not propagate.

* A manual `pct cleanup console` CLI tool.
 Verdict: cleanup shouldn't be manual.

* An ExecStopPost= hook on pve-container@.service.

Changes v1->v2:
---------------
* scrap the idea of binding console sessions to container lifetime via
  systemd scoopes.

Previous versions
-----------------
v1: https://lore.proxmox.com/pve-devel/20261007083726.26067-1-e.huhsovitz@proxmox.com/

References
----------
[1] https://lore.proxmox.com/pve-devel/82fceb8e-f52d-4e56-9dcf-5cd61393fa26@proxmox.com/
[2] https://lore.proxmox.com/pve-devel/20260121112335.84473-1-f.schauer@proxmox.com/

Summary of Changes
------------------
 PVE/Service/pvestatd.pm | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 34f9da71..cec66e00 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -444,20 +444,6 @@ sub update_qemu_status {
     PVE::PullMetric::update($pull_txn, 'qemu', $vmstatus, $ctime);
 }
 
-sub remove_stale_lxc_consoles {
-
-    my $vmstatus = PVE::LXC::vmstatus();
-    my $pidhash = PVE::LXC::find_lxc_console_pids();
-
-    foreach my $vmid (keys %$pidhash) {
-        next if defined($vmstatus->{$vmid});
-        syslog('info', "remove stale lxc-console for CT $vmid");
-        foreach my $pid (@{ $pidhash->{$vmid} }) {
-            kill(9, $pid);
-        }
-    }
-}
-
 my $rebalance_error_count = {};
 
 my $NO_REBALANCE;
@@ -848,10 +834,6 @@ sub update_status {
     $err = $@;
     syslog('err', "storage status update error: $err") if $err;
 
-    eval { remove_stale_lxc_consoles(); };
-    $err = $@;
-    syslog('err', "lxc console cleanup error: $err") if $err;
-
     eval { rotate_authkeys(); };
     $err = $@;
     syslog('err', "authkey rotation error: $err") if $err;
-- 
2.47.3

Re: [PATCH manager v2] fix #8093: pvestatd: remove stale lxc console reaper

[Filip Schauer]

As far as my testing is concerned, the only way to produce an orphaned
lxc-console process is to manually send a SIGSTOP to the lxc-console
process and then stop the container. So I don't think this cleanup
serves any purpose.

As a test I started 50 containers and measured how long the
remove_stale_lxc_console call was taking: It took about 0.8 seconds on
my machine. Which is quite long considering it runs every 10 seconds.

Calling lxc-info twice for every running container every 10 seconds
sounds inefficient. So I am in favor of removing this so we at least
drop this from two lxc-info calls down to one lxc-info call per running
container every 10 seconds.

I applied this patch locally and bulk shutdown hundreds of containers
with lxc-consoles. Not a single stale lxc-console remained.

Tested-by: Filip Schauer <f.schauer@proxmox.com>
Reviewed-by: Filip Schauer <f.schauer@proxmox.com>


On 09/10/2026 14:38, Elias Huhsovitz wrote:
> remove_stale_lxc_consoles() calls PVE::LXC::vmstatus() every cycle,
> which runs `lxc-info` once per running container. Since
> update_lxc_status() already calls vmstatus() in the same cycle, this
> doubles the overhead of running `lxc-info`. Thus, this overhead is
> scaling linearly with the number of containers.
> 
> Under normal operation, lxc-console and dtach exit automatically when
> the container stops. The reaper only fires in broken system states, such
> as deleting the container config file while the container is still
> running. A continuous polling reaper is not an appropriate mechanism for
> these situations.

Even in this situation (deleting the container config while the
container is running) the reaper is not needed, since lxc-console exits
automatically when the container stops.

> 
> Remove the reaper entirely.
> 
> Signed-off-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
> ---
> This patch removes the reaper entirely. No replacement mechanism is
> added. If a console process becomes orphaned due to manual system
> interference, the administrator can identify and kill it directly.
> 
> A type of "replacement" machenanism that bridges this gap is
> currently being discussed [1], based on patch [2].
> 
> Alternatives considered
> -----------------------
> * Wrapping console commands in transient systemd scopes bound via
>    PartOf= to the container service.
>    Verdict: rejected because lxc-stop ends the
>    service without a systemd stop job, so PartOf= does not propagate.
> 
> * A manual `pct cleanup console` CLI tool.
>   Verdict: cleanup shouldn't be manual.
> 
> * An ExecStopPost= hook on pve-container@.service.
> 
> Changes v1->v2:
> ---------------
> * scrap the idea of binding console sessions to container lifetime via
>    systemd scoopes.
> 
> Previous versions
> -----------------
> v1: https://lore.proxmox.com/pve-devel/20261007083726.26067-1-e.huhsovitz@proxmox.com/
> 
> References
> ----------
> [1] https://lore.proxmox.com/pve-devel/82fceb8e-f52d-4e56-9dcf-5cd61393fa26@proxmox.com/
> [2] https://lore.proxmox.com/pve-devel/20260121112335.84473-1-f.schauer@proxmox.com/
> 
> Summary of Changes
> ------------------
>   PVE/Service/pvestatd.pm | 18 ------------------
>   1 file changed, 18 deletions(-)
> 
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index 34f9da71..cec66e00 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -444,20 +444,6 @@ sub update_qemu_status {
>       PVE::PullMetric::update($pull_txn, 'qemu', $vmstatus, $ctime);
>   }
>   
> -sub remove_stale_lxc_consoles {
> -
> -    my $vmstatus = PVE::LXC::vmstatus();
> -    my $pidhash = PVE::LXC::find_lxc_console_pids();
> -
> -    foreach my $vmid (keys %$pidhash) {
> -        next if defined($vmstatus->{$vmid});
> -        syslog('info', "remove stale lxc-console for CT $vmid");
> -        foreach my $pid (@{ $pidhash->{$vmid} }) {
> -            kill(9, $pid);
> -        }
> -    }
> -}
> -
>   my $rebalance_error_count = {};
>   
>   my $NO_REBALANCE;
> @@ -848,10 +834,6 @@ sub update_status {
>       $err = $@;
>       syslog('err', "storage status update error: $err") if $err;
>   
> -    eval { remove_stale_lxc_consoles(); };
> -    $err = $@;
> -    syslog('err', "lxc console cleanup error: $err") if $err;
> -
>       eval { rotate_authkeys(); };
>       $err = $@;
>       syslog('err', "authkey rotation error: $err") if $err;

applied: [PATCH manager v2] fix #8093: pvestatd: remove stale lxc console reaper

[Thomas Lamprecht]

On Fri, 09 Oct 2026 14:38:19 +0200, Elias Huhsovitz wrote:
> remove_stale_lxc_consoles() calls PVE::LXC::vmstatus() every cycle,
> which runs `lxc-info` once per running container. Since
> update_lxc_status() already calls vmstatus() in the same cycle, this
> doubles the overhead of running `lxc-info`. Thus, this overhead is
> scaling linearly with the number of containers.
> 
> Under normal operation, lxc-console and dtach exit automatically when
> the container stops. The reaper only fires in broken system states, such
> as deleting the container config file while the container is still
> running. A continuous polling reaper is not an appropriate mechanism for
> these situations.
> 
> [...]

Applied, thanks!

[1/1] fix #8093: pvestatd: remove stale lxc console reaper
      https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=888c136dd303f41d7950a5c9c7381a6f14cea33b