public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH common 0/2] Adding D state checking to the run command
@ 2026-11-19 14:50 Jordyn Hamilton
  2026-11-19 14:50 ` [PATCH common 1/2] Added subroutine to check process status Jordyn Hamilton
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: Jordyn Hamilton @ 2026-11-19 14:50 UTC (permalink / raw)
  To: pve-devel; +Cc: j.hamilton, c.hughes

Currently the run command does not have any logic branches to check for
a D state. This can cause certain processes like `pvereport` to hang
indefinitely if it encounters a subprocess that ends up in this state.

Adding a logic branch off the timeout error branch to check for a D
state process that, if encountered, throws a meaningful error and
terminates. Due to the nature of D states, there is no way to
programatically fix or terminate a process in a D state, so the error
message contains the command, the process id, and first steps towards
debugging.

This change wasn't designed as opt-in/out due to the fact that a D
state would cause whatever process that calls it to hang indefinitely,
without proper termination or error messaging, and universally, having a
proper termination with an error message would provide a better user and
support experience.

Jordyn Hamilton (2):
  Added subroutine to check process status
  Added logic to check for d state to run subroutine

 src/PVE/Cmd.pm | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

-- 
2.47.3





^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH common 1/2] Added subroutine to check process status
  2026-11-19 14:50 [PATCH common 0/2] Adding D state checking to the run command Jordyn Hamilton
@ 2026-11-19 14:50 ` Jordyn Hamilton
  2026-11-19 15:33   ` Shannon Sterz
  2026-11-19 14:50 ` [PATCH common 2/2] Added logic to check for d state to run subroutine Jordyn Hamilton
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 7+ messages in thread
From: Jordyn Hamilton @ 2026-11-19 14:50 UTC (permalink / raw)
  To: pve-devel; +Cc: j.hamilton, c.hughes

Added a subroutine to check if a process is in a D state. It does so by
reading the processes `stat` file from `/proc/$pid/stat`, then using
regex to grab a group containing the status character. It then returns
true or false if the status is 'D'. Not checking for any other status
types as D state is the only one that needs special logic.

Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
---
 src/PVE/Cmd.pm | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
index 65d00a7..773e55a 100644
--- a/src/PVE/Cmd.pm
+++ b/src/PVE/Cmd.pm
@@ -44,6 +44,18 @@ sub split_args($str) {
     return $str ? [Text::ParseWords::shellwords($str)] : [];
 }
 
+# check a subprocesses `/proc/<PID>/stat` file for a D state
+sub in_d_state($pid) {
+    open my $pstat_h, '<', "/proc/$pid/stat" or die $!;
+    my $stat_in = <$pstat_h>;
+    close $pstat_h;
+
+    # extracting the stat character from the input
+    my $status = $1 if $stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/;
+
+    return ($status eq 'D');
+}
+
 # The $cmd parameter can be:
 #  -) a string
 #    This is generally executed by passing it to the shell with the -c option.
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH common 2/2] Added logic to check for d state to run subroutine
  2026-11-19 14:50 [PATCH common 0/2] Adding D state checking to the run command Jordyn Hamilton
  2026-11-19 14:50 ` [PATCH common 1/2] Added subroutine to check process status Jordyn Hamilton
@ 2026-11-19 14:50 ` Jordyn Hamilton
  2026-11-19 15:33   ` Shannon Sterz
  2026-11-19 15:33 ` [PATCH common 0/2] Adding D state checking to the run command Shannon Sterz
  2026-11-19 18:49 ` superseded " Jordyn Hamilton
  3 siblings, 1 reply; 7+ messages in thread
From: Jordyn Hamilton @ 2026-11-19 14:50 UTC (permalink / raw)
  To: pve-devel; +Cc: j.hamilton, c.hughes

Within the timeout error logic branch, I added flow to check for a D
state and customize the error message to reflect that state. I chose the
timeout branch as the d state check would only be required if the
process takes long enough to time out.

Logic tree queues a `SIGKILL` prior to any D state checking, as the
system would queue the signal if the process is in a D state, meaning it
would auto terminate once the state resolves. This also means the
standard timeout error checking remains untouched, it still sends the
signal, waits for the process to terminate then dies.

Since there is no way to terminate D state processes programatically,
the program only overwrites the error message then terminates. It
directs the user in the right direction for debugging and provides all
the information necessary for beginning that process including the
`$pid` and `$cmd_str`.

Modified the d state error message to include that the process can only
be terminated by resolving the errors or performing a system reboot.

Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
---
 src/PVE/Cmd.pm | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
index 773e55a..20cd62b 100644
--- a/src/PVE/Cmd.pm
+++ b/src/PVE/Cmd.pm
@@ -53,7 +53,7 @@ sub in_d_state($pid) {
     # extracting the stat character from the input
     my $status = $1 if $stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/;
 
-    return ($status eq 'D');
+    return $status eq 'D';
 }
 
 # The $cmd parameter can be:
@@ -326,7 +326,17 @@ sub run($cmd, %param) {
     if ($err) {
         if ($pid && ($err eq "got timeout\n")) {
             kill(9, $pid);
-            waitpid($pid, 0);
+            if (in_d_state($pid)) {
+                $err = "process '$pid' is in an uninterruptible sleep " .
+                    "and cannot resolve or be terminated by this program.\n" .
+                    "This process can only be terminated by resolving the errors " .
+                    "or performing a system reboot.\n" .
+                    "This is due to underlying kernel or hardware errors, " .
+                    "check dmesg for errors or " .
+                    "kernel messages.\n";
+            }else {
+                waitpid($pid, 0);
+            }
             die "command '$cmdstr' failed: $err";
         }
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH common 0/2] Adding D state checking to the run command
  2026-11-19 14:50 [PATCH common 0/2] Adding D state checking to the run command Jordyn Hamilton
  2026-11-19 14:50 ` [PATCH common 1/2] Added subroutine to check process status Jordyn Hamilton
  2026-11-19 14:50 ` [PATCH common 2/2] Added logic to check for d state to run subroutine Jordyn Hamilton
@ 2026-11-19 15:33 ` Shannon Sterz
  2026-11-19 18:49 ` superseded " Jordyn Hamilton
  3 siblings, 0 replies; 7+ messages in thread
From: Shannon Sterz @ 2026-11-19 15:33 UTC (permalink / raw)
  To: Jordyn Hamilton, pve-devel; +Cc: c.hughes

On Thu Nov 19, 2026 at 3:50 PM CET, Jordyn Hamilton wrote:
> Currently the run command does not have any logic branches to check for
> a D state. This can cause certain processes like `pvereport` to hang
> indefinitely if it encounters a subprocess that ends up in this state.
>
> Adding a logic branch off the timeout error branch to check for a D
> state process that, if encountered, throws a meaningful error and
> terminates. Due to the nature of D states, there is no way to
> programatically fix or terminate a process in a D state, so the error
> message contains the command, the process id, and first steps towards
> debugging.
>
> This change wasn't designed as opt-in/out due to the fact that a D
> state would cause whatever process that calls it to hang indefinitely,
> without proper termination or error messaging, and universally, having a
> proper termination with an error message would provide a better user and
> support experience.
>
> Jordyn Hamilton (2):
>   Added subroutine to check process status
>   Added logic to check for d state to run subroutine
>
>  src/PVE/Cmd.pm | 24 +++++++++++++++++++++++-
>  1 file changed, 23 insertions(+), 1 deletion(-)


Hi Jordyn,

thank you for your patches! Sadly they don't apply on top of current
master cleanly anymore, so I couldn't test them. I left a few nits, and
imo, it would be fine to combine the changes here into a single patch.




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH common 1/2] Added subroutine to check process status
  2026-11-19 14:50 ` [PATCH common 1/2] Added subroutine to check process status Jordyn Hamilton
@ 2026-11-19 15:33   ` Shannon Sterz
  0 siblings, 0 replies; 7+ messages in thread
From: Shannon Sterz @ 2026-11-19 15:33 UTC (permalink / raw)
  To: Jordyn Hamilton, pve-devel; +Cc: c.hughes

On Thu Nov 19, 2026 at 3:50 PM CET, Jordyn Hamilton wrote:
> Added a subroutine to check if a process is in a D state. It does so by
> reading the processes `stat` file from `/proc/$pid/stat`, then using
> regex to grab a group containing the status character. It then returns
> true or false if the status is 'D'. Not checking for any other status
> types as D state is the only one that needs special logic.
>
> Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
> ---
>  src/PVE/Cmd.pm | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
> index 65d00a7..773e55a 100644
> --- a/src/PVE/Cmd.pm
> +++ b/src/PVE/Cmd.pm
> @@ -44,6 +44,18 @@ sub split_args($str) {
>      return $str ? [Text::ParseWords::shellwords($str)] : [];
>  }
>
> +# check a subprocesses `/proc/<PID>/stat` file for a D state
> +sub in_d_state($pid) {
> +    open my $pstat_h, '<', "/proc/$pid/stat" or die $!;
> +    my $stat_in = <$pstat_h>;
> +    close $pstat_h;

nit: our style guide prefers the use of parentheses for function calls
[1].

[1]: https://pve.proxmox.com/wiki/Perl_Style_Guide#Perl_syntax_choices

> +
> +    # extracting the stat character from the input
> +    my $status = $1 if $stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/;
> +
> +    return ($status eq 'D');
> +}
> +
>  # The $cmd parameter can be:
>  #  -) a string
>  #    This is generally executed by passing it to the shell with the -c option.





^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH common 2/2] Added logic to check for d state to run subroutine
  2026-11-19 14:50 ` [PATCH common 2/2] Added logic to check for d state to run subroutine Jordyn Hamilton
@ 2026-11-19 15:33   ` Shannon Sterz
  0 siblings, 0 replies; 7+ messages in thread
From: Shannon Sterz @ 2026-11-19 15:33 UTC (permalink / raw)
  To: Jordyn Hamilton, pve-devel; +Cc: c.hughes

this patch doesn't apply any longer

On Thu Nov 19, 2026 at 3:50 PM CET, Jordyn Hamilton wrote:
> Within the timeout error logic branch, I added flow to check for a D
> state and customize the error message to reflect that state. I chose the
> timeout branch as the d state check would only be required if the
> process takes long enough to time out.
>
> Logic tree queues a `SIGKILL` prior to any D state checking, as the
> system would queue the signal if the process is in a D state, meaning it
> would auto terminate once the state resolves. This also means the
> standard timeout error checking remains untouched, it still sends the
> signal, waits for the process to terminate then dies.
>
> Since there is no way to terminate D state processes programatically,
> the program only overwrites the error message then terminates. It
> directs the user in the right direction for debugging and provides all
> the information necessary for beginning that process including the
> `$pid` and `$cmd_str`.
>
> Modified the d state error message to include that the process can only
> be terminated by resolving the errors or performing a system reboot.
>
> Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
> ---
>  src/PVE/Cmd.pm | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
> index 773e55a..20cd62b 100644
> --- a/src/PVE/Cmd.pm
> +++ b/src/PVE/Cmd.pm
> @@ -53,7 +53,7 @@ sub in_d_state($pid) {
>      # extracting the stat character from the input
>      my $status = $1 if $stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/;
>
> -    return ($status eq 'D');
> +    return $status eq 'D';

please refrain from modifying could trivial here. this line is added in
the previous commit, so this change should have just been part of that
patch. otherwise, this creates more review overhead. thank you!

>  }
>
>  # The $cmd parameter can be:
> @@ -326,7 +326,17 @@ sub run($cmd, %param) {
>      if ($err) {
>          if ($pid && ($err eq "got timeout\n")) {
>              kill(9, $pid);
> -            waitpid($pid, 0);
> +            if (in_d_state($pid)) {
> +                $err = "process '$pid' is in an uninterruptible sleep " .
> +                    "and cannot resolve or be terminated by this program.\n" .
> +                    "This process can only be terminated by resolving the errors " .
> +                    "or performing a system reboot.\n" .
> +                    "This is due to underlying kernel or hardware errors, " .
> +                    "check dmesg for errors or " .
> +                    "kernel messages.\n";

nit: this string isn't wrapped according to our style guide [1].

[1]: https://pve.proxmox.com/wiki/Perl_Style_Guide#Wrapping_Strings

> +            }else {
> +                waitpid($pid, 0);
> +            }
>              die "command '$cmdstr' failed: $err";
>          }
>





^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: superseded [PATCH common 0/2] Adding D state checking to the run command
  2026-11-19 14:50 [PATCH common 0/2] Adding D state checking to the run command Jordyn Hamilton
                   ` (2 preceding siblings ...)
  2026-11-19 15:33 ` [PATCH common 0/2] Adding D state checking to the run command Shannon Sterz
@ 2026-11-19 18:49 ` Jordyn Hamilton
  3 siblings, 0 replies; 7+ messages in thread
From: Jordyn Hamilton @ 2026-11-19 18:49 UTC (permalink / raw)
  To: pve-devel; +Cc: c.hughes

Superseded by 
https://lore.proxmox.com/pve-devel/20261119184502.22359-1-j.hamilton@proxmox.com/T/#u

On 2026-11-19 09:50, Jordyn Hamilton wrote:
> Currently the run command does not have any logic branches to check for
> a D state. This can cause certain processes like `pvereport` to hang
> indefinitely if it encounters a subprocess that ends up in this state.
>
> Adding a logic branch off the timeout error branch to check for a D
> state process that, if encountered, throws a meaningful error and
> terminates. Due to the nature of D states, there is no way to
> programatically fix or terminate a process in a D state, so the error
> message contains the command, the process id, and first steps towards
> debugging.
>
> This change wasn't designed as opt-in/out due to the fact that a D
> state would cause whatever process that calls it to hang indefinitely,
> without proper termination or error messaging, and universally, having a
> proper termination with an error message would provide a better user and
> support experience.
>
> Jordyn Hamilton (2):
>    Added subroutine to check process status
>    Added logic to check for d state to run subroutine
>
>   src/PVE/Cmd.pm | 24 +++++++++++++++++++++++-
>   1 file changed, 23 insertions(+), 1 deletion(-)
>




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-19 18:49 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 14:50 [PATCH common 0/2] Adding D state checking to the run command Jordyn Hamilton
2026-11-19 14:50 ` [PATCH common 1/2] Added subroutine to check process status Jordyn Hamilton
2026-11-19 15:33   ` Shannon Sterz
2026-11-19 14:50 ` [PATCH common 2/2] Added logic to check for d state to run subroutine Jordyn Hamilton
2026-11-19 15:33   ` Shannon Sterz
2026-11-19 15:33 ` [PATCH common 0/2] Adding D state checking to the run command Shannon Sterz
2026-11-19 18:49 ` superseded " Jordyn Hamilton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal