* [PATCH common v2] Added D state checking to `run` subroutine
@ 2026-11-19 18:45 Jordyn Hamilton
2026-11-19 22:00 ` Thomas Lamprecht
0 siblings, 1 reply; 2+ messages in thread
From: Jordyn Hamilton @ 2026-11-19 18:45 UTC (permalink / raw)
To: pve-devel; +Cc: j.hamilton, c.hughes
Added a subroutine to check if a process is in a D state. It does so by
reading the processes `stat` file from `/proc/$pid/stat`, then using
regex to grab a group containing the status character. It then returns
true or false if the status is 'D'. Not checking for any other status
types as D state is the only one that needs special logic.
Added flow to check for a D state and customize the error message to
reflect that state within the timeout logic tree. The timeout branch was
chosen since the d state check would only be required if the process
takes long enough to time out.
Logic tree queues a `SIGKILL` prior to any D state checking, as the
system would queue the signal if the process is in a D state, meaning it
would auto terminate once the state resolves. This also means the
standard timeout error checking remains untouched, it still sends the
signal, waits for the process to terminate then dies.
Since there is no way to terminate D state processes programatically,
the program only overwrites the error message then dies, orphaning the
D state process while allowing future processes relying on the
completion of this one to run. It directs the user in the right
direction for debugging and provides all the information necessary for
beginning that process including the `$pid` and `$cmd_str`.
from v2:
- Pulled master from remote and rebased branch off master to resolve
merge conflicts.
- Modified function calls to use parentheses in accordance with the perl
style guide
- Modified error message string to be fomatted by perl style guides
- Squashed commits into one
Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
---
src/PVE/Cmd.pm | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
index 7dee654..9715a06 100644
--- a/src/PVE/Cmd.pm
+++ b/src/PVE/Cmd.pm
@@ -44,6 +44,18 @@ sub split_args($str) {
return $str ? [Text::ParseWords::shellwords($str)] : [];
}
+# check a subprocesses `/proc/<PID>/stat` file for a D state
+sub in_d_state($pid) {
+ open(my $pstat_h, '<', "/proc/$pid/stat") or die $!;
+ my $stat_in = <$pstat_h>;
+ close($pstat_h);
+
+ # extracting the stat character from the input
+ my $status = $1 if ($stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/);
+
+ return ($status eq 'D');
+}
+
# The $cmd parameter can be:
# -) a string
# This is generally executed by passing it to the shell with the -c option.
@@ -321,6 +333,18 @@ sub run($cmd, %param) {
if ($err) {
if ($pid && ($err eq "got timeout\n")) {
+ kill(9, $pid);
+ if (in_d_state($pid)) {
+ $err = "process '$pid' is in an uninterruptible sleep"
+ . " and cannot resolve or be terminated by this program.\n"
+ . "This process can only be terminated by"
+ . " resolving the errors or performing a system reboot.\n"
+ . "This is due to underlying kernel or hardware errors,"
+ . " check dmesg for errors or"
+ . " kernel messages.\n";
+ }else {
+ waitpid($pid, 0);
+ }
die "command '$cmdstr' failed: $err";
}
--
2.47.3
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH common v2] Added D state checking to `run` subroutine
2026-11-19 18:45 [PATCH common v2] Added D state checking to `run` subroutine Jordyn Hamilton
@ 2026-11-19 22:00 ` Thomas Lamprecht
0 siblings, 0 replies; 2+ messages in thread
From: Thomas Lamprecht @ 2026-11-19 22:00 UTC (permalink / raw)
To: Jordyn Hamilton, pve-devel; +Cc: c.hughes
Am 19.11.26 um 19:45 schrieb Jordyn Hamilton:
> Added a subroutine to check if a process is in a D state. It does so by
> reading the processes `stat` file from `/proc/$pid/stat`, then using
> regex to grab a group containing the status character. It then returns
> true or false if the status is 'D'. Not checking for any other status
> types as D state is the only one that needs special logic.
>
> Added flow to check for a D state and customize the error message to
> reflect that state within the timeout logic tree. The timeout branch was
> chosen since the d state check would only be required if the process
> takes long enough to time out.
This is a very literal description of the code, I'm faster understanding
what the patch does when reading the code change over having to read this
in full. This makes it rather noise and goes against the developer
documentations guideline's [0]:
"Do not put in a literal description of the code diff, especially not for simpler changes."
-- https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages
>
> Logic tree queues a `SIGKILL` prior to any D state checking, as the
Sorry, what tree?
> system would queue the signal if the process is in a D state, meaning it
> would auto terminate once the state resolves. This also means the
> standard timeout error checking remains untouched, it still sends the
> signal, waits for the process to terminate then dies.
>
> Since there is no way to terminate D state processes programatically,
s/programatically/programmatically/
> the program only overwrites the error message then dies, orphaning the
> D state process while allowing future processes relying on the
> completion of this one to run. It directs the user in the right
> direction for debugging and provides all the information necessary for
> beginning that process including the `$pid` and `$cmd_str`.
>
I'd propose the following subject and body (certainly not perfect
language-wise, but to better show what we would rather see here):
----8<----
run cmd: report a distinct error when a timed-out process is in D-state
Query the status of the process that run() spawned, and if it's in
the infamous D-state (uninterruptible sleep), add that to the warning
message to allow easier triaging of such events.
---->8----
That basically tells the full intention (most important) and gives
a high level quick overview (good enough for such a patch). This is
a example. it could be a bit longer and still be fine.
> from v2:
> - Pulled master from remote and rebased branch off master to resolve
> merge conflicts.
> - Modified function calls to use parentheses in accordance with the perl
> style guide
> - Modified error message string to be fomatted by perl style guides
> - Squashed commits into one
Thank you for providing a changelog, that's definitively appreciated, but
placing it here makes it a permanent part of the git history, but this
is relevant for review only. When applying to git we "pretended" all
previous revisions did not happen and we made that perfect commit on
the first go. Well not really, but for the git log the history of a
change is secondary. Using the b4 tool we auto-link to the mailing-list
on applying, so if one is interested they can easily go back to the ML
and figure out the history for the few cases its relevant. Otherwise
it adds rather just noise for git log, especially on bigger series and
many revisions. You can place it instead ...
>
> Signed-off-by: Jordyn Hamilton <j.hamilton@proxmox.com>
> ---
... here, that's the diffstat section and is ignored by git am.
> src/PVE/Cmd.pm | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
>
> diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
> index 7dee654..9715a06 100644
> --- a/src/PVE/Cmd.pm
> +++ b/src/PVE/Cmd.pm
> @@ -44,6 +44,18 @@ sub split_args($str) {
> return $str ? [Text::ParseWords::shellwords($str)] : [];
> }
>
> +# check a subprocesses `/proc/<PID>/stat` file for a D state
> +sub in_d_state($pid) {
If we do this here manually I'd make it a private sub using the `my` prefix,
i.e.:
my sub in_d_state($pid) {
> + open(my $pstat_h, '<', "/proc/$pid/stat") or die $!;
This also dies if the process was already reaped and the proc entry
is gone, one would check the ENOENT error and ignore it in that case.
Also some context in the die would be nice, getting some generic
"file does not exist" error in a report is not really helpful.
Like:
open(my $pstat_h, '<', "/proc/$pid/stat") or $!{ENOENT} or die "failed to open $pid stat file - $!";
return if !defined(pstat_h);
But FWIW, might be more sensible to reuse the existing
PVE::ProcFSTools::read_proc_pid_stat() here, albeit ProcFSTools is not
included yet here in the leaner Cmd module, so there is some benefit
of a local helper.
> + my $stat_in = <$pstat_h>;
> + close($pstat_h);
> +
> + # extracting the stat character from the input
would drop above comment, it's not really telling one much and just describing
the code in literal form. Comments should get added only for subtle or
non-obvious rationale or complexity.
> + my $status = $1 if ($stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/);
Two issues, for one the command name itself can contain parentheses
itself, most linux filenames allow all bytes inside file names but /
and \0.
Further these post-of variable declarations is undefined behavior
and rather dangerous, as in practice perl keeps the value from the
last true evaluation of the if expression, so the next time its false
you'll still get the previous value in $status and compare stale data
below.
https://pve.proxmox.com/wiki/Perl_Style_Guide#:~:text=declaration%20behind%20post%2Dif%20is%20undefined
(also skim the rest of that Perl Style guide, and employ proxmox-perltidy)
> +
> + return ($status eq 'D');
> +}
> +
> # The $cmd parameter can be:
> # -) a string
> # This is generally executed by passing it to the shell with the -c option.
> @@ -321,6 +333,18 @@ sub run($cmd, %param) {
>
> if ($err) {
> if ($pid && ($err eq "got timeout\n")) {
> + kill(9, $pid);
that already happened farther above, and thus
> + if (in_d_state($pid)) {
here the check runs after the child is already killed and reaped, which will
cause the open inside to fail, which make me question if this was ever tested
in practice?
> + $err = "process '$pid' is in an uninterruptible sleep"
> + . " and cannot resolve or be terminated by this program.\n"
> + . "This process can only be terminated by"
> + . " resolving the errors or performing a system reboot.\n"
> + . "This is due to underlying kernel or hardware errors,"
> + . " check dmesg for errors or"
> + . " kernel messages.\n";
why that many (unbalanced) line-wraps? lets use the full 100cc text-width here
to avoid line-bloat.
> + }else {
missing space before else (proxmox-perltidy handles that for you).
> + waitpid($pid, 0);
> + }
> die "command '$cmdstr' failed: $err";
> }
>
When reviewing this I noticed that a89e2d8 ("cmd: run: kill the command
when run dies") actually introduced a regression, I fixed that in [0]. That
also should address some (all?) issues you see here, especially if it happens
since a few months.
[0]: https://git.proxmox.com/?p=pve-common.git;a=commitdiff;h=8f3c3eccbd706d10e4dcab25c600d6bb7c18d50b
You can still add the D-state distinction on top, I'd suggested something like:
----8<----
diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm
index 31f0a94..d629952 100644
--- a/src/PVE/Cmd.pm
+++ b/src/PVE/Cmd.pm
@@ -44,6 +44,16 @@ sub split_args($str) {
return $str ? [Text::ParseWords::shellwords($str)] : [];
}
+# Returns the state letter of process $pid as in /proc/<pid>/stat, or undef if that cannot be read.
+# Kept minimal on purpose, to avoid pulling in PVE::ProcFSTools here.
+my sub process_state($pid) {
+ open(my $fh, '<', "/proc/$pid/stat") or return undef;
+ my $stat = do { local $/; <$fh> };
+ close($fh);
+ # the command name in parentheses can contain parentheses itself, the state follows the last one
+ return defined($stat) && $stat =~ /^.*\) (\S) /s ? $1 : undef;
+}
+
# The $cmd parameter can be:
# -) a string
# This is generally executed by passing it to the shell with the -c option.
@@ -299,8 +309,13 @@ sub run($cmd, %param) {
my $give_up_at = Time::HiRes::time() + 5;
while (!waitpid($pid, POSIX::WNOHANG)) {
if (Time::HiRes::time() >= $give_up_at) {
- warn "command '$cmdstr' (PID $pid) did not exit after being killed, not waiting"
- . " for it any longer\n";
+ my $msg = "command '$cmdstr' (PID $pid) did not exit after being killed,"
+ . " not waiting for it any longer";
+ if ((process_state($pid) // '') eq 'D') {
+ $msg .= " - it is in uninterruptible sleep, check the kernel log for storage"
+ . " problems";
+ }
+ warn "$msg\n";
last;
}
usleep(10_000);
^ permalink raw reply related [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-19 22:00 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 18:45 [PATCH common v2] Added D state checking to `run` subroutine Jordyn Hamilton
2026-11-19 22:00 ` Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox