From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id F40151FF0DA for ; Thu, 19 Nov 2026 23:00:16 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B8A252182C; Thu, 19 Nov 2026 23:00:13 +0100 (CET) Message-ID: Date: Thu, 19 Nov 2026 23:00:06 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH common v2] Added D state checking to `run` subroutine To: Jordyn Hamilton , pve-devel@lists.proxmox.com References: <20261119184502.22359-1-j.hamilton@proxmox.com> Content-Language: en-US From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20261119184502.22359-1-j.hamilton@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795125606923 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.149 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LOL5DG34UWL45SBOJRP3RRTOBF2C4ALX X-Message-ID-Hash: LOL5DG34UWL45SBOJRP3RRTOBF2C4ALX X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: c.hughes@proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 19.11.26 um 19:45 schrieb Jordyn Hamilton: > Added a subroutine to check if a process is in a D state. It does so by > reading the processes `stat` file from `/proc/$pid/stat`, then using > regex to grab a group containing the status character. It then returns > true or false if the status is 'D'. Not checking for any other status > types as D state is the only one that needs special logic. > > Added flow to check for a D state and customize the error message to > reflect that state within the timeout logic tree. The timeout branch was > chosen since the d state check would only be required if the process > takes long enough to time out. This is a very literal description of the code, I'm faster understanding what the patch does when reading the code change over having to read this in full. This makes it rather noise and goes against the developer documentations guideline's [0]: "Do not put in a literal description of the code diff, especially not for simpler changes." -- https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages > > Logic tree queues a `SIGKILL` prior to any D state checking, as the Sorry, what tree? > system would queue the signal if the process is in a D state, meaning it > would auto terminate once the state resolves. This also means the > standard timeout error checking remains untouched, it still sends the > signal, waits for the process to terminate then dies. > > Since there is no way to terminate D state processes programatically, s/programatically/programmatically/ > the program only overwrites the error message then dies, orphaning the > D state process while allowing future processes relying on the > completion of this one to run. It directs the user in the right > direction for debugging and provides all the information necessary for > beginning that process including the `$pid` and `$cmd_str`. > I'd propose the following subject and body (certainly not perfect language-wise, but to better show what we would rather see here): ----8<---- run cmd: report a distinct error when a timed-out process is in D-state Query the status of the process that run() spawned, and if it's in the infamous D-state (uninterruptible sleep), add that to the warning message to allow easier triaging of such events. ---->8---- That basically tells the full intention (most important) and gives a high level quick overview (good enough for such a patch). This is a example. it could be a bit longer and still be fine. > from v2: > - Pulled master from remote and rebased branch off master to resolve > merge conflicts. > - Modified function calls to use parentheses in accordance with the perl > style guide > - Modified error message string to be fomatted by perl style guides > - Squashed commits into one Thank you for providing a changelog, that's definitively appreciated, but placing it here makes it a permanent part of the git history, but this is relevant for review only. When applying to git we "pretended" all previous revisions did not happen and we made that perfect commit on the first go. Well not really, but for the git log the history of a change is secondary. Using the b4 tool we auto-link to the mailing-list on applying, so if one is interested they can easily go back to the ML and figure out the history for the few cases its relevant. Otherwise it adds rather just noise for git log, especially on bigger series and many revisions. You can place it instead ... > > Signed-off-by: Jordyn Hamilton > --- ... here, that's the diffstat section and is ignored by git am. > src/PVE/Cmd.pm | 24 ++++++++++++++++++++++++ > 1 file changed, 24 insertions(+) > > diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm > index 7dee654..9715a06 100644 > --- a/src/PVE/Cmd.pm > +++ b/src/PVE/Cmd.pm > @@ -44,6 +44,18 @@ sub split_args($str) { > return $str ? [Text::ParseWords::shellwords($str)] : []; > } > > +# check a subprocesses `/proc//stat` file for a D state > +sub in_d_state($pid) { If we do this here manually I'd make it a private sub using the `my` prefix, i.e.: my sub in_d_state($pid) { > + open(my $pstat_h, '<', "/proc/$pid/stat") or die $!; This also dies if the process was already reaped and the proc entry is gone, one would check the ENOENT error and ignore it in that case. Also some context in the die would be nice, getting some generic "file does not exist" error in a report is not really helpful. Like: open(my $pstat_h, '<', "/proc/$pid/stat") or $!{ENOENT} or die "failed to open $pid stat file - $!"; return if !defined(pstat_h); But FWIW, might be more sensible to reuse the existing PVE::ProcFSTools::read_proc_pid_stat() here, albeit ProcFSTools is not included yet here in the leaner Cmd module, so there is some benefit of a local helper. > + my $stat_in = <$pstat_h>; > + close($pstat_h); > + > + # extracting the stat character from the input would drop above comment, it's not really telling one much and just describing the code in literal form. Comments should get added only for subtle or non-obvious rationale or complexity. > + my $status = $1 if ($stat_in =~ /^\d+\s+\(.*\)\s+([A-Z])/); Two issues, for one the command name itself can contain parentheses itself, most linux filenames allow all bytes inside file names but / and \0. Further these post-of variable declarations is undefined behavior and rather dangerous, as in practice perl keeps the value from the last true evaluation of the if expression, so the next time its false you'll still get the previous value in $status and compare stale data below. https://pve.proxmox.com/wiki/Perl_Style_Guide#:~:text=declaration%20behind%20post%2Dif%20is%20undefined (also skim the rest of that Perl Style guide, and employ proxmox-perltidy) > + > + return ($status eq 'D'); > +} > + > # The $cmd parameter can be: > # -) a string > # This is generally executed by passing it to the shell with the -c option. > @@ -321,6 +333,18 @@ sub run($cmd, %param) { > > if ($err) { > if ($pid && ($err eq "got timeout\n")) { > + kill(9, $pid); that already happened farther above, and thus > + if (in_d_state($pid)) { here the check runs after the child is already killed and reaped, which will cause the open inside to fail, which make me question if this was ever tested in practice? > + $err = "process '$pid' is in an uninterruptible sleep" > + . " and cannot resolve or be terminated by this program.\n" > + . "This process can only be terminated by" > + . " resolving the errors or performing a system reboot.\n" > + . "This is due to underlying kernel or hardware errors," > + . " check dmesg for errors or" > + . " kernel messages.\n"; why that many (unbalanced) line-wraps? lets use the full 100cc text-width here to avoid line-bloat. > + }else { missing space before else (proxmox-perltidy handles that for you). > + waitpid($pid, 0); > + } > die "command '$cmdstr' failed: $err"; > } > When reviewing this I noticed that a89e2d8 ("cmd: run: kill the command when run dies") actually introduced a regression, I fixed that in [0]. That also should address some (all?) issues you see here, especially if it happens since a few months. [0]: https://git.proxmox.com/?p=pve-common.git;a=commitdiff;h=8f3c3eccbd706d10e4dcab25c600d6bb7c18d50b You can still add the D-state distinction on top, I'd suggested something like: ----8<---- diff --git a/src/PVE/Cmd.pm b/src/PVE/Cmd.pm index 31f0a94..d629952 100644 --- a/src/PVE/Cmd.pm +++ b/src/PVE/Cmd.pm @@ -44,6 +44,16 @@ sub split_args($str) { return $str ? [Text::ParseWords::shellwords($str)] : []; } +# Returns the state letter of process $pid as in /proc//stat, or undef if that cannot be read. +# Kept minimal on purpose, to avoid pulling in PVE::ProcFSTools here. +my sub process_state($pid) { + open(my $fh, '<', "/proc/$pid/stat") or return undef; + my $stat = do { local $/; <$fh> }; + close($fh); + # the command name in parentheses can contain parentheses itself, the state follows the last one + return defined($stat) && $stat =~ /^.*\) (\S) /s ? $1 : undef; +} + # The $cmd parameter can be: # -) a string # This is generally executed by passing it to the shell with the -c option. @@ -299,8 +309,13 @@ sub run($cmd, %param) { my $give_up_at = Time::HiRes::time() + 5; while (!waitpid($pid, POSIX::WNOHANG)) { if (Time::HiRes::time() >= $give_up_at) { - warn "command '$cmdstr' (PID $pid) did not exit after being killed, not waiting" - . " for it any longer\n"; + my $msg = "command '$cmdstr' (PID $pid) did not exit after being killed," + . " not waiting for it any longer"; + if ((process_state($pid) // '') eq 'D') { + $msg .= " - it is in uninterruptible sleep, check the kernel log for storage" + . " problems"; + } + warn "$msg\n"; last; } usleep(10_000);