[PATCH manager] fix #4986: jobs: do not get stuck if the task log is missing

[Michal Fox <me@dualfroz.com>]

When the scheduler checks whether the last run of a job has finished,
it dies if the task log of that run does not exist. The job is then
skipped with 'could not update job state, skipping - no such task' on
every iteration, so it never runs again until its state file is
cleaned up manually.

The task log can be gone, for example after a crash. If the task is
not running anymore, treat it as stopped with an error status instead,
so that the job gets scheduled again, like Proxmox Backup Server does
since commit 48fbce07 ("fix #4895: scheduled jobs: ignore task-log not
found error").

The check now comes after the one whether the task is still running,
so a running task does not get marked as stopped.

Signed-off-by: Michal Fox <me@dualfroz.com>
---
Tested with a script that runs run_jobs() with a mocked jobs.cfg on a
job state whose task log does not exist: before, the job was skipped
with 'no such task' on every run, now it is marked as stopped and
started again, while a job whose task is still running stays started.

 PVE/Jobs.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/PVE/Jobs.pm b/PVE/Jobs.pm
index 40caf257..75f87060 100644
--- a/PVE/Jobs.pm
+++ b/PVE/Jobs.pm
@@ -109,13 +109,15 @@ my $get_job_task_status = sub {
 
     my ($task, $filename) = PVE::Tools::upid_decode($state->{upid}, 1);
     die "unable to parse worker upid - $state->{upid}\n" if !$task;
-    die "no such task\n" if !-f $filename;
 
     my $pstart = PVE::ProcFSTools::read_proc_starttime($task->{pid});
     if ($pstart && $pstart == $task->{pstart}) {
         return; # still running
     }
 
+    # the task log might be gone, e.g. after a crash, do not let that block the job forever
+    return "task log not found" if !-f $filename;
+
     return PVE::Tools::upid_read_status($state->{upid});
 };
 
-- 
2.43.0