[PATCH proxmox-backup v3 0/3] fix #7400: improve handling of corrupted job statefiles

["Michael Köppl" <m.koeppl@proxmox.com>]

This patch series fixes a problem where an empty or corrupted job state
file (due to I/O error, abrupt shutdown, ...) would cause API endpoints
for listing jobs to return an error, breaking the web UI for users
because they could not view any of their configured jobs of that type.
It would also cause proxmox-backup-proxy to indefinitely skip the jobs
until a user manually triggered it to rewrite the statefile.

1/3 is a preparatory patch that centralizes job statefile loading
in compute_schedule_status instead of having every handler function
open the statefile, handle potential errors and then passing the
JobState to compute_schedule_status.

2/3 introduces a new JobState `Unknown`, representing cases in which the
job state could not be determined. In addition, the patch also updates
the scheduling functions such that errors during reading the statefiles
will result in the Unknown state.

3/3 then utilizes this Unknown state and adapts the scheduling functions
such that the Unknown state will then lead to the statefile being
overwritten with a new Created state and the job running again at its
next scheduled run.

changes since v2:
- introduced the Unknown state in 2/3, adapted 3/3 accordingly (thanks,
  @Fabian and @Christian)
- make sure the "could not open statefile" error is also printed in
  garbage_collection_status if status_in_memory.upid is None (thanks,
  @Christian)
- inline jobtype and err variables in error logging

changes since v1:
- added preparatory patch 1/3, centralizing the statefile loading before
  adapting the handling of the error case in that centralized place
  (compute_schedule_status). Thanks, Christian for the suggestion!
- adapted the error message if job statefile loading fails to make clear
  that the default status will be returned as a fallback

proxmox-backup:

Michael Köppl (3):
  api: move statefile loading into compute_schedule_status
  fix #7400: api: gracefully handle corrupted job statefiles
  fix #7400: proxy: self-heal corrupted job statefiles

 src/api2/admin/datastore.rs     | 15 ++++------
 src/api2/admin/prune.rs         |  9 ++----
 src/api2/admin/sync.rs          |  9 ++----
 src/api2/admin/verify.rs        |  9 ++----
 src/api2/tape/backup.rs         |  9 ++----
 src/bin/proxmox-backup-proxy.rs |  4 ++-
 src/server/jobstate.rs          | 52 +++++++++++++++++++++++++++++----
 7 files changed, 67 insertions(+), 40 deletions(-)


Summary over all repositories:
  7 files changed, 67 insertions(+), 40 deletions(-)

-- 
Generated by murpp 0.11.0