[PATCH datacenter-manager/proxmox 00/15] task cache improvements (archive corruption handling, error handling)

[Lukas Wagner <l.wagner@proxmox.com>]

This series bundles a couple of improvements and some refactoring efforts for
the task cache implementation in PDM.

Highlights: 

- Fix an issue that led to PBS task fetch cutoffs not being recorded in the
state file, leading to always the full task history being requested

- Fail more gracefully if any of the zstd-archive files were corrupted (before,
you would end up in an endless loop when reading tasks due to the iterator not
making any progress)

- When creating a new zstd-compressed file, make sure create the file
atomically. This avoids creating a .zst file without a proper zst file header
if the process crashes at the wrong moment.

- Detect archive corruption during read/write access (when applying journal and when
compressing archive files). Attempt basic recovery in the next task fetching
cycle by resetting cutoff timestamps to the lower boundary of the corrupted
file.

- Improve logging, trying to include path to the affected archive file if there
is an error

- Statically compute static paths and only instantiate TaskCache once

The series also contains a patch for proxmox-sys, which is technically not
needed for this series, but the improvement opportunity popped up when working
on this series.


proxmox:

Lukas Wagner (1):
  sys: fs: don't replace file extension make_tmp_file

 proxmox-sys/src/fs/file.rs | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)


proxmox-datacenter-manager:

Lukas Wagner (14):
  task cache: fix missing cutoff state for PBS remotes
  task cache: refresh task: don't apply journal if the archive was
    rotated
  task cache: rotate: align timestamp for new files to UTC midnight
  task cache: add test case for task cache rotation
  task cache: pre-compute static paths during initialization
  task cache: only initialize `TaskCache` struct once
  task cache: archive iterator: don't yield more items if reading from
    file failed
  task cache: include archive file path in error log messages
  task cache: introduce ArchiveFileWriter
  task cache: use ArchiveFileWriter when creating new archive files
  task cache: trigger repair of corruption when applying journal
  task cache: trigger repair of corruption when compressing archive
    files
  task cache: trigger repair of corruption after read-accesses
  task cache: handle potentially duplicated archive files after
    'compress_archive_file'

 server/src/remote_tasks/mod.rs          |   53 +-
 server/src/remote_tasks/refresh_task.rs |   51 +-
 server/src/remote_tasks/task_cache.rs   | 1055 ++++++++++++++++++-----
 3 files changed, 922 insertions(+), 237 deletions(-)


Summary over all repositories:
  4 files changed, 955 insertions(+), 238 deletions(-)

-- 
Generated by murpp 0.12.0