Re: [PATCH datacenter-manager/proxmox 00/15] task cache improvements (archive corruption handling, error handling)

[Dominik Csapak <d.csapak@proxmox.com>]

sent a few comments and nits, some things are probably just 
misunderstandings of my side (since i'm not super deep into the task
caching code)...

On 7/2/26 11:23 AM, Lukas Wagner wrote:
> This series bundles a couple of improvements and some refactoring efforts for
> the task cache implementation in PDM.
> 
> Highlights:
> 
> - Fix an issue that led to PBS task fetch cutoffs not being recorded in the
> state file, leading to always the full task history being requested
> 
> - Fail more gracefully if any of the zstd-archive files were corrupted (before,
> you would end up in an endless loop when reading tasks due to the iterator not
> making any progress)
> 
> - When creating a new zstd-compressed file, make sure create the file
> atomically. This avoids creating a .zst file without a proper zst file header
> if the process crashes at the wrong moment.
> 
> - Detect archive corruption during read/write access (when applying journal and when
> compressing archive files). Attempt basic recovery in the next task fetching
> cycle by resetting cutoff timestamps to the lower boundary of the corrupted
> file.
> 
> - Improve logging, trying to include path to the affected archive file if there
> is an error
> 
> - Statically compute static paths and only instantiate TaskCache once
> 
> The series also contains a patch for proxmox-sys, which is technically not
> needed for this series, but the improvement opportunity popped up when working
> on this series.
> 
> 
> proxmox:
> 
> Lukas Wagner (1):
>    sys: fs: don't replace file extension make_tmp_file
> 
>   proxmox-sys/src/fs/file.rs | 34 +++++++++++++++++++++++++++++++++-
>   1 file changed, 33 insertions(+), 1 deletion(-)
> 
> 
> proxmox-datacenter-manager:
> 
> Lukas Wagner (14):
>    task cache: fix missing cutoff state for PBS remotes
>    task cache: refresh task: don't apply journal if the archive was
>      rotated
>    task cache: rotate: align timestamp for new files to UTC midnight
>    task cache: add test case for task cache rotation
>    task cache: pre-compute static paths during initialization
>    task cache: only initialize `TaskCache` struct once
>    task cache: archive iterator: don't yield more items if reading from
>      file failed
>    task cache: include archive file path in error log messages
>    task cache: introduce ArchiveFileWriter
>    task cache: use ArchiveFileWriter when creating new archive files
>    task cache: trigger repair of corruption when applying journal
>    task cache: trigger repair of corruption when compressing archive
>      files
>    task cache: trigger repair of corruption after read-accesses
>    task cache: handle potentially duplicated archive files after
>      'compress_archive_file'
> 
>   server/src/remote_tasks/mod.rs          |   53 +-
>   server/src/remote_tasks/refresh_task.rs |   51 +-
>   server/src/remote_tasks/task_cache.rs   | 1055 ++++++++++++++++++-----
>   3 files changed, 922 insertions(+), 237 deletions(-)
> 
> 
> Summary over all repositories:
>    4 files changed, 955 insertions(+), 238 deletions(-)
>