From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 060BE1FF135 for ; Thu, 02 Jul 2026 11:23:21 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2D31121429; Thu, 02 Jul 2026 11:23:20 +0200 (CEST) From: Lukas Wagner To: pdm-devel@lists.proxmox.com Subject: [PATCH datacenter-manager/proxmox 00/15] task cache improvements (archive corruption handling, error handling) Date: Thu, 2 Jul 2026 11:22:43 +0200 Message-ID: <20260702092258.174740-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1782984189201 X-SPAM-LEVEL: Spam detection results: 0 DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LLA3GYVDBYEYOQLCHOXXLJKTOIDSLWOL X-Message-ID-Hash: LLA3GYVDBYEYOQLCHOXXLJKTOIDSLWOL X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series bundles a couple of improvements and some refactoring efforts for the task cache implementation in PDM. Highlights: - Fix an issue that led to PBS task fetch cutoffs not being recorded in the state file, leading to always the full task history being requested - Fail more gracefully if any of the zstd-archive files were corrupted (before, you would end up in an endless loop when reading tasks due to the iterator not making any progress) - When creating a new zstd-compressed file, make sure create the file atomically. This avoids creating a .zst file without a proper zst file header if the process crashes at the wrong moment. - Detect archive corruption during read/write access (when applying journal and when compressing archive files). Attempt basic recovery in the next task fetching cycle by resetting cutoff timestamps to the lower boundary of the corrupted file. - Improve logging, trying to include path to the affected archive file if there is an error - Statically compute static paths and only instantiate TaskCache once The series also contains a patch for proxmox-sys, which is technically not needed for this series, but the improvement opportunity popped up when working on this series. proxmox: Lukas Wagner (1): sys: fs: don't replace file extension make_tmp_file proxmox-sys/src/fs/file.rs | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) proxmox-datacenter-manager: Lukas Wagner (14): task cache: fix missing cutoff state for PBS remotes task cache: refresh task: don't apply journal if the archive was rotated task cache: rotate: align timestamp for new files to UTC midnight task cache: add test case for task cache rotation task cache: pre-compute static paths during initialization task cache: only initialize `TaskCache` struct once task cache: archive iterator: don't yield more items if reading from file failed task cache: include archive file path in error log messages task cache: introduce ArchiveFileWriter task cache: use ArchiveFileWriter when creating new archive files task cache: trigger repair of corruption when applying journal task cache: trigger repair of corruption when compressing archive files task cache: trigger repair of corruption after read-accesses task cache: handle potentially duplicated archive files after 'compress_archive_file' server/src/remote_tasks/mod.rs | 53 +- server/src/remote_tasks/refresh_task.rs | 51 +- server/src/remote_tasks/task_cache.rs | 1055 ++++++++++++++++++----- 3 files changed, 922 insertions(+), 237 deletions(-) Summary over all repositories: 4 files changed, 955 insertions(+), 238 deletions(-) -- Generated by murpp 0.12.0