From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 0B9E71FF0E7 for ; Fri, 10 Jul 2026 13:37:16 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id CDE2021454; Fri, 10 Jul 2026 13:37:15 +0200 (CEST) Message-ID: <981253f7-30e7-4095-949b-5a6984c2902d@proxmox.com> Date: Fri, 10 Jul 2026 13:36:39 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH datacenter-manager/proxmox 00/15] task cache improvements (archive corruption handling, error handling) To: Lukas Wagner , pdm-devel@lists.proxmox.com References: <20260702092258.174740-1-l.wagner@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: <20260702092258.174740-1-l.wagner@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783683388239 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.053 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KADUBQ2OXV3TJRBADWLZBXPK65VZ65KO X-Message-ID-Hash: KADUBQ2OXV3TJRBADWLZBXPK65VZ65KO X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: sent a few comments and nits, some things are probably just misunderstandings of my side (since i'm not super deep into the task caching code)... On 7/2/26 11:23 AM, Lukas Wagner wrote: > This series bundles a couple of improvements and some refactoring efforts for > the task cache implementation in PDM. > > Highlights: > > - Fix an issue that led to PBS task fetch cutoffs not being recorded in the > state file, leading to always the full task history being requested > > - Fail more gracefully if any of the zstd-archive files were corrupted (before, > you would end up in an endless loop when reading tasks due to the iterator not > making any progress) > > - When creating a new zstd-compressed file, make sure create the file > atomically. This avoids creating a .zst file without a proper zst file header > if the process crashes at the wrong moment. > > - Detect archive corruption during read/write access (when applying journal and when > compressing archive files). Attempt basic recovery in the next task fetching > cycle by resetting cutoff timestamps to the lower boundary of the corrupted > file. > > - Improve logging, trying to include path to the affected archive file if there > is an error > > - Statically compute static paths and only instantiate TaskCache once > > The series also contains a patch for proxmox-sys, which is technically not > needed for this series, but the improvement opportunity popped up when working > on this series. > > > proxmox: > > Lukas Wagner (1): > sys: fs: don't replace file extension make_tmp_file > > proxmox-sys/src/fs/file.rs | 34 +++++++++++++++++++++++++++++++++- > 1 file changed, 33 insertions(+), 1 deletion(-) > > > proxmox-datacenter-manager: > > Lukas Wagner (14): > task cache: fix missing cutoff state for PBS remotes > task cache: refresh task: don't apply journal if the archive was > rotated > task cache: rotate: align timestamp for new files to UTC midnight > task cache: add test case for task cache rotation > task cache: pre-compute static paths during initialization > task cache: only initialize `TaskCache` struct once > task cache: archive iterator: don't yield more items if reading from > file failed > task cache: include archive file path in error log messages > task cache: introduce ArchiveFileWriter > task cache: use ArchiveFileWriter when creating new archive files > task cache: trigger repair of corruption when applying journal > task cache: trigger repair of corruption when compressing archive > files > task cache: trigger repair of corruption after read-accesses > task cache: handle potentially duplicated archive files after > 'compress_archive_file' > > server/src/remote_tasks/mod.rs | 53 +- > server/src/remote_tasks/refresh_task.rs | 51 +- > server/src/remote_tasks/task_cache.rs | 1055 ++++++++++++++++++----- > 3 files changed, 922 insertions(+), 237 deletions(-) > > > Summary over all repositories: > 4 files changed, 955 insertions(+), 238 deletions(-) >