From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id CD0A11FF0AF for ; Fri, 20 Nov 2026 14:51:22 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 36A6421719; Fri, 20 Nov 2026 14:51:22 +0100 (CET) Message-ID: <0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com> Date: Fri, 20 Nov 2026 14:51:18 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task To: Christian Ebner , pbs-devel@lists.proxmox.com References: <20261120082710.65728-1-c.ebner@proxmox.com> <20261120082710.65728-4-c.ebner@proxmox.com> Content-Language: en-US From: Samuel Rufinatscha In-Reply-To: <20261120082710.65728-4-c.ebner@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.152 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: X5PO3GGLKZNJT7VC5HEK2X7ACWLNXZZE X-Message-ID-Hash: X5PO3GGLKZNJT7VC5HEK2X7ACWLNXZZE X-MailFrom: s.rufinatscha@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 11/20/26 9:27 AM, Christian Ebner wrote: > Determining the cutoff timestamp for old task log cleanup previously > only inspected the oldest entry in the task archive, which only tracks > finished tasks. Still running tasks are listed exclusively in the > active tasks file without an end time. > > If a long-running task started before the calculated archive cutoff > time and has not written any new log lines since then, its log file's > mtime is older than the cutoff time and cleanup_old_tasks() would > remove the log file while the task is still active. > > Prevent this by also reading the active task list while holding the > shared task list lock during cutoff time gathering and clamping the > cutoff timestamp to the earliest active task's start time. Since a > task's log file is created after its UPID start time is generated, > its mtime is guaranteed to be at or after that timestamp. > > Signed-off-by: Christian Ebner > --- > proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++--- > 1 file changed, 13 insertions(+), 3 deletions(-) > > diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs > index 80889451..04803556 100644 > --- a/proxmox-rest-server/src/worker_task.rs > +++ b/proxmox-rest-server/src/worker_task.rs > @@ -264,7 +264,8 @@ impl WorkerTaskSetup { > } > > /// Identify the oldest finished task entry's timestamp by iterating the task > - /// archive and return the calculated cutoff time. > + /// archive, clamped to the earliest active task's start time, and return the > + /// calculated cutoff time. > /// > /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`]. > fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result, Error> { > @@ -273,11 +274,20 @@ impl WorkerTaskSetup { > let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?; > > if let Some(last_file) = logrotate.files().last() { > + let oldest_active = read_task_file_from_path(&self.active_tasks_fn)? > + .into_iter() > + .map(|info| info.upid.starttime) > + .min(); > + > let reader = BufReader::new(last_file); > for line in reader.lines() { > let line = line?; > if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { > - let timestamp = state.endtime(); > + let mut timestamp = state.endtime(); > + > + if let Some(oldest_active) = oldest_active { > + timestamp = timestamp.min(oldest_active); > + } > > let cutoff_time = if timestamp > 0 { > SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) > @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive( > } > > /// removes all task logs that are older than the oldest task entry in the > -/// task archive > +/// task archive and not currently active > pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { > let setup = worker_task_setup()?; > One edge case that I think we should double check: Let's say the last kernel clock tic was at 08:59:59.999 and a task is started. Suppose 1.2 ms have passed since the last tick and the UPID records the task start at 09:00:00. Then the log file would be created, however with the last cached clock tic which may be still at 08:59:59.999. Lets say the log content would stay as it is. Now we would have a situation where the logfile mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it would prematurely delete this log file. Was able to reproduce this with: use proxmox_rest_server::{WorkerTask, cleanup_old_tasks, init_worker_tasks, upid_log_path}; use proxmox_time::epoch_i64; #[tokio::test] async fn quiet_log_survives() -> anyhow::Result<()> { let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(), None)?; init_worker_tasks(dir.clone(), Default::default())?; // Aim for a log mtime just before the UPID's start second. let second = epoch_i64(); while epoch_i64() == second { std::hint::spin_loop(); } let (quiet, _logger) = WorkerTask::new("quiet", None, "root@pam".into(), false)?; tokio::time::sleep(Duration::from_secs(1)).await; let (done, logger) = WorkerTask::new("done", None, "root@pam".into(), false)?; proxmox_log::LogContext::new(logger).sync_scope(|| done.log_result(&Ok(()))); cleanup_old_tasks(true)?; let active = fs::read_to_string(dir.join("tasks/active"))?; let kept = upid_log_path(quiet.upid())?.try_exists()?; fs::remove_dir_all(dir)?; assert_eq!(active.trim_end(), quiet.to_string()); assert!(kept, "cleanup removed the active worker's log"); Ok(()) } Since we already read tasks/active we could maybe just skip their logs.