From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task
Date: Fri, 20 Nov 2026 14:51:18 +0100 [thread overview]
Message-ID: <0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com> (raw)
In-Reply-To: <20261120082710.65728-4-c.ebner@proxmox.com>
On 11/20/26 9:27 AM, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
>
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
>
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..04803556 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
> }
>
> /// Identify the oldest finished task entry's timestamp by iterating the task
> - /// archive and return the calculated cutoff time.
> + /// archive, clamped to the earliest active task's start time, and return the
> + /// calculated cutoff time.
> ///
> /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
> fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -273,11 +274,20 @@ impl WorkerTaskSetup {
> let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
>
> if let Some(last_file) = logrotate.files().last() {
> + let oldest_active = read_task_file_from_path(&self.active_tasks_fn)?
> + .into_iter()
> + .map(|info| info.upid.starttime)
> + .min();
> +
> let reader = BufReader::new(last_file);
> for line in reader.lines() {
> let line = line?;
> if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> - let timestamp = state.endtime();
> + let mut timestamp = state.endtime();
> +
> + if let Some(oldest_active) = oldest_active {
> + timestamp = timestamp.min(oldest_active);
> + }
>
> let cutoff_time = if timestamp > 0 {
> SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive(
> }
>
> /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
> pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
> let setup = worker_task_setup()?;
>
One edge case that I think we should double check:
Let's say the last kernel clock tic was at 08:59:59.999 and a task is
started. Suppose 1.2 ms have passed since the last tick and the UPID
records the task start at 09:00:00.
Then the log file would be created, however with the last cached clock
tic which may be still at 08:59:59.999. Lets say the log content would
stay as it is. Now we would have a situation where the logfile
mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it
would prematurely delete this log file.
Was able to reproduce this with:
use proxmox_rest_server::{WorkerTask, cleanup_old_tasks,
init_worker_tasks, upid_log_path};
use proxmox_time::epoch_i64;
#[tokio::test]
async fn quiet_log_survives() -> anyhow::Result<()> {
let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(),
None)?;
init_worker_tasks(dir.clone(), Default::default())?;
// Aim for a log mtime just before the UPID's start second.
let second = epoch_i64();
while epoch_i64() == second {
std::hint::spin_loop();
}
let (quiet, _logger) = WorkerTask::new("quiet", None,
"root@pam".into(), false)?;
tokio::time::sleep(Duration::from_secs(1)).await;
let (done, logger) = WorkerTask::new("done", None,
"root@pam".into(), false)?;
proxmox_log::LogContext::new(logger).sync_scope(||
done.log_result(&Ok(())));
cleanup_old_tasks(true)?;
let active = fs::read_to_string(dir.join("tasks/active"))?;
let kept = upid_log_path(quiet.upid())?.try_exists()?;
fs::remove_dir_all(dir)?;
assert_eq!(active.trim_end(), quiet.to_string());
assert!(kept, "cleanup removed the active worker's log");
Ok(())
}
Since we already read tasks/active we could maybe just skip their logs.
prev parent reply other threads:[~2026-11-20 13:51 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-20 13:51 ` Samuel Rufinatscha [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com \
--to=s.rufinatscha@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox