From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task
Date: Fri, 20 Nov 2026 14:51:18 +0100 [thread overview]
Message-ID: <0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com> (raw)
In-Reply-To: <20261120082710.65728-4-c.ebner@proxmox.com>
On 11/20/26 9:27 AM, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
>
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
>
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..04803556 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
> }
>
> /// Identify the oldest finished task entry's timestamp by iterating the task
> - /// archive and return the calculated cutoff time.
> + /// archive, clamped to the earliest active task's start time, and return the
> + /// calculated cutoff time.
> ///
> /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
> fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -273,11 +274,20 @@ impl WorkerTaskSetup {
> let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
>
> if let Some(last_file) = logrotate.files().last() {
> + let oldest_active = read_task_file_from_path(&self.active_tasks_fn)?
> + .into_iter()
> + .map(|info| info.upid.starttime)
> + .min();
> +
> let reader = BufReader::new(last_file);
> for line in reader.lines() {
> let line = line?;
> if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> - let timestamp = state.endtime();
> + let mut timestamp = state.endtime();
> +
> + if let Some(oldest_active) = oldest_active {
> + timestamp = timestamp.min(oldest_active);
> + }
>
> let cutoff_time = if timestamp > 0 {
> SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive(
> }
>
> /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
> pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
> let setup = worker_task_setup()?;
>
One edge case that I think we should double check:
Let's say the last kernel clock tic was at 08:59:59.999 and a task is
started. Suppose 1.2 ms have passed since the last tick and the UPID
records the task start at 09:00:00.
Then the log file would be created, however with the last cached clock
tic which may be still at 08:59:59.999. Lets say the log content would
stay as it is. Now we would have a situation where the logfile
mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it
would prematurely delete this log file.
Was able to reproduce this with:
use proxmox_rest_server::{WorkerTask, cleanup_old_tasks,
init_worker_tasks, upid_log_path};
use proxmox_time::epoch_i64;
#[tokio::test]
async fn quiet_log_survives() -> anyhow::Result<()> {
let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(),
None)?;
init_worker_tasks(dir.clone(), Default::default())?;
// Aim for a log mtime just before the UPID's start second.
let second = epoch_i64();
while epoch_i64() == second {
std::hint::spin_loop();
}
let (quiet, _logger) = WorkerTask::new("quiet", None,
"root@pam".into(), false)?;
tokio::time::sleep(Duration::from_secs(1)).await;
let (done, logger) = WorkerTask::new("done", None,
"root@pam".into(), false)?;
proxmox_log::LogContext::new(logger).sync_scope(||
done.log_result(&Ok(())));
cleanup_old_tasks(true)?;
let active = fs::read_to_string(dir.join("tasks/active"))?;
let kept = upid_log_path(quiet.upid())?.try_exists()?;
fs::remove_dir_all(dir)?;
assert_eq!(active.trim_end(), quiet.to_string());
assert!(kept, "cleanup removed the active worker's log");
Ok(())
}
Since we already read tasks/active we could maybe just skip their logs.
prev parent reply other threads:[~2026-11-20 13:51 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-20 13:51 ` Samuel Rufinatscha [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com \
--to=s.rufinatscha@proxmox.com \
--cc=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.