public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>, pbs-devel@lists.proxmox.com
Subject: Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task
Date: Fri, 20 Nov 2026 14:51:18 +0100	[thread overview]
Message-ID: <0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com> (raw)
In-Reply-To: <20261120082710.65728-4-c.ebner@proxmox.com>

On 11/20/26 9:27 AM, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
> 
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
> 
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>   proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++---
>   1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..04803556 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
>       }
>   
>       /// Identify the oldest finished task entry's timestamp by iterating the task
> -    /// archive and return the calculated cutoff time.
> +    /// archive, clamped to the earliest active task's start time, and  return the
> +    /// calculated cutoff time.
>       ///
>       /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
>       fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -273,11 +274,20 @@ impl WorkerTaskSetup {
>           let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
>   
>           if let Some(last_file) = logrotate.files().last() {
> +            let oldest_active = read_task_file_from_path(&self.active_tasks_fn)?
> +                .into_iter()
> +                .map(|info| info.upid.starttime)
> +                .min();
> +
>               let reader = BufReader::new(last_file);
>               for line in reader.lines() {
>                   let line = line?;
>                   if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> -                    let timestamp = state.endtime();
> +                    let mut timestamp = state.endtime();
> +
> +                    if let Some(oldest_active) = oldest_active {
> +                        timestamp = timestamp.min(oldest_active);
> +                    }
>   
>                       let cutoff_time = if timestamp > 0 {
>                           SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive(
>   }
>   
>   /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
>   pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
>       let setup = worker_task_setup()?;
>   

One edge case that I think we should double check:
Let's say the last kernel clock tic was at 08:59:59.999 and a task is
started. Suppose 1.2 ms have passed since the last tick and the UPID
records the task start at 09:00:00.
Then the log file would be created, however with the last cached clock
tic which may be still at 08:59:59.999. Lets say the log content would
stay as it is. Now we would have a situation where the logfile
mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it
would prematurely delete this log file.

Was able to reproduce this with:

     use proxmox_rest_server::{WorkerTask, cleanup_old_tasks, 
init_worker_tasks, upid_log_path};
     use proxmox_time::epoch_i64;

     #[tokio::test]
     async fn quiet_log_survives() -> anyhow::Result<()> {
         let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(), 
None)?;
         init_worker_tasks(dir.clone(), Default::default())?;

         // Aim for a log mtime just before the UPID's start second.
         let second = epoch_i64();
         while epoch_i64() == second {
             std::hint::spin_loop();
         }
         let (quiet, _logger) = WorkerTask::new("quiet", None, 
"root@pam".into(), false)?;

         tokio::time::sleep(Duration::from_secs(1)).await;
         let (done, logger) = WorkerTask::new("done", None, 
"root@pam".into(), false)?;
         proxmox_log::LogContext::new(logger).sync_scope(|| 
done.log_result(&Ok(())));
         cleanup_old_tasks(true)?;

         let active = fs::read_to_string(dir.join("tasks/active"))?;
         let kept = upid_log_path(quiet.upid())?.try_exists()?;
         fs::remove_dir_all(dir)?;
         assert_eq!(active.trim_end(), quiet.to_string());
         assert!(kept, "cleanup removed the active worker's log");
         Ok(())
     }

Since we already read tasks/active we could maybe just skip their logs.




      reply	other threads:[~2026-11-20 13:51 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20  8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-20 13:51   ` Samuel Rufinatscha [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0a828cbf-9e9d-427a-9a96-fc16a315b205@proxmox.com \
    --to=s.rufinatscha@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal