all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup
@ 2026-11-20  8:27 Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Christian Ebner @ 2026-11-20  8:27 UTC (permalink / raw)
  To: pbs-devel

When rotating the task log archive and cleaning up old task logs
(scheduled daily at midnight in Proxmox Backup Server), and exclusive
lock was previously held for the entire duration of the cleanup.

On systems with a large number of task logs or slow storage,
directory traversal during task log cleanup can easily exceed the
15 second lock acquisition timeout. Because state updates on any
worker task requires an exclusive lock on `.active.lock` to update the
active task list, tasks running concurrently around midnight failed
with:
```
Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
```

Since determining the cutoff timestamp only requires reading the
oldest entry from the task archive and is not required at all when
removing the log files for which mtime is older than the cutoff time,
acquire a shared lock only and drop it after the cutoff time has been
calculated.

Link to the bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=8341

Further cleanups are suggested in https://bugzilla.proxmox.com/show_bug.cgi?id=5960
are not part of this series.

Changes since version 1 (thanks a lot @Robert for feedback):
- Dropped patch changing error message since the signal interrupting the flock
  call can be unrelated to the timeout.
- Pull the latest active task gathering out of the loop which checks the
  finished tasks end time. This was needless to be repeated for each finished
  task as the active task list is still locked from being updated anyways.


proxmox:

Christian Ebner (3):
  rest-server: use shared instead of exclusive lock for old task cleanup
  fix #8341: rest-server: do not lock task list during old log cleanup
  rest-server: clamp old task log cleanup cutoff to earliest active task

 proxmox-rest-server/src/worker_task.rs | 68 +++++++++++++++++---------
 1 file changed, 44 insertions(+), 24 deletions(-)


Summary over all repositories:
  1 files changed, 44 insertions(+), 24 deletions(-)

-- 
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup
  2026-11-20  8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
@ 2026-11-20  8:27 ` Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
  2 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2026-11-20  8:27 UTC (permalink / raw)
  To: pbs-devel

Cleanup of old task logs happens by identifying the oldest task entry
in the task archive. To protect against races by concurrent task
archive file updates, this is protected by a file lock.

Currently an exclusive lock is acquired, the oldest task cleanup
however only reads, never modifies the files so a shared lock is
sufficient. Since active worker updates and task log archive rotation
still acquire an exclusive lock, they cannot interfere.

Parts of the iteration via the task list info iterator might now
happen concurrently, but these cannot interfere since both read the
same task from the archive, which does not include old task logs to
be cleaned up and active worker updates during iteration still
require the exclusive lock.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index dfd0def6..d43c42ba 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -358,7 +358,7 @@ pub fn rotate_task_log_archive(
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-    let _lock = setup.lock_task_list_files(true)?;
+    let _lock = setup.lock_task_list_files(false)?;
 
     let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup
  2026-11-20  8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
@ 2026-11-20  8:27 ` Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
  2 siblings, 0 replies; 5+ messages in thread
From: Christian Ebner @ 2026-11-20  8:27 UTC (permalink / raw)
  To: pbs-devel

Cleaning up old task logs held the task list lock while walking all
task log directories, checking every log file and removing the
outdated ones. With many task logs, potentially being stored on slow
storage, this can take well over the 15 second timeout used when
trying to acquire the lock by concurrent tasks.

Since every task state update requires that lock, tasks running
concurrently to the daily log rotation scheduled at midnight for PBS
failed with being unable to acquire the lock.

Only determining the cutoff time used for task log files mtime
comparison reads the task archive and requires the lock. The task log
files themselves are not protected by it.

Hold the shared lock only while computing the cutoff and perform the
old log cleanup without holding it, since logs of new tasks are
always newer than any cutoff time generated beforehand and concurrent
log rotation can only move the oldest archived entry forwards, never
backwards.

To make this more explicit, refactor the cutoff time gathering into
a dedicated method of WorkerTaskSetup, naturally limiting the lock
scope.

This will reduce lock contention during log rotation in PBS, being
scheduled at midnight, which can lead to concurrent tasks such as
backup jobs failing as reported next to the bugzilla issue several
times also in the community forum and via enterprise support.

Link: https://forum.proxmox.com/threads/155613/
Link: https://forum.proxmox.com/threads/183852/
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++-----------
 1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index d43c42ba..80889451 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -262,6 +262,37 @@ impl WorkerTaskSetup {
         })
         .map_err(|err: Error| format_err!("unable to create task log dir - {}", err))
     }
+
+    /// Identify the oldest finished task entry's timestamp by iterating the task
+    /// archive and return the calculated cutoff time.
+    ///
+    /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
+    fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
+        let _lock = self.lock_task_list_files(false)?;
+
+        let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
+
+        if let Some(last_file) = logrotate.files().last() {
+            let reader = BufReader::new(last_file);
+            for line in reader.lines() {
+                let line = line?;
+                if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
+                    let timestamp = state.endtime();
+
+                    let cutoff_time = if timestamp > 0 {
+                        SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
+                    } else {
+                        SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
+                    }
+                    .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
+
+                    return Ok(Some(cutoff_time));
+                }
+            }
+        }
+
+        Ok(None)
+    }
 }
 
 /// Initialize the WorkerTask library
@@ -358,34 +389,13 @@ pub fn rotate_task_log_archive(
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-    let _lock = setup.lock_task_list_files(false)?;
-
-    let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
-
-    let mut timestamp = None;
-    if let Some(last_file) = logrotate.files().last() {
-        let reader = BufReader::new(last_file);
-        for line in reader.lines() {
-            let line = line?;
-            if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
-                timestamp = Some(state.endtime());
-                break;
-            }
-        }
-    }
+    let cutoff_time = setup.oldest_task_cutoff_time(compressed)?;
 
     fn get_modified(entry: std::fs::DirEntry) -> Result<SystemTime, std::io::Error> {
         entry.metadata()?.modified()
     }
 
-    if let Some(timestamp) = timestamp {
-        let cutoff_time = if timestamp > 0 {
-            SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
-        } else {
-            SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
-        }
-        .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
-
+    if let Some(cutoff_time) = cutoff_time {
         for i in 0..256 {
             let path = setup.taskdir.join(format!("{i:02X}"));
             let files = match std::fs::read_dir(path) {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task
  2026-11-20  8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
  2026-11-20  8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
@ 2026-11-20  8:27 ` Christian Ebner
  2026-11-20 13:51   ` Samuel Rufinatscha
  2 siblings, 1 reply; 5+ messages in thread
From: Christian Ebner @ 2026-11-20  8:27 UTC (permalink / raw)
  To: pbs-devel

Determining the cutoff timestamp for old task log cleanup previously
only inspected the oldest entry in the task archive, which only tracks
finished tasks. Still running tasks are listed exclusively in the
active tasks file without an end time.

If a long-running task started before the calculated archive cutoff
time and has not written any new log lines since then, its log file's
mtime is older than the cutoff time and cleanup_old_tasks() would
remove the log file while the task is still active.

Prevent this by also reading the active task list while holding the
shared task list lock during cutoff time gathering and clamping the
cutoff timestamp to the earliest active task's start time. Since a
task's log file is created after its UPID start time is generated,
its mtime is guaranteed to be at or after that timestamp.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index 80889451..04803556 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -264,7 +264,8 @@ impl WorkerTaskSetup {
     }
 
     /// Identify the oldest finished task entry's timestamp by iterating the task
-    /// archive and return the calculated cutoff time.
+    /// archive, clamped to the earliest active task's start time, and  return the
+    /// calculated cutoff time.
     ///
     /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
     fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
@@ -273,11 +274,20 @@ impl WorkerTaskSetup {
         let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
 
         if let Some(last_file) = logrotate.files().last() {
+            let oldest_active = read_task_file_from_path(&self.active_tasks_fn)?
+                .into_iter()
+                .map(|info| info.upid.starttime)
+                .min();
+
             let reader = BufReader::new(last_file);
             for line in reader.lines() {
                 let line = line?;
                 if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
-                    let timestamp = state.endtime();
+                    let mut timestamp = state.endtime();
+
+                    if let Some(oldest_active) = oldest_active {
+                        timestamp = timestamp.min(oldest_active);
+                    }
 
                     let cutoff_time = if timestamp > 0 {
                         SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
@@ -385,7 +395,7 @@ pub fn rotate_task_log_archive(
 }
 
 /// removes all task logs that are older than the oldest task entry in the
-/// task archive
+/// task archive and not currently active
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task
  2026-11-20  8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
@ 2026-11-20 13:51   ` Samuel Rufinatscha
  0 siblings, 0 replies; 5+ messages in thread
From: Samuel Rufinatscha @ 2026-11-20 13:51 UTC (permalink / raw)
  To: Christian Ebner, pbs-devel

On 11/20/26 9:27 AM, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
> 
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
> 
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>   proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++---
>   1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..04803556 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
>       }
>   
>       /// Identify the oldest finished task entry's timestamp by iterating the task
> -    /// archive and return the calculated cutoff time.
> +    /// archive, clamped to the earliest active task's start time, and  return the
> +    /// calculated cutoff time.
>       ///
>       /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
>       fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -273,11 +274,20 @@ impl WorkerTaskSetup {
>           let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
>   
>           if let Some(last_file) = logrotate.files().last() {
> +            let oldest_active = read_task_file_from_path(&self.active_tasks_fn)?
> +                .into_iter()
> +                .map(|info| info.upid.starttime)
> +                .min();
> +
>               let reader = BufReader::new(last_file);
>               for line in reader.lines() {
>                   let line = line?;
>                   if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> -                    let timestamp = state.endtime();
> +                    let mut timestamp = state.endtime();
> +
> +                    if let Some(oldest_active) = oldest_active {
> +                        timestamp = timestamp.min(oldest_active);
> +                    }
>   
>                       let cutoff_time = if timestamp > 0 {
>                           SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive(
>   }
>   
>   /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
>   pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
>       let setup = worker_task_setup()?;
>   

One edge case that I think we should double check:
Let's say the last kernel clock tic was at 08:59:59.999 and a task is
started. Suppose 1.2 ms have passed since the last tick and the UPID
records the task start at 09:00:00.
Then the log file would be created, however with the last cached clock
tic which may be still at 08:59:59.999. Lets say the log content would
stay as it is. Now we would have a situation where the logfile
mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it
would prematurely delete this log file.

Was able to reproduce this with:

     use proxmox_rest_server::{WorkerTask, cleanup_old_tasks, 
init_worker_tasks, upid_log_path};
     use proxmox_time::epoch_i64;

     #[tokio::test]
     async fn quiet_log_survives() -> anyhow::Result<()> {
         let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(), 
None)?;
         init_worker_tasks(dir.clone(), Default::default())?;

         // Aim for a log mtime just before the UPID's start second.
         let second = epoch_i64();
         while epoch_i64() == second {
             std::hint::spin_loop();
         }
         let (quiet, _logger) = WorkerTask::new("quiet", None, 
"root@pam".into(), false)?;

         tokio::time::sleep(Duration::from_secs(1)).await;
         let (done, logger) = WorkerTask::new("done", None, 
"root@pam".into(), false)?;
         proxmox_log::LogContext::new(logger).sync_scope(|| 
done.log_result(&Ok(())));
         cleanup_old_tasks(true)?;

         let active = fs::read_to_string(dir.join("tasks/active"))?;
         let kept = upid_log_path(quiet.upid())?.try_exists()?;
         fs::remove_dir_all(dir)?;
         assert_eq!(active.trim_end(), quiet.to_string());
         assert!(kept, "cleanup removed the active worker's log");
         Ok(())
     }

Since we already read tasks/active we could maybe just skip their logs.




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-20 13:51 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-20  8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-20 13:51   ` Samuel Rufinatscha

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal