all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup
@ 2026-11-18 18:02 Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
  To: pbs-devel

When rotating the task log archive and cleaning up old task logs
(scheduled daily at midnight in Proxmox Backup Server), and exclusive
lock was previously held for the entire duration of the cleanup.

On systems with a large number of task logs or slow storage,
directory traversal during task log cleanup can easily exceed the
15 second lock acquisition timeout. Because state updates on any
worker task requires an exclusive lock on `.active.lock` to update the
active task list, tasks running concurrently around midnight failed
with:
```
Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
```

Since determining the cutoff timestamp only requires reading the
oldest entry from the task archive and is not required at all when
removing the log files for which mtime is older than the cutoff time,
acquire a shared lock only and drop it after the cutoff time has been
calculated. While at it, improve also the error message to make it
clear that a lock acquisition failed because of the timeout.

Link to the bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=8341

Further cleanups are suggested in https://bugzilla.proxmox.com/show_bug.cgi?id=5960
are not part of this series.


proxmox:

Christian Ebner (4):
  sys: fs: report lock timeouts as such instead of EINTR
  rest-server: use shared instead of exclusive lock for old task cleanup
  fix #8341: rest-server: do not lock task list during old log cleanup
  rest-server: clamp old task log cleanup cutoff to earliest active task

 proxmox-rest-server/src/worker_task.rs | 67 +++++++++++++++++---------
 proxmox-sys/src/fs/file.rs             |  3 ++
 2 files changed, 46 insertions(+), 24 deletions(-)


Summary over all repositories:
  2 files changed, 46 insertions(+), 24 deletions(-)

-- 
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
  2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
  2026-11-19 10:18   ` Robert Obkircher
  2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
  To: pbs-devel

Improves the error handling for file locking failing because of a
timeout. Since the timeout is implemented by a timer signal that
interrupts the blocking flock call, the propagated error message
states, e.g.:
```
Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
```
Instead, report that lock acquisition timed out after waiting for
the requested duration, which is more telling. Other errors kinds
remain unchanged.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-sys/src/fs/file.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
index 4b7cfa8a..9c5d471a 100644
--- a/proxmox-sys/src/fs/file.rs
+++ b/proxmox-sys/src/fs/file.rs
@@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
 
     match lock_file(&mut file, exclusive, Some(timeout)) {
         Ok(_) => Ok(file),
+        Err(err) if err.kind() == io::ErrorKind::Interrupted => {
+            bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
+        }
         Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
     }
 }
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup
  2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
  3 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
  To: pbs-devel

Cleanup of old task logs happens by identifying the oldest task entry
in the task archive. To protect against races by concurrent task
archive file updates, this is protected by a file lock.

Currently an exclusive lock is acquired, the oldest task cleanup
however only reads, never modifies the files so a shared lock is
sufficient. Since active worker updates and task log archive rotation
still acquire an exclusive lock, they cannot interfere.

Parts of the iteration via the task list info iterator might now
happen concurrently, but these cannot interfere since both read the
same task from the archive, which does not include old task logs to
be cleaned up and active worker updates during iteration still
require the exclusive lock.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index dfd0def6..d43c42ba 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -358,7 +358,7 @@ pub fn rotate_task_log_archive(
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-    let _lock = setup.lock_task_list_files(true)?;
+    let _lock = setup.lock_task_list_files(false)?;
 
     let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup
  2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
  2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
  3 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
  To: pbs-devel

Cleaning up old task logs held the task list lock while walking all
task log directories, checking every log file and removing the
outdated ones. With many task logs, potentially being stored on slow
storage, this can take well over the 15 second timeout used when
trying to acquire the lock by concurrent tasks.

Since every task state update requires that lock, tasks running
concurrently to the daily log rotation scheduled at midnight for PBS
failed with being unable to acquire the lock.

Only determining the cutoff time used for task log files mtime
comparison reads the task archive and requires the lock. The task log
files themselves are not protected by it.

Hold the shared lock only while computing the cutoff and perform the
old log cleanup without holding it, since logs of new tasks are
always newer than any cutoff time generated beforehand and concurrent
log rotation can only move the oldest archived entry forwards, never
backwards.

To make this more explicit, refactor the cutoff time gathering into
a dedicated method of WorkerTaskSetup, naturally limiting the lock
scope.

This will reduce lock contention during log rotation in PBS, being
scheduled at midnight, which can lead to concurrent tasks such as
backup jobs failing as reported next to the bugzilla issue several
times also in the community forum and via enterprise support.

Link: https://forum.proxmox.com/threads/155613/
Link: https://forum.proxmox.com/threads/183852/
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++-----------
 1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index d43c42ba..80889451 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -262,6 +262,37 @@ impl WorkerTaskSetup {
         })
         .map_err(|err: Error| format_err!("unable to create task log dir - {}", err))
     }
+
+    /// Identify the oldest finished task entry's timestamp by iterating the task
+    /// archive and return the calculated cutoff time.
+    ///
+    /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
+    fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
+        let _lock = self.lock_task_list_files(false)?;
+
+        let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
+
+        if let Some(last_file) = logrotate.files().last() {
+            let reader = BufReader::new(last_file);
+            for line in reader.lines() {
+                let line = line?;
+                if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
+                    let timestamp = state.endtime();
+
+                    let cutoff_time = if timestamp > 0 {
+                        SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
+                    } else {
+                        SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
+                    }
+                    .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
+
+                    return Ok(Some(cutoff_time));
+                }
+            }
+        }
+
+        Ok(None)
+    }
 }
 
 /// Initialize the WorkerTask library
@@ -358,34 +389,13 @@ pub fn rotate_task_log_archive(
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-    let _lock = setup.lock_task_list_files(false)?;
-
-    let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
-
-    let mut timestamp = None;
-    if let Some(last_file) = logrotate.files().last() {
-        let reader = BufReader::new(last_file);
-        for line in reader.lines() {
-            let line = line?;
-            if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
-                timestamp = Some(state.endtime());
-                break;
-            }
-        }
-    }
+    let cutoff_time = setup.oldest_task_cutoff_time(compressed)?;
 
     fn get_modified(entry: std::fs::DirEntry) -> Result<SystemTime, std::io::Error> {
         entry.metadata()?.modified()
     }
 
-    if let Some(timestamp) = timestamp {
-        let cutoff_time = if timestamp > 0 {
-            SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
-        } else {
-            SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
-        }
-        .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
-
+    if let Some(cutoff_time) = cutoff_time {
         for i in 0..256 {
             let path = setup.taskdir.join(format!("{i:02X}"));
             let files = match std::fs::read_dir(path) {
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
  2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
                   ` (2 preceding siblings ...)
  2026-11-18 18:02 ` [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
  2026-11-19 10:55   ` Robert Obkircher
  3 siblings, 1 reply; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
  To: pbs-devel

Determining the cutoff timestamp for old task log cleanup previously
only inspected the oldest entry in the task archive, which only tracks
finished tasks. Still running tasks are listed exclusively in the
active tasks file without an end time.

If a long-running task started before the calculated archive cutoff
time and has not written any new log lines since then, its log file's
mtime is older than the cutoff time and cleanup_old_tasks() would
remove the log file while the task is still active.

Prevent this by also reading the active task list while holding the
shared task list lock during cutoff time gathering and clamping the
cutoff timestamp to the earliest active task's start time. Since a
task's log file is created after its UPID start time is generated,
its mtime is guaranteed to be at or after that timestamp.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index 80889451..f1f27ec2 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -264,7 +264,8 @@ impl WorkerTaskSetup {
     }
 
     /// Identify the oldest finished task entry's timestamp by iterating the task
-    /// archive and return the calculated cutoff time.
+    /// archive, clamped to the earliest active task's start time, and  return the
+    /// calculated cutoff time.
     ///
     /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
     fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
@@ -277,7 +278,15 @@ impl WorkerTaskSetup {
             for line in reader.lines() {
                 let line = line?;
                 if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
-                    let timestamp = state.endtime();
+                    let mut timestamp = state.endtime();
+
+                    if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
+                        .into_iter()
+                        .map(|info| info.upid.starttime)
+                        .min()
+                    {
+                        timestamp = timestamp.min(oldest_active);
+                    }
 
                     let cutoff_time = if timestamp > 0 {
                         SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
@@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
 }
 
 /// removes all task logs that are older than the oldest task entry in the
-/// task archive
+/// task archive and not currently active
 pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
     let setup = worker_task_setup()?;
 
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
  2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
@ 2026-11-19 10:18   ` Robert Obkircher
  2026-11-19 10:53     ` Christian Ebner
  0 siblings, 1 reply; 9+ messages in thread
From: Robert Obkircher @ 2026-11-19 10:18 UTC (permalink / raw)
  To: Christian Ebner, pbs-devel


On 18.11.26 19:02, Christian Ebner wrote:
> Improves the error handling for file locking failing because of a
> timeout. Since the timeout is implemented by a timer signal that
> interrupts the blocking flock call, the propagated error message
> states, e.g.:
> ```
> Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
> ```
> Instead, report that lock acquisition timed out after waiting for
> the requested duration, which is more telling. Other errors kinds
> remain unchanged.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  proxmox-sys/src/fs/file.rs | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
> index 4b7cfa8a..9c5d471a 100644
> --- a/proxmox-sys/src/fs/file.rs
> +++ b/proxmox-sys/src/fs/file.rs
> @@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
>  
>      match lock_file(&mut file, exclusive, Some(timeout)) {
>          Ok(_) => Ok(file),
> +        Err(err) if err.kind() == io::ErrorKind::Interrupted => {
> +            bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
> +        }
The timeout value would be misleading if another signal interrupted
the call earlier than expected.

>          Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
>      }
>  }




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
  2026-11-19 10:18   ` Robert Obkircher
@ 2026-11-19 10:53     ` Christian Ebner
  0 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-19 10:53 UTC (permalink / raw)
  To: Robert Obkircher, pbs-devel

On 11/19/26 11:18 AM, Robert Obkircher wrote:
> 
> On 18.11.26 19:02, Christian Ebner wrote:
>> Improves the error handling for file locking failing because of a
>> timeout. Since the timeout is implemented by a timer signal that
>> interrupts the blocking flock call, the propagated error message
>> states, e.g.:
>> ```
>> Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
>> ```
>> Instead, report that lock acquisition timed out after waiting for
>> the requested duration, which is more telling. Other errors kinds
>> remain unchanged.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>>   proxmox-sys/src/fs/file.rs | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
>> index 4b7cfa8a..9c5d471a 100644
>> --- a/proxmox-sys/src/fs/file.rs
>> +++ b/proxmox-sys/src/fs/file.rs
>> @@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
>>   
>>       match lock_file(&mut file, exclusive, Some(timeout)) {
>>           Ok(_) => Ok(file),
>> +        Err(err) if err.kind() == io::ErrorKind::Interrupted => {
>> +            bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
>> +        }
> The timeout value would be misleading if another signal interrupted
> the call earlier than expected.

Good catch, let's drop this patch then as it was intended as additional 
cosmetic improvement.

> 
>>           Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
>>       }
>>   }





^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
  2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
@ 2026-11-19 10:55   ` Robert Obkircher
  2026-11-19 11:06     ` Christian Ebner
  0 siblings, 1 reply; 9+ messages in thread
From: Robert Obkircher @ 2026-11-19 10:55 UTC (permalink / raw)
  To: Christian Ebner, pbs-devel


On 18.11.26 19:02, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
>
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
>
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..f1f27ec2 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
>      }
>  
>      /// Identify the oldest finished task entry's timestamp by iterating the task
> -    /// archive and return the calculated cutoff time.
> +    /// archive, clamped to the earliest active task's start time, and  return the
> +    /// calculated cutoff time.
>      ///
>      /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
>      fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -277,7 +278,15 @@ impl WorkerTaskSetup {
>              for line in reader.lines() {
>                  let line = line?;
>                  if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> -                    let timestamp = state.endtime();
> +                    let mut timestamp = state.endtime();
> +
> +                    if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
Couldn't this be read once outside of the loops?
> +                        .into_iter()
> +                        .map(|info| info.upid.starttime)
> +                        .min()
> +                    {
> +                        timestamp = timestamp.min(oldest_active);
> +                    }
>  
>                      let cutoff_time = if timestamp > 0 {
>                          SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
>  }
>  
>  /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
>  pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
>      let setup = worker_task_setup()?;
>  




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
  2026-11-19 10:55   ` Robert Obkircher
@ 2026-11-19 11:06     ` Christian Ebner
  0 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-19 11:06 UTC (permalink / raw)
  To: Robert Obkircher, pbs-devel

On 11/19/26 11:55 AM, Robert Obkircher wrote:
> 
> On 18.11.26 19:02, Christian Ebner wrote:
>> Determining the cutoff timestamp for old task log cleanup previously
>> only inspected the oldest entry in the task archive, which only tracks
>> finished tasks. Still running tasks are listed exclusively in the
>> active tasks file without an end time.
>>
>> If a long-running task started before the calculated archive cutoff
>> time and has not written any new log lines since then, its log file's
>> mtime is older than the cutoff time and cleanup_old_tasks() would
>> remove the log file while the task is still active.
>>
>> Prevent this by also reading the active task list while holding the
>> shared task list lock during cutoff time gathering and clamping the
>> cutoff timestamp to the earliest active task's start time. Since a
>> task's log file is created after its UPID start time is generated,
>> its mtime is guaranteed to be at or after that timestamp.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>>   proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
>>   1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
>> index 80889451..f1f27ec2 100644
>> --- a/proxmox-rest-server/src/worker_task.rs
>> +++ b/proxmox-rest-server/src/worker_task.rs
>> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
>>       }
>>   
>>       /// Identify the oldest finished task entry's timestamp by iterating the task
>> -    /// archive and return the calculated cutoff time.
>> +    /// archive, clamped to the earliest active task's start time, and  return the
>> +    /// calculated cutoff time.
>>       ///
>>       /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
>>       fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
>> @@ -277,7 +278,15 @@ impl WorkerTaskSetup {
>>               for line in reader.lines() {
>>                   let line = line?;
>>                   if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
>> -                    let timestamp = state.endtime();
>> +                    let mut timestamp = state.endtime();
>> +
>> +                    if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
> Couldn't this be read once outside of the loops?

Yeah, better to gather the latest active right after locking. Will send 
a v2 dropping patch 1 and pulling this out of  the loop, thanks.

>> +                        .into_iter()
>> +                        .map(|info| info.upid.starttime)
>> +                        .min()
>> +                    {
>> +                        timestamp = timestamp.min(oldest_active);
>> +                    }
>>   
>>                       let cutoff_time = if timestamp > 0 {
>>                           SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
>> @@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
>>   }
>>   
>>   /// removes all task logs that are older than the oldest task entry in the
>> -/// task archive
>> +/// task archive and not currently active
>>   pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
>>       let setup = worker_task_setup()?;
>>   





^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-19 11:06 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
2026-11-19 10:18   ` Robert Obkircher
2026-11-19 10:53     ` Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-19 10:55   ` Robert Obkircher
2026-11-19 11:06     ` Christian Ebner

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal