* [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup
@ 2026-11-20 8:27 Christian Ebner
2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Christian Ebner @ 2026-11-20 8:27 UTC (permalink / raw)
To: pbs-devel
When rotating the task log archive and cleaning up old task logs
(scheduled daily at midnight in Proxmox Backup Server), and exclusive
lock was previously held for the entire duration of the cleanup.
On systems with a large number of task logs or slow storage,
directory traversal during task log cleanup can easily exceed the
15 second lock acquisition timeout. Because state updates on any
worker task requires an exclusive lock on `.active.lock` to update the
active task list, tasks running concurrently around midnight failed
with:
```
Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
```
Since determining the cutoff timestamp only requires reading the
oldest entry from the task archive and is not required at all when
removing the log files for which mtime is older than the cutoff time,
acquire a shared lock only and drop it after the cutoff time has been
calculated.
Link to the bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=8341
Further cleanups are suggested in https://bugzilla.proxmox.com/show_bug.cgi?id=5960
are not part of this series.
Changes since version 1 (thanks a lot @Robert for feedback):
- Dropped patch changing error message since the signal interrupting the flock
call can be unrelated to the timeout.
- Pull the latest active task gathering out of the loop which checks the
finished tasks end time. This was needless to be repeated for each finished
task as the active task list is still locked from being updated anyways.
proxmox:
Christian Ebner (3):
rest-server: use shared instead of exclusive lock for old task cleanup
fix #8341: rest-server: do not lock task list during old log cleanup
rest-server: clamp old task log cleanup cutoff to earliest active task
proxmox-rest-server/src/worker_task.rs | 68 +++++++++++++++++---------
1 file changed, 44 insertions(+), 24 deletions(-)
Summary over all repositories:
1 files changed, 44 insertions(+), 24 deletions(-)
--
Generated by murpp 0.11.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup 2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner @ 2026-11-20 8:27 ` Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner 2 siblings, 0 replies; 5+ messages in thread From: Christian Ebner @ 2026-11-20 8:27 UTC (permalink / raw) To: pbs-devel Cleanup of old task logs happens by identifying the oldest task entry in the task archive. To protect against races by concurrent task archive file updates, this is protected by a file lock. Currently an exclusive lock is acquired, the oldest task cleanup however only reads, never modifies the files so a shared lock is sufficient. Since active worker updates and task log archive rotation still acquire an exclusive lock, they cannot interfere. Parts of the iteration via the task list info iterator might now happen concurrently, but these cannot interfere since both read the same task from the archive, which does not include old task logs to be cleaned up and active worker updates during iteration still require the exclusive lock. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- proxmox-rest-server/src/worker_task.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs index dfd0def6..d43c42ba 100644 --- a/proxmox-rest-server/src/worker_task.rs +++ b/proxmox-rest-server/src/worker_task.rs @@ -358,7 +358,7 @@ pub fn rotate_task_log_archive( pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { let setup = worker_task_setup()?; - let _lock = setup.lock_task_list_files(true)?; + let _lock = setup.lock_task_list_files(false)?; let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?; -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup 2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner @ 2026-11-20 8:27 ` Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner 2 siblings, 0 replies; 5+ messages in thread From: Christian Ebner @ 2026-11-20 8:27 UTC (permalink / raw) To: pbs-devel Cleaning up old task logs held the task list lock while walking all task log directories, checking every log file and removing the outdated ones. With many task logs, potentially being stored on slow storage, this can take well over the 15 second timeout used when trying to acquire the lock by concurrent tasks. Since every task state update requires that lock, tasks running concurrently to the daily log rotation scheduled at midnight for PBS failed with being unable to acquire the lock. Only determining the cutoff time used for task log files mtime comparison reads the task archive and requires the lock. The task log files themselves are not protected by it. Hold the shared lock only while computing the cutoff and perform the old log cleanup without holding it, since logs of new tasks are always newer than any cutoff time generated beforehand and concurrent log rotation can only move the oldest archived entry forwards, never backwards. To make this more explicit, refactor the cutoff time gathering into a dedicated method of WorkerTaskSetup, naturally limiting the lock scope. This will reduce lock contention during log rotation in PBS, being scheduled at midnight, which can lead to concurrent tasks such as backup jobs failing as reported next to the bugzilla issue several times also in the community forum and via enterprise support. Link: https://forum.proxmox.com/threads/155613/ Link: https://forum.proxmox.com/threads/183852/ Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341 Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++----------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs index d43c42ba..80889451 100644 --- a/proxmox-rest-server/src/worker_task.rs +++ b/proxmox-rest-server/src/worker_task.rs @@ -262,6 +262,37 @@ impl WorkerTaskSetup { }) .map_err(|err: Error| format_err!("unable to create task log dir - {}", err)) } + + /// Identify the oldest finished task entry's timestamp by iterating the task + /// archive and return the calculated cutoff time. + /// + /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`]. + fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> { + let _lock = self.lock_task_list_files(false)?; + + let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?; + + if let Some(last_file) = logrotate.files().last() { + let reader = BufReader::new(last_file); + for line in reader.lines() { + let line = line?; + if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { + let timestamp = state.endtime(); + + let cutoff_time = if timestamp > 0 { + SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) + } else { + SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64)) + } + .ok_or_else(|| format_err!("could not calculate cutoff time"))?; + + return Ok(Some(cutoff_time)); + } + } + } + + Ok(None) + } } /// Initialize the WorkerTask library @@ -358,34 +389,13 @@ pub fn rotate_task_log_archive( pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { let setup = worker_task_setup()?; - let _lock = setup.lock_task_list_files(false)?; - - let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?; - - let mut timestamp = None; - if let Some(last_file) = logrotate.files().last() { - let reader = BufReader::new(last_file); - for line in reader.lines() { - let line = line?; - if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { - timestamp = Some(state.endtime()); - break; - } - } - } + let cutoff_time = setup.oldest_task_cutoff_time(compressed)?; fn get_modified(entry: std::fs::DirEntry) -> Result<SystemTime, std::io::Error> { entry.metadata()?.modified() } - if let Some(timestamp) = timestamp { - let cutoff_time = if timestamp > 0 { - SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) - } else { - SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64)) - } - .ok_or_else(|| format_err!("could not calculate cutoff time"))?; - + if let Some(cutoff_time) = cutoff_time { for i in 0..256 { let path = setup.taskdir.join(format!("{i:02X}")); let files = match std::fs::read_dir(path) { -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task 2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner @ 2026-11-20 8:27 ` Christian Ebner 2026-11-20 13:51 ` Samuel Rufinatscha 2 siblings, 1 reply; 5+ messages in thread From: Christian Ebner @ 2026-11-20 8:27 UTC (permalink / raw) To: pbs-devel Determining the cutoff timestamp for old task log cleanup previously only inspected the oldest entry in the task archive, which only tracks finished tasks. Still running tasks are listed exclusively in the active tasks file without an end time. If a long-running task started before the calculated archive cutoff time and has not written any new log lines since then, its log file's mtime is older than the cutoff time and cleanup_old_tasks() would remove the log file while the task is still active. Prevent this by also reading the active task list while holding the shared task list lock during cutoff time gathering and clamping the cutoff timestamp to the earliest active task's start time. Since a task's log file is created after its UPID start time is generated, its mtime is guaranteed to be at or after that timestamp. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs index 80889451..04803556 100644 --- a/proxmox-rest-server/src/worker_task.rs +++ b/proxmox-rest-server/src/worker_task.rs @@ -264,7 +264,8 @@ impl WorkerTaskSetup { } /// Identify the oldest finished task entry's timestamp by iterating the task - /// archive and return the calculated cutoff time. + /// archive, clamped to the earliest active task's start time, and return the + /// calculated cutoff time. /// /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`]. fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> { @@ -273,11 +274,20 @@ impl WorkerTaskSetup { let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?; if let Some(last_file) = logrotate.files().last() { + let oldest_active = read_task_file_from_path(&self.active_tasks_fn)? + .into_iter() + .map(|info| info.upid.starttime) + .min(); + let reader = BufReader::new(last_file); for line in reader.lines() { let line = line?; if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { - let timestamp = state.endtime(); + let mut timestamp = state.endtime(); + + if let Some(oldest_active) = oldest_active { + timestamp = timestamp.min(oldest_active); + } let cutoff_time = if timestamp > 0 { SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive( } /// removes all task logs that are older than the oldest task entry in the -/// task archive +/// task archive and not currently active pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { let setup = worker_task_setup()?; -- 2.47.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task 2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner @ 2026-11-20 13:51 ` Samuel Rufinatscha 0 siblings, 0 replies; 5+ messages in thread From: Samuel Rufinatscha @ 2026-11-20 13:51 UTC (permalink / raw) To: Christian Ebner, pbs-devel On 11/20/26 9:27 AM, Christian Ebner wrote: > Determining the cutoff timestamp for old task log cleanup previously > only inspected the oldest entry in the task archive, which only tracks > finished tasks. Still running tasks are listed exclusively in the > active tasks file without an end time. > > If a long-running task started before the calculated archive cutoff > time and has not written any new log lines since then, its log file's > mtime is older than the cutoff time and cleanup_old_tasks() would > remove the log file while the task is still active. > > Prevent this by also reading the active task list while holding the > shared task list lock during cutoff time gathering and clamping the > cutoff timestamp to the earliest active task's start time. Since a > task's log file is created after its UPID start time is generated, > its mtime is guaranteed to be at or after that timestamp. > > Signed-off-by: Christian Ebner <c.ebner@proxmox.com> > --- > proxmox-rest-server/src/worker_task.rs | 16 +++++++++++++--- > 1 file changed, 13 insertions(+), 3 deletions(-) > > diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs > index 80889451..04803556 100644 > --- a/proxmox-rest-server/src/worker_task.rs > +++ b/proxmox-rest-server/src/worker_task.rs > @@ -264,7 +264,8 @@ impl WorkerTaskSetup { > } > > /// Identify the oldest finished task entry's timestamp by iterating the task > - /// archive and return the calculated cutoff time. > + /// archive, clamped to the earliest active task's start time, and return the > + /// calculated cutoff time. > /// > /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`]. > fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> { > @@ -273,11 +274,20 @@ impl WorkerTaskSetup { > let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?; > > if let Some(last_file) = logrotate.files().last() { > + let oldest_active = read_task_file_from_path(&self.active_tasks_fn)? > + .into_iter() > + .map(|info| info.upid.starttime) > + .min(); > + > let reader = BufReader::new(last_file); > for line in reader.lines() { > let line = line?; > if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { > - let timestamp = state.endtime(); > + let mut timestamp = state.endtime(); > + > + if let Some(oldest_active) = oldest_active { > + timestamp = timestamp.min(oldest_active); > + } > > let cutoff_time = if timestamp > 0 { > SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) > @@ -385,7 +395,7 @@ pub fn rotate_task_log_archive( > } > > /// removes all task logs that are older than the oldest task entry in the > -/// task archive > +/// task archive and not currently active > pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { > let setup = worker_task_setup()?; > One edge case that I think we should double check: Let's say the last kernel clock tic was at 08:59:59.999 and a task is started. Suppose 1.2 ms have passed since the last tick and the UPID records the task start at 09:00:00. Then the log file would be created, however with the last cached clock tic which may be still at 08:59:59.999. Lets say the log content would stay as it is. Now we would have a situation where the logfile mtime < worker start time. If cleanup would use 09:00:00 as cutoff, it would prematurely delete this log file. Was able to reproduce this with: use proxmox_rest_server::{WorkerTask, cleanup_old_tasks, init_worker_tasks, upid_log_path}; use proxmox_time::epoch_i64; #[tokio::test] async fn quiet_log_survives() -> anyhow::Result<()> { let dir = proxmox_sys::fs::make_tmp_dir(std::env::temp_dir(), None)?; init_worker_tasks(dir.clone(), Default::default())?; // Aim for a log mtime just before the UPID's start second. let second = epoch_i64(); while epoch_i64() == second { std::hint::spin_loop(); } let (quiet, _logger) = WorkerTask::new("quiet", None, "root@pam".into(), false)?; tokio::time::sleep(Duration::from_secs(1)).await; let (done, logger) = WorkerTask::new("done", None, "root@pam".into(), false)?; proxmox_log::LogContext::new(logger).sync_scope(|| done.log_result(&Ok(()))); cleanup_old_tasks(true)?; let active = fs::read_to_string(dir.join("tasks/active"))?; let kept = upid_log_path(quiet.upid())?.try_exists()?; fs::remove_dir_all(dir)?; assert_eq!(active.trim_end(), quiet.to_string()); assert!(kept, "cleanup removed the active worker's log"); Ok(()) } Since we already read tasks/active we could maybe just skip their logs. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-20 13:51 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-20 8:27 [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 1/3] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 2/3] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner 2026-11-20 8:27 ` [PATCH proxmox v2 3/3] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner 2026-11-20 13:51 ` Samuel Rufinatscha
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.