* [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
2026-11-19 10:18 ` Robert Obkircher
2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
To: pbs-devel
Improves the error handling for file locking failing because of a
timeout. Since the timeout is implemented by a timer signal that
interrupts the blocking flock call, the propagated error message
states, e.g.:
```
Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
```
Instead, report that lock acquisition timed out after waiting for
the requested duration, which is more telling. Other errors kinds
remain unchanged.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-sys/src/fs/file.rs | 3 +++
1 file changed, 3 insertions(+)
diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
index 4b7cfa8a..9c5d471a 100644
--- a/proxmox-sys/src/fs/file.rs
+++ b/proxmox-sys/src/fs/file.rs
@@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
match lock_file(&mut file, exclusive, Some(timeout)) {
Ok(_) => Ok(file),
+ Err(err) if err.kind() == io::ErrorKind::Interrupted => {
+ bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
+ }
Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
}
}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
@ 2026-11-19 10:18 ` Robert Obkircher
2026-11-19 10:53 ` Christian Ebner
0 siblings, 1 reply; 9+ messages in thread
From: Robert Obkircher @ 2026-11-19 10:18 UTC (permalink / raw)
To: Christian Ebner, pbs-devel
On 18.11.26 19:02, Christian Ebner wrote:
> Improves the error handling for file locking failing because of a
> timeout. Since the timeout is implemented by a timer signal that
> interrupts the blocking flock call, the propagated error message
> states, e.g.:
> ```
> Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
> ```
> Instead, report that lock acquisition timed out after waiting for
> the requested duration, which is more telling. Other errors kinds
> remain unchanged.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> proxmox-sys/src/fs/file.rs | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
> index 4b7cfa8a..9c5d471a 100644
> --- a/proxmox-sys/src/fs/file.rs
> +++ b/proxmox-sys/src/fs/file.rs
> @@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
>
> match lock_file(&mut file, exclusive, Some(timeout)) {
> Ok(_) => Ok(file),
> + Err(err) if err.kind() == io::ErrorKind::Interrupted => {
> + bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
> + }
The timeout value would be misleading if another signal interrupted
the call earlier than expected.
> Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
> }
> }
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR
2026-11-19 10:18 ` Robert Obkircher
@ 2026-11-19 10:53 ` Christian Ebner
0 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-19 10:53 UTC (permalink / raw)
To: Robert Obkircher, pbs-devel
On 11/19/26 11:18 AM, Robert Obkircher wrote:
>
> On 18.11.26 19:02, Christian Ebner wrote:
>> Improves the error handling for file locking failing because of a
>> timeout. Since the timeout is implemented by a timer signal that
>> interrupts the blocking flock call, the propagated error message
>> states, e.g.:
>> ```
>> Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4)
>> ```
>> Instead, report that lock acquisition timed out after waiting for
>> the requested duration, which is more telling. Other errors kinds
>> remain unchanged.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>> proxmox-sys/src/fs/file.rs | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/proxmox-sys/src/fs/file.rs b/proxmox-sys/src/fs/file.rs
>> index 4b7cfa8a..9c5d471a 100644
>> --- a/proxmox-sys/src/fs/file.rs
>> +++ b/proxmox-sys/src/fs/file.rs
>> @@ -474,6 +474,9 @@ pub fn open_file_locked<P: AsRef<Path>>(
>>
>> match lock_file(&mut file, exclusive, Some(timeout)) {
>> Ok(_) => Ok(file),
>> + Err(err) if err.kind() == io::ErrorKind::Interrupted => {
>> + bail!("Unable to acquire lock {path:?} - timed out after {timeout:?}")
>> + }
> The timeout value would be misleading if another signal interrupted
> the call earlier than expected.
Good catch, let's drop this patch then as it was intended as additional
cosmetic improvement.
>
>> Err(err) => bail!("Unable to acquire lock {:?} - {}", path, err),
>> }
>> }
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
3 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
To: pbs-devel
Cleanup of old task logs happens by identifying the oldest task entry
in the task archive. To protect against races by concurrent task
archive file updates, this is protected by a file lock.
Currently an exclusive lock is acquired, the oldest task cleanup
however only reads, never modifies the files so a shared lock is
sufficient. Since active worker updates and task log archive rotation
still acquire an exclusive lock, they cannot interfere.
Parts of the iteration via the task list info iterator might now
happen concurrently, but these cannot interfere since both read the
same task from the archive, which does not include old task logs to
be cleaned up and active worker updates during iteration still
require the exclusive lock.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-rest-server/src/worker_task.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index dfd0def6..d43c42ba 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -358,7 +358,7 @@ pub fn rotate_task_log_archive(
pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
let setup = worker_task_setup()?;
- let _lock = setup.lock_task_list_files(true)?;
+ let _lock = setup.lock_task_list_files(false)?;
let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
3 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
To: pbs-devel
Cleaning up old task logs held the task list lock while walking all
task log directories, checking every log file and removing the
outdated ones. With many task logs, potentially being stored on slow
storage, this can take well over the 15 second timeout used when
trying to acquire the lock by concurrent tasks.
Since every task state update requires that lock, tasks running
concurrently to the daily log rotation scheduled at midnight for PBS
failed with being unable to acquire the lock.
Only determining the cutoff time used for task log files mtime
comparison reads the task archive and requires the lock. The task log
files themselves are not protected by it.
Hold the shared lock only while computing the cutoff and perform the
old log cleanup without holding it, since logs of new tasks are
always newer than any cutoff time generated beforehand and concurrent
log rotation can only move the oldest archived entry forwards, never
backwards.
To make this more explicit, refactor the cutoff time gathering into
a dedicated method of WorkerTaskSetup, naturally limiting the lock
scope.
This will reduce lock contention during log rotation in PBS, being
scheduled at midnight, which can lead to concurrent tasks such as
backup jobs failing as reported next to the bugzilla issue several
times also in the community forum and via enterprise support.
Link: https://forum.proxmox.com/threads/155613/
Link: https://forum.proxmox.com/threads/183852/
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++-----------
1 file changed, 33 insertions(+), 23 deletions(-)
diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index d43c42ba..80889451 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -262,6 +262,37 @@ impl WorkerTaskSetup {
})
.map_err(|err: Error| format_err!("unable to create task log dir - {}", err))
}
+
+ /// Identify the oldest finished task entry's timestamp by iterating the task
+ /// archive and return the calculated cutoff time.
+ ///
+ /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
+ fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
+ let _lock = self.lock_task_list_files(false)?;
+
+ let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
+
+ if let Some(last_file) = logrotate.files().last() {
+ let reader = BufReader::new(last_file);
+ for line in reader.lines() {
+ let line = line?;
+ if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
+ let timestamp = state.endtime();
+
+ let cutoff_time = if timestamp > 0 {
+ SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
+ } else {
+ SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
+ }
+ .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
+
+ return Ok(Some(cutoff_time));
+ }
+ }
+ }
+
+ Ok(None)
+ }
}
/// Initialize the WorkerTask library
@@ -358,34 +389,13 @@ pub fn rotate_task_log_archive(
pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
let setup = worker_task_setup()?;
- let _lock = setup.lock_task_list_files(false)?;
-
- let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
-
- let mut timestamp = None;
- if let Some(last_file) = logrotate.files().last() {
- let reader = BufReader::new(last_file);
- for line in reader.lines() {
- let line = line?;
- if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
- timestamp = Some(state.endtime());
- break;
- }
- }
- }
+ let cutoff_time = setup.oldest_task_cutoff_time(compressed)?;
fn get_modified(entry: std::fs::DirEntry) -> Result<SystemTime, std::io::Error> {
entry.metadata()?.modified()
}
- if let Some(timestamp) = timestamp {
- let cutoff_time = if timestamp > 0 {
- SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
- } else {
- SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
- }
- .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
-
+ if let Some(cutoff_time) = cutoff_time {
for i in 0..256 {
let path = setup.taskdir.join(format!("{i:02X}"));
let files = match std::fs::read_dir(path) {
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
` (2 preceding siblings ...)
2026-11-18 18:02 ` [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Christian Ebner
@ 2026-11-18 18:02 ` Christian Ebner
2026-11-19 10:55 ` Robert Obkircher
3 siblings, 1 reply; 9+ messages in thread
From: Christian Ebner @ 2026-11-18 18:02 UTC (permalink / raw)
To: pbs-devel
Determining the cutoff timestamp for old task log cleanup previously
only inspected the oldest entry in the task archive, which only tracks
finished tasks. Still running tasks are listed exclusively in the
active tasks file without an end time.
If a long-running task started before the calculated archive cutoff
time and has not written any new log lines since then, its log file's
mtime is older than the cutoff time and cleanup_old_tasks() would
remove the log file while the task is still active.
Prevent this by also reading the active task list while holding the
shared task list lock during cutoff time gathering and clamping the
cutoff timestamp to the earliest active task's start time. Since a
task's log file is created after its UPID start time is generated,
its mtime is guaranteed to be at or after that timestamp.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index 80889451..f1f27ec2 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -264,7 +264,8 @@ impl WorkerTaskSetup {
}
/// Identify the oldest finished task entry's timestamp by iterating the task
- /// archive and return the calculated cutoff time.
+ /// archive, clamped to the earliest active task's start time, and return the
+ /// calculated cutoff time.
///
/// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
@@ -277,7 +278,15 @@ impl WorkerTaskSetup {
for line in reader.lines() {
let line = line?;
if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
- let timestamp = state.endtime();
+ let mut timestamp = state.endtime();
+
+ if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
+ .into_iter()
+ .map(|info| info.upid.starttime)
+ .min()
+ {
+ timestamp = timestamp.min(oldest_active);
+ }
let cutoff_time = if timestamp > 0 {
SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
@@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
}
/// removes all task logs that are older than the oldest task entry in the
-/// task archive
+/// task archive and not currently active
pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
let setup = worker_task_setup()?;
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
@ 2026-11-19 10:55 ` Robert Obkircher
2026-11-19 11:06 ` Christian Ebner
0 siblings, 1 reply; 9+ messages in thread
From: Robert Obkircher @ 2026-11-19 10:55 UTC (permalink / raw)
To: Christian Ebner, pbs-devel
On 18.11.26 19:02, Christian Ebner wrote:
> Determining the cutoff timestamp for old task log cleanup previously
> only inspected the oldest entry in the task archive, which only tracks
> finished tasks. Still running tasks are listed exclusively in the
> active tasks file without an end time.
>
> If a long-running task started before the calculated archive cutoff
> time and has not written any new log lines since then, its log file's
> mtime is older than the cutoff time and cleanup_old_tasks() would
> remove the log file while the task is still active.
>
> Prevent this by also reading the active task list while holding the
> shared task list lock during cutoff time gathering and clamping the
> cutoff timestamp to the earliest active task's start time. Since a
> task's log file is created after its UPID start time is generated,
> its mtime is guaranteed to be at or after that timestamp.
>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
> 1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
> index 80889451..f1f27ec2 100644
> --- a/proxmox-rest-server/src/worker_task.rs
> +++ b/proxmox-rest-server/src/worker_task.rs
> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
> }
>
> /// Identify the oldest finished task entry's timestamp by iterating the task
> - /// archive and return the calculated cutoff time.
> + /// archive, clamped to the earliest active task's start time, and return the
> + /// calculated cutoff time.
> ///
> /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
> fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
> @@ -277,7 +278,15 @@ impl WorkerTaskSetup {
> for line in reader.lines() {
> let line = line?;
> if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
> - let timestamp = state.endtime();
> + let mut timestamp = state.endtime();
> +
> + if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
Couldn't this be read once outside of the loops?
> + .into_iter()
> + .map(|info| info.upid.starttime)
> + .min()
> + {
> + timestamp = timestamp.min(oldest_active);
> + }
>
> let cutoff_time = if timestamp > 0 {
> SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
> @@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
> }
>
> /// removes all task logs that are older than the oldest task entry in the
> -/// task archive
> +/// task archive and not currently active
> pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
> let setup = worker_task_setup()?;
>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task
2026-11-19 10:55 ` Robert Obkircher
@ 2026-11-19 11:06 ` Christian Ebner
0 siblings, 0 replies; 9+ messages in thread
From: Christian Ebner @ 2026-11-19 11:06 UTC (permalink / raw)
To: Robert Obkircher, pbs-devel
On 11/19/26 11:55 AM, Robert Obkircher wrote:
>
> On 18.11.26 19:02, Christian Ebner wrote:
>> Determining the cutoff timestamp for old task log cleanup previously
>> only inspected the oldest entry in the task archive, which only tracks
>> finished tasks. Still running tasks are listed exclusively in the
>> active tasks file without an end time.
>>
>> If a long-running task started before the calculated archive cutoff
>> time and has not written any new log lines since then, its log file's
>> mtime is older than the cutoff time and cleanup_old_tasks() would
>> remove the log file while the task is still active.
>>
>> Prevent this by also reading the active task list while holding the
>> shared task list lock during cutoff time gathering and clamping the
>> cutoff timestamp to the earliest active task's start time. Since a
>> task's log file is created after its UPID start time is generated,
>> its mtime is guaranteed to be at or after that timestamp.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>> proxmox-rest-server/src/worker_task.rs | 15 ++++++++++++---
>> 1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
>> index 80889451..f1f27ec2 100644
>> --- a/proxmox-rest-server/src/worker_task.rs
>> +++ b/proxmox-rest-server/src/worker_task.rs
>> @@ -264,7 +264,8 @@ impl WorkerTaskSetup {
>> }
>>
>> /// Identify the oldest finished task entry's timestamp by iterating the task
>> - /// archive and return the calculated cutoff time.
>> + /// archive, clamped to the earliest active task's start time, and return the
>> + /// calculated cutoff time.
>> ///
>> /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
>> fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
>> @@ -277,7 +278,15 @@ impl WorkerTaskSetup {
>> for line in reader.lines() {
>> let line = line?;
>> if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
>> - let timestamp = state.endtime();
>> + let mut timestamp = state.endtime();
>> +
>> + if let Some(oldest_active) = read_task_file_from_path(&self.active_tasks_fn)?
> Couldn't this be read once outside of the loops?
Yeah, better to gather the latest active right after locking. Will send
a v2 dropping patch 1 and pulling this out of the loop, thanks.
>> + .into_iter()
>> + .map(|info| info.upid.starttime)
>> + .min()
>> + {
>> + timestamp = timestamp.min(oldest_active);
>> + }
>>
>> let cutoff_time = if timestamp > 0 {
>> SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
>> @@ -385,7 +394,7 @@ pub fn rotate_task_log_archive(
>> }
>>
>> /// removes all task logs that are older than the oldest task entry in the
>> -/// task archive
>> +/// task archive and not currently active
>> pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
>> let setup = worker_task_setup()?;
>>
^ permalink raw reply [flat|nested] 9+ messages in thread