From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup
Date: Wed, 18 Nov 2026 19:02:27 +0100 [thread overview]
Message-ID: <20261118180228.341628-4-c.ebner@proxmox.com> (raw)
In-Reply-To: <20261118180228.341628-1-c.ebner@proxmox.com>
Cleaning up old task logs held the task list lock while walking all
task log directories, checking every log file and removing the
outdated ones. With many task logs, potentially being stored on slow
storage, this can take well over the 15 second timeout used when
trying to acquire the lock by concurrent tasks.
Since every task state update requires that lock, tasks running
concurrently to the daily log rotation scheduled at midnight for PBS
failed with being unable to acquire the lock.
Only determining the cutoff time used for task log files mtime
comparison reads the task archive and requires the lock. The task log
files themselves are not protected by it.
Hold the shared lock only while computing the cutoff and perform the
old log cleanup without holding it, since logs of new tasks are
always newer than any cutoff time generated beforehand and concurrent
log rotation can only move the oldest archived entry forwards, never
backwards.
To make this more explicit, refactor the cutoff time gathering into
a dedicated method of WorkerTaskSetup, naturally limiting the lock
scope.
This will reduce lock contention during log rotation in PBS, being
scheduled at midnight, which can lead to concurrent tasks such as
backup jobs failing as reported next to the bugzilla issue several
times also in the community forum and via enterprise support.
Link: https://forum.proxmox.com/threads/155613/
Link: https://forum.proxmox.com/threads/183852/
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++-----------
1 file changed, 33 insertions(+), 23 deletions(-)
diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index d43c42ba..80889451 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -262,6 +262,37 @@ impl WorkerTaskSetup {
})
.map_err(|err: Error| format_err!("unable to create task log dir - {}", err))
}
+
+ /// Identify the oldest finished task entry's timestamp by iterating the task
+ /// archive and return the calculated cutoff time.
+ ///
+ /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`].
+ fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result<Option<SystemTime>, Error> {
+ let _lock = self.lock_task_list_files(false)?;
+
+ let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?;
+
+ if let Some(last_file) = logrotate.files().last() {
+ let reader = BufReader::new(last_file);
+ for line in reader.lines() {
+ let line = line?;
+ if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
+ let timestamp = state.endtime();
+
+ let cutoff_time = if timestamp > 0 {
+ SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
+ } else {
+ SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
+ }
+ .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
+
+ return Ok(Some(cutoff_time));
+ }
+ }
+ }
+
+ Ok(None)
+ }
}
/// Initialize the WorkerTask library
@@ -358,34 +389,13 @@ pub fn rotate_task_log_archive(
pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> {
let setup = worker_task_setup()?;
- let _lock = setup.lock_task_list_files(false)?;
-
- let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?;
-
- let mut timestamp = None;
- if let Some(last_file) = logrotate.files().last() {
- let reader = BufReader::new(last_file);
- for line in reader.lines() {
- let line = line?;
- if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) {
- timestamp = Some(state.endtime());
- break;
- }
- }
- }
+ let cutoff_time = setup.oldest_task_cutoff_time(compressed)?;
fn get_modified(entry: std::fs::DirEntry) -> Result<SystemTime, std::io::Error> {
entry.metadata()?.modified()
}
- if let Some(timestamp) = timestamp {
- let cutoff_time = if timestamp > 0 {
- SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64))
- } else {
- SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64))
- }
- .ok_or_else(|| format_err!("could not calculate cutoff time"))?;
-
+ if let Some(cutoff_time) = cutoff_time {
for i in 0..256 {
let path = setup.taskdir.join(format!("{i:02X}"));
let files = match std::fs::read_dir(path) {
--
2.47.3
next prev parent reply other threads:[~2026-11-18 18:02 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 18:02 [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 1/4] sys: fs: report lock timeouts as such instead of EINTR Christian Ebner
2026-11-19 10:18 ` Robert Obkircher
2026-11-19 10:53 ` Christian Ebner
2026-11-18 18:02 ` [PATCH proxmox 2/4] rest-server: use shared instead of exclusive lock for old task cleanup Christian Ebner
2026-11-18 18:02 ` Christian Ebner [this message]
2026-11-18 18:02 ` [PATCH proxmox 4/4] rest-server: clamp old task log cleanup cutoff to earliest active task Christian Ebner
2026-11-19 10:55 ` Robert Obkircher
2026-11-19 11:06 ` Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118180228.341628-4-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox