From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 979B51FF0C0 for ; Wed, 18 Nov 2026 19:02:59 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A94F62178D; Wed, 18 Nov 2026 19:02:50 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox 3/4] fix #8341: rest-server: do not lock task list during old log cleanup Date: Wed, 18 Nov 2026 19:02:27 +0100 Message-ID: <20261118180228.341628-4-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118180228.341628-1-c.ebner@proxmox.com> References: <20261118180228.341628-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795024965626 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.361 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: BYCAOIGC72X45D72WNKGX67VO7M5NHMM X-Message-ID-Hash: BYCAOIGC72X45D72WNKGX67VO7M5NHMM X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Cleaning up old task logs held the task list lock while walking all task log directories, checking every log file and removing the outdated ones. With many task logs, potentially being stored on slow storage, this can take well over the 15 second timeout used when trying to acquire the lock by concurrent tasks. Since every task state update requires that lock, tasks running concurrently to the daily log rotation scheduled at midnight for PBS failed with being unable to acquire the lock. Only determining the cutoff time used for task log files mtime comparison reads the task archive and requires the lock. The task log files themselves are not protected by it. Hold the shared lock only while computing the cutoff and perform the old log cleanup without holding it, since logs of new tasks are always newer than any cutoff time generated beforehand and concurrent log rotation can only move the oldest archived entry forwards, never backwards. To make this more explicit, refactor the cutoff time gathering into a dedicated method of WorkerTaskSetup, naturally limiting the lock scope. This will reduce lock contention during log rotation in PBS, being scheduled at midnight, which can lead to concurrent tasks such as backup jobs failing as reported next to the bugzilla issue several times also in the community forum and via enterprise support. Link: https://forum.proxmox.com/threads/155613/ Link: https://forum.proxmox.com/threads/183852/ Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8341 Signed-off-by: Christian Ebner --- proxmox-rest-server/src/worker_task.rs | 56 +++++++++++++++----------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs index d43c42ba..80889451 100644 --- a/proxmox-rest-server/src/worker_task.rs +++ b/proxmox-rest-server/src/worker_task.rs @@ -262,6 +262,37 @@ impl WorkerTaskSetup { }) .map_err(|err: Error| format_err!("unable to create task log dir - {}", err)) } + + /// Identify the oldest finished task entry's timestamp by iterating the task + /// archive and return the calculated cutoff time. + /// + /// Note: acquired a shared lock on [`WorkerTaskSetup::task_lock_fn`]. + fn oldest_task_cutoff_time(&self, compressed: bool,) -> Result, Error> { + let _lock = self.lock_task_list_files(false)?; + + let logrotate = LogRotate::new(&self.task_archive_fn, compressed, None, None)?; + + if let Some(last_file) = logrotate.files().last() { + let reader = BufReader::new(last_file); + for line in reader.lines() { + let line = line?; + if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { + let timestamp = state.endtime(); + + let cutoff_time = if timestamp > 0 { + SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) + } else { + SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64)) + } + .ok_or_else(|| format_err!("could not calculate cutoff time"))?; + + return Ok(Some(cutoff_time)); + } + } + } + + Ok(None) + } } /// Initialize the WorkerTask library @@ -358,34 +389,13 @@ pub fn rotate_task_log_archive( pub fn cleanup_old_tasks(compressed: bool) -> Result<(), Error> { let setup = worker_task_setup()?; - let _lock = setup.lock_task_list_files(false)?; - - let logrotate = LogRotate::new(&setup.task_archive_fn, compressed, None, None)?; - - let mut timestamp = None; - if let Some(last_file) = logrotate.files().last() { - let reader = BufReader::new(last_file); - for line in reader.lines() { - let line = line?; - if let Ok((_, _, Some(state))) = parse_worker_status_line(&line) { - timestamp = Some(state.endtime()); - break; - } - } - } + let cutoff_time = setup.oldest_task_cutoff_time(compressed)?; fn get_modified(entry: std::fs::DirEntry) -> Result { entry.metadata()?.modified() } - if let Some(timestamp) = timestamp { - let cutoff_time = if timestamp > 0 { - SystemTime::UNIX_EPOCH.checked_add(Duration::from_secs(timestamp as u64)) - } else { - SystemTime::UNIX_EPOCH.checked_sub(Duration::from_secs(-timestamp as u64)) - } - .ok_or_else(|| format_err!("could not calculate cutoff time"))?; - + if let Some(cutoff_time) = cutoff_time { for i in 0..256 { let path = setup.taskdir.join(format!("{i:02X}")); let files = match std::fs::read_dir(path) { -- 2.47.3