From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 95A321FF0AF for ; Fri, 20 Nov 2026 09:27:31 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BEC2821688; Fri, 20 Nov 2026 09:27:30 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox v2 0/3] fix #8341: reduce task list lock contention during old task log cleanup Date: Fri, 20 Nov 2026 09:27:07 +0100 Message-ID: <20261120082710.65728-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795163246473 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.358 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: IQLYJBWKM64ZUOB2VLY7EWKRK7WVWQCI X-Message-ID-Hash: IQLYJBWKM64ZUOB2VLY7EWKRK7WVWQCI X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: When rotating the task log archive and cleaning up old task logs (scheduled daily at midnight in Proxmox Backup Server), and exclusive lock was previously held for the entire duration of the cleanup. On systems with a large number of task logs or slow storage, directory traversal during task log cleanup can easily exceed the 15 second lock acquisition timeout. Because state updates on any worker task requires an exclusive lock on `.active.lock` to update the active task list, tasks running concurrently around midnight failed with: ``` Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4) ``` Since determining the cutoff timestamp only requires reading the oldest entry from the task archive and is not required at all when removing the log files for which mtime is older than the cutoff time, acquire a shared lock only and drop it after the cutoff time has been calculated. Link to the bugtracker: https://bugzilla.proxmox.com/show_bug.cgi?id=8341 Further cleanups are suggested in https://bugzilla.proxmox.com/show_bug.cgi?id=5960 are not part of this series. Changes since version 1 (thanks a lot @Robert for feedback): - Dropped patch changing error message since the signal interrupting the flock call can be unrelated to the timeout. - Pull the latest active task gathering out of the loop which checks the finished tasks end time. This was needless to be repeated for each finished task as the active task list is still locked from being updated anyways. proxmox: Christian Ebner (3): rest-server: use shared instead of exclusive lock for old task cleanup fix #8341: rest-server: do not lock task list during old log cleanup rest-server: clamp old task log cleanup cutoff to earliest active task proxmox-rest-server/src/worker_task.rs | 68 +++++++++++++++++--------- 1 file changed, 44 insertions(+), 24 deletions(-) Summary over all repositories: 1 files changed, 44 insertions(+), 24 deletions(-) -- Generated by murpp 0.11.0