From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 90BDC1FF0C0 for ; Wed, 18 Nov 2026 19:02:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id EF9B521724; Wed, 18 Nov 2026 19:02:49 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox 0/4] fix #8341: reduce task list lock contention during old task log cleanup Date: Wed, 18 Nov 2026 19:02:24 +0100 Message-ID: <20261118180228.341628-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795024964823 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.362 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YQ66FAMEH7XASV7ZGVMMFKJTQPQK6DX2 X-Message-ID-Hash: YQ66FAMEH7XASV7ZGVMMFKJTQPQK6DX2 X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: When rotating the task log archive and cleaning up old task logs (scheduled daily at midnight in Proxmox Backup Server), and exclusive lock was previously held for the entire duration of the cleanup. On systems with a large number of task logs or slow storage, directory traversal during task log cleanup can easily exceed the 15 second lock acquisition timeout. Because state updates on any worker task requires an exclusive lock on `.active.lock` to update the active task list, tasks running concurrently around midnight failed with: ``` Unable to acquire lock "/var/log/proxmox-backup/tasks/.active.lock" - Interrupted system call (os error 4) ``` Since determining the cutoff timestamp only requires reading the oldest entry from the task archive and is not required at all when removing the log files for which mtime is older than the cutoff time, acquire a shared lock only and drop it after the cutoff time has been calculated. While at it, improve also the error message to make it clear that a lock acquisition failed because of the timeout. Link to the bugtracker: https://bugzilla.proxmox.com/show_bug.cgi?id=8341 Further cleanups are suggested in https://bugzilla.proxmox.com/show_bug.cgi?id=5960 are not part of this series. proxmox: Christian Ebner (4): sys: fs: report lock timeouts as such instead of EINTR rest-server: use shared instead of exclusive lock for old task cleanup fix #8341: rest-server: do not lock task list during old log cleanup rest-server: clamp old task log cleanup cutoff to earliest active task proxmox-rest-server/src/worker_task.rs | 67 +++++++++++++++++--------- proxmox-sys/src/fs/file.rs | 3 ++ 2 files changed, 46 insertions(+), 24 deletions(-) Summary over all repositories: 2 files changed, 46 insertions(+), 24 deletions(-) -- Generated by murpp 0.11.0