From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 801241FF125 for ; Mon, 03 Aug 2026 11:08:20 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 58943214CE; Mon, 03 Aug 2026 11:08:09 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{,-backup} v4 00/14] keep datastore config unlocked during long running operations Date: Mon, 3 Aug 2026 11:07:33 +0200 Message-ID: <20260803090747.265683-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1785748072052 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.139 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WH3KUKVHX4S52Y53IMELN3AEAEHCTPWY X-Message-ID-Hash: WH3KUKVHX4S52Y53IMELN3AEAEHCTPWY X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Currently an s3-refresh for datastores backed by s3, mount and unmount for removalbe datastores as well as datastore creation (independent of datastore type) run while holding the datastore config lock for consistency. However, this also blocks config access by all unrelated datastores. Instead, implement per-datastore locks for maintenance-mode changes, and rely on block unexpected maintenance-mode changes during ongoing operations. This allows to concurrently create datastores, run s3 refresh and perform mount/unmount without blocking config access longer than necessary. Based on a report in the community forum: https://forum.proxmox.com/threads/183244/ Changes since version 3 (thanks @Robert for comments): - Refactor datastore creation validation checks and combine logic into common helpers, so datastore config checks and operations happen independent from the datastore creation. - Add MaintenanceModeLock wrapper type to clearly distinguish from datastore lock. - Extend file lock inode check to also check for device number. - Limit DATASTORE_LOCK_DIR to pub(crate). Changes since version 2: - Rebased onto current master Changes since version 1 (thanks @Fabian for review): - Heavily reworked locking helpers for s3-refresh and mount/unmount operations - Include new maintenance type create, used to unblock concurrent datastore creation - Refactor and code cleanup, especially with respect to chunk store scope restriction proxmox: Christian Ebner (1): pbs-api-types: add datastore create maintenance-mode type pbs-api-types/src/datastore.rs | 8 ++++++++ pbs-api-types/src/maintenance.rs | 4 ++++ 2 files changed, 12 insertions(+) proxmox-backup: Christian Ebner (13): api: config: early perform user access checks for datastore creation api: config: unlocked s3 bucket access check for datastore creation api: config: rearrange independent code block for datastore creation api/datastore: refactor datastore creation helper logic datastore: restrict chunk store scope to pbs-datastore crate datastore: move lock files base path constant to central location datastore: move file lock helper to centralized place datastore: add additional check for device number on lock helper datastore: create lockdir with correct mode for backup user access api/datastore: use maintenance-mode lock to protect against changes pbs-config: add helper to check new datastore config sections api: datastore: move prune job and s3 backend logic to create helper datastore: protect datastore creation by maintenance-mode pbs-config/src/datastore.rs | 45 +++- pbs-datastore/src/backup_info.rs | 36 +-- pbs-datastore/src/chunk_store.rs | 22 +- pbs-datastore/src/datastore.rs | 129 +++++++++- pbs-datastore/src/lib.rs | 78 +++++- .../src/local_datastore_lru_cache.rs | 2 +- pbs-datastore/src/move_journal.rs | 2 +- proxmox-backup-client/src/main.rs | 15 +- src/api2/admin/datastore.rs | 65 ++--- src/api2/config/datastore.rs | 229 ++++-------------- src/api2/helpers.rs | 78 +++++- src/api2/node/disks/directory.rs | 19 +- src/api2/node/disks/zfs.rs | 21 +- 13 files changed, 438 insertions(+), 303 deletions(-) Summary over all repositories: 15 files changed, 450 insertions(+), 303 deletions(-) -- Generated by murpp 0.11.0