From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id A6A971FF15E for <inbox@lore.proxmox.com>; Tue, 11 Mar 2025 14:52:23 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 29EB511662; Tue, 11 Mar 2025 14:52:16 +0100 (CET) From: Shannon Sterz <s.sterz@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Tue, 11 Mar 2025 14:51:56 +0100 Message-Id: <20250311135200.309896-1-s.sterz@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.013 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: [pbs-devel] [PATCH proxmox-backup v7 0/4] refactor datastore locking to use tmpfs X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> The goal of this series is to make it safer to remove backup groups & snapshots by separating the corresponding directories from their lock files. By moving the lock files to the tmpfs-backed '/run' directory, we also make sure that the lock files get cleaned up when the system reboots. This series refactors the locking mechanism inside the `DataStore`, `BackupDir` and `BackupGroup` traits. In a first step locking methods are added and the existing code is refactored to use them. Commit two derives a lock file name under '/run' for each group/snapshot. It also adds double stat'ing. To avoid issues when upgrading, the file `/run/proxmox-backup/old-locking` is created which is used to determine whether the system has been rebooted and we can safely use the new locking mechanism. The third commit refactors locking for manifests and brings it in-line with the group/snapshot locks. Finally, the last commit fixes a race condition when changing the owner of a datastore. This version is based on a fairly old series from 2022 [1]. [1]: https://lore.proxmox.com/pbs-devel/20220824124829.392189-1-s.sterz@proxmox.com/ ---- changes from v6: * add old locking safe guards to avoid different versions of the locking mechanism being used at the same time (see discussion here [2]). [2]: https://lore.proxmox.com/pbs-devel/20250306120810.361035-1-m.sandoval@proxmox.com/T/#u changes from v5: * re-phrase commit messages to make it clear which commit actually fixes the issue and what the commit implies in-terms of semantic changes for error messages (thanks @ Thomas Lamprecht) * make it so the series applies cleanly again and clean up a newly added usage of `lock_dir_noblock` changes from v4 (thanks @ Wolfgang Bumiller): * stop using `to_string_lossy()` * switch funtion signature of `create_locked_backup_group()` and `create_locked_backup_dir` to use the `Arc` version of a datastore. * smaller clippy fixes * rebased on current master changes from v3: * moved patch 2 to the front so it can be applied separatelly more easily * rebased on current master changes from v2: * different encoding scheme for lock file names * refactored locking methods to be used by the new BackupDir and BackupGroup traits * adapted lock file names to include namespaces changes from v1 (thanks @ Wolfgang Bumiller & Thomas Lamprecht): * split adding locking helpers and move '/run' into two commits * instead of stat'ing the path of lock file twice, only use the file descriptor for one of the stat'ing procedures instead * several improvements to helper functions and documentation Shannon Sterz (4): datastore/api/backup: prepare for fix of #3935 by adding lock helpers fix #3935: datastore/api/backup: move datastore locking to '/run' fix #3935: datastore: move manifest locking to new locking method fix: api: avoid race condition in set_backup_owner Cargo.toml | 2 +- debian/postinst | 5 + pbs-config/src/lib.rs | 32 +++- pbs-datastore/Cargo.toml | 1 + pbs-datastore/src/backup_info.rs | 230 ++++++++++++++++++++++++--- pbs-datastore/src/datastore.rs | 70 +++----- pbs-datastore/src/snapshot_reader.rs | 20 ++- src/api2/admin/datastore.rs | 9 +- src/api2/backup/environment.rs | 14 +- src/api2/backup/mod.rs | 8 +- src/api2/reader/mod.rs | 7 +- src/backup/verify.rs | 12 +- src/server/sync.rs | 10 +- 13 files changed, 302 insertions(+), 118 deletions(-) -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel