From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 7CF441FF16C for ; Tue, 3 Sep 2024 14:34:10 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4168696F1; Tue, 3 Sep 2024 14:34:41 +0200 (CEST) From: Hannes Laimer To: pbs-devel@lists.proxmox.com Date: Tue, 3 Sep 2024 14:33:51 +0200 Message-Id: <20240903123401.91513-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.017 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs, datastore.rs, namespace.rs, pull.rs, drive.rs, hierarchy.rs, status.rs, mod.rs, prune.rs, verify.rs, backup.rs, proxmox-backup-proxy.rs, environment.rs] Subject: [pbs-devel] [PATCH proxmox-backup RFC 00/10] introduce typestate for datastore/chunkstore X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" This patch series introduces two traits, CanRead and CanWrite, to define whether a datastore reference is readable, writable, or neither. Functions that read or write are now implemented in `impl` or `impl` blocks, ensuring that they are only available to references that are supposed to read/write. Motivation: Currently, we track the number of read/write references of a datastore but we don't track Lookup operations as they don't read or write, they still need a chunkstore, so eventhough they don't neccessarily directly do IO, they hold an open file handle. This is a problem for things like unmounting, currently lookup operations are only really short, so you'd need really unlucky timing to actually run into problems, but still, if a datastore is in "offline" maintenance mode, we shouldn't open filehandles on it. By encoding state in the type: 1. We can assign non-readable/writable references for lookup operations. 2. The compiler ensures correct usage of references. Since it is easy to miss what might happen a few function calls down the line, having the compiler yell at you for easily missed things like this, is a really good thing I think. Changes: * Added CanRead and CanWrite traits. * Separated functions into impl or impl. * Introduced three new datastore lookup functions that return concrete types implementing CanRead, CanWrite, or neither. * Renamed lookup_datastore() to open_datastore() and made it private. The main downside is needing separate datastore caches for read and write references due to concrete type requirements in the cache HashMap. Almost all changes are either adding generics or moving functions into the appropriate trait implementations. The logic itself is only touched twice, once in datastore_lookup() and once check_privs_and_load_store() in /api/admin/datastore, this function now only checks the privs, the datastore opening happens in the endpoint function directly. Hannes Laimer (10): chunkstore: add CanRead and CanWrite trait chunkstore: separate functions into impl block datastore: add generics and new lookup functions datastore: separate functions into impl block backup_info: add generics and separate functions into impl blocks pbs-datastore: add generics and separate functions into impl blocks api: replace datastore_lookup with new, state-typed datastore returning functions server/bin: replace datastore_lookup with new, state-typed datastore returning functions api: add generics and separate functions into impl blocks backup/server/tape: add generics and separate functions into impl blocks pbs-datastore/src/backup_info.rs | 179 +- pbs-datastore/src/chunk_store.rs | 228 ++- pbs-datastore/src/datastore.rs | 1688 ++++++++++--------- pbs-datastore/src/dynamic_index.rs | 22 +- pbs-datastore/src/fixed_index.rs | 50 +- pbs-datastore/src/hierarchy.rs | 92 +- pbs-datastore/src/local_chunk_reader.rs | 13 +- pbs-datastore/src/prune.rs | 19 +- pbs-datastore/src/snapshot_reader.rs | 30 +- src/api2/admin/datastore.rs | 170 +- src/api2/admin/namespace.rs | 8 +- src/api2/backup/environment.rs | 176 +- src/api2/backup/mod.rs | 25 +- src/api2/backup/upload_chunk.rs | 19 +- src/api2/reader/environment.rs | 31 +- src/api2/reader/mod.rs | 9 +- src/api2/status.rs | 8 +- src/api2/tape/backup.rs | 21 +- src/api2/tape/drive.rs | 2 +- src/api2/tape/restore.rs | 75 +- src/backup/hierarchy.rs | 26 +- src/backup/verify.rs | 53 +- src/bin/proxmox-backup-proxy.rs | 4 +- src/server/gc_job.rs | 8 +- src/server/prune_job.rs | 9 +- src/server/pull.rs | 29 +- src/server/verify_job.rs | 4 +- src/tape/file_formats/snapshot_archive.rs | 5 +- src/tape/pool_writer/mod.rs | 11 +- src/tape/pool_writer/new_chunks_iterator.rs | 7 +- 30 files changed, 1585 insertions(+), 1436 deletions(-) -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel