From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id CB5378223 for ; Mon, 21 Aug 2023 15:44:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A2813157B for ; Mon, 21 Aug 2023 15:44:54 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 21 Aug 2023 15:44:54 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id B52A042DB8 for ; Mon, 21 Aug 2023 15:44:53 +0200 (CEST) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Mon, 21 Aug 2023 15:44:37 +0200 Message-Id: <20230821134444.620021-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.038 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Aug 2023 13:44:54 -0000 This patch series introduces a caching mechanism for expensive status update calls made from pvestatd. As a first example for this RFC I introduced the cache to the arguably most expensive call, namely `storage_info` from pve-storage. Instead of caching the results of the `storage_info` function as a whole, we only cache the results from status updates from individual storage plugins. Regarding the cache implementation: I really tried it keep it as simple as possible. Cached values are not queried terribly often and are rather small, so I ended up with with simply creating a json-file per cache key in a directory in tmpfs. Benefits of this approach is that it does not need any locking mechanisms (files are replaced atomically). Also it is easily debugable/introspectable. Open questions: - not sure what a good expiration time for cached entries is. For now I picked 30s, but there was not much thought behind that value. The first three patches for `proxmox` are purely preparatory and cleanup. proxmox: Lukas Wagner (5): sys: fs: move tests to a sub-module sys: add make_tmp_dir sys: fs: remove unnecessary clippy allow directive cache: add new crate 'proxmox-cache' cache: add debian packaging Cargo.toml | 1 + proxmox-cache/Cargo.toml | 20 ++ proxmox-cache/debian/changelog | 5 + proxmox-cache/debian/control | 47 +++++ proxmox-cache/debian/copyright | 18 ++ proxmox-cache/debian/debcargo.toml | 7 + proxmox-cache/examples/performance.rs | 82 ++++++++ proxmox-cache/src/lib.rs | 40 ++++ proxmox-cache/src/shared_cache.rs | 263 ++++++++++++++++++++++++++ proxmox-sys/src/fs/dir.rs | 106 +++++++++-- proxmox-sys/src/fs/mod.rs | 2 - 11 files changed, 571 insertions(+), 20 deletions(-) create mode 100644 proxmox-cache/Cargo.toml create mode 100644 proxmox-cache/debian/changelog create mode 100644 proxmox-cache/debian/control create mode 100644 proxmox-cache/debian/copyright create mode 100644 proxmox-cache/debian/debcargo.toml create mode 100644 proxmox-cache/examples/performance.rs create mode 100644 proxmox-cache/src/lib.rs create mode 100644 proxmox-cache/src/shared_cache.rs proxmox-perl-rs: Lukas Wagner (1): cache: add bindings for `SharedCache` from `proxmox-cache` common/pkg/Makefile | 1 + common/src/cache.rs | 59 +++++++++++++++++++++++++++++++++++++++++++++ common/src/mod.rs | 1 + pve-rs/Cargo.toml | 5 ++++ pve-rs/src/lib.rs | 1 + 5 files changed, 67 insertions(+) create mode 100644 common/src/cache.rs pve-storage: Lukas Wagner (1): stats: api: cache storage plugin status src/PVE/API2/Storage/Config.pm | 10 +++++++++ src/PVE/Storage.pm | 40 ++++++++++++++++++++++++---------- 2 files changed, 39 insertions(+), 11 deletions(-) Summary over all repositories: 18 files changed, 677 insertions(+), 31 deletions(-) -- murpp v0.4.0