From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id CF2D8EFBD for ; Thu, 28 Sep 2023 13:50:17 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B212215913 for ; Thu, 28 Sep 2023 13:50:17 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 28 Sep 2023 13:50:15 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id A171348E51 for ; Thu, 28 Sep 2023 13:50:15 +0200 (CEST) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Thu, 28 Sep 2023 13:50:05 +0200 Message-Id: <20230928115012.326777-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH v2 storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Sep 2023 11:50:17 -0000 This patch series introduces a caching mechanism for expensive status update calls made from pvestatd. As a first step, I introduced the cache to the arguably most expensive call, namely `storage_info` from pve-storage. Instead of caching the results of the `storage_info` function as a whole, we only cache the results from status updates from individual storage plugins. Regarding the cache implementation: I really tried it keep it as simple as possible. Cached values are not queried terribly often and are rather small, so I ended up with with simply creating a json-file per cache key in a directory in tmpfs. Changes from v1 -> v2: - Incorporated changes from the first review (thx @Wolfgang, thx @Max) - Made sure to invalidate the cache for storage plugin if necessary: - When changing storage config in some way - When allocating volumes - When uploading ISOs/Images (might change the amount of free space) - This allows to increase the expiration time - I've set it now to 60s - Added a flag to storage status API/CLI that allows one to ignore cached values, in case somebody needs up-to-date values at all cost (--ignore-cache) - Added functions that allow the user of proxmox-cache to lock cache entries (locking is accomplished using lock files and `flock`) - Added get_or_update helper - this one tries to get the cached value, and if it does not exist/is expired computes the new value from a closure/function passed in as a parameter, all while holding the lock on the cache entry. Locking while updating the value eliminates possible race conditions if multiple processes want to update the value at the same time. Versions of this patch series: - v1: https://lists.proxmox.com/pipermail/pve-devel/2023-August/058806.html proxmox: Lukas Wagner (5): sys: fs: remove unnecessary clippy allow directive sys: fs: let CreateOptions::apply_to take RawFd instead of File sys: fs: use inline formatting for bail! macro sys: add make_tmp_dir cache: add new crate 'proxmox-shared-cache' Cargo.toml | 1 + proxmox-shared-cache/Cargo.toml | 18 + proxmox-shared-cache/debian/changelog | 5 + proxmox-shared-cache/debian/control | 53 ++ proxmox-shared-cache/debian/copyright | 18 + proxmox-shared-cache/debian/debcargo.toml | 7 + proxmox-shared-cache/examples/performance.rs | 113 +++++ proxmox-shared-cache/src/lib.rs | 485 +++++++++++++++++++ proxmox-shared-memory/src/lib.rs | 4 +- proxmox-sys/src/fs/dir.rs | 76 ++- proxmox-sys/src/fs/file.rs | 4 +- proxmox-sys/src/fs/mod.rs | 15 +- 12 files changed, 780 insertions(+), 19 deletions(-) create mode 100644 proxmox-shared-cache/Cargo.toml create mode 100644 proxmox-shared-cache/debian/changelog create mode 100644 proxmox-shared-cache/debian/control create mode 100644 proxmox-shared-cache/debian/copyright create mode 100644 proxmox-shared-cache/debian/debcargo.toml create mode 100644 proxmox-shared-cache/examples/performance.rs create mode 100644 proxmox-shared-cache/src/lib.rs proxmox-perl-rs: Lukas Wagner (1): cache: add bindings for `SharedCache` common/pkg/Makefile | 1 + common/pkg/Proxmox/RS/SharedCache.pm | 46 ++++++++++++++ common/src/mod.rs | 1 + common/src/shared_cache.rs | 89 ++++++++++++++++++++++++++++ pve-rs/Cargo.toml | 1 + 5 files changed, 138 insertions(+) create mode 100644 common/pkg/Proxmox/RS/SharedCache.pm create mode 100644 common/src/shared_cache.rs pve-storage: Lukas Wagner (1): stats: api: cache storage plugin status src/PVE/API2/Storage/Config.pm | 18 +++++++ src/PVE/API2/Storage/Content.pm | 15 ++++++ src/PVE/API2/Storage/Status.pm | 38 ++++++++++++++- src/PVE/Storage.pm | 84 ++++++++++++++++++++++++--------- 4 files changed, 131 insertions(+), 24 deletions(-) Summary over all repositories: 21 files changed, 1049 insertions(+), 43 deletions(-) -- murpp v0.4.0