From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 1F5EC1FF14F for ; Fri, 08 May 2026 17:04:13 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id F2C771C394; Fri, 8 May 2026 17:04:12 +0200 (CEST) From: Lukas Wagner To: pdm-devel@lists.proxmox.com Subject: [RFC datacenter-manager 0/4] add generic, per-remote (and global) cache for remote API responses Date: Fri, 8 May 2026 17:03:26 +0200 Message-ID: <20260508150330.363622-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1778252506836 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.054 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [resources.rs,proxmox-datacenter-privileged-api.rs,lib.rs] Message-ID-Hash: LE7YW3BL4CNHEGPOTNKWMCYSL373M4G3 X-Message-ID-Hash: LE7YW3BL4CNHEGPOTNKWMCYSL373M4G3 X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The main intention is to avoid a sprawl of different caching approaches by establishing a simple, easy to use cache implementation that can be used to persistently cache API responses from remotes (and derived aggregations). Open questions: - is the per-namespace lock too coarse? Should we rather lock per key? Anyways, one should not hold the lock during longer periods of time (e.g. when doing API requests), so the namespace-level lock seemed fine to me. A per-namespace (so, per-remote) lock is nicer when one wants to update several keys in one go. - Base directory for cache, currently it is /var/cache/proxmox-datacenter-manager/cache But this seems both redundant and generic to me, so maybe 'api-cache'? - Went with a max_age param on `get` instead of a `set` with an expiry time, I think it's quite common to have cache readers with different requirements to value freshness, so this might be a better fit. Also, we use the max-age mechanism in the API already, so this is a seamless fit then. Does this make sense? Or this we rather have redis-style `set` with expiry time/TTL? The `namespaced_cache` module is pretty generic and can be moved to proxmox.git (maybe in proxmox-shared-cache) once it has sufficiently stabilized. proxmox-datacenter-manager: Lukas Wagner (4): add persistent, generic, namespaced key-value cache implementation add pdm_cache cache as a specialized wrapper around the namespaced cache api: resources: subscriptions: switch over to pdm_cache remote-updates: switch over to pdm_cache Cargo.toml | 1 + server/Cargo.toml | 2 + server/src/api/resources.rs | 82 ++--- .../bin/proxmox-datacenter-privileged-api.rs | 7 + server/src/lib.rs | 2 + server/src/namespaced_cache.rs | 324 ++++++++++++++++++ server/src/pdm_cache.rs | 69 ++++ server/src/remote_updates.rs | 52 +-- 8 files changed, 452 insertions(+), 87 deletions(-) create mode 100644 server/src/namespaced_cache.rs create mode 100644 server/src/pdm_cache.rs Summary over all repositories: 8 files changed, 452 insertions(+), 87 deletions(-) -- Generated by murpp 0.12.0