From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id A74A11FF15E for ; Tue, 28 Jan 2025 13:26:24 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9F084CD4A; Tue, 28 Jan 2025 13:26:23 +0100 (CET) From: Lukas Wagner To: pdm-devel@lists.proxmox.com Date: Tue, 28 Jan 2025 13:25:05 +0100 Message-Id: <20250128122520.167796-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.015 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pdm-devel] [PATCH proxmox-datacenter-manager 00/15] change task cache mechanism from time-based to max-size FIFO X-BeenThere: pdm-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Datacenter Manager development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pdm-devel-bounces@lists.proxmox.com Sender: "pdm-devel" This patch series changes the remote task caching behavior from being purely time-based cache to a FIFO cache-replacement policy with a maximum number of cached tasks per remote. If the maximum number is exceeded, the oldest tasks are dropped from the cache. When calling the remote-tasks API, the latest missing task data which is not yet in the cache is requested from the remotes. There we limit this to once every 5 minutes at the moment, with the option for a force-refresh (to be triggered by a refresh button in the UI). As before, we augment the cached task data with the currently running tracked tasks which were started by PDM. Some words about the cache storage implementation: Note that the storage backend for this cache probably needs some more love in the future. Right now its just a single JSON file for everything, mainly because this was the quickest approach to implement to unblock UI development work. The problem with the approach is that it does not perform well with setups with a large number of remotes, since every update to the cache rewrites the entire cache file when the cache is persisted, causing additional CPU and IO load. In the future, we should use a similar mechanism as the task archive in PBS. I'm not sure if the exact same mechanism can be used due to some different requirements, but the general direct probably fits quite well. If we can reuse it 1:1, we have to break it out of (iirc) the WorkerTask struct to be reusable. It will probably require some experimentation and benchmarking to find an ideal approach. We probably don't want a separate archive per remote, since we do not want to read hundres/thousands of files when we request an aggregated remote task history via the API. But having the complete archive in a single file also seems quite challenging - we need to keep the data sorted, while we also need to handle task data arriving out of order from different remotes. Keeping the data sorted when new data arrives leads us to the same problem as with JSON file, being that we have to rewrite the file over and over again, causing load and writes. The good news is that since this is just a cache, we are pretty free to change the storage implementation without too much trouble; we don't even have to migrate the old data, since it should not be an issue to simply request the data from the remotes again. This is the main reason why I've opted to keep the JSON file for now; I or somebody else can revisit this at a later time. proxmox-datacenter-manager: Lukas Wagner (15): pdm-api-types: derive Debug and PartialEq for TaskListItem test support: add NamedTempFile helper task cache: add basic test for TaskCache task cache: remove max-age machanism task cache: add FIFO cache replacement policy remote tasks: move to dir based module task cache: move to its own submodule task cache: fetch every 5mins, requesting only missing tasks remote tasks: return tasks in stable order remote tasks: allow to force-fetch latest tasks fake remote: add missing fields to make the debug feature compile again fake remote: generate fake task data task cache: tests: improve test coverage remote tasks: fix unused variable warning remote-tasks: restrict function visibility lib/pdm-api-types/src/lib.rs | 2 +- server/src/api/pve/mod.rs | 4 +- server/src/api/remote_tasks.rs | 18 +- server/src/lib.rs | 4 +- .../{task_cache.rs => remote_tasks/mod.rs} | 253 ++++-------- server/src/remote_tasks/task_cache.rs | 365 ++++++++++++++++++ server/src/test_support/fake_remote.rs | 81 +++- server/src/test_support/mod.rs | 4 + server/src/test_support/temp.rs | 33 ++ 9 files changed, 563 insertions(+), 201 deletions(-) rename server/src/{task_cache.rs => remote_tasks/mod.rs} (62%) create mode 100644 server/src/remote_tasks/task_cache.rs create mode 100644 server/src/test_support/temp.rs Summary over all repositories: 9 files changed, 563 insertions(+), 201 deletions(-) -- Generated by git-murpp 0.8.0 _______________________________________________ pdm-devel mailing list pdm-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel