From: Lukas Wagner <l.wagner@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls
Date: Mon, 21 Aug 2023 15:44:37 +0200 [thread overview]
Message-ID: <20230821134444.620021-1-l.wagner@proxmox.com> (raw)
This patch series introduces a caching mechanism for expensive status
update calls made from pvestatd.
As a first example for this RFC I introduced the cache to the arguably
most expensive call, namely `storage_info` from pve-storage. Instead
of caching the results of the `storage_info` function as a whole, we
only cache the results from status updates from individual storage
plugins.
Regarding the cache implementation: I really tried it keep it as simple
as possible. Cached values are not queried terribly often and are
rather small, so I ended up with with simply creating a json-file per
cache key in a directory in tmpfs. Benefits of this approach is that
it does not need any locking mechanisms (files are replaced atomically).
Also it is easily debugable/introspectable.
Open questions:
- not sure what a good expiration time for cached entries is. For
now I picked 30s, but there was not much thought behind that value.
The first three patches for `proxmox` are purely preparatory and cleanup.
proxmox:
Lukas Wagner (5):
sys: fs: move tests to a sub-module
sys: add make_tmp_dir
sys: fs: remove unnecessary clippy allow directive
cache: add new crate 'proxmox-cache'
cache: add debian packaging
Cargo.toml | 1 +
proxmox-cache/Cargo.toml | 20 ++
proxmox-cache/debian/changelog | 5 +
proxmox-cache/debian/control | 47 +++++
proxmox-cache/debian/copyright | 18 ++
proxmox-cache/debian/debcargo.toml | 7 +
proxmox-cache/examples/performance.rs | 82 ++++++++
proxmox-cache/src/lib.rs | 40 ++++
proxmox-cache/src/shared_cache.rs | 263 ++++++++++++++++++++++++++
proxmox-sys/src/fs/dir.rs | 106 +++++++++--
proxmox-sys/src/fs/mod.rs | 2 -
11 files changed, 571 insertions(+), 20 deletions(-)
create mode 100644 proxmox-cache/Cargo.toml
create mode 100644 proxmox-cache/debian/changelog
create mode 100644 proxmox-cache/debian/control
create mode 100644 proxmox-cache/debian/copyright
create mode 100644 proxmox-cache/debian/debcargo.toml
create mode 100644 proxmox-cache/examples/performance.rs
create mode 100644 proxmox-cache/src/lib.rs
create mode 100644 proxmox-cache/src/shared_cache.rs
proxmox-perl-rs:
Lukas Wagner (1):
cache: add bindings for `SharedCache` from `proxmox-cache`
common/pkg/Makefile | 1 +
common/src/cache.rs | 59 +++++++++++++++++++++++++++++++++++++++++++++
common/src/mod.rs | 1 +
pve-rs/Cargo.toml | 5 ++++
pve-rs/src/lib.rs | 1 +
5 files changed, 67 insertions(+)
create mode 100644 common/src/cache.rs
pve-storage:
Lukas Wagner (1):
stats: api: cache storage plugin status
src/PVE/API2/Storage/Config.pm | 10 +++++++++
src/PVE/Storage.pm | 40 ++++++++++++++++++++++++----------
2 files changed, 39 insertions(+), 11 deletions(-)
Summary over all repositories:
18 files changed, 677 insertions(+), 31 deletions(-)
--
murpp v0.4.0
next reply other threads:[~2023-08-21 13:44 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-21 13:44 Lukas Wagner [this message]
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 1/7] sys: fs: move tests to a sub-module Lukas Wagner
2023-08-30 15:38 ` [pve-devel] applied: " Thomas Lamprecht
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 2/7] sys: add make_tmp_dir Lukas Wagner
2023-08-22 8:39 ` Wolfgang Bumiller
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 3/7] sys: fs: remove unnecessary clippy allow directive Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 4/7] cache: add new crate 'proxmox-cache' Lukas Wagner
2023-08-22 10:08 ` Max Carrara
2023-08-22 11:33 ` Lukas Wagner
2023-08-22 12:01 ` Wolfgang Bumiller
2023-08-22 11:56 ` Wolfgang Bumiller
2023-08-22 13:52 ` Max Carrara
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 5/7] cache: add debian packaging Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC proxmox-perl-rs 6/7] cache: add bindings for `SharedCache` from `proxmox-cache` Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC pve-storage 7/7] stats: api: cache storage plugin status Lukas Wagner
2023-08-22 8:51 ` Lukas Wagner
2023-08-22 9:17 ` [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls Fiona Ebner
2023-08-22 11:25 ` Wolfgang Bumiller
2023-08-30 17:07 ` Wolf Noble
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230821134444.620021-1-l.wagner@proxmox.com \
--to=l.wagner@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox