all lists on lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls
@ 2023-08-21 13:44 Lukas Wagner
  2023-08-21 13:44 ` [pve-devel] [RFC proxmox 1/7] sys: fs: move tests to a sub-module Lukas Wagner
                   ` (7 more replies)
  0 siblings, 8 replies; 19+ messages in thread
From: Lukas Wagner @ 2023-08-21 13:44 UTC (permalink / raw)
  To: pve-devel

This patch series introduces a caching mechanism for expensive status
update calls made from pvestatd.

As a first example for this RFC I introduced the cache to the arguably
most expensive call, namely `storage_info` from pve-storage. Instead
of caching the results of the `storage_info` function as a whole, we
only cache the results from status updates from individual storage
plugins.

Regarding the cache implementation: I really tried it keep it as simple
as possible. Cached values are not queried terribly often and are
rather small, so I ended up with with simply creating a json-file per
cache key in a  directory in tmpfs. Benefits of this approach is that
it does not need any locking mechanisms (files are replaced atomically).
Also it is easily debugable/introspectable.

Open questions:
  - not sure what a good expiration time for cached entries is. For
    now I picked 30s, but there was not much thought behind that value.

The first three patches for `proxmox` are purely preparatory and cleanup.



proxmox:

Lukas Wagner (5):
  sys: fs: move tests to a sub-module
  sys: add make_tmp_dir
  sys: fs: remove unnecessary clippy allow directive
  cache: add new crate 'proxmox-cache'
  cache: add debian packaging

 Cargo.toml                            |   1 +
 proxmox-cache/Cargo.toml              |  20 ++
 proxmox-cache/debian/changelog        |   5 +
 proxmox-cache/debian/control          |  47 +++++
 proxmox-cache/debian/copyright        |  18 ++
 proxmox-cache/debian/debcargo.toml    |   7 +
 proxmox-cache/examples/performance.rs |  82 ++++++++
 proxmox-cache/src/lib.rs              |  40 ++++
 proxmox-cache/src/shared_cache.rs     | 263 ++++++++++++++++++++++++++
 proxmox-sys/src/fs/dir.rs             | 106 +++++++++--
 proxmox-sys/src/fs/mod.rs             |   2 -
 11 files changed, 571 insertions(+), 20 deletions(-)
 create mode 100644 proxmox-cache/Cargo.toml
 create mode 100644 proxmox-cache/debian/changelog
 create mode 100644 proxmox-cache/debian/control
 create mode 100644 proxmox-cache/debian/copyright
 create mode 100644 proxmox-cache/debian/debcargo.toml
 create mode 100644 proxmox-cache/examples/performance.rs
 create mode 100644 proxmox-cache/src/lib.rs
 create mode 100644 proxmox-cache/src/shared_cache.rs


proxmox-perl-rs:

Lukas Wagner (1):
  cache: add bindings for `SharedCache` from `proxmox-cache`

 common/pkg/Makefile |  1 +
 common/src/cache.rs | 59 +++++++++++++++++++++++++++++++++++++++++++++
 common/src/mod.rs   |  1 +
 pve-rs/Cargo.toml   |  5 ++++
 pve-rs/src/lib.rs   |  1 +
 5 files changed, 67 insertions(+)
 create mode 100644 common/src/cache.rs


pve-storage:

Lukas Wagner (1):
  stats: api: cache storage plugin status

 src/PVE/API2/Storage/Config.pm | 10 +++++++++
 src/PVE/Storage.pm             | 40 ++++++++++++++++++++++++----------
 2 files changed, 39 insertions(+), 11 deletions(-)


Summary over all repositories:
  18 files changed, 677 insertions(+), 31 deletions(-)

-- 
murpp v0.4.0





^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2023-08-30 17:08 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-08-21 13:44 [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 1/7] sys: fs: move tests to a sub-module Lukas Wagner
2023-08-30 15:38   ` [pve-devel] applied: " Thomas Lamprecht
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 2/7] sys: add make_tmp_dir Lukas Wagner
2023-08-22  8:39   ` Wolfgang Bumiller
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 3/7] sys: fs: remove unnecessary clippy allow directive Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 4/7] cache: add new crate 'proxmox-cache' Lukas Wagner
2023-08-22 10:08   ` Max Carrara
2023-08-22 11:33     ` Lukas Wagner
2023-08-22 12:01       ` Wolfgang Bumiller
2023-08-22 11:56     ` Wolfgang Bumiller
2023-08-22 13:52       ` Max Carrara
2023-08-21 13:44 ` [pve-devel] [RFC proxmox 5/7] cache: add debian packaging Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC proxmox-perl-rs 6/7] cache: add bindings for `SharedCache` from `proxmox-cache` Lukas Wagner
2023-08-21 13:44 ` [pve-devel] [RFC pve-storage 7/7] stats: api: cache storage plugin status Lukas Wagner
2023-08-22  8:51   ` Lukas Wagner
2023-08-22  9:17 ` [pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls Fiona Ebner
2023-08-22 11:25   ` Wolfgang Bumiller
2023-08-30 17:07   ` Wolf Noble

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal