[pve-devel] [RFC storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls

[Lukas Wagner <l.wagner@proxmox.com>]

This patch series introduces a caching mechanism for expensive status
update calls made from pvestatd.

As a first example for this RFC I introduced the cache to the arguably
most expensive call, namely `storage_info` from pve-storage. Instead
of caching the results of the `storage_info` function as a whole, we
only cache the results from status updates from individual storage
plugins.

Regarding the cache implementation: I really tried it keep it as simple
as possible. Cached values are not queried terribly often and are
rather small, so I ended up with with simply creating a json-file per
cache key in a  directory in tmpfs. Benefits of this approach is that
it does not need any locking mechanisms (files are replaced atomically).
Also it is easily debugable/introspectable.

Open questions:
  - not sure what a good expiration time for cached entries is. For
    now I picked 30s, but there was not much thought behind that value.

The first three patches for `proxmox` are purely preparatory and cleanup.



proxmox:

Lukas Wagner (5):
  sys: fs: move tests to a sub-module
  sys: add make_tmp_dir
  sys: fs: remove unnecessary clippy allow directive
  cache: add new crate 'proxmox-cache'
  cache: add debian packaging

 Cargo.toml                            |   1 +
 proxmox-cache/Cargo.toml              |  20 ++
 proxmox-cache/debian/changelog        |   5 +
 proxmox-cache/debian/control          |  47 +++++
 proxmox-cache/debian/copyright        |  18 ++
 proxmox-cache/debian/debcargo.toml    |   7 +
 proxmox-cache/examples/performance.rs |  82 ++++++++
 proxmox-cache/src/lib.rs              |  40 ++++
 proxmox-cache/src/shared_cache.rs     | 263 ++++++++++++++++++++++++++
 proxmox-sys/src/fs/dir.rs             | 106 +++++++++--
 proxmox-sys/src/fs/mod.rs             |   2 -
 11 files changed, 571 insertions(+), 20 deletions(-)
 create mode 100644 proxmox-cache/Cargo.toml
 create mode 100644 proxmox-cache/debian/changelog
 create mode 100644 proxmox-cache/debian/control
 create mode 100644 proxmox-cache/debian/copyright
 create mode 100644 proxmox-cache/debian/debcargo.toml
 create mode 100644 proxmox-cache/examples/performance.rs
 create mode 100644 proxmox-cache/src/lib.rs
 create mode 100644 proxmox-cache/src/shared_cache.rs


proxmox-perl-rs:

Lukas Wagner (1):
  cache: add bindings for `SharedCache` from `proxmox-cache`

 common/pkg/Makefile |  1 +
 common/src/cache.rs | 59 +++++++++++++++++++++++++++++++++++++++++++++
 common/src/mod.rs   |  1 +
 pve-rs/Cargo.toml   |  5 ++++
 pve-rs/src/lib.rs   |  1 +
 5 files changed, 67 insertions(+)
 create mode 100644 common/src/cache.rs


pve-storage:

Lukas Wagner (1):
  stats: api: cache storage plugin status

 src/PVE/API2/Storage/Config.pm | 10 +++++++++
 src/PVE/Storage.pm             | 40 ++++++++++++++++++++++++----------
 2 files changed, 39 insertions(+), 11 deletions(-)


Summary over all repositories:
  18 files changed, 677 insertions(+), 31 deletions(-)

-- 
murpp v0.4.0