[pve-devel] [PATCH v2 storage/proxmox{, -perl-rs} 0/7] cache storage plugin status for pvestatd/API status update calls

[Lukas Wagner <l.wagner@proxmox.com>]

This patch series introduces a caching mechanism for expensive status
update calls made from pvestatd.

As a first step, I introduced the cache to the arguably
most expensive call, namely `storage_info` from pve-storage. Instead
of caching the results of the `storage_info` function as a whole, we
only cache the results from status updates from individual storage
plugins.

Regarding the cache implementation: I really tried it keep it as simple
as possible. Cached values are not queried terribly often and are
rather small, so I ended up with with simply creating a json-file per
cache key in a directory in tmpfs.

Changes from v1 -> v2:
  - Incorporated changes from the first review (thx @Wolfgang, thx @Max)
  - Made sure to invalidate the cache for storage plugin if necessary:
     - When changing storage config in some way
     - When allocating volumes
     - When uploading ISOs/Images (might change the amount of free space)
  - This allows to increase the expiration time - I've set it now to 60s
  - Added a flag to storage status API/CLI that allows one to ignore cached values,
    in case somebody needs up-to-date values at all cost (--ignore-cache)
  - Added functions that allow the user of proxmox-cache to lock cache entries
    (locking is accomplished using lock files and `flock`)
  - Added get_or_update helper - this one tries to get the cached value, and if 
    it does not exist/is expired computes the new value from a closure/function
    passed in as a parameter, all while holding the lock on the cache entry.
    Locking while updating the value eliminates possible race conditions if 
    multiple processes want to update the value at the same time.

Versions of this patch series:
  - v1: https://lists.proxmox.com/pipermail/pve-devel/2023-August/058806.html



proxmox:

Lukas Wagner (5):
  sys: fs: remove unnecessary clippy allow directive
  sys: fs: let CreateOptions::apply_to take RawFd instead of File
  sys: fs: use inline formatting for bail! macro
  sys: add make_tmp_dir
  cache: add new crate 'proxmox-shared-cache'

 Cargo.toml                                   |   1 +
 proxmox-shared-cache/Cargo.toml              |  18 +
 proxmox-shared-cache/debian/changelog        |   5 +
 proxmox-shared-cache/debian/control          |  53 ++
 proxmox-shared-cache/debian/copyright        |  18 +
 proxmox-shared-cache/debian/debcargo.toml    |   7 +
 proxmox-shared-cache/examples/performance.rs | 113 +++++
 proxmox-shared-cache/src/lib.rs              | 485 +++++++++++++++++++
 proxmox-shared-memory/src/lib.rs             |   4 +-
 proxmox-sys/src/fs/dir.rs                    |  76 ++-
 proxmox-sys/src/fs/file.rs                   |   4 +-
 proxmox-sys/src/fs/mod.rs                    |  15 +-
 12 files changed, 780 insertions(+), 19 deletions(-)
 create mode 100644 proxmox-shared-cache/Cargo.toml
 create mode 100644 proxmox-shared-cache/debian/changelog
 create mode 100644 proxmox-shared-cache/debian/control
 create mode 100644 proxmox-shared-cache/debian/copyright
 create mode 100644 proxmox-shared-cache/debian/debcargo.toml
 create mode 100644 proxmox-shared-cache/examples/performance.rs
 create mode 100644 proxmox-shared-cache/src/lib.rs


proxmox-perl-rs:

Lukas Wagner (1):
  cache: add bindings for `SharedCache`

 common/pkg/Makefile                  |  1 +
 common/pkg/Proxmox/RS/SharedCache.pm | 46 ++++++++++++++
 common/src/mod.rs                    |  1 +
 common/src/shared_cache.rs           | 89 ++++++++++++++++++++++++++++
 pve-rs/Cargo.toml                    |  1 +
 5 files changed, 138 insertions(+)
 create mode 100644 common/pkg/Proxmox/RS/SharedCache.pm
 create mode 100644 common/src/shared_cache.rs


pve-storage:

Lukas Wagner (1):
  stats: api: cache storage plugin status

 src/PVE/API2/Storage/Config.pm  | 18 +++++++
 src/PVE/API2/Storage/Content.pm | 15 ++++++
 src/PVE/API2/Storage/Status.pm  | 38 ++++++++++++++-
 src/PVE/Storage.pm              | 84 ++++++++++++++++++++++++---------
 4 files changed, 131 insertions(+), 24 deletions(-)


Summary over all repositories:
  21 files changed, 1049 insertions(+), 43 deletions(-)

-- 
murpp v0.4.0