public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements
@ 2021-10-13  8:24 Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
                   ` (15 more replies)
  0 siblings, 16 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

- use a journal. This way we can reduce the overall number of bytes
  written by increasing the flush/commit interval (30 minutes).

- the new CBOR base format is more flexible, and we store much more
  data points now.

We previously wrote about 7MB/h. With the new format and journal, we now write about 3MB/h.

Dietmar Maurer (15):
  proxmox-rrd: use a journal to reduce amount of bytes written
  RRD_CACHE: use a OnceCell instead of lazy_static
  proxmox-backup-proxy: use tokio::task::spawn_blocking instead of
    block_in_place
  proxmox-rrd: implement new CBOR based format
  proxmox-rrd: remove dependency to proxmox-rrd-api-types
  proxmox-rrd: extract_data: include values from current slot
  remove proxmox-rrd-api-types crate,
    s/RRDTimeFrameResolution/RRDTimeFrame/
  proxmox-rrd: support CF::Last
  proxmox-rrd: split out load_rrd (cleanup)
  proxmox-rrd: add binary to create/manage rrd files
  proxmox-rrd: avoid % inside loop
  proxmox-rrd: new helper methods - slot() and slot_end_time()
  proxmox-rrd: protect against negative update time
  proxmox-rrd: rename last_counter to last_value
  proxmox-rrd: add more commands to the rrd cli tool

 Cargo.toml                       |   2 -
 Makefile                         |   1 -
 pbs-api-types/Cargo.toml         |   1 -
 pbs-api-types/src/lib.rs         |  30 +-
 proxmox-rrd-api-types/Cargo.toml |  11 -
 proxmox-rrd-api-types/src/lib.rs |  32 --
 proxmox-rrd/Cargo.toml           |   8 +-
 proxmox-rrd/src/bin/rrd.rs       | 412 +++++++++++++++++++++++
 proxmox-rrd/src/cache.rs         | 283 +++++++++++++---
 proxmox-rrd/src/lib.rs           |  17 +-
 proxmox-rrd/src/rrd.rs           | 539 ++++++++++++++++---------------
 proxmox-rrd/src/rrd_v1.rs        | 296 +++++++++++++++++
 src/api2/admin/datastore.rs      |   6 +-
 src/api2/node/rrd.rs             |  52 +--
 src/api2/status.rs               |  14 +-
 src/bin/proxmox-backup-api.rs    |   4 -
 src/bin/proxmox-backup-proxy.rs  | 166 +++++-----
 src/lib.rs                       |  86 +++--
 18 files changed, 1465 insertions(+), 495 deletions(-)
 delete mode 100644 proxmox-rrd-api-types/Cargo.toml
 delete mode 100644 proxmox-rrd-api-types/src/lib.rs
 create mode 100644 proxmox-rrd/src/bin/rrd.rs
 create mode 100644 proxmox-rrd/src/rrd_v1.rs

-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
                   ` (14 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

Apply the journal every 30 seconds.
---
 proxmox-rrd/Cargo.toml          |   1 +
 proxmox-rrd/src/cache.rs        | 223 ++++++++++++++++++++++++++++----
 proxmox-rrd/src/lib.rs          |   6 +-
 proxmox-rrd/src/rrd.rs          |  30 +++++
 src/bin/proxmox-backup-api.rs   |   2 +-
 src/bin/proxmox-backup-proxy.rs |  54 ++++----
 src/lib.rs                      |   5 +-
 7 files changed, 261 insertions(+), 60 deletions(-)

diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index 7225be8e..c66344ac 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -9,6 +9,7 @@ description = "Simple RRD database implementation."
 anyhow = "1.0"
 bitflags = "1.2.1"
 log = "0.4"
+nix = "0.19.1"
 
 proxmox = { version = "0.14.0" }
 proxmox-time = "1"
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index fe28aeda..7c56e047 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -1,24 +1,46 @@
+use std::fs::File;
 use std::path::{Path, PathBuf};
 use std::collections::HashMap;
-use std::sync::{RwLock};
+use std::sync::RwLock;
+use std::io::Write;
+use std::io::{BufRead, BufReader};
+use std::os::unix::io::AsRawFd;
 
-use anyhow::{format_err, Error};
+use anyhow::{format_err, bail, Error};
+use nix::fcntl::OFlag;
 
-use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions};
 
 use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
 
 use crate::{DST, rrd::RRD};
 
+const RRD_JOURNAL_NAME: &str = "rrd.journal";
+
 /// RRD cache - keep RRD data in RAM, but write updates to disk
 ///
 /// This cache is designed to run as single instance (no concurrent
 /// access from other processes).
 pub struct RRDCache {
+    apply_interval: f64,
     basedir: PathBuf,
     file_options: CreateOptions,
     dir_options: CreateOptions,
-    cache: RwLock<HashMap<String, RRD>>,
+    state: RwLock<RRDCacheState>,
+}
+
+// shared state behind RwLock
+struct RRDCacheState {
+    rrd_map: HashMap<String, RRD>,
+    journal: File,
+    last_journal_flush: f64,
+}
+
+struct JournalEntry {
+    time: f64,
+    value: f64,
+    dst: DST,
+    rel_path: String,
 }
 
 impl RRDCache {
@@ -28,21 +50,166 @@ impl RRDCache {
         basedir: P,
         file_options: Option<CreateOptions>,
         dir_options: Option<CreateOptions>,
-    ) -> Self {
+        apply_interval: f64,
+    ) -> Result<Self, Error> {
         let basedir = basedir.as_ref().to_owned();
-        Self {
+
+        let file_options = file_options.unwrap_or_else(|| CreateOptions::new());
+        let dir_options = dir_options.unwrap_or_else(|| CreateOptions::new());
+
+        create_path(&basedir, Some(dir_options.clone()), Some(dir_options.clone()))
+            .map_err(|err: Error| format_err!("unable to create rrdb stat dir - {}", err))?;
+
+        let mut journal_path = basedir.clone();
+        journal_path.push(RRD_JOURNAL_NAME);
+
+        let flags = OFlag::O_CLOEXEC|OFlag::O_WRONLY|OFlag::O_APPEND;
+        let journal = atomic_open_or_create_file(&journal_path, flags,  &[], file_options.clone())?;
+
+        let state = RRDCacheState {
+            journal,
+            rrd_map: HashMap::new(),
+            last_journal_flush: 0.0,
+        };
+
+        Ok(Self {
             basedir,
-            file_options: file_options.unwrap_or_else(|| CreateOptions::new()),
-            dir_options: dir_options.unwrap_or_else(|| CreateOptions::new()),
-            cache: RwLock::new(HashMap::new()),
+            file_options,
+            dir_options,
+            apply_interval,
+            state: RwLock::new(state),
+        })
+    }
+
+    fn parse_journal_line(line: &str) -> Result<JournalEntry, Error> {
+
+        let line = line.trim();
+
+        let parts: Vec<&str> = line.splitn(4, ':').collect();
+        if parts.len() != 4 {
+            bail!("wrong numper of components");
         }
+
+        let time: f64 = parts[0].parse()
+            .map_err(|_| format_err!("unable to parse time"))?;
+        let value: f64 = parts[1].parse()
+            .map_err(|_| format_err!("unable to parse value"))?;
+        let dst: u8 = parts[2].parse()
+            .map_err(|_| format_err!("unable to parse data source type"))?;
+
+        let dst = match dst {
+            0 => DST::Gauge,
+            1 => DST::Derive,
+            _ => bail!("got strange value for data source type '{}'", dst),
+        };
+
+        let rel_path = parts[3].to_string();
+
+        Ok(JournalEntry { time, value, dst, rel_path })
+    }
+
+    fn append_journal_entry(
+        state: &mut RRDCacheState,
+        time: f64,
+        value: f64,
+        dst: DST,
+        rel_path: &str,
+    ) -> Result<(), Error> {
+        let journal_entry = format!("{}:{}:{}:{}\n", time, value, dst as u8, rel_path);
+        state.journal.write_all(journal_entry.as_bytes())?;
+        Ok(())
     }
 
-    /// Create rrdd stat dir with correct permission
-    pub fn create_rrdb_dir(&self) -> Result<(), Error> {
+    pub fn apply_journal(&self) -> Result<(), Error> {
+        let mut state = self.state.write().unwrap(); // block writers
+        self.apply_journal_locked(&mut state)
+    }
 
-        create_path(&self.basedir, Some(self.dir_options.clone()), Some(self.dir_options.clone()))
-            .map_err(|err: Error| format_err!("unable to create rrdb stat dir - {}", err))?;
+    fn apply_journal_locked(&self, state: &mut RRDCacheState) -> Result<(), Error> {
+
+        log::info!("applying rrd journal");
+
+        state.last_journal_flush = proxmox_time::epoch_f64();
+
+        let mut journal_path = self.basedir.clone();
+        journal_path.push(RRD_JOURNAL_NAME);
+
+        let flags = OFlag::O_CLOEXEC|OFlag::O_RDONLY;
+        let journal = atomic_open_or_create_file(&journal_path, flags,  &[], self.file_options.clone())?;
+        let mut journal = BufReader::new(journal);
+
+        let mut last_update_map = HashMap::new();
+
+        let mut get_last_update = |rel_path: &str, rrd: &RRD| {
+            if let Some(time) = last_update_map.get(rel_path) {
+                return *time;
+            }
+            let last_update =  rrd.last_update();
+            last_update_map.insert(rel_path.to_string(), last_update);
+            last_update
+        };
+
+        let mut linenr = 0;
+        loop {
+            linenr += 1;
+            let mut line = String::new();
+            let len = journal.read_line(&mut line)?;
+            if len == 0 { break; }
+
+            let entry = match Self::parse_journal_line(&line) {
+                Ok(entry) => entry,
+                Err(err) => {
+                    log::warn!("unable to parse rrd journal line {} (skip) - {}", linenr, err);
+                    continue; // skip unparsable lines
+                }
+            };
+
+            if let Some(rrd) = state.rrd_map.get_mut(&entry.rel_path) {
+                if entry.time > get_last_update(&entry.rel_path, &rrd) {
+                    rrd.update(entry.time, entry.value);
+                }
+            } else {
+                let mut path = self.basedir.clone();
+                path.push(&entry.rel_path);
+                create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
+
+                let mut rrd = match RRD::load(&path) {
+                    Ok(rrd) => rrd,
+                    Err(err) => {
+                        if err.kind() != std::io::ErrorKind::NotFound {
+                            log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
+                        }
+                        RRD::new(entry.dst)
+                    },
+                };
+                if entry.time > get_last_update(&entry.rel_path, &rrd) {
+                    rrd.update(entry.time, entry.value);
+                }
+                state.rrd_map.insert(entry.rel_path.clone(), rrd);
+            }
+        }
+
+        // save all RRDs
+
+        let mut errors = 0;
+        for (rel_path, rrd) in state.rrd_map.iter() {
+            let mut path = self.basedir.clone();
+            path.push(&rel_path);
+            if let Err(err) = rrd.save(&path, self.file_options.clone()) {
+                errors += 1;
+                log::error!("unable to save {:?}: {}", path, err);
+            }
+        }
+
+        // if everything went ok, commit the journal
+
+        if errors == 0 {
+            nix::unistd::ftruncate(state.journal.as_raw_fd(), 0)
+                .map_err(|err| format_err!("unable to truncate journal - {}", err))?;
+            log::info!("rrd journal successfully committed");
+        } else {
+            log::error!("errors during rrd flush - unable to commit rrd journal");
+        }
 
         Ok(())
     }
@@ -53,21 +220,26 @@ impl RRDCache {
         rel_path: &str,
         value: f64,
         dst: DST,
-        save: bool,
     ) -> Result<(), Error> {
 
-        let mut path = self.basedir.clone();
-        path.push(rel_path);
-
-        create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.file_options.clone()))?;
+        let mut state = self.state.write().unwrap(); // block other writers
 
-        let mut map = self.cache.write().unwrap();
         let now = proxmox_time::epoch_f64();
 
-        if let Some(rrd) = map.get_mut(rel_path) {
+        if (now - state.last_journal_flush) > self.apply_interval {
+            if let Err(err) = self.apply_journal_locked(&mut state) {
+                log::error!("apply journal failed: {}", err);
+            }
+        }
+
+        Self::append_journal_entry(&mut state, now, value, dst, rel_path)?;
+
+        if let Some(rrd) = state.rrd_map.get_mut(rel_path) {
             rrd.update(now, value);
-            if save { rrd.save(&path, self.file_options.clone())?; }
         } else {
+            let mut path = self.basedir.clone();
+            path.push(rel_path);
+            create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
             let mut rrd = match RRD::load(&path) {
                 Ok(rrd) => rrd,
                 Err(err) => {
@@ -78,10 +250,7 @@ impl RRDCache {
                 },
             };
             rrd.update(now, value);
-            if save {
-                rrd.save(&path, self.file_options.clone())?;
-            }
-            map.insert(rel_path.into(), rrd);
+            state.rrd_map.insert(rel_path.into(), rrd);
         }
 
         Ok(())
@@ -97,9 +266,9 @@ impl RRDCache {
         mode: RRDMode,
     ) -> Option<(u64, u64, Vec<Option<f64>>)> {
 
-        let map = self.cache.read().unwrap();
+        let state = self.state.read().unwrap();
 
-        match map.get(&format!("{}/{}", base, name)) {
+        match state.rrd_map.get(&format!("{}/{}", base, name)) {
             Some(rrd) => Some(rrd.extract_data(now, timeframe, mode)),
             None => None,
         }
diff --git a/proxmox-rrd/src/lib.rs b/proxmox-rrd/src/lib.rs
index 303cd55d..d83e6ffc 100644
--- a/proxmox-rrd/src/lib.rs
+++ b/proxmox-rrd/src/lib.rs
@@ -13,9 +13,11 @@ mod cache;
 pub use cache::*;
 
 /// RRD data source tyoe
+#[repr(u8)]
+#[derive(Copy, Clone)]
 pub enum DST {
     /// Gauge values are stored unmodified.
-    Gauge,
+    Gauge = 0,
     /// Stores the difference to the previous value.
-    Derive,
+    Derive = 1,
 }
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index f4c08909..026498ed 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -336,6 +336,36 @@ impl RRD {
         replace_file(filename, rrd_slice, options)
     }
 
+    pub fn last_update(&self) -> f64 {
+
+        let mut last_update = 0.0;
+
+        {
+            let mut check_last_update = |rra: &RRA| {
+                if rra.last_update > last_update {
+                    last_update = rra.last_update;
+                }
+            };
+
+            check_last_update(&self.hour_avg);
+            check_last_update(&self.hour_max);
+
+            check_last_update(&self.day_avg);
+            check_last_update(&self.day_max);
+
+            check_last_update(&self.week_avg);
+            check_last_update(&self.week_max);
+
+            check_last_update(&self.month_avg);
+            check_last_update(&self.month_max);
+
+            check_last_update(&self.year_avg);
+            check_last_update(&self.year_max);
+        }
+
+        last_update
+    }
+
     /// Update the value (in memory)
     ///
     /// Note: This does not call [Self::save].
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index c98fba25..3c963a77 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -74,7 +74,7 @@ async fn run() -> Result<(), Error> {
 
     proxmox_backup::server::create_run_dir()?;
 
-    RRD_CACHE.create_rrdb_dir()?;
+    RRD_CACHE.apply_journal()?;
 
     proxmox_backup::server::jobstate::create_jobstate_dir()?;
     proxmox_backup::tape::create_tape_status_dir()?;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 4bdb8ce9..8da98ff8 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -889,14 +889,10 @@ async fn command_reopen_auth_logfiles() -> Result<(), Error> {
 
 async fn run_stat_generator() {
 
-    let mut count = 0;
     loop {
-        count += 1;
-        let save = if count >= 6 { count = 0; true } else { false };
-
         let delay_target = Instant::now() +  Duration::from_secs(10);
 
-        generate_host_stats(save).await;
+        generate_host_stats().await;
 
         tokio::time::sleep_until(tokio::time::Instant::from_std(delay_target)).await;
 
@@ -904,19 +900,19 @@ async fn run_stat_generator() {
 
 }
 
-fn rrd_update_gauge(name: &str, value: f64, save: bool) {
-    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge, save) {
+fn rrd_update_gauge(name: &str, value: f64) {
+    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge) {
         eprintln!("rrd::update_value '{}' failed - {}", name, err);
     }
 }
 
-fn rrd_update_derive(name: &str, value: f64, save: bool) {
-    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive, save) {
+fn rrd_update_derive(name: &str, value: f64) {
+    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive) {
         eprintln!("rrd::update_value '{}' failed - {}", name, err);
     }
 }
 
-async fn generate_host_stats(save: bool) {
+async fn generate_host_stats() {
     use proxmox::sys::linux::procfs::{
         read_meminfo, read_proc_stat, read_proc_net_dev, read_loadavg};
 
@@ -924,8 +920,8 @@ async fn generate_host_stats(save: bool) {
 
         match read_proc_stat() {
             Ok(stat) => {
-                rrd_update_gauge("host/cpu", stat.cpu, save);
-                rrd_update_gauge("host/iowait", stat.iowait_percent, save);
+                rrd_update_gauge("host/cpu", stat.cpu);
+                rrd_update_gauge("host/iowait", stat.iowait_percent);
             }
             Err(err) => {
                 eprintln!("read_proc_stat failed - {}", err);
@@ -934,10 +930,10 @@ async fn generate_host_stats(save: bool) {
 
         match read_meminfo() {
             Ok(meminfo) => {
-                rrd_update_gauge("host/memtotal", meminfo.memtotal as f64, save);
-                rrd_update_gauge("host/memused", meminfo.memused as f64, save);
-                rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64, save);
-                rrd_update_gauge("host/swapused", meminfo.swapused as f64, save);
+                rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
+                rrd_update_gauge("host/memused", meminfo.memused as f64);
+                rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
+                rrd_update_gauge("host/swapused", meminfo.swapused as f64);
             }
             Err(err) => {
                 eprintln!("read_meminfo failed - {}", err);
@@ -954,8 +950,8 @@ async fn generate_host_stats(save: bool) {
                     netin += item.receive;
                     netout += item.send;
                 }
-                rrd_update_derive("host/netin", netin as f64, save);
-                rrd_update_derive("host/netout", netout as f64, save);
+                rrd_update_derive("host/netin", netin as f64);
+                rrd_update_derive("host/netout", netout as f64);
             }
             Err(err) => {
                 eprintln!("read_prox_net_dev failed - {}", err);
@@ -964,7 +960,7 @@ async fn generate_host_stats(save: bool) {
 
         match read_loadavg() {
             Ok(loadavg) => {
-                rrd_update_gauge("host/loadavg", loadavg.0 as f64, save);
+                rrd_update_gauge("host/loadavg", loadavg.0 as f64);
             }
             Err(err) => {
                 eprintln!("read_loadavg failed - {}", err);
@@ -973,7 +969,7 @@ async fn generate_host_stats(save: bool) {
 
         let disk_manager = DiskManage::new();
 
-        gather_disk_stats(disk_manager.clone(), Path::new("/"), "host", save);
+        gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
 
         match pbs_config::datastore::config() {
             Ok((config, _)) => {
@@ -984,7 +980,7 @@ async fn generate_host_stats(save: bool) {
 
                     let rrd_prefix = format!("datastore/{}", config.name);
                     let path = std::path::Path::new(&config.path);
-                    gather_disk_stats(disk_manager.clone(), path, &rrd_prefix, save);
+                    gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
                 }
             }
             Err(err) => {
@@ -1025,14 +1021,14 @@ fn check_schedule(worker_type: &str, event_str: &str, id: &str) -> bool {
     next <= now
 }
 
-fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &str, save: bool) {
+fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &str) {
 
     match proxmox_backup::tools::disks::disk_usage(path) {
         Ok(status) => {
             let rrd_key = format!("{}/total", rrd_prefix);
-            rrd_update_gauge(&rrd_key, status.total as f64, save);
+            rrd_update_gauge(&rrd_key, status.total as f64);
             let rrd_key = format!("{}/used", rrd_prefix);
-            rrd_update_gauge(&rrd_key, status.used as f64, save);
+            rrd_update_gauge(&rrd_key, status.used as f64);
         }
         Err(err) => {
             eprintln!("read disk_usage on {:?} failed - {}", path, err);
@@ -1064,17 +1060,17 @@ fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &st
             }
             if let Some(stat) = device_stat {
                 let rrd_key = format!("{}/read_ios", rrd_prefix);
-                rrd_update_derive(&rrd_key, stat.read_ios as f64, save);
+                rrd_update_derive(&rrd_key, stat.read_ios as f64);
                 let rrd_key = format!("{}/read_bytes", rrd_prefix);
-                rrd_update_derive(&rrd_key, (stat.read_sectors*512) as f64, save);
+                rrd_update_derive(&rrd_key, (stat.read_sectors*512) as f64);
 
                 let rrd_key = format!("{}/write_ios", rrd_prefix);
-                rrd_update_derive(&rrd_key, stat.write_ios as f64, save);
+                rrd_update_derive(&rrd_key, stat.write_ios as f64);
                 let rrd_key = format!("{}/write_bytes", rrd_prefix);
-                rrd_update_derive(&rrd_key, (stat.write_sectors*512) as f64, save);
+                rrd_update_derive(&rrd_key, (stat.write_sectors*512) as f64);
 
                 let rrd_key = format!("{}/io_ticks", rrd_prefix);
-                rrd_update_derive(&rrd_key, (stat.io_ticks as f64)/1000.0, save);
+                rrd_update_derive(&rrd_key, (stat.io_ticks as f64)/1000.0);
             }
         }
         Err(err) => {
diff --git a/src/lib.rs b/src/lib.rs
index 98b6b987..5d2b4590 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -51,10 +51,13 @@ lazy_static::lazy_static!{
             .owner(backup_user.uid)
             .group(backup_user.gid);
 
+        let apply_interval = 30.0*60.0; // 30 minutes
+
         RRDCache::new(
             "/var/lib/proxmox-backup/rrdb",
             Some(file_options),
             Some(dir_options),
-        )
+            apply_interval,
+        ).unwrap()
     };
 }
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
                   ` (13 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

And initialize only with proxmox-backup-proxy. Other binaries dont need it.
---
 src/api2/node/rrd.rs            |  6 ++--
 src/api2/status.rs              |  7 ++--
 src/bin/proxmox-backup-api.rs   |  4 ---
 src/bin/proxmox-backup-proxy.rs | 17 +++++++---
 src/lib.rs                      | 59 +++++++++++++++++++++------------
 5 files changed, 59 insertions(+), 34 deletions(-)

diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index c59a35a4..b53076d7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -10,7 +10,7 @@ use pbs_api_types::{
 
 use proxmox_rrd::rrd::RRD_DATA_ENTRIES;
 
-use crate::RRD_CACHE;
+use crate::get_rrd_cache;
 
 pub fn create_value_from_rrd(
     basedir: &str,
@@ -22,8 +22,10 @@ pub fn create_value_from_rrd(
     let mut result = Vec::new();
     let now = proxmox_time::epoch_f64();
 
+    let rrd_cache = get_rrd_cache()?;
+
     for name in list {
-        let (start, reso, list) = match RRD_CACHE.extract_cached_data(basedir, name, now, timeframe, cf) {
+        let (start, reso, list) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf) {
             Some(result) => result,
             None => continue,
         };
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 9c7d4f7d..40a9e5b4 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -22,7 +22,7 @@ use pbs_datastore::DataStore;
 use pbs_config::CachedUserInfo;
 
 use crate::tools::statistics::{linear_regression};
-use crate::RRD_CACHE;
+use crate::get_rrd_cache;
 
 #[api(
     returns: {
@@ -90,6 +90,8 @@ pub fn datastore_status(
 
     let mut list = Vec::new();
 
+    let rrd_cache = get_rrd_cache()?;
+
     for (store, (_, _)) in &config.sections {
         let user_privs = user_info.lookup_privs(&auth_id, &["datastore", &store]);
         let allowed = (user_privs & (PRIV_DATASTORE_AUDIT| PRIV_DATASTORE_BACKUP)) != 0;
@@ -123,7 +125,8 @@ pub fn datastore_status(
         let rrd_dir = format!("datastore/{}", store);
         let now = proxmox_time::epoch_f64();
 
-        let get_rrd = |what: &str| RRD_CACHE.extract_cached_data(
+
+        let get_rrd = |what: &str| rrd_cache.extract_cached_data(
             &rrd_dir,
             what,
             now,
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 3c963a77..8d6c3170 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -16,7 +16,6 @@ use proxmox_rest_server::{daemon, AuthError, ApiConfig, RestServer, RestEnvironm
 
 use proxmox_backup::server::auth::check_pbs_auth;
 use proxmox_backup::auth_helpers::*;
-use proxmox_backup::RRD_CACHE;
 use proxmox_backup::config;
 
 fn main() {
@@ -73,9 +72,6 @@ async fn run() -> Result<(), Error> {
     config::update_self_signed_cert(false)?;
 
     proxmox_backup::server::create_run_dir()?;
-
-    RRD_CACHE.apply_journal()?;
-
     proxmox_backup::server::jobstate::create_jobstate_dir()?;
     proxmox_backup::tape::create_tape_status_dir()?;
     proxmox_backup::tape::create_drive_state_dir()?;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 8da98ff8..66d81bdb 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -32,7 +32,7 @@ use proxmox_rest_server::{
 };
 
 use proxmox_backup::{
-    RRD_CACHE,
+    get_rrd_cache, initialize_rrd_cache,
     server::{
         auth::check_pbs_auth,
         jobstate::{
@@ -208,6 +208,9 @@ async fn run() -> Result<(), Error> {
     let _ = public_auth_key(); // load with lazy_static
     let _ = csrf_secret(); // load with lazy_static
 
+    let rrd_cache = initialize_rrd_cache()?;
+    rrd_cache.apply_journal()?;
+
     let mut config = ApiConfig::new(
         pbs_buildcfg::JS_DIR,
         &proxmox_backup::api2::ROUTER,
@@ -901,14 +904,18 @@ async fn run_stat_generator() {
 }
 
 fn rrd_update_gauge(name: &str, value: f64) {
-    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge) {
-        eprintln!("rrd::update_value '{}' failed - {}", name, err);
+    if let Ok(rrd_cache) = get_rrd_cache() {
+        if let Err(err) = rrd_cache.update_value(name, value, DST::Gauge) {
+            eprintln!("rrd::update_value '{}' failed - {}", name, err);
+        }
     }
 }
 
 fn rrd_update_derive(name: &str, value: f64) {
-    if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive) {
-        eprintln!("rrd::update_value '{}' failed - {}", name, err);
+    if let Ok(rrd_cache) = get_rrd_cache() {
+        if let Err(err) = rrd_cache.update_value(name, value, DST::Derive) {
+            eprintln!("rrd::update_value '{}' failed - {}", name, err);
+        }
     }
 }
 
diff --git a/src/lib.rs b/src/lib.rs
index 5d2b4590..a1ac23bf 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -5,6 +5,9 @@
 
 use std::path::PathBuf;
 
+use once_cell::sync::OnceCell;
+use anyhow::{format_err, Error};
+
 use proxmox::tools::fs::CreateOptions;
 
 use pbs_buildcfg::configdir;
@@ -39,25 +42,39 @@ pub fn cert_info() -> Result<CertInfo, anyhow::Error> {
     CertInfo::from_path(PathBuf::from(configdir!("/proxy.pem")))
 }
 
-lazy_static::lazy_static!{
-    /// Proxmox Backup Server RRD cache instance
-    pub static ref RRD_CACHE: RRDCache = {
-        let backup_user = pbs_config::backup_user().unwrap();
-        let file_options = CreateOptions::new()
-            .owner(backup_user.uid)
-            .group(backup_user.gid);
-
-       let dir_options = CreateOptions::new()
-            .owner(backup_user.uid)
-            .group(backup_user.gid);
-
-        let apply_interval = 30.0*60.0; // 30 minutes
-
-        RRDCache::new(
-            "/var/lib/proxmox-backup/rrdb",
-            Some(file_options),
-            Some(dir_options),
-            apply_interval,
-        ).unwrap()
-    };
+pub static RRD_CACHE: OnceCell<RRDCache> = OnceCell::new();
+
+/// Get the RRD cache instance
+pub fn get_rrd_cache() -> Result<&'static RRDCache, Error> {
+    RRD_CACHE.get().ok_or_else(|| format_err!("RRD cache not initialized!"))
+}
+
+/// Initialize the RRD cache instance
+///
+/// Note: Only a single process must do this (proxmox-backup-proxy)
+pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
+
+    let backup_user = pbs_config::backup_user()?;
+
+    let file_options = CreateOptions::new()
+        .owner(backup_user.uid)
+        .group(backup_user.gid);
+
+    let dir_options = CreateOptions::new()
+        .owner(backup_user.uid)
+        .group(backup_user.gid);
+
+    let apply_interval = 30.0*60.0; // 30 minutes
+
+    let cache = RRDCache::new(
+        "/var/lib/proxmox-backup/rrdb",
+        Some(file_options),
+        Some(dir_options),
+        apply_interval,
+    )?;
+
+    RRD_CACHE.set(cache)
+        .map_err(|_| format_err!("RRD cache already initialized!"))?;
+
+    Ok(RRD_CACHE.get().unwrap())
 }
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
                   ` (12 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 src/bin/proxmox-backup-proxy.rs | 119 ++++++++++++++++----------------
 1 file changed, 61 insertions(+), 58 deletions(-)

diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 66d81bdb..0f0f6f59 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -920,82 +920,85 @@ fn rrd_update_derive(name: &str, value: f64) {
 }
 
 async fn generate_host_stats() {
+    match tokio::task::spawn_blocking(generate_host_stats_sync).await {
+        Ok(()) => (),
+        Err(err) => log::error!("generate_host_stats paniced: {}", err),
+    }
+}
+
+fn generate_host_stats_sync() {
     use proxmox::sys::linux::procfs::{
         read_meminfo, read_proc_stat, read_proc_net_dev, read_loadavg};
 
-    pbs_runtime::block_in_place(move || {
-
-        match read_proc_stat() {
-            Ok(stat) => {
-                rrd_update_gauge("host/cpu", stat.cpu);
-                rrd_update_gauge("host/iowait", stat.iowait_percent);
-            }
-            Err(err) => {
-                eprintln!("read_proc_stat failed - {}", err);
-            }
+    match read_proc_stat() {
+        Ok(stat) => {
+            rrd_update_gauge("host/cpu", stat.cpu);
+            rrd_update_gauge("host/iowait", stat.iowait_percent);
+        }
+        Err(err) => {
+            eprintln!("read_proc_stat failed - {}", err);
         }
+    }
 
-        match read_meminfo() {
-            Ok(meminfo) => {
-                rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
-                rrd_update_gauge("host/memused", meminfo.memused as f64);
-                rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
-                rrd_update_gauge("host/swapused", meminfo.swapused as f64);
-            }
-            Err(err) => {
-                eprintln!("read_meminfo failed - {}", err);
-            }
+    match read_meminfo() {
+        Ok(meminfo) => {
+            rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
+            rrd_update_gauge("host/memused", meminfo.memused as f64);
+            rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
+            rrd_update_gauge("host/swapused", meminfo.swapused as f64);
         }
+        Err(err) => {
+            eprintln!("read_meminfo failed - {}", err);
+        }
+    }
 
-        match read_proc_net_dev() {
-            Ok(netdev) => {
-                use pbs_config::network::is_physical_nic;
-                let mut netin = 0;
-                let mut netout = 0;
-                for item in netdev {
-                    if !is_physical_nic(&item.device) { continue; }
-                    netin += item.receive;
-                    netout += item.send;
-                }
-                rrd_update_derive("host/netin", netin as f64);
-                rrd_update_derive("host/netout", netout as f64);
-            }
-            Err(err) => {
-                eprintln!("read_prox_net_dev failed - {}", err);
+    match read_proc_net_dev() {
+        Ok(netdev) => {
+            use pbs_config::network::is_physical_nic;
+            let mut netin = 0;
+            let mut netout = 0;
+            for item in netdev {
+                if !is_physical_nic(&item.device) { continue; }
+                netin += item.receive;
+                netout += item.send;
             }
+            rrd_update_derive("host/netin", netin as f64);
+            rrd_update_derive("host/netout", netout as f64);
+        }
+        Err(err) => {
+            eprintln!("read_prox_net_dev failed - {}", err);
         }
+    }
 
-        match read_loadavg() {
-            Ok(loadavg) => {
-                rrd_update_gauge("host/loadavg", loadavg.0 as f64);
-            }
-            Err(err) => {
-                eprintln!("read_loadavg failed - {}", err);
-            }
+    match read_loadavg() {
+        Ok(loadavg) => {
+            rrd_update_gauge("host/loadavg", loadavg.0 as f64);
+        }
+        Err(err) => {
+            eprintln!("read_loadavg failed - {}", err);
         }
+    }
 
-        let disk_manager = DiskManage::new();
+    let disk_manager = DiskManage::new();
 
-        gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
+    gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
 
-        match pbs_config::datastore::config() {
-            Ok((config, _)) => {
-                let datastore_list: Vec<DataStoreConfig> =
-                    config.convert_to_typed_array("datastore").unwrap_or_default();
+    match pbs_config::datastore::config() {
+        Ok((config, _)) => {
+            let datastore_list: Vec<DataStoreConfig> =
+                config.convert_to_typed_array("datastore").unwrap_or_default();
 
-                for config in datastore_list {
+            for config in datastore_list {
 
-                    let rrd_prefix = format!("datastore/{}", config.name);
-                    let path = std::path::Path::new(&config.path);
-                    gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
-                }
-            }
-            Err(err) => {
-                eprintln!("read datastore config failed - {}", err);
+                let rrd_prefix = format!("datastore/{}", config.name);
+                let path = std::path::Path::new(&config.path);
+                gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
             }
         }
-
-    });
+        Err(err) => {
+            eprintln!("read datastore config failed - {}", err);
+        }
+    }
 }
 
 fn check_schedule(worker_type: &str, event_str: &str, id: &str) -> bool {
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (2 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
                   ` (11 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

Storing much more data points now got get better graphs.
---
 proxmox-rrd-api-types/src/lib.rs |  23 +-
 proxmox-rrd/Cargo.toml           |   4 +
 proxmox-rrd/src/cache.rs         |  51 ++-
 proxmox-rrd/src/lib.rs           |  19 +-
 proxmox-rrd/src/rrd.rs           | 534 +++++++++++++++----------------
 proxmox-rrd/src/rrd_v1.rs        | 296 +++++++++++++++++
 src/api2/node/rrd.rs             |  39 ++-
 src/api2/status.rs               |   4 +-
 src/bin/proxmox-backup-proxy.rs  |   2 +-
 9 files changed, 641 insertions(+), 331 deletions(-)
 create mode 100644 proxmox-rrd/src/rrd_v1.rs

diff --git a/proxmox-rrd-api-types/src/lib.rs b/proxmox-rrd-api-types/src/lib.rs
index b5e62e73..32601477 100644
--- a/proxmox-rrd-api-types/src/lib.rs
+++ b/proxmox-rrd-api-types/src/lib.rs
@@ -14,19 +14,20 @@ pub enum RRDMode {
 }
 
 #[api()]
-#[repr(u64)]
 #[derive(Copy, Clone, Serialize, Deserialize)]
 #[serde(rename_all = "lowercase")]
 /// RRD time frame resolution
 pub enum RRDTimeFrameResolution {
-    ///  1 min => last 70 minutes
-    Hour = 60,
-    /// 30 min => last 35 hours
-    Day = 60*30,
-    /// 3 hours => about 8 days
-    Week = 60*180,
-    /// 12 hours => last 35 days
-    Month = 60*720,
-    /// 1 week => last 490 days
-    Year = 60*10080,
+    /// Hour
+    Hour,
+    /// Day
+    Day,
+    /// Week
+    Week,
+    /// Month
+    Month,
+    /// Year
+    Year,
+    /// Decade (10 years)
+    Decade,
 }
diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index c66344ac..b3dd02c3 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -10,8 +10,12 @@ anyhow = "1.0"
 bitflags = "1.2.1"
 log = "0.4"
 nix = "0.19.1"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+serde_cbor = "0.11.1"
 
 proxmox = { version = "0.14.0" }
 proxmox-time = "1"
+proxmox-schema = { version = "1", features = [ "api-macro" ] }
 
 proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index 7c56e047..f14837fc 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -13,7 +13,7 @@ use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions}
 
 use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
 
-use crate::{DST, rrd::RRD};
+use crate::rrd::{DST, CF, RRD, RRA};
 
 const RRD_JOURNAL_NAME: &str = "rrd.journal";
 
@@ -81,6 +81,29 @@ impl RRDCache {
         })
     }
 
+    fn create_default_rrd(dst: DST) -> RRD {
+
+        let mut rra_list = Vec::new();
+
+        // 1min * 1440 => 1day
+        rra_list.push(RRA::new(CF::Average, 60, 1440));
+        rra_list.push(RRA::new(CF::Maximum, 60, 1440));
+
+        // 30min * 1440 => 30days = 1month
+        rra_list.push(RRA::new(CF::Average, 30*60, 1440));
+        rra_list.push(RRA::new(CF::Maximum, 30*60, 1440));
+
+        // 6h * 1440 => 360days = 1year
+        rra_list.push(RRA::new(CF::Average, 6*3600, 1440));
+        rra_list.push(RRA::new(CF::Maximum, 6*3600, 1440));
+
+        // 1week * 570 => 10years
+        rra_list.push(RRA::new(CF::Average, 7*86400, 570));
+        rra_list.push(RRA::new(CF::Maximum, 7*86400, 570));
+
+        RRD::new(dst, rra_list)
+    }
+
     fn parse_journal_line(line: &str) -> Result<JournalEntry, Error> {
 
         let line = line.trim();
@@ -179,7 +202,7 @@ impl RRDCache {
                         if err.kind() != std::io::ErrorKind::NotFound {
                             log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
                         }
-                        RRD::new(entry.dst)
+                        Self::create_default_rrd(entry.dst)
                     },
                 };
                 if entry.time > get_last_update(&entry.rel_path, &rrd) {
@@ -246,7 +269,7 @@ impl RRDCache {
                     if err.kind() != std::io::ErrorKind::NotFound {
                         log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
                     }
-                    RRD::new(dst)
+                    Self::create_default_rrd(dst)
                 },
             };
             rrd.update(now, value);
@@ -264,13 +287,29 @@ impl RRDCache {
         now: f64,
         timeframe: RRDTimeFrameResolution,
         mode: RRDMode,
-    ) -> Option<(u64, u64, Vec<Option<f64>>)> {
+    ) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
 
         let state = self.state.read().unwrap();
 
+        let cf = match mode {
+            RRDMode::Max => CF::Maximum,
+            RRDMode::Average => CF::Average,
+        };
+
+        let now = now as u64;
+
+        let (start, resolution) = match timeframe {
+            RRDTimeFrameResolution::Hour => (now - 3600, 60),
+            RRDTimeFrameResolution::Day => (now - 3600*24, 60),
+            RRDTimeFrameResolution::Week => (now - 3600*24*7, 30*60),
+            RRDTimeFrameResolution::Month => (now - 3600*24*30, 30*60),
+            RRDTimeFrameResolution::Year => (now - 3600*24*365, 6*60*60),
+            RRDTimeFrameResolution::Decade => (now - 10*3600*24*366, 7*86400),
+        };
+
         match state.rrd_map.get(&format!("{}/{}", base, name)) {
-            Some(rrd) => Some(rrd.extract_data(now, timeframe, mode)),
-            None => None,
+            Some(rrd) => Ok(Some(rrd.extract_data(start, now, cf, resolution)?)),
+            None => Ok(None),
         }
     }
 }
diff --git a/proxmox-rrd/src/lib.rs b/proxmox-rrd/src/lib.rs
index d83e6ffc..2038170d 100644
--- a/proxmox-rrd/src/lib.rs
+++ b/proxmox-rrd/src/lib.rs
@@ -1,23 +1,14 @@
-//! # Simple Round Robin Database files with fixed format
+//! # Round Robin Database files
 //!
 //! ## Features
 //!
 //! * One file stores a single data source
-//! * Small/constant file size (6008 bytes)
-//! * Stores avarage and maximum values
-//! * Stores data for different time resolution ([RRDTimeFrameResolution](proxmox_rrd_api_types::RRDTimeFrameResolution))
+//! * Stores data for different time resolution
+//! * Simple cache implementation with journal support
+
+mod rrd_v1;
 
 pub mod rrd;
 
 mod cache;
 pub use cache::*;
-
-/// RRD data source tyoe
-#[repr(u8)]
-#[derive(Copy, Clone)]
-pub enum DST {
-    /// Gauge values are stored unmodified.
-    Gauge = 0,
-    /// Stores the difference to the previous value.
-    Derive = 1,
-}
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 026498ed..82fa5a3a 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -1,82 +1,175 @@
-//! # Round Robin Database file format
+//! # Proxmox RRD format version 2
+//!
+//! The new format uses
+//! [CBOR](https://datatracker.ietf.org/doc/html/rfc8949) as storage
+//! format. This way we can use the serde serialization framework,
+//! which make our code more flexible, much nicer and type safe.
+//!
+//! ## Features
+//!
+//! * Well defined data format [CBOR](https://datatracker.ietf.org/doc/html/rfc8949)
+//! * Plattform independent (big endian f64, hopefully a standard format?)
+//! * Arbitrary number of RRAs (dynamically changeable)
 
-use std::io::Read;
 use std::path::Path;
 
 use anyhow::{bail, Error};
-use bitflags::bitflags;
 
-use proxmox::tools::{fs::replace_file, fs::CreateOptions};
+use serde::{Serialize, Deserialize};
+
+use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_schema::api;
+
+use crate::rrd_v1;
+
+/// Proxmox RRD v2 file magic number
+// openssl::sha::sha256(b"Proxmox Round Robin Database file v2.0")[0..8];
+pub const PROXMOX_RRD_MAGIC_2_0: [u8; 8] = [224, 200, 228, 27, 239, 112, 122, 159];
+
+#[api()]
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// RRD data source type
+pub enum DST {
+    /// Gauge values are stored unmodified.
+    Gauge,
+    /// Stores the difference to the previous value.
+    Derive,
+    /// Stores the difference to the previous value (like Derive), but
+    /// detect counter overflow (and ignores that value)
+    Counter,
+}
+
+#[api()]
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Consolidation function
+pub enum CF {
+    /// Average
+    Average,
+    /// Maximum
+    Maximum,
+    /// Minimum
+    Minimum,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct DataSource {
+    /// Data source type
+    pub dst: DST,
+    /// Last update time (epoch)
+    pub last_update: f64,
+    /// Stores the last value, used to compute differential value for
+    /// derive/counters
+    pub counter_value: f64,
+}
 
-use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
+impl DataSource {
 
-/// The number of data entries per RRA
-pub const RRD_DATA_ENTRIES: usize = 70;
+    pub fn new(dst: DST) -> Self {
+        Self {
+            dst,
+            last_update: 0.0,
+            counter_value: f64::NAN,
+        }
+    }
 
-/// Proxmox RRD file magic number
-// openssl::sha::sha256(b"Proxmox Round Robin Database file v1.0")[0..8];
-pub const PROXMOX_RRD_MAGIC_1_0: [u8; 8] =  [206, 46, 26, 212, 172, 158, 5, 186];
+    fn compute_new_value(&mut self, time: f64, mut value: f64) -> Result<f64, Error> {
+        if time <= self.last_update {
+            bail!("time in past ({} < {})", time, self.last_update);
+        }
 
-use crate::DST;
+        if value.is_nan() {
+            bail!("new value is NAN");
+        }
 
-bitflags!{
-    /// Flags to specify the data soure type and consolidation function
-    pub struct RRAFlags: u64 {
-        // Data Source Types
-        const DST_GAUGE  = 1;
-        const DST_DERIVE = 2;
-        const DST_COUNTER = 4;
-        const DST_MASK   = 255; // first 8 bits
+        // derive counter value
+        let is_counter = self.dst == DST::Counter;
+
+        if is_counter || self.dst == DST::Derive {
+            let time_diff = time - self.last_update;
+
+            let diff = if self.counter_value.is_nan() {
+                0.0
+            } else if is_counter && value < 0.0 {
+                bail!("got negative value for counter");
+            } else if is_counter && value < self.counter_value {
+                // Note: We do not try automatic overflow corrections, but
+                // we update counter_value anyways, so that we can compute the diff
+                // next time.
+                self.counter_value = value;
+                bail!("conter overflow/reset detected");
+            } else {
+                value - self.counter_value
+            };
+            self.counter_value = value;
+            value = diff/time_diff;
+        }
 
-        // Consolidation Functions
-        const CF_AVERAGE = 1 << 8;
-        const CF_MAX     = 2 << 8;
-        const CF_MASK    = 255 << 8;
+        Ok(value)
     }
+
+
 }
 
-/// Round Robin Archive with [RRD_DATA_ENTRIES] data slots.
-///
-/// This data structure is used inside [RRD] and directly written to the
-/// RRD files.
-#[repr(C)]
+#[derive(Serialize, Deserialize)]
 pub struct RRA {
-    /// Defined the data soure type and consolidation function
-    pub flags: RRAFlags,
-    /// Resulution (seconds) from [RRDTimeFrameResolution]
     pub resolution: u64,
-    /// Last update time (epoch)
-    pub last_update: f64,
+    pub cf: CF,
     /// Count values computed inside this update interval
     pub last_count: u64,
-    /// Stores the last value, used to compute differential value for derive/counters
-    pub counter_value: f64,
-    /// Data slots
-    pub data: [f64; RRD_DATA_ENTRIES],
+    /// The actual data
+    pub data: Vec<f64>,
 }
 
 impl RRA {
-    fn new(flags: RRAFlags, resolution: u64) -> Self {
+
+    pub fn new(cf: CF, resolution: u64, points: usize) -> Self {
         Self {
-            flags, resolution,
-            last_update: 0.0,
+            cf,
+            resolution,
             last_count: 0,
-            counter_value: f64::NAN,
-            data: [f64::NAN; RRD_DATA_ENTRIES],
+            data: vec![f64::NAN; points],
+        }
+    }
+
+    // directly overwrite data slots
+    // the caller need to set last_update value on the DataSource manually.
+    pub(crate) fn insert_data(
+        &mut self,
+        start: u64,
+        resolution: u64,
+        data: Vec<Option<f64>>,
+    ) -> Result<(), Error> {
+        if resolution != self.resolution {
+            bail!("inser_data failed: got wrong resolution");
+        }
+        let num_entries = self.data.len() as u64;
+        let mut index = ((start/self.resolution) % num_entries) as usize;
+
+        for i in 0..data.len() {
+            if let Some(v) = data[i] {
+                self.data[index] = v;
+            }
+            index += 1;
+            if index >= self.data.len() { index = 0; }
         }
+        Ok(())
     }
 
-    fn delete_old(&mut self, time: f64) {
+    fn delete_old_slots(&mut self, time: f64, last_update: f64) {
         let epoch = time as u64;
-        let last_update = self.last_update as u64;
+        let last_update = last_update as u64;
         let reso = self.resolution;
+        let num_entries = self.data.len() as u64;
 
-        let min_time = epoch - (RRD_DATA_ENTRIES as u64)*reso;
+        let min_time = epoch - num_entries*reso;
         let min_time = (min_time/reso + 1)*reso;
-        let mut t = last_update.saturating_sub((RRD_DATA_ENTRIES as u64)*reso);
-        let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
-        for _ in 0..RRD_DATA_ENTRIES {
-            t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+        let mut t = last_update.saturating_sub(num_entries*reso);
+        let mut index = ((t/reso) % num_entries) as usize;
+        for _ in 0..num_entries {
+            t += reso;
+            index = (index + 1) % (num_entries as usize);
             if t < min_time {
                 self.data[index] = f64::NAN;
             } else {
@@ -85,13 +178,14 @@ impl RRA {
         }
     }
 
-    fn compute_new_value(&mut self, time: f64, value: f64) {
+    fn compute_new_value(&mut self, time: f64, last_update: f64, value: f64) {
         let epoch = time as u64;
-        let last_update = self.last_update as u64;
+        let last_update = last_update as u64;
         let reso = self.resolution;
+        let num_entries = self.data.len() as u64;
 
-        let index = ((epoch/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
-        let last_index = ((last_update/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
+        let index = ((epoch/reso) % num_entries) as usize;
+        let last_index = ((last_update/reso) % num_entries) as usize;
 
         if (epoch - (last_update as u64)) > reso || index != last_index {
             self.last_count = 0;
@@ -112,258 +206,111 @@ impl RRA {
             self.data[index] = value;
             self.last_count = 1;
         } else {
-            let new_value = if self.flags.contains(RRAFlags::CF_MAX) {
-                if last_value > value { last_value } else { value }
-            } else if self.flags.contains(RRAFlags::CF_AVERAGE) {
-                (last_value*(self.last_count as f64))/(new_count as f64)
-                    + value/(new_count as f64)
-            } else {
-                log::error!("rrdb update failed - unknown CF");
-                return;
+            let new_value = match self.cf {
+                CF::Maximum => if last_value > value { last_value } else { value },
+                CF::Minimum => if last_value < value { last_value } else { value },
+                CF::Average => {
+                    (last_value*(self.last_count as f64))/(new_count as f64)
+                        + value/(new_count as f64)
+                }
             };
             self.data[index] = new_value;
             self.last_count = new_count;
         }
-        self.last_update = time;
     }
 
-    // Note: This may update the state even in case of errors (see counter overflow)
-    fn update(&mut self, time: f64, mut value: f64) -> Result<(), Error> {
-
-        if time <= self.last_update {
-            bail!("time in past ({} < {})", time, self.last_update);
-        }
-
-        if value.is_nan() {
-            bail!("new value is NAN");
-        }
-
-        // derive counter value
-        if self.flags.intersects(RRAFlags::DST_DERIVE | RRAFlags::DST_COUNTER) {
-            let time_diff = time - self.last_update;
-            let is_counter = self.flags.contains(RRAFlags::DST_COUNTER);
-
-            let diff = if self.counter_value.is_nan() {
-                0.0
-            } else if is_counter && value < 0.0 {
-                bail!("got negative value for counter");
-            } else if is_counter && value < self.counter_value {
-                // Note: We do not try automatic overflow corrections, but
-                // we update counter_value anyways, so that we can compute the diff
-                // next time.
-                self.counter_value = value;
-                bail!("conter overflow/reset detected");
-            } else {
-                value - self.counter_value
-            };
-            self.counter_value = value;
-            value = diff/time_diff;
-        }
-
-        self.delete_old(time);
-        self.compute_new_value(time, value);
-
-        Ok(())
-    }
-}
-
-/// Round Robin Database file format with fixed number of [RRA]s
-#[repr(C)]
-// Note: Avoid alignment problems by using 8byte types only
-pub struct RRD {
-    /// The magic number to identify the file type
-    pub magic: [u8; 8],
-    /// Hourly data (average values)
-    pub hour_avg: RRA,
-    /// Hourly data (maximum values)
-    pub hour_max: RRA,
-    /// Dayly data (average values)
-    pub day_avg: RRA,
-    /// Dayly data (maximum values)
-    pub day_max: RRA,
-    /// Weekly data (average values)
-    pub week_avg: RRA,
-    /// Weekly data (maximum values)
-    pub week_max: RRA,
-    /// Monthly data (average values)
-    pub month_avg: RRA,
-    /// Monthly data (maximum values)
-    pub month_max: RRA,
-    /// Yearly data (average values)
-    pub year_avg: RRA,
-    /// Yearly data (maximum values)
-    pub year_max: RRA,
-}
-
-impl RRD {
-
-    /// Create a new empty instance
-    pub fn new(dst: DST) -> Self {
-        let flags = match dst {
-            DST::Gauge => RRAFlags::DST_GAUGE,
-            DST::Derive => RRAFlags::DST_DERIVE,
-        };
-
-        Self {
-            magic: PROXMOX_RRD_MAGIC_1_0,
-            hour_avg: RRA::new(
-                flags | RRAFlags::CF_AVERAGE,
-                RRDTimeFrameResolution::Hour as u64,
-            ),
-            hour_max: RRA::new(
-                flags |  RRAFlags::CF_MAX,
-                RRDTimeFrameResolution::Hour as u64,
-            ),
-            day_avg: RRA::new(
-                flags |  RRAFlags::CF_AVERAGE,
-                RRDTimeFrameResolution::Day as u64,
-            ),
-            day_max: RRA::new(
-                flags |  RRAFlags::CF_MAX,
-                RRDTimeFrameResolution::Day as u64,
-            ),
-            week_avg: RRA::new(
-                flags |  RRAFlags::CF_AVERAGE,
-                RRDTimeFrameResolution::Week as u64,
-            ),
-            week_max: RRA::new(
-                flags |  RRAFlags::CF_MAX,
-                RRDTimeFrameResolution::Week as u64,
-            ),
-            month_avg: RRA::new(
-                flags |  RRAFlags::CF_AVERAGE,
-                RRDTimeFrameResolution::Month as u64,
-            ),
-            month_max: RRA::new(
-                flags |  RRAFlags::CF_MAX,
-                RRDTimeFrameResolution::Month as u64,
-            ),
-            year_avg: RRA::new(
-                flags |  RRAFlags::CF_AVERAGE,
-                RRDTimeFrameResolution::Year as u64,
-            ),
-            year_max: RRA::new(
-                flags |  RRAFlags::CF_MAX,
-                RRDTimeFrameResolution::Year as u64,
-            ),
-        }
-    }
-
-    /// Extract data from the archive
-    pub fn extract_data(
+    fn extract_data(
         &self,
-        time: f64,
-        timeframe: RRDTimeFrameResolution,
-        mode: RRDMode,
+        start: u64,
+        end: u64,
+        last_update: f64,
     ) -> (u64, u64, Vec<Option<f64>>) {
-
-        let epoch = time as u64;
-        let reso = timeframe as u64;
-
-        let end = reso*(epoch/reso + 1);
-        let start = end - reso*(RRD_DATA_ENTRIES as u64);
+        let last_update = last_update as u64;
+        let reso = self.resolution;
+        let num_entries = self.data.len() as u64;
 
         let mut list = Vec::new();
 
-        let raa = match (mode, timeframe) {
-            (RRDMode::Average, RRDTimeFrameResolution::Hour) => &self.hour_avg,
-            (RRDMode::Max, RRDTimeFrameResolution::Hour) => &self.hour_max,
-            (RRDMode::Average, RRDTimeFrameResolution::Day) => &self.day_avg,
-            (RRDMode::Max, RRDTimeFrameResolution::Day) => &self.day_max,
-            (RRDMode::Average, RRDTimeFrameResolution::Week) => &self.week_avg,
-            (RRDMode::Max, RRDTimeFrameResolution::Week) => &self.week_max,
-            (RRDMode::Average, RRDTimeFrameResolution::Month) => &self.month_avg,
-            (RRDMode::Max, RRDTimeFrameResolution::Month) => &self.month_max,
-            (RRDMode::Average, RRDTimeFrameResolution::Year) => &self.year_avg,
-            (RRDMode::Max, RRDTimeFrameResolution::Year) => &self.year_max,
-        };
-
-        let rrd_end = reso*((raa.last_update as u64)/reso);
-        let rrd_start = rrd_end - reso*(RRD_DATA_ENTRIES as u64);
+        let rrd_end = reso*(last_update/reso);
+        let rrd_start = rrd_end.saturating_sub(reso*num_entries);
 
         let mut t = start;
-        let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
-        for _ in 0..RRD_DATA_ENTRIES {
+        let mut index = ((t/reso) % num_entries) as usize;
+        for _ in 0..num_entries {
+            if t > end { break; };
             if t < rrd_start || t > rrd_end {
                 list.push(None);
             } else {
-                let value = raa.data[index];
+                let value = self.data[index];
                 if value.is_nan() {
                     list.push(None);
                 } else {
                     list.push(Some(value));
                 }
             }
-            t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+            t += reso; index = (index + 1) % (num_entries as usize);
         }
 
         (start, reso, list)
     }
+}
 
-    /// Create instance from raw data, testing data len and magic number
-    pub fn from_raw(mut raw: &[u8]) -> Result<Self, std::io::Error> {
-        let expected_len = std::mem::size_of::<RRD>();
-        if raw.len() != expected_len {
-            let msg = format!("wrong data size ({} != {})", raw.len(), expected_len);
-            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
-        }
+#[derive(Serialize, Deserialize)]
+pub struct RRD {
+    pub source: DataSource,
+    pub rra_list: Vec<RRA>,
+}
 
-        let mut rrd: RRD = unsafe { std::mem::zeroed() };
-        unsafe {
-            let rrd_slice = std::slice::from_raw_parts_mut(&mut rrd as *mut _ as *mut u8, expected_len);
-            raw.read_exact(rrd_slice)?;
-        }
+impl RRD {
 
-        if rrd.magic != PROXMOX_RRD_MAGIC_1_0 {
-            let msg = "wrong magic number".to_string();
-            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+    pub fn new(dst: DST, rra_list: Vec<RRA>) -> RRD {
+
+        let source = DataSource::new(dst);
+
+        RRD {
+            source,
+            rra_list,
         }
 
-        Ok(rrd)
     }
 
     /// Load data from a file
     pub fn load(path: &Path) -> Result<Self, std::io::Error> {
         let raw = std::fs::read(path)?;
-        Self::from_raw(&raw)
+        if raw.len() < 8 {
+            let msg = format!("not an rrd file - file is too small ({})", raw.len());
+            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+        }
+
+        if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
+            let v1 = rrd_v1::RRDv1::from_raw(&raw)?;
+            v1.to_rrd_v2()
+                .map_err(|err| {
+                    let msg = format!("unable to convert from old V1 format - {}", err);
+                    std::io::Error::new(std::io::ErrorKind::Other, msg)
+                })
+        } else if raw[0..8] == PROXMOX_RRD_MAGIC_2_0 {
+            serde_cbor::from_slice(&raw[8..])
+                .map_err(|err| {
+                    let msg = format!("unable to decode RRD file - {}", err);
+                    std::io::Error::new(std::io::ErrorKind::Other, msg)
+                })
+         } else {
+            let msg = format!("not an rrd file - unknown magic number");
+            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+        }
     }
 
     /// Store data into a file (atomic replace file)
     pub fn save(&self, filename: &Path, options: CreateOptions) -> Result<(), Error> {
-        let rrd_slice = unsafe {
-            std::slice::from_raw_parts(self as *const _ as *const u8, std::mem::size_of::<RRD>())
-        };
-        replace_file(filename, rrd_slice, options)
+        let mut data: Vec<u8> = Vec::new();
+        data.extend(&PROXMOX_RRD_MAGIC_2_0);
+        serde_cbor::to_writer(&mut data, self)?;
+        replace_file(filename, &data, options)
     }
 
     pub fn last_update(&self) -> f64 {
-
-        let mut last_update = 0.0;
-
-        {
-            let mut check_last_update = |rra: &RRA| {
-                if rra.last_update > last_update {
-                    last_update = rra.last_update;
-                }
-            };
-
-            check_last_update(&self.hour_avg);
-            check_last_update(&self.hour_max);
-
-            check_last_update(&self.day_avg);
-            check_last_update(&self.day_max);
-
-            check_last_update(&self.week_avg);
-            check_last_update(&self.week_max);
-
-            check_last_update(&self.month_avg);
-            check_last_update(&self.month_max);
-
-            check_last_update(&self.year_avg);
-            check_last_update(&self.year_max);
-        }
-
-        last_update
+        self.source.last_update
     }
 
     /// Update the value (in memory)
@@ -371,32 +318,53 @@ impl RRD {
     /// Note: This does not call [Self::save].
     pub fn update(&mut self, time: f64, value: f64) {
 
-        let mut log_error = true;
-
-        let mut update_rra = |rra: &mut RRA| {
-            if let Err(err) = rra.update(time, value) {
-                if log_error {
-                    log::error!("rrd update failed: {}", err);
-                    // we only log the first error, because it is very
-                    // likely other calls produce the same error
-                    log_error = false;
-                }
+        let value = match self.source.compute_new_value(time, value) {
+            Ok(value) => value,
+            Err(err) => {
+                log::error!("rrd update failed: {}", err);
+                return;
             }
         };
 
-        update_rra(&mut self.hour_avg);
-        update_rra(&mut self.hour_max);
-
-        update_rra(&mut self.day_avg);
-        update_rra(&mut self.day_max);
+        let last_update = self.source.last_update;
+        self.source.last_update = time;
 
-        update_rra(&mut self.week_avg);
-        update_rra(&mut self.week_max);
+        for rra in self.rra_list.iter_mut() {
+            rra.delete_old_slots(time, last_update);
+            rra.compute_new_value(time, last_update, value);
+        }
+    }
 
-        update_rra(&mut self.month_avg);
-        update_rra(&mut self.month_max);
+    /// Extract data from the archive
+    ///
+    /// This selects the RRA with specified [CF] and (minimum)
+    /// resolution, and extract data from `start` to `end`.
+    pub fn extract_data(
+        &self,
+        start: u64,
+        end: u64,
+        cf: CF,
+        resolution: u64,
+    ) -> Result<(u64, u64, Vec<Option<f64>>), Error> {
+
+        let mut rra: Option<&RRA> = None;
+        for item in self.rra_list.iter() {
+            if item.cf != cf { continue; }
+            if item.resolution > resolution { continue; }
+
+            if let Some(current) = rra {
+                if item.resolution > current.resolution {
+                    rra = Some(item);
+                }
+            } else {
+                rra = Some(item);
+            }
+        }
 
-        update_rra(&mut self.year_avg);
-        update_rra(&mut self.year_max);
+        match rra {
+            Some(rra) => Ok(rra.extract_data(start, end, self.source.last_update)),
+            None => bail!("unable to find RRA suitable ({:?}:{})", cf, resolution),
+        }
     }
+
 }
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
new file mode 100644
index 00000000..919896f0
--- /dev/null
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -0,0 +1,296 @@
+use std::io::Read;
+
+use anyhow::Error;
+use bitflags::bitflags;
+
+/// The number of data entries per RRA
+pub const RRD_DATA_ENTRIES: usize = 70;
+
+/// Proxmox RRD file magic number
+// openssl::sha::sha256(b"Proxmox Round Robin Database file v1.0")[0..8];
+pub const PROXMOX_RRD_MAGIC_1_0: [u8; 8] =  [206, 46, 26, 212, 172, 158, 5, 186];
+
+use crate::rrd::{RRD, RRA, CF, DST, DataSource};
+
+bitflags!{
+    /// Flags to specify the data soure type and consolidation function
+    pub struct RRAFlags: u64 {
+        // Data Source Types
+        const DST_GAUGE  = 1;
+        const DST_DERIVE = 2;
+        const DST_COUNTER = 4;
+        const DST_MASK   = 255; // first 8 bits
+
+        // Consolidation Functions
+        const CF_AVERAGE = 1 << 8;
+        const CF_MAX     = 2 << 8;
+        const CF_MASK    = 255 << 8;
+    }
+}
+
+/// Round Robin Archive with [RRD_DATA_ENTRIES] data slots.
+///
+/// This data structure is used inside [RRD] and directly written to the
+/// RRD files.
+#[repr(C)]
+pub struct RRAv1 {
+    /// Defined the data soure type and consolidation function
+    pub flags: RRAFlags,
+    /// Resulution (seconds) from [RRDTimeFrameResolution]
+    pub resolution: u64,
+    /// Last update time (epoch)
+    pub last_update: f64,
+    /// Count values computed inside this update interval
+    pub last_count: u64,
+    /// Stores the last value, used to compute differential value for derive/counters
+    pub counter_value: f64,
+    /// Data slots
+    pub data: [f64; RRD_DATA_ENTRIES],
+}
+
+impl RRAv1 {
+
+    fn extract_data(
+        &self,
+    ) -> (u64, u64, Vec<Option<f64>>) {
+        let reso = self.resolution;
+
+        let mut list = Vec::new();
+
+        let rra_end = reso*((self.last_update as u64)/reso);
+        let rra_start = rra_end - reso*(RRD_DATA_ENTRIES as u64);
+
+        let mut t = rra_start;
+        let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
+        for _ in 0..RRD_DATA_ENTRIES {
+            let value = self.data[index];
+            if value.is_nan() {
+                list.push(None);
+            } else {
+                list.push(Some(value));
+            }
+
+            t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+        }
+
+        (rra_start, reso, list)
+    }
+}
+
+/// Round Robin Database file format with fixed number of [RRA]s
+#[repr(C)]
+// Note: Avoid alignment problems by using 8byte types only
+pub struct RRDv1 {
+    /// The magic number to identify the file type
+    pub magic: [u8; 8],
+    /// Hourly data (average values)
+    pub hour_avg: RRAv1,
+    /// Hourly data (maximum values)
+    pub hour_max: RRAv1,
+    /// Dayly data (average values)
+    pub day_avg: RRAv1,
+    /// Dayly data (maximum values)
+    pub day_max: RRAv1,
+    /// Weekly data (average values)
+    pub week_avg: RRAv1,
+    /// Weekly data (maximum values)
+    pub week_max: RRAv1,
+    /// Monthly data (average values)
+    pub month_avg: RRAv1,
+    /// Monthly data (maximum values)
+    pub month_max: RRAv1,
+    /// Yearly data (average values)
+    pub year_avg: RRAv1,
+    /// Yearly data (maximum values)
+    pub year_max: RRAv1,
+}
+
+impl RRDv1 {
+
+    pub fn from_raw(mut raw: &[u8]) -> Result<Self, std::io::Error> {
+
+        let expected_len = std::mem::size_of::<RRDv1>();
+
+        if raw.len() != expected_len {
+            let msg = format!("wrong data size ({} != {})", raw.len(), expected_len);
+            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+        }
+
+        let mut rrd: RRDv1 = unsafe { std::mem::zeroed() };
+        unsafe {
+            let rrd_slice = std::slice::from_raw_parts_mut(&mut rrd as *mut _ as *mut u8, expected_len);
+            raw.read_exact(rrd_slice)?;
+        }
+
+        if rrd.magic != PROXMOX_RRD_MAGIC_1_0 {
+            let msg = "wrong magic number".to_string();
+            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+        }
+
+        Ok(rrd)
+    }
+
+    pub fn to_rrd_v2(&self) -> Result<RRD, Error> {
+
+        let mut rra_list = Vec::new();
+
+        // old format v1:
+        //
+        // hour      1 min,   70 points
+        // day      30 min,   70 points
+        // week      3 hours, 70 points
+        // month    12 hours, 70 points
+        // year      1 week,  70 points
+        //
+        // new default for RRD v2:
+        //
+        // day      1 min,      1440 points
+        // month   30 min,      1440 points
+        // year   365 min (6h), 1440 points
+        // decade   1 week,      570 points
+
+        // Linear extrapolation
+        fn extrapolate_data(start: u64, reso: u64, factor: u64, data: Vec<Option<f64>>) -> (u64, u64, Vec<Option<f64>>) {
+
+            let mut new = Vec::new();
+
+            for i in 0..data.len() {
+                let mut next = i + 1;
+                if next >= data.len() { next = 0 };
+                let v = data[i];
+                let v1 = data[next];
+                match (v, v1) {
+                    (Some(v), Some(v1)) => {
+                        let diff = (v1 - v)/(factor as f64);
+                        for j in 0..factor {
+                            new.push(Some(v + diff*(j as f64)));
+                        }
+                    }
+                    (Some(v), None) => {
+                        new.push(Some(v));
+                        for _ in 0..factor-1 {
+                            new.push(None);
+                        }
+                    }
+                    (None, Some(v1)) => {
+                        for _ in 0..factor-1 {
+                            new.push(None);
+                        }
+                        new.push(Some(v1));
+                    }
+                    (None, None) => {
+                        for _ in 0..factor {
+                            new.push(None);
+                        }
+                    }
+                }
+            }
+
+            (start, reso/factor, new)
+        }
+
+        // Try to convert to new, higher capacity format
+
+        // compute daily average (merge old self.day_avg and self.hour_avg
+        let mut day_avg = RRA::new(CF::Average, 60, 1440);
+
+        let (start, reso, data) = self.day_avg.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 30, data);
+        day_avg.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.hour_avg.extract_data();
+        day_avg.insert_data(start, reso, data)?;
+
+        // compute daily maximum (merge old self.day_max and self.hour_max
+        let mut day_max = RRA::new(CF::Maximum, 60, 1440);
+
+        let (start, reso, data) = self.day_max.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 30, data);
+        day_max.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.hour_max.extract_data();
+        day_max.insert_data(start, reso, data)?;
+
+        // compute montly average (merge old self.month_avg,
+        // self.week_avg and self.day_avg)
+        let mut month_avg = RRA::new(CF::Average, 30*60, 1440);
+
+        let (start, reso, data) = self.month_avg.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 24, data);
+        month_avg.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.week_avg.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 6, data);
+        month_avg.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.day_avg.extract_data();
+        month_avg.insert_data(start, reso, data)?;
+
+        // compute montly maximum (merge old self.month_max,
+        // self.week_max and self.day_max)
+        let mut month_max = RRA::new(CF::Maximum, 30*60, 1440);
+
+        let (start, reso, data) = self.month_max.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 24, data);
+        month_max.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.week_max.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 6, data);
+        month_max.insert_data(start, reso, data)?;
+
+        let (start, reso, data) = self.day_max.extract_data();
+        month_max.insert_data(start, reso, data)?;
+
+        // compute yearly average (merge old self.year_avg)
+        let mut year_avg = RRA::new(CF::Average, 6*3600, 1440);
+
+        let (start, reso, data) = self.year_avg.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 28, data);
+        year_avg.insert_data(start, reso, data)?;
+
+        // compute yearly maximum (merge old self.year_avg)
+        let mut year_max = RRA::new(CF::Maximum, 6*3600, 1440);
+
+        let (start, reso, data) = self.year_max.extract_data();
+        let (start, reso, data) = extrapolate_data(start, reso, 28, data);
+        year_max.insert_data(start, reso, data)?;
+
+        // compute decade average (merge old self.year_avg)
+        let mut decade_avg = RRA::new(CF::Average, 7*86400, 570);
+        let (start, reso, data) = self.year_avg.extract_data();
+        decade_avg.insert_data(start, reso, data)?;
+
+        // compute decade maximum (merge old self.year_max)
+        let mut decade_max = RRA::new(CF::Maximum, 7*86400, 570);
+        let (start, reso, data) = self.year_max.extract_data();
+        decade_max.insert_data(start, reso, data)?;
+
+        rra_list.push(day_avg);
+        rra_list.push(day_max);
+        rra_list.push(month_avg);
+        rra_list.push(month_max);
+        rra_list.push(year_avg);
+        rra_list.push(year_max);
+        rra_list.push(decade_avg);
+        rra_list.push(decade_max);
+
+        // use values from hour_avg for source (all RRAv1 must have the same config)
+        let dst = if self.hour_avg.flags.contains(RRAFlags::DST_COUNTER) {
+            DST::Counter
+        } else if self.hour_avg.flags.contains(RRAFlags::DST_DERIVE) {
+            DST::Derive
+        } else {
+            DST::Gauge
+        };
+
+        let source = DataSource {
+            dst,
+            counter_value: f64::NAN,
+            last_update:  self.hour_avg.last_update, // IMPORTANT!
+        };
+        Ok(RRD {
+            source,
+            rra_list,
+        })
+    }
+}
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index b53076d7..4fe18f46 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -1,5 +1,6 @@
-use anyhow::Error;
+use anyhow::{bail, Error};
 use serde_json::{Value, json};
+use std::collections::BTreeMap;
 
 use proxmox_router::{Permission, Router};
 use proxmox_schema::api;
@@ -8,8 +9,6 @@ use pbs_api_types::{
     NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
 };
 
-use proxmox_rrd::rrd::RRD_DATA_ENTRIES;
-
 use crate::get_rrd_cache;
 
 pub fn create_value_from_rrd(
@@ -19,32 +18,44 @@ pub fn create_value_from_rrd(
     cf: RRDMode,
 ) -> Result<Value, Error> {
 
-    let mut result = Vec::new();
+    let mut result: Vec<Value> = Vec::new();
     let now = proxmox_time::epoch_f64();
 
     let rrd_cache = get_rrd_cache()?;
 
+    let mut timemap = BTreeMap::new();
+
+    let mut last_resolution = None;
+
     for name in list {
-        let (start, reso, list) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf) {
+        let (start, reso, data) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf)? {
             Some(result) => result,
             None => continue,
         };
 
+        if let Some(expected_resolution) = last_resolution  {
+            if reso != expected_resolution {
+                bail!("got unexpected RRD resolution ({} != {})", reso, expected_resolution);
+            }
+        } else {
+            last_resolution = Some(reso);
+        }
+
         let mut t = start;
-        for index in 0..RRD_DATA_ENTRIES {
-            if result.len() <= index {
-                if let Some(value) = list[index] {
-                    result.push(json!({ "time": t, *name: value }));
-                } else {
-                    result.push(json!({ "time": t }));
-                }
-            } else if let Some(value) = list[index] {
-                result[index][name] = value.into();
+
+        for index in 0..data.len() {
+            let entry = timemap.entry(t).or_insert(json!({ "time": t }));
+            if let Some(value) = data[index] {
+                entry[*name] = value.into();
             }
             t += reso;
         }
     }
 
+    for item in timemap.values() {
+        result.push(item.clone());
+    }
+
     Ok(result.into())
 }
 
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 40a9e5b4..2476fe97 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -134,8 +134,8 @@ pub fn datastore_status(
             RRDMode::Average,
         );
 
-        let total_res = get_rrd("total");
-        let used_res = get_rrd("used");
+        let total_res = get_rrd("total")?;
+        let used_res = get_rrd("used")?;
 
         if let (Some((start, reso, total_list)), Some((_, _, used_list))) = (total_res, used_res) {
             let mut usage_list: Vec<f64> = Vec::new();
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 0f0f6f59..2abc3eaa 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -24,7 +24,7 @@ use proxmox_router::{RpcEnvironment, RpcEnvironmentType, UserInformation};
 
 use pbs_tools::{task_log, task_warn};
 use pbs_datastore::DataStore;
-use proxmox_rrd::DST;
+use proxmox_rrd::rrd::DST;
 
 use proxmox_rest_server::{
     rotate_task_log_archive, extract_cookie , AuthError, ApiConfig, RestServer, RestEnvironment,
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (3 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
                   ` (10 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/Cargo.toml   |  2 --
 proxmox-rrd/src/cache.rs | 30 ++++++++----------------------
 proxmox-rrd/src/rrd.rs   | 13 ++++++++++---
 src/api2/node/rrd.rs     |  9 +++------
 src/api2/status.rs       |  9 ++-------
 src/lib.rs               | 32 +++++++++++++++++++++++++++++++-
 6 files changed, 54 insertions(+), 41 deletions(-)

diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index b3dd02c3..69a68530 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -17,5 +17,3 @@ serde_cbor = "0.11.1"
 proxmox = { version = "0.14.0" }
 proxmox-time = "1"
 proxmox-schema = { version = "1", features = [ "api-macro" ] }
-
-proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index f14837fc..5e359d9b 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -11,8 +11,6 @@ use nix::fcntl::OFlag;
 
 use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions};
 
-use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
-
 use crate::rrd::{DST, CF, RRD, RRA};
 
 const RRD_JOURNAL_NAME: &str = "rrd.journal";
@@ -280,35 +278,23 @@ impl RRDCache {
     }
 
     /// Extract data from cached RRD
+    ///
+    /// `start`: Start time. If not sepecified, we simply extract 10 data points.
+    /// `end`: End time. Default is to use the current time.
     pub fn extract_cached_data(
         &self,
         base: &str,
         name: &str,
-        now: f64,
-        timeframe: RRDTimeFrameResolution,
-        mode: RRDMode,
+        cf: CF,
+        resolution: u64,
+        start: Option<u64>,
+        end: Option<u64>,
     ) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
 
         let state = self.state.read().unwrap();
 
-        let cf = match mode {
-            RRDMode::Max => CF::Maximum,
-            RRDMode::Average => CF::Average,
-        };
-
-        let now = now as u64;
-
-        let (start, resolution) = match timeframe {
-            RRDTimeFrameResolution::Hour => (now - 3600, 60),
-            RRDTimeFrameResolution::Day => (now - 3600*24, 60),
-            RRDTimeFrameResolution::Week => (now - 3600*24*7, 30*60),
-            RRDTimeFrameResolution::Month => (now - 3600*24*30, 30*60),
-            RRDTimeFrameResolution::Year => (now - 3600*24*365, 6*60*60),
-            RRDTimeFrameResolution::Decade => (now - 10*3600*24*366, 7*86400),
-        };
-
         match state.rrd_map.get(&format!("{}/{}", base, name)) {
-            Some(rrd) => Ok(Some(rrd.extract_data(start, now, cf, resolution)?)),
+            Some(rrd) => Ok(Some(rrd.extract_data(cf, resolution, start, end)?)),
             None => Ok(None),
         }
     }
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 82fa5a3a..c97be96d 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -339,12 +339,15 @@ impl RRD {
     ///
     /// This selects the RRA with specified [CF] and (minimum)
     /// resolution, and extract data from `start` to `end`.
+    ///
+    /// `start`: Start time. If not sepecified, we simply extract 10 data points.
+    /// `end`: End time. Default is to use the current time.
     pub fn extract_data(
         &self,
-        start: u64,
-        end: u64,
         cf: CF,
         resolution: u64,
+        start: Option<u64>,
+        end: Option<u64>,
     ) -> Result<(u64, u64, Vec<Option<f64>>), Error> {
 
         let mut rra: Option<&RRA> = None;
@@ -362,7 +365,11 @@ impl RRD {
         }
 
         match rra {
-            Some(rra) => Ok(rra.extract_data(start, end, self.source.last_update)),
+            Some(rra) => {
+                let end = end.unwrap_or_else(|| proxmox_time::epoch_f64() as u64);
+                let start = start.unwrap_or(end - 10*rra.resolution);
+                Ok(rra.extract_data(start, end, self.source.last_update))
+            }
             None => bail!("unable to find RRA suitable ({:?}:{})", cf, resolution),
         }
     }
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index 4fe18f46..21a55bb7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -9,26 +9,23 @@ use pbs_api_types::{
     NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
 };
 
-use crate::get_rrd_cache;
+use crate::extract_rrd_data;
 
 pub fn create_value_from_rrd(
     basedir: &str,
     list: &[&str],
     timeframe: RRDTimeFrameResolution,
-    cf: RRDMode,
+    mode: RRDMode,
 ) -> Result<Value, Error> {
 
     let mut result: Vec<Value> = Vec::new();
-    let now = proxmox_time::epoch_f64();
-
-    let rrd_cache = get_rrd_cache()?;
 
     let mut timemap = BTreeMap::new();
 
     let mut last_resolution = None;
 
     for name in list {
-        let (start, reso, data) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf)? {
+        let (start, reso, data) = match extract_rrd_data(basedir, name, timeframe, mode)? {
             Some(result) => result,
             None => continue,
         };
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 2476fe97..9a72ce52 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -22,7 +22,7 @@ use pbs_datastore::DataStore;
 use pbs_config::CachedUserInfo;
 
 use crate::tools::statistics::{linear_regression};
-use crate::get_rrd_cache;
+use crate::extract_rrd_data;
 
 #[api(
     returns: {
@@ -90,8 +90,6 @@ pub fn datastore_status(
 
     let mut list = Vec::new();
 
-    let rrd_cache = get_rrd_cache()?;
-
     for (store, (_, _)) in &config.sections {
         let user_privs = user_info.lookup_privs(&auth_id, &["datastore", &store]);
         let allowed = (user_privs & (PRIV_DATASTORE_AUDIT| PRIV_DATASTORE_BACKUP)) != 0;
@@ -123,13 +121,10 @@ pub fn datastore_status(
         });
 
         let rrd_dir = format!("datastore/{}", store);
-        let now = proxmox_time::epoch_f64();
-
 
-        let get_rrd = |what: &str| rrd_cache.extract_cached_data(
+        let get_rrd = |what: &str| extract_rrd_data(
             &rrd_dir,
             what,
-            now,
             RRDTimeFrameResolution::Month,
             RRDMode::Average,
         );
diff --git a/src/lib.rs b/src/lib.rs
index a1ac23bf..6745627a 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -10,9 +10,10 @@ use anyhow::{format_err, Error};
 
 use proxmox::tools::fs::CreateOptions;
 
+use pbs_api_types::{RRDMode, RRDTimeFrameResolution};
 use pbs_buildcfg::configdir;
 use pbs_tools::cert::CertInfo;
-use proxmox_rrd::RRDCache;
+use proxmox_rrd::{rrd::CF, RRDCache};
 
 #[macro_use]
 pub mod tools;
@@ -78,3 +79,32 @@ pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
 
     Ok(RRD_CACHE.get().unwrap())
 }
+
+/// Extracts data for the specified time frame from from RRD cache
+pub fn extract_rrd_data(
+    basedir: &str,
+    name: &str,
+    timeframe: RRDTimeFrameResolution,
+    mode: RRDMode,
+) ->  Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
+
+    let end = proxmox_time::epoch_f64() as u64;
+
+    let (start, resolution) = match timeframe {
+        RRDTimeFrameResolution::Hour => (end - 3600, 60),
+        RRDTimeFrameResolution::Day => (end - 3600*24, 60),
+        RRDTimeFrameResolution::Week => (end - 3600*24*7, 30*60),
+        RRDTimeFrameResolution::Month => (end - 3600*24*30, 30*60),
+        RRDTimeFrameResolution::Year => (end - 3600*24*365, 6*60*60),
+        RRDTimeFrameResolution::Decade => (end - 10*3600*24*366, 7*86400),
+    };
+
+    let cf = match mode {
+        RRDMode::Max => CF::Maximum,
+        RRDMode::Average => CF::Average,
+    };
+
+    let rrd_cache = get_rrd_cache()?;
+
+    rrd_cache.extract_cached_data(basedir, name, cf, resolution, Some(start), Some(end))
+}
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (4 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
                   ` (9 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/rrd.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index c97be96d..328ac9f2 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -231,7 +231,7 @@ impl RRA {
 
         let mut list = Vec::new();
 
-        let rrd_end = reso*(last_update/reso);
+        let rrd_end = reso*(last_update/reso + 1);
         let rrd_start = rrd_end.saturating_sub(reso*num_entries);
 
         let mut t = start;
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (5 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
                   ` (8 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features
---
 Cargo.toml                       |  2 --
 Makefile                         |  1 -
 pbs-api-types/Cargo.toml         |  1 -
 pbs-api-types/src/lib.rs         | 30 ++++++++++++++++++++++++++++-
 proxmox-rrd-api-types/Cargo.toml | 11 -----------
 proxmox-rrd-api-types/src/lib.rs | 33 --------------------------------
 proxmox-rrd/src/rrd_v1.rs        |  2 +-
 src/api2/admin/datastore.rs      |  6 +++---
 src/api2/node/rrd.rs             |  8 ++++----
 src/api2/status.rs               |  4 ++--
 src/lib.rs                       | 16 ++++++++--------
 11 files changed, 47 insertions(+), 67 deletions(-)
 delete mode 100644 proxmox-rrd-api-types/Cargo.toml
 delete mode 100644 proxmox-rrd-api-types/src/lib.rs

diff --git a/Cargo.toml b/Cargo.toml
index d2e3fa1f..772588f6 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -27,7 +27,6 @@ members = [
     "pbs-fuse-loop",
     "pbs-runtime",
     "proxmox-rest-server",
-    "proxmox-rrd-api-types",
     "proxmox-rrd",
     "proxmox-systemd",
     "pbs-tape",
@@ -119,7 +118,6 @@ pbs-config = { path = "pbs-config" }
 pbs-datastore = { path = "pbs-datastore" }
 pbs-runtime = { path = "pbs-runtime" }
 proxmox-rest-server = { path = "proxmox-rest-server" }
-proxmox-rrd-api-types = { path = "proxmox-rrd-api-types" }
 proxmox-rrd = { path = "proxmox-rrd" }
 proxmox-systemd = { path = "proxmox-systemd" }
 pbs-tools = { path = "pbs-tools" }
diff --git a/Makefile b/Makefile
index 6b098aa1..d6951c9a 100644
--- a/Makefile
+++ b/Makefile
@@ -40,7 +40,6 @@ SUBCRATES := \
 	pbs-fuse-loop \
 	pbs-runtime \
 	proxmox-rest-server \
-	proxmox-rrd-api-types \
 	proxmox-rrd \
 	proxmox-systemd \
 	pbs-tape \
diff --git a/pbs-api-types/Cargo.toml b/pbs-api-types/Cargo.toml
index 2a51bd3a..11644399 100644
--- a/pbs-api-types/Cargo.toml
+++ b/pbs-api-types/Cargo.toml
@@ -20,6 +20,5 @@ proxmox-schema = { version = "1.0.0", features = [ "api-macro" ] }
 proxmox-time = "1.0.0"
 proxmox-uuid = { version = "1.0.0", features = [ "serde" ] }
 
-proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
 proxmox-systemd = { path = "../proxmox-systemd" }
 pbs-tools = { path = "../pbs-tools" }
diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index cdf765a1..96ac657b 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -424,4 +424,32 @@ pub const NODE_TASKS_LIST_TASKS_RETURN_TYPE: ReturnType = ReturnType {
     ).schema(),
 };
 
-pub use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
+#[api()]
+#[derive(Copy, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+/// RRD consolidation mode
+pub enum RRDMode {
+    /// Maximum
+    Max,
+    /// Average
+    Average,
+}
+
+#[api()]
+#[derive(Copy, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+/// RRD time frame
+pub enum RRDTimeFrame {
+    /// Hour
+    Hour,
+    /// Day
+    Day,
+    /// Week
+    Week,
+    /// Month
+    Month,
+    /// Year
+    Year,
+    /// Decade (10 years)
+    Decade,
+}
diff --git a/proxmox-rrd-api-types/Cargo.toml b/proxmox-rrd-api-types/Cargo.toml
deleted file mode 100644
index 816f7fde..00000000
--- a/proxmox-rrd-api-types/Cargo.toml
+++ /dev/null
@@ -1,11 +0,0 @@
-[package]
-name = "proxmox-rrd-api-types"
-version = "0.1.0"
-authors = ["Proxmox Support Team <support@proxmox.com>"]
-edition = "2018"
-description = "API type definitions for proxmox-rrd crate."
-
-
-[dependencies]
-serde = { version = "1.0", features = ["derive"] }
-proxmox-schema = { version = "1", features = ["api-macro"] }
diff --git a/proxmox-rrd-api-types/src/lib.rs b/proxmox-rrd-api-types/src/lib.rs
deleted file mode 100644
index 32601477..00000000
--- a/proxmox-rrd-api-types/src/lib.rs
+++ /dev/null
@@ -1,33 +0,0 @@
-use serde::{Deserialize, Serialize};
-
-use proxmox_schema::api;
-
-#[api()]
-#[derive(Copy, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "UPPERCASE")]
-/// RRD consolidation mode
-pub enum RRDMode {
-    /// Maximum
-    Max,
-    /// Average
-    Average,
-}
-
-#[api()]
-#[derive(Copy, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-/// RRD time frame resolution
-pub enum RRDTimeFrameResolution {
-    /// Hour
-    Hour,
-    /// Day
-    Day,
-    /// Week
-    Week,
-    /// Month
-    Month,
-    /// Year
-    Year,
-    /// Decade (10 years)
-    Decade,
-}
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
index 919896f0..511b510b 100644
--- a/proxmox-rrd/src/rrd_v1.rs
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -36,7 +36,7 @@ bitflags!{
 pub struct RRAv1 {
     /// Defined the data soure type and consolidation function
     pub flags: RRAFlags,
-    /// Resulution (seconds) from [RRDTimeFrameResolution]
+    /// Resulution (seconds)
     pub resolution: u64,
     /// Last update time (epoch)
     pub last_update: f64,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7e98b4a9..fb5647ce 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -28,7 +28,7 @@ use pxar::EntryKind;
 use pbs_api_types::{ Authid, BackupContent, Counts, CryptMode,
     DataStoreListItem, GarbageCollectionStatus, GroupListItem,
     SnapshotListItem, SnapshotVerifyState, PruneOptions,
-    DataStoreStatus, RRDMode, RRDTimeFrameResolution,
+    DataStoreStatus, RRDMode, RRDTimeFrame,
     BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_TIME_SCHEMA,
     BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA,
     IGNORE_VERIFIED_BACKUPS_SCHEMA, UPID_SCHEMA,
@@ -1537,7 +1537,7 @@ pub fn pxar_file_download(
                 schema: DATASTORE_SCHEMA,
             },
             timeframe: {
-                type: RRDTimeFrameResolution,
+                type: RRDTimeFrame,
             },
             cf: {
                 type: RRDMode,
@@ -1551,7 +1551,7 @@ pub fn pxar_file_download(
 /// Read datastore stats
 pub fn get_rrd_stats(
     store: String,
-    timeframe: RRDTimeFrameResolution,
+    timeframe: RRDTimeFrame,
     cf: RRDMode,
     _param: Value,
 ) -> Result<Value, Error> {
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index 21a55bb7..9b6780b7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -6,7 +6,7 @@ use proxmox_router::{Permission, Router};
 use proxmox_schema::api;
 
 use pbs_api_types::{
-    NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
+    NODE_SCHEMA, RRDMode, RRDTimeFrame, PRIV_SYS_AUDIT,
 };
 
 use crate::extract_rrd_data;
@@ -14,7 +14,7 @@ use crate::extract_rrd_data;
 pub fn create_value_from_rrd(
     basedir: &str,
     list: &[&str],
-    timeframe: RRDTimeFrameResolution,
+    timeframe: RRDTimeFrame,
     mode: RRDMode,
 ) -> Result<Value, Error> {
 
@@ -63,7 +63,7 @@ pub fn create_value_from_rrd(
                 schema: NODE_SCHEMA,
             },
             timeframe: {
-                type: RRDTimeFrameResolution,
+                type: RRDTimeFrame,
             },
             cf: {
                 type: RRDMode,
@@ -76,7 +76,7 @@ pub fn create_value_from_rrd(
 )]
 /// Read node stats
 fn get_node_stats(
-    timeframe: RRDTimeFrameResolution,
+    timeframe: RRDTimeFrame,
     cf: RRDMode,
     _param: Value,
 ) -> Result<Value, Error> {
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 9a72ce52..a3456ad4 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -14,7 +14,7 @@ use proxmox_router::{
 use proxmox_router::list_subdirs_api_method;
 
 use pbs_api_types::{
-    Authid, DATASTORE_SCHEMA, RRDMode, RRDTimeFrameResolution,
+    Authid, DATASTORE_SCHEMA, RRDMode, RRDTimeFrame,
     PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
 };
 
@@ -125,7 +125,7 @@ pub fn datastore_status(
         let get_rrd = |what: &str| extract_rrd_data(
             &rrd_dir,
             what,
-            RRDTimeFrameResolution::Month,
+            RRDTimeFrame::Month,
             RRDMode::Average,
         );
 
diff --git a/src/lib.rs b/src/lib.rs
index 6745627a..a5a28190 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -10,7 +10,7 @@ use anyhow::{format_err, Error};
 
 use proxmox::tools::fs::CreateOptions;
 
-use pbs_api_types::{RRDMode, RRDTimeFrameResolution};
+use pbs_api_types::{RRDMode, RRDTimeFrame};
 use pbs_buildcfg::configdir;
 use pbs_tools::cert::CertInfo;
 use proxmox_rrd::{rrd::CF, RRDCache};
@@ -84,19 +84,19 @@ pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
 pub fn extract_rrd_data(
     basedir: &str,
     name: &str,
-    timeframe: RRDTimeFrameResolution,
+    timeframe: RRDTimeFrame,
     mode: RRDMode,
 ) ->  Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
 
     let end = proxmox_time::epoch_f64() as u64;
 
     let (start, resolution) = match timeframe {
-        RRDTimeFrameResolution::Hour => (end - 3600, 60),
-        RRDTimeFrameResolution::Day => (end - 3600*24, 60),
-        RRDTimeFrameResolution::Week => (end - 3600*24*7, 30*60),
-        RRDTimeFrameResolution::Month => (end - 3600*24*30, 30*60),
-        RRDTimeFrameResolution::Year => (end - 3600*24*365, 6*60*60),
-        RRDTimeFrameResolution::Decade => (end - 10*3600*24*366, 7*86400),
+        RRDTimeFrame::Hour => (end - 3600, 60),
+        RRDTimeFrame::Day => (end - 3600*24, 60),
+        RRDTimeFrame::Week => (end - 3600*24*7, 30*60),
+        RRDTimeFrame::Month => (end - 3600*24*30, 30*60),
+        RRDTimeFrame::Year => (end - 3600*24*365, 6*60*60),
+        RRDTimeFrame::Decade => (end - 10*3600*24*366, 7*86400),
     };
 
     let cf = match mode {
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (6 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
                   ` (7 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/rrd.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 328ac9f2..7a9ce94a 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -51,6 +51,8 @@ pub enum CF {
     Maximum,
     /// Minimum
     Minimum,
+    /// Use the last value
+    Last,
 }
 
 #[derive(Serialize, Deserialize)]
@@ -209,6 +211,7 @@ impl RRA {
             let new_value = match self.cf {
                 CF::Maximum => if last_value > value { last_value } else { value },
                 CF::Minimum => if last_value < value { last_value } else { value },
+                CF::Last => value,
                 CF::Average => {
                     (last_value*(self.last_count as f64))/(new_count as f64)
                         + value/(new_count as f64)
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup)
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (7 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
                   ` (6 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/cache.rs | 37 ++++++++++++++++++-------------------
 1 file changed, 18 insertions(+), 19 deletions(-)

diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index 5e359d9b..5225bd49 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -194,15 +194,8 @@ impl RRDCache {
                 path.push(&entry.rel_path);
                 create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
 
-                let mut rrd = match RRD::load(&path) {
-                    Ok(rrd) => rrd,
-                    Err(err) => {
-                        if err.kind() != std::io::ErrorKind::NotFound {
-                            log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
-                        }
-                        Self::create_default_rrd(entry.dst)
-                    },
-                };
+                let mut rrd = Self::load_rrd(&path, entry.dst);
+
                 if entry.time > get_last_update(&entry.rel_path, &rrd) {
                     rrd.update(entry.time, entry.value);
                 }
@@ -235,7 +228,19 @@ impl RRDCache {
         Ok(())
     }
 
-    /// Update data in RAM and write file back to disk (if `save` is set)
+    fn load_rrd(path: &Path, dst: DST) -> RRD {
+        match RRD::load(path) {
+            Ok(rrd) => rrd,
+            Err(err) => {
+                if err.kind() != std::io::ErrorKind::NotFound {
+                    log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
+                }
+                Self::create_default_rrd(dst)
+            },
+        }
+    }
+
+    /// Update data in RAM and write file back to disk (journal)
     pub fn update_value(
         &self,
         rel_path: &str,
@@ -261,15 +266,9 @@ impl RRDCache {
             let mut path = self.basedir.clone();
             path.push(rel_path);
             create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
-            let mut rrd = match RRD::load(&path) {
-                Ok(rrd) => rrd,
-                Err(err) => {
-                    if err.kind() != std::io::ErrorKind::NotFound {
-                        log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
-                    }
-                    Self::create_default_rrd(dst)
-                },
-            };
+
+            let mut rrd = Self::load_rrd(&path, dst);
+
             rrd.update(now, value);
             state.rrd_map.insert(rel_path.into(), rrd);
         }
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (8 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
                   ` (5 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/Cargo.toml     |   1 +
 proxmox-rrd/src/bin/rrd.rs | 215 +++++++++++++++++++++++++++++++++++++
 2 files changed, 216 insertions(+)
 create mode 100644 proxmox-rrd/src/bin/rrd.rs

diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index 69a68530..e3ab5380 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -16,4 +16,5 @@ serde_cbor = "0.11.1"
 
 proxmox = { version = "0.14.0" }
 proxmox-time = "1"
+proxmox-router = "1"
 proxmox-schema = { version = "1", features = [ "api-macro" ] }
diff --git a/proxmox-rrd/src/bin/rrd.rs b/proxmox-rrd/src/bin/rrd.rs
new file mode 100644
index 00000000..fdb61ffd
--- /dev/null
+++ b/proxmox-rrd/src/bin/rrd.rs
@@ -0,0 +1,215 @@
+//! RRD toolkit - create/manage/update proxmox RRD (v2) file
+
+use std::path::PathBuf;
+
+use anyhow::{bail, Error};
+use serde::{Serialize, Deserialize};
+
+use proxmox_router::RpcEnvironment;
+use proxmox_router::cli::{run_cli_command, CliCommand, CliCommandMap, CliEnvironment};
+use proxmox_schema::{api, parse_property_string};
+use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema};
+
+use proxmox::tools::fs::CreateOptions;
+
+use proxmox_rrd::rrd::{CF, DST, RRA, RRD};
+
+pub const RRA_CONFIG_STRING_SCHEMA: Schema = StringSchema::new(
+    "RRA configuration")
+    .format(&ApiStringFormat::PropertyString(&RRAConfig::API_SCHEMA))
+    .schema();
+
+#[api(
+    properties: {},
+    default_key: "cf",
+)]
+#[derive(Debug, Serialize, Deserialize)]
+/// RRA configuration
+pub struct RRAConfig {
+    /// Time resolution
+    pub r: u64,
+    pub cf: CF,
+    /// Number of data points
+    pub n: u64,
+}
+
+#[api(
+   input: {
+       properties: {
+          path: {
+              description: "The filename."
+          },
+       },
+   },
+)]
+/// Dump the RRDB database in JSON format
+pub fn dump_rrdb(path: String) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+    serde_json::to_writer_pretty(std::io::stdout(), &rrd)?;
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+           time: {
+               description: "Update time.",
+               optional: true,
+           },
+           value: {
+               description: "Update value.",
+           },
+       },
+   },
+)]
+/// Update the RRDB database
+pub fn update_rrdb(
+    path: String,
+    time: Option<u64>,
+    value: f64,
+) -> Result<(), Error> {
+
+    let path = PathBuf::from(path);
+
+    let time = time.map(|v| v as f64)
+        .unwrap_or_else(proxmox_time::epoch_f64);
+
+    let mut rrd = RRD::load(&path)?;
+    rrd.update(time, value);
+
+    rrd.save(&path, CreateOptions::new())?;
+
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+           cf: {
+               type: CF,
+           },
+           resolution: {
+               description: "Time resulution",
+           },
+           start: {
+               description: "Start time. If not sepecified, we simply extract 10 data points.",
+               optional: true,
+           },
+           end: {
+               description: "End time (Unix Epoch). Default is the last update time.",
+               optional: true,
+           },
+       },
+   },
+)]
+/// Fetch data from the RRDB database
+pub fn fetch_rrdb(
+    path: String,
+    cf: CF,
+    resolution: u64,
+    start: Option<u64>,
+    end: Option<u64>,
+) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+
+    let data = rrd.extract_data(cf, resolution, start, end)?;
+
+    println!("{}", serde_json::to_string_pretty(&data)?);
+
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+           dst: {
+               type: DST,
+           },
+           path: {
+               description: "The filename to create."
+           },
+           rra: {
+               description: "Configuration of contained RRAs.",
+               type: Array,
+               items: {
+                   schema:  RRA_CONFIG_STRING_SCHEMA,
+               }
+           },
+       },
+   },
+)]
+/// Create a new RRDB database file
+pub fn create_rrdb(
+    dst: DST,
+    path: String,
+    rra: Vec<String>,
+) -> Result<(), Error> {
+
+    let mut rra_list = Vec::new();
+
+    for item in rra.iter() {
+        let rra: RRAConfig = serde_json::from_value(
+            parse_property_string(item, &RRAConfig::API_SCHEMA)?
+        )?;
+        println!("GOT {:?}", rra);
+        rra_list.push(RRA::new(rra.cf, rra.r, rra.n as usize));
+    }
+
+    let path = PathBuf::from(path);
+
+    let rrd = RRD::new(dst, rra_list);
+
+    rrd.save(&path, CreateOptions::new())?;
+
+    Ok(())
+}
+
+
+fn main() -> Result<(), Error> {
+
+    let uid = nix::unistd::Uid::current();
+
+    let username = match nix::unistd::User::from_uid(uid)? {
+        Some(user) => user.name,
+        None => bail!("unable to get user name"),
+    };
+
+    let cmd_def = CliCommandMap::new()
+        .insert(
+            "create",
+            CliCommand::new(&API_METHOD_CREATE_RRDB)
+                .arg_param(&["path"])
+        )
+        .insert(
+            "update",
+            CliCommand::new(&API_METHOD_UPDATE_RRDB)
+                .arg_param(&["path"])
+        )
+        .insert(
+            "fetch",
+            CliCommand::new(&API_METHOD_FETCH_RRDB)
+                .arg_param(&["path"])
+        )
+        .insert(
+            "dump",
+            CliCommand::new(&API_METHOD_DUMP_RRDB)
+                .arg_param(&["path"])
+        )
+        ;
+
+    let mut rpcenv = CliEnvironment::new();
+    rpcenv.set_auth_id(Some(format!("{}@pam", username)));
+
+    run_cli_command(cmd_def, rpcenv, None);
+
+    Ok(())
+
+}
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (9 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
                   ` (4 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

Modulo is very slow, so we try to avoid it inside loops.
---
 proxmox-rrd/src/rrd.rs | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 7a9ce94a..54cb8b48 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -153,8 +153,7 @@ impl RRA {
             if let Some(v) = data[i] {
                 self.data[index] = v;
             }
-            index += 1;
-            if index >= self.data.len() { index = 0; }
+            index += 1; if index >= self.data.len() { index = 0; }
         }
         Ok(())
     }
@@ -171,7 +170,7 @@ impl RRA {
         let mut index = ((t/reso) % num_entries) as usize;
         for _ in 0..num_entries {
             t += reso;
-            index = (index + 1) % (num_entries as usize);
+            index += 1; if index >= self.data.len() { index = 0; }
             if t < min_time {
                 self.data[index] = f64::NAN;
             } else {
@@ -251,7 +250,8 @@ impl RRA {
                     list.push(Some(value));
                 }
             }
-            t += reso; index = (index + 1) % (num_entries as usize);
+            t += reso;
+            index += 1; if index >= self.data.len() { index = 0; }
         }
 
         (start, reso, list)
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time()
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (10 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
                   ` (3 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/rrd.rs | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 54cb8b48..12d2ec8c 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -135,6 +135,14 @@ impl RRA {
         }
     }
 
+    fn slot_end_time(&self, time: u64) -> u64 {
+        self.resolution*(time/self.resolution + 1)
+    }
+
+    fn slot(&self, time: u64) -> usize {
+        ((time/self.resolution) as usize) % self.data.len()
+    }
+
     // directly overwrite data slots
     // the caller need to set last_update value on the DataSource manually.
     pub(crate) fn insert_data(
@@ -146,8 +154,8 @@ impl RRA {
         if resolution != self.resolution {
             bail!("inser_data failed: got wrong resolution");
         }
-        let num_entries = self.data.len() as u64;
-        let mut index = ((start/self.resolution) % num_entries) as usize;
+
+        let mut index = self.slot(start);
 
         for i in 0..data.len() {
             if let Some(v) = data[i] {
@@ -167,7 +175,9 @@ impl RRA {
         let min_time = epoch - num_entries*reso;
         let min_time = (min_time/reso + 1)*reso;
         let mut t = last_update.saturating_sub(num_entries*reso);
-        let mut index = ((t/reso) % num_entries) as usize;
+
+        let mut index = self.slot(t);
+
         for _ in 0..num_entries {
             t += reso;
             index += 1; if index >= self.data.len() { index = 0; }
@@ -183,12 +193,11 @@ impl RRA {
         let epoch = time as u64;
         let last_update = last_update as u64;
         let reso = self.resolution;
-        let num_entries = self.data.len() as u64;
 
-        let index = ((epoch/reso) % num_entries) as usize;
-        let last_index = ((last_update/reso) % num_entries) as usize;
+        let index = self.slot(epoch);
+        let last_index = self.slot(last_update);
 
-        if (epoch - (last_update as u64)) > reso || index != last_index {
+        if (epoch - last_update) > reso || index != last_index {
             self.last_count = 0;
         }
 
@@ -233,11 +242,11 @@ impl RRA {
 
         let mut list = Vec::new();
 
-        let rrd_end = reso*(last_update/reso + 1);
+        let rrd_end = self.slot_end_time(last_update);
         let rrd_start = rrd_end.saturating_sub(reso*num_entries);
 
         let mut t = start;
-        let mut index = ((t/reso) % num_entries) as usize;
+        let mut index = self.slot(t);
         for _ in 0..num_entries {
             if t > end { break; };
             if t < rrd_start || t > rrd_end {
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (11 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
                   ` (2 subsequent siblings)
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/rrd.rs | 45 +++++++++++++++++++++++++-----------------
 1 file changed, 27 insertions(+), 18 deletions(-)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 12d2ec8c..7901fe39 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -13,7 +13,7 @@
 
 use std::path::Path;
 
-use anyhow::{bail, Error};
+use anyhow::{bail, format_err, Error};
 
 use serde::{Serialize, Deserialize};
 
@@ -77,6 +77,9 @@ impl DataSource {
     }
 
     fn compute_new_value(&mut self, time: f64, mut value: f64) -> Result<f64, Error> {
+        if time < 0.0 {
+            bail!("got negative time");
+        }
         if time <= self.last_update {
             bail!("time in past ({} < {})", time, self.last_update);
         }
@@ -286,30 +289,36 @@ impl RRD {
 
     }
 
-    /// Load data from a file
-    pub fn load(path: &Path) -> Result<Self, std::io::Error> {
-        let raw = std::fs::read(path)?;
+    fn from_raw(raw: &[u8]) -> Result<Self, Error> {
         if raw.len() < 8 {
-            let msg = format!("not an rrd file - file is too small ({})", raw.len());
-            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+            bail!("not an rrd file - file is too small ({})", raw.len());
         }
 
-        if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
+        let rrd = if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
             let v1 = rrd_v1::RRDv1::from_raw(&raw)?;
             v1.to_rrd_v2()
-                .map_err(|err| {
-                    let msg = format!("unable to convert from old V1 format - {}", err);
-                    std::io::Error::new(std::io::ErrorKind::Other, msg)
-                })
+                .map_err(|err| format_err!("unable to convert from old V1 format - {}", err))?
         } else if raw[0..8] == PROXMOX_RRD_MAGIC_2_0 {
             serde_cbor::from_slice(&raw[8..])
-                .map_err(|err| {
-                    let msg = format!("unable to decode RRD file - {}", err);
-                    std::io::Error::new(std::io::ErrorKind::Other, msg)
-                })
-         } else {
-            let msg = format!("not an rrd file - unknown magic number");
-            return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+                .map_err(|err| format_err!("unable to decode RRD file - {}", err))?
+        } else {
+            bail!("not an rrd file - unknown magic number");
+        };
+
+        if rrd.source.last_update < 0.0 {
+            bail!("rrd file has negative last_update time");
+        }
+
+        Ok(rrd)
+    }
+
+    /// Load data from a file
+    pub fn load(path: &Path) -> Result<Self, std::io::Error> {
+        let raw = std::fs::read(path)?;
+
+        match Self::from_raw(&raw) {
+            Ok(rrd) => Ok(rrd),
+            Err(err) =>  Err(std::io::Error::new(std::io::ErrorKind::Other, err.to_string())),
         }
     }
 
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (12 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
  2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/rrd.rs    | 25 +++++++++++++++----------
 proxmox-rrd/src/rrd_v1.rs |  2 +-
 2 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 7901fe39..72769bfd 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -63,7 +63,7 @@ pub struct DataSource {
     pub last_update: f64,
     /// Stores the last value, used to compute differential value for
     /// derive/counters
-    pub counter_value: f64,
+    pub last_value: f64,
 }
 
 impl DataSource {
@@ -72,7 +72,7 @@ impl DataSource {
         Self {
             dst,
             last_update: 0.0,
-            counter_value: f64::NAN,
+            last_value: f64::NAN,
         }
     }
 
@@ -94,23 +94,24 @@ impl DataSource {
         if is_counter || self.dst == DST::Derive {
             let time_diff = time - self.last_update;
 
-            let diff = if self.counter_value.is_nan() {
+            let diff = if self.last_value.is_nan() {
                 0.0
             } else if is_counter && value < 0.0 {
                 bail!("got negative value for counter");
-            } else if is_counter && value < self.counter_value {
+            } else if is_counter && value < self.last_value {
                 // Note: We do not try automatic overflow corrections, but
-                // we update counter_value anyways, so that we can compute the diff
+                // we update last_value anyways, so that we can compute the diff
                 // next time.
-                self.counter_value = value;
+                self.last_value = value;
                 bail!("conter overflow/reset detected");
             } else {
-                value - self.counter_value
+                value - self.last_value
             };
-            self.counter_value = value;
             value = diff/time_diff;
         }
 
+        self.last_value = value;
+
         Ok(value)
     }
 
@@ -138,11 +139,15 @@ impl RRA {
         }
     }
 
-    fn slot_end_time(&self, time: u64) -> u64 {
+    pub fn slot_end_time(&self, time: u64) -> u64 {
         self.resolution*(time/self.resolution + 1)
     }
 
-    fn slot(&self, time: u64) -> usize {
+    pub fn slot_start_time(&self, time: u64) -> u64 {
+        self.resolution*(time/self.resolution)
+    }
+
+    pub fn slot(&self, time: u64) -> usize {
         ((time/self.resolution) as usize) % self.data.len()
     }
 
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
index 511b510b..7e4b97c2 100644
--- a/proxmox-rrd/src/rrd_v1.rs
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -285,7 +285,7 @@ impl RRDv1 {
 
         let source = DataSource {
             dst,
-            counter_value: f64::NAN,
+            last_value: f64::NAN,
             last_update:  self.hour_avg.last_update, // IMPORTANT!
         };
         Ok(RRD {
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (13 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
@ 2021-10-13  8:24 ` Dietmar Maurer
  2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht
  15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13  8:24 UTC (permalink / raw)
  To: pbs-devel

---
 proxmox-rrd/src/bin/rrd.rs | 229 ++++++++++++++++++++++++++++++++++---
 proxmox-rrd/src/rrd.rs     |   4 +-
 2 files changed, 215 insertions(+), 18 deletions(-)

diff --git a/proxmox-rrd/src/bin/rrd.rs b/proxmox-rrd/src/bin/rrd.rs
index fdb61ffd..bf2817c4 100644
--- a/proxmox-rrd/src/bin/rrd.rs
+++ b/proxmox-rrd/src/bin/rrd.rs
@@ -4,16 +4,22 @@ use std::path::PathBuf;
 
 use anyhow::{bail, Error};
 use serde::{Serialize, Deserialize};
+use serde_json::json;
 
 use proxmox_router::RpcEnvironment;
 use proxmox_router::cli::{run_cli_command, CliCommand, CliCommandMap, CliEnvironment};
 use proxmox_schema::{api, parse_property_string};
-use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema};
+use proxmox_schema::{ApiStringFormat, ApiType, IntegerSchema, Schema, StringSchema};
 
 use proxmox::tools::fs::CreateOptions;
 
 use proxmox_rrd::rrd::{CF, DST, RRA, RRD};
 
+pub const RRA_INDEX_SCHEMA: Schema = IntegerSchema::new(
+    "Index of the RRA.")
+    .minimum(0)
+    .schema();
+
 pub const RRA_CONFIG_STRING_SCHEMA: Schema = StringSchema::new(
     "RRA configuration")
     .format(&ApiStringFormat::PropertyString(&RRAConfig::API_SCHEMA))
@@ -42,11 +48,36 @@ pub struct RRAConfig {
        },
    },
 )]
-/// Dump the RRDB database in JSON format
-pub fn dump_rrdb(path: String) -> Result<(), Error> {
+/// Dump the RRD file in JSON format
+pub fn dump_rrd(path: String) -> Result<(), Error> {
 
     let rrd = RRD::load(&PathBuf::from(path))?;
     serde_json::to_writer_pretty(std::io::stdout(), &rrd)?;
+    println!("");
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+          path: {
+              description: "The filename."
+          },
+       },
+   },
+)]
+/// RRD file information
+pub fn rrd_info(path: String) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+
+    println!("DST: {:?}", rrd.source.dst);
+
+    for (i, rra) in rrd.rra_list.iter().enumerate() {
+        // use RRAConfig property string format
+        println!("RRA[{}]: {:?},r={},n={}", i, rra.cf, rra.resolution, rra.data.len());
+    }
+
     Ok(())
 }
 
@@ -66,8 +97,8 @@ pub fn dump_rrdb(path: String) -> Result<(), Error> {
        },
    },
 )]
-/// Update the RRDB database
-pub fn update_rrdb(
+/// Update the RRD database
+pub fn update_rrd(
     path: String,
     time: Option<u64>,
     value: f64,
@@ -109,8 +140,8 @@ pub fn update_rrdb(
        },
    },
 )]
-/// Fetch data from the RRDB database
-pub fn fetch_rrdb(
+/// Fetch data from the RRD file
+pub fn fetch_rrd(
     path: String,
     cf: CF,
     resolution: u64,
@@ -127,6 +158,80 @@ pub fn fetch_rrdb(
     Ok(())
 }
 
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+           "rra-index": {
+               schema: RRA_INDEX_SCHEMA,
+           },
+       },
+   },
+)]
+/// Return the Unix timestamp of the first time slot inside the
+/// specified RRA (slot start time)
+pub fn first_update_time(
+    path: String,
+    rra_index: usize,
+) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+
+    if rra_index >= rrd.rra_list.len() {
+        bail!("rra-index is out of range");
+    }
+    let rra = &rrd.rra_list[rra_index];
+    let duration =  (rra.data.len() as u64)*rra.resolution;
+    let first = rra.slot_start_time((rrd.source.last_update as u64).saturating_sub(duration));
+
+    println!("{}", first);
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+       },
+   },
+)]
+/// Return the Unix timestamp of the last update
+pub fn last_update_time(path: String) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+
+    println!("{}", rrd.source.last_update);
+    Ok(())
+}
+
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+       },
+   },
+)]
+/// Return the time and value from the last update
+pub fn last_update(path: String) -> Result<(), Error> {
+
+    let rrd = RRD::load(&PathBuf::from(path))?;
+
+    let result = json!({
+        "time": rrd.source.last_update,
+        "value": rrd.source.last_value,
+    });
+
+    println!("{}", serde_json::to_string_pretty(&result)?);
+
+    Ok(())
+}
+
 #[api(
    input: {
        properties: {
@@ -146,8 +251,8 @@ pub fn fetch_rrdb(
        },
    },
 )]
-/// Create a new RRDB database file
-pub fn create_rrdb(
+/// Create a new RRD file
+pub fn create_rrd(
     dst: DST,
     path: String,
     rra: Vec<String>,
@@ -172,6 +277,64 @@ pub fn create_rrdb(
     Ok(())
 }
 
+#[api(
+   input: {
+       properties: {
+           path: {
+               description: "The filename."
+           },
+           "rra-index": {
+               schema: RRA_INDEX_SCHEMA,
+           },
+           slots: {
+               description: "The number of slots you want to add or remove.",
+               type: i64,
+           },
+       },
+   },
+)]
+/// Resize. Change the number of data slots for the specified RRA.
+pub fn resize_rrd(
+    path: String,
+    rra_index: usize,
+    slots: i64,
+) -> Result<(), Error> {
+
+    let path = PathBuf::from(&path);
+
+    let mut rrd = RRD::load(&path)?;
+
+    if rra_index >= rrd.rra_list.len() {
+        bail!("rra-index is out of range");
+    }
+
+    let rra = &rrd.rra_list[rra_index];
+
+    let new_slots = (rra.data.len() as i64) + slots;
+
+    if new_slots < 1 {
+        bail!("numer of new slots is too small ('{}' < 1)", new_slots);
+    }
+
+    if new_slots > 1024*1024 {
+        bail!("numer of new slots is too big ('{}' > 1M)", new_slots);
+    }
+
+    let rra_end = rra.slot_end_time(rrd.source.last_update as u64);
+    let rra_start = rra_end - rra.resolution*(rra.data.len() as u64);
+    let (start, reso, data) = rra.extract_data(rra_start, rra_end, rrd.source.last_update);
+
+    let mut new_rra = RRA::new(rra.cf, rra.resolution, new_slots as usize);
+    new_rra.last_count = rra.last_count;
+
+    new_rra.insert_data(start, reso, data)?;
+
+    rrd.rra_list[rra_index] = new_rra;
+
+    rrd.save(&path, CreateOptions::new())?;
+
+    Ok(())
+}
 
 fn main() -> Result<(), Error> {
 
@@ -185,23 +348,57 @@ fn main() -> Result<(), Error> {
     let cmd_def = CliCommandMap::new()
         .insert(
             "create",
-            CliCommand::new(&API_METHOD_CREATE_RRDB)
+            CliCommand::new(&API_METHOD_CREATE_RRD)
                 .arg_param(&["path"])
+                //.completion_cb("path", pbs_tools::fs::complete_file_name)
         )
         .insert(
-            "update",
-            CliCommand::new(&API_METHOD_UPDATE_RRDB)
+            "dump",
+            CliCommand::new(&API_METHOD_DUMP_RRD)
                 .arg_param(&["path"])
-        )
+                //.completion_cb("path", pbs_tools::fs::complete_file_name)
+         )
         .insert(
             "fetch",
-            CliCommand::new(&API_METHOD_FETCH_RRDB)
+            CliCommand::new(&API_METHOD_FETCH_RRD)
                 .arg_param(&["path"])
+                //.completion_cb("path", pbs_tools::fs::complete_file_name)
+         )
+        .insert(
+            "first",
+            CliCommand::new(&API_METHOD_FIRST_UPDATE_TIME)
+                .arg_param(&["path"])
+                //.completion_cb("path", pbs_tools::fs::complete_file_name)
         )
         .insert(
-            "dump",
-            CliCommand::new(&API_METHOD_DUMP_RRDB)
+            "info",
+            CliCommand::new(&API_METHOD_RRD_INFO)
+                .arg_param(&["path"])
+                //.completion_cb("path", pbs_tools::fs::complete_file_name)
+        )
+        .insert(
+            "last",
+            CliCommand::new(&API_METHOD_LAST_UPDATE_TIME)
+                .arg_param(&["path"])
+            //.completion_cb("path", pbs_tools::fs::complete_file_name)
+        )
+        .insert(
+            "lastupdate",
+            CliCommand::new(&API_METHOD_LAST_UPDATE)
+                .arg_param(&["path"])
+            //.completion_cb("path", pbs_tools::fs::complete_file_name)
+        )
+        .insert(
+            "resize",
+            CliCommand::new(&API_METHOD_RESIZE_RRD)
+                .arg_param(&["path"])
+            //.completion_cb("path", pbs_tools::fs::complete_file_name)
+        )
+        .insert(
+            "update",
+            CliCommand::new(&API_METHOD_UPDATE_RRD)
                 .arg_param(&["path"])
+            //.completion_cb("path", pbs_tools::fs::complete_file_name)
         )
         ;
 
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 72769bfd..c6996e42 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -153,7 +153,7 @@ impl RRA {
 
     // directly overwrite data slots
     // the caller need to set last_update value on the DataSource manually.
-    pub(crate) fn insert_data(
+    pub fn insert_data(
         &mut self,
         start: u64,
         resolution: u64,
@@ -238,7 +238,7 @@ impl RRA {
         }
     }
 
-    fn extract_data(
+    pub fn extract_data(
         &self,
         start: u64,
         end: u64,
-- 
2.30.2





^ permalink raw reply	[flat|nested] 17+ messages in thread

* [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements
  2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
                   ` (14 preceding siblings ...)
  2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
@ 2021-10-13 11:58 ` Thomas Lamprecht
  15 siblings, 0 replies; 17+ messages in thread
From: Thomas Lamprecht @ 2021-10-13 11:58 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Dietmar Maurer

On 13.10.21 10:24, Dietmar Maurer wrote:
> - use a journal. This way we can reduce the overall number of bytes
>   written by increasing the flush/commit interval (30 minutes).
> 
> - the new CBOR base format is more flexible, and we store much more
>   data points now.
> 
> We previously wrote about 7MB/h. With the new format and journal, we now write about 3MB/h.
> 
> Dietmar Maurer (15):
>   proxmox-rrd: use a journal to reduce amount of bytes written
>   RRD_CACHE: use a OnceCell instead of lazy_static
>   proxmox-backup-proxy: use tokio::task::spawn_blocking instead of
>     block_in_place
>   proxmox-rrd: implement new CBOR based format
>   proxmox-rrd: remove dependency to proxmox-rrd-api-types
>   proxmox-rrd: extract_data: include values from current slot
>   remove proxmox-rrd-api-types crate,
>     s/RRDTimeFrameResolution/RRDTimeFrame/
>   proxmox-rrd: support CF::Last
>   proxmox-rrd: split out load_rrd (cleanup)
>   proxmox-rrd: add binary to create/manage rrd files
>   proxmox-rrd: avoid % inside loop
>   proxmox-rrd: new helper methods - slot() and slot_end_time()
>   proxmox-rrd: protect against negative update time
>   proxmox-rrd: rename last_counter to last_value
>   proxmox-rrd: add more commands to the rrd cli tool

applied whole series, with the diff you replied on patch 14/15 squashed in and the rrd
debug tool moved to examples as quick "fix" for avoiding proxmox-router as hard dependency
on the crate for now, thanks!




^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2021-10-13 11:58 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-13  8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
2021-10-13  8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal