* [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements
@ 2021-10-13 8:24 Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
` (15 more replies)
0 siblings, 16 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
- use a journal. This way we can reduce the overall number of bytes
written by increasing the flush/commit interval (30 minutes).
- the new CBOR base format is more flexible, and we store much more
data points now.
We previously wrote about 7MB/h. With the new format and journal, we now write about 3MB/h.
Dietmar Maurer (15):
proxmox-rrd: use a journal to reduce amount of bytes written
RRD_CACHE: use a OnceCell instead of lazy_static
proxmox-backup-proxy: use tokio::task::spawn_blocking instead of
block_in_place
proxmox-rrd: implement new CBOR based format
proxmox-rrd: remove dependency to proxmox-rrd-api-types
proxmox-rrd: extract_data: include values from current slot
remove proxmox-rrd-api-types crate,
s/RRDTimeFrameResolution/RRDTimeFrame/
proxmox-rrd: support CF::Last
proxmox-rrd: split out load_rrd (cleanup)
proxmox-rrd: add binary to create/manage rrd files
proxmox-rrd: avoid % inside loop
proxmox-rrd: new helper methods - slot() and slot_end_time()
proxmox-rrd: protect against negative update time
proxmox-rrd: rename last_counter to last_value
proxmox-rrd: add more commands to the rrd cli tool
Cargo.toml | 2 -
Makefile | 1 -
pbs-api-types/Cargo.toml | 1 -
pbs-api-types/src/lib.rs | 30 +-
proxmox-rrd-api-types/Cargo.toml | 11 -
proxmox-rrd-api-types/src/lib.rs | 32 --
proxmox-rrd/Cargo.toml | 8 +-
proxmox-rrd/src/bin/rrd.rs | 412 +++++++++++++++++++++++
proxmox-rrd/src/cache.rs | 283 +++++++++++++---
proxmox-rrd/src/lib.rs | 17 +-
proxmox-rrd/src/rrd.rs | 539 ++++++++++++++++---------------
proxmox-rrd/src/rrd_v1.rs | 296 +++++++++++++++++
src/api2/admin/datastore.rs | 6 +-
src/api2/node/rrd.rs | 52 +--
src/api2/status.rs | 14 +-
src/bin/proxmox-backup-api.rs | 4 -
src/bin/proxmox-backup-proxy.rs | 166 +++++-----
src/lib.rs | 86 +++--
18 files changed, 1465 insertions(+), 495 deletions(-)
delete mode 100644 proxmox-rrd-api-types/Cargo.toml
delete mode 100644 proxmox-rrd-api-types/src/lib.rs
create mode 100644 proxmox-rrd/src/bin/rrd.rs
create mode 100644 proxmox-rrd/src/rrd_v1.rs
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
` (14 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
Apply the journal every 30 seconds.
---
proxmox-rrd/Cargo.toml | 1 +
proxmox-rrd/src/cache.rs | 223 ++++++++++++++++++++++++++++----
proxmox-rrd/src/lib.rs | 6 +-
proxmox-rrd/src/rrd.rs | 30 +++++
src/bin/proxmox-backup-api.rs | 2 +-
src/bin/proxmox-backup-proxy.rs | 54 ++++----
src/lib.rs | 5 +-
7 files changed, 261 insertions(+), 60 deletions(-)
diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index 7225be8e..c66344ac 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -9,6 +9,7 @@ description = "Simple RRD database implementation."
anyhow = "1.0"
bitflags = "1.2.1"
log = "0.4"
+nix = "0.19.1"
proxmox = { version = "0.14.0" }
proxmox-time = "1"
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index fe28aeda..7c56e047 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -1,24 +1,46 @@
+use std::fs::File;
use std::path::{Path, PathBuf};
use std::collections::HashMap;
-use std::sync::{RwLock};
+use std::sync::RwLock;
+use std::io::Write;
+use std::io::{BufRead, BufReader};
+use std::os::unix::io::AsRawFd;
-use anyhow::{format_err, Error};
+use anyhow::{format_err, bail, Error};
+use nix::fcntl::OFlag;
-use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions};
use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
use crate::{DST, rrd::RRD};
+const RRD_JOURNAL_NAME: &str = "rrd.journal";
+
/// RRD cache - keep RRD data in RAM, but write updates to disk
///
/// This cache is designed to run as single instance (no concurrent
/// access from other processes).
pub struct RRDCache {
+ apply_interval: f64,
basedir: PathBuf,
file_options: CreateOptions,
dir_options: CreateOptions,
- cache: RwLock<HashMap<String, RRD>>,
+ state: RwLock<RRDCacheState>,
+}
+
+// shared state behind RwLock
+struct RRDCacheState {
+ rrd_map: HashMap<String, RRD>,
+ journal: File,
+ last_journal_flush: f64,
+}
+
+struct JournalEntry {
+ time: f64,
+ value: f64,
+ dst: DST,
+ rel_path: String,
}
impl RRDCache {
@@ -28,21 +50,166 @@ impl RRDCache {
basedir: P,
file_options: Option<CreateOptions>,
dir_options: Option<CreateOptions>,
- ) -> Self {
+ apply_interval: f64,
+ ) -> Result<Self, Error> {
let basedir = basedir.as_ref().to_owned();
- Self {
+
+ let file_options = file_options.unwrap_or_else(|| CreateOptions::new());
+ let dir_options = dir_options.unwrap_or_else(|| CreateOptions::new());
+
+ create_path(&basedir, Some(dir_options.clone()), Some(dir_options.clone()))
+ .map_err(|err: Error| format_err!("unable to create rrdb stat dir - {}", err))?;
+
+ let mut journal_path = basedir.clone();
+ journal_path.push(RRD_JOURNAL_NAME);
+
+ let flags = OFlag::O_CLOEXEC|OFlag::O_WRONLY|OFlag::O_APPEND;
+ let journal = atomic_open_or_create_file(&journal_path, flags, &[], file_options.clone())?;
+
+ let state = RRDCacheState {
+ journal,
+ rrd_map: HashMap::new(),
+ last_journal_flush: 0.0,
+ };
+
+ Ok(Self {
basedir,
- file_options: file_options.unwrap_or_else(|| CreateOptions::new()),
- dir_options: dir_options.unwrap_or_else(|| CreateOptions::new()),
- cache: RwLock::new(HashMap::new()),
+ file_options,
+ dir_options,
+ apply_interval,
+ state: RwLock::new(state),
+ })
+ }
+
+ fn parse_journal_line(line: &str) -> Result<JournalEntry, Error> {
+
+ let line = line.trim();
+
+ let parts: Vec<&str> = line.splitn(4, ':').collect();
+ if parts.len() != 4 {
+ bail!("wrong numper of components");
}
+
+ let time: f64 = parts[0].parse()
+ .map_err(|_| format_err!("unable to parse time"))?;
+ let value: f64 = parts[1].parse()
+ .map_err(|_| format_err!("unable to parse value"))?;
+ let dst: u8 = parts[2].parse()
+ .map_err(|_| format_err!("unable to parse data source type"))?;
+
+ let dst = match dst {
+ 0 => DST::Gauge,
+ 1 => DST::Derive,
+ _ => bail!("got strange value for data source type '{}'", dst),
+ };
+
+ let rel_path = parts[3].to_string();
+
+ Ok(JournalEntry { time, value, dst, rel_path })
+ }
+
+ fn append_journal_entry(
+ state: &mut RRDCacheState,
+ time: f64,
+ value: f64,
+ dst: DST,
+ rel_path: &str,
+ ) -> Result<(), Error> {
+ let journal_entry = format!("{}:{}:{}:{}\n", time, value, dst as u8, rel_path);
+ state.journal.write_all(journal_entry.as_bytes())?;
+ Ok(())
}
- /// Create rrdd stat dir with correct permission
- pub fn create_rrdb_dir(&self) -> Result<(), Error> {
+ pub fn apply_journal(&self) -> Result<(), Error> {
+ let mut state = self.state.write().unwrap(); // block writers
+ self.apply_journal_locked(&mut state)
+ }
- create_path(&self.basedir, Some(self.dir_options.clone()), Some(self.dir_options.clone()))
- .map_err(|err: Error| format_err!("unable to create rrdb stat dir - {}", err))?;
+ fn apply_journal_locked(&self, state: &mut RRDCacheState) -> Result<(), Error> {
+
+ log::info!("applying rrd journal");
+
+ state.last_journal_flush = proxmox_time::epoch_f64();
+
+ let mut journal_path = self.basedir.clone();
+ journal_path.push(RRD_JOURNAL_NAME);
+
+ let flags = OFlag::O_CLOEXEC|OFlag::O_RDONLY;
+ let journal = atomic_open_or_create_file(&journal_path, flags, &[], self.file_options.clone())?;
+ let mut journal = BufReader::new(journal);
+
+ let mut last_update_map = HashMap::new();
+
+ let mut get_last_update = |rel_path: &str, rrd: &RRD| {
+ if let Some(time) = last_update_map.get(rel_path) {
+ return *time;
+ }
+ let last_update = rrd.last_update();
+ last_update_map.insert(rel_path.to_string(), last_update);
+ last_update
+ };
+
+ let mut linenr = 0;
+ loop {
+ linenr += 1;
+ let mut line = String::new();
+ let len = journal.read_line(&mut line)?;
+ if len == 0 { break; }
+
+ let entry = match Self::parse_journal_line(&line) {
+ Ok(entry) => entry,
+ Err(err) => {
+ log::warn!("unable to parse rrd journal line {} (skip) - {}", linenr, err);
+ continue; // skip unparsable lines
+ }
+ };
+
+ if let Some(rrd) = state.rrd_map.get_mut(&entry.rel_path) {
+ if entry.time > get_last_update(&entry.rel_path, &rrd) {
+ rrd.update(entry.time, entry.value);
+ }
+ } else {
+ let mut path = self.basedir.clone();
+ path.push(&entry.rel_path);
+ create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
+
+ let mut rrd = match RRD::load(&path) {
+ Ok(rrd) => rrd,
+ Err(err) => {
+ if err.kind() != std::io::ErrorKind::NotFound {
+ log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
+ }
+ RRD::new(entry.dst)
+ },
+ };
+ if entry.time > get_last_update(&entry.rel_path, &rrd) {
+ rrd.update(entry.time, entry.value);
+ }
+ state.rrd_map.insert(entry.rel_path.clone(), rrd);
+ }
+ }
+
+ // save all RRDs
+
+ let mut errors = 0;
+ for (rel_path, rrd) in state.rrd_map.iter() {
+ let mut path = self.basedir.clone();
+ path.push(&rel_path);
+ if let Err(err) = rrd.save(&path, self.file_options.clone()) {
+ errors += 1;
+ log::error!("unable to save {:?}: {}", path, err);
+ }
+ }
+
+ // if everything went ok, commit the journal
+
+ if errors == 0 {
+ nix::unistd::ftruncate(state.journal.as_raw_fd(), 0)
+ .map_err(|err| format_err!("unable to truncate journal - {}", err))?;
+ log::info!("rrd journal successfully committed");
+ } else {
+ log::error!("errors during rrd flush - unable to commit rrd journal");
+ }
Ok(())
}
@@ -53,21 +220,26 @@ impl RRDCache {
rel_path: &str,
value: f64,
dst: DST,
- save: bool,
) -> Result<(), Error> {
- let mut path = self.basedir.clone();
- path.push(rel_path);
-
- create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.file_options.clone()))?;
+ let mut state = self.state.write().unwrap(); // block other writers
- let mut map = self.cache.write().unwrap();
let now = proxmox_time::epoch_f64();
- if let Some(rrd) = map.get_mut(rel_path) {
+ if (now - state.last_journal_flush) > self.apply_interval {
+ if let Err(err) = self.apply_journal_locked(&mut state) {
+ log::error!("apply journal failed: {}", err);
+ }
+ }
+
+ Self::append_journal_entry(&mut state, now, value, dst, rel_path)?;
+
+ if let Some(rrd) = state.rrd_map.get_mut(rel_path) {
rrd.update(now, value);
- if save { rrd.save(&path, self.file_options.clone())?; }
} else {
+ let mut path = self.basedir.clone();
+ path.push(rel_path);
+ create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
let mut rrd = match RRD::load(&path) {
Ok(rrd) => rrd,
Err(err) => {
@@ -78,10 +250,7 @@ impl RRDCache {
},
};
rrd.update(now, value);
- if save {
- rrd.save(&path, self.file_options.clone())?;
- }
- map.insert(rel_path.into(), rrd);
+ state.rrd_map.insert(rel_path.into(), rrd);
}
Ok(())
@@ -97,9 +266,9 @@ impl RRDCache {
mode: RRDMode,
) -> Option<(u64, u64, Vec<Option<f64>>)> {
- let map = self.cache.read().unwrap();
+ let state = self.state.read().unwrap();
- match map.get(&format!("{}/{}", base, name)) {
+ match state.rrd_map.get(&format!("{}/{}", base, name)) {
Some(rrd) => Some(rrd.extract_data(now, timeframe, mode)),
None => None,
}
diff --git a/proxmox-rrd/src/lib.rs b/proxmox-rrd/src/lib.rs
index 303cd55d..d83e6ffc 100644
--- a/proxmox-rrd/src/lib.rs
+++ b/proxmox-rrd/src/lib.rs
@@ -13,9 +13,11 @@ mod cache;
pub use cache::*;
/// RRD data source tyoe
+#[repr(u8)]
+#[derive(Copy, Clone)]
pub enum DST {
/// Gauge values are stored unmodified.
- Gauge,
+ Gauge = 0,
/// Stores the difference to the previous value.
- Derive,
+ Derive = 1,
}
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index f4c08909..026498ed 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -336,6 +336,36 @@ impl RRD {
replace_file(filename, rrd_slice, options)
}
+ pub fn last_update(&self) -> f64 {
+
+ let mut last_update = 0.0;
+
+ {
+ let mut check_last_update = |rra: &RRA| {
+ if rra.last_update > last_update {
+ last_update = rra.last_update;
+ }
+ };
+
+ check_last_update(&self.hour_avg);
+ check_last_update(&self.hour_max);
+
+ check_last_update(&self.day_avg);
+ check_last_update(&self.day_max);
+
+ check_last_update(&self.week_avg);
+ check_last_update(&self.week_max);
+
+ check_last_update(&self.month_avg);
+ check_last_update(&self.month_max);
+
+ check_last_update(&self.year_avg);
+ check_last_update(&self.year_max);
+ }
+
+ last_update
+ }
+
/// Update the value (in memory)
///
/// Note: This does not call [Self::save].
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index c98fba25..3c963a77 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -74,7 +74,7 @@ async fn run() -> Result<(), Error> {
proxmox_backup::server::create_run_dir()?;
- RRD_CACHE.create_rrdb_dir()?;
+ RRD_CACHE.apply_journal()?;
proxmox_backup::server::jobstate::create_jobstate_dir()?;
proxmox_backup::tape::create_tape_status_dir()?;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 4bdb8ce9..8da98ff8 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -889,14 +889,10 @@ async fn command_reopen_auth_logfiles() -> Result<(), Error> {
async fn run_stat_generator() {
- let mut count = 0;
loop {
- count += 1;
- let save = if count >= 6 { count = 0; true } else { false };
-
let delay_target = Instant::now() + Duration::from_secs(10);
- generate_host_stats(save).await;
+ generate_host_stats().await;
tokio::time::sleep_until(tokio::time::Instant::from_std(delay_target)).await;
@@ -904,19 +900,19 @@ async fn run_stat_generator() {
}
-fn rrd_update_gauge(name: &str, value: f64, save: bool) {
- if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge, save) {
+fn rrd_update_gauge(name: &str, value: f64) {
+ if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge) {
eprintln!("rrd::update_value '{}' failed - {}", name, err);
}
}
-fn rrd_update_derive(name: &str, value: f64, save: bool) {
- if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive, save) {
+fn rrd_update_derive(name: &str, value: f64) {
+ if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive) {
eprintln!("rrd::update_value '{}' failed - {}", name, err);
}
}
-async fn generate_host_stats(save: bool) {
+async fn generate_host_stats() {
use proxmox::sys::linux::procfs::{
read_meminfo, read_proc_stat, read_proc_net_dev, read_loadavg};
@@ -924,8 +920,8 @@ async fn generate_host_stats(save: bool) {
match read_proc_stat() {
Ok(stat) => {
- rrd_update_gauge("host/cpu", stat.cpu, save);
- rrd_update_gauge("host/iowait", stat.iowait_percent, save);
+ rrd_update_gauge("host/cpu", stat.cpu);
+ rrd_update_gauge("host/iowait", stat.iowait_percent);
}
Err(err) => {
eprintln!("read_proc_stat failed - {}", err);
@@ -934,10 +930,10 @@ async fn generate_host_stats(save: bool) {
match read_meminfo() {
Ok(meminfo) => {
- rrd_update_gauge("host/memtotal", meminfo.memtotal as f64, save);
- rrd_update_gauge("host/memused", meminfo.memused as f64, save);
- rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64, save);
- rrd_update_gauge("host/swapused", meminfo.swapused as f64, save);
+ rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
+ rrd_update_gauge("host/memused", meminfo.memused as f64);
+ rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
+ rrd_update_gauge("host/swapused", meminfo.swapused as f64);
}
Err(err) => {
eprintln!("read_meminfo failed - {}", err);
@@ -954,8 +950,8 @@ async fn generate_host_stats(save: bool) {
netin += item.receive;
netout += item.send;
}
- rrd_update_derive("host/netin", netin as f64, save);
- rrd_update_derive("host/netout", netout as f64, save);
+ rrd_update_derive("host/netin", netin as f64);
+ rrd_update_derive("host/netout", netout as f64);
}
Err(err) => {
eprintln!("read_prox_net_dev failed - {}", err);
@@ -964,7 +960,7 @@ async fn generate_host_stats(save: bool) {
match read_loadavg() {
Ok(loadavg) => {
- rrd_update_gauge("host/loadavg", loadavg.0 as f64, save);
+ rrd_update_gauge("host/loadavg", loadavg.0 as f64);
}
Err(err) => {
eprintln!("read_loadavg failed - {}", err);
@@ -973,7 +969,7 @@ async fn generate_host_stats(save: bool) {
let disk_manager = DiskManage::new();
- gather_disk_stats(disk_manager.clone(), Path::new("/"), "host", save);
+ gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
match pbs_config::datastore::config() {
Ok((config, _)) => {
@@ -984,7 +980,7 @@ async fn generate_host_stats(save: bool) {
let rrd_prefix = format!("datastore/{}", config.name);
let path = std::path::Path::new(&config.path);
- gather_disk_stats(disk_manager.clone(), path, &rrd_prefix, save);
+ gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
}
}
Err(err) => {
@@ -1025,14 +1021,14 @@ fn check_schedule(worker_type: &str, event_str: &str, id: &str) -> bool {
next <= now
}
-fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &str, save: bool) {
+fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &str) {
match proxmox_backup::tools::disks::disk_usage(path) {
Ok(status) => {
let rrd_key = format!("{}/total", rrd_prefix);
- rrd_update_gauge(&rrd_key, status.total as f64, save);
+ rrd_update_gauge(&rrd_key, status.total as f64);
let rrd_key = format!("{}/used", rrd_prefix);
- rrd_update_gauge(&rrd_key, status.used as f64, save);
+ rrd_update_gauge(&rrd_key, status.used as f64);
}
Err(err) => {
eprintln!("read disk_usage on {:?} failed - {}", path, err);
@@ -1064,17 +1060,17 @@ fn gather_disk_stats(disk_manager: Arc<DiskManage>, path: &Path, rrd_prefix: &st
}
if let Some(stat) = device_stat {
let rrd_key = format!("{}/read_ios", rrd_prefix);
- rrd_update_derive(&rrd_key, stat.read_ios as f64, save);
+ rrd_update_derive(&rrd_key, stat.read_ios as f64);
let rrd_key = format!("{}/read_bytes", rrd_prefix);
- rrd_update_derive(&rrd_key, (stat.read_sectors*512) as f64, save);
+ rrd_update_derive(&rrd_key, (stat.read_sectors*512) as f64);
let rrd_key = format!("{}/write_ios", rrd_prefix);
- rrd_update_derive(&rrd_key, stat.write_ios as f64, save);
+ rrd_update_derive(&rrd_key, stat.write_ios as f64);
let rrd_key = format!("{}/write_bytes", rrd_prefix);
- rrd_update_derive(&rrd_key, (stat.write_sectors*512) as f64, save);
+ rrd_update_derive(&rrd_key, (stat.write_sectors*512) as f64);
let rrd_key = format!("{}/io_ticks", rrd_prefix);
- rrd_update_derive(&rrd_key, (stat.io_ticks as f64)/1000.0, save);
+ rrd_update_derive(&rrd_key, (stat.io_ticks as f64)/1000.0);
}
}
Err(err) => {
diff --git a/src/lib.rs b/src/lib.rs
index 98b6b987..5d2b4590 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -51,10 +51,13 @@ lazy_static::lazy_static!{
.owner(backup_user.uid)
.group(backup_user.gid);
+ let apply_interval = 30.0*60.0; // 30 minutes
+
RRDCache::new(
"/var/lib/proxmox-backup/rrdb",
Some(file_options),
Some(dir_options),
- )
+ apply_interval,
+ ).unwrap()
};
}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
` (13 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
And initialize only with proxmox-backup-proxy. Other binaries dont need it.
---
src/api2/node/rrd.rs | 6 ++--
src/api2/status.rs | 7 ++--
src/bin/proxmox-backup-api.rs | 4 ---
src/bin/proxmox-backup-proxy.rs | 17 +++++++---
src/lib.rs | 59 +++++++++++++++++++++------------
5 files changed, 59 insertions(+), 34 deletions(-)
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index c59a35a4..b53076d7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -10,7 +10,7 @@ use pbs_api_types::{
use proxmox_rrd::rrd::RRD_DATA_ENTRIES;
-use crate::RRD_CACHE;
+use crate::get_rrd_cache;
pub fn create_value_from_rrd(
basedir: &str,
@@ -22,8 +22,10 @@ pub fn create_value_from_rrd(
let mut result = Vec::new();
let now = proxmox_time::epoch_f64();
+ let rrd_cache = get_rrd_cache()?;
+
for name in list {
- let (start, reso, list) = match RRD_CACHE.extract_cached_data(basedir, name, now, timeframe, cf) {
+ let (start, reso, list) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf) {
Some(result) => result,
None => continue,
};
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 9c7d4f7d..40a9e5b4 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -22,7 +22,7 @@ use pbs_datastore::DataStore;
use pbs_config::CachedUserInfo;
use crate::tools::statistics::{linear_regression};
-use crate::RRD_CACHE;
+use crate::get_rrd_cache;
#[api(
returns: {
@@ -90,6 +90,8 @@ pub fn datastore_status(
let mut list = Vec::new();
+ let rrd_cache = get_rrd_cache()?;
+
for (store, (_, _)) in &config.sections {
let user_privs = user_info.lookup_privs(&auth_id, &["datastore", &store]);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT| PRIV_DATASTORE_BACKUP)) != 0;
@@ -123,7 +125,8 @@ pub fn datastore_status(
let rrd_dir = format!("datastore/{}", store);
let now = proxmox_time::epoch_f64();
- let get_rrd = |what: &str| RRD_CACHE.extract_cached_data(
+
+ let get_rrd = |what: &str| rrd_cache.extract_cached_data(
&rrd_dir,
what,
now,
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 3c963a77..8d6c3170 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -16,7 +16,6 @@ use proxmox_rest_server::{daemon, AuthError, ApiConfig, RestServer, RestEnvironm
use proxmox_backup::server::auth::check_pbs_auth;
use proxmox_backup::auth_helpers::*;
-use proxmox_backup::RRD_CACHE;
use proxmox_backup::config;
fn main() {
@@ -73,9 +72,6 @@ async fn run() -> Result<(), Error> {
config::update_self_signed_cert(false)?;
proxmox_backup::server::create_run_dir()?;
-
- RRD_CACHE.apply_journal()?;
-
proxmox_backup::server::jobstate::create_jobstate_dir()?;
proxmox_backup::tape::create_tape_status_dir()?;
proxmox_backup::tape::create_drive_state_dir()?;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 8da98ff8..66d81bdb 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -32,7 +32,7 @@ use proxmox_rest_server::{
};
use proxmox_backup::{
- RRD_CACHE,
+ get_rrd_cache, initialize_rrd_cache,
server::{
auth::check_pbs_auth,
jobstate::{
@@ -208,6 +208,9 @@ async fn run() -> Result<(), Error> {
let _ = public_auth_key(); // load with lazy_static
let _ = csrf_secret(); // load with lazy_static
+ let rrd_cache = initialize_rrd_cache()?;
+ rrd_cache.apply_journal()?;
+
let mut config = ApiConfig::new(
pbs_buildcfg::JS_DIR,
&proxmox_backup::api2::ROUTER,
@@ -901,14 +904,18 @@ async fn run_stat_generator() {
}
fn rrd_update_gauge(name: &str, value: f64) {
- if let Err(err) = RRD_CACHE.update_value(name, value, DST::Gauge) {
- eprintln!("rrd::update_value '{}' failed - {}", name, err);
+ if let Ok(rrd_cache) = get_rrd_cache() {
+ if let Err(err) = rrd_cache.update_value(name, value, DST::Gauge) {
+ eprintln!("rrd::update_value '{}' failed - {}", name, err);
+ }
}
}
fn rrd_update_derive(name: &str, value: f64) {
- if let Err(err) = RRD_CACHE.update_value(name, value, DST::Derive) {
- eprintln!("rrd::update_value '{}' failed - {}", name, err);
+ if let Ok(rrd_cache) = get_rrd_cache() {
+ if let Err(err) = rrd_cache.update_value(name, value, DST::Derive) {
+ eprintln!("rrd::update_value '{}' failed - {}", name, err);
+ }
}
}
diff --git a/src/lib.rs b/src/lib.rs
index 5d2b4590..a1ac23bf 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -5,6 +5,9 @@
use std::path::PathBuf;
+use once_cell::sync::OnceCell;
+use anyhow::{format_err, Error};
+
use proxmox::tools::fs::CreateOptions;
use pbs_buildcfg::configdir;
@@ -39,25 +42,39 @@ pub fn cert_info() -> Result<CertInfo, anyhow::Error> {
CertInfo::from_path(PathBuf::from(configdir!("/proxy.pem")))
}
-lazy_static::lazy_static!{
- /// Proxmox Backup Server RRD cache instance
- pub static ref RRD_CACHE: RRDCache = {
- let backup_user = pbs_config::backup_user().unwrap();
- let file_options = CreateOptions::new()
- .owner(backup_user.uid)
- .group(backup_user.gid);
-
- let dir_options = CreateOptions::new()
- .owner(backup_user.uid)
- .group(backup_user.gid);
-
- let apply_interval = 30.0*60.0; // 30 minutes
-
- RRDCache::new(
- "/var/lib/proxmox-backup/rrdb",
- Some(file_options),
- Some(dir_options),
- apply_interval,
- ).unwrap()
- };
+pub static RRD_CACHE: OnceCell<RRDCache> = OnceCell::new();
+
+/// Get the RRD cache instance
+pub fn get_rrd_cache() -> Result<&'static RRDCache, Error> {
+ RRD_CACHE.get().ok_or_else(|| format_err!("RRD cache not initialized!"))
+}
+
+/// Initialize the RRD cache instance
+///
+/// Note: Only a single process must do this (proxmox-backup-proxy)
+pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
+
+ let backup_user = pbs_config::backup_user()?;
+
+ let file_options = CreateOptions::new()
+ .owner(backup_user.uid)
+ .group(backup_user.gid);
+
+ let dir_options = CreateOptions::new()
+ .owner(backup_user.uid)
+ .group(backup_user.gid);
+
+ let apply_interval = 30.0*60.0; // 30 minutes
+
+ let cache = RRDCache::new(
+ "/var/lib/proxmox-backup/rrdb",
+ Some(file_options),
+ Some(dir_options),
+ apply_interval,
+ )?;
+
+ RRD_CACHE.set(cache)
+ .map_err(|_| format_err!("RRD cache already initialized!"))?;
+
+ Ok(RRD_CACHE.get().unwrap())
}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
` (12 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
src/bin/proxmox-backup-proxy.rs | 119 ++++++++++++++++----------------
1 file changed, 61 insertions(+), 58 deletions(-)
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 66d81bdb..0f0f6f59 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -920,82 +920,85 @@ fn rrd_update_derive(name: &str, value: f64) {
}
async fn generate_host_stats() {
+ match tokio::task::spawn_blocking(generate_host_stats_sync).await {
+ Ok(()) => (),
+ Err(err) => log::error!("generate_host_stats paniced: {}", err),
+ }
+}
+
+fn generate_host_stats_sync() {
use proxmox::sys::linux::procfs::{
read_meminfo, read_proc_stat, read_proc_net_dev, read_loadavg};
- pbs_runtime::block_in_place(move || {
-
- match read_proc_stat() {
- Ok(stat) => {
- rrd_update_gauge("host/cpu", stat.cpu);
- rrd_update_gauge("host/iowait", stat.iowait_percent);
- }
- Err(err) => {
- eprintln!("read_proc_stat failed - {}", err);
- }
+ match read_proc_stat() {
+ Ok(stat) => {
+ rrd_update_gauge("host/cpu", stat.cpu);
+ rrd_update_gauge("host/iowait", stat.iowait_percent);
+ }
+ Err(err) => {
+ eprintln!("read_proc_stat failed - {}", err);
}
+ }
- match read_meminfo() {
- Ok(meminfo) => {
- rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
- rrd_update_gauge("host/memused", meminfo.memused as f64);
- rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
- rrd_update_gauge("host/swapused", meminfo.swapused as f64);
- }
- Err(err) => {
- eprintln!("read_meminfo failed - {}", err);
- }
+ match read_meminfo() {
+ Ok(meminfo) => {
+ rrd_update_gauge("host/memtotal", meminfo.memtotal as f64);
+ rrd_update_gauge("host/memused", meminfo.memused as f64);
+ rrd_update_gauge("host/swaptotal", meminfo.swaptotal as f64);
+ rrd_update_gauge("host/swapused", meminfo.swapused as f64);
}
+ Err(err) => {
+ eprintln!("read_meminfo failed - {}", err);
+ }
+ }
- match read_proc_net_dev() {
- Ok(netdev) => {
- use pbs_config::network::is_physical_nic;
- let mut netin = 0;
- let mut netout = 0;
- for item in netdev {
- if !is_physical_nic(&item.device) { continue; }
- netin += item.receive;
- netout += item.send;
- }
- rrd_update_derive("host/netin", netin as f64);
- rrd_update_derive("host/netout", netout as f64);
- }
- Err(err) => {
- eprintln!("read_prox_net_dev failed - {}", err);
+ match read_proc_net_dev() {
+ Ok(netdev) => {
+ use pbs_config::network::is_physical_nic;
+ let mut netin = 0;
+ let mut netout = 0;
+ for item in netdev {
+ if !is_physical_nic(&item.device) { continue; }
+ netin += item.receive;
+ netout += item.send;
}
+ rrd_update_derive("host/netin", netin as f64);
+ rrd_update_derive("host/netout", netout as f64);
+ }
+ Err(err) => {
+ eprintln!("read_prox_net_dev failed - {}", err);
}
+ }
- match read_loadavg() {
- Ok(loadavg) => {
- rrd_update_gauge("host/loadavg", loadavg.0 as f64);
- }
- Err(err) => {
- eprintln!("read_loadavg failed - {}", err);
- }
+ match read_loadavg() {
+ Ok(loadavg) => {
+ rrd_update_gauge("host/loadavg", loadavg.0 as f64);
+ }
+ Err(err) => {
+ eprintln!("read_loadavg failed - {}", err);
}
+ }
- let disk_manager = DiskManage::new();
+ let disk_manager = DiskManage::new();
- gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
+ gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
- match pbs_config::datastore::config() {
- Ok((config, _)) => {
- let datastore_list: Vec<DataStoreConfig> =
- config.convert_to_typed_array("datastore").unwrap_or_default();
+ match pbs_config::datastore::config() {
+ Ok((config, _)) => {
+ let datastore_list: Vec<DataStoreConfig> =
+ config.convert_to_typed_array("datastore").unwrap_or_default();
- for config in datastore_list {
+ for config in datastore_list {
- let rrd_prefix = format!("datastore/{}", config.name);
- let path = std::path::Path::new(&config.path);
- gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
- }
- }
- Err(err) => {
- eprintln!("read datastore config failed - {}", err);
+ let rrd_prefix = format!("datastore/{}", config.name);
+ let path = std::path::Path::new(&config.path);
+ gather_disk_stats(disk_manager.clone(), path, &rrd_prefix);
}
}
-
- });
+ Err(err) => {
+ eprintln!("read datastore config failed - {}", err);
+ }
+ }
}
fn check_schedule(worker_type: &str, event_str: &str, id: &str) -> bool {
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (2 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
` (11 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
Storing much more data points now got get better graphs.
---
proxmox-rrd-api-types/src/lib.rs | 23 +-
proxmox-rrd/Cargo.toml | 4 +
proxmox-rrd/src/cache.rs | 51 ++-
proxmox-rrd/src/lib.rs | 19 +-
proxmox-rrd/src/rrd.rs | 534 +++++++++++++++----------------
proxmox-rrd/src/rrd_v1.rs | 296 +++++++++++++++++
src/api2/node/rrd.rs | 39 ++-
src/api2/status.rs | 4 +-
src/bin/proxmox-backup-proxy.rs | 2 +-
9 files changed, 641 insertions(+), 331 deletions(-)
create mode 100644 proxmox-rrd/src/rrd_v1.rs
diff --git a/proxmox-rrd-api-types/src/lib.rs b/proxmox-rrd-api-types/src/lib.rs
index b5e62e73..32601477 100644
--- a/proxmox-rrd-api-types/src/lib.rs
+++ b/proxmox-rrd-api-types/src/lib.rs
@@ -14,19 +14,20 @@ pub enum RRDMode {
}
#[api()]
-#[repr(u64)]
#[derive(Copy, Clone, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
/// RRD time frame resolution
pub enum RRDTimeFrameResolution {
- /// 1 min => last 70 minutes
- Hour = 60,
- /// 30 min => last 35 hours
- Day = 60*30,
- /// 3 hours => about 8 days
- Week = 60*180,
- /// 12 hours => last 35 days
- Month = 60*720,
- /// 1 week => last 490 days
- Year = 60*10080,
+ /// Hour
+ Hour,
+ /// Day
+ Day,
+ /// Week
+ Week,
+ /// Month
+ Month,
+ /// Year
+ Year,
+ /// Decade (10 years)
+ Decade,
}
diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index c66344ac..b3dd02c3 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -10,8 +10,12 @@ anyhow = "1.0"
bitflags = "1.2.1"
log = "0.4"
nix = "0.19.1"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+serde_cbor = "0.11.1"
proxmox = { version = "0.14.0" }
proxmox-time = "1"
+proxmox-schema = { version = "1", features = [ "api-macro" ] }
proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index 7c56e047..f14837fc 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -13,7 +13,7 @@ use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions}
use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
-use crate::{DST, rrd::RRD};
+use crate::rrd::{DST, CF, RRD, RRA};
const RRD_JOURNAL_NAME: &str = "rrd.journal";
@@ -81,6 +81,29 @@ impl RRDCache {
})
}
+ fn create_default_rrd(dst: DST) -> RRD {
+
+ let mut rra_list = Vec::new();
+
+ // 1min * 1440 => 1day
+ rra_list.push(RRA::new(CF::Average, 60, 1440));
+ rra_list.push(RRA::new(CF::Maximum, 60, 1440));
+
+ // 30min * 1440 => 30days = 1month
+ rra_list.push(RRA::new(CF::Average, 30*60, 1440));
+ rra_list.push(RRA::new(CF::Maximum, 30*60, 1440));
+
+ // 6h * 1440 => 360days = 1year
+ rra_list.push(RRA::new(CF::Average, 6*3600, 1440));
+ rra_list.push(RRA::new(CF::Maximum, 6*3600, 1440));
+
+ // 1week * 570 => 10years
+ rra_list.push(RRA::new(CF::Average, 7*86400, 570));
+ rra_list.push(RRA::new(CF::Maximum, 7*86400, 570));
+
+ RRD::new(dst, rra_list)
+ }
+
fn parse_journal_line(line: &str) -> Result<JournalEntry, Error> {
let line = line.trim();
@@ -179,7 +202,7 @@ impl RRDCache {
if err.kind() != std::io::ErrorKind::NotFound {
log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
}
- RRD::new(entry.dst)
+ Self::create_default_rrd(entry.dst)
},
};
if entry.time > get_last_update(&entry.rel_path, &rrd) {
@@ -246,7 +269,7 @@ impl RRDCache {
if err.kind() != std::io::ErrorKind::NotFound {
log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
}
- RRD::new(dst)
+ Self::create_default_rrd(dst)
},
};
rrd.update(now, value);
@@ -264,13 +287,29 @@ impl RRDCache {
now: f64,
timeframe: RRDTimeFrameResolution,
mode: RRDMode,
- ) -> Option<(u64, u64, Vec<Option<f64>>)> {
+ ) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
let state = self.state.read().unwrap();
+ let cf = match mode {
+ RRDMode::Max => CF::Maximum,
+ RRDMode::Average => CF::Average,
+ };
+
+ let now = now as u64;
+
+ let (start, resolution) = match timeframe {
+ RRDTimeFrameResolution::Hour => (now - 3600, 60),
+ RRDTimeFrameResolution::Day => (now - 3600*24, 60),
+ RRDTimeFrameResolution::Week => (now - 3600*24*7, 30*60),
+ RRDTimeFrameResolution::Month => (now - 3600*24*30, 30*60),
+ RRDTimeFrameResolution::Year => (now - 3600*24*365, 6*60*60),
+ RRDTimeFrameResolution::Decade => (now - 10*3600*24*366, 7*86400),
+ };
+
match state.rrd_map.get(&format!("{}/{}", base, name)) {
- Some(rrd) => Some(rrd.extract_data(now, timeframe, mode)),
- None => None,
+ Some(rrd) => Ok(Some(rrd.extract_data(start, now, cf, resolution)?)),
+ None => Ok(None),
}
}
}
diff --git a/proxmox-rrd/src/lib.rs b/proxmox-rrd/src/lib.rs
index d83e6ffc..2038170d 100644
--- a/proxmox-rrd/src/lib.rs
+++ b/proxmox-rrd/src/lib.rs
@@ -1,23 +1,14 @@
-//! # Simple Round Robin Database files with fixed format
+//! # Round Robin Database files
//!
//! ## Features
//!
//! * One file stores a single data source
-//! * Small/constant file size (6008 bytes)
-//! * Stores avarage and maximum values
-//! * Stores data for different time resolution ([RRDTimeFrameResolution](proxmox_rrd_api_types::RRDTimeFrameResolution))
+//! * Stores data for different time resolution
+//! * Simple cache implementation with journal support
+
+mod rrd_v1;
pub mod rrd;
mod cache;
pub use cache::*;
-
-/// RRD data source tyoe
-#[repr(u8)]
-#[derive(Copy, Clone)]
-pub enum DST {
- /// Gauge values are stored unmodified.
- Gauge = 0,
- /// Stores the difference to the previous value.
- Derive = 1,
-}
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 026498ed..82fa5a3a 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -1,82 +1,175 @@
-//! # Round Robin Database file format
+//! # Proxmox RRD format version 2
+//!
+//! The new format uses
+//! [CBOR](https://datatracker.ietf.org/doc/html/rfc8949) as storage
+//! format. This way we can use the serde serialization framework,
+//! which make our code more flexible, much nicer and type safe.
+//!
+//! ## Features
+//!
+//! * Well defined data format [CBOR](https://datatracker.ietf.org/doc/html/rfc8949)
+//! * Plattform independent (big endian f64, hopefully a standard format?)
+//! * Arbitrary number of RRAs (dynamically changeable)
-use std::io::Read;
use std::path::Path;
use anyhow::{bail, Error};
-use bitflags::bitflags;
-use proxmox::tools::{fs::replace_file, fs::CreateOptions};
+use serde::{Serialize, Deserialize};
+
+use proxmox::tools::fs::{replace_file, CreateOptions};
+use proxmox_schema::api;
+
+use crate::rrd_v1;
+
+/// Proxmox RRD v2 file magic number
+// openssl::sha::sha256(b"Proxmox Round Robin Database file v2.0")[0..8];
+pub const PROXMOX_RRD_MAGIC_2_0: [u8; 8] = [224, 200, 228, 27, 239, 112, 122, 159];
+
+#[api()]
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// RRD data source type
+pub enum DST {
+ /// Gauge values are stored unmodified.
+ Gauge,
+ /// Stores the difference to the previous value.
+ Derive,
+ /// Stores the difference to the previous value (like Derive), but
+ /// detect counter overflow (and ignores that value)
+ Counter,
+}
+
+#[api()]
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Consolidation function
+pub enum CF {
+ /// Average
+ Average,
+ /// Maximum
+ Maximum,
+ /// Minimum
+ Minimum,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct DataSource {
+ /// Data source type
+ pub dst: DST,
+ /// Last update time (epoch)
+ pub last_update: f64,
+ /// Stores the last value, used to compute differential value for
+ /// derive/counters
+ pub counter_value: f64,
+}
-use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
+impl DataSource {
-/// The number of data entries per RRA
-pub const RRD_DATA_ENTRIES: usize = 70;
+ pub fn new(dst: DST) -> Self {
+ Self {
+ dst,
+ last_update: 0.0,
+ counter_value: f64::NAN,
+ }
+ }
-/// Proxmox RRD file magic number
-// openssl::sha::sha256(b"Proxmox Round Robin Database file v1.0")[0..8];
-pub const PROXMOX_RRD_MAGIC_1_0: [u8; 8] = [206, 46, 26, 212, 172, 158, 5, 186];
+ fn compute_new_value(&mut self, time: f64, mut value: f64) -> Result<f64, Error> {
+ if time <= self.last_update {
+ bail!("time in past ({} < {})", time, self.last_update);
+ }
-use crate::DST;
+ if value.is_nan() {
+ bail!("new value is NAN");
+ }
-bitflags!{
- /// Flags to specify the data soure type and consolidation function
- pub struct RRAFlags: u64 {
- // Data Source Types
- const DST_GAUGE = 1;
- const DST_DERIVE = 2;
- const DST_COUNTER = 4;
- const DST_MASK = 255; // first 8 bits
+ // derive counter value
+ let is_counter = self.dst == DST::Counter;
+
+ if is_counter || self.dst == DST::Derive {
+ let time_diff = time - self.last_update;
+
+ let diff = if self.counter_value.is_nan() {
+ 0.0
+ } else if is_counter && value < 0.0 {
+ bail!("got negative value for counter");
+ } else if is_counter && value < self.counter_value {
+ // Note: We do not try automatic overflow corrections, but
+ // we update counter_value anyways, so that we can compute the diff
+ // next time.
+ self.counter_value = value;
+ bail!("conter overflow/reset detected");
+ } else {
+ value - self.counter_value
+ };
+ self.counter_value = value;
+ value = diff/time_diff;
+ }
- // Consolidation Functions
- const CF_AVERAGE = 1 << 8;
- const CF_MAX = 2 << 8;
- const CF_MASK = 255 << 8;
+ Ok(value)
}
+
+
}
-/// Round Robin Archive with [RRD_DATA_ENTRIES] data slots.
-///
-/// This data structure is used inside [RRD] and directly written to the
-/// RRD files.
-#[repr(C)]
+#[derive(Serialize, Deserialize)]
pub struct RRA {
- /// Defined the data soure type and consolidation function
- pub flags: RRAFlags,
- /// Resulution (seconds) from [RRDTimeFrameResolution]
pub resolution: u64,
- /// Last update time (epoch)
- pub last_update: f64,
+ pub cf: CF,
/// Count values computed inside this update interval
pub last_count: u64,
- /// Stores the last value, used to compute differential value for derive/counters
- pub counter_value: f64,
- /// Data slots
- pub data: [f64; RRD_DATA_ENTRIES],
+ /// The actual data
+ pub data: Vec<f64>,
}
impl RRA {
- fn new(flags: RRAFlags, resolution: u64) -> Self {
+
+ pub fn new(cf: CF, resolution: u64, points: usize) -> Self {
Self {
- flags, resolution,
- last_update: 0.0,
+ cf,
+ resolution,
last_count: 0,
- counter_value: f64::NAN,
- data: [f64::NAN; RRD_DATA_ENTRIES],
+ data: vec![f64::NAN; points],
+ }
+ }
+
+ // directly overwrite data slots
+ // the caller need to set last_update value on the DataSource manually.
+ pub(crate) fn insert_data(
+ &mut self,
+ start: u64,
+ resolution: u64,
+ data: Vec<Option<f64>>,
+ ) -> Result<(), Error> {
+ if resolution != self.resolution {
+ bail!("inser_data failed: got wrong resolution");
+ }
+ let num_entries = self.data.len() as u64;
+ let mut index = ((start/self.resolution) % num_entries) as usize;
+
+ for i in 0..data.len() {
+ if let Some(v) = data[i] {
+ self.data[index] = v;
+ }
+ index += 1;
+ if index >= self.data.len() { index = 0; }
}
+ Ok(())
}
- fn delete_old(&mut self, time: f64) {
+ fn delete_old_slots(&mut self, time: f64, last_update: f64) {
let epoch = time as u64;
- let last_update = self.last_update as u64;
+ let last_update = last_update as u64;
let reso = self.resolution;
+ let num_entries = self.data.len() as u64;
- let min_time = epoch - (RRD_DATA_ENTRIES as u64)*reso;
+ let min_time = epoch - num_entries*reso;
let min_time = (min_time/reso + 1)*reso;
- let mut t = last_update.saturating_sub((RRD_DATA_ENTRIES as u64)*reso);
- let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
- for _ in 0..RRD_DATA_ENTRIES {
- t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+ let mut t = last_update.saturating_sub(num_entries*reso);
+ let mut index = ((t/reso) % num_entries) as usize;
+ for _ in 0..num_entries {
+ t += reso;
+ index = (index + 1) % (num_entries as usize);
if t < min_time {
self.data[index] = f64::NAN;
} else {
@@ -85,13 +178,14 @@ impl RRA {
}
}
- fn compute_new_value(&mut self, time: f64, value: f64) {
+ fn compute_new_value(&mut self, time: f64, last_update: f64, value: f64) {
let epoch = time as u64;
- let last_update = self.last_update as u64;
+ let last_update = last_update as u64;
let reso = self.resolution;
+ let num_entries = self.data.len() as u64;
- let index = ((epoch/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
- let last_index = ((last_update/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
+ let index = ((epoch/reso) % num_entries) as usize;
+ let last_index = ((last_update/reso) % num_entries) as usize;
if (epoch - (last_update as u64)) > reso || index != last_index {
self.last_count = 0;
@@ -112,258 +206,111 @@ impl RRA {
self.data[index] = value;
self.last_count = 1;
} else {
- let new_value = if self.flags.contains(RRAFlags::CF_MAX) {
- if last_value > value { last_value } else { value }
- } else if self.flags.contains(RRAFlags::CF_AVERAGE) {
- (last_value*(self.last_count as f64))/(new_count as f64)
- + value/(new_count as f64)
- } else {
- log::error!("rrdb update failed - unknown CF");
- return;
+ let new_value = match self.cf {
+ CF::Maximum => if last_value > value { last_value } else { value },
+ CF::Minimum => if last_value < value { last_value } else { value },
+ CF::Average => {
+ (last_value*(self.last_count as f64))/(new_count as f64)
+ + value/(new_count as f64)
+ }
};
self.data[index] = new_value;
self.last_count = new_count;
}
- self.last_update = time;
}
- // Note: This may update the state even in case of errors (see counter overflow)
- fn update(&mut self, time: f64, mut value: f64) -> Result<(), Error> {
-
- if time <= self.last_update {
- bail!("time in past ({} < {})", time, self.last_update);
- }
-
- if value.is_nan() {
- bail!("new value is NAN");
- }
-
- // derive counter value
- if self.flags.intersects(RRAFlags::DST_DERIVE | RRAFlags::DST_COUNTER) {
- let time_diff = time - self.last_update;
- let is_counter = self.flags.contains(RRAFlags::DST_COUNTER);
-
- let diff = if self.counter_value.is_nan() {
- 0.0
- } else if is_counter && value < 0.0 {
- bail!("got negative value for counter");
- } else if is_counter && value < self.counter_value {
- // Note: We do not try automatic overflow corrections, but
- // we update counter_value anyways, so that we can compute the diff
- // next time.
- self.counter_value = value;
- bail!("conter overflow/reset detected");
- } else {
- value - self.counter_value
- };
- self.counter_value = value;
- value = diff/time_diff;
- }
-
- self.delete_old(time);
- self.compute_new_value(time, value);
-
- Ok(())
- }
-}
-
-/// Round Robin Database file format with fixed number of [RRA]s
-#[repr(C)]
-// Note: Avoid alignment problems by using 8byte types only
-pub struct RRD {
- /// The magic number to identify the file type
- pub magic: [u8; 8],
- /// Hourly data (average values)
- pub hour_avg: RRA,
- /// Hourly data (maximum values)
- pub hour_max: RRA,
- /// Dayly data (average values)
- pub day_avg: RRA,
- /// Dayly data (maximum values)
- pub day_max: RRA,
- /// Weekly data (average values)
- pub week_avg: RRA,
- /// Weekly data (maximum values)
- pub week_max: RRA,
- /// Monthly data (average values)
- pub month_avg: RRA,
- /// Monthly data (maximum values)
- pub month_max: RRA,
- /// Yearly data (average values)
- pub year_avg: RRA,
- /// Yearly data (maximum values)
- pub year_max: RRA,
-}
-
-impl RRD {
-
- /// Create a new empty instance
- pub fn new(dst: DST) -> Self {
- let flags = match dst {
- DST::Gauge => RRAFlags::DST_GAUGE,
- DST::Derive => RRAFlags::DST_DERIVE,
- };
-
- Self {
- magic: PROXMOX_RRD_MAGIC_1_0,
- hour_avg: RRA::new(
- flags | RRAFlags::CF_AVERAGE,
- RRDTimeFrameResolution::Hour as u64,
- ),
- hour_max: RRA::new(
- flags | RRAFlags::CF_MAX,
- RRDTimeFrameResolution::Hour as u64,
- ),
- day_avg: RRA::new(
- flags | RRAFlags::CF_AVERAGE,
- RRDTimeFrameResolution::Day as u64,
- ),
- day_max: RRA::new(
- flags | RRAFlags::CF_MAX,
- RRDTimeFrameResolution::Day as u64,
- ),
- week_avg: RRA::new(
- flags | RRAFlags::CF_AVERAGE,
- RRDTimeFrameResolution::Week as u64,
- ),
- week_max: RRA::new(
- flags | RRAFlags::CF_MAX,
- RRDTimeFrameResolution::Week as u64,
- ),
- month_avg: RRA::new(
- flags | RRAFlags::CF_AVERAGE,
- RRDTimeFrameResolution::Month as u64,
- ),
- month_max: RRA::new(
- flags | RRAFlags::CF_MAX,
- RRDTimeFrameResolution::Month as u64,
- ),
- year_avg: RRA::new(
- flags | RRAFlags::CF_AVERAGE,
- RRDTimeFrameResolution::Year as u64,
- ),
- year_max: RRA::new(
- flags | RRAFlags::CF_MAX,
- RRDTimeFrameResolution::Year as u64,
- ),
- }
- }
-
- /// Extract data from the archive
- pub fn extract_data(
+ fn extract_data(
&self,
- time: f64,
- timeframe: RRDTimeFrameResolution,
- mode: RRDMode,
+ start: u64,
+ end: u64,
+ last_update: f64,
) -> (u64, u64, Vec<Option<f64>>) {
-
- let epoch = time as u64;
- let reso = timeframe as u64;
-
- let end = reso*(epoch/reso + 1);
- let start = end - reso*(RRD_DATA_ENTRIES as u64);
+ let last_update = last_update as u64;
+ let reso = self.resolution;
+ let num_entries = self.data.len() as u64;
let mut list = Vec::new();
- let raa = match (mode, timeframe) {
- (RRDMode::Average, RRDTimeFrameResolution::Hour) => &self.hour_avg,
- (RRDMode::Max, RRDTimeFrameResolution::Hour) => &self.hour_max,
- (RRDMode::Average, RRDTimeFrameResolution::Day) => &self.day_avg,
- (RRDMode::Max, RRDTimeFrameResolution::Day) => &self.day_max,
- (RRDMode::Average, RRDTimeFrameResolution::Week) => &self.week_avg,
- (RRDMode::Max, RRDTimeFrameResolution::Week) => &self.week_max,
- (RRDMode::Average, RRDTimeFrameResolution::Month) => &self.month_avg,
- (RRDMode::Max, RRDTimeFrameResolution::Month) => &self.month_max,
- (RRDMode::Average, RRDTimeFrameResolution::Year) => &self.year_avg,
- (RRDMode::Max, RRDTimeFrameResolution::Year) => &self.year_max,
- };
-
- let rrd_end = reso*((raa.last_update as u64)/reso);
- let rrd_start = rrd_end - reso*(RRD_DATA_ENTRIES as u64);
+ let rrd_end = reso*(last_update/reso);
+ let rrd_start = rrd_end.saturating_sub(reso*num_entries);
let mut t = start;
- let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
- for _ in 0..RRD_DATA_ENTRIES {
+ let mut index = ((t/reso) % num_entries) as usize;
+ for _ in 0..num_entries {
+ if t > end { break; };
if t < rrd_start || t > rrd_end {
list.push(None);
} else {
- let value = raa.data[index];
+ let value = self.data[index];
if value.is_nan() {
list.push(None);
} else {
list.push(Some(value));
}
}
- t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+ t += reso; index = (index + 1) % (num_entries as usize);
}
(start, reso, list)
}
+}
- /// Create instance from raw data, testing data len and magic number
- pub fn from_raw(mut raw: &[u8]) -> Result<Self, std::io::Error> {
- let expected_len = std::mem::size_of::<RRD>();
- if raw.len() != expected_len {
- let msg = format!("wrong data size ({} != {})", raw.len(), expected_len);
- return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
- }
+#[derive(Serialize, Deserialize)]
+pub struct RRD {
+ pub source: DataSource,
+ pub rra_list: Vec<RRA>,
+}
- let mut rrd: RRD = unsafe { std::mem::zeroed() };
- unsafe {
- let rrd_slice = std::slice::from_raw_parts_mut(&mut rrd as *mut _ as *mut u8, expected_len);
- raw.read_exact(rrd_slice)?;
- }
+impl RRD {
- if rrd.magic != PROXMOX_RRD_MAGIC_1_0 {
- let msg = "wrong magic number".to_string();
- return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ pub fn new(dst: DST, rra_list: Vec<RRA>) -> RRD {
+
+ let source = DataSource::new(dst);
+
+ RRD {
+ source,
+ rra_list,
}
- Ok(rrd)
}
/// Load data from a file
pub fn load(path: &Path) -> Result<Self, std::io::Error> {
let raw = std::fs::read(path)?;
- Self::from_raw(&raw)
+ if raw.len() < 8 {
+ let msg = format!("not an rrd file - file is too small ({})", raw.len());
+ return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ }
+
+ if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
+ let v1 = rrd_v1::RRDv1::from_raw(&raw)?;
+ v1.to_rrd_v2()
+ .map_err(|err| {
+ let msg = format!("unable to convert from old V1 format - {}", err);
+ std::io::Error::new(std::io::ErrorKind::Other, msg)
+ })
+ } else if raw[0..8] == PROXMOX_RRD_MAGIC_2_0 {
+ serde_cbor::from_slice(&raw[8..])
+ .map_err(|err| {
+ let msg = format!("unable to decode RRD file - {}", err);
+ std::io::Error::new(std::io::ErrorKind::Other, msg)
+ })
+ } else {
+ let msg = format!("not an rrd file - unknown magic number");
+ return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ }
}
/// Store data into a file (atomic replace file)
pub fn save(&self, filename: &Path, options: CreateOptions) -> Result<(), Error> {
- let rrd_slice = unsafe {
- std::slice::from_raw_parts(self as *const _ as *const u8, std::mem::size_of::<RRD>())
- };
- replace_file(filename, rrd_slice, options)
+ let mut data: Vec<u8> = Vec::new();
+ data.extend(&PROXMOX_RRD_MAGIC_2_0);
+ serde_cbor::to_writer(&mut data, self)?;
+ replace_file(filename, &data, options)
}
pub fn last_update(&self) -> f64 {
-
- let mut last_update = 0.0;
-
- {
- let mut check_last_update = |rra: &RRA| {
- if rra.last_update > last_update {
- last_update = rra.last_update;
- }
- };
-
- check_last_update(&self.hour_avg);
- check_last_update(&self.hour_max);
-
- check_last_update(&self.day_avg);
- check_last_update(&self.day_max);
-
- check_last_update(&self.week_avg);
- check_last_update(&self.week_max);
-
- check_last_update(&self.month_avg);
- check_last_update(&self.month_max);
-
- check_last_update(&self.year_avg);
- check_last_update(&self.year_max);
- }
-
- last_update
+ self.source.last_update
}
/// Update the value (in memory)
@@ -371,32 +318,53 @@ impl RRD {
/// Note: This does not call [Self::save].
pub fn update(&mut self, time: f64, value: f64) {
- let mut log_error = true;
-
- let mut update_rra = |rra: &mut RRA| {
- if let Err(err) = rra.update(time, value) {
- if log_error {
- log::error!("rrd update failed: {}", err);
- // we only log the first error, because it is very
- // likely other calls produce the same error
- log_error = false;
- }
+ let value = match self.source.compute_new_value(time, value) {
+ Ok(value) => value,
+ Err(err) => {
+ log::error!("rrd update failed: {}", err);
+ return;
}
};
- update_rra(&mut self.hour_avg);
- update_rra(&mut self.hour_max);
-
- update_rra(&mut self.day_avg);
- update_rra(&mut self.day_max);
+ let last_update = self.source.last_update;
+ self.source.last_update = time;
- update_rra(&mut self.week_avg);
- update_rra(&mut self.week_max);
+ for rra in self.rra_list.iter_mut() {
+ rra.delete_old_slots(time, last_update);
+ rra.compute_new_value(time, last_update, value);
+ }
+ }
- update_rra(&mut self.month_avg);
- update_rra(&mut self.month_max);
+ /// Extract data from the archive
+ ///
+ /// This selects the RRA with specified [CF] and (minimum)
+ /// resolution, and extract data from `start` to `end`.
+ pub fn extract_data(
+ &self,
+ start: u64,
+ end: u64,
+ cf: CF,
+ resolution: u64,
+ ) -> Result<(u64, u64, Vec<Option<f64>>), Error> {
+
+ let mut rra: Option<&RRA> = None;
+ for item in self.rra_list.iter() {
+ if item.cf != cf { continue; }
+ if item.resolution > resolution { continue; }
+
+ if let Some(current) = rra {
+ if item.resolution > current.resolution {
+ rra = Some(item);
+ }
+ } else {
+ rra = Some(item);
+ }
+ }
- update_rra(&mut self.year_avg);
- update_rra(&mut self.year_max);
+ match rra {
+ Some(rra) => Ok(rra.extract_data(start, end, self.source.last_update)),
+ None => bail!("unable to find RRA suitable ({:?}:{})", cf, resolution),
+ }
}
+
}
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
new file mode 100644
index 00000000..919896f0
--- /dev/null
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -0,0 +1,296 @@
+use std::io::Read;
+
+use anyhow::Error;
+use bitflags::bitflags;
+
+/// The number of data entries per RRA
+pub const RRD_DATA_ENTRIES: usize = 70;
+
+/// Proxmox RRD file magic number
+// openssl::sha::sha256(b"Proxmox Round Robin Database file v1.0")[0..8];
+pub const PROXMOX_RRD_MAGIC_1_0: [u8; 8] = [206, 46, 26, 212, 172, 158, 5, 186];
+
+use crate::rrd::{RRD, RRA, CF, DST, DataSource};
+
+bitflags!{
+ /// Flags to specify the data soure type and consolidation function
+ pub struct RRAFlags: u64 {
+ // Data Source Types
+ const DST_GAUGE = 1;
+ const DST_DERIVE = 2;
+ const DST_COUNTER = 4;
+ const DST_MASK = 255; // first 8 bits
+
+ // Consolidation Functions
+ const CF_AVERAGE = 1 << 8;
+ const CF_MAX = 2 << 8;
+ const CF_MASK = 255 << 8;
+ }
+}
+
+/// Round Robin Archive with [RRD_DATA_ENTRIES] data slots.
+///
+/// This data structure is used inside [RRD] and directly written to the
+/// RRD files.
+#[repr(C)]
+pub struct RRAv1 {
+ /// Defined the data soure type and consolidation function
+ pub flags: RRAFlags,
+ /// Resulution (seconds) from [RRDTimeFrameResolution]
+ pub resolution: u64,
+ /// Last update time (epoch)
+ pub last_update: f64,
+ /// Count values computed inside this update interval
+ pub last_count: u64,
+ /// Stores the last value, used to compute differential value for derive/counters
+ pub counter_value: f64,
+ /// Data slots
+ pub data: [f64; RRD_DATA_ENTRIES],
+}
+
+impl RRAv1 {
+
+ fn extract_data(
+ &self,
+ ) -> (u64, u64, Vec<Option<f64>>) {
+ let reso = self.resolution;
+
+ let mut list = Vec::new();
+
+ let rra_end = reso*((self.last_update as u64)/reso);
+ let rra_start = rra_end - reso*(RRD_DATA_ENTRIES as u64);
+
+ let mut t = rra_start;
+ let mut index = ((t/reso) % (RRD_DATA_ENTRIES as u64)) as usize;
+ for _ in 0..RRD_DATA_ENTRIES {
+ let value = self.data[index];
+ if value.is_nan() {
+ list.push(None);
+ } else {
+ list.push(Some(value));
+ }
+
+ t += reso; index = (index + 1) % RRD_DATA_ENTRIES;
+ }
+
+ (rra_start, reso, list)
+ }
+}
+
+/// Round Robin Database file format with fixed number of [RRA]s
+#[repr(C)]
+// Note: Avoid alignment problems by using 8byte types only
+pub struct RRDv1 {
+ /// The magic number to identify the file type
+ pub magic: [u8; 8],
+ /// Hourly data (average values)
+ pub hour_avg: RRAv1,
+ /// Hourly data (maximum values)
+ pub hour_max: RRAv1,
+ /// Dayly data (average values)
+ pub day_avg: RRAv1,
+ /// Dayly data (maximum values)
+ pub day_max: RRAv1,
+ /// Weekly data (average values)
+ pub week_avg: RRAv1,
+ /// Weekly data (maximum values)
+ pub week_max: RRAv1,
+ /// Monthly data (average values)
+ pub month_avg: RRAv1,
+ /// Monthly data (maximum values)
+ pub month_max: RRAv1,
+ /// Yearly data (average values)
+ pub year_avg: RRAv1,
+ /// Yearly data (maximum values)
+ pub year_max: RRAv1,
+}
+
+impl RRDv1 {
+
+ pub fn from_raw(mut raw: &[u8]) -> Result<Self, std::io::Error> {
+
+ let expected_len = std::mem::size_of::<RRDv1>();
+
+ if raw.len() != expected_len {
+ let msg = format!("wrong data size ({} != {})", raw.len(), expected_len);
+ return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ }
+
+ let mut rrd: RRDv1 = unsafe { std::mem::zeroed() };
+ unsafe {
+ let rrd_slice = std::slice::from_raw_parts_mut(&mut rrd as *mut _ as *mut u8, expected_len);
+ raw.read_exact(rrd_slice)?;
+ }
+
+ if rrd.magic != PROXMOX_RRD_MAGIC_1_0 {
+ let msg = "wrong magic number".to_string();
+ return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ }
+
+ Ok(rrd)
+ }
+
+ pub fn to_rrd_v2(&self) -> Result<RRD, Error> {
+
+ let mut rra_list = Vec::new();
+
+ // old format v1:
+ //
+ // hour 1 min, 70 points
+ // day 30 min, 70 points
+ // week 3 hours, 70 points
+ // month 12 hours, 70 points
+ // year 1 week, 70 points
+ //
+ // new default for RRD v2:
+ //
+ // day 1 min, 1440 points
+ // month 30 min, 1440 points
+ // year 365 min (6h), 1440 points
+ // decade 1 week, 570 points
+
+ // Linear extrapolation
+ fn extrapolate_data(start: u64, reso: u64, factor: u64, data: Vec<Option<f64>>) -> (u64, u64, Vec<Option<f64>>) {
+
+ let mut new = Vec::new();
+
+ for i in 0..data.len() {
+ let mut next = i + 1;
+ if next >= data.len() { next = 0 };
+ let v = data[i];
+ let v1 = data[next];
+ match (v, v1) {
+ (Some(v), Some(v1)) => {
+ let diff = (v1 - v)/(factor as f64);
+ for j in 0..factor {
+ new.push(Some(v + diff*(j as f64)));
+ }
+ }
+ (Some(v), None) => {
+ new.push(Some(v));
+ for _ in 0..factor-1 {
+ new.push(None);
+ }
+ }
+ (None, Some(v1)) => {
+ for _ in 0..factor-1 {
+ new.push(None);
+ }
+ new.push(Some(v1));
+ }
+ (None, None) => {
+ for _ in 0..factor {
+ new.push(None);
+ }
+ }
+ }
+ }
+
+ (start, reso/factor, new)
+ }
+
+ // Try to convert to new, higher capacity format
+
+ // compute daily average (merge old self.day_avg and self.hour_avg
+ let mut day_avg = RRA::new(CF::Average, 60, 1440);
+
+ let (start, reso, data) = self.day_avg.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 30, data);
+ day_avg.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.hour_avg.extract_data();
+ day_avg.insert_data(start, reso, data)?;
+
+ // compute daily maximum (merge old self.day_max and self.hour_max
+ let mut day_max = RRA::new(CF::Maximum, 60, 1440);
+
+ let (start, reso, data) = self.day_max.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 30, data);
+ day_max.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.hour_max.extract_data();
+ day_max.insert_data(start, reso, data)?;
+
+ // compute montly average (merge old self.month_avg,
+ // self.week_avg and self.day_avg)
+ let mut month_avg = RRA::new(CF::Average, 30*60, 1440);
+
+ let (start, reso, data) = self.month_avg.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 24, data);
+ month_avg.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.week_avg.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 6, data);
+ month_avg.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.day_avg.extract_data();
+ month_avg.insert_data(start, reso, data)?;
+
+ // compute montly maximum (merge old self.month_max,
+ // self.week_max and self.day_max)
+ let mut month_max = RRA::new(CF::Maximum, 30*60, 1440);
+
+ let (start, reso, data) = self.month_max.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 24, data);
+ month_max.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.week_max.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 6, data);
+ month_max.insert_data(start, reso, data)?;
+
+ let (start, reso, data) = self.day_max.extract_data();
+ month_max.insert_data(start, reso, data)?;
+
+ // compute yearly average (merge old self.year_avg)
+ let mut year_avg = RRA::new(CF::Average, 6*3600, 1440);
+
+ let (start, reso, data) = self.year_avg.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 28, data);
+ year_avg.insert_data(start, reso, data)?;
+
+ // compute yearly maximum (merge old self.year_avg)
+ let mut year_max = RRA::new(CF::Maximum, 6*3600, 1440);
+
+ let (start, reso, data) = self.year_max.extract_data();
+ let (start, reso, data) = extrapolate_data(start, reso, 28, data);
+ year_max.insert_data(start, reso, data)?;
+
+ // compute decade average (merge old self.year_avg)
+ let mut decade_avg = RRA::new(CF::Average, 7*86400, 570);
+ let (start, reso, data) = self.year_avg.extract_data();
+ decade_avg.insert_data(start, reso, data)?;
+
+ // compute decade maximum (merge old self.year_max)
+ let mut decade_max = RRA::new(CF::Maximum, 7*86400, 570);
+ let (start, reso, data) = self.year_max.extract_data();
+ decade_max.insert_data(start, reso, data)?;
+
+ rra_list.push(day_avg);
+ rra_list.push(day_max);
+ rra_list.push(month_avg);
+ rra_list.push(month_max);
+ rra_list.push(year_avg);
+ rra_list.push(year_max);
+ rra_list.push(decade_avg);
+ rra_list.push(decade_max);
+
+ // use values from hour_avg for source (all RRAv1 must have the same config)
+ let dst = if self.hour_avg.flags.contains(RRAFlags::DST_COUNTER) {
+ DST::Counter
+ } else if self.hour_avg.flags.contains(RRAFlags::DST_DERIVE) {
+ DST::Derive
+ } else {
+ DST::Gauge
+ };
+
+ let source = DataSource {
+ dst,
+ counter_value: f64::NAN,
+ last_update: self.hour_avg.last_update, // IMPORTANT!
+ };
+ Ok(RRD {
+ source,
+ rra_list,
+ })
+ }
+}
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index b53076d7..4fe18f46 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -1,5 +1,6 @@
-use anyhow::Error;
+use anyhow::{bail, Error};
use serde_json::{Value, json};
+use std::collections::BTreeMap;
use proxmox_router::{Permission, Router};
use proxmox_schema::api;
@@ -8,8 +9,6 @@ use pbs_api_types::{
NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
};
-use proxmox_rrd::rrd::RRD_DATA_ENTRIES;
-
use crate::get_rrd_cache;
pub fn create_value_from_rrd(
@@ -19,32 +18,44 @@ pub fn create_value_from_rrd(
cf: RRDMode,
) -> Result<Value, Error> {
- let mut result = Vec::new();
+ let mut result: Vec<Value> = Vec::new();
let now = proxmox_time::epoch_f64();
let rrd_cache = get_rrd_cache()?;
+ let mut timemap = BTreeMap::new();
+
+ let mut last_resolution = None;
+
for name in list {
- let (start, reso, list) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf) {
+ let (start, reso, data) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf)? {
Some(result) => result,
None => continue,
};
+ if let Some(expected_resolution) = last_resolution {
+ if reso != expected_resolution {
+ bail!("got unexpected RRD resolution ({} != {})", reso, expected_resolution);
+ }
+ } else {
+ last_resolution = Some(reso);
+ }
+
let mut t = start;
- for index in 0..RRD_DATA_ENTRIES {
- if result.len() <= index {
- if let Some(value) = list[index] {
- result.push(json!({ "time": t, *name: value }));
- } else {
- result.push(json!({ "time": t }));
- }
- } else if let Some(value) = list[index] {
- result[index][name] = value.into();
+
+ for index in 0..data.len() {
+ let entry = timemap.entry(t).or_insert(json!({ "time": t }));
+ if let Some(value) = data[index] {
+ entry[*name] = value.into();
}
t += reso;
}
}
+ for item in timemap.values() {
+ result.push(item.clone());
+ }
+
Ok(result.into())
}
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 40a9e5b4..2476fe97 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -134,8 +134,8 @@ pub fn datastore_status(
RRDMode::Average,
);
- let total_res = get_rrd("total");
- let used_res = get_rrd("used");
+ let total_res = get_rrd("total")?;
+ let used_res = get_rrd("used")?;
if let (Some((start, reso, total_list)), Some((_, _, used_list))) = (total_res, used_res) {
let mut usage_list: Vec<f64> = Vec::new();
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 0f0f6f59..2abc3eaa 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -24,7 +24,7 @@ use proxmox_router::{RpcEnvironment, RpcEnvironmentType, UserInformation};
use pbs_tools::{task_log, task_warn};
use pbs_datastore::DataStore;
-use proxmox_rrd::DST;
+use proxmox_rrd::rrd::DST;
use proxmox_rest_server::{
rotate_task_log_archive, extract_cookie , AuthError, ApiConfig, RestServer, RestEnvironment,
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (3 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
` (10 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/Cargo.toml | 2 --
proxmox-rrd/src/cache.rs | 30 ++++++++----------------------
proxmox-rrd/src/rrd.rs | 13 ++++++++++---
src/api2/node/rrd.rs | 9 +++------
src/api2/status.rs | 9 ++-------
src/lib.rs | 32 +++++++++++++++++++++++++++++++-
6 files changed, 54 insertions(+), 41 deletions(-)
diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index b3dd02c3..69a68530 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -17,5 +17,3 @@ serde_cbor = "0.11.1"
proxmox = { version = "0.14.0" }
proxmox-time = "1"
proxmox-schema = { version = "1", features = [ "api-macro" ] }
-
-proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index f14837fc..5e359d9b 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -11,8 +11,6 @@ use nix::fcntl::OFlag;
use proxmox::tools::fs::{atomic_open_or_create_file, create_path, CreateOptions};
-use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
-
use crate::rrd::{DST, CF, RRD, RRA};
const RRD_JOURNAL_NAME: &str = "rrd.journal";
@@ -280,35 +278,23 @@ impl RRDCache {
}
/// Extract data from cached RRD
+ ///
+ /// `start`: Start time. If not sepecified, we simply extract 10 data points.
+ /// `end`: End time. Default is to use the current time.
pub fn extract_cached_data(
&self,
base: &str,
name: &str,
- now: f64,
- timeframe: RRDTimeFrameResolution,
- mode: RRDMode,
+ cf: CF,
+ resolution: u64,
+ start: Option<u64>,
+ end: Option<u64>,
) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
let state = self.state.read().unwrap();
- let cf = match mode {
- RRDMode::Max => CF::Maximum,
- RRDMode::Average => CF::Average,
- };
-
- let now = now as u64;
-
- let (start, resolution) = match timeframe {
- RRDTimeFrameResolution::Hour => (now - 3600, 60),
- RRDTimeFrameResolution::Day => (now - 3600*24, 60),
- RRDTimeFrameResolution::Week => (now - 3600*24*7, 30*60),
- RRDTimeFrameResolution::Month => (now - 3600*24*30, 30*60),
- RRDTimeFrameResolution::Year => (now - 3600*24*365, 6*60*60),
- RRDTimeFrameResolution::Decade => (now - 10*3600*24*366, 7*86400),
- };
-
match state.rrd_map.get(&format!("{}/{}", base, name)) {
- Some(rrd) => Ok(Some(rrd.extract_data(start, now, cf, resolution)?)),
+ Some(rrd) => Ok(Some(rrd.extract_data(cf, resolution, start, end)?)),
None => Ok(None),
}
}
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 82fa5a3a..c97be96d 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -339,12 +339,15 @@ impl RRD {
///
/// This selects the RRA with specified [CF] and (minimum)
/// resolution, and extract data from `start` to `end`.
+ ///
+ /// `start`: Start time. If not sepecified, we simply extract 10 data points.
+ /// `end`: End time. Default is to use the current time.
pub fn extract_data(
&self,
- start: u64,
- end: u64,
cf: CF,
resolution: u64,
+ start: Option<u64>,
+ end: Option<u64>,
) -> Result<(u64, u64, Vec<Option<f64>>), Error> {
let mut rra: Option<&RRA> = None;
@@ -362,7 +365,11 @@ impl RRD {
}
match rra {
- Some(rra) => Ok(rra.extract_data(start, end, self.source.last_update)),
+ Some(rra) => {
+ let end = end.unwrap_or_else(|| proxmox_time::epoch_f64() as u64);
+ let start = start.unwrap_or(end - 10*rra.resolution);
+ Ok(rra.extract_data(start, end, self.source.last_update))
+ }
None => bail!("unable to find RRA suitable ({:?}:{})", cf, resolution),
}
}
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index 4fe18f46..21a55bb7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -9,26 +9,23 @@ use pbs_api_types::{
NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
};
-use crate::get_rrd_cache;
+use crate::extract_rrd_data;
pub fn create_value_from_rrd(
basedir: &str,
list: &[&str],
timeframe: RRDTimeFrameResolution,
- cf: RRDMode,
+ mode: RRDMode,
) -> Result<Value, Error> {
let mut result: Vec<Value> = Vec::new();
- let now = proxmox_time::epoch_f64();
-
- let rrd_cache = get_rrd_cache()?;
let mut timemap = BTreeMap::new();
let mut last_resolution = None;
for name in list {
- let (start, reso, data) = match rrd_cache.extract_cached_data(basedir, name, now, timeframe, cf)? {
+ let (start, reso, data) = match extract_rrd_data(basedir, name, timeframe, mode)? {
Some(result) => result,
None => continue,
};
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 2476fe97..9a72ce52 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -22,7 +22,7 @@ use pbs_datastore::DataStore;
use pbs_config::CachedUserInfo;
use crate::tools::statistics::{linear_regression};
-use crate::get_rrd_cache;
+use crate::extract_rrd_data;
#[api(
returns: {
@@ -90,8 +90,6 @@ pub fn datastore_status(
let mut list = Vec::new();
- let rrd_cache = get_rrd_cache()?;
-
for (store, (_, _)) in &config.sections {
let user_privs = user_info.lookup_privs(&auth_id, &["datastore", &store]);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT| PRIV_DATASTORE_BACKUP)) != 0;
@@ -123,13 +121,10 @@ pub fn datastore_status(
});
let rrd_dir = format!("datastore/{}", store);
- let now = proxmox_time::epoch_f64();
-
- let get_rrd = |what: &str| rrd_cache.extract_cached_data(
+ let get_rrd = |what: &str| extract_rrd_data(
&rrd_dir,
what,
- now,
RRDTimeFrameResolution::Month,
RRDMode::Average,
);
diff --git a/src/lib.rs b/src/lib.rs
index a1ac23bf..6745627a 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -10,9 +10,10 @@ use anyhow::{format_err, Error};
use proxmox::tools::fs::CreateOptions;
+use pbs_api_types::{RRDMode, RRDTimeFrameResolution};
use pbs_buildcfg::configdir;
use pbs_tools::cert::CertInfo;
-use proxmox_rrd::RRDCache;
+use proxmox_rrd::{rrd::CF, RRDCache};
#[macro_use]
pub mod tools;
@@ -78,3 +79,32 @@ pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
Ok(RRD_CACHE.get().unwrap())
}
+
+/// Extracts data for the specified time frame from from RRD cache
+pub fn extract_rrd_data(
+ basedir: &str,
+ name: &str,
+ timeframe: RRDTimeFrameResolution,
+ mode: RRDMode,
+) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
+
+ let end = proxmox_time::epoch_f64() as u64;
+
+ let (start, resolution) = match timeframe {
+ RRDTimeFrameResolution::Hour => (end - 3600, 60),
+ RRDTimeFrameResolution::Day => (end - 3600*24, 60),
+ RRDTimeFrameResolution::Week => (end - 3600*24*7, 30*60),
+ RRDTimeFrameResolution::Month => (end - 3600*24*30, 30*60),
+ RRDTimeFrameResolution::Year => (end - 3600*24*365, 6*60*60),
+ RRDTimeFrameResolution::Decade => (end - 10*3600*24*366, 7*86400),
+ };
+
+ let cf = match mode {
+ RRDMode::Max => CF::Maximum,
+ RRDMode::Average => CF::Average,
+ };
+
+ let rrd_cache = get_rrd_cache()?;
+
+ rrd_cache.extract_cached_data(basedir, name, cf, resolution, Some(start), Some(end))
+}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (4 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
` (9 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index c97be96d..328ac9f2 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -231,7 +231,7 @@ impl RRA {
let mut list = Vec::new();
- let rrd_end = reso*(last_update/reso);
+ let rrd_end = reso*(last_update/reso + 1);
let rrd_start = rrd_end.saturating_sub(reso*num_entries);
let mut t = start;
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (5 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
` (8 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
---
Cargo.toml | 2 --
Makefile | 1 -
pbs-api-types/Cargo.toml | 1 -
pbs-api-types/src/lib.rs | 30 ++++++++++++++++++++++++++++-
proxmox-rrd-api-types/Cargo.toml | 11 -----------
proxmox-rrd-api-types/src/lib.rs | 33 --------------------------------
proxmox-rrd/src/rrd_v1.rs | 2 +-
src/api2/admin/datastore.rs | 6 +++---
src/api2/node/rrd.rs | 8 ++++----
src/api2/status.rs | 4 ++--
src/lib.rs | 16 ++++++++--------
11 files changed, 47 insertions(+), 67 deletions(-)
delete mode 100644 proxmox-rrd-api-types/Cargo.toml
delete mode 100644 proxmox-rrd-api-types/src/lib.rs
diff --git a/Cargo.toml b/Cargo.toml
index d2e3fa1f..772588f6 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -27,7 +27,6 @@ members = [
"pbs-fuse-loop",
"pbs-runtime",
"proxmox-rest-server",
- "proxmox-rrd-api-types",
"proxmox-rrd",
"proxmox-systemd",
"pbs-tape",
@@ -119,7 +118,6 @@ pbs-config = { path = "pbs-config" }
pbs-datastore = { path = "pbs-datastore" }
pbs-runtime = { path = "pbs-runtime" }
proxmox-rest-server = { path = "proxmox-rest-server" }
-proxmox-rrd-api-types = { path = "proxmox-rrd-api-types" }
proxmox-rrd = { path = "proxmox-rrd" }
proxmox-systemd = { path = "proxmox-systemd" }
pbs-tools = { path = "pbs-tools" }
diff --git a/Makefile b/Makefile
index 6b098aa1..d6951c9a 100644
--- a/Makefile
+++ b/Makefile
@@ -40,7 +40,6 @@ SUBCRATES := \
pbs-fuse-loop \
pbs-runtime \
proxmox-rest-server \
- proxmox-rrd-api-types \
proxmox-rrd \
proxmox-systemd \
pbs-tape \
diff --git a/pbs-api-types/Cargo.toml b/pbs-api-types/Cargo.toml
index 2a51bd3a..11644399 100644
--- a/pbs-api-types/Cargo.toml
+++ b/pbs-api-types/Cargo.toml
@@ -20,6 +20,5 @@ proxmox-schema = { version = "1.0.0", features = [ "api-macro" ] }
proxmox-time = "1.0.0"
proxmox-uuid = { version = "1.0.0", features = [ "serde" ] }
-proxmox-rrd-api-types = { path = "../proxmox-rrd-api-types" }
proxmox-systemd = { path = "../proxmox-systemd" }
pbs-tools = { path = "../pbs-tools" }
diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index cdf765a1..96ac657b 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -424,4 +424,32 @@ pub const NODE_TASKS_LIST_TASKS_RETURN_TYPE: ReturnType = ReturnType {
).schema(),
};
-pub use proxmox_rrd_api_types::{RRDMode, RRDTimeFrameResolution};
+#[api()]
+#[derive(Copy, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+/// RRD consolidation mode
+pub enum RRDMode {
+ /// Maximum
+ Max,
+ /// Average
+ Average,
+}
+
+#[api()]
+#[derive(Copy, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+/// RRD time frame
+pub enum RRDTimeFrame {
+ /// Hour
+ Hour,
+ /// Day
+ Day,
+ /// Week
+ Week,
+ /// Month
+ Month,
+ /// Year
+ Year,
+ /// Decade (10 years)
+ Decade,
+}
diff --git a/proxmox-rrd-api-types/Cargo.toml b/proxmox-rrd-api-types/Cargo.toml
deleted file mode 100644
index 816f7fde..00000000
--- a/proxmox-rrd-api-types/Cargo.toml
+++ /dev/null
@@ -1,11 +0,0 @@
-[package]
-name = "proxmox-rrd-api-types"
-version = "0.1.0"
-authors = ["Proxmox Support Team <support@proxmox.com>"]
-edition = "2018"
-description = "API type definitions for proxmox-rrd crate."
-
-
-[dependencies]
-serde = { version = "1.0", features = ["derive"] }
-proxmox-schema = { version = "1", features = ["api-macro"] }
diff --git a/proxmox-rrd-api-types/src/lib.rs b/proxmox-rrd-api-types/src/lib.rs
deleted file mode 100644
index 32601477..00000000
--- a/proxmox-rrd-api-types/src/lib.rs
+++ /dev/null
@@ -1,33 +0,0 @@
-use serde::{Deserialize, Serialize};
-
-use proxmox_schema::api;
-
-#[api()]
-#[derive(Copy, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "UPPERCASE")]
-/// RRD consolidation mode
-pub enum RRDMode {
- /// Maximum
- Max,
- /// Average
- Average,
-}
-
-#[api()]
-#[derive(Copy, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-/// RRD time frame resolution
-pub enum RRDTimeFrameResolution {
- /// Hour
- Hour,
- /// Day
- Day,
- /// Week
- Week,
- /// Month
- Month,
- /// Year
- Year,
- /// Decade (10 years)
- Decade,
-}
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
index 919896f0..511b510b 100644
--- a/proxmox-rrd/src/rrd_v1.rs
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -36,7 +36,7 @@ bitflags!{
pub struct RRAv1 {
/// Defined the data soure type and consolidation function
pub flags: RRAFlags,
- /// Resulution (seconds) from [RRDTimeFrameResolution]
+ /// Resulution (seconds)
pub resolution: u64,
/// Last update time (epoch)
pub last_update: f64,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7e98b4a9..fb5647ce 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -28,7 +28,7 @@ use pxar::EntryKind;
use pbs_api_types::{ Authid, BackupContent, Counts, CryptMode,
DataStoreListItem, GarbageCollectionStatus, GroupListItem,
SnapshotListItem, SnapshotVerifyState, PruneOptions,
- DataStoreStatus, RRDMode, RRDTimeFrameResolution,
+ DataStoreStatus, RRDMode, RRDTimeFrame,
BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_TIME_SCHEMA,
BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA,
IGNORE_VERIFIED_BACKUPS_SCHEMA, UPID_SCHEMA,
@@ -1537,7 +1537,7 @@ pub fn pxar_file_download(
schema: DATASTORE_SCHEMA,
},
timeframe: {
- type: RRDTimeFrameResolution,
+ type: RRDTimeFrame,
},
cf: {
type: RRDMode,
@@ -1551,7 +1551,7 @@ pub fn pxar_file_download(
/// Read datastore stats
pub fn get_rrd_stats(
store: String,
- timeframe: RRDTimeFrameResolution,
+ timeframe: RRDTimeFrame,
cf: RRDMode,
_param: Value,
) -> Result<Value, Error> {
diff --git a/src/api2/node/rrd.rs b/src/api2/node/rrd.rs
index 21a55bb7..9b6780b7 100644
--- a/src/api2/node/rrd.rs
+++ b/src/api2/node/rrd.rs
@@ -6,7 +6,7 @@ use proxmox_router::{Permission, Router};
use proxmox_schema::api;
use pbs_api_types::{
- NODE_SCHEMA, RRDMode, RRDTimeFrameResolution, PRIV_SYS_AUDIT,
+ NODE_SCHEMA, RRDMode, RRDTimeFrame, PRIV_SYS_AUDIT,
};
use crate::extract_rrd_data;
@@ -14,7 +14,7 @@ use crate::extract_rrd_data;
pub fn create_value_from_rrd(
basedir: &str,
list: &[&str],
- timeframe: RRDTimeFrameResolution,
+ timeframe: RRDTimeFrame,
mode: RRDMode,
) -> Result<Value, Error> {
@@ -63,7 +63,7 @@ pub fn create_value_from_rrd(
schema: NODE_SCHEMA,
},
timeframe: {
- type: RRDTimeFrameResolution,
+ type: RRDTimeFrame,
},
cf: {
type: RRDMode,
@@ -76,7 +76,7 @@ pub fn create_value_from_rrd(
)]
/// Read node stats
fn get_node_stats(
- timeframe: RRDTimeFrameResolution,
+ timeframe: RRDTimeFrame,
cf: RRDMode,
_param: Value,
) -> Result<Value, Error> {
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 9a72ce52..a3456ad4 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -14,7 +14,7 @@ use proxmox_router::{
use proxmox_router::list_subdirs_api_method;
use pbs_api_types::{
- Authid, DATASTORE_SCHEMA, RRDMode, RRDTimeFrameResolution,
+ Authid, DATASTORE_SCHEMA, RRDMode, RRDTimeFrame,
PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
};
@@ -125,7 +125,7 @@ pub fn datastore_status(
let get_rrd = |what: &str| extract_rrd_data(
&rrd_dir,
what,
- RRDTimeFrameResolution::Month,
+ RRDTimeFrame::Month,
RRDMode::Average,
);
diff --git a/src/lib.rs b/src/lib.rs
index 6745627a..a5a28190 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -10,7 +10,7 @@ use anyhow::{format_err, Error};
use proxmox::tools::fs::CreateOptions;
-use pbs_api_types::{RRDMode, RRDTimeFrameResolution};
+use pbs_api_types::{RRDMode, RRDTimeFrame};
use pbs_buildcfg::configdir;
use pbs_tools::cert::CertInfo;
use proxmox_rrd::{rrd::CF, RRDCache};
@@ -84,19 +84,19 @@ pub fn initialize_rrd_cache() -> Result<&'static RRDCache, Error> {
pub fn extract_rrd_data(
basedir: &str,
name: &str,
- timeframe: RRDTimeFrameResolution,
+ timeframe: RRDTimeFrame,
mode: RRDMode,
) -> Result<Option<(u64, u64, Vec<Option<f64>>)>, Error> {
let end = proxmox_time::epoch_f64() as u64;
let (start, resolution) = match timeframe {
- RRDTimeFrameResolution::Hour => (end - 3600, 60),
- RRDTimeFrameResolution::Day => (end - 3600*24, 60),
- RRDTimeFrameResolution::Week => (end - 3600*24*7, 30*60),
- RRDTimeFrameResolution::Month => (end - 3600*24*30, 30*60),
- RRDTimeFrameResolution::Year => (end - 3600*24*365, 6*60*60),
- RRDTimeFrameResolution::Decade => (end - 10*3600*24*366, 7*86400),
+ RRDTimeFrame::Hour => (end - 3600, 60),
+ RRDTimeFrame::Day => (end - 3600*24, 60),
+ RRDTimeFrame::Week => (end - 3600*24*7, 30*60),
+ RRDTimeFrame::Month => (end - 3600*24*30, 30*60),
+ RRDTimeFrame::Year => (end - 3600*24*365, 6*60*60),
+ RRDTimeFrame::Decade => (end - 10*3600*24*366, 7*86400),
};
let cf = match mode {
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (6 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
` (7 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 3 +++
1 file changed, 3 insertions(+)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 328ac9f2..7a9ce94a 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -51,6 +51,8 @@ pub enum CF {
Maximum,
/// Minimum
Minimum,
+ /// Use the last value
+ Last,
}
#[derive(Serialize, Deserialize)]
@@ -209,6 +211,7 @@ impl RRA {
let new_value = match self.cf {
CF::Maximum => if last_value > value { last_value } else { value },
CF::Minimum => if last_value < value { last_value } else { value },
+ CF::Last => value,
CF::Average => {
(last_value*(self.last_count as f64))/(new_count as f64)
+ value/(new_count as f64)
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup)
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (7 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
` (6 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/cache.rs | 37 ++++++++++++++++++-------------------
1 file changed, 18 insertions(+), 19 deletions(-)
diff --git a/proxmox-rrd/src/cache.rs b/proxmox-rrd/src/cache.rs
index 5e359d9b..5225bd49 100644
--- a/proxmox-rrd/src/cache.rs
+++ b/proxmox-rrd/src/cache.rs
@@ -194,15 +194,8 @@ impl RRDCache {
path.push(&entry.rel_path);
create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
- let mut rrd = match RRD::load(&path) {
- Ok(rrd) => rrd,
- Err(err) => {
- if err.kind() != std::io::ErrorKind::NotFound {
- log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
- }
- Self::create_default_rrd(entry.dst)
- },
- };
+ let mut rrd = Self::load_rrd(&path, entry.dst);
+
if entry.time > get_last_update(&entry.rel_path, &rrd) {
rrd.update(entry.time, entry.value);
}
@@ -235,7 +228,19 @@ impl RRDCache {
Ok(())
}
- /// Update data in RAM and write file back to disk (if `save` is set)
+ fn load_rrd(path: &Path, dst: DST) -> RRD {
+ match RRD::load(path) {
+ Ok(rrd) => rrd,
+ Err(err) => {
+ if err.kind() != std::io::ErrorKind::NotFound {
+ log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
+ }
+ Self::create_default_rrd(dst)
+ },
+ }
+ }
+
+ /// Update data in RAM and write file back to disk (journal)
pub fn update_value(
&self,
rel_path: &str,
@@ -261,15 +266,9 @@ impl RRDCache {
let mut path = self.basedir.clone();
path.push(rel_path);
create_path(path.parent().unwrap(), Some(self.dir_options.clone()), Some(self.dir_options.clone()))?;
- let mut rrd = match RRD::load(&path) {
- Ok(rrd) => rrd,
- Err(err) => {
- if err.kind() != std::io::ErrorKind::NotFound {
- log::warn!("overwriting RRD file {:?}, because of load error: {}", path, err);
- }
- Self::create_default_rrd(dst)
- },
- };
+
+ let mut rrd = Self::load_rrd(&path, dst);
+
rrd.update(now, value);
state.rrd_map.insert(rel_path.into(), rrd);
}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (8 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
` (5 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/Cargo.toml | 1 +
proxmox-rrd/src/bin/rrd.rs | 215 +++++++++++++++++++++++++++++++++++++
2 files changed, 216 insertions(+)
create mode 100644 proxmox-rrd/src/bin/rrd.rs
diff --git a/proxmox-rrd/Cargo.toml b/proxmox-rrd/Cargo.toml
index 69a68530..e3ab5380 100644
--- a/proxmox-rrd/Cargo.toml
+++ b/proxmox-rrd/Cargo.toml
@@ -16,4 +16,5 @@ serde_cbor = "0.11.1"
proxmox = { version = "0.14.0" }
proxmox-time = "1"
+proxmox-router = "1"
proxmox-schema = { version = "1", features = [ "api-macro" ] }
diff --git a/proxmox-rrd/src/bin/rrd.rs b/proxmox-rrd/src/bin/rrd.rs
new file mode 100644
index 00000000..fdb61ffd
--- /dev/null
+++ b/proxmox-rrd/src/bin/rrd.rs
@@ -0,0 +1,215 @@
+//! RRD toolkit - create/manage/update proxmox RRD (v2) file
+
+use std::path::PathBuf;
+
+use anyhow::{bail, Error};
+use serde::{Serialize, Deserialize};
+
+use proxmox_router::RpcEnvironment;
+use proxmox_router::cli::{run_cli_command, CliCommand, CliCommandMap, CliEnvironment};
+use proxmox_schema::{api, parse_property_string};
+use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema};
+
+use proxmox::tools::fs::CreateOptions;
+
+use proxmox_rrd::rrd::{CF, DST, RRA, RRD};
+
+pub const RRA_CONFIG_STRING_SCHEMA: Schema = StringSchema::new(
+ "RRA configuration")
+ .format(&ApiStringFormat::PropertyString(&RRAConfig::API_SCHEMA))
+ .schema();
+
+#[api(
+ properties: {},
+ default_key: "cf",
+)]
+#[derive(Debug, Serialize, Deserialize)]
+/// RRA configuration
+pub struct RRAConfig {
+ /// Time resolution
+ pub r: u64,
+ pub cf: CF,
+ /// Number of data points
+ pub n: u64,
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ },
+ },
+)]
+/// Dump the RRDB database in JSON format
+pub fn dump_rrdb(path: String) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+ serde_json::to_writer_pretty(std::io::stdout(), &rrd)?;
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ time: {
+ description: "Update time.",
+ optional: true,
+ },
+ value: {
+ description: "Update value.",
+ },
+ },
+ },
+)]
+/// Update the RRDB database
+pub fn update_rrdb(
+ path: String,
+ time: Option<u64>,
+ value: f64,
+) -> Result<(), Error> {
+
+ let path = PathBuf::from(path);
+
+ let time = time.map(|v| v as f64)
+ .unwrap_or_else(proxmox_time::epoch_f64);
+
+ let mut rrd = RRD::load(&path)?;
+ rrd.update(time, value);
+
+ rrd.save(&path, CreateOptions::new())?;
+
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ cf: {
+ type: CF,
+ },
+ resolution: {
+ description: "Time resulution",
+ },
+ start: {
+ description: "Start time. If not sepecified, we simply extract 10 data points.",
+ optional: true,
+ },
+ end: {
+ description: "End time (Unix Epoch). Default is the last update time.",
+ optional: true,
+ },
+ },
+ },
+)]
+/// Fetch data from the RRDB database
+pub fn fetch_rrdb(
+ path: String,
+ cf: CF,
+ resolution: u64,
+ start: Option<u64>,
+ end: Option<u64>,
+) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+
+ let data = rrd.extract_data(cf, resolution, start, end)?;
+
+ println!("{}", serde_json::to_string_pretty(&data)?);
+
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ dst: {
+ type: DST,
+ },
+ path: {
+ description: "The filename to create."
+ },
+ rra: {
+ description: "Configuration of contained RRAs.",
+ type: Array,
+ items: {
+ schema: RRA_CONFIG_STRING_SCHEMA,
+ }
+ },
+ },
+ },
+)]
+/// Create a new RRDB database file
+pub fn create_rrdb(
+ dst: DST,
+ path: String,
+ rra: Vec<String>,
+) -> Result<(), Error> {
+
+ let mut rra_list = Vec::new();
+
+ for item in rra.iter() {
+ let rra: RRAConfig = serde_json::from_value(
+ parse_property_string(item, &RRAConfig::API_SCHEMA)?
+ )?;
+ println!("GOT {:?}", rra);
+ rra_list.push(RRA::new(rra.cf, rra.r, rra.n as usize));
+ }
+
+ let path = PathBuf::from(path);
+
+ let rrd = RRD::new(dst, rra_list);
+
+ rrd.save(&path, CreateOptions::new())?;
+
+ Ok(())
+}
+
+
+fn main() -> Result<(), Error> {
+
+ let uid = nix::unistd::Uid::current();
+
+ let username = match nix::unistd::User::from_uid(uid)? {
+ Some(user) => user.name,
+ None => bail!("unable to get user name"),
+ };
+
+ let cmd_def = CliCommandMap::new()
+ .insert(
+ "create",
+ CliCommand::new(&API_METHOD_CREATE_RRDB)
+ .arg_param(&["path"])
+ )
+ .insert(
+ "update",
+ CliCommand::new(&API_METHOD_UPDATE_RRDB)
+ .arg_param(&["path"])
+ )
+ .insert(
+ "fetch",
+ CliCommand::new(&API_METHOD_FETCH_RRDB)
+ .arg_param(&["path"])
+ )
+ .insert(
+ "dump",
+ CliCommand::new(&API_METHOD_DUMP_RRDB)
+ .arg_param(&["path"])
+ )
+ ;
+
+ let mut rpcenv = CliEnvironment::new();
+ rpcenv.set_auth_id(Some(format!("{}@pam", username)));
+
+ run_cli_command(cmd_def, rpcenv, None);
+
+ Ok(())
+
+}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (9 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
` (4 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
Modulo is very slow, so we try to avoid it inside loops.
---
proxmox-rrd/src/rrd.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 7a9ce94a..54cb8b48 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -153,8 +153,7 @@ impl RRA {
if let Some(v) = data[i] {
self.data[index] = v;
}
- index += 1;
- if index >= self.data.len() { index = 0; }
+ index += 1; if index >= self.data.len() { index = 0; }
}
Ok(())
}
@@ -171,7 +170,7 @@ impl RRA {
let mut index = ((t/reso) % num_entries) as usize;
for _ in 0..num_entries {
t += reso;
- index = (index + 1) % (num_entries as usize);
+ index += 1; if index >= self.data.len() { index = 0; }
if t < min_time {
self.data[index] = f64::NAN;
} else {
@@ -251,7 +250,8 @@ impl RRA {
list.push(Some(value));
}
}
- t += reso; index = (index + 1) % (num_entries as usize);
+ t += reso;
+ index += 1; if index >= self.data.len() { index = 0; }
}
(start, reso, list)
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time()
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (10 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
` (3 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 54cb8b48..12d2ec8c 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -135,6 +135,14 @@ impl RRA {
}
}
+ fn slot_end_time(&self, time: u64) -> u64 {
+ self.resolution*(time/self.resolution + 1)
+ }
+
+ fn slot(&self, time: u64) -> usize {
+ ((time/self.resolution) as usize) % self.data.len()
+ }
+
// directly overwrite data slots
// the caller need to set last_update value on the DataSource manually.
pub(crate) fn insert_data(
@@ -146,8 +154,8 @@ impl RRA {
if resolution != self.resolution {
bail!("inser_data failed: got wrong resolution");
}
- let num_entries = self.data.len() as u64;
- let mut index = ((start/self.resolution) % num_entries) as usize;
+
+ let mut index = self.slot(start);
for i in 0..data.len() {
if let Some(v) = data[i] {
@@ -167,7 +175,9 @@ impl RRA {
let min_time = epoch - num_entries*reso;
let min_time = (min_time/reso + 1)*reso;
let mut t = last_update.saturating_sub(num_entries*reso);
- let mut index = ((t/reso) % num_entries) as usize;
+
+ let mut index = self.slot(t);
+
for _ in 0..num_entries {
t += reso;
index += 1; if index >= self.data.len() { index = 0; }
@@ -183,12 +193,11 @@ impl RRA {
let epoch = time as u64;
let last_update = last_update as u64;
let reso = self.resolution;
- let num_entries = self.data.len() as u64;
- let index = ((epoch/reso) % num_entries) as usize;
- let last_index = ((last_update/reso) % num_entries) as usize;
+ let index = self.slot(epoch);
+ let last_index = self.slot(last_update);
- if (epoch - (last_update as u64)) > reso || index != last_index {
+ if (epoch - last_update) > reso || index != last_index {
self.last_count = 0;
}
@@ -233,11 +242,11 @@ impl RRA {
let mut list = Vec::new();
- let rrd_end = reso*(last_update/reso + 1);
+ let rrd_end = self.slot_end_time(last_update);
let rrd_start = rrd_end.saturating_sub(reso*num_entries);
let mut t = start;
- let mut index = ((t/reso) % num_entries) as usize;
+ let mut index = self.slot(t);
for _ in 0..num_entries {
if t > end { break; };
if t < rrd_start || t > rrd_end {
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (11 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
` (2 subsequent siblings)
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 45 +++++++++++++++++++++++++-----------------
1 file changed, 27 insertions(+), 18 deletions(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 12d2ec8c..7901fe39 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -13,7 +13,7 @@
use std::path::Path;
-use anyhow::{bail, Error};
+use anyhow::{bail, format_err, Error};
use serde::{Serialize, Deserialize};
@@ -77,6 +77,9 @@ impl DataSource {
}
fn compute_new_value(&mut self, time: f64, mut value: f64) -> Result<f64, Error> {
+ if time < 0.0 {
+ bail!("got negative time");
+ }
if time <= self.last_update {
bail!("time in past ({} < {})", time, self.last_update);
}
@@ -286,30 +289,36 @@ impl RRD {
}
- /// Load data from a file
- pub fn load(path: &Path) -> Result<Self, std::io::Error> {
- let raw = std::fs::read(path)?;
+ fn from_raw(raw: &[u8]) -> Result<Self, Error> {
if raw.len() < 8 {
- let msg = format!("not an rrd file - file is too small ({})", raw.len());
- return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ bail!("not an rrd file - file is too small ({})", raw.len());
}
- if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
+ let rrd = if raw[0..8] == rrd_v1::PROXMOX_RRD_MAGIC_1_0 {
let v1 = rrd_v1::RRDv1::from_raw(&raw)?;
v1.to_rrd_v2()
- .map_err(|err| {
- let msg = format!("unable to convert from old V1 format - {}", err);
- std::io::Error::new(std::io::ErrorKind::Other, msg)
- })
+ .map_err(|err| format_err!("unable to convert from old V1 format - {}", err))?
} else if raw[0..8] == PROXMOX_RRD_MAGIC_2_0 {
serde_cbor::from_slice(&raw[8..])
- .map_err(|err| {
- let msg = format!("unable to decode RRD file - {}", err);
- std::io::Error::new(std::io::ErrorKind::Other, msg)
- })
- } else {
- let msg = format!("not an rrd file - unknown magic number");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, msg));
+ .map_err(|err| format_err!("unable to decode RRD file - {}", err))?
+ } else {
+ bail!("not an rrd file - unknown magic number");
+ };
+
+ if rrd.source.last_update < 0.0 {
+ bail!("rrd file has negative last_update time");
+ }
+
+ Ok(rrd)
+ }
+
+ /// Load data from a file
+ pub fn load(path: &Path) -> Result<Self, std::io::Error> {
+ let raw = std::fs::read(path)?;
+
+ match Self::from_raw(&raw) {
+ Ok(rrd) => Ok(rrd),
+ Err(err) => Err(std::io::Error::new(std::io::ErrorKind::Other, err.to_string())),
}
}
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (12 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 25 +++++++++++++++----------
proxmox-rrd/src/rrd_v1.rs | 2 +-
2 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 7901fe39..72769bfd 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -63,7 +63,7 @@ pub struct DataSource {
pub last_update: f64,
/// Stores the last value, used to compute differential value for
/// derive/counters
- pub counter_value: f64,
+ pub last_value: f64,
}
impl DataSource {
@@ -72,7 +72,7 @@ impl DataSource {
Self {
dst,
last_update: 0.0,
- counter_value: f64::NAN,
+ last_value: f64::NAN,
}
}
@@ -94,23 +94,24 @@ impl DataSource {
if is_counter || self.dst == DST::Derive {
let time_diff = time - self.last_update;
- let diff = if self.counter_value.is_nan() {
+ let diff = if self.last_value.is_nan() {
0.0
} else if is_counter && value < 0.0 {
bail!("got negative value for counter");
- } else if is_counter && value < self.counter_value {
+ } else if is_counter && value < self.last_value {
// Note: We do not try automatic overflow corrections, but
- // we update counter_value anyways, so that we can compute the diff
+ // we update last_value anyways, so that we can compute the diff
// next time.
- self.counter_value = value;
+ self.last_value = value;
bail!("conter overflow/reset detected");
} else {
- value - self.counter_value
+ value - self.last_value
};
- self.counter_value = value;
value = diff/time_diff;
}
+ self.last_value = value;
+
Ok(value)
}
@@ -138,11 +139,15 @@ impl RRA {
}
}
- fn slot_end_time(&self, time: u64) -> u64 {
+ pub fn slot_end_time(&self, time: u64) -> u64 {
self.resolution*(time/self.resolution + 1)
}
- fn slot(&self, time: u64) -> usize {
+ pub fn slot_start_time(&self, time: u64) -> u64 {
+ self.resolution*(time/self.resolution)
+ }
+
+ pub fn slot(&self, time: u64) -> usize {
((time/self.resolution) as usize) % self.data.len()
}
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
index 511b510b..7e4b97c2 100644
--- a/proxmox-rrd/src/rrd_v1.rs
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -285,7 +285,7 @@ impl RRDv1 {
let source = DataSource {
dst,
- counter_value: f64::NAN,
+ last_value: f64::NAN,
last_update: self.hour_avg.last_update, // IMPORTANT!
};
Ok(RRD {
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (13 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht
15 siblings, 0 replies; 17+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/bin/rrd.rs | 229 ++++++++++++++++++++++++++++++++++---
proxmox-rrd/src/rrd.rs | 4 +-
2 files changed, 215 insertions(+), 18 deletions(-)
diff --git a/proxmox-rrd/src/bin/rrd.rs b/proxmox-rrd/src/bin/rrd.rs
index fdb61ffd..bf2817c4 100644
--- a/proxmox-rrd/src/bin/rrd.rs
+++ b/proxmox-rrd/src/bin/rrd.rs
@@ -4,16 +4,22 @@ use std::path::PathBuf;
use anyhow::{bail, Error};
use serde::{Serialize, Deserialize};
+use serde_json::json;
use proxmox_router::RpcEnvironment;
use proxmox_router::cli::{run_cli_command, CliCommand, CliCommandMap, CliEnvironment};
use proxmox_schema::{api, parse_property_string};
-use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema};
+use proxmox_schema::{ApiStringFormat, ApiType, IntegerSchema, Schema, StringSchema};
use proxmox::tools::fs::CreateOptions;
use proxmox_rrd::rrd::{CF, DST, RRA, RRD};
+pub const RRA_INDEX_SCHEMA: Schema = IntegerSchema::new(
+ "Index of the RRA.")
+ .minimum(0)
+ .schema();
+
pub const RRA_CONFIG_STRING_SCHEMA: Schema = StringSchema::new(
"RRA configuration")
.format(&ApiStringFormat::PropertyString(&RRAConfig::API_SCHEMA))
@@ -42,11 +48,36 @@ pub struct RRAConfig {
},
},
)]
-/// Dump the RRDB database in JSON format
-pub fn dump_rrdb(path: String) -> Result<(), Error> {
+/// Dump the RRD file in JSON format
+pub fn dump_rrd(path: String) -> Result<(), Error> {
let rrd = RRD::load(&PathBuf::from(path))?;
serde_json::to_writer_pretty(std::io::stdout(), &rrd)?;
+ println!("");
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ },
+ },
+)]
+/// RRD file information
+pub fn rrd_info(path: String) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+
+ println!("DST: {:?}", rrd.source.dst);
+
+ for (i, rra) in rrd.rra_list.iter().enumerate() {
+ // use RRAConfig property string format
+ println!("RRA[{}]: {:?},r={},n={}", i, rra.cf, rra.resolution, rra.data.len());
+ }
+
Ok(())
}
@@ -66,8 +97,8 @@ pub fn dump_rrdb(path: String) -> Result<(), Error> {
},
},
)]
-/// Update the RRDB database
-pub fn update_rrdb(
+/// Update the RRD database
+pub fn update_rrd(
path: String,
time: Option<u64>,
value: f64,
@@ -109,8 +140,8 @@ pub fn update_rrdb(
},
},
)]
-/// Fetch data from the RRDB database
-pub fn fetch_rrdb(
+/// Fetch data from the RRD file
+pub fn fetch_rrd(
path: String,
cf: CF,
resolution: u64,
@@ -127,6 +158,80 @@ pub fn fetch_rrdb(
Ok(())
}
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ "rra-index": {
+ schema: RRA_INDEX_SCHEMA,
+ },
+ },
+ },
+)]
+/// Return the Unix timestamp of the first time slot inside the
+/// specified RRA (slot start time)
+pub fn first_update_time(
+ path: String,
+ rra_index: usize,
+) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+
+ if rra_index >= rrd.rra_list.len() {
+ bail!("rra-index is out of range");
+ }
+ let rra = &rrd.rra_list[rra_index];
+ let duration = (rra.data.len() as u64)*rra.resolution;
+ let first = rra.slot_start_time((rrd.source.last_update as u64).saturating_sub(duration));
+
+ println!("{}", first);
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ },
+ },
+)]
+/// Return the Unix timestamp of the last update
+pub fn last_update_time(path: String) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+
+ println!("{}", rrd.source.last_update);
+ Ok(())
+}
+
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ },
+ },
+)]
+/// Return the time and value from the last update
+pub fn last_update(path: String) -> Result<(), Error> {
+
+ let rrd = RRD::load(&PathBuf::from(path))?;
+
+ let result = json!({
+ "time": rrd.source.last_update,
+ "value": rrd.source.last_value,
+ });
+
+ println!("{}", serde_json::to_string_pretty(&result)?);
+
+ Ok(())
+}
+
#[api(
input: {
properties: {
@@ -146,8 +251,8 @@ pub fn fetch_rrdb(
},
},
)]
-/// Create a new RRDB database file
-pub fn create_rrdb(
+/// Create a new RRD file
+pub fn create_rrd(
dst: DST,
path: String,
rra: Vec<String>,
@@ -172,6 +277,64 @@ pub fn create_rrdb(
Ok(())
}
+#[api(
+ input: {
+ properties: {
+ path: {
+ description: "The filename."
+ },
+ "rra-index": {
+ schema: RRA_INDEX_SCHEMA,
+ },
+ slots: {
+ description: "The number of slots you want to add or remove.",
+ type: i64,
+ },
+ },
+ },
+)]
+/// Resize. Change the number of data slots for the specified RRA.
+pub fn resize_rrd(
+ path: String,
+ rra_index: usize,
+ slots: i64,
+) -> Result<(), Error> {
+
+ let path = PathBuf::from(&path);
+
+ let mut rrd = RRD::load(&path)?;
+
+ if rra_index >= rrd.rra_list.len() {
+ bail!("rra-index is out of range");
+ }
+
+ let rra = &rrd.rra_list[rra_index];
+
+ let new_slots = (rra.data.len() as i64) + slots;
+
+ if new_slots < 1 {
+ bail!("numer of new slots is too small ('{}' < 1)", new_slots);
+ }
+
+ if new_slots > 1024*1024 {
+ bail!("numer of new slots is too big ('{}' > 1M)", new_slots);
+ }
+
+ let rra_end = rra.slot_end_time(rrd.source.last_update as u64);
+ let rra_start = rra_end - rra.resolution*(rra.data.len() as u64);
+ let (start, reso, data) = rra.extract_data(rra_start, rra_end, rrd.source.last_update);
+
+ let mut new_rra = RRA::new(rra.cf, rra.resolution, new_slots as usize);
+ new_rra.last_count = rra.last_count;
+
+ new_rra.insert_data(start, reso, data)?;
+
+ rrd.rra_list[rra_index] = new_rra;
+
+ rrd.save(&path, CreateOptions::new())?;
+
+ Ok(())
+}
fn main() -> Result<(), Error> {
@@ -185,23 +348,57 @@ fn main() -> Result<(), Error> {
let cmd_def = CliCommandMap::new()
.insert(
"create",
- CliCommand::new(&API_METHOD_CREATE_RRDB)
+ CliCommand::new(&API_METHOD_CREATE_RRD)
.arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
)
.insert(
- "update",
- CliCommand::new(&API_METHOD_UPDATE_RRDB)
+ "dump",
+ CliCommand::new(&API_METHOD_DUMP_RRD)
.arg_param(&["path"])
- )
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
.insert(
"fetch",
- CliCommand::new(&API_METHOD_FETCH_RRDB)
+ CliCommand::new(&API_METHOD_FETCH_RRD)
.arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
+ .insert(
+ "first",
+ CliCommand::new(&API_METHOD_FIRST_UPDATE_TIME)
+ .arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
)
.insert(
- "dump",
- CliCommand::new(&API_METHOD_DUMP_RRDB)
+ "info",
+ CliCommand::new(&API_METHOD_RRD_INFO)
+ .arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
+ .insert(
+ "last",
+ CliCommand::new(&API_METHOD_LAST_UPDATE_TIME)
+ .arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
+ .insert(
+ "lastupdate",
+ CliCommand::new(&API_METHOD_LAST_UPDATE)
+ .arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
+ .insert(
+ "resize",
+ CliCommand::new(&API_METHOD_RESIZE_RRD)
+ .arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
+ )
+ .insert(
+ "update",
+ CliCommand::new(&API_METHOD_UPDATE_RRD)
.arg_param(&["path"])
+ //.completion_cb("path", pbs_tools::fs::complete_file_name)
)
;
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 72769bfd..c6996e42 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -153,7 +153,7 @@ impl RRA {
// directly overwrite data slots
// the caller need to set last_update value on the DataSource manually.
- pub(crate) fn insert_data(
+ pub fn insert_data(
&mut self,
start: u64,
resolution: u64,
@@ -238,7 +238,7 @@ impl RRA {
}
}
- fn extract_data(
+ pub fn extract_data(
&self,
start: u64,
end: u64,
--
2.30.2
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
` (14 preceding siblings ...)
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
@ 2021-10-13 11:58 ` Thomas Lamprecht
15 siblings, 0 replies; 17+ messages in thread
From: Thomas Lamprecht @ 2021-10-13 11:58 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Dietmar Maurer
On 13.10.21 10:24, Dietmar Maurer wrote:
> - use a journal. This way we can reduce the overall number of bytes
> written by increasing the flush/commit interval (30 minutes).
>
> - the new CBOR base format is more flexible, and we store much more
> data points now.
>
> We previously wrote about 7MB/h. With the new format and journal, we now write about 3MB/h.
>
> Dietmar Maurer (15):
> proxmox-rrd: use a journal to reduce amount of bytes written
> RRD_CACHE: use a OnceCell instead of lazy_static
> proxmox-backup-proxy: use tokio::task::spawn_blocking instead of
> block_in_place
> proxmox-rrd: implement new CBOR based format
> proxmox-rrd: remove dependency to proxmox-rrd-api-types
> proxmox-rrd: extract_data: include values from current slot
> remove proxmox-rrd-api-types crate,
> s/RRDTimeFrameResolution/RRDTimeFrame/
> proxmox-rrd: support CF::Last
> proxmox-rrd: split out load_rrd (cleanup)
> proxmox-rrd: add binary to create/manage rrd files
> proxmox-rrd: avoid % inside loop
> proxmox-rrd: new helper methods - slot() and slot_end_time()
> proxmox-rrd: protect against negative update time
> proxmox-rrd: rename last_counter to last_value
> proxmox-rrd: add more commands to the rrd cli tool
applied whole series, with the diff you replied on patch 14/15 squashed in and the rrd
debug tool moved to examples as quick "fix" for avoiding proxmox-router as hard dependency
on the crate for now, thanks!
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2021-10-13 11:58 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 01/15] proxmox-rrd: use a journal to reduce amount of bytes written Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 02/15] RRD_CACHE: use a OnceCell instead of lazy_static Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 03/15] proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 04/15] proxmox-rrd: implement new CBOR based format Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 05/15] proxmox-rrd: remove dependency to proxmox-rrd-api-types Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 06/15] proxmox-rrd: extract_data: include values from current slot Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 07/15] remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 08/15] proxmox-rrd: support CF::Last Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 09/15] proxmox-rrd: split out load_rrd (cleanup) Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 10/15] proxmox-rrd: add binary to create/manage rrd files Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 11/15] proxmox-rrd: avoid % inside loop Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 12/15] proxmox-rrd: new helper methods - slot() and slot_end_time() Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 13/15] proxmox-rrd: protect against negative update time Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 15/15] proxmox-rrd: add more commands to the rrd cli tool Dietmar Maurer
2021-10-13 11:58 ` [pbs-devel] applied-series: [PATCH proxmox-backup 00/15] RRD database improvements Thomas Lamprecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.