* Re: [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value
@ 2021-10-13 9:51 Dietmar Maurer
0 siblings, 0 replies; 2+ messages in thread
From: Dietmar Maurer @ 2021-10-13 9:51 UTC (permalink / raw)
To: pbs-devel
This introduces a bug: here is the fix
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index c6996e42..eb6154e7 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -107,11 +107,12 @@ impl DataSource {
} else {
value - self.last_value
};
+ self.last_value = value;
value = diff/time_diff;
+ } else {
+ self.last_value = value;
}
- self.last_value = value;
-
Ok(value)
}
> On 10/13/2021 10:24 AM Dietmar Maurer <dietmar@proxmox.com> wrote:
>
>
> ---
> proxmox-rrd/src/rrd.rs | 25 +++++++++++++++----------
> proxmox-rrd/src/rrd_v1.rs | 2 +-
> 2 files changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
> index 7901fe39..72769bfd 100644
> --- a/proxmox-rrd/src/rrd.rs
> +++ b/proxmox-rrd/src/rrd.rs
> @@ -63,7 +63,7 @@ pub struct DataSource {
> pub last_update: f64,
> /// Stores the last value, used to compute differential value for
> /// derive/counters
> - pub counter_value: f64,
> + pub last_value: f64,
> }
>
> impl DataSource {
> @@ -72,7 +72,7 @@ impl DataSource {
> Self {
> dst,
> last_update: 0.0,
> - counter_value: f64::NAN,
> + last_value: f64::NAN,
> }
> }
>
> @@ -94,23 +94,24 @@ impl DataSource {
> if is_counter || self.dst == DST::Derive {
> let time_diff = time - self.last_update;
>
> - let diff = if self.counter_value.is_nan() {
> + let diff = if self.last_value.is_nan() {
> 0.0
> } else if is_counter && value < 0.0 {
> bail!("got negative value for counter");
> - } else if is_counter && value < self.counter_value {
> + } else if is_counter && value < self.last_value {
> // Note: We do not try automatic overflow corrections, but
> - // we update counter_value anyways, so that we can compute the diff
> + // we update last_value anyways, so that we can compute the diff
> // next time.
> - self.counter_value = value;
> + self.last_value = value;
> bail!("conter overflow/reset detected");
> } else {
> - value - self.counter_value
> + value - self.last_value
> };
> - self.counter_value = value;
> value = diff/time_diff;
> }
>
> + self.last_value = value;
> +
> Ok(value)
> }
>
> @@ -138,11 +139,15 @@ impl RRA {
> }
> }
>
> - fn slot_end_time(&self, time: u64) -> u64 {
> + pub fn slot_end_time(&self, time: u64) -> u64 {
> self.resolution*(time/self.resolution + 1)
> }
>
> - fn slot(&self, time: u64) -> usize {
> + pub fn slot_start_time(&self, time: u64) -> u64 {
> + self.resolution*(time/self.resolution)
> + }
> +
> + pub fn slot(&self, time: u64) -> usize {
> ((time/self.resolution) as usize) % self.data.len()
> }
>
> diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
> index 511b510b..7e4b97c2 100644
> --- a/proxmox-rrd/src/rrd_v1.rs
> +++ b/proxmox-rrd/src/rrd_v1.rs
> @@ -285,7 +285,7 @@ impl RRDv1 {
>
> let source = DataSource {
> dst,
> - counter_value: f64::NAN,
> + last_value: f64::NAN,
> last_update: self.hour_avg.last_update, // IMPORTANT!
> };
> Ok(RRD {
> --
> 2.30.2
^ permalink raw reply [flat|nested] 2+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements
@ 2021-10-13 8:24 Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
0 siblings, 1 reply; 2+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
- use a journal. This way we can reduce the overall number of bytes
written by increasing the flush/commit interval (30 minutes).
- the new CBOR base format is more flexible, and we store much more
data points now.
We previously wrote about 7MB/h. With the new format and journal, we now write about 3MB/h.
Dietmar Maurer (15):
proxmox-rrd: use a journal to reduce amount of bytes written
RRD_CACHE: use a OnceCell instead of lazy_static
proxmox-backup-proxy: use tokio::task::spawn_blocking instead of
block_in_place
proxmox-rrd: implement new CBOR based format
proxmox-rrd: remove dependency to proxmox-rrd-api-types
proxmox-rrd: extract_data: include values from current slot
remove proxmox-rrd-api-types crate,
s/RRDTimeFrameResolution/RRDTimeFrame/
proxmox-rrd: support CF::Last
proxmox-rrd: split out load_rrd (cleanup)
proxmox-rrd: add binary to create/manage rrd files
proxmox-rrd: avoid % inside loop
proxmox-rrd: new helper methods - slot() and slot_end_time()
proxmox-rrd: protect against negative update time
proxmox-rrd: rename last_counter to last_value
proxmox-rrd: add more commands to the rrd cli tool
Cargo.toml | 2 -
Makefile | 1 -
pbs-api-types/Cargo.toml | 1 -
pbs-api-types/src/lib.rs | 30 +-
proxmox-rrd-api-types/Cargo.toml | 11 -
proxmox-rrd-api-types/src/lib.rs | 32 --
proxmox-rrd/Cargo.toml | 8 +-
proxmox-rrd/src/bin/rrd.rs | 412 +++++++++++++++++++++++
proxmox-rrd/src/cache.rs | 283 +++++++++++++---
proxmox-rrd/src/lib.rs | 17 +-
proxmox-rrd/src/rrd.rs | 539 ++++++++++++++++---------------
proxmox-rrd/src/rrd_v1.rs | 296 +++++++++++++++++
src/api2/admin/datastore.rs | 6 +-
src/api2/node/rrd.rs | 52 +--
src/api2/status.rs | 14 +-
src/bin/proxmox-backup-api.rs | 4 -
src/bin/proxmox-backup-proxy.rs | 166 +++++-----
src/lib.rs | 86 +++--
18 files changed, 1465 insertions(+), 495 deletions(-)
delete mode 100644 proxmox-rrd-api-types/Cargo.toml
delete mode 100644 proxmox-rrd-api-types/src/lib.rs
create mode 100644 proxmox-rrd/src/bin/rrd.rs
create mode 100644 proxmox-rrd/src/rrd_v1.rs
--
2.30.2
^ permalink raw reply [flat|nested] 2+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
@ 2021-10-13 8:24 ` Dietmar Maurer
0 siblings, 0 replies; 2+ messages in thread
From: Dietmar Maurer @ 2021-10-13 8:24 UTC (permalink / raw)
To: pbs-devel
---
proxmox-rrd/src/rrd.rs | 25 +++++++++++++++----------
proxmox-rrd/src/rrd_v1.rs | 2 +-
2 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/proxmox-rrd/src/rrd.rs b/proxmox-rrd/src/rrd.rs
index 7901fe39..72769bfd 100644
--- a/proxmox-rrd/src/rrd.rs
+++ b/proxmox-rrd/src/rrd.rs
@@ -63,7 +63,7 @@ pub struct DataSource {
pub last_update: f64,
/// Stores the last value, used to compute differential value for
/// derive/counters
- pub counter_value: f64,
+ pub last_value: f64,
}
impl DataSource {
@@ -72,7 +72,7 @@ impl DataSource {
Self {
dst,
last_update: 0.0,
- counter_value: f64::NAN,
+ last_value: f64::NAN,
}
}
@@ -94,23 +94,24 @@ impl DataSource {
if is_counter || self.dst == DST::Derive {
let time_diff = time - self.last_update;
- let diff = if self.counter_value.is_nan() {
+ let diff = if self.last_value.is_nan() {
0.0
} else if is_counter && value < 0.0 {
bail!("got negative value for counter");
- } else if is_counter && value < self.counter_value {
+ } else if is_counter && value < self.last_value {
// Note: We do not try automatic overflow corrections, but
- // we update counter_value anyways, so that we can compute the diff
+ // we update last_value anyways, so that we can compute the diff
// next time.
- self.counter_value = value;
+ self.last_value = value;
bail!("conter overflow/reset detected");
} else {
- value - self.counter_value
+ value - self.last_value
};
- self.counter_value = value;
value = diff/time_diff;
}
+ self.last_value = value;
+
Ok(value)
}
@@ -138,11 +139,15 @@ impl RRA {
}
}
- fn slot_end_time(&self, time: u64) -> u64 {
+ pub fn slot_end_time(&self, time: u64) -> u64 {
self.resolution*(time/self.resolution + 1)
}
- fn slot(&self, time: u64) -> usize {
+ pub fn slot_start_time(&self, time: u64) -> u64 {
+ self.resolution*(time/self.resolution)
+ }
+
+ pub fn slot(&self, time: u64) -> usize {
((time/self.resolution) as usize) % self.data.len()
}
diff --git a/proxmox-rrd/src/rrd_v1.rs b/proxmox-rrd/src/rrd_v1.rs
index 511b510b..7e4b97c2 100644
--- a/proxmox-rrd/src/rrd_v1.rs
+++ b/proxmox-rrd/src/rrd_v1.rs
@@ -285,7 +285,7 @@ impl RRDv1 {
let source = DataSource {
dst,
- counter_value: f64::NAN,
+ last_value: f64::NAN,
last_update: self.hour_avg.last_update, // IMPORTANT!
};
Ok(RRD {
--
2.30.2
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-10-13 9:52 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-13 9:51 [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
-- strict thread matches above, loose matches on Subject: below --
2021-10-13 8:24 [pbs-devel] [PATCH proxmox-backup 00/15] RRD database improvements Dietmar Maurer
2021-10-13 8:24 ` [pbs-devel] [PATCH proxmox-backup 14/15] proxmox-rrd: rename last_counter to last_value Dietmar Maurer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox