* [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type
@ 2021-11-17 13:37 Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 1/4] pbs-api-types: " Dietmar Maurer
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-11-17 13:37 UTC (permalink / raw)
To: pbs-devel
Changes since v1:
- do not expose auto_unit functions
- store size as u64 (suggested by thomas)
- do not store unit (suggested by thomas)
The disadvantage is that the config value does not exactly
match the value you set, for example:
proxmox-backup-manager traffic-control update rule0 --rate-out 1.2mib
config contains "1.198 MiB"
proxmox-backup-manager traffic-control update rule0 --rate-out 1200kib
config contains "1.17 MiB"
Dietmar Maurer (4):
pbs-api-types: more flexible HumanByte type
use HumanByte for traffic-control config
HumanByte: use u64 instead of f64 to store size
HumanByte: do not store unit (always compute canonical form)
pbs-api-types/src/human_byte.rs | 326 +++++++++++++++++++++++++++
pbs-api-types/src/lib.rs | 3 +
pbs-api-types/src/traffic_control.rs | 18 +-
pbs-client/src/backup_writer.rs | 19 +-
pbs-datastore/src/datastore.rs | 13 +-
pbs-tools/Cargo.toml | 1 +
pbs-tools/src/format.rs | 55 +----
src/bin/proxmox-tape.rs | 7 +-
src/cached_traffic_control.rs | 18 +-
src/server/email_notifications.rs | 5 +-
10 files changed, 376 insertions(+), 89 deletions(-)
create mode 100644 pbs-api-types/src/human_byte.rs
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [RFC proxmox-backup v2 1/4] pbs-api-types: more flexible HumanByte type
2021-11-17 13:37 [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type Dietmar Maurer
@ 2021-11-17 13:37 ` Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 2/4] use HumanByte for traffic-control config Dietmar Maurer
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-11-17 13:37 UTC (permalink / raw)
To: pbs-devel
This implements a parser for human readabkle byte sizes.
Examples: "1KB", "1.5 Mib", "3T"
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
---
pbs-api-types/src/human_byte.rs | 343 ++++++++++++++++++++++++++++++
pbs-api-types/src/lib.rs | 3 +
pbs-client/src/backup_writer.rs | 19 +-
pbs-datastore/src/datastore.rs | 13 +-
pbs-tools/Cargo.toml | 1 +
pbs-tools/src/format.rs | 55 +----
src/bin/proxmox-tape.rs | 7 +-
src/server/email_notifications.rs | 5 +-
8 files changed, 372 insertions(+), 74 deletions(-)
create mode 100644 pbs-api-types/src/human_byte.rs
diff --git a/pbs-api-types/src/human_byte.rs b/pbs-api-types/src/human_byte.rs
new file mode 100644
index 00000000..df90d446
--- /dev/null
+++ b/pbs-api-types/src/human_byte.rs
@@ -0,0 +1,343 @@
+use std::str::FromStr;
+
+use anyhow::{bail, Error};
+
+use proxmox_schema::{ApiStringFormat, ApiType, StringSchema, Schema, UpdaterType};
+
+#[derive(Debug,Copy,Clone, PartialEq)]
+pub enum SizeUnit {
+ None,
+ Byte,
+ Kilo,
+ KByte,
+ Mega,
+ MByte,
+ Giga,
+ GByte,
+ Tera,
+ TByte,
+ Peta,
+ PByte,
+ Kibi,
+ Mebi,
+ Gibi,
+ Tebi,
+ Pebi,
+}
+
+impl SizeUnit {
+ pub fn factor(&self) -> f64 {
+ match self {
+ SizeUnit::None => 1.0,
+ SizeUnit::Byte => 1.0,
+
+ SizeUnit::Kilo | SizeUnit::KByte => 1_000.0,
+ SizeUnit::Mega | SizeUnit::MByte => 1_000_000.0,
+ SizeUnit::Giga | SizeUnit::GByte => 1_000_000_000.0,
+ SizeUnit::Tera | SizeUnit::TByte => 1_000_000_000_000.0,
+ SizeUnit::Peta | SizeUnit::PByte => 1_000_000_000_000_000.0,
+
+ SizeUnit::Kibi => 1024.0,
+ SizeUnit::Mebi => 1024.0*1024.0,
+ SizeUnit::Gibi => 1024.0*1024.0*1024.0,
+ SizeUnit::Tebi => 1024.0*1024.0*1024.0*1024.0,
+ SizeUnit::Pebi => 1024.0*1024.0*1024.0*1024.0*1024.0,
+ }
+ }
+
+ pub fn unit_str(&self) -> &'static str {
+ match self {
+ SizeUnit::None => "",
+ SizeUnit::Byte => "B",
+
+ SizeUnit::Kilo => "K",
+ SizeUnit::Mega => "M",
+ SizeUnit::Giga => "G",
+ SizeUnit::Tera => "T",
+ SizeUnit::Peta => "P",
+
+ SizeUnit::KByte => "KB",
+ SizeUnit::MByte => "MB",
+ SizeUnit::GByte => "GB",
+ SizeUnit::TByte => "TB",
+ SizeUnit::PByte => "PB",
+
+ SizeUnit::Kibi => "KiB",
+ SizeUnit::Mebi => "MiB",
+ SizeUnit::Gibi => "GiB",
+ SizeUnit::Tebi => "TiB",
+ SizeUnit::Pebi => "PiB",
+ }
+ }
+}
+
+fn strip_unit(v: &str) -> (&str, SizeUnit) {
+
+ if let Some(v) = v.strip_suffix(&['k', 'K'][..]) {
+ return (v, SizeUnit::Kilo);
+ }
+ if let Some(v) = v.strip_suffix(&['m', 'M'][..]) {
+ return (v, SizeUnit::Mega);
+ }
+ if let Some(v) = v.strip_suffix(&['g', 'G'][..]) {
+ return (v, SizeUnit::Giga);
+ }
+ if let Some(v) = v.strip_suffix(&['t', 'T'][..]) {
+ return (v, SizeUnit::Tera);
+ }
+ if let Some(v) = v.strip_suffix(&['p', 'P'][..]) {
+ return (v, SizeUnit::Peta);
+ }
+
+ if let Some(mut v) = v.strip_suffix(&['b', 'B'][..]) {
+
+ let binary = if let Some(n) = v.strip_suffix('i') {
+ v = n;
+ true
+ } else {
+ false
+ };
+
+ if let Some(v) = v.strip_suffix(&['k', 'K'][..]) {
+ return (v, if binary { SizeUnit::Kibi } else { SizeUnit::KByte });
+ }
+ if let Some(v) = v.strip_suffix(&['m', 'M'][..]) {
+ return (v, if binary { SizeUnit::Mebi } else { SizeUnit::MByte });
+ }
+ if let Some(v) = v.strip_suffix(&['g', 'G'][..]) {
+ return (v, if binary { SizeUnit::Gibi } else { SizeUnit::GByte });
+ }
+ if let Some(v) = v.strip_suffix(&['t', 'T'][..]) {
+ return (v, if binary { SizeUnit::Tebi } else { SizeUnit::TByte });
+ }
+ if let Some(v) = v.strip_suffix(&['p', 'P'][..]) {
+ return (v, if binary { SizeUnit::Pebi } else { SizeUnit::PByte });
+ }
+ return (v, SizeUnit::Byte);
+ }
+
+ return (v, SizeUnit::None);
+}
+
+#[derive(Debug, Copy, Clone, UpdaterType)]
+/// Byte size with unit
+pub struct HumanByte {
+ size: f64,
+ unit: SizeUnit,
+}
+
+fn verify_human_byte(s: &str) -> Result<(), Error> {
+ match s.parse::<HumanByte>() {
+ Ok(_) => Ok(()),
+ Err(err) => bail!("byte size parse error: {}", err),
+ }
+}
+
+impl ApiType for HumanByte {
+ const API_SCHEMA: Schema = StringSchema::new("Byte size with optional unit (KB, MB, GB, KiB, MiB, Gib, ...).")
+ .format(&ApiStringFormat::VerifyFn(verify_human_byte))
+ .min_length(1)
+ .max_length(64)
+ .schema();
+}
+
+impl HumanByte {
+
+ pub fn new_decimal(size: u64) -> Self {
+ let this = HumanByte { size: size as f64, unit: SizeUnit::None };
+ this.auto_unit_decimal()
+ }
+
+ pub fn new_binary(size: u64) -> Self {
+ let this = HumanByte { size: size as f64, unit: SizeUnit::None };
+ this.auto_unit_binary()
+ }
+
+ pub fn as_u64(&self) -> u64 {
+ self.size as u64
+ }
+
+ pub fn auto_unit_binary(mut self) -> Self {
+ let size = self.size as u64;
+ let zeroes = size.leading_zeros();
+ let bits = 63 - zeroes;
+ self.unit = if bits >= 50 {
+ SizeUnit::Pebi
+ } else if bits >= 40 {
+ SizeUnit::Tebi
+ } else if bits >= 30 {
+ SizeUnit::Gibi
+ } else if bits >= 20 {
+ SizeUnit::Mebi
+ } else if bits >= 10 {
+ SizeUnit::Kibi
+ } else {
+ SizeUnit::None
+ };
+ self
+ }
+
+ pub fn auto_unit_decimal(mut self) -> Self {
+ self.unit = if self.size >= 1_000_000_000_000_000.0 {
+ SizeUnit::PByte
+ } else if self.size >= 1_000_000_000_000.0 {
+ SizeUnit::TByte
+ } else if self.size >= 1_000_000_000.0 {
+ SizeUnit::GByte
+ } else if self.size >= 1_000_000.0 {
+ SizeUnit::MByte
+ } else if self.size >= 1_000.0 {
+ SizeUnit::KByte
+ } else {
+ SizeUnit::None
+ };
+ self
+ }
+}
+
+impl std::fmt::Display for HumanByte {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ let size = self.size;
+
+ let unit = self.unit;
+
+ let size = size/unit.factor();
+ let unit_str = unit.unit_str();
+
+ if unit == SizeUnit::Byte || unit == SizeUnit::None {
+ if unit_str.is_empty() {
+ write!(f, "{}", size as u64)
+ } else {
+ write!(f, "{} {}", size as u64, unit.unit_str())
+ }
+ } else {
+ let size = ((size*1000.0).floor())/1000.0;
+
+ if unit_str.is_empty() {
+ write!(f, "{}", size)
+ } else {
+ write!(f, "{} {}", size, unit.unit_str())
+ }
+ }
+ }
+}
+
+impl FromStr for HumanByte {
+ type Err = Error;
+
+ fn from_str(v: &str) -> Result<Self, Error> {
+ let (mut v, unit) = strip_unit(v);
+
+ if unit != SizeUnit::None {
+ v = v.trim_end();
+ }
+ let size: f64 = v.parse()?;
+
+ if size < 0.0 {
+ bail!("size may not be negative");
+ }
+
+ let size = size*unit.factor();
+
+ Ok(Self { size: size, unit })
+ }
+
+}
+
+proxmox::forward_deserialize_to_from_str!(HumanByte);
+proxmox::forward_serialize_to_display!(HumanByte);
+
+#[test]
+fn test_human_byte_parser() -> Result<(), Error> {
+
+ assert!("-10".parse::<HumanByte>().is_err()); // negative size
+
+ fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
+ let h: HumanByte = v.parse()?;
+
+ if h.size != size {
+ bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
+ }
+ if h.unit != unit {
+ bail!("got unexpected unit for '{}' ({:?} != {:?})", v, h.unit, unit);
+ }
+
+ let new = h.to_string();
+ if &new != as_str {
+ bail!("to_string failed for '{}' ({:?} != {:?})", v, new, as_str);
+
+ }
+
+ Ok(())
+ }
+
+ test("14.4", 14.4, SizeUnit::None, "14")?;
+
+ test("14", 14.0, SizeUnit::None, "14")?;
+ test("987654321", 987654321.0, SizeUnit::None, "987654321")?;
+
+ test("1300b", 1300.0, SizeUnit::Byte, "1300 B")?;
+ test("1300B", 1300.0, SizeUnit::Byte, "1300 B")?;
+
+ test("1.5KB", 1500.0, SizeUnit::KByte, "1.5 KB")?;
+ test("1.5kb", 1500.0, SizeUnit::KByte, "1.5 KB")?;
+ test("1.654321MB", 1_654_321.0, SizeUnit::MByte, "1.654 MB")?;
+
+ test("2.0GB", 2_000_000_000.0, SizeUnit::GByte, "2 GB")?;
+
+ test("1.4TB", 1_400_000_000_000.0, SizeUnit::TByte, "1.4 TB")?;
+ test("1.4tb", 1_400_000_000_000.0, SizeUnit::TByte, "1.4 TB")?;
+
+ test("2KiB", 2048.0, SizeUnit::Kibi, "2 KiB")?;
+ test("2kib", 2048.0, SizeUnit::Kibi, "2 KiB")?;
+
+ test("2.3456MiB", 2.3456*1024.0*1024.0, SizeUnit::Mebi, "2.345 MiB")?;
+
+ test("4gib", 4.0*1024.0*1024.0*1024.0, SizeUnit::Gibi, "4 GiB")?;
+
+ Ok(())
+}
+
+#[test]
+fn test_human_byte_auto_unit_decimal() {
+ fn convert(b: u64) -> String {
+ HumanByte::new_decimal(b).to_string()
+ }
+ assert_eq!(convert(987), "987");
+ assert_eq!(convert(1022), "1.022 KB");
+ assert_eq!(convert(9_000), "9 KB");
+ assert_eq!(convert(1_000), "1 KB");
+ assert_eq!(convert(1_000_000), "1 MB");
+ assert_eq!(convert(1_000_000_000), "1 GB");
+ assert_eq!(convert(1_000_000_000_000), "1 TB");
+ assert_eq!(convert(1_000_000_000_000_000), "1 PB");
+
+ assert_eq!(convert((1<<30) + 103 * (1<<20)), "1.181 GB");
+ assert_eq!(convert((1<<30) + 128 * (1<<20)), "1.207 GB");
+ assert_eq!(convert((2<<50) + 500 * (1<<40)), "2.801 PB");
+}
+
+#[test]
+fn test_human_byte_auto_unit_binary() {
+ fn convert(b: u64) -> String {
+ HumanByte::new_binary(b).to_string()
+ }
+ assert_eq!(convert(987), "987");
+ assert_eq!(convert(1022), "1022");
+ assert_eq!(convert(9_000), "8.789 KiB");
+ assert_eq!(convert(10_000_000), "9.536 MiB");
+ assert_eq!(convert(10_000_000_000), "9.313 GiB");
+ assert_eq!(convert(10_000_000_000_000), "9.094 TiB");
+
+ assert_eq!(convert(1<<10), "1 KiB");
+ assert_eq!(convert((1<<10)*10), "10 KiB");
+ assert_eq!(convert(1<<20), "1 MiB");
+ assert_eq!(convert(1<<30), "1 GiB");
+ assert_eq!(convert(1<<40), "1 TiB");
+ assert_eq!(convert(1<<50), "1 PiB");
+
+ assert_eq!(convert((1<<30) + 103 * (1<<20)), "1.1 GiB");
+ assert_eq!(convert((1<<30) + 128 * (1<<20)), "1.125 GiB");
+ assert_eq!(convert((1<<40) + 128 * (1<<30)), "1.125 TiB");
+ assert_eq!(convert((2<<50) + 512 * (1<<40)), "2.5 PiB");
+}
diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index 01c14cc4..daf8fd54 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -39,6 +39,9 @@ pub use acl::*;
mod datastore;
pub use datastore::*;
+mod human_byte;
+pub use human_byte::HumanByte;
+
mod jobs;
pub use jobs::*;
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index 531de2e8..98f78652 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -3,6 +3,7 @@ use std::future::Future;
use std::os::unix::fs::OpenOptionsExt;
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
+use std::convert::TryInto;
use anyhow::{bail, format_err, Error};
use futures::future::{self, AbortHandle, Either, FutureExt, TryFutureExt};
@@ -14,8 +15,8 @@ use tokio_stream::wrappers::ReceiverStream;
use proxmox::tools::digest_to_hex;
+use pbs_api_types::HumanByte;
use pbs_tools::crypt_config::CryptConfig;
-use pbs_tools::format::HumanByte;
use pbs_datastore::{CATALOG_NAME, PROXMOX_BACKUP_PROTOCOL_ID_V1};
use pbs_datastore::data_blob::{ChunkInfo, DataBlob, DataChunkBuilder};
use pbs_datastore::dynamic_index::DynamicIndexReader;
@@ -336,17 +337,19 @@ impl BackupWriter {
.await?;
let size_dirty = upload_stats.size - upload_stats.size_reused;
- let size: HumanByte = upload_stats.size.into();
+ let size = HumanByte::new_binary(upload_stats.size as u64);
let archive = if self.verbose {
archive_name
} else {
pbs_tools::format::strip_server_file_extension(archive_name)
};
if archive_name != CATALOG_NAME {
- let speed: HumanByte =
- ((size_dirty * 1_000_000) / (upload_stats.duration.as_micros() as usize)).into();
- let size_dirty: HumanByte = size_dirty.into();
- let size_compressed: HumanByte = upload_stats.size_compressed.into();
+ let speed = HumanByte::new_binary(
+ ((size_dirty as u128 * 1_000_000) / (upload_stats.duration.as_micros()))
+ .try_into()?
+ );
+ let size_dirty = HumanByte::new_binary(size_dirty as u64);
+ let size_compressed = HumanByte::new_binary(upload_stats.size_compressed as u64);
println!(
"{}: had to backup {} of {} (compressed {}) in {:.2}s",
archive,
@@ -362,7 +365,7 @@ impl BackupWriter {
if upload_stats.size_reused > 0 && upload_stats.size > 1024 * 1024 {
let reused_percent = upload_stats.size_reused as f64 * 100. / upload_stats.size as f64;
- let reused: HumanByte = upload_stats.size_reused.into();
+ let reused = HumanByte::new_binary(upload_stats.size_reused as u64);
println!(
"{}: backup was done incrementally, reused {} ({:.1}%)",
archive, reused, reused_percent
@@ -376,7 +379,7 @@ impl BackupWriter {
println!(
"{}: Average chunk size was {}.",
archive,
- HumanByte::from(upload_stats.size / upload_stats.chunk_count)
+ HumanByte::new_binary((upload_stats.size / upload_stats.chunk_count) as u64),
);
println!(
"{}: Average time per request: {} microseconds.",
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 5049cb3d..f76554f2 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -11,8 +11,7 @@ use lazy_static::lazy_static;
use proxmox::tools::fs::{replace_file, file_read_optional_string, CreateOptions};
-use pbs_api_types::{UPID, DataStoreConfig, Authid, GarbageCollectionStatus};
-use pbs_tools::format::HumanByte;
+use pbs_api_types::{UPID, DataStoreConfig, Authid, GarbageCollectionStatus, HumanByte};
use pbs_tools::fs::{lock_dir_noblock, DirLockGuard};
use pbs_tools::process_locker::ProcessLockSharedGuard;
use pbs_tools::{task_log, task_warn, task::WorkerTaskContext};
@@ -654,14 +653,14 @@ impl DataStore {
task_log!(
worker,
"Removed garbage: {}",
- HumanByte::from(gc_status.removed_bytes),
+ HumanByte::new_binary(gc_status.removed_bytes),
);
task_log!(worker, "Removed chunks: {}", gc_status.removed_chunks);
if gc_status.pending_bytes > 0 {
task_log!(
worker,
"Pending removals: {} (in {} chunks)",
- HumanByte::from(gc_status.pending_bytes),
+ HumanByte::new_binary(gc_status.pending_bytes),
gc_status.pending_chunks,
);
}
@@ -676,7 +675,7 @@ impl DataStore {
task_log!(
worker,
"Original data usage: {}",
- HumanByte::from(gc_status.index_data_bytes),
+ HumanByte::new_binary(gc_status.index_data_bytes),
);
if gc_status.index_data_bytes > 0 {
@@ -684,7 +683,7 @@ impl DataStore {
task_log!(
worker,
"On-Disk usage: {} ({:.2}%)",
- HumanByte::from(gc_status.disk_bytes),
+ HumanByte::new_binary(gc_status.disk_bytes),
comp_per,
);
}
@@ -701,7 +700,7 @@ impl DataStore {
if gc_status.disk_chunks > 0 {
let avg_chunk = gc_status.disk_bytes/(gc_status.disk_chunks as u64);
- task_log!(worker, "Average chunk size: {}", HumanByte::from(avg_chunk));
+ task_log!(worker, "Average chunk size: {}", HumanByte::new_binary(avg_chunk));
}
if let Ok(serialized) = serde_json::to_string(&gc_status) {
diff --git a/pbs-tools/Cargo.toml b/pbs-tools/Cargo.toml
index 635933e1..8c5f4671 100644
--- a/pbs-tools/Cargo.toml
+++ b/pbs-tools/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-time = { version = "1" }
pbs-buildcfg = { path = "../pbs-buildcfg" }
pbs-runtime = { path = "../pbs-runtime" }
+pbs-api-types = { path = "../pbs-api-types" }
[dev-dependencies]
tokio = { version = "1.6", features = [ "macros" ] }
diff --git a/pbs-tools/src/format.rs b/pbs-tools/src/format.rs
index feabd8f1..048a588d 100644
--- a/pbs-tools/src/format.rs
+++ b/pbs-tools/src/format.rs
@@ -3,6 +3,8 @@ use std::borrow::Borrow;
use anyhow::{Error};
use serde_json::Value;
+use pbs_api_types::HumanByte;
+
pub fn strip_server_file_extension(name: &str) -> &str {
if name.ends_with(".didx") || name.ends_with(".fidx") || name.ends_with(".blob") {
&name[..name.len()-5]
@@ -55,7 +57,7 @@ pub fn render_bytes_human_readable(value: &Value, _record: &Value) -> Result<Str
if value.is_null() { return Ok(String::new()); }
let text = match value.as_u64() {
Some(bytes) => {
- HumanByte::from(bytes).to_string()
+ HumanByte::new_binary(bytes).to_string()
}
None => {
value.to_string()
@@ -63,54 +65,3 @@ pub fn render_bytes_human_readable(value: &Value, _record: &Value) -> Result<Str
};
Ok(text)
}
-
-pub struct HumanByte {
- b: usize,
-}
-impl std::fmt::Display for HumanByte {
- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- if self.b < 1024 {
- return write!(f, "{} B", self.b);
- }
- let kb: f64 = self.b as f64 / 1024.0;
- if kb < 1024.0 {
- return write!(f, "{:.2} KiB", kb);
- }
- let mb: f64 = kb / 1024.0;
- if mb < 1024.0 {
- return write!(f, "{:.2} MiB", mb);
- }
- let gb: f64 = mb / 1024.0;
- if gb < 1024.0 {
- return write!(f, "{:.2} GiB", gb);
- }
- let tb: f64 = gb / 1024.0;
- if tb < 1024.0 {
- return write!(f, "{:.2} TiB", tb);
- }
- let pb: f64 = tb / 1024.0;
- return write!(f, "{:.2} PiB", pb);
- }
-}
-impl From<usize> for HumanByte {
- fn from(v: usize) -> Self {
- HumanByte { b: v }
- }
-}
-impl From<u64> for HumanByte {
- fn from(v: u64) -> Self {
- HumanByte { b: v as usize }
- }
-}
-
-#[test]
-fn correct_byte_convert() {
- fn convert(b: usize) -> String {
- HumanByte::from(b).to_string()
- }
- assert_eq!(convert(1023), "1023 B");
- assert_eq!(convert(1<<10), "1.00 KiB");
- assert_eq!(convert(1<<20), "1.00 MiB");
- assert_eq!(convert((1<<30) + 103 * (1<<20)), "1.10 GiB");
- assert_eq!(convert((2<<50) + 500 * (1<<40)), "2.49 PiB");
-}
diff --git a/src/bin/proxmox-tape.rs b/src/bin/proxmox-tape.rs
index 5de727c1..8f3a1248 100644
--- a/src/bin/proxmox-tape.rs
+++ b/src/bin/proxmox-tape.rs
@@ -10,7 +10,6 @@ use proxmox_time::strftime_local;
use pbs_client::view_task_result;
use pbs_tools::format::{
- HumanByte,
render_epoch,
render_bytes_human_readable,
};
@@ -21,7 +20,7 @@ use pbs_config::datastore::complete_datastore_name;
use pbs_api_types::{
Userid, Authid, DATASTORE_SCHEMA, DATASTORE_MAP_LIST_SCHEMA,
- DRIVE_NAME_SCHEMA, MEDIA_LABEL_SCHEMA, MEDIA_POOL_NAME_SCHEMA,
+ DRIVE_NAME_SCHEMA, HumanByte, MEDIA_LABEL_SCHEMA, MEDIA_POOL_NAME_SCHEMA,
TAPE_RESTORE_SNAPSHOT_SCHEMA,
};
use pbs_tape::{
@@ -599,7 +598,7 @@ fn debug_scan(mut param: Value) -> Result<(), Error> {
println!("got content header: {}", name);
println!(" uuid: {}", header.content_uuid());
println!(" ctime: {}", strftime_local("%c", header.ctime)?);
- println!(" hsize: {}", HumanByte::from(header.size as usize));
+ println!(" hsize: {}", HumanByte::new_binary(header.size as u64));
println!(" part: {}", header.part_number);
} else {
println!("got unknown content header: {:?}", header.content_magic);
@@ -610,7 +609,7 @@ fn debug_scan(mut param: Value) -> Result<(), Error> {
}
}
let bytes = reader.skip_data()?;
- println!("skipped {}", HumanByte::from(bytes));
+ println!("skipped {}", HumanByte::new_binary(bytes as u64));
if let Ok(true) = reader.has_end_marker() {
if reader.is_incomplete()? {
println!("WARNING: file is incomplete");
diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index b462bd4c..8b57a67a 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -7,10 +7,9 @@ use proxmox::tools::email::sendmail;
use proxmox_lang::try_block;
use proxmox_schema::{parse_property_string, ApiType};
-use pbs_tools::format::HumanByte;
use pbs_api_types::{
User, TapeBackupJobSetup, SyncJobConfig, VerificationJobConfig,
- APTUpdateInfo, GarbageCollectionStatus,
+ APTUpdateInfo, GarbageCollectionStatus, HumanByte,
Userid, Notify, DatastoreNotify, DataStoreConfig,
};
@@ -601,7 +600,7 @@ fn handlebars_humam_bytes_helper(
.flatten()
.ok_or_else(|| RenderError::new("human-bytes: param not found"))?;
- out.write(&HumanByte::from(param).to_string())?;
+ out.write(&HumanByte::new_binary(param).to_string())?;
Ok(())
}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [RFC proxmox-backup v2 2/4] use HumanByte for traffic-control config
2021-11-17 13:37 [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 1/4] pbs-api-types: " Dietmar Maurer
@ 2021-11-17 13:37 ` Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 3/4] HumanByte: use u64 instead of f64 to store size Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 4/4] HumanByte: do not store unit (always compute canonical form) Dietmar Maurer
3 siblings, 0 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-11-17 13:37 UTC (permalink / raw)
To: pbs-devel
---
pbs-api-types/src/traffic_control.rs | 18 +++++++++---------
src/cached_traffic_control.rs | 18 ++++++++++++------
2 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/pbs-api-types/src/traffic_control.rs b/pbs-api-types/src/traffic_control.rs
index 0dd7ed58..210f53ac 100644
--- a/pbs-api-types/src/traffic_control.rs
+++ b/pbs-api-types/src/traffic_control.rs
@@ -3,7 +3,7 @@ use serde::{Deserialize, Serialize};
use proxmox_schema::{api, Schema, IntegerSchema, StringSchema, Updater};
use crate::{
- CIDR_SCHEMA, DAILY_DURATION_FORMAT,
+ HumanByte, CIDR_SCHEMA, DAILY_DURATION_FORMAT,
PROXMOX_SAFE_ID_FORMAT, SINGLE_LINE_COMMENT_SCHEMA,
};
@@ -38,19 +38,19 @@ pub const TRAFFIC_CONTROL_BURST_SCHEMA: Schema = IntegerSchema::new(
schema: SINGLE_LINE_COMMENT_SCHEMA,
},
"rate-in": {
- schema: TRAFFIC_CONTROL_RATE_SCHEMA,
+ type: HumanByte,
optional: true,
},
"burst-in": {
- schema: TRAFFIC_CONTROL_BURST_SCHEMA,
+ type: HumanByte,
optional: true,
},
"rate-out": {
- schema: TRAFFIC_CONTROL_RATE_SCHEMA,
+ type: HumanByte,
optional: true,
},
"burst-out": {
- schema: TRAFFIC_CONTROL_BURST_SCHEMA,
+ type: HumanByte,
optional: true,
},
network: {
@@ -79,13 +79,13 @@ pub struct TrafficControlRule {
/// Rule applies to Source IPs within this networks
pub network: Vec<String>,
#[serde(skip_serializing_if="Option::is_none")]
- pub rate_in: Option<u64>,
+ pub rate_in: Option<HumanByte>,
#[serde(skip_serializing_if="Option::is_none")]
- pub burst_in: Option<u64>,
+ pub burst_in: Option<HumanByte>,
#[serde(skip_serializing_if="Option::is_none")]
- pub rate_out: Option<u64>,
+ pub rate_out: Option<HumanByte>,
#[serde(skip_serializing_if="Option::is_none")]
- pub burst_out: Option<u64>,
+ pub burst_out: Option<HumanByte>,
// fixme: expose this?
// /// Bandwidth is shared accross all connections
// #[serde(skip_serializing_if="Option::is_none")]
diff --git a/src/cached_traffic_control.rs b/src/cached_traffic_control.rs
index be554f6c..b45e564c 100644
--- a/src/cached_traffic_control.rs
+++ b/src/cached_traffic_control.rs
@@ -224,7 +224,10 @@ impl TrafficControlCache {
Some(ref read_limiter) => {
match rule.rate_in {
Some(rate_in) => {
- read_limiter.update_rate(rate_in, rule.burst_in.unwrap_or(rate_in));
+ read_limiter.update_rate(
+ rate_in.as_u64(),
+ rule.burst_in.unwrap_or(rate_in).as_u64(),
+ );
}
None => entry.0 = None,
}
@@ -235,8 +238,8 @@ impl TrafficControlCache {
let limiter = create_limiter(
self.use_shared_memory,
&name,
- rate_in,
- rule.burst_in.unwrap_or(rate_in),
+ rate_in.as_u64(),
+ rule.burst_in.unwrap_or(rate_in).as_u64(),
)?;
entry.0 = Some(limiter);
}
@@ -247,7 +250,10 @@ impl TrafficControlCache {
Some(ref write_limiter) => {
match rule.rate_out {
Some(rate_out) => {
- write_limiter.update_rate(rate_out, rule.burst_out.unwrap_or(rate_out));
+ write_limiter.update_rate(
+ rate_out.as_u64(),
+ rule.burst_out.unwrap_or(rate_out).as_u64(),
+ );
}
None => entry.1 = None,
}
@@ -258,8 +264,8 @@ impl TrafficControlCache {
let limiter = create_limiter(
self.use_shared_memory,
&name,
- rate_out,
- rule.burst_out.unwrap_or(rate_out),
+ rate_out.as_u64(),
+ rule.burst_out.unwrap_or(rate_out).as_u64(),
)?;
entry.1 = Some(limiter);
}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [RFC proxmox-backup v2 3/4] HumanByte: use u64 instead of f64 to store size
2021-11-17 13:37 [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 1/4] pbs-api-types: " Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 2/4] use HumanByte for traffic-control config Dietmar Maurer
@ 2021-11-17 13:37 ` Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 4/4] HumanByte: do not store unit (always compute canonical form) Dietmar Maurer
3 siblings, 0 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-11-17 13:37 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
---
pbs-api-types/src/human_byte.rs | 84 ++++++++++++++++-----------------
1 file changed, 42 insertions(+), 42 deletions(-)
diff --git a/pbs-api-types/src/human_byte.rs b/pbs-api-types/src/human_byte.rs
index df90d446..4d8c4f21 100644
--- a/pbs-api-types/src/human_byte.rs
+++ b/pbs-api-types/src/human_byte.rs
@@ -26,22 +26,22 @@ pub enum SizeUnit {
}
impl SizeUnit {
- pub fn factor(&self) -> f64 {
+ pub fn factor(&self) -> u64 {
match self {
- SizeUnit::None => 1.0,
- SizeUnit::Byte => 1.0,
-
- SizeUnit::Kilo | SizeUnit::KByte => 1_000.0,
- SizeUnit::Mega | SizeUnit::MByte => 1_000_000.0,
- SizeUnit::Giga | SizeUnit::GByte => 1_000_000_000.0,
- SizeUnit::Tera | SizeUnit::TByte => 1_000_000_000_000.0,
- SizeUnit::Peta | SizeUnit::PByte => 1_000_000_000_000_000.0,
-
- SizeUnit::Kibi => 1024.0,
- SizeUnit::Mebi => 1024.0*1024.0,
- SizeUnit::Gibi => 1024.0*1024.0*1024.0,
- SizeUnit::Tebi => 1024.0*1024.0*1024.0*1024.0,
- SizeUnit::Pebi => 1024.0*1024.0*1024.0*1024.0*1024.0,
+ SizeUnit::None => 1,
+ SizeUnit::Byte => 1,
+
+ SizeUnit::Kilo | SizeUnit::KByte => 1_000,
+ SizeUnit::Mega | SizeUnit::MByte => 1_000_000,
+ SizeUnit::Giga | SizeUnit::GByte => 1_000_000_000,
+ SizeUnit::Tera | SizeUnit::TByte => 1_000_000_000_000,
+ SizeUnit::Peta | SizeUnit::PByte => 1_000_000_000_000_000,
+
+ SizeUnit::Kibi => 1024,
+ SizeUnit::Mebi => 1024*1024,
+ SizeUnit::Gibi => 1024*1024*1024,
+ SizeUnit::Tebi => 1024*1024*1024*1024,
+ SizeUnit::Pebi => 1024*1024*1024*1024*1024,
}
}
@@ -122,7 +122,7 @@ fn strip_unit(v: &str) -> (&str, SizeUnit) {
#[derive(Debug, Copy, Clone, UpdaterType)]
/// Byte size with unit
pub struct HumanByte {
- size: f64,
+ size: u64,
unit: SizeUnit,
}
@@ -144,12 +144,12 @@ impl ApiType for HumanByte {
impl HumanByte {
pub fn new_decimal(size: u64) -> Self {
- let this = HumanByte { size: size as f64, unit: SizeUnit::None };
+ let this = HumanByte { size: size, unit: SizeUnit::None };
this.auto_unit_decimal()
}
pub fn new_binary(size: u64) -> Self {
- let this = HumanByte { size: size as f64, unit: SizeUnit::None };
+ let this = HumanByte { size: size, unit: SizeUnit::None };
this.auto_unit_binary()
}
@@ -178,15 +178,15 @@ impl HumanByte {
}
pub fn auto_unit_decimal(mut self) -> Self {
- self.unit = if self.size >= 1_000_000_000_000_000.0 {
+ self.unit = if self.size >= 1_000_000_000_000_000 {
SizeUnit::PByte
- } else if self.size >= 1_000_000_000_000.0 {
+ } else if self.size >= 1_000_000_000_000 {
SizeUnit::TByte
- } else if self.size >= 1_000_000_000.0 {
+ } else if self.size >= 1_000_000_000 {
SizeUnit::GByte
- } else if self.size >= 1_000_000.0 {
+ } else if self.size >= 1_000_000 {
SizeUnit::MByte
- } else if self.size >= 1_000.0 {
+ } else if self.size >= 1_000 {
SizeUnit::KByte
} else {
SizeUnit::None
@@ -201,7 +201,7 @@ impl std::fmt::Display for HumanByte {
let unit = self.unit;
- let size = size/unit.factor();
+ let size = (size as f64)/(unit.factor() as f64);
let unit_str = unit.unit_str();
if unit == SizeUnit::Byte || unit == SizeUnit::None {
@@ -237,9 +237,9 @@ impl FromStr for HumanByte {
bail!("size may not be negative");
}
- let size = size*unit.factor();
+ let size = (size*(unit.factor() as f64)) as u64;
- Ok(Self { size: size, unit })
+ Ok(Self { size, unit })
}
}
@@ -252,7 +252,7 @@ fn test_human_byte_parser() -> Result<(), Error> {
assert!("-10".parse::<HumanByte>().is_err()); // negative size
- fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
+ fn test(v: &str, size: u64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
let h: HumanByte = v.parse()?;
if h.size != size {
@@ -271,29 +271,29 @@ fn test_human_byte_parser() -> Result<(), Error> {
Ok(())
}
- test("14.4", 14.4, SizeUnit::None, "14")?;
+ test("14.4", 14, SizeUnit::None, "14")?;
- test("14", 14.0, SizeUnit::None, "14")?;
- test("987654321", 987654321.0, SizeUnit::None, "987654321")?;
+ test("14", 14, SizeUnit::None, "14")?;
+ test("987654321", 987654321, SizeUnit::None, "987654321")?;
- test("1300b", 1300.0, SizeUnit::Byte, "1300 B")?;
- test("1300B", 1300.0, SizeUnit::Byte, "1300 B")?;
+ test("1300b", 1300, SizeUnit::Byte, "1300 B")?;
+ test("1300B", 1300, SizeUnit::Byte, "1300 B")?;
- test("1.5KB", 1500.0, SizeUnit::KByte, "1.5 KB")?;
- test("1.5kb", 1500.0, SizeUnit::KByte, "1.5 KB")?;
- test("1.654321MB", 1_654_321.0, SizeUnit::MByte, "1.654 MB")?;
+ test("1.5KB", 1500, SizeUnit::KByte, "1.5 KB")?;
+ test("1.5kb", 1500, SizeUnit::KByte, "1.5 KB")?;
+ test("1.654321MB", 1_654_321, SizeUnit::MByte, "1.654 MB")?;
- test("2.0GB", 2_000_000_000.0, SizeUnit::GByte, "2 GB")?;
+ test("2.0GB", 2_000_000_000, SizeUnit::GByte, "2 GB")?;
- test("1.4TB", 1_400_000_000_000.0, SizeUnit::TByte, "1.4 TB")?;
- test("1.4tb", 1_400_000_000_000.0, SizeUnit::TByte, "1.4 TB")?;
+ test("1.4TB", 1_400_000_000_000, SizeUnit::TByte, "1.4 TB")?;
+ test("1.4tb", 1_400_000_000_000, SizeUnit::TByte, "1.4 TB")?;
- test("2KiB", 2048.0, SizeUnit::Kibi, "2 KiB")?;
- test("2kib", 2048.0, SizeUnit::Kibi, "2 KiB")?;
+ test("2KiB", 2048, SizeUnit::Kibi, "2 KiB")?;
+ test("2kib", 2048, SizeUnit::Kibi, "2 KiB")?;
- test("2.3456MiB", 2.3456*1024.0*1024.0, SizeUnit::Mebi, "2.345 MiB")?;
+ test("2.3456MiB", (2.3456*1024.0*1024.0) as u64, SizeUnit::Mebi, "2.345 MiB")?;
- test("4gib", 4.0*1024.0*1024.0*1024.0, SizeUnit::Gibi, "4 GiB")?;
+ test("4gib", (4.0*1024.0*1024.0*1024.0) as u64, SizeUnit::Gibi, "4 GiB")?;
Ok(())
}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread
* [pbs-devel] [RFC proxmox-backup v2 4/4] HumanByte: do not store unit (always compute canonical form)
2021-11-17 13:37 [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type Dietmar Maurer
` (2 preceding siblings ...)
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 3/4] HumanByte: use u64 instead of f64 to store size Dietmar Maurer
@ 2021-11-17 13:37 ` Dietmar Maurer
3 siblings, 0 replies; 5+ messages in thread
From: Dietmar Maurer @ 2021-11-17 13:37 UTC (permalink / raw)
To: pbs-devel
---
pbs-api-types/src/human_byte.rs | 161 ++++++++++++++------------------
src/cached_traffic_control.rs | 16 ++--
2 files changed, 80 insertions(+), 97 deletions(-)
diff --git a/pbs-api-types/src/human_byte.rs b/pbs-api-types/src/human_byte.rs
index 4d8c4f21..f233c9fb 100644
--- a/pbs-api-types/src/human_byte.rs
+++ b/pbs-api-types/src/human_byte.rs
@@ -8,15 +8,10 @@ use proxmox_schema::{ApiStringFormat, ApiType, StringSchema, Schema, UpdaterType
pub enum SizeUnit {
None,
Byte,
- Kilo,
KByte,
- Mega,
MByte,
- Giga,
GByte,
- Tera,
TByte,
- Peta,
PByte,
Kibi,
Mebi,
@@ -31,11 +26,11 @@ impl SizeUnit {
SizeUnit::None => 1,
SizeUnit::Byte => 1,
- SizeUnit::Kilo | SizeUnit::KByte => 1_000,
- SizeUnit::Mega | SizeUnit::MByte => 1_000_000,
- SizeUnit::Giga | SizeUnit::GByte => 1_000_000_000,
- SizeUnit::Tera | SizeUnit::TByte => 1_000_000_000_000,
- SizeUnit::Peta | SizeUnit::PByte => 1_000_000_000_000_000,
+ SizeUnit::KByte => 1_000,
+ SizeUnit::MByte => 1_000_000,
+ SizeUnit::GByte => 1_000_000_000,
+ SizeUnit::TByte => 1_000_000_000_000,
+ SizeUnit::PByte => 1_000_000_000_000_000,
SizeUnit::Kibi => 1024,
SizeUnit::Mebi => 1024*1024,
@@ -50,12 +45,6 @@ impl SizeUnit {
SizeUnit::None => "",
SizeUnit::Byte => "B",
- SizeUnit::Kilo => "K",
- SizeUnit::Mega => "M",
- SizeUnit::Giga => "G",
- SizeUnit::Tera => "T",
- SizeUnit::Peta => "P",
-
SizeUnit::KByte => "KB",
SizeUnit::MByte => "MB",
SizeUnit::GByte => "GB",
@@ -74,19 +63,19 @@ impl SizeUnit {
fn strip_unit(v: &str) -> (&str, SizeUnit) {
if let Some(v) = v.strip_suffix(&['k', 'K'][..]) {
- return (v, SizeUnit::Kilo);
+ return (v, SizeUnit::KByte);
}
if let Some(v) = v.strip_suffix(&['m', 'M'][..]) {
- return (v, SizeUnit::Mega);
+ return (v, SizeUnit::MByte);
}
if let Some(v) = v.strip_suffix(&['g', 'G'][..]) {
- return (v, SizeUnit::Giga);
+ return (v, SizeUnit::GByte);
}
if let Some(v) = v.strip_suffix(&['t', 'T'][..]) {
- return (v, SizeUnit::Tera);
+ return (v, SizeUnit::TByte);
}
if let Some(v) = v.strip_suffix(&['p', 'P'][..]) {
- return (v, SizeUnit::Peta);
+ return (v, SizeUnit::PByte);
}
if let Some(mut v) = v.strip_suffix(&['b', 'B'][..]) {
@@ -122,8 +111,8 @@ fn strip_unit(v: &str) -> (&str, SizeUnit) {
#[derive(Debug, Copy, Clone, UpdaterType)]
/// Byte size with unit
pub struct HumanByte {
- size: u64,
- unit: SizeUnit,
+ pub size: u64,
+ binary: bool,
}
fn verify_human_byte(s: &str) -> Result<(), Error> {
@@ -144,64 +133,54 @@ impl ApiType for HumanByte {
impl HumanByte {
pub fn new_decimal(size: u64) -> Self {
- let this = HumanByte { size: size, unit: SizeUnit::None };
- this.auto_unit_decimal()
+ HumanByte { size: size, binary: false }
}
pub fn new_binary(size: u64) -> Self {
- let this = HumanByte { size: size, unit: SizeUnit::None };
- this.auto_unit_binary()
- }
-
- pub fn as_u64(&self) -> u64 {
- self.size as u64
+ HumanByte { size: size, binary: true }
}
- pub fn auto_unit_binary(mut self) -> Self {
- let size = self.size as u64;
- let zeroes = size.leading_zeros();
- let bits = 63 - zeroes;
- self.unit = if bits >= 50 {
- SizeUnit::Pebi
- } else if bits >= 40 {
- SizeUnit::Tebi
- } else if bits >= 30 {
- SizeUnit::Gibi
- } else if bits >= 20 {
- SizeUnit::Mebi
- } else if bits >= 10 {
- SizeUnit::Kibi
- } else {
- SizeUnit::None
- };
- self
- }
-
- pub fn auto_unit_decimal(mut self) -> Self {
- self.unit = if self.size >= 1_000_000_000_000_000 {
- SizeUnit::PByte
- } else if self.size >= 1_000_000_000_000 {
- SizeUnit::TByte
- } else if self.size >= 1_000_000_000 {
- SizeUnit::GByte
- } else if self.size >= 1_000_000 {
- SizeUnit::MByte
- } else if self.size >= 1_000 {
- SizeUnit::KByte
+ fn auto_unit(&self) -> SizeUnit {
+ if self.binary {
+ let zeroes = self.size.leading_zeros();
+ let bits = 63 - zeroes;
+ if bits >= 50 {
+ SizeUnit::Pebi
+ } else if bits >= 40 {
+ SizeUnit::Tebi
+ } else if bits >= 30 {
+ SizeUnit::Gibi
+ } else if bits >= 20 {
+ SizeUnit::Mebi
+ } else if bits >= 10 {
+ SizeUnit::Kibi
+ } else {
+ SizeUnit::None
+ }
} else {
- SizeUnit::None
- };
- self
+ if self.size >= 1_000_000_000_000_000 {
+ SizeUnit::PByte
+ } else if self.size >= 1_000_000_000_000 {
+ SizeUnit::TByte
+ } else if self.size >= 1_000_000_000 {
+ SizeUnit::GByte
+ } else if self.size >= 1_000_000 {
+ SizeUnit::MByte
+ } else if self.size >= 1_000 {
+ SizeUnit::KByte
+ } else {
+ SizeUnit::None
+ }
+ }
}
}
impl std::fmt::Display for HumanByte {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- let size = self.size;
- let unit = self.unit;
+ let unit = self.auto_unit();
- let size = (size as f64)/(unit.factor() as f64);
+ let size = (self.size as f64)/(unit.factor() as f64);
let unit_str = unit.unit_str();
if unit == SizeUnit::Byte || unit == SizeUnit::None {
@@ -239,9 +218,16 @@ impl FromStr for HumanByte {
let size = (size*(unit.factor() as f64)) as u64;
- Ok(Self { size, unit })
- }
+ let binary = match unit {
+ SizeUnit::Kibi | SizeUnit::Mebi | SizeUnit::Gibi |
+ SizeUnit::Tebi | SizeUnit::Pebi => true,
+ SizeUnit::None | SizeUnit::Byte | SizeUnit::KByte |
+ SizeUnit::MByte | SizeUnit::GByte | SizeUnit::TByte |
+ SizeUnit::PByte => false,
+ };
+ Ok(Self { size, binary})
+ }
}
proxmox::forward_deserialize_to_from_str!(HumanByte);
@@ -252,15 +238,12 @@ fn test_human_byte_parser() -> Result<(), Error> {
assert!("-10".parse::<HumanByte>().is_err()); // negative size
- fn test(v: &str, size: u64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
+ fn test(v: &str, size: u64, as_str: &str) -> Result<(), Error> {
let h: HumanByte = v.parse()?;
if h.size != size {
bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
}
- if h.unit != unit {
- bail!("got unexpected unit for '{}' ({:?} != {:?})", v, h.unit, unit);
- }
let new = h.to_string();
if &new != as_str {
@@ -271,29 +254,29 @@ fn test_human_byte_parser() -> Result<(), Error> {
Ok(())
}
- test("14.4", 14, SizeUnit::None, "14")?;
+ test("14.4", 14, "14")?;
- test("14", 14, SizeUnit::None, "14")?;
- test("987654321", 987654321, SizeUnit::None, "987654321")?;
+ test("14", 14, "14")?;
+ test("987654321", 987654321, "987.654 MB")?;
- test("1300b", 1300, SizeUnit::Byte, "1300 B")?;
- test("1300B", 1300, SizeUnit::Byte, "1300 B")?;
+ test("1300b", 1300, "1.3 KB")?;
+ test("1300B", 1300, "1.3 KB")?;
- test("1.5KB", 1500, SizeUnit::KByte, "1.5 KB")?;
- test("1.5kb", 1500, SizeUnit::KByte, "1.5 KB")?;
- test("1.654321MB", 1_654_321, SizeUnit::MByte, "1.654 MB")?;
+ test("1.5KB", 1500, "1.5 KB")?;
+ test("1.5kb", 1500, "1.5 KB")?;
+ test("1.654321MB", 1_654_321, "1.654 MB")?;
- test("2.0GB", 2_000_000_000, SizeUnit::GByte, "2 GB")?;
+ test("2.0GB", 2_000_000_000, "2 GB")?;
- test("1.4TB", 1_400_000_000_000, SizeUnit::TByte, "1.4 TB")?;
- test("1.4tb", 1_400_000_000_000, SizeUnit::TByte, "1.4 TB")?;
+ test("1.4TB", 1_400_000_000_000, "1.4 TB")?;
+ test("1.4tb", 1_400_000_000_000, "1.4 TB")?;
- test("2KiB", 2048, SizeUnit::Kibi, "2 KiB")?;
- test("2kib", 2048, SizeUnit::Kibi, "2 KiB")?;
+ test("2KiB", 2048, "2 KiB")?;
+ test("2kib", 2048, "2 KiB")?;
- test("2.3456MiB", (2.3456*1024.0*1024.0) as u64, SizeUnit::Mebi, "2.345 MiB")?;
+ test("2.3456MiB", (2.3456*1024.0*1024.0) as u64, "2.345 MiB")?;
- test("4gib", (4.0*1024.0*1024.0*1024.0) as u64, SizeUnit::Gibi, "4 GiB")?;
+ test("4gib", (4.0*1024.0*1024.0*1024.0) as u64, "4 GiB")?;
Ok(())
}
diff --git a/src/cached_traffic_control.rs b/src/cached_traffic_control.rs
index b45e564c..05bf8a60 100644
--- a/src/cached_traffic_control.rs
+++ b/src/cached_traffic_control.rs
@@ -225,8 +225,8 @@ impl TrafficControlCache {
match rule.rate_in {
Some(rate_in) => {
read_limiter.update_rate(
- rate_in.as_u64(),
- rule.burst_in.unwrap_or(rate_in).as_u64(),
+ rate_in.size,
+ rule.burst_in.unwrap_or(rate_in).size,
);
}
None => entry.0 = None,
@@ -238,8 +238,8 @@ impl TrafficControlCache {
let limiter = create_limiter(
self.use_shared_memory,
&name,
- rate_in.as_u64(),
- rule.burst_in.unwrap_or(rate_in).as_u64(),
+ rate_in.size,
+ rule.burst_in.unwrap_or(rate_in).size,
)?;
entry.0 = Some(limiter);
}
@@ -251,8 +251,8 @@ impl TrafficControlCache {
match rule.rate_out {
Some(rate_out) => {
write_limiter.update_rate(
- rate_out.as_u64(),
- rule.burst_out.unwrap_or(rate_out).as_u64(),
+ rate_out.size,
+ rule.burst_out.unwrap_or(rate_out).size,
);
}
None => entry.1 = None,
@@ -264,8 +264,8 @@ impl TrafficControlCache {
let limiter = create_limiter(
self.use_shared_memory,
&name,
- rate_out.as_u64(),
- rule.burst_out.unwrap_or(rate_out).as_u64(),
+ rate_out.size,
+ rule.burst_out.unwrap_or(rate_out).size,
)?;
entry.1 = Some(limiter);
}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-11-17 13:37 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-17 13:37 [pbs-devel] [RFC proxmox-backup v2 0/4] more flexible HumanByte type Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 1/4] pbs-api-types: " Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 2/4] use HumanByte for traffic-control config Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 3/4] HumanByte: use u64 instead of f64 to store size Dietmar Maurer
2021-11-17 13:37 ` [pbs-devel] [RFC proxmox-backup v2 4/4] HumanByte: do not store unit (always compute canonical form) Dietmar Maurer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.