* [pbs-devel] [PATCH proxmox 1/3] add `proxmox-human-byte` crate
2023-05-08 10:01 [pbs-devel] [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Lukas Wagner
@ 2023-05-08 10:01 ` Lukas Wagner
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox 2/3] human-byte: move tests to their sub module Lukas Wagner
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Lukas Wagner @ 2023-05-08 10:01 UTC (permalink / raw)
To: pbs-devel
The module previously lived in `pbs-api-types`, however turned out to
be useful in other places as well (POM, proxmox-notify), so it is moved
here as its own micro-crate.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
Cargo.toml | 1 +
proxmox-human-byte/Cargo.toml | 15 +
proxmox-human-byte/debian/changelog | 5 +
proxmox-human-byte/debian/control | 43 +++
proxmox-human-byte/debian/copyright | 16 ++
proxmox-human-byte/debian/debcargo.toml | 7 +
proxmox-human-byte/src/lib.rs | 358 ++++++++++++++++++++++++
7 files changed, 445 insertions(+)
create mode 100644 proxmox-human-byte/Cargo.toml
create mode 100644 proxmox-human-byte/debian/changelog
create mode 100644 proxmox-human-byte/debian/control
create mode 100644 proxmox-human-byte/debian/copyright
create mode 100644 proxmox-human-byte/debian/debcargo.toml
create mode 100644 proxmox-human-byte/src/lib.rs
diff --git a/Cargo.toml b/Cargo.toml
index a32e1c6..ac2bd7e 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -6,6 +6,7 @@ members = [
"proxmox-borrow",
"proxmox-compression",
"proxmox-http",
+ "proxmox-human-byte",
"proxmox-io",
"proxmox-lang",
"proxmox-ldap",
diff --git a/proxmox-human-byte/Cargo.toml b/proxmox-human-byte/Cargo.toml
new file mode 100644
index 0000000..4cdbe6c
--- /dev/null
+++ b/proxmox-human-byte/Cargo.toml
@@ -0,0 +1,15 @@
+[package]
+name = "proxmox-human-byte"
+version = "0.1.0"
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+exclude.workspace = true
+description = "Proxmox library for formatting byte sizes (IEC or SI)"
+
+[dependencies]
+anyhow.workspace = true
+proxmox-schema = { workspace = true, features = ["api-macro"]}
+proxmox-serde.workspace = true
+serde.workspace = true
diff --git a/proxmox-human-byte/debian/changelog b/proxmox-human-byte/debian/changelog
new file mode 100644
index 0000000..7f7603b
--- /dev/null
+++ b/proxmox-human-byte/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-human-byte (0.1.0-1) stable; urgency=medium
+
+ * Initial release.
+
+ -- Proxmox Support Team <support@proxmox.com> Thu, 12 Jan 2023 11:42:11 +0200
diff --git a/proxmox-human-byte/debian/control b/proxmox-human-byte/debian/control
new file mode 100644
index 0000000..6aae2a5
--- /dev/null
+++ b/proxmox-human-byte/debian/control
@@ -0,0 +1,43 @@
+Source: rust-proxmox-human-byte
+Section: rust
+Priority: optional
+Build-Depends: debhelper (>= 12),
+ dh-cargo (>= 25),
+ cargo:native <!nocheck>,
+ rustc:native <!nocheck>,
+ libstd-rust-dev <!nocheck>,
+ librust-anyhow-1+default-dev <!nocheck>,
+ librust-proxmox-schema-1+api-macro-dev (>= 1.3.7-~~) <!nocheck>,
+ librust-proxmox-schema-1+default-dev (>= 1.3.7-~~) <!nocheck>,
+ librust-proxmox-serde-0.1+default-dev (>= 0.1.1-~~) <!nocheck>,
+ librust-proxmox-serde-0.1+serde-json-dev (>= 0.1.1-~~) <!nocheck>,
+ librust-serde-1+default-dev <!nocheck>
+Maintainer: Proxmox Support Team <support@proxmox.com>
+Standards-Version: 4.6.1
+Vcs-Git: git://git.proxmox.com/git/proxmox.git
+Vcs-Browser: https://git.proxmox.com/?p=proxmox.git
+X-Cargo-Crate: proxmox-human-byte
+Rules-Requires-Root: no
+
+Package: librust-proxmox-human-byte-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-anyhow-1+default-dev,
+ librust-proxmox-schema-1+api-macro-dev (>= 1.3.7-~~),
+ librust-proxmox-schema-1+default-dev (>= 1.3.7-~~),
+ librust-proxmox-serde-0.1+default-dev (>= 0.1.1-~~),
+ librust-proxmox-serde-0.1+serde-json-dev (>= 0.1.1-~~),
+ librust-serde-1+default-dev
+Provides:
+ librust-proxmox-human-byte+default-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0+default-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0.1-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0.1+default-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0.1.0-dev (= ${binary:Version}),
+ librust-proxmox-human-byte-0.1.0+default-dev (= ${binary:Version})
+Description: Proxmox library for formatting byte sizes (IEC or SI) - Rust source code
+ This package contains the source for the Rust proxmox-human-byte crate,
+ packaged by debcargo for use with cargo and dh-cargo.
diff --git a/proxmox-human-byte/debian/copyright b/proxmox-human-byte/debian/copyright
new file mode 100644
index 0000000..4fce23a
--- /dev/null
+++ b/proxmox-human-byte/debian/copyright
@@ -0,0 +1,16 @@
+Copyright (C) 2023 Proxmox Server Solutions GmbH
+
+This software is written by Proxmox Server Solutions GmbH <support@proxmox.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
diff --git a/proxmox-human-byte/debian/debcargo.toml b/proxmox-human-byte/debian/debcargo.toml
new file mode 100644
index 0000000..b7864cd
--- /dev/null
+++ b/proxmox-human-byte/debian/debcargo.toml
@@ -0,0 +1,7 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+
+[source]
+vcs_git = "git://git.proxmox.com/git/proxmox.git"
+vcs_browser = "https://git.proxmox.com/?p=proxmox.git"
diff --git a/proxmox-human-byte/src/lib.rs b/proxmox-human-byte/src/lib.rs
new file mode 100644
index 0000000..7be16a5
--- /dev/null
+++ b/proxmox-human-byte/src/lib.rs
@@ -0,0 +1,358 @@
+use anyhow::{bail, Error};
+
+use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema, UpdaterType};
+
+/// Size units for byte sizes
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum SizeUnit {
+ Byte,
+ // SI (base 10)
+ KByte,
+ MByte,
+ GByte,
+ TByte,
+ PByte,
+ // IEC (base 2)
+ Kibi,
+ Mebi,
+ Gibi,
+ Tebi,
+ Pebi,
+}
+
+impl SizeUnit {
+ /// Returns the scaling factor
+ pub fn factor(&self) -> f64 {
+ match self {
+ SizeUnit::Byte => 1.0,
+ // SI (base 10)
+ SizeUnit::KByte => 1_000.0,
+ SizeUnit::MByte => 1_000_000.0,
+ SizeUnit::GByte => 1_000_000_000.0,
+ SizeUnit::TByte => 1_000_000_000_000.0,
+ SizeUnit::PByte => 1_000_000_000_000_000.0,
+ // IEC (base 2)
+ SizeUnit::Kibi => 1024.0,
+ SizeUnit::Mebi => 1024.0 * 1024.0,
+ SizeUnit::Gibi => 1024.0 * 1024.0 * 1024.0,
+ SizeUnit::Tebi => 1024.0 * 1024.0 * 1024.0 * 1024.0,
+ SizeUnit::Pebi => 1024.0 * 1024.0 * 1024.0 * 1024.0 * 1024.0,
+ }
+ }
+
+ /// gets the biggest possible unit still having a value greater zero before the decimal point
+ /// 'binary' specifies if IEC (base 2) units should be used or SI (base 10) ones
+ pub fn auto_scale(size: f64, binary: bool) -> SizeUnit {
+ if binary {
+ let bits = 64 - (size as u64).leading_zeros();
+ match bits {
+ 51.. => SizeUnit::Pebi,
+ 41..=50 => SizeUnit::Tebi,
+ 31..=40 => SizeUnit::Gibi,
+ 21..=30 => SizeUnit::Mebi,
+ 11..=20 => SizeUnit::Kibi,
+ _ => SizeUnit::Byte,
+ }
+ } else if size >= 1_000_000_000_000_000.0 {
+ SizeUnit::PByte
+ } else if size >= 1_000_000_000_000.0 {
+ SizeUnit::TByte
+ } else if size >= 1_000_000_000.0 {
+ SizeUnit::GByte
+ } else if size >= 1_000_000.0 {
+ SizeUnit::MByte
+ } else if size >= 1_000.0 {
+ SizeUnit::KByte
+ } else {
+ SizeUnit::Byte
+ }
+ }
+}
+
+/// Returns the string representation
+impl std::fmt::Display for SizeUnit {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ SizeUnit::Byte => write!(f, "B"),
+ // SI (base 10)
+ SizeUnit::KByte => write!(f, "KB"),
+ SizeUnit::MByte => write!(f, "MB"),
+ SizeUnit::GByte => write!(f, "GB"),
+ SizeUnit::TByte => write!(f, "TB"),
+ SizeUnit::PByte => write!(f, "PB"),
+ // IEC (base 2)
+ SizeUnit::Kibi => write!(f, "KiB"),
+ SizeUnit::Mebi => write!(f, "MiB"),
+ SizeUnit::Gibi => write!(f, "GiB"),
+ SizeUnit::Tebi => write!(f, "TiB"),
+ SizeUnit::Pebi => write!(f, "PiB"),
+ }
+ }
+}
+
+/// Strips a trailing SizeUnit inclusive trailing whitespace
+/// Supports both IEC and SI based scales, the B/b byte symbol is optional.
+fn strip_unit(v: &str) -> (&str, SizeUnit) {
+ let v = v.strip_suffix(&['b', 'B'][..]).unwrap_or(v); // byte is implied anyway
+
+ let (v, binary) = match v.strip_suffix('i') {
+ Some(n) => (n, true),
+ None => (v, false),
+ };
+
+ let mut unit = SizeUnit::Byte;
+ #[rustfmt::skip]
+ let value = v.strip_suffix(|c: char| match c {
+ 'k' | 'K' if !binary => { unit = SizeUnit::KByte; true }
+ 'm' | 'M' if !binary => { unit = SizeUnit::MByte; true }
+ 'g' | 'G' if !binary => { unit = SizeUnit::GByte; true }
+ 't' | 'T' if !binary => { unit = SizeUnit::TByte; true }
+ 'p' | 'P' if !binary => { unit = SizeUnit::PByte; true }
+ // binary (IEC recommended) variants
+ 'k' | 'K' if binary => { unit = SizeUnit::Kibi; true }
+ 'm' | 'M' if binary => { unit = SizeUnit::Mebi; true }
+ 'g' | 'G' if binary => { unit = SizeUnit::Gibi; true }
+ 't' | 'T' if binary => { unit = SizeUnit::Tebi; true }
+ 'p' | 'P' if binary => { unit = SizeUnit::Pebi; true }
+ _ => false
+ }).unwrap_or(v).trim_end();
+
+ (value, unit)
+}
+
+/// Byte size which can be displayed in a human friendly way
+#[derive(Debug, Copy, Clone, UpdaterType, PartialEq)]
+pub struct HumanByte {
+ /// The siginficant value, it does not includes any factor of the `unit`
+ size: f64,
+ /// The scale/unit of the value
+ unit: SizeUnit,
+}
+
+fn verify_human_byte(s: &str) -> Result<(), Error> {
+ match s.parse::<HumanByte>() {
+ Ok(_) => Ok(()),
+ Err(err) => bail!("byte-size parse error for '{}': {}", s, err),
+ }
+}
+impl ApiType for HumanByte {
+ const API_SCHEMA: Schema = StringSchema::new(
+ "Byte size with optional unit (B, KB (base 10), MB, GB, ..., KiB (base 2), MiB, Gib, ...).",
+ )
+ .format(&ApiStringFormat::VerifyFn(verify_human_byte))
+ .min_length(1)
+ .max_length(64)
+ .schema();
+}
+
+impl HumanByte {
+ /// Create instance with size and unit (size must be positive)
+ pub fn with_unit(size: f64, unit: SizeUnit) -> Result<Self, Error> {
+ if size < 0.0 {
+ bail!("byte size may not be negative");
+ }
+ Ok(HumanByte { size, unit })
+ }
+
+ /// Create a new instance with optimal binary unit computed
+ pub fn new_binary(size: f64) -> Self {
+ let unit = SizeUnit::auto_scale(size, true);
+ HumanByte {
+ size: size / unit.factor(),
+ unit,
+ }
+ }
+
+ /// Create a new instance with optimal decimal unit computed
+ pub fn new_decimal(size: f64) -> Self {
+ let unit = SizeUnit::auto_scale(size, false);
+ HumanByte {
+ size: size / unit.factor(),
+ unit,
+ }
+ }
+
+ /// Returns the size as u64 number of bytes
+ pub fn as_u64(&self) -> u64 {
+ self.as_f64() as u64
+ }
+
+ /// Returns the size as f64 number of bytes
+ pub fn as_f64(&self) -> f64 {
+ self.size * self.unit.factor()
+ }
+
+ /// Returns a copy with optimal binary unit computed
+ pub fn auto_scale_binary(self) -> Self {
+ HumanByte::new_binary(self.as_f64())
+ }
+
+ /// Returns a copy with optimal decimal unit computed
+ pub fn auto_scale_decimal(self) -> Self {
+ HumanByte::new_decimal(self.as_f64())
+ }
+}
+
+impl From<u64> for HumanByte {
+ fn from(v: u64) -> Self {
+ HumanByte::new_binary(v as f64)
+ }
+}
+impl From<usize> for HumanByte {
+ fn from(v: usize) -> Self {
+ HumanByte::new_binary(v as f64)
+ }
+}
+
+impl std::fmt::Display for HumanByte {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ let precision = f.precision().unwrap_or(3) as f64;
+ let precision_factor = 1.0 * 10.0_f64.powf(precision);
+ // this could cause loss of information, rust has sadly no shortest-max-X flt2dec fmt yet
+ let size = ((self.size * precision_factor).round()) / precision_factor;
+ write!(f, "{} {}", size, self.unit)
+ }
+}
+
+impl std::str::FromStr for HumanByte {
+ type Err = Error;
+
+ fn from_str(v: &str) -> Result<Self, Error> {
+ let (v, unit) = strip_unit(v);
+ HumanByte::with_unit(v.parse()?, unit)
+ }
+}
+
+proxmox_serde::forward_deserialize_to_from_str!(HumanByte);
+proxmox_serde::forward_serialize_to_display!(HumanByte);
+
+#[test]
+fn test_human_byte_parser() -> Result<(), Error> {
+ assert!("-10".parse::<HumanByte>().is_err()); // negative size
+
+ fn do_test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
+ let h: HumanByte = v.parse()?;
+
+ if h.size != size {
+ bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
+ }
+ if h.unit != unit {
+ bail!(
+ "got unexpected unit for '{}' ({:?} != {:?})",
+ v,
+ h.unit,
+ unit
+ );
+ }
+
+ let new = h.to_string();
+ if new != *as_str {
+ bail!("to_string failed for '{}' ({:?} != {:?})", v, new, as_str);
+ }
+ Ok(())
+ }
+ fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> bool {
+ match do_test(v, size, unit, as_str) {
+ Ok(_) => true,
+ Err(err) => {
+ eprintln!("{}", err); // makes debugging easier
+ false
+ }
+ }
+ }
+
+ assert!(test("14", 14.0, SizeUnit::Byte, "14 B"));
+ assert!(test("14.4", 14.4, SizeUnit::Byte, "14.4 B"));
+ assert!(test("14.45", 14.45, SizeUnit::Byte, "14.45 B"));
+ assert!(test("14.456", 14.456, SizeUnit::Byte, "14.456 B"));
+ assert!(test("14.4567", 14.4567, SizeUnit::Byte, "14.457 B"));
+
+ let h: HumanByte = "1.2345678".parse()?;
+ assert_eq!(&format!("{:.0}", h), "1 B");
+ assert_eq!(&format!("{:.0}", h.as_f64()), "1"); // use as_f64 to get raw bytes without unit
+ assert_eq!(&format!("{:.1}", h), "1.2 B");
+ assert_eq!(&format!("{:.2}", h), "1.23 B");
+ assert_eq!(&format!("{:.3}", h), "1.235 B");
+ assert_eq!(&format!("{:.4}", h), "1.2346 B");
+ assert_eq!(&format!("{:.5}", h), "1.23457 B");
+ assert_eq!(&format!("{:.6}", h), "1.234568 B");
+ assert_eq!(&format!("{:.7}", h), "1.2345678 B");
+ assert_eq!(&format!("{:.8}", h), "1.2345678 B");
+
+ assert!(test(
+ "987654321",
+ 987654321.0,
+ SizeUnit::Byte,
+ "987654321 B"
+ ));
+
+ assert!(test("1300b", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300B", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300 B", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300 b", 1300.0, SizeUnit::Byte, "1300 B"));
+
+ assert!(test("1.5KB", 1.5, SizeUnit::KByte, "1.5 KB"));
+ assert!(test("1.5kb", 1.5, SizeUnit::KByte, "1.5 KB"));
+ assert!(test("1.654321MB", 1.654_321, SizeUnit::MByte, "1.654 MB"));
+
+ assert!(test("2.0GB", 2.0, SizeUnit::GByte, "2 GB"));
+
+ assert!(test("1.4TB", 1.4, SizeUnit::TByte, "1.4 TB"));
+ assert!(test("1.4tb", 1.4, SizeUnit::TByte, "1.4 TB"));
+
+ assert!(test("2KiB", 2.0, SizeUnit::Kibi, "2 KiB"));
+ assert!(test("2Ki", 2.0, SizeUnit::Kibi, "2 KiB"));
+ assert!(test("2kib", 2.0, SizeUnit::Kibi, "2 KiB"));
+
+ assert!(test("2.3454MiB", 2.3454, SizeUnit::Mebi, "2.345 MiB"));
+ assert!(test("2.3456MiB", 2.3456, SizeUnit::Mebi, "2.346 MiB"));
+
+ assert!(test("4gib", 4.0, SizeUnit::Gibi, "4 GiB"));
+
+ Ok(())
+}
+
+#[test]
+fn test_human_byte_auto_unit_decimal() {
+ fn convert(b: u64) -> String {
+ HumanByte::new_decimal(b as f64).to_string()
+ }
+ assert_eq!(convert(987), "987 B");
+ assert_eq!(convert(1022), "1.022 KB");
+ assert_eq!(convert(9_000), "9 KB");
+ assert_eq!(convert(1_000), "1 KB");
+ assert_eq!(convert(1_000_000), "1 MB");
+ assert_eq!(convert(1_000_000_000), "1 GB");
+ assert_eq!(convert(1_000_000_000_000), "1 TB");
+ assert_eq!(convert(1_000_000_000_000_000), "1 PB");
+
+ assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.182 GB");
+ assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.208 GB");
+ assert_eq!(convert((2 << 50) + 500 * (1 << 40)), "2.802 PB");
+}
+
+#[test]
+fn test_human_byte_auto_unit_binary() {
+ fn convert(b: u64) -> String {
+ HumanByte::from(b).to_string()
+ }
+ assert_eq!(convert(0), "0 B");
+ assert_eq!(convert(987), "987 B");
+ assert_eq!(convert(1022), "1022 B");
+ assert_eq!(convert(9_000), "8.789 KiB");
+ assert_eq!(convert(10_000_000), "9.537 MiB");
+ assert_eq!(convert(10_000_000_000), "9.313 GiB");
+ assert_eq!(convert(10_000_000_000_000), "9.095 TiB");
+
+ assert_eq!(convert(1 << 10), "1 KiB");
+ assert_eq!(convert((1 << 10) * 10), "10 KiB");
+ assert_eq!(convert(1 << 20), "1 MiB");
+ assert_eq!(convert(1 << 30), "1 GiB");
+ assert_eq!(convert(1 << 40), "1 TiB");
+ assert_eq!(convert(1 << 50), "1 PiB");
+
+ assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.101 GiB");
+ assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.125 GiB");
+ assert_eq!(convert((1 << 40) + 128 * (1 << 30)), "1.125 TiB");
+ assert_eq!(convert((2 << 50) + 512 * (1 << 40)), "2.5 PiB");
+}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread* [pbs-devel] [PATCH proxmox 2/3] human-byte: move tests to their sub module
2023-05-08 10:01 [pbs-devel] [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Lukas Wagner
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox 1/3] add `proxmox-human-byte` crate Lukas Wagner
@ 2023-05-08 10:01 ` Lukas Wagner
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox-backup 3/3] api-types: client: datastore: tools: use proxmox-human-bytes crate Lukas Wagner
2023-06-26 11:57 ` [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Wolfgang Bumiller
3 siblings, 0 replies; 5+ messages in thread
From: Lukas Wagner @ 2023-05-08 10:01 UTC (permalink / raw)
To: pbs-devel
The `#[cfg(test)]` directive ensures that the tests are not compiled
for non-test builds.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
proxmox-human-byte/src/lib.rs | 241 +++++++++++++++++-----------------
1 file changed, 123 insertions(+), 118 deletions(-)
diff --git a/proxmox-human-byte/src/lib.rs b/proxmox-human-byte/src/lib.rs
index 7be16a5..ef680b4 100644
--- a/proxmox-human-byte/src/lib.rs
+++ b/proxmox-human-byte/src/lib.rs
@@ -226,133 +226,138 @@ impl std::str::FromStr for HumanByte {
proxmox_serde::forward_deserialize_to_from_str!(HumanByte);
proxmox_serde::forward_serialize_to_display!(HumanByte);
-#[test]
-fn test_human_byte_parser() -> Result<(), Error> {
- assert!("-10".parse::<HumanByte>().is_err()); // negative size
+#[cfg(test)]
+mod tests {
+ use super::*;
- fn do_test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
- let h: HumanByte = v.parse()?;
+ #[test]
+ fn test_human_byte_parser() -> Result<(), Error> {
+ assert!("-10".parse::<HumanByte>().is_err()); // negative size
- if h.size != size {
- bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
- }
- if h.unit != unit {
- bail!(
- "got unexpected unit for '{}' ({:?} != {:?})",
- v,
- h.unit,
- unit
- );
- }
+ fn do_test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
+ let h: HumanByte = v.parse()?;
+
+ if h.size != size {
+ bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
+ }
+ if h.unit != unit {
+ bail!(
+ "got unexpected unit for '{}' ({:?} != {:?})",
+ v,
+ h.unit,
+ unit
+ );
+ }
- let new = h.to_string();
- if new != *as_str {
- bail!("to_string failed for '{}' ({:?} != {:?})", v, new, as_str);
+ let new = h.to_string();
+ if new != *as_str {
+ bail!("to_string failed for '{}' ({:?} != {:?})", v, new, as_str);
+ }
+ Ok(())
}
- Ok(())
- }
- fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> bool {
- match do_test(v, size, unit, as_str) {
- Ok(_) => true,
- Err(err) => {
- eprintln!("{}", err); // makes debugging easier
- false
+ fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> bool {
+ match do_test(v, size, unit, as_str) {
+ Ok(_) => true,
+ Err(err) => {
+ eprintln!("{}", err); // makes debugging easier
+ false
+ }
}
}
- }
- assert!(test("14", 14.0, SizeUnit::Byte, "14 B"));
- assert!(test("14.4", 14.4, SizeUnit::Byte, "14.4 B"));
- assert!(test("14.45", 14.45, SizeUnit::Byte, "14.45 B"));
- assert!(test("14.456", 14.456, SizeUnit::Byte, "14.456 B"));
- assert!(test("14.4567", 14.4567, SizeUnit::Byte, "14.457 B"));
-
- let h: HumanByte = "1.2345678".parse()?;
- assert_eq!(&format!("{:.0}", h), "1 B");
- assert_eq!(&format!("{:.0}", h.as_f64()), "1"); // use as_f64 to get raw bytes without unit
- assert_eq!(&format!("{:.1}", h), "1.2 B");
- assert_eq!(&format!("{:.2}", h), "1.23 B");
- assert_eq!(&format!("{:.3}", h), "1.235 B");
- assert_eq!(&format!("{:.4}", h), "1.2346 B");
- assert_eq!(&format!("{:.5}", h), "1.23457 B");
- assert_eq!(&format!("{:.6}", h), "1.234568 B");
- assert_eq!(&format!("{:.7}", h), "1.2345678 B");
- assert_eq!(&format!("{:.8}", h), "1.2345678 B");
-
- assert!(test(
- "987654321",
- 987654321.0,
- SizeUnit::Byte,
- "987654321 B"
- ));
-
- assert!(test("1300b", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300B", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300 B", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300 b", 1300.0, SizeUnit::Byte, "1300 B"));
-
- assert!(test("1.5KB", 1.5, SizeUnit::KByte, "1.5 KB"));
- assert!(test("1.5kb", 1.5, SizeUnit::KByte, "1.5 KB"));
- assert!(test("1.654321MB", 1.654_321, SizeUnit::MByte, "1.654 MB"));
-
- assert!(test("2.0GB", 2.0, SizeUnit::GByte, "2 GB"));
-
- assert!(test("1.4TB", 1.4, SizeUnit::TByte, "1.4 TB"));
- assert!(test("1.4tb", 1.4, SizeUnit::TByte, "1.4 TB"));
-
- assert!(test("2KiB", 2.0, SizeUnit::Kibi, "2 KiB"));
- assert!(test("2Ki", 2.0, SizeUnit::Kibi, "2 KiB"));
- assert!(test("2kib", 2.0, SizeUnit::Kibi, "2 KiB"));
-
- assert!(test("2.3454MiB", 2.3454, SizeUnit::Mebi, "2.345 MiB"));
- assert!(test("2.3456MiB", 2.3456, SizeUnit::Mebi, "2.346 MiB"));
-
- assert!(test("4gib", 4.0, SizeUnit::Gibi, "4 GiB"));
-
- Ok(())
-}
+ assert!(test("14", 14.0, SizeUnit::Byte, "14 B"));
+ assert!(test("14.4", 14.4, SizeUnit::Byte, "14.4 B"));
+ assert!(test("14.45", 14.45, SizeUnit::Byte, "14.45 B"));
+ assert!(test("14.456", 14.456, SizeUnit::Byte, "14.456 B"));
+ assert!(test("14.4567", 14.4567, SizeUnit::Byte, "14.457 B"));
+
+ let h: HumanByte = "1.2345678".parse()?;
+ assert_eq!(&format!("{:.0}", h), "1 B");
+ assert_eq!(&format!("{:.0}", h.as_f64()), "1"); // use as_f64 to get raw bytes without unit
+ assert_eq!(&format!("{:.1}", h), "1.2 B");
+ assert_eq!(&format!("{:.2}", h), "1.23 B");
+ assert_eq!(&format!("{:.3}", h), "1.235 B");
+ assert_eq!(&format!("{:.4}", h), "1.2346 B");
+ assert_eq!(&format!("{:.5}", h), "1.23457 B");
+ assert_eq!(&format!("{:.6}", h), "1.234568 B");
+ assert_eq!(&format!("{:.7}", h), "1.2345678 B");
+ assert_eq!(&format!("{:.8}", h), "1.2345678 B");
+
+ assert!(test(
+ "987654321",
+ 987654321.0,
+ SizeUnit::Byte,
+ "987654321 B"
+ ));
+
+ assert!(test("1300b", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300B", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300 B", 1300.0, SizeUnit::Byte, "1300 B"));
+ assert!(test("1300 b", 1300.0, SizeUnit::Byte, "1300 B"));
+
+ assert!(test("1.5KB", 1.5, SizeUnit::KByte, "1.5 KB"));
+ assert!(test("1.5kb", 1.5, SizeUnit::KByte, "1.5 KB"));
+ assert!(test("1.654321MB", 1.654_321, SizeUnit::MByte, "1.654 MB"));
+
+ assert!(test("2.0GB", 2.0, SizeUnit::GByte, "2 GB"));
+
+ assert!(test("1.4TB", 1.4, SizeUnit::TByte, "1.4 TB"));
+ assert!(test("1.4tb", 1.4, SizeUnit::TByte, "1.4 TB"));
+
+ assert!(test("2KiB", 2.0, SizeUnit::Kibi, "2 KiB"));
+ assert!(test("2Ki", 2.0, SizeUnit::Kibi, "2 KiB"));
+ assert!(test("2kib", 2.0, SizeUnit::Kibi, "2 KiB"));
+
+ assert!(test("2.3454MiB", 2.3454, SizeUnit::Mebi, "2.345 MiB"));
+ assert!(test("2.3456MiB", 2.3456, SizeUnit::Mebi, "2.346 MiB"));
+
+ assert!(test("4gib", 4.0, SizeUnit::Gibi, "4 GiB"));
-#[test]
-fn test_human_byte_auto_unit_decimal() {
- fn convert(b: u64) -> String {
- HumanByte::new_decimal(b as f64).to_string()
+ Ok(())
}
- assert_eq!(convert(987), "987 B");
- assert_eq!(convert(1022), "1.022 KB");
- assert_eq!(convert(9_000), "9 KB");
- assert_eq!(convert(1_000), "1 KB");
- assert_eq!(convert(1_000_000), "1 MB");
- assert_eq!(convert(1_000_000_000), "1 GB");
- assert_eq!(convert(1_000_000_000_000), "1 TB");
- assert_eq!(convert(1_000_000_000_000_000), "1 PB");
-
- assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.182 GB");
- assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.208 GB");
- assert_eq!(convert((2 << 50) + 500 * (1 << 40)), "2.802 PB");
-}
-#[test]
-fn test_human_byte_auto_unit_binary() {
- fn convert(b: u64) -> String {
- HumanByte::from(b).to_string()
+ #[test]
+ fn test_human_byte_auto_unit_decimal() {
+ fn convert(b: u64) -> String {
+ HumanByte::new_decimal(b as f64).to_string()
+ }
+ assert_eq!(convert(987), "987 B");
+ assert_eq!(convert(1022), "1.022 KB");
+ assert_eq!(convert(9_000), "9 KB");
+ assert_eq!(convert(1_000), "1 KB");
+ assert_eq!(convert(1_000_000), "1 MB");
+ assert_eq!(convert(1_000_000_000), "1 GB");
+ assert_eq!(convert(1_000_000_000_000), "1 TB");
+ assert_eq!(convert(1_000_000_000_000_000), "1 PB");
+
+ assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.182 GB");
+ assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.208 GB");
+ assert_eq!(convert((2 << 50) + 500 * (1 << 40)), "2.802 PB");
+ }
+
+ #[test]
+ fn test_human_byte_auto_unit_binary() {
+ fn convert(b: u64) -> String {
+ HumanByte::from(b).to_string()
+ }
+ assert_eq!(convert(0), "0 B");
+ assert_eq!(convert(987), "987 B");
+ assert_eq!(convert(1022), "1022 B");
+ assert_eq!(convert(9_000), "8.789 KiB");
+ assert_eq!(convert(10_000_000), "9.537 MiB");
+ assert_eq!(convert(10_000_000_000), "9.313 GiB");
+ assert_eq!(convert(10_000_000_000_000), "9.095 TiB");
+
+ assert_eq!(convert(1 << 10), "1 KiB");
+ assert_eq!(convert((1 << 10) * 10), "10 KiB");
+ assert_eq!(convert(1 << 20), "1 MiB");
+ assert_eq!(convert(1 << 30), "1 GiB");
+ assert_eq!(convert(1 << 40), "1 TiB");
+ assert_eq!(convert(1 << 50), "1 PiB");
+
+ assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.101 GiB");
+ assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.125 GiB");
+ assert_eq!(convert((1 << 40) + 128 * (1 << 30)), "1.125 TiB");
+ assert_eq!(convert((2 << 50) + 512 * (1 << 40)), "2.5 PiB");
}
- assert_eq!(convert(0), "0 B");
- assert_eq!(convert(987), "987 B");
- assert_eq!(convert(1022), "1022 B");
- assert_eq!(convert(9_000), "8.789 KiB");
- assert_eq!(convert(10_000_000), "9.537 MiB");
- assert_eq!(convert(10_000_000_000), "9.313 GiB");
- assert_eq!(convert(10_000_000_000_000), "9.095 TiB");
-
- assert_eq!(convert(1 << 10), "1 KiB");
- assert_eq!(convert((1 << 10) * 10), "10 KiB");
- assert_eq!(convert(1 << 20), "1 MiB");
- assert_eq!(convert(1 << 30), "1 GiB");
- assert_eq!(convert(1 << 40), "1 TiB");
- assert_eq!(convert(1 << 50), "1 PiB");
-
- assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.101 GiB");
- assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.125 GiB");
- assert_eq!(convert((1 << 40) + 128 * (1 << 30)), "1.125 TiB");
- assert_eq!(convert((2 << 50) + 512 * (1 << 40)), "2.5 PiB");
}
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread* [pbs-devel] [PATCH proxmox-backup 3/3] api-types: client: datastore: tools: use proxmox-human-bytes crate
2023-05-08 10:01 [pbs-devel] [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Lukas Wagner
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox 1/3] add `proxmox-human-byte` crate Lukas Wagner
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox 2/3] human-byte: move tests to their sub module Lukas Wagner
@ 2023-05-08 10:01 ` Lukas Wagner
2023-06-26 11:57 ` [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Wolfgang Bumiller
3 siblings, 0 replies; 5+ messages in thread
From: Lukas Wagner @ 2023-05-08 10:01 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
Cargo.toml | 2 +
pbs-api-types/Cargo.toml | 1 +
pbs-api-types/src/human_byte.rs | 358 ---------------------------
pbs-api-types/src/lib.rs | 3 -
pbs-api-types/src/traffic_control.rs | 4 +-
pbs-client/Cargo.toml | 1 +
pbs-client/src/backup_writer.rs | 4 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 3 +-
pbs-tools/Cargo.toml | 3 +-
pbs-tools/src/format.rs | 2 +-
proxmox-backup-client/Cargo.toml | 1 +
proxmox-backup-client/src/main.rs | 5 +-
src/api2/tape/restore.rs | 8 +-
src/bin/proxmox-tape.rs | 8 +-
src/bin/proxmox_backup_debug/diff.rs | 3 +-
src/server/email_notifications.rs | 3 +-
17 files changed, 31 insertions(+), 379 deletions(-)
delete mode 100644 pbs-api-types/src/human_byte.rs
diff --git a/Cargo.toml b/Cargo.toml
index ce0f7bec..933af600 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -59,6 +59,7 @@ proxmox-auth-api = "0.1"
proxmox-borrow = "1"
proxmox-compression = "0.1.1"
proxmox-fuse = "0.1.3"
+proxmox-human-byte = "0.1"
proxmox-http = { version = "0.8.0", features = [ "client", "http-helpers", "websocket" ] } # see below
proxmox-io = "1.0.1" # tools and client use "tokio" feature
proxmox-lang = "1.1"
@@ -205,6 +206,7 @@ proxmox-async.workspace = true
proxmox-auth-api = { workspace = true, features = [ "api", "pam-authenticator" ] }
proxmox-compression.workspace = true
proxmox-http = { workspace = true, features = [ "client-trait", "proxmox-async", "rate-limited-stream" ] } # pbs-client doesn't use these
+proxmox-human-byte.workspace = true
proxmox-io.workspace = true
proxmox-lang.workspace = true
proxmox-ldap.workspace = true
diff --git a/pbs-api-types/Cargo.toml b/pbs-api-types/Cargo.toml
index b0092813..31b69f62 100644
--- a/pbs-api-types/Cargo.toml
+++ b/pbs-api-types/Cargo.toml
@@ -15,6 +15,7 @@ serde.workspace = true
serde_plain.workspace = true
proxmox-auth-api = { workspace = true, features = [ "api-types" ] }
+proxmox-human-byte.workspace = true
proxmox-lang.workspace=true
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
proxmox-serde.workspace = true
diff --git a/pbs-api-types/src/human_byte.rs b/pbs-api-types/src/human_byte.rs
deleted file mode 100644
index 189a645c..00000000
--- a/pbs-api-types/src/human_byte.rs
+++ /dev/null
@@ -1,358 +0,0 @@
-use anyhow::{bail, Error};
-
-use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema, UpdaterType};
-
-/// Size units for byte sizes
-#[derive(Debug, Copy, Clone, PartialEq, Eq)]
-pub enum SizeUnit {
- Byte,
- // SI (base 10)
- KByte,
- MByte,
- GByte,
- TByte,
- PByte,
- // IEC (base 2)
- Kibi,
- Mebi,
- Gibi,
- Tebi,
- Pebi,
-}
-
-impl SizeUnit {
- /// Returns the scaling factor
- pub fn factor(&self) -> f64 {
- match self {
- SizeUnit::Byte => 1.0,
- // SI (base 10)
- SizeUnit::KByte => 1_000.0,
- SizeUnit::MByte => 1_000_000.0,
- SizeUnit::GByte => 1_000_000_000.0,
- SizeUnit::TByte => 1_000_000_000_000.0,
- SizeUnit::PByte => 1_000_000_000_000_000.0,
- // IEC (base 2)
- SizeUnit::Kibi => 1024.0,
- SizeUnit::Mebi => 1024.0 * 1024.0,
- SizeUnit::Gibi => 1024.0 * 1024.0 * 1024.0,
- SizeUnit::Tebi => 1024.0 * 1024.0 * 1024.0 * 1024.0,
- SizeUnit::Pebi => 1024.0 * 1024.0 * 1024.0 * 1024.0 * 1024.0,
- }
- }
-
- /// gets the biggest possible unit still having a value greater zero before the decimal point
- /// 'binary' specifies if IEC (base 2) units should be used or SI (base 10) ones
- pub fn auto_scale(size: f64, binary: bool) -> SizeUnit {
- if binary {
- let bits = 64 - (size as u64).leading_zeros();
- match bits {
- 51.. => SizeUnit::Pebi,
- 41..=50 => SizeUnit::Tebi,
- 31..=40 => SizeUnit::Gibi,
- 21..=30 => SizeUnit::Mebi,
- 11..=20 => SizeUnit::Kibi,
- _ => SizeUnit::Byte,
- }
- } else if size >= 1_000_000_000_000_000.0 {
- SizeUnit::PByte
- } else if size >= 1_000_000_000_000.0 {
- SizeUnit::TByte
- } else if size >= 1_000_000_000.0 {
- SizeUnit::GByte
- } else if size >= 1_000_000.0 {
- SizeUnit::MByte
- } else if size >= 1_000.0 {
- SizeUnit::KByte
- } else {
- SizeUnit::Byte
- }
- }
-}
-
-/// Returns the string representation
-impl std::fmt::Display for SizeUnit {
- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- match self {
- SizeUnit::Byte => write!(f, "B"),
- // SI (base 10)
- SizeUnit::KByte => write!(f, "KB"),
- SizeUnit::MByte => write!(f, "MB"),
- SizeUnit::GByte => write!(f, "GB"),
- SizeUnit::TByte => write!(f, "TB"),
- SizeUnit::PByte => write!(f, "PB"),
- // IEC (base 2)
- SizeUnit::Kibi => write!(f, "KiB"),
- SizeUnit::Mebi => write!(f, "MiB"),
- SizeUnit::Gibi => write!(f, "GiB"),
- SizeUnit::Tebi => write!(f, "TiB"),
- SizeUnit::Pebi => write!(f, "PiB"),
- }
- }
-}
-
-/// Strips a trailing SizeUnit inclusive trailing whitespace
-/// Supports both IEC and SI based scales, the B/b byte symbol is optional.
-fn strip_unit(v: &str) -> (&str, SizeUnit) {
- let v = v.strip_suffix(&['b', 'B'][..]).unwrap_or(v); // byte is implied anyway
-
- let (v, binary) = match v.strip_suffix('i') {
- Some(n) => (n, true),
- None => (v, false),
- };
-
- let mut unit = SizeUnit::Byte;
- #[rustfmt::skip]
- let value = v.strip_suffix(|c: char| match c {
- 'k' | 'K' if !binary => { unit = SizeUnit::KByte; true }
- 'm' | 'M' if !binary => { unit = SizeUnit::MByte; true }
- 'g' | 'G' if !binary => { unit = SizeUnit::GByte; true }
- 't' | 'T' if !binary => { unit = SizeUnit::TByte; true }
- 'p' | 'P' if !binary => { unit = SizeUnit::PByte; true }
- // binary (IEC recommended) variants
- 'k' | 'K' if binary => { unit = SizeUnit::Kibi; true }
- 'm' | 'M' if binary => { unit = SizeUnit::Mebi; true }
- 'g' | 'G' if binary => { unit = SizeUnit::Gibi; true }
- 't' | 'T' if binary => { unit = SizeUnit::Tebi; true }
- 'p' | 'P' if binary => { unit = SizeUnit::Pebi; true }
- _ => false
- }).unwrap_or(v).trim_end();
-
- (value, unit)
-}
-
-/// Byte size which can be displayed in a human friendly way
-#[derive(Debug, Copy, Clone, UpdaterType, PartialEq)]
-pub struct HumanByte {
- /// The siginficant value, it does not includes any factor of the `unit`
- size: f64,
- /// The scale/unit of the value
- unit: SizeUnit,
-}
-
-fn verify_human_byte(s: &str) -> Result<(), Error> {
- match s.parse::<HumanByte>() {
- Ok(_) => Ok(()),
- Err(err) => bail!("byte-size parse error for '{}': {}", s, err),
- }
-}
-impl ApiType for HumanByte {
- const API_SCHEMA: Schema = StringSchema::new(
- "Byte size with optional unit (B, KB (base 10), MB, GB, ..., KiB (base 2), MiB, Gib, ...).",
- )
- .format(&ApiStringFormat::VerifyFn(verify_human_byte))
- .min_length(1)
- .max_length(64)
- .schema();
-}
-
-impl HumanByte {
- /// Create instance with size and unit (size must be positive)
- pub fn with_unit(size: f64, unit: SizeUnit) -> Result<Self, Error> {
- if size < 0.0 {
- bail!("byte size may not be negative");
- }
- Ok(HumanByte { size, unit })
- }
-
- /// Create a new instance with optimal binary unit computed
- pub fn new_binary(size: f64) -> Self {
- let unit = SizeUnit::auto_scale(size, true);
- HumanByte {
- size: size / unit.factor(),
- unit,
- }
- }
-
- /// Create a new instance with optimal decimal unit computed
- pub fn new_decimal(size: f64) -> Self {
- let unit = SizeUnit::auto_scale(size, false);
- HumanByte {
- size: size / unit.factor(),
- unit,
- }
- }
-
- /// Returns the size as u64 number of bytes
- pub fn as_u64(&self) -> u64 {
- self.as_f64() as u64
- }
-
- /// Returns the size as f64 number of bytes
- pub fn as_f64(&self) -> f64 {
- self.size * self.unit.factor()
- }
-
- /// Returns a copy with optimal binary unit computed
- pub fn auto_scale_binary(self) -> Self {
- HumanByte::new_binary(self.as_f64())
- }
-
- /// Returns a copy with optimal decimal unit computed
- pub fn auto_scale_decimal(self) -> Self {
- HumanByte::new_decimal(self.as_f64())
- }
-}
-
-impl From<u64> for HumanByte {
- fn from(v: u64) -> Self {
- HumanByte::new_binary(v as f64)
- }
-}
-impl From<usize> for HumanByte {
- fn from(v: usize) -> Self {
- HumanByte::new_binary(v as f64)
- }
-}
-
-impl std::fmt::Display for HumanByte {
- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- let precision = f.precision().unwrap_or(3) as f64;
- let precision_factor = 1.0 * 10.0_f64.powf(precision);
- // this could cause loss of information, rust has sadly no shortest-max-X flt2dec fmt yet
- let size = ((self.size * precision_factor).round()) / precision_factor;
- write!(f, "{} {}", size, self.unit)
- }
-}
-
-impl std::str::FromStr for HumanByte {
- type Err = Error;
-
- fn from_str(v: &str) -> Result<Self, Error> {
- let (v, unit) = strip_unit(v);
- HumanByte::with_unit(v.parse()?, unit)
- }
-}
-
-proxmox_serde::forward_deserialize_to_from_str!(HumanByte);
-proxmox_serde::forward_serialize_to_display!(HumanByte);
-
-#[test]
-fn test_human_byte_parser() -> Result<(), Error> {
- assert!("-10".parse::<HumanByte>().is_err()); // negative size
-
- fn do_test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> Result<(), Error> {
- let h: HumanByte = v.parse()?;
-
- if h.size != size {
- bail!("got unexpected size for '{}' ({} != {})", v, h.size, size);
- }
- if h.unit != unit {
- bail!(
- "got unexpected unit for '{}' ({:?} != {:?})",
- v,
- h.unit,
- unit
- );
- }
-
- let new = h.to_string();
- if new != *as_str {
- bail!("to_string failed for '{}' ({:?} != {:?})", v, new, as_str);
- }
- Ok(())
- }
- fn test(v: &str, size: f64, unit: SizeUnit, as_str: &str) -> bool {
- match do_test(v, size, unit, as_str) {
- Ok(_) => true,
- Err(err) => {
- eprintln!("{}", err); // makes debugging easier
- false
- }
- }
- }
-
- assert!(test("14", 14.0, SizeUnit::Byte, "14 B"));
- assert!(test("14.4", 14.4, SizeUnit::Byte, "14.4 B"));
- assert!(test("14.45", 14.45, SizeUnit::Byte, "14.45 B"));
- assert!(test("14.456", 14.456, SizeUnit::Byte, "14.456 B"));
- assert!(test("14.4567", 14.4567, SizeUnit::Byte, "14.457 B"));
-
- let h: HumanByte = "1.2345678".parse()?;
- assert_eq!(&format!("{:.0}", h), "1 B");
- assert_eq!(&format!("{:.0}", h.as_f64()), "1"); // use as_f64 to get raw bytes without unit
- assert_eq!(&format!("{:.1}", h), "1.2 B");
- assert_eq!(&format!("{:.2}", h), "1.23 B");
- assert_eq!(&format!("{:.3}", h), "1.235 B");
- assert_eq!(&format!("{:.4}", h), "1.2346 B");
- assert_eq!(&format!("{:.5}", h), "1.23457 B");
- assert_eq!(&format!("{:.6}", h), "1.234568 B");
- assert_eq!(&format!("{:.7}", h), "1.2345678 B");
- assert_eq!(&format!("{:.8}", h), "1.2345678 B");
-
- assert!(test(
- "987654321",
- 987654321.0,
- SizeUnit::Byte,
- "987654321 B"
- ));
-
- assert!(test("1300b", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300B", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300 B", 1300.0, SizeUnit::Byte, "1300 B"));
- assert!(test("1300 b", 1300.0, SizeUnit::Byte, "1300 B"));
-
- assert!(test("1.5KB", 1.5, SizeUnit::KByte, "1.5 KB"));
- assert!(test("1.5kb", 1.5, SizeUnit::KByte, "1.5 KB"));
- assert!(test("1.654321MB", 1.654_321, SizeUnit::MByte, "1.654 MB"));
-
- assert!(test("2.0GB", 2.0, SizeUnit::GByte, "2 GB"));
-
- assert!(test("1.4TB", 1.4, SizeUnit::TByte, "1.4 TB"));
- assert!(test("1.4tb", 1.4, SizeUnit::TByte, "1.4 TB"));
-
- assert!(test("2KiB", 2.0, SizeUnit::Kibi, "2 KiB"));
- assert!(test("2Ki", 2.0, SizeUnit::Kibi, "2 KiB"));
- assert!(test("2kib", 2.0, SizeUnit::Kibi, "2 KiB"));
-
- assert!(test("2.3454MiB", 2.3454, SizeUnit::Mebi, "2.345 MiB"));
- assert!(test("2.3456MiB", 2.3456, SizeUnit::Mebi, "2.346 MiB"));
-
- assert!(test("4gib", 4.0, SizeUnit::Gibi, "4 GiB"));
-
- Ok(())
-}
-
-#[test]
-fn test_human_byte_auto_unit_decimal() {
- fn convert(b: u64) -> String {
- HumanByte::new_decimal(b as f64).to_string()
- }
- assert_eq!(convert(987), "987 B");
- assert_eq!(convert(1022), "1.022 KB");
- assert_eq!(convert(9_000), "9 KB");
- assert_eq!(convert(1_000), "1 KB");
- assert_eq!(convert(1_000_000), "1 MB");
- assert_eq!(convert(1_000_000_000), "1 GB");
- assert_eq!(convert(1_000_000_000_000), "1 TB");
- assert_eq!(convert(1_000_000_000_000_000), "1 PB");
-
- assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.182 GB");
- assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.208 GB");
- assert_eq!(convert((2 << 50) + 500 * (1 << 40)), "2.802 PB");
-}
-
-#[test]
-fn test_human_byte_auto_unit_binary() {
- fn convert(b: u64) -> String {
- HumanByte::from(b).to_string()
- }
- assert_eq!(convert(0), "0 B");
- assert_eq!(convert(987), "987 B");
- assert_eq!(convert(1022), "1022 B");
- assert_eq!(convert(9_000), "8.789 KiB");
- assert_eq!(convert(10_000_000), "9.537 MiB");
- assert_eq!(convert(10_000_000_000), "9.313 GiB");
- assert_eq!(convert(10_000_000_000_000), "9.095 TiB");
-
- assert_eq!(convert(1 << 10), "1 KiB");
- assert_eq!(convert((1 << 10) * 10), "10 KiB");
- assert_eq!(convert(1 << 20), "1 MiB");
- assert_eq!(convert(1 << 30), "1 GiB");
- assert_eq!(convert(1 << 40), "1 TiB");
- assert_eq!(convert(1 << 50), "1 PiB");
-
- assert_eq!(convert((1 << 30) + 103 * (1 << 20)), "1.101 GiB");
- assert_eq!(convert((1 << 30) + 128 * (1 << 20)), "1.125 GiB");
- assert_eq!(convert((1 << 40) + 128 * (1 << 30)), "1.125 TiB");
- assert_eq!(convert((2 << 50) + 512 * (1 << 40)), "2.5 PiB");
-}
diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index 2a5c1932..4764c51a 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -72,9 +72,6 @@ pub use acl::*;
mod datastore;
pub use datastore::*;
-mod human_byte;
-pub use human_byte::HumanByte;
-
mod jobs;
pub use jobs::*;
diff --git a/pbs-api-types/src/traffic_control.rs b/pbs-api-types/src/traffic_control.rs
index 947df38a..24195e44 100644
--- a/pbs-api-types/src/traffic_control.rs
+++ b/pbs-api-types/src/traffic_control.rs
@@ -1,10 +1,10 @@
use serde::{Deserialize, Serialize};
+use proxmox_human_byte::HumanByte;
use proxmox_schema::{api, IntegerSchema, Schema, StringSchema, Updater};
use crate::{
- HumanByte, CIDR_SCHEMA, DAILY_DURATION_FORMAT, PROXMOX_SAFE_ID_FORMAT,
- SINGLE_LINE_COMMENT_SCHEMA,
+ CIDR_SCHEMA, DAILY_DURATION_FORMAT, PROXMOX_SAFE_ID_FORMAT, SINGLE_LINE_COMMENT_SCHEMA,
};
pub const TRAFFIC_CONTROL_TIMEFRAME_SCHEMA: Schema =
diff --git a/pbs-client/Cargo.toml b/pbs-client/Cargo.toml
index 2b16150c..ed7d651d 100644
--- a/pbs-client/Cargo.toml
+++ b/pbs-client/Cargo.toml
@@ -37,6 +37,7 @@ proxmox-async.workspace = true
proxmox-auth-api.workspace = true
proxmox-compression.workspace = true
proxmox-http = { workspace = true, features = [ "rate-limiter" ] }
+proxmox-human-byte.workspace = true
proxmox-io = { workspace = true, features = [ "tokio" ] }
proxmox-lang.workspace = true
proxmox-router = { workspace = true, features = [ "cli", "server" ] }
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index be6da2a6..15927a17 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -12,7 +12,7 @@ use tokio::io::AsyncReadExt;
use tokio::sync::{mpsc, oneshot};
use tokio_stream::wrappers::ReceiverStream;
-use pbs_api_types::{BackupDir, BackupNamespace, HumanByte};
+use pbs_api_types::{BackupDir, BackupNamespace};
use pbs_datastore::data_blob::{ChunkInfo, DataBlob, DataChunkBuilder};
use pbs_datastore::dynamic_index::DynamicIndexReader;
use pbs_datastore::fixed_index::FixedIndexReader;
@@ -21,6 +21,8 @@ use pbs_datastore::manifest::{ArchiveType, BackupManifest, MANIFEST_BLOB_NAME};
use pbs_datastore::{CATALOG_NAME, PROXMOX_BACKUP_PROTOCOL_ID_V1};
use pbs_tools::crypt_config::CryptConfig;
+use proxmox_human_byte::HumanByte;
+
use super::merge_known_chunks::{MergeKnownChunks, MergedChunkInfo};
use super::{H2Client, HttpClient};
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index a6f7b771..b793dc5b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -28,6 +28,7 @@ pxar.workspace = true
proxmox-borrow.workspace = true
proxmox-io.workspace = true
+proxmox-human-byte.workspace = true
proxmox-lang.workspace=true
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index bf951305..8edd0ba6 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -9,6 +9,7 @@ use anyhow::{bail, format_err, Error};
use lazy_static::lazy_static;
use nix::unistd::{unlinkat, UnlinkatFlags};
+use proxmox_human_byte::HumanByte;
use proxmox_schema::ApiType;
use proxmox_sys::error::SysError;
@@ -20,7 +21,7 @@ use proxmox_sys::{task_log, task_warn};
use pbs_api_types::{
Authid, BackupNamespace, BackupType, ChunkOrder, DataStoreConfig, DatastoreFSyncLevel,
- DatastoreTuning, GarbageCollectionStatus, HumanByte, Operation, UPID,
+ DatastoreTuning, GarbageCollectionStatus, Operation, UPID,
};
use crate::backup_info::{BackupDir, BackupGroup};
diff --git a/pbs-tools/Cargo.toml b/pbs-tools/Cargo.toml
index 61132734..3dcae88a 100644
--- a/pbs-tools/Cargo.toml
+++ b/pbs-tools/Cargo.toml
@@ -31,7 +31,8 @@ walkdir.workspace = true
zstd.workspace = true
proxmox-async.workspace = true
-proxmox-io= { workspace = true, features = [ "tokio" ] }
+proxmox-io = { workspace = true, features = [ "tokio" ] }
+proxmox-human-byte.workspace = true
proxmox-lang.workspace=true
proxmox-sys.workspace = true
proxmox-time.workspace = true
diff --git a/pbs-tools/src/format.rs b/pbs-tools/src/format.rs
index 038362a0..9f915d6c 100644
--- a/pbs-tools/src/format.rs
+++ b/pbs-tools/src/format.rs
@@ -3,7 +3,7 @@ use std::borrow::Borrow;
use anyhow::Error;
use serde_json::Value;
-use pbs_api_types::HumanByte;
+use proxmox_human_byte::HumanByte;
pub fn strip_server_file_extension(name: &str) -> &str {
if name.ends_with(".didx") || name.ends_with(".fidx") || name.ends_with(".blob") {
diff --git a/proxmox-backup-client/Cargo.toml b/proxmox-backup-client/Cargo.toml
index 89a47111..40de2450 100644
--- a/proxmox-backup-client/Cargo.toml
+++ b/proxmox-backup-client/Cargo.toml
@@ -25,6 +25,7 @@ pxar.workspace = true
proxmox-async.workspace = true
proxmox-fuse.workspace = true
+proxmox-human-byte.workspace = true
proxmox-io.workspace = true
proxmox-router = { workspace = true, features = [ "cli" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 55198108..a4dff5c8 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -15,6 +15,7 @@ use xdg::BaseDirectories;
use pathpatterns::{MatchEntry, MatchType, PatternFlag};
use proxmox_async::blocking::TokioWriterAdapter;
+use proxmox_human_byte::HumanByte;
use proxmox_io::StdChannelWriter;
use proxmox_router::{cli::*, ApiMethod, RpcEnvironment};
use proxmox_schema::api;
@@ -24,8 +25,8 @@ use pxar::accessor::{MaybeReady, ReadAt, ReadAtOperation};
use pbs_api_types::{
Authid, BackupDir, BackupGroup, BackupNamespace, BackupPart, BackupType, CryptMode,
- Fingerprint, GroupListItem, HumanByte, PruneJobOptions, PruneListItem, RateLimitConfig,
- SnapshotListItem, StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
+ Fingerprint, GroupListItem, PruneJobOptions, PruneListItem, RateLimitConfig, SnapshotListItem,
+ StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
BACKUP_TYPE_SCHEMA, TRAFFIC_CONTROL_BURST_SCHEMA, TRAFFIC_CONTROL_RATE_SCHEMA,
};
use pbs_client::catalog_shell::Shell;
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 43d34473..ffec86a5 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -7,6 +7,7 @@ use std::sync::Arc;
use anyhow::{bail, format_err, Error};
use serde_json::Value;
+use proxmox_human_byte::HumanByte;
use proxmox_io::ReadExt;
use proxmox_router::{Permission, Router, RpcEnvironment, RpcEnvironmentType};
use proxmox_schema::{api, ApiType};
@@ -17,10 +18,9 @@ use proxmox_uuid::Uuid;
use pbs_api_types::{
parse_ns_and_snapshot, print_ns_and_snapshot, Authid, BackupDir, BackupNamespace, CryptMode,
- HumanByte, Operation, TapeRestoreNamespace, Userid, DATASTORE_MAP_ARRAY_SCHEMA,
- DATASTORE_MAP_LIST_SCHEMA, DRIVE_NAME_SCHEMA, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_BACKUP,
- PRIV_DATASTORE_MODIFY, PRIV_TAPE_READ, TAPE_RESTORE_NAMESPACE_SCHEMA,
- TAPE_RESTORE_SNAPSHOT_SCHEMA, UPID_SCHEMA,
+ Operation, TapeRestoreNamespace, Userid, DATASTORE_MAP_ARRAY_SCHEMA, DATASTORE_MAP_LIST_SCHEMA,
+ DRIVE_NAME_SCHEMA, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
+ PRIV_TAPE_READ, TAPE_RESTORE_NAMESPACE_SCHEMA, TAPE_RESTORE_SNAPSHOT_SCHEMA, UPID_SCHEMA,
};
use pbs_config::CachedUserInfo;
use pbs_datastore::dynamic_index::DynamicIndexReader;
diff --git a/src/bin/proxmox-tape.rs b/src/bin/proxmox-tape.rs
index 59254540..83793c34 100644
--- a/src/bin/proxmox-tape.rs
+++ b/src/bin/proxmox-tape.rs
@@ -3,6 +3,7 @@ use std::collections::HashMap;
use anyhow::{bail, format_err, Error};
use serde_json::{json, Value};
+use proxmox_human_byte::HumanByte;
use proxmox_io::ReadExt;
use proxmox_router::cli::*;
use proxmox_router::RpcEnvironment;
@@ -18,10 +19,9 @@ use pbs_config::drive::complete_drive_name;
use pbs_config::media_pool::complete_pool_name;
use pbs_api_types::{
- Authid, BackupNamespace, GroupListItem, HumanByte, Userid, DATASTORE_MAP_LIST_SCHEMA,
- DATASTORE_SCHEMA, DRIVE_NAME_SCHEMA, GROUP_FILTER_LIST_SCHEMA, MEDIA_LABEL_SCHEMA,
- MEDIA_POOL_NAME_SCHEMA, NS_MAX_DEPTH_SCHEMA, TAPE_RESTORE_NAMESPACE_SCHEMA,
- TAPE_RESTORE_SNAPSHOT_SCHEMA,
+ Authid, BackupNamespace, GroupListItem, Userid, DATASTORE_MAP_LIST_SCHEMA, DATASTORE_SCHEMA,
+ DRIVE_NAME_SCHEMA, GROUP_FILTER_LIST_SCHEMA, MEDIA_LABEL_SCHEMA, MEDIA_POOL_NAME_SCHEMA,
+ NS_MAX_DEPTH_SCHEMA, TAPE_RESTORE_NAMESPACE_SCHEMA, TAPE_RESTORE_SNAPSHOT_SCHEMA,
};
use pbs_tape::{BlockReadError, MediaContentHeader, PROXMOX_BACKUP_CONTENT_HEADER_MAGIC_1_0};
diff --git a/src/bin/proxmox_backup_debug/diff.rs b/src/bin/proxmox_backup_debug/diff.rs
index 288d35ce..9924fb7b 100644
--- a/src/bin/proxmox_backup_debug/diff.rs
+++ b/src/bin/proxmox_backup_debug/diff.rs
@@ -9,10 +9,11 @@ use anyhow::{bail, Context as AnyhowContext, Error};
use futures::future::BoxFuture;
use futures::FutureExt;
+use proxmox_human_byte::HumanByte;
use proxmox_router::cli::{CliCommand, CliCommandMap, CommandLineInterface};
use proxmox_schema::api;
-use pbs_api_types::{BackupNamespace, BackupPart, HumanByte};
+use pbs_api_types::{BackupNamespace, BackupPart};
use pbs_client::tools::key_source::{
crypto_parameters, format_key_source, get_encryption_key_password, KEYFD_SCHEMA,
};
diff --git a/src/server/email_notifications.rs b/src/server/email_notifications.rs
index 1224191c..7ef67cdb 100644
--- a/src/server/email_notifications.rs
+++ b/src/server/email_notifications.rs
@@ -5,12 +5,13 @@ use handlebars::{
Context, Handlebars, Helper, HelperResult, Output, RenderContext, RenderError, TemplateError,
};
+use proxmox_human_byte::HumanByte;
use proxmox_lang::try_block;
use proxmox_schema::ApiType;
use proxmox_sys::email::sendmail;
use pbs_api_types::{
- APTUpdateInfo, DataStoreConfig, DatastoreNotify, GarbageCollectionStatus, HumanByte, Notify,
+ APTUpdateInfo, DataStoreConfig, DatastoreNotify, GarbageCollectionStatus, Notify,
SyncJobConfig, TapeBackupJobSetup, User, Userid, VerificationJobConfig,
};
--
2.30.2
^ permalink raw reply [flat|nested] 5+ messages in thread* [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate
2023-05-08 10:01 [pbs-devel] [PATCH proxmox{, -backup} 0/3] move implementation `human-byte` module to its own crate Lukas Wagner
` (2 preceding siblings ...)
2023-05-08 10:01 ` [pbs-devel] [PATCH proxmox-backup 3/3] api-types: client: datastore: tools: use proxmox-human-bytes crate Lukas Wagner
@ 2023-06-26 11:57 ` Wolfgang Bumiller
3 siblings, 0 replies; 5+ messages in thread
From: Wolfgang Bumiller @ 2023-06-26 11:57 UTC (permalink / raw)
To: Lukas Wagner; +Cc: pbs-devel
applied series, thanks
^ permalink raw reply [flat|nested] 5+ messages in thread