[pbs-devel] [PATCH backup v5 1/3] fix #4077: Estimated Full metric on ext4 file systems

[pbs-devel] [PATCH backup v5 1/3] fix #4077: Estimated Full metric on ext4 file systems

[Daniel Tschlatscher]

The rrd data now includes tracking the available field in disk usage.
The calculation for the estimated_time_full was adapted to use the
total for the unpriviliged user, which is the sum of used + available.

The total for unprivileged users is preferable, because datastores are
always written to by the backup user. Which means that any storage
space reserved for root is unusable for our purposes.

To avoid resetting the estimate when switching to this new version,
the backend will try to use the available value to calculate the
unprivileged total. When that is not an option, it will fall back to
using the absolute total.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
---
Changes from v4:
* Revised the implementation of calculating the estimated_full_date
  field. The new implementation now tries to calculate the total with
  the new rrd field 'available' + 'used', and if that fails, falls
  back to using the 'total' that was used up until now.

 src/api2/status.rs              | 49 ++++++++++++++++-----------------
 src/bin/proxmox-backup-proxy.rs |  2 ++
 2 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/src/api2/status.rs b/src/api2/status.rs
index 50d401c2..87450d63 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -86,42 +86,39 @@ pub async fn datastore_status(
 
         let total_res = get_rrd("total")?;
         let used_res = get_rrd("used")?;
+        let avail_res = get_rrd("available")?;
 
-        if let (
-            Some(proxmox_rrd::Entry {
-                start,
-                resolution,
-                data: total_list,
-            }),
-            Some(proxmox_rrd::Entry {
-                data: used_list, ..
-            }),
-        ) = (total_res, used_res)
-        {
+        if let Some(((total_entry, used), avail)) = total_res.zip(used_res).zip(avail_res) {
             let mut usage_list: Vec<f64> = Vec::new();
             let mut time_list: Vec<u64> = Vec::new();
             let mut history = Vec::new();
 
-            for (idx, used) in used_list.iter().enumerate() {
-                let total = if idx < total_list.len() {
-                    total_list[idx]
+            for (idx, used) in used.data.iter().enumerate() {
+                let used = match used {
+                    Some(used) => used,
+                    _ => {
+                        history.push(None);
+                        continue;
+                    }
+                };
+
+                let total = if idx < avail.data.len() && avail.data[idx].is_some() {
+                    avail.data[idx].unwrap() + used
+                } else if idx < total_entry.data.len() && total_entry.data[idx].is_some() {
+                    total_entry.data[idx].unwrap()
                 } else {
-                    None
+                    history.push(None);
+                    continue;
                 };
 
-                match (total, used) {
-                    (Some(total), Some(used)) if total != 0.0 => {
-                        time_list.push(start + (idx as u64) * resolution);
-                        let usage = used / total;
-                        usage_list.push(usage);
-                        history.push(Some(usage));
-                    }
-                    _ => history.push(None),
-                }
+                let usage = used / total;
+                time_list.push(total_entry.start + (idx as u64) * total_entry.resolution);
+                usage_list.push(usage);
+                history.push(Some(usage));
             }
 
-            entry.history_start = Some(start);
-            entry.history_delta = Some(resolution);
+            entry.history_start = Some(total_entry.start);
+            entry.history_delta = Some(total_entry.resolution);
             entry.history = Some(history);
 
             // we skip the calculation for datastores with not enough data
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 69863481..ec2a8744 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -1255,6 +1255,8 @@ fn rrd_update_disk_stat(disk: &DiskStat, rrd_prefix: &str) {
         rrd_update_gauge(&rrd_key, status.total as f64);
         let rrd_key = format!("{}/used", rrd_prefix);
         rrd_update_gauge(&rrd_key, status.used as f64);
+        let rrd_key = format!("{}/available", rrd_prefix);
+        rrd_update_gauge(&rrd_key, status.available as f64);
     }
 
     if let Some(stat) = &disk.dev {
-- 
2.30.2

[pbs-devel] [PATCH backup v5 2/3] 'available' field in rrd data in the API and change usage of 'total'

[Daniel Tschlatscher]

The API now exposes the field 'available' as well, with which the
unprivileged total is calculated in all corresponsing views in the
frontend.
The rrd charts now also display the total as the unprivileged total
if available, otherwise the absolute total is used.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
---
Changes from v4:
* The API now no longer uses the 'unpriv-total' field but instead
  exposes the field 'available as well'. Subsequently, the calculation
  for the unprivileged total is now made in the GUI.
* The Usage display in the datastore summary now also displays the
  total for the unprivileged user
* RRD charts now also use the unprivileged total

 src/api2/admin/datastore.rs           |  1 +
 www/dashboard/DataStoreStatistics.js  |  7 ++++++-
 www/datastore/DataStoreListSummary.js |  5 +++--
 www/datastore/Summary.js              | 18 ++++++++++++++++--
 4 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index e66e1c0b..c8e86b07 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -1804,6 +1804,7 @@ pub fn get_rrd_stats(
 
     let mut rrd_fields = vec![
         "total",
+        "available",
         "used",
         "read_ios",
         "read_bytes",
diff --git a/www/dashboard/DataStoreStatistics.js b/www/dashboard/DataStoreStatistics.js
index 38f7a2fe..0510fc7d 100644
--- a/www/dashboard/DataStoreStatistics.js
+++ b/www/dashboard/DataStoreStatistics.js
@@ -3,7 +3,12 @@ Ext.define('pbs-datastore-statistics', {
 
     fields: [
 	'store',
-	'total',
+	{
+	    name: 'total',
+	    calculate: function(data) {
+		return data.avail + data.used;
+	    },
+	},
 	'used',
 	'avail',
 	'estimated-full-date',
diff --git a/www/datastore/DataStoreListSummary.js b/www/datastore/DataStoreListSummary.js
index c7b67d56..bec0562d 100644
--- a/www/datastore/DataStoreListSummary.js
+++ b/www/datastore/DataStoreListSummary.js
@@ -52,10 +52,11 @@ Ext.define('PBS.datastore.DataStoreListSummary', {
 	    vm.set('maintenance', '');
 	}
 
-	let usage = statusData.used/statusData.total;
+	let total = statusData.avail + statusData.used;
+	let usage = statusData.used / total;
 	let usagetext = Ext.String.format(gettext('{0} of {1}'),
 	    Proxmox.Utils.format_size(statusData.used, true),
-	    Proxmox.Utils.format_size(statusData.total, true),
+	    Proxmox.Utils.format_size(total, true),
 	);
 
 	let usagePanel = me.lookup('usage');
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index 94be9559..d67e81cc 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -3,6 +3,20 @@ Ext.define('pve-rrd-datastore', {
     fields: [
 	'used',
 	'total',
+	{
+	    name: 'unpriv-total', // Can't resuse 'total' here as that creates a stack overflow
+	    calculate: function(data) {
+		let used = data.used;
+		let avail = data.available;
+
+		if (avail && used) {
+		    return avail + used;
+		}
+
+		return data.total;
+	    },
+	},
+	'available',
 	'read_ios',
 	'read_bytes',
 	'write_ios',
@@ -66,8 +80,8 @@ Ext.define('PBS.DataStoreInfo', {
 	    let vm = me.getViewModel();
 
 	    let counts = store.getById('counts').data.value;
-	    let total = store.getById('total').data.value;
 	    let used = store.getById('used').data.value;
+	    let total = store.getById('avail').data.value + used;
 
 	    let usage = Proxmox.Utils.render_size_usage(used, total, true);
 	    vm.set('usagetext', usage);
@@ -236,7 +250,7 @@ Ext.define('PBS.DataStoreSummary', {
 	{
 	    xtype: 'proxmoxRRDChart',
 	    title: gettext('Storage usage (bytes)'),
-	    fields: ['total', 'used'],
+	    fields: ['unpriv-total', 'used'],
 	    fieldTitles: [gettext('Total'), gettext('Storage usage')],
 	},
 	{
-- 
2.30.2

[pbs-devel] [PATCH backup v5 3/3] gui: change reporting of the estimated_time_full to "Full" if no space

[Daniel Tschlatscher]

is left in the datastore. Before, the GUI would report "Never" for the
estimated time full, because the value provided in the backend was in
the past. To get around this, the GUI now reports "Full" if the value
for available reaches 0.

Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
---
No changes from v4

 www/Utils.js                          | 6 +++++-
 www/datastore/DataStoreListSummary.js | 3 ++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/www/Utils.js b/www/Utils.js
index ad451c9f..f6d353ef 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -285,7 +285,11 @@ Ext.define('PBS.Utils', {
 	return tokenid.match(/^(.+)!([^!]+)$/)[2];
     },
 
-    render_estimate: function(value) {
+    render_estimate: function(value, metaData, record) {
+	if (record.data.avail === 0) {
+	    return gettext("Full");
+	}
+
 	if (value === undefined) {
 	    return gettext('Not enough data');
 	}
diff --git a/www/datastore/DataStoreListSummary.js b/www/datastore/DataStoreListSummary.js
index bec0562d..968239b0 100644
--- a/www/datastore/DataStoreListSummary.js
+++ b/www/datastore/DataStoreListSummary.js
@@ -62,7 +62,8 @@ Ext.define('PBS.datastore.DataStoreListSummary', {
 	let usagePanel = me.lookup('usage');
 	usagePanel.updateValue(usage, usagetext);
 
-	let estimate = PBS.Utils.render_estimate(statusData['estimated-full-date']);
+	let estimate = PBS.Utils.render_estimate(statusData['estimated-full-date'], null, { data: statusData });
+
 	vm.set('full', estimate);
 	vm.set('deduplication', PBS.Utils.calculate_dedup_factor(statusData['gc-status']).toFixed(2));
 	vm.set('stillbad', statusData['gc-status']['still-bad']);
-- 
2.30.2

[pbs-devel] applied-series: [PATCH backup v5 1/3] fix #4077: Estimated Full metric on ext4 file systems

[Wolfgang Bumiller]

applied series, thanks