* [pbs-devel] [PATCH proxmox-backup v4 1/2] close #4723: ui: new gc view
@ 2023-09-27 13:06 Gabriel Goller
2023-09-27 13:06 ` [pbs-devel] [PATCH proxmox-backup v4 2/2] close #4723: api: added endpoint for gc status Gabriel Goller
0 siblings, 1 reply; 3+ messages in thread
From: Gabriel Goller @ 2023-09-27 13:06 UTC (permalink / raw)
To: pbs-devel
Moved GCView to different file and enhanced it with last,
next run, status and duration. Added button to show task
log.
Changed `render_task_status()` to also take in account upids stored in
other 'rows'.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
---
update v4:
- separate commits for ui/api
- added 'show task log' button
update v3:
- ui: removed `required` attribute on items to get the sorting right
- made `pending_chunks` and `removed_chunks` options, so that they
are not shown when no gc run exists
update v2:
- skip serializing if value is `None`
- return just the schedule if `upid` doesn't exist (means no gc has been
run)
- ui: removed default values on timestamps
- ui: removed flex and minHeight properties
www/Makefile | 1 +
www/Utils.js | 12 ++--
www/config/GCView.js | 129 ++++++++++++++++++++++++++++++++++++
www/datastore/PruneAndGC.js | 91 +------------------------
4 files changed, 138 insertions(+), 95 deletions(-)
create mode 100644 www/config/GCView.js
diff --git a/www/Makefile b/www/Makefile
index 04c12b31..b67fd73f 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -62,6 +62,7 @@ JSSRC= \
config/SyncView.js \
config/VerifyView.js \
config/PruneView.js \
+ config/GCView.js \
config/WebauthnView.js \
config/CertificateView.js \
config/NodeOptionView.js \
diff --git a/www/Utils.js b/www/Utils.js
index 2eca600e..66293b17 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -199,14 +199,14 @@ Ext.define('PBS.Utils', {
return fingerprint.substring(0, 23);
},
- render_task_status: function(value, metadata, record) {
- if (!record.data['last-run-upid']) {
- return '-';
+ render_task_status: function(value, metadata, record, rowIndex, colIndex, store) {
+ if (!record.data['last-run-upid'] && !store.getById('last-run-upid')?.data.value) {
+ return '-';
}
- if (!record.data['last-run-endtime']) {
- metadata.tdCls = 'x-grid-row-loading';
- return '';
+ if (!record.data['last-run-endtime'] && !store.getById('last-run-endtime')?.data.value) {
+ metadata.tdCls = 'x-grid-row-loading';
+ return '';
}
let parsed = Proxmox.Utils.parse_task_status(value);
diff --git a/www/config/GCView.js b/www/config/GCView.js
new file mode 100644
index 00000000..b8f8812e
--- /dev/null
+++ b/www/config/GCView.js
@@ -0,0 +1,129 @@
+Ext.define('PBS.Datastore.GCOptions', {
+ extend: 'Proxmox.grid.ObjectGrid',
+ alias: 'widget.pbsGCJobView',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ onlineHelp: 'maintenance_pruning',
+
+ cbindData: function(initial) {
+ let me = this;
+
+ me.datastore = encodeURIComponent(me.datastore);
+ me.url = `/api2/json/admin/datastore/${me.datastore}/gc_info`;
+ me.editorConfig = {
+ url: `/api2/extjs/config/datastore/${me.datastore}`,
+ };
+ return {};
+ },
+
+ controller: {
+ xclass: 'Ext.app.ViewController',
+
+ edit: function() {
+ this.getView().run_editor();
+ },
+
+ garbageCollect: function() {
+ let me = this;
+ let view = me.getView();
+ Proxmox.Utils.API2Request({
+ url: `/admin/datastore/${view.datastore}/gc`,
+ method: 'POST',
+ failure: function(response) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, options) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ }).show();
+ },
+ });
+ },
+
+ showTaskLog: function() {
+ let me = this;
+
+ let upid = me.getView().getStore().getById('last-run-upid')?.data.value;
+ if (!upid) return;
+
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid,
+ }).show();
+ },
+ },
+
+ tbar: [
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Edit'),
+ enableFn: (rec) => rec.id === 'schedule',
+ disabled: true,
+ handler: 'edit',
+ },
+ '-',
+ {
+ xtype: 'proxmoxButton',
+ selModel: null,
+ text: gettext('Start Garbage Collection'),
+ handler: 'garbageCollect',
+ },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Show Task Log'),
+ enableFn: (rec) => !!rec.store.getById('last-run-upid')?.data.value,
+ disabled: true,
+ handler: 'showTaskLog',
+ },
+ ],
+
+ listeners: {
+ activate: function() { this.rstore.startUpdate(); },
+ destroy: function() { this.rstore.stopUpdate(); },
+ deactivate: function() { this.rstore.stopUpdate(); },
+ itemdblclick: 'edit',
+ },
+
+ rows: {
+ "schedule": {
+ required: true,
+ defaultValue: Proxmox.Utils.NoneText,
+ header: gettext('Schedule'),
+ editor: {
+ xtype: 'proxmoxWindowEdit',
+ title: gettext('GC Schedule'),
+ onlineHelp: 'maintenance_gc',
+ items: {
+ xtype: 'pbsCalendarEvent',
+ name: 'gc-schedule',
+ fieldLabel: gettext("GC Schedule"),
+ emptyText: Proxmox.Utils.NoneText,
+ deleteEmpty: true,
+ },
+ },
+ },
+ "last-run-state": {
+ required: true,
+ header: gettext('State'),
+ renderer: PBS.Utils.render_task_status,
+ },
+ "duration": {
+ header: gettext('Duration'),
+ renderer: Proxmox.Utils.render_duration,
+ },
+ "last-run-endtime": {
+ header: gettext('Last Run'),
+ renderer: PBS.Utils.render_optional_timestamp,
+ },
+ "next-run": {
+ header: gettext('Next Run'),
+ renderer: PBS.Utils.render_next_task_run,
+ },
+ "pending-chunks": {
+ header: gettext('Pending Chunks'),
+ },
+ "removed-chunks": {
+ header: gettext('Removed Chunks'),
+ },
+ "last-run-upid": { visible: false },
+ },
+});
diff --git a/www/datastore/PruneAndGC.js b/www/datastore/PruneAndGC.js
index aab98dad..3b9878a3 100644
--- a/www/datastore/PruneAndGC.js
+++ b/www/datastore/PruneAndGC.js
@@ -1,88 +1,3 @@
-Ext.define('PBS.Datastore.GCOptions', {
- extend: 'Proxmox.grid.ObjectGrid',
- alias: 'widget.pbsDatastoreGCOpts',
- mixins: ['Proxmox.Mixin.CBind'],
-
- onlineHelp: 'maintenance_pruning',
-
- cbindData: function(initial) {
- let me = this;
-
- me.datastore = encodeURIComponent(me.datastore);
- me.url = `/api2/json/config/datastore/${me.datastore}`;
- me.editorConfig = {
- url: `/api2/extjs/config/datastore/${me.datastore}`,
- };
- return {};
- },
-
- controller: {
- xclass: 'Ext.app.ViewController',
-
- edit: function() { this.getView().run_editor(); },
-
- garbageCollect: function() {
- let me = this;
- let view = me.getView();
- Proxmox.Utils.API2Request({
- url: `/admin/datastore/${view.datastore}/gc`,
- method: 'POST',
- failure: function(response) {
- Ext.Msg.alert(gettext('Error'), response.htmlStatus);
- },
- success: function(response, options) {
- Ext.create('Proxmox.window.TaskViewer', {
- upid: response.result.data,
- }).show();
- },
- });
- },
- },
-
- tbar: [
- {
- xtype: 'proxmoxButton',
- text: gettext('Edit'),
- disabled: true,
- handler: 'edit',
- },
- '-',
- {
- xtype: 'proxmoxButton',
- text: gettext('Start Garbage Collection'),
- selModel: null,
- handler: 'garbageCollect',
- },
- ],
-
- listeners: {
- activate: function() { this.rstore.startUpdate(); },
- destroy: function() { this.rstore.stopUpdate(); },
- deactivate: function() { this.rstore.stopUpdate(); },
- itemdblclick: 'edit',
- },
-
- rows: {
- "gc-schedule": {
- required: true,
- defaultValue: Proxmox.Utils.NoneText,
- header: gettext('Garbage Collection Schedule'),
- editor: {
- xtype: 'proxmoxWindowEdit',
- title: gettext('GC Schedule'),
- onlineHelp: 'maintenance_gc',
- items: {
- xtype: 'pbsCalendarEvent',
- name: 'gc-schedule',
- fieldLabel: gettext("GC Schedule"),
- emptyText: Proxmox.Utils.noneText,
- deleteEmpty: true,
- },
- },
- },
- },
-});
-
Ext.define('PBS.Datastore.PruneAndGC', {
extend: 'Ext.panel.Panel',
alias: 'widget.pbsDatastorePruneAndGC',
@@ -99,9 +14,9 @@ Ext.define('PBS.Datastore.PruneAndGC', {
},
items: [
{
- xtype: 'pbsDatastoreGCOpts',
+ xtype: 'pbsGCJobView',
title: gettext('Garbage Collection'),
- itemId: 'datastore-gc',
+ itemId: 'datastore-gc-jobs',
nodename: 'localhost',
cbind: {
datastore: '{datastore}',
@@ -111,8 +26,6 @@ Ext.define('PBS.Datastore.PruneAndGC', {
xtype: 'pbsPruneJobView',
nodename: 'localhost',
itemId: 'datastore-prune-jobs',
- flex: 1,
- minHeight: 200,
cbind: {
datastore: '{datastore}',
},
--
2.39.2
^ permalink raw reply [flat|nested] 3+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 2/2] close #4723: api: added endpoint for gc status
2023-09-27 13:06 [pbs-devel] [PATCH proxmox-backup v4 1/2] close #4723: ui: new gc view Gabriel Goller
@ 2023-09-27 13:06 ` Gabriel Goller
2023-09-27 15:19 ` Thomas Lamprecht
0 siblings, 1 reply; 3+ messages in thread
From: Gabriel Goller @ 2023-09-27 13:06 UTC (permalink / raw)
To: pbs-devel
Returns general info on the garbage collection such as:
- Schedule
- State (of last run)
- Duration (of last run)
- Last Run
- Next Run
- Pending Chunks (of last run)
- Removed Chunks (of last run)
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
---
update v4:
- separate commits for ui/api
- cleaned up code
- show 'next scheduled run', when no gc has ever been run
update v3:
- ui: removed `required` attribute on items to get the sorting right
- made `pending_chunks` and `removed_chunks` options, so that they
are not shown when no gc run exists
update v2:
- skip serializing if value is `None`
- return just the schedule if `upid` doesn't exist (means no gc has been
run)
- ui: removed default values on timestamps
- ui: removed flex and minHeight properties
pbs-api-types/src/datastore.rs | 38 +++++++++
src/api2/admin/datastore.rs | 138 +++++++++++++++++++++++++++++++--
2 files changed, 168 insertions(+), 8 deletions(-)
mode change 100644 => 100755 src/api2/admin/datastore.rs
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 73c4890e..55caa963 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1250,6 +1250,44 @@ pub struct GarbageCollectionStatus {
pub still_bad: usize,
}
+#[api(
+ properties: {
+ "last-run-upid": {
+ optional: true,
+ type: UPID,
+ },
+ },
+)]
+#[derive(Clone, Default, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Garbage Collection general info
+pub struct GarbageCollectionInfo {
+ /// upid of the last run gc job
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub last_run_upid: Option<String>,
+ /// Number of removed chunks
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub removed_chunks: Option<usize>,
+ /// Number of pending chunks
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub pending_chunks: Option<usize>,
+ /// Schedule of the gc job
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub schedule: Option<String>,
+ /// Time of the next gc run
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub next_run: Option<i64>,
+ /// Endtime of the last gc run
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub last_run_endtime: Option<i64>,
+ /// State of the last gc run
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub last_run_state: Option<String>,
+ /// Duration of last gc run
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub duration: Option<i64>,
+}
+
#[api(
properties: {
"gc-status": {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
old mode 100644
new mode 100755
index a95031e7..8a86c233
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -10,6 +10,7 @@ use anyhow::{bail, format_err, Error};
use futures::*;
use hyper::http::request::Parts;
use hyper::{header, Body, Response, StatusCode};
+use proxmox_time::CalendarEvent;
use serde::Deserialize;
use serde_json::{json, Value};
use tokio_stream::wrappers::ReceiverStream;
@@ -33,13 +34,14 @@ use pxar::EntryKind;
use pbs_api_types::{
print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
- Counts, CryptMode, DataStoreListItem, DataStoreStatus, GarbageCollectionStatus, GroupListItem,
- KeepOptions, Operation, PruneJobOptions, RRDMode, RRDTimeFrame, SnapshotListItem,
- SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
- BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA,
- MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
- PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY,
- UPID_SCHEMA, VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus, GarbageCollectionInfo,
+ GarbageCollectionStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
+ PruneJobOptions, RRDMode, RRDTimeFrame, SnapshotListItem, SnapshotVerifyState,
+ BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
+ BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
+ NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
+ PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
+ VERIFICATION_OUTDATED_AFTER_SCHEMA,
};
use pbs_client::pxar::{create_tar, create_zip};
use pbs_config::CachedUserInfo;
@@ -67,7 +69,7 @@ use crate::backup::{
ListAccessibleBackupGroups, NS_PRIVS_OK,
};
-use crate::server::jobstate::Job;
+use crate::server::jobstate::{compute_schedule_status, Job, JobState};
const GROUP_NOTES_FILE_NAME: &str = "notes";
@@ -1199,6 +1201,122 @@ pub fn garbage_collection_status(
Ok(status)
}
+#[api(
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ },
+ },
+ returns: {
+ type: GarbageCollectionInfo,
+ },
+ access: {
+ permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
+ },
+)]
+/// Garbage collection status.
+pub fn garbage_collection_info(
+ store: String,
+ _info: &ApiMethod,
+ _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<GarbageCollectionInfo, Error> {
+ let (config, _) = pbs_config::datastore::config()?;
+ let store_config: DataStoreConfig = config.lookup("datastore", &store)?;
+
+ let mut info = GarbageCollectionInfo {
+ schedule: store_config.gc_schedule,
+ ..Default::default()
+ };
+
+ let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+ let status_in_memory = datastore.last_gc_status();
+ let state_file = JobState::load("garbage_collection", &store)
+ .map_err(|err| {
+ log::error!(
+ "could not open statefile for {:?}: {}",
+ info.last_run_upid,
+ err
+ )
+ })
+ .ok();
+
+ let mut selected_upid = None;
+ if status_in_memory.upid.is_some() {
+ selected_upid = status_in_memory.upid;
+ } else if let Some(JobState::Finished { upid, .. }) = &state_file {
+ selected_upid = Some(upid.to_owned());
+ }
+
+ info.last_run_upid = selected_upid.clone();
+
+ match selected_upid {
+ Some(upid) => {
+ info.removed_chunks = Some(status_in_memory.removed_chunks);
+ info.pending_chunks = Some(status_in_memory.pending_chunks);
+
+ let mut computed_schedule: JobScheduleStatus = JobScheduleStatus::default();
+ let mut duration = None;
+ if let Some(state) = state_file {
+ if let Ok(cs) = compute_schedule_status(&state, info.last_run_upid.as_deref()) {
+ computed_schedule = cs;
+ }
+ }
+
+ if let Some(endtime) = computed_schedule.last_run_endtime {
+ computed_schedule.next_run = info
+ .schedule
+ .as_ref()
+ .and_then(|s| {
+ s.parse::<CalendarEvent>()
+ .map_err(|err| log::error!("{err}"))
+ .ok()
+ })
+ .and_then(|e| {
+ e.compute_next_event(endtime)
+ .map_err(|err| log::error!("{err}"))
+ .ok()
+ })
+ .and_then(|ne| ne);
+
+ if let Ok(parsed_upid) = upid.parse::<UPID>() {
+ duration = Some(endtime - parsed_upid.starttime);
+ }
+ }
+
+ info.next_run = computed_schedule.next_run;
+ info.last_run_endtime = computed_schedule.last_run_endtime;
+ info.last_run_state = computed_schedule.last_run_state;
+ info.duration = duration;
+ }
+ None => {
+ if let Some(schedule) = &info.schedule {
+ info.next_run = schedule
+ .parse::<CalendarEvent>()
+ .map_err(|err| log::error!("{err}"))
+ .ok()
+ .and_then(|e| {
+ e.compute_next_event(proxmox_time::epoch_i64())
+ .map_err(|err| log::error!("{err}"))
+ .ok()
+ })
+ .and_then(|ne| ne);
+
+ if let Ok(event) = schedule.parse::<CalendarEvent>() {
+ if let Ok(next_event) = event.compute_next_event(proxmox_time::epoch_i64()) {
+ info.next_run = next_event;
+ }
+ }
+ } else {
+ return Ok(info);
+ }
+ }
+ }
+
+ Ok(info)
+}
+
#[api(
returns: {
description: "List the accessible datastores.",
@@ -2265,6 +2383,10 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
.get(&API_METHOD_GARBAGE_COLLECTION_STATUS)
.post(&API_METHOD_START_GARBAGE_COLLECTION),
),
+ (
+ "gc_info",
+ &Router::new().get(&API_METHOD_GARBAGE_COLLECTION_INFO),
+ ),
(
"group-notes",
&Router::new()
--
2.39.2
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v4 2/2] close #4723: api: added endpoint for gc status
2023-09-27 13:06 ` [pbs-devel] [PATCH proxmox-backup v4 2/2] close #4723: api: added endpoint for gc status Gabriel Goller
@ 2023-09-27 15:19 ` Thomas Lamprecht
0 siblings, 0 replies; 3+ messages in thread
From: Thomas Lamprecht @ 2023-09-27 15:19 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Gabriel Goller
Am 27/09/2023 um 15:06 schrieb Gabriel Goller:
> update v4:
> - separate commits for ui/api
No thorough review, I'm afraid, but two things I noticed when skimming
this:
1. Please order patches such that the dependency order is correct.
Here you added the UI using a new API endpoint before adding said
endpoint.
It's easier to review, and less error-prone, e.g., if there are patches
that seem like they could be applied already, while some other parts of
such a patch series might still need rework.
I hope I (or maybe someone else) can give this a more thorough review.
If the order and the endpoint name (see below) will stay the only two
issues, there's no hard need for a v3, I can also fix those up on
applying.
> - cleaned up code
> - show 'next scheduled run', when no gc has ever been run
> @@ -2265,6 +2383,10 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> .get(&API_METHOD_GARBAGE_COLLECTION_STATUS)
> .post(&API_METHOD_START_GARBAGE_COLLECTION),
> ),
> + (
> + "gc_info",
2. Please use kepbab-case, not snake_case, for things part of the public
API, like the URL here (but also parameters, which you already handle
correctly).
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-09-27 15:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-27 13:06 [pbs-devel] [PATCH proxmox-backup v4 1/2] close #4723: ui: new gc view Gabriel Goller
2023-09-27 13:06 ` [pbs-devel] [PATCH proxmox-backup v4 2/2] close #4723: api: added endpoint for gc status Gabriel Goller
2023-09-27 15:19 ` Thomas Lamprecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal