[pbs-devel] [PATCH proxmox-backup 00/11] add 'prune all' button to datastore content

[pbs-devel] [PATCH proxmox-backup 00/11] add 'prune all' button to datastore content

[Dominik Csapak]

similar to the 'verify all' button, it makes sense that a user can
prune all groups (which he has access to) on demand with custom setings.

this adds a new api call, since pruning all groups could take a while,
depending on the number of groups snapshots, and it does not make sense
to have that in a synchronous api call (that we already have per group),
so modifying the existing api call to have the group optional was not
really sensible IMHO.

patches 1-6 are simply refactoring, to make it more easy to use
 the PruneOptions, 'prune_datastore', etc. later
7,8 add new parameter/functionality to the 'prune_datastore' method
9 really adds the api call
10 is the button in the gui
11 is just a ui improvement for the other prune window
 (could be applied seperately)

Dominik Csapak (11):
  api-types: move PRUNE_SCHEMA_KEEP_* to pbs-api-types
  pbs-datastore/prune: make PruneOptions an api type
  client: simplify prune api method
  api: admin/datastore: simplify prune api call
  backup/datastore: refactor check_backup_owner there
  server/prune_job: factor out 'prune_datastore'
  server/prune_job: add 'keep_all' logic to 'prune_datastore'
  server/prune_job: add proper permission checks to 'prune_datastore'
  api: admin/datastore: add new 'prune-datastore' api call
  ui: datastore/Content: add 'Prune All' button
  ui: datastore/Prune: improve title of group prune window

 pbs-api-types/src/lib.rs         |  30 +++++
 pbs-datastore/src/prune.rs       |  50 +++++++-
 src/api2/admin/datastore.rs      | 208 ++++++++++++++++---------------
 src/api2/types/mod.rs            |  30 -----
 src/backup/datastore.rs          |  20 +++
 src/bin/proxmox-backup-client.rs |  95 +++++++-------
 src/server/prune_job.rs          | 139 +++++++++++++--------
 www/datastore/Content.js         |  34 +++++
 www/datastore/Prune.js           |   2 +-
 www/window/DataStoreEdit.js      |  15 +++
 10 files changed, 390 insertions(+), 233 deletions(-)

-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 01/11] api-types: move PRUNE_SCHEMA_KEEP_* to pbs-api-types

[Dominik Csapak]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pbs-api-types/src/lib.rs | 30 ++++++++++++++++++++++++++++++
 src/api2/types/mod.rs    | 30 ------------------------------
 2 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index f9fe5cfb..c07699d1 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -146,6 +146,36 @@ pub const CERT_FINGERPRINT_SHA256_SCHEMA: Schema =
         .format(&FINGERPRINT_SHA256_FORMAT)
         .schema();
 
+pub const PRUNE_SCHEMA_KEEP_DAILY: Schema = IntegerSchema::new(
+    "Number of daily backups to keep.")
+    .minimum(1)
+    .schema();
+
+pub const PRUNE_SCHEMA_KEEP_HOURLY: Schema = IntegerSchema::new(
+    "Number of hourly backups to keep.")
+    .minimum(1)
+    .schema();
+
+pub const PRUNE_SCHEMA_KEEP_LAST: Schema = IntegerSchema::new(
+    "Number of backups to keep.")
+    .minimum(1)
+    .schema();
+
+pub const PRUNE_SCHEMA_KEEP_MONTHLY: Schema = IntegerSchema::new(
+    "Number of monthly backups to keep.")
+    .minimum(1)
+    .schema();
+
+pub const PRUNE_SCHEMA_KEEP_WEEKLY: Schema = IntegerSchema::new(
+    "Number of weekly backups to keep.")
+    .minimum(1)
+    .schema();
+
+pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema = IntegerSchema::new(
+    "Number of yearly backups to keep.")
+    .minimum(1)
+    .schema();
+
 pub const PROXMOX_SAFE_ID_FORMAT: ApiStringFormat =
     ApiStringFormat::Pattern(&PROXMOX_SAFE_ID_REGEX);
 
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index d3c16b96..b8b42a5e 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -357,36 +357,6 @@ pub const REALM_ID_SCHEMA: Schema = StringSchema::new("Realm name.")
 
 // Complex type definitions
 
-pub const PRUNE_SCHEMA_KEEP_DAILY: Schema = IntegerSchema::new(
-    "Number of daily backups to keep.")
-    .minimum(1)
-    .schema();
-
-pub const PRUNE_SCHEMA_KEEP_HOURLY: Schema = IntegerSchema::new(
-    "Number of hourly backups to keep.")
-    .minimum(1)
-    .schema();
-
-pub const PRUNE_SCHEMA_KEEP_LAST: Schema = IntegerSchema::new(
-    "Number of backups to keep.")
-    .minimum(1)
-    .schema();
-
-pub const PRUNE_SCHEMA_KEEP_MONTHLY: Schema = IntegerSchema::new(
-    "Number of monthly backups to keep.")
-    .minimum(1)
-    .schema();
-
-pub const PRUNE_SCHEMA_KEEP_WEEKLY: Schema = IntegerSchema::new(
-    "Number of weekly backups to keep.")
-    .minimum(1)
-    .schema();
-
-pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema = IntegerSchema::new(
-    "Number of yearly backups to keep.")
-    .minimum(1)
-    .schema();
-
 #[api()]
 #[derive(Default, Serialize, Deserialize)]
 /// Storage space usage information.
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 02/11] pbs-datastore/prune: make PruneOptions an api type

[Dominik Csapak]

so that we can reuse it from here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pbs-datastore/src/prune.rs | 50 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 49 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/prune.rs b/pbs-datastore/src/prune.rs
index d0d8ca16..4605e26f 100644
--- a/pbs-datastore/src/prune.rs
+++ b/pbs-datastore/src/prune.rs
@@ -2,6 +2,18 @@ use std::collections::{HashMap, HashSet};
 use std::path::PathBuf;
 
 use anyhow::{Error};
+use serde::{Deserialize, Serialize};
+
+use proxmox::api::api;
+
+use pbs_api_types::{
+    PRUNE_SCHEMA_KEEP_LAST,
+    PRUNE_SCHEMA_KEEP_HOURLY,
+    PRUNE_SCHEMA_KEEP_DAILY,
+    PRUNE_SCHEMA_KEEP_WEEKLY,
+    PRUNE_SCHEMA_KEEP_MONTHLY,
+    PRUNE_SCHEMA_KEEP_YEARLY,
+};
 
 use super::BackupInfo;
 
@@ -68,13 +80,49 @@ fn remove_incomplete_snapshots(
     }
 }
 
-#[derive(Default)]
+#[api(
+    properties: {
+        "keep-last": {
+            schema: PRUNE_SCHEMA_KEEP_LAST,
+            optional: true,
+        },
+        "keep-hourly": {
+            schema: PRUNE_SCHEMA_KEEP_HOURLY,
+            optional: true,
+        },
+        "keep-daily": {
+            schema: PRUNE_SCHEMA_KEEP_DAILY,
+            optional: true,
+        },
+        "keep-weekly": {
+            schema: PRUNE_SCHEMA_KEEP_WEEKLY,
+            optional: true,
+        },
+        "keep-monthly": {
+            schema: PRUNE_SCHEMA_KEEP_MONTHLY,
+            optional: true,
+        },
+        "keep-yearly": {
+            schema: PRUNE_SCHEMA_KEEP_YEARLY,
+            optional: true,
+        },
+    }
+)]
+#[derive(Serialize, Deserialize, Default)]
+#[serde(rename_all = "kebab-case")]
+/// Common pruning options
 pub struct PruneOptions {
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_last: Option<u64>,
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_hourly: Option<u64>,
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_daily: Option<u64>,
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_weekly: Option<u64>,
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_monthly: Option<u64>,
+    #[serde(skip_serializing_if="Option::is_none")]
     pub keep_yearly: Option<u64>,
 }
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 03/11] client: simplify prune api method

[Dominik Csapak]

by using the api macro on the async method and reusing the PruneOptions
from pbs-datastore with 'flatten: true'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/bin/proxmox-backup-client.rs | 95 ++++++++++++++++----------------
 1 file changed, 49 insertions(+), 46 deletions(-)

diff --git a/src/bin/proxmox-backup-client.rs b/src/bin/proxmox-backup-client.rs
index b110763e..a5810538 100644
--- a/src/bin/proxmox-backup-client.rs
+++ b/src/bin/proxmox-backup-client.rs
@@ -6,7 +6,6 @@ use std::sync::{Arc, Mutex};
 use std::task::Context;
 
 use anyhow::{bail, format_err, Error};
-use futures::future::FutureExt;
 use futures::stream::{StreamExt, TryStreamExt};
 use serde_json::{json, Value};
 use tokio::sync::mpsc;
@@ -21,10 +20,8 @@ use proxmox::{
     },
     api::{
         api,
-        ApiHandler,
         ApiMethod,
         RpcEnvironment,
-        schema::*,
         cli::*,
     },
 };
@@ -65,6 +62,7 @@ use proxmox_backup::backup::{
     IndexFile,
     MANIFEST_BLOB_NAME,
     Shell,
+    PruneOptions,
 };
 
 mod proxmox_backup_client;
@@ -1225,60 +1223,65 @@ async fn restore(param: Value) -> Result<Value, Error> {
     Ok(Value::Null)
 }
 
-const API_METHOD_PRUNE: ApiMethod = ApiMethod::new(
-    &ApiHandler::Async(&prune),
-    &ObjectSchema::new(
-        "Prune a backup repository.",
-        &proxmox_backup::add_common_prune_prameters!([
-            ("dry-run", true, &BooleanSchema::new(
-                "Just show what prune would do, but do not delete anything.")
-             .schema()),
-            ("group", false, &StringSchema::new("Backup group.").schema()),
-        ], [
-            ("output-format", true, &OUTPUT_FORMAT),
-            (
-                "quiet",
-                true,
-                &BooleanSchema::new("Minimal output - only show removals.")
-                    .schema()
-            ),
-            ("repository", true, &REPO_URL_SCHEMA),
-        ])
-    )
-);
-
-fn prune<'a>(
-    param: Value,
-    _info: &ApiMethod,
-    _rpcenv: &'a mut dyn RpcEnvironment,
-) -> proxmox::api::ApiFuture<'a> {
-    async move {
-        prune_async(param).await
-    }.boxed()
-}
-
-async fn prune_async(mut param: Value) -> Result<Value, Error> {
+#[api(
+    input: {
+        properties: {
+            "dry-run": {
+                type: bool,
+                optional: true,
+                description: "Just show what prune would do, but do not delete anything.",
+            },
+            group: {
+                type: String,
+                description: "Backup group",
+            },
+            "prune-options": {
+                type: PruneOptions,
+                flatten: true,
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+            quiet: {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Minimal output - only show removals.",
+            },
+            repository: {
+                schema: REPO_URL_SCHEMA,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Prune a backup repository.
+async fn prune(
+    dry_run: Option<bool>,
+    group: String,
+    prune_options: PruneOptions,
+    quiet: bool,
+    mut param: Value
+) -> Result<Value, Error> {
     let repo = extract_repository_from_value(&param)?;
 
     let mut client = connect(&repo)?;
 
     let path = format!("api2/json/admin/datastore/{}/prune", repo.store());
 
-    let group = tools::required_string_param(&param, "group")?;
     let group: BackupGroup = group.parse()?;
 
     let output_format = extract_output_format(&mut param);
 
-    let quiet = param["quiet"].as_bool().unwrap_or(false);
-
-    param.as_object_mut().unwrap().remove("repository");
-    param.as_object_mut().unwrap().remove("group");
-    param.as_object_mut().unwrap().remove("quiet");
-
-    param["backup-type"] = group.backup_type().into();
-    param["backup-id"] = group.backup_id().into();
+    let mut api_param = serde_json::to_value(prune_options)?;
+    if let Some(dry_run) = dry_run {
+        api_param["dry-run"] = dry_run.into();
+    }
+    api_param["backup-type"] = group.backup_type().into();
+    api_param["backup-id"] = group.backup_id().into();
 
-    let mut result = client.post(&path, Some(param)).await?;
+    let mut result = client.post(&path, Some(api_param)).await?;
 
     record_repository(&repo);
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 04/11] api: admin/datastore: simplify prune api call

[Dominik Csapak]

by using the api macro and reusing the PruneOptions from pbs-datastore

this means we can now drop the 'add_common_prune_prameters' macro

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/admin/datastore.rs | 135 ++++++++++++------------------------
 1 file changed, 45 insertions(+), 90 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 184def5a..0bf6a86b 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -16,7 +16,7 @@ use proxmox::api::{
     api, ApiResponseFuture, ApiHandler, ApiMethod, Router,
     RpcEnvironment, RpcEnvironmentType, Permission
 };
-use proxmox::api::router::{ReturnType, SubdirMap};
+use proxmox::api::router::SubdirMap;
 use proxmox::api::schema::*;
 use proxmox::tools::fs::{
     file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
@@ -797,106 +797,61 @@ pub fn verify(
     Ok(json!(upid_str))
 }
 
-#[macro_export]
-macro_rules! add_common_prune_prameters {
-    ( [ $( $list1:tt )* ] ) => {
-        add_common_prune_prameters!([$( $list1 )* ] ,  [])
-    };
-    ( [ $( $list1:tt )* ] ,  [ $( $list2:tt )* ] ) => {
-        [
-            $( $list1 )*
-            (
-                "keep-daily",
-                true,
-                &PRUNE_SCHEMA_KEEP_DAILY,
-            ),
-            (
-                "keep-hourly",
-                true,
-                &PRUNE_SCHEMA_KEEP_HOURLY,
-            ),
-            (
-                "keep-last",
-                true,
-                &PRUNE_SCHEMA_KEEP_LAST,
-            ),
-            (
-                "keep-monthly",
-                true,
-                &PRUNE_SCHEMA_KEEP_MONTHLY,
-            ),
-            (
-                "keep-weekly",
-                true,
-                &PRUNE_SCHEMA_KEEP_WEEKLY,
-            ),
-            (
-                "keep-yearly",
-                true,
-                &PRUNE_SCHEMA_KEEP_YEARLY,
-            ),
-            $( $list2 )*
-        ]
-    }
-}
-
-pub const API_RETURN_SCHEMA_PRUNE: Schema = ArraySchema::new(
-    "Returns the list of snapshots and a flag indicating if there are kept or removed.",
-    &PruneListItem::API_SCHEMA
-).schema();
-
-pub const API_METHOD_PRUNE: ApiMethod = ApiMethod::new(
-    &ApiHandler::Sync(&prune),
-    &ObjectSchema::new(
-        "Prune the datastore.",
-        &add_common_prune_prameters!([
-            ("backup-id", false, &BACKUP_ID_SCHEMA),
-            ("backup-type", false, &BACKUP_TYPE_SCHEMA),
-            ("dry-run", true, &BooleanSchema::new(
-                "Just show what prune would do, but do not delete anything.")
-             .schema()
-            ),
-        ],[
-            ("store", false, &DATASTORE_SCHEMA),
-        ])
-    ))
-    .returns(ReturnType::new(false, &API_RETURN_SCHEMA_PRUNE))
-    .access(None, &Permission::Privilege(
-    &["datastore", "{store}"],
-    PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_PRUNE,
-    true)
-);
-
+#[api(
+    input: {
+        properties: {
+            "backup-id": {
+                schema: BACKUP_ID_SCHEMA,
+            },
+            "backup-type": {
+                schema: BACKUP_TYPE_SCHEMA,
+            },
+            "dry-run": {
+                optional: true,
+                type: bool,
+                default: false,
+                description: "Just show what prune would do, but do not delete anything.",
+            },
+            "prune-options": {
+                type: PruneOptions,
+                flatten: true,
+            },
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+        },
+    },
+    returns: {
+        type: Array,
+        description: "Returns the list of snapshots and a flag indicating if there are kept or removed.",
+        items: {
+            type: PruneListItem,
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_PRUNE, true),
+    },
+)]
+/// Prune the datastore
 pub fn prune(
-    param: Value,
-    _info: &ApiMethod,
+    backup_id: String,
+    backup_type: String,
+    dry_run: bool,
+    prune_options: PruneOptions,
+    store: String,
+    _param: Value,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
 
-    let store = tools::required_string_param(&param, "store")?;
-    let backup_type = tools::required_string_param(&param, "backup-type")?;
-    let backup_id = tools::required_string_param(&param, "backup-id")?;
-
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
 
-    let dry_run = param["dry-run"].as_bool().unwrap_or(false);
-
-    let group = BackupGroup::new(backup_type, backup_id);
+    let group = BackupGroup::new(&backup_type, &backup_id);
 
     let datastore = DataStore::lookup_datastore(&store)?;
 
     check_priv_or_backup_owner(&datastore, &group, &auth_id, PRIV_DATASTORE_MODIFY)?;
 
-    let prune_options = PruneOptions {
-        keep_last: param["keep-last"].as_u64(),
-        keep_hourly: param["keep-hourly"].as_u64(),
-        keep_daily: param["keep-daily"].as_u64(),
-        keep_weekly: param["keep-weekly"].as_u64(),
-        keep_monthly: param["keep-monthly"].as_u64(),
-        keep_yearly: param["keep-yearly"].as_u64(),
-    };
-
-    let worker_id = format!("{}:{}/{}", store, backup_type, backup_id);
+    let worker_id = format!("{}:{}/{}", store, &backup_type, &backup_id);
 
     let mut prune_result = Vec::new();
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 05/11] backup/datastore: refactor check_backup_owner there

[Dominik Csapak]

and add a 'owns_backup' convenience function

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/admin/datastore.rs | 12 ------------
 src/backup/datastore.rs     | 20 ++++++++++++++++++++
 2 files changed, 20 insertions(+), 12 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 0bf6a86b..79ab97e7 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -74,18 +74,6 @@ fn check_priv_or_backup_owner(
     Ok(())
 }
 
-fn check_backup_owner(
-    owner: &Authid,
-    auth_id: &Authid,
-) -> Result<(), Error> {
-    let correct_owner = owner == auth_id
-        || (owner.is_token() && &Authid::from(owner.user().clone()) == auth_id);
-    if !correct_owner {
-        bail!("backup owner check failed ({} != {})", auth_id, owner);
-    }
-    Ok(())
-}
-
 fn read_backup_index(
     store: &DataStore,
     backup_dir: &BackupDir,
diff --git a/src/backup/datastore.rs b/src/backup/datastore.rs
index d47c412b..29700846 100644
--- a/src/backup/datastore.rs
+++ b/src/backup/datastore.rs
@@ -37,6 +37,20 @@ lazy_static! {
     static ref DATASTORE_MAP: Mutex<HashMap<String, Arc<DataStore>>> = Mutex::new(HashMap::new());
 }
 
+/// checks if auth_id is owner, or, if owner is a token, if
+/// auth_id is the user of the token
+pub fn check_backup_owner(
+    owner: &Authid,
+    auth_id: &Authid,
+) -> Result<(), Error> {
+    let correct_owner = owner == auth_id
+        || (owner.is_token() && &Authid::from(owner.user().clone()) == auth_id);
+    if !correct_owner {
+        bail!("backup owner check failed ({} != {})", auth_id, owner);
+    }
+    Ok(())
+}
+
 /// Datastore Management
 ///
 /// A Datastore can store severals backups, and provides the
@@ -338,6 +352,12 @@ impl DataStore {
         Ok(owner.trim_end().parse()?) // remove trailing newline
     }
 
+    pub fn owns_backup(&self, backup_group: &BackupGroup, auth_id: &Authid) -> Result<bool, Error> {
+        let owner = self.get_owner(backup_group)?;
+
+        Ok(check_backup_owner(owner, auth_id).is_ok())
+    }
+
     /// Set the backup owner.
     pub fn set_owner(
         &self,
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 06/11] server/prune_job: factor out 'prune_datastore'

[Dominik Csapak]

we want to use that outside of a prune job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/server/prune_job.rs | 114 +++++++++++++++++++++-------------------
 1 file changed, 61 insertions(+), 53 deletions(-)

diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 248068ea..7ea42fcb 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -1,6 +1,6 @@
-use anyhow::Error;
+use std::sync::Arc;
 
-use proxmox::try_block;
+use anyhow::Error;
 
 use pbs_datastore::{task_log, task_warn};
 
@@ -11,6 +11,61 @@ use crate::{
     server::WorkerTask,
 };
 
+pub fn prune_datastore(
+    worker: Arc<WorkerTask>,
+    prune_options: PruneOptions,
+    store: &str,
+    datastore: Arc<DataStore>,
+) -> Result<(), Error> {
+    task_log!(worker, "Starting datastore prune on store \"{}\"", store);
+
+    task_log!(
+        worker,
+        "retention options: {}",
+        prune_options.cli_options_string()
+    );
+
+    let base_path = datastore.base_path();
+
+    let groups = BackupInfo::list_backup_groups(&base_path)?;
+    for group in groups {
+        let list = group.list_backups(&base_path)?;
+        let mut prune_info = compute_prune_info(list, &prune_options)?;
+        prune_info.reverse(); // delete older snapshots first
+
+        task_log!(
+            worker,
+            "Starting prune on store \"{}\" group \"{}/{}\"",
+            store,
+            group.backup_type(),
+            group.backup_id()
+        );
+
+        for (info, keep) in prune_info {
+            task_log!(
+                worker,
+                "{} {}/{}/{}",
+                if keep { "keep" } else { "remove" },
+                group.backup_type(),
+                group.backup_id(),
+                info.backup_dir.backup_time_string()
+            );
+            if !keep {
+                if let Err(err) = datastore.remove_backup_dir(&info.backup_dir, false) {
+                    task_warn!(
+                        worker,
+                        "failed to remove dir {:?}: {}",
+                        info.backup_dir.relative_path(),
+                        err,
+                    );
+                }
+            }
+        }
+    }
+
+    Ok(())
+}
+
 pub fn do_prune_job(
     mut job: Job,
     prune_options: PruneOptions,
@@ -29,58 +84,11 @@ pub fn do_prune_job(
         move |worker| {
             job.start(&worker.upid().to_string())?;
 
-            let result = try_block!({
-                task_log!(worker, "Starting datastore prune on store \"{}\"", store);
-
-                if let Some(event_str) = schedule {
-                    task_log!(worker, "task triggered by schedule '{}'", event_str);
-                }
-
-                task_log!(
-                    worker,
-                    "retention options: {}",
-                    prune_options.cli_options_string()
-                );
-
-                let base_path = datastore.base_path();
-
-                let groups = BackupInfo::list_backup_groups(&base_path)?;
-                for group in groups {
-                    let list = group.list_backups(&base_path)?;
-                    let mut prune_info = compute_prune_info(list, &prune_options)?;
-                    prune_info.reverse(); // delete older snapshots first
-
-                    task_log!(
-                        worker,
-                        "Starting prune on store \"{}\" group \"{}/{}\"",
-                        store,
-                        group.backup_type(),
-                        group.backup_id()
-                    );
+            if let Some(event_str) = schedule {
+                task_log!(worker, "task triggered by schedule '{}'", event_str);
+            }
 
-                    for (info, keep) in prune_info {
-                        task_log!(
-                            worker,
-                            "{} {}/{}/{}",
-                            if keep { "keep" } else { "remove" },
-                            group.backup_type(),
-                            group.backup_id(),
-                            info.backup_dir.backup_time_string()
-                        );
-                        if !keep {
-                            if let Err(err) = datastore.remove_backup_dir(&info.backup_dir, false) {
-                                task_warn!(
-                                    worker,
-                                    "failed to remove dir {:?}: {}",
-                                    info.backup_dir.relative_path(),
-                                    err,
-                                );
-                            }
-                        }
-                    }
-                }
-                Ok(())
-            });
+            let result = prune_datastore(worker.clone(), prune_options, &store, datastore);
 
             let status = worker.create_state(&result);
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 07/11] server/prune_job: add 'keep_all' logic to 'prune_datastore'

[Dominik Csapak]

it is the same as when pruning single groups.
for prune_jobs, we never start the worker if there is no prune option set.
but if we want to call 'prune_datastore' from somewhere else, we
have to check it here again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/server/prune_job.rs | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 7ea42fcb..40ed555f 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -19,11 +19,17 @@ pub fn prune_datastore(
 ) -> Result<(), Error> {
     task_log!(worker, "Starting datastore prune on store \"{}\"", store);
 
-    task_log!(
-        worker,
-        "retention options: {}",
-        prune_options.cli_options_string()
-    );
+    let keep_all = !prune_options.keeps_something();
+
+    if keep_all {
+        task_log!(worker, "No prune selection - keeping all files.");
+    } else {
+        task_log!(
+            worker,
+            "retention options: {}",
+            prune_options.cli_options_string()
+        );
+    }
 
     let base_path = datastore.base_path();
 
@@ -41,7 +47,8 @@ pub fn prune_datastore(
             group.backup_id()
         );
 
-        for (info, keep) in prune_info {
+        for (info, mut keep) in prune_info {
+            if keep_all { keep = true; }
             task_log!(
                 worker,
                 "{} {}/{}/{}",
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 08/11] server/prune_job: add proper permission checks to 'prune_datastore'

[Dominik Csapak]

checks for PRIV_DATASTORE_MODIFY, or else if the auth_id is the backup
owner, and skips the group if not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/backup/datastore.rs |  2 +-
 src/server/prune_job.rs | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/src/backup/datastore.rs b/src/backup/datastore.rs
index 29700846..0a5a52d1 100644
--- a/src/backup/datastore.rs
+++ b/src/backup/datastore.rs
@@ -355,7 +355,7 @@ impl DataStore {
     pub fn owns_backup(&self, backup_group: &BackupGroup, auth_id: &Authid) -> Result<bool, Error> {
         let owner = self.get_owner(backup_group)?;
 
-        Ok(check_backup_owner(owner, auth_id).is_ok())
+        Ok(check_backup_owner(&owner, auth_id).is_ok())
     }
 
     /// Set the backup owner.
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 40ed555f..bbf53ade 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -6,6 +6,8 @@ use pbs_datastore::{task_log, task_warn};
 
 use crate::{
     api2::types::*,
+    config::acl::PRIV_DATASTORE_MODIFY,
+    config::cached_user_info::CachedUserInfo,
     backup::{compute_prune_info, BackupInfo, DataStore, PruneOptions},
     server::jobstate::Job,
     server::WorkerTask,
@@ -13,6 +15,7 @@ use crate::{
 
 pub fn prune_datastore(
     worker: Arc<WorkerTask>,
+    auth_id: Authid,
     prune_options: PruneOptions,
     store: &str,
     datastore: Arc<DataStore>,
@@ -31,11 +34,20 @@ pub fn prune_datastore(
         );
     }
 
+    let user_info = CachedUserInfo::new()?;
+    let privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
+    let has_privs = privs & PRIV_DATASTORE_MODIFY != 0;
+
     let base_path = datastore.base_path();
 
     let groups = BackupInfo::list_backup_groups(&base_path)?;
     for group in groups {
         let list = group.list_backups(&base_path)?;
+
+        if !has_privs && !datastore.owns_backup(&group, &auth_id)? {
+            continue;
+        }
+
         let mut prune_info = compute_prune_info(list, &prune_options)?;
         prune_info.reverse(); // delete older snapshots first
 
@@ -83,6 +95,7 @@ pub fn do_prune_job(
     let datastore = DataStore::lookup_datastore(&store)?;
 
     let worker_type = job.jobtype().to_string();
+    let auth_id = auth_id.clone();
     let upid_str = WorkerTask::new_thread(
         &worker_type,
         Some(job.jobname().to_string()),
@@ -95,7 +108,7 @@ pub fn do_prune_job(
                 task_log!(worker, "task triggered by schedule '{}'", event_str);
             }
 
-            let result = prune_datastore(worker.clone(), prune_options, &store, datastore);
+            let result = prune_datastore(worker.clone(), auth_id, prune_options, &store, datastore);
 
             let status = worker.create_state(&result);
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 09/11] api: admin/datastore: add new 'prune-datastore' api call

[Dominik Csapak]

to prune the whole datastore at once, with the given parameters.
We need a new api call since this can take a while and we need to start
a worker for this. The exisiting api call returns a list of removed/kept
snapshots and is synchronous.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/admin/datastore.rs | 63 ++++++++++++++++++++++++++++++++++++-
 src/server/prune_job.rs     |  9 ++++--
 2 files changed, 69 insertions(+), 3 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 79ab97e7..60941501 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -820,7 +820,7 @@ pub fn verify(
         permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_PRUNE, true),
     },
 )]
-/// Prune the datastore
+/// Prune a group on the datastore
 pub fn prune(
     backup_id: String,
     backup_type: String,
@@ -922,6 +922,62 @@ pub fn prune(
     Ok(json!(prune_result))
 }
 
+#[api(
+    input: {
+        properties: {
+            "dry-run": {
+                optional: true,
+                type: bool,
+                default: false,
+                description: "Just show what prune would do, but do not delete anything.",
+            },
+            "prune-options": {
+                type: PruneOptions,
+                flatten: true,
+            },
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+        },
+    },
+    returns: {
+        schema: UPID_SCHEMA,
+    },
+    access: {
+        permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_PRUNE, true),
+    },
+)]
+/// Prune the datastore
+pub fn prune_datastore(
+    dry_run: bool,
+    prune_options: PruneOptions,
+    store: String,
+    _param: Value,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<String, Error> {
+
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+
+    let datastore = DataStore::lookup_datastore(&store)?;
+
+    let upid_str = WorkerTask::new_thread(
+        "prune",
+        Some(store.clone()),
+        auth_id.clone(),
+        false,
+        move |worker| crate::server::prune_datastore(
+            worker.clone(),
+            auth_id,
+            prune_options,
+            &store,
+            datastore,
+            dry_run
+        ),
+    )?;
+
+    Ok(upid_str)
+}
+
 #[api(
     input: {
         properties: {
@@ -1844,6 +1900,11 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
         &Router::new()
             .post(&API_METHOD_PRUNE)
     ),
+    (
+        "prune-datastore",
+        &Router::new()
+            .post(&API_METHOD_PRUNE_DATASTORE)
+    ),
     (
         "pxar-file-download",
         &Router::new()
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index bbf53ade..255e21ce 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -19,9 +19,14 @@ pub fn prune_datastore(
     prune_options: PruneOptions,
     store: &str,
     datastore: Arc<DataStore>,
+    dry_run: bool,
 ) -> Result<(), Error> {
     task_log!(worker, "Starting datastore prune on store \"{}\"", store);
 
+    if dry_run {
+        task_log!(worker, "(dry test run)");
+    }
+
     let keep_all = !prune_options.keeps_something();
 
     if keep_all {
@@ -69,7 +74,7 @@ pub fn prune_datastore(
                 group.backup_id(),
                 info.backup_dir.backup_time_string()
             );
-            if !keep {
+            if !keep && !dry_run {
                 if let Err(err) = datastore.remove_backup_dir(&info.backup_dir, false) {
                     task_warn!(
                         worker,
@@ -108,7 +113,7 @@ pub fn do_prune_job(
                 task_log!(worker, "task triggered by schedule '{}'", event_str);
             }
 
-            let result = prune_datastore(worker.clone(), auth_id, prune_options, &store, datastore);
+            let result = prune_datastore(worker.clone(), auth_id, prune_options, &store, datastore, false);
 
             let status = worker.create_state(&result);
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 10/11] ui: datastore/Content: add 'Prune All' button

[Dominik Csapak]

since the api call always starts a real worker, we cannot have a
preview. It would also be very hard to show that for all groups in a
non-confusing way. We reuse the pbsPruneInputPanel and add the dry-run
field there conditionally.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 www/datastore/Content.js    | 34 ++++++++++++++++++++++++++++++++++
 www/window/DataStoreEdit.js | 15 +++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/www/datastore/Content.js b/www/datastore/Content.js
index ac8ae1cc..57693785 100644
--- a/www/datastore/Content.js
+++ b/www/datastore/Content.js
@@ -340,6 +340,35 @@ Ext.define('PBS.DataStoreContent', {
 	    });
 	},
 
+	pruneAll: function() {
+	    let me = this;
+	    let view = me.getView();
+
+	    if (!view.datastore) return;
+
+	    Ext.create('Proxmox.window.Edit', {
+		title: `Prune Datastore '${view.datastore}'`,
+		onlineHelp: 'maintenance_pruning',
+
+		method: 'POST',
+		submitText: "Prune",
+		autoShow: true,
+		isCreate: true,
+		showTaskViewer: true,
+
+		taskDone: () => me.reload(),
+
+		url: `/api2/extjs/admin/datastore/${view.datastore}/prune-datastore`,
+
+		items: [
+		    {
+			xtype: 'pbsPruneInputPanel',
+			dryrun: true,
+		    },
+		],
+	    });
+	},
+
 	onVerify: function(view, rI, cI, item, e, rec) {
 	    let me = this;
 	    view = me.getView();
@@ -865,6 +894,11 @@ Ext.define('PBS.DataStoreContent', {
 	    confirmMsg: gettext('Do you want to verify all snapshots now?'),
 	    handler: 'verifyAll',
 	},
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Prune All'),
+	    handler: 'pruneAll',
+	},
 	'->',
 	{
 	    xtype: 'tbtext',
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index fbf0da5b..ed23ad11 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -6,6 +6,9 @@ Ext.define('PBS.panel.PruneInputPanel', {
 
     onlineHelp: 'maintenance_pruning',
 
+    // show/hide dry-run field
+    dryrun: false,
+
     cbindData: function() {
 	let me = this;
 	me.isCreate = !!me.isCreate;
@@ -65,6 +68,18 @@ Ext.define('PBS.panel.PruneInputPanel', {
 	},
     ],
 
+    columnB: [
+	{
+	    xtype: 'proxmoxcheckbox',
+	    name: 'dry-run',
+	    fieldLabel: gettext('Dry Run'),
+	    cbind: {
+		hidden: '{!dryrun}',
+		disabled: '{!dryrun}',
+	    },
+	},
+    ],
+
 });
 Ext.define('PBS.DataStoreEdit', {
     extend: 'Proxmox.window.Edit',
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 11/11] ui: datastore/Prune: improve title of group prune window

[Dominik Csapak]

we are not actually pruning the whole datastore, but only the single
group, so set that as a title

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 www/datastore/Prune.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/www/datastore/Prune.js b/www/datastore/Prune.js
index 10191679..3e74269f 100644
--- a/www/datastore/Prune.js
+++ b/www/datastore/Prune.js
@@ -251,7 +251,7 @@ Ext.define('PBS.DataStorePrune', {
 
 	Ext.apply(me, {
 	    url: '/api2/extjs/admin/datastore/' + me.datastore + "/prune",
-	    title: "Prune Datastore '" + me.datastore + "'",
+	    title: `Prune Group '${me.datastore}:${me.backup_type}/${me.backup_id}'`,
 	    items: [{
 		xtype: 'pbsDataStorePruneInputPanel',
 		url: '/api2/extjs/admin/datastore/' + me.datastore + "/prune",
-- 
2.30.2

[pbs-devel] applied: [PATCH proxmox-backup 00/11] add 'prune all' button to datastore content

[Dietmar Maurer]

applied

On 7/16/21 10:53 AM, Dominik Csapak wrote:
> similar to the 'verify all' button, it makes sense that a user can
> prune all groups (which he has access to) on demand with custom setings.
>
> this adds a new api call, since pruning all groups could take a while,
> depending on the number of groups snapshots, and it does not make sense
> to have that in a synchronous api call (that we already have per group),
> so modifying the existing api call to have the group optional was not
> really sensible IMHO.
>
> patches 1-6 are simply refactoring, to make it more easy to use
>   the PruneOptions, 'prune_datastore', etc. later
> 7,8 add new parameter/functionality to the 'prune_datastore' method
> 9 really adds the api call
> 10 is the button in the gui
> 11 is just a ui improvement for the other prune window
>   (could be applied seperately)
>
> Dominik Csapak (11):
>    api-types: move PRUNE_SCHEMA_KEEP_* to pbs-api-types
>    pbs-datastore/prune: make PruneOptions an api type
>    client: simplify prune api method
>    api: admin/datastore: simplify prune api call
>    backup/datastore: refactor check_backup_owner there
>    server/prune_job: factor out 'prune_datastore'
>    server/prune_job: add 'keep_all' logic to 'prune_datastore'
>    server/prune_job: add proper permission checks to 'prune_datastore'
>    api: admin/datastore: add new 'prune-datastore' api call
>    ui: datastore/Content: add 'Prune All' button
>    ui: datastore/Prune: improve title of group prune window
>
>   pbs-api-types/src/lib.rs         |  30 +++++
>   pbs-datastore/src/prune.rs       |  50 +++++++-
>   src/api2/admin/datastore.rs      | 208 ++++++++++++++++---------------
>   src/api2/types/mod.rs            |  30 -----
>   src/backup/datastore.rs          |  20 +++
>   src/bin/proxmox-backup-client.rs |  95 +++++++-------
>   src/server/prune_job.rs          | 139 +++++++++++++--------
>   www/datastore/Content.js         |  34 +++++
>   www/datastore/Prune.js           |   2 +-
>   www/window/DataStoreEdit.js      |  15 +++
>   10 files changed, 390 insertions(+), 233 deletions(-)
>