[pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool

[pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool

[Dominik Csapak]

this series adds the tool 'pbs-shell', similar to 'pvesh' and 'pmgsh'
this tool is intended mainly for debugging, but can be useful for
api calls not exposed via the client/manager.

proxmox (and the dependency in proxmox-backup) need to be bumped

proxmox-backup patches 2-5 are not strictly necessary, but improve
the api so that the 'ls' command works better
(there are still some api paths to be fixed)

proxmox:

Dominik Csapak (1):
  proxmox: generate_usage_str: don't require static lifetimes

 proxmox/src/api/cli/format.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

proxmox-backup:

Dominik Csapak (5):
  add 'pbs-shell' utility
  api2: add missing token list match_all property
  api2: disks/directory: refactor BASE_MOUNT_DIR
  api2: disks/directory: add 'name' property to directory mount listing
  api2: nodes: add missing node list api call

 Makefile                                     |   2 +
 debian/pbs-shell.bc                          |   3 +
 debian/proxmox-backup-server.bash-completion |   1 +
 debian/proxmox-backup-server.install         |   3 +
 docs/Makefile                                |   8 +
 docs/pbs-shell/description.rst               |   3 +
 docs/pbs-shell/man1.rst                      |  40 ++
 src/api2/access/user.rs                      |  35 +-
 src/api2/mod.rs                              |   4 +-
 src/api2/node/disks/directory.rs             |  15 +-
 src/api2/node/mod.rs                         |  12 +-
 src/bin/pbs-shell.rs                         | 502 +++++++++++++++++++
 zsh-completions/_pbs-shell                   |  13 +
 13 files changed, 628 insertions(+), 13 deletions(-)
 create mode 100644 debian/pbs-shell.bc
 create mode 100644 docs/pbs-shell/description.rst
 create mode 100644 docs/pbs-shell/man1.rst
 create mode 100644 src/bin/pbs-shell.rs
 create mode 100644 zsh-completions/_pbs-shell

-- 
2.30.2

[pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes

[Dominik Csapak]

this prevents us from using it under certain conditions and it's
actually not necessary, so drop them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox/src/api/cli/format.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proxmox/src/api/cli/format.rs b/proxmox/src/api/cli/format.rs
index 17efcd6..a4fb78d 100644
--- a/proxmox/src/api/cli/format.rs
+++ b/proxmox/src/api/cli/format.rs
@@ -60,7 +60,7 @@ pub fn generate_usage_str(
     cli_cmd: &CliCommand,
     format: DocumentationFormat,
     indent: &str,
-    skip_options: &'static [&'static str],
+    skip_options: &[&str],
 ) -> String {
     let arg_param = cli_cmd.arg_param;
     let fixed_param = &cli_cmd.fixed_param;
-- 
2.30.2

[pbs-devel] applied: [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes

[Thomas Lamprecht]

On 09.09.21 15:48, Dominik Csapak wrote:
> this prevents us from using it under certain conditions and it's
> actually not necessary, so drop them
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  proxmox/src/api/cli/format.rs | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
>

applied this one already, thanks!

[pbs-devel] [PATCH proxmox-backup 1/5] add 'pbs-shell' utility

[Dominik Csapak]

similar to pve/pmg, a user can call the api with this utility without
going through the proxy/daemon, as well as list the api endpoints
(with child links) and get the api description of endpoints

this is mainly intended for debugging, but it is also useful for
situations where some api calls do not have an equivalent in a binary
and a user does not want to go through the api

not implemented are the http2 api calls (since it is a separate api and
it wouldn't be that easy to do)

there are a few quirks though, related to the 'ls' command:
i extract the 'child-link' from the property name of the
'match_all' statement of the router, but this does not
always match with the property from the relevant 'get' api call
so it fails there (e.g. /tape/drive )

this can be fixed in the respective api calls (e.g. by renaming
the parameter that comes from the path)

includes bash/zsh completion helpers and a basic manpage

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 Makefile                                     |   2 +
 debian/pbs-shell.bc                          |   3 +
 debian/proxmox-backup-server.bash-completion |   1 +
 debian/proxmox-backup-server.install         |   3 +
 docs/Makefile                                |   8 +
 docs/pbs-shell/description.rst               |   3 +
 docs/pbs-shell/man1.rst                      |  40 ++
 src/bin/pbs-shell.rs                         | 502 +++++++++++++++++++
 zsh-completions/_pbs-shell                   |  13 +
 9 files changed, 575 insertions(+)
 create mode 100644 debian/pbs-shell.bc
 create mode 100644 docs/pbs-shell/description.rst
 create mode 100644 docs/pbs-shell/man1.rst
 create mode 100644 src/bin/pbs-shell.rs
 create mode 100644 zsh-completions/_pbs-shell

diff --git a/Makefile b/Makefile
index c1aecf61..abeaff37 100644
--- a/Makefile
+++ b/Makefile
@@ -11,6 +11,7 @@ USR_BIN := \
 	proxmox-backup-client 	\
 	proxmox-file-restore	\
 	pxar			\
+	pbs-shell		\
 	proxmox-tape		\
 	pmtx			\
 	pmt
@@ -172,6 +173,7 @@ $(COMPILED_BINS) $(COMPILEDIR)/dump-catalog-shell-cli $(COMPILEDIR)/docgen: .do-
 	    --bin proxmox-backup-api \
 	    --bin proxmox-backup-proxy \
 	    --bin proxmox-backup-manager \
+	    --bin pbs-shell \
 	    --bin docgen
 	$(CARGO) build $(CARGO_BUILD_ARGS) \
 	    --package proxmox-backup-banner \
diff --git a/debian/pbs-shell.bc b/debian/pbs-shell.bc
new file mode 100644
index 00000000..3d17187c
--- /dev/null
+++ b/debian/pbs-shell.bc
@@ -0,0 +1,3 @@
+# pbs-shell bash completion
+
+complete -C 'pbs-shell bashcomplete' pbs-shell
diff --git a/debian/proxmox-backup-server.bash-completion b/debian/proxmox-backup-server.bash-completion
index a2165699..8d6a7047 100644
--- a/debian/proxmox-backup-server.bash-completion
+++ b/debian/proxmox-backup-server.bash-completion
@@ -2,3 +2,4 @@ debian/proxmox-backup-manager.bc proxmox-backup-manager
 debian/proxmox-tape.bc proxmox-tape
 debian/pmtx.bc pmtx
 debian/pmt.bc pmt
+debian/pbs-shell.bc pbs-shell
diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
index 6e2219b4..5e1071fa 100644
--- a/debian/proxmox-backup-server.install
+++ b/debian/proxmox-backup-server.install
@@ -14,6 +14,7 @@ usr/sbin/proxmox-backup-manager
 usr/bin/pmtx
 usr/bin/pmt
 usr/bin/proxmox-tape
+usr/bin/pbs-shell
 usr/share/javascript/proxmox-backup/index.hbs
 usr/share/javascript/proxmox-backup/css/ext6-pbs.css
 usr/share/javascript/proxmox-backup/images
@@ -24,6 +25,7 @@ usr/share/man/man1/proxmox-backup-proxy.1
 usr/share/man/man1/proxmox-tape.1
 usr/share/man/man1/pmtx.1
 usr/share/man/man1/pmt.1
+usr/share/man/man1/pbs-shell.1
 usr/share/man/man5/acl.cfg.5
 usr/share/man/man5/datastore.cfg.5
 usr/share/man/man5/user.cfg.5
@@ -38,3 +40,4 @@ usr/share/zsh/vendor-completions/_proxmox-backup-manager
 usr/share/zsh/vendor-completions/_proxmox-tape
 usr/share/zsh/vendor-completions/_pmtx
 usr/share/zsh/vendor-completions/_pmt
+usr/share/zsh/vendor-completions/_pbs-shell
diff --git a/docs/Makefile b/docs/Makefile
index 5e37f7d1..e67df2ea 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -10,6 +10,7 @@ GENERATED_SYNOPSIS := 						\
 	pxar/synopsis.rst					\
 	pmtx/synopsis.rst					\
 	pmt/synopsis.rst					\
+	pbs-shell/synopsis.rst					\
 	config/media-pool/config.rst				\
 	config/tape/config.rst					\
 	config/tape-job/config.rst				\
@@ -24,6 +25,7 @@ MAN1_PAGES := 				\
 	pxar.1				\
 	pmtx.1				\
 	pmt.1				\
+	pbs-shell.1			\
 	proxmox-tape.1			\
 	proxmox-backup-proxy.1		\
 	proxmox-backup-client.1		\
@@ -117,6 +119,12 @@ pmt/synopsis.rst: ${COMPILEDIR}/pmt
 pmt.1: pmt/man1.rst  pmt/description.rst pmt/options.rst pmt/synopsis.rst
 	rst2man $< >$@
 
+pbs-shell/synopsis.rst: ${COMPILEDIR}/pbs-shell
+	${COMPILEDIR}/pbs-shell printdoc > pbs-shell/synopsis.rst
+
+pbs-shell.1: pbs-shell/man1.rst  pbs-shell/description.rst pbs-shell/synopsis.rst
+	rst2man $< >$@
+
 config/datastore/config.rst: ${COMPILEDIR}/docgen
 	${COMPILEDIR}/docgen datastore.cfg >$@
 
diff --git a/docs/pbs-shell/description.rst b/docs/pbs-shell/description.rst
new file mode 100644
index 00000000..8dfcae15
--- /dev/null
+++ b/docs/pbs-shell/description.rst
@@ -0,0 +1,3 @@
+The ``pbs-shell`` command can show and execute api calls and their parameters.
+It is mainly intended for use during debugging.
+
diff --git a/docs/pbs-shell/man1.rst b/docs/pbs-shell/man1.rst
new file mode 100644
index 00000000..d0a1d07b
--- /dev/null
+++ b/docs/pbs-shell/man1.rst
@@ -0,0 +1,40 @@
+==========================
+pbs-shell
+==========================
+
+.. include:: ../epilog.rst
+
+-------------------------------------------------------------
+Show and execute PBS API calls
+-------------------------------------------------------------
+
+:Author: |AUTHOR|
+:Version: Version |VERSION|
+:Manual section: 1
+
+
+Synopsis
+==========
+
+.. include:: synopsis.rst
+
+
+Common Options
+==============
+
+Commands generating output supports the ``--output-format``
+parameter. It accepts the following values:
+
+:``text``: Text format (default). Human readable.
+
+:``json``: JSON (single line).
+
+:``json-pretty``: JSON (multiple lines, nicely formatted).
+
+
+Description
+============
+
+.. include:: description.rst
+
+.. include:: ../pbs-copyright.rst
diff --git a/src/bin/pbs-shell.rs b/src/bin/pbs-shell.rs
new file mode 100644
index 00000000..ce64617b
--- /dev/null
+++ b/src/bin/pbs-shell.rs
@@ -0,0 +1,502 @@
+use anyhow::{bail, format_err, Error};
+use hyper::Method;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+use std::collections::HashMap;
+
+use proxmox::api::{
+    api,
+    cli::*,
+    format::DocumentationFormat,
+    schema::{parse_parameter_strings, ApiType, ParameterSchema, Schema},
+    ApiHandler, ApiMethod, RpcEnvironment, SubRoute,
+};
+
+use pbs_client::{connect_to_localhost, display_task_log};
+
+const PROG_NAME: &str = "pbs-shell";
+
+fn complete_api_path(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, None).await })
+}
+
+fn complete_api_path_get(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, Some("r")).await })
+}
+
+fn complete_api_path_set(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, Some("w")).await })
+}
+
+fn complete_api_path_create(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, Some("c")).await })
+}
+
+fn complete_api_path_delete(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, Some("d")).await })
+}
+
+fn complete_api_path_ls(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> {
+    pbs_runtime::main(async { complete_api_path_do(complete_me, Some("D")).await })
+}
+
+async fn complete_api_path_do(mut complete_me: &str, capability: Option<&str>) -> Vec<String> {
+    if complete_me.is_empty() {
+        complete_me = "/";
+    }
+
+    let mut list = Vec::new();
+
+    let mut lookup_path = complete_me.to_string();
+    let mut filter = "";
+    let last_path_index = complete_me.rfind('/');
+    if let Some(index) = last_path_index {
+        if index != complete_me.len() - 1 {
+            lookup_path = complete_me[..(index + 1)].to_string();
+            if index < complete_me.len() - 1 {
+                filter = &complete_me[(index + 1)..];
+            }
+        }
+    }
+
+    let uid = nix::unistd::Uid::current();
+
+    let username = match nix::unistd::User::from_uid(uid) {
+        Ok(Some(user)) => user.name,
+        _ => "root@pam".to_string(),
+    };
+    let mut rpcenv = CliEnvironment::new();
+    rpcenv.set_auth_id(Some(format!("{}@pam", username)));
+
+    while let Ok(children) = get_api_children(lookup_path.clone(), &mut rpcenv).await {
+        let old_len = list.len();
+        for entry in children {
+            let name = entry.name;
+            let caps = entry.capabilities;
+
+            if filter.is_empty() || name.starts_with(filter) {
+                let mut path = format!("{}{}", lookup_path, name);
+                if caps.contains('D') {
+                    path.push('/');
+                    list.push(path.clone());
+                } else if let Some(cap) = capability {
+                    if caps.contains(cap) {
+                        list.push(path);
+                    }
+                } else {
+                    list.push(path);
+                }
+            }
+        }
+
+        if list.len() == 1 && old_len != 1 && list[0].ends_with('/') {
+            // we added only one match and it was a directory, lookup again
+            lookup_path = list[0].clone();
+            filter = "";
+            continue;
+        }
+
+        break;
+    }
+
+    list
+}
+
+async fn get_child_links(
+    path: &str,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<String>, Error> {
+    let mut uri_param = HashMap::new();
+    let (path, components) = proxmox_backup::tools::normalize_uri_path(&path)?;
+
+    let info = &proxmox_backup::api2::ROUTER
+        .find_route(&components, &mut uri_param)
+        .ok_or_else(|| format_err!("no such resource"))?;
+
+    match info.subroute {
+        Some(SubRoute::Map(map)) => Ok(map.iter().map(|(name, _)| name.to_string()).collect()),
+        Some(SubRoute::MatchAll { param_name, .. }) => {
+            let get_call = info.get.ok_or_else(|| format_err!("no such resource"))?;
+            let list = call_api(get_call, rpcenv, serde_json::to_value(uri_param)?).await?;
+            Ok(list
+                .as_array()
+                .ok_or_else(|| format_err!("{} did not return an array", path))?
+                .iter()
+                .map(|item| {
+                    item[param_name]
+                        .as_str()
+                        .map(|c| c.to_string())
+                        .ok_or_else(|| format_err!("no such property {}", param_name))
+                })
+                .collect::<Result<Vec<_>, _>>()?)
+        }
+        None => bail!("link does not define child links"),
+    }
+}
+
+fn get_api_method(
+    method: Method,
+    path: &str,
+) -> Result<(&'static ApiMethod, HashMap<String, String>), Error> {
+    let mut uri_param = HashMap::new();
+    let (path, components) = proxmox_backup::tools::normalize_uri_path(&path)?;
+    if let Some(method) =
+        &proxmox_backup::api2::ROUTER.find_method(&components, method.clone(), &mut uri_param)
+    {
+        Ok((method, uri_param))
+    } else {
+        bail!("no {} handler defined for '{}'", method, path);
+    }
+}
+
+fn merge_parameters(
+    uri_param: HashMap<String, String>,
+    param: Value,
+    schema: ParameterSchema,
+) -> Result<Value, Error> {
+    let mut param_list: Vec<(String, String)> = vec![];
+
+    for (k, v) in uri_param {
+        param_list.push((k.clone(), v.clone()));
+    }
+
+    if let Some(map) = param.as_object() {
+        for (k, v) in map {
+            param_list.push((k.clone(), v.as_str().unwrap().to_string()));
+        }
+    }
+
+    let params = parse_parameter_strings(&param_list, schema, true)?;
+
+    Ok(params)
+}
+
+async fn call_api(
+    method: &'static ApiMethod,
+    rpcenv: &mut dyn RpcEnvironment,
+    params: Value,
+) -> Result<Value, Error> {
+    match method.handler {
+        ApiHandler::AsyncHttp(_handler) => {
+            bail!("not implemented");
+        }
+        ApiHandler::Sync(handler) => (handler)(params, method, rpcenv),
+        ApiHandler::Async(handler) => (handler)(params, method, rpcenv).await,
+    }
+}
+
+async fn call_api_and_format_result(
+    method: Method,
+    path: String,
+    mut param: Value,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let mut output_format = extract_output_format(&mut param);
+    let (method, uri_param) = get_api_method(method, &path)?;
+    let params = merge_parameters(uri_param, param, method.parameters)?;
+
+    let mut result = call_api(method, rpcenv, params).await?;
+
+    if output_format == "text" {
+        if let Some(upid) = result.as_str() {
+            let mut client = connect_to_localhost()?;
+            display_task_log(&mut client, upid, true).await?;
+            return Ok(());
+        }
+    }
+
+    let options = default_table_format_options();
+    let return_type = &method.returns;
+    if matches!(return_type.schema, Schema::Null) {
+        output_format = "json-pretty".to_string();
+    }
+
+    format_and_print_result_full(&mut result, return_type, &output_format, &options);
+
+    Ok(())
+}
+
+#[api(
+    input: {
+        additional_properties: true,
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Call API PUT on <path>
+async fn set(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    call_api_and_format_result(Method::PUT, path, param, rpcenv).await
+}
+
+#[api(
+    input: {
+        additional_properties: true,
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Call API POST on <path>
+async fn create(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    call_api_and_format_result(Method::POST, path, param, rpcenv).await
+}
+
+#[api(
+    input: {
+        additional_properties: true,
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Call API GET on <path>
+async fn get(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    call_api_and_format_result(Method::GET, path, param, rpcenv).await
+}
+
+#[api(
+    input: {
+        additional_properties: true,
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Call API DELETE on <path>
+async fn delete(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    call_api_and_format_result(Method::DELETE, path, param, rpcenv).await
+}
+
+#[api(
+    input: {
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            verbose: {
+                type: Boolean,
+                description: "Verbose output format.",
+                optional: true,
+                default: false,
+            }
+        },
+    },
+)]
+/// Get API usage information for <path>
+async fn usage(
+    path: String,
+    verbose: bool,
+    _param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let docformat = if verbose {
+        DocumentationFormat::Full
+    } else {
+        DocumentationFormat::Short
+    };
+    let mut found = false;
+    for command in  &["get", "set", "create", "delete"] {
+        let http_method = match *command {
+            "get" => Method::GET,
+            "set" => Method::PUT,
+            "create" => Method::POST,
+            "delete" => Method::DELETE,
+            _ => unreachable!(),
+        };
+        let (info, uri_params) = match get_api_method(http_method, &path) {
+            Ok(some) => some,
+            Err(_) => continue,
+        };
+        found = true;
+
+        let skip_params: Vec<&str> = uri_params.keys().map(|s| &**s).collect();
+
+        let cmd = CliCommand::new(info);
+        let prefix = format!("USAGE: {} {} {}", PROG_NAME, command, path);
+
+        print!(
+            "{}",
+            generate_usage_str(&prefix, &cmd, docformat, "", &skip_params)
+        );
+    }
+
+    if !found {
+        bail!("no such resource '{}'", path);
+    }
+    Ok(())
+}
+
+#[api()]
+#[derive(Debug, Serialize, Deserialize)]
+/// A child link with capabilities
+struct ApiDirEntry {
+    /// The name of the link
+    name: String,
+    /// The capabilities of the path (format Drwcd)
+    capabilities: String,
+}
+
+const LS_SCHEMA: &proxmox::api::schema::Schema =
+    &proxmox::api::schema::ArraySchema::new("List of child links", &ApiDirEntry::API_SCHEMA)
+        .schema();
+
+async fn get_api_children(
+    path: String,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<ApiDirEntry>, Error> {
+    let mut res = Vec::new();
+    for link in get_child_links(&path, rpcenv).await? {
+        let path = format!("{}/{}", path, link);
+        let (path, _) = proxmox_backup::tools::normalize_uri_path(&path)?;
+        let mut cap = String::new();
+
+        if get_child_links(&path, rpcenv).await.is_ok() {
+            cap.push('D');
+        } else {
+            cap.push('-');
+        }
+
+        let cap_list = &[
+            (Method::GET,    'r'),
+            (Method::PUT,    'w'),
+            (Method::POST,   'c'),
+            (Method::DELETE, 'd'),
+        ];
+
+        for (method, c) in cap_list {
+            if get_api_method(method.clone(), &path).is_ok() {
+                cap.push(*c);
+            } else {
+                cap.push('-');
+            }
+        }
+
+        res.push(ApiDirEntry {
+            name: link.to_string(),
+            capabilities: cap,
+        });
+    }
+
+    Ok(res)
+}
+
+#[api(
+    input: {
+        properties: {
+            path: {
+                type: String,
+                description: "API path.",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Get API usage information for <path>
+async fn ls(path: String, mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    let output_format = extract_output_format(&mut param);
+
+    let options = TableFormatOptions::new()
+        .noborder(true)
+        .noheader(true)
+        .sortby("name", false);
+
+    let res = get_api_children(path, rpcenv).await?;
+
+    format_and_print_result_full(
+        &mut serde_json::to_value(res)?,
+        &proxmox::api::schema::ReturnType {
+            optional: false,
+            schema: &LS_SCHEMA,
+        },
+        &output_format,
+        &options,
+    );
+
+    Ok(())
+}
+
+fn main() -> Result<(), Error> {
+    let cmd_def = CliCommandMap::new()
+        .insert(
+            "get",
+            CliCommand::new(&API_METHOD_GET)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path_get),
+        )
+        .insert(
+            "set",
+            CliCommand::new(&API_METHOD_SET)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path_set),
+        )
+        .insert(
+            "create",
+            CliCommand::new(&API_METHOD_CREATE)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path_create),
+        )
+        .insert(
+            "delete",
+            CliCommand::new(&API_METHOD_DELETE)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path_delete),
+        )
+        .insert(
+            "ls",
+            CliCommand::new(&API_METHOD_LS)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path_ls),
+        )
+        .insert(
+            "usage",
+            CliCommand::new(&API_METHOD_USAGE)
+                .arg_param(&["path"])
+                .completion_cb("path", complete_api_path),
+        );
+
+    let uid = nix::unistd::Uid::current();
+
+    let username = match nix::unistd::User::from_uid(uid)? {
+        Some(user) => user.name,
+        None => bail!("unable to get user name"),
+    };
+    let mut rpcenv = CliEnvironment::new();
+    rpcenv.set_auth_id(Some(format!("{}@pam", username)));
+
+    pbs_runtime::main(run_async_cli_command(cmd_def, rpcenv));
+    Ok(())
+}
diff --git a/zsh-completions/_pbs-shell b/zsh-completions/_pbs-shell
new file mode 100644
index 00000000..507f15ae
--- /dev/null
+++ b/zsh-completions/_pbs-shell
@@ -0,0 +1,13 @@
+#compdef _pbs-shell() pbs-shell
+
+function _pbs-shell() {
+    local cwords line point cmd curr prev
+    cwords=${#words[@]}
+    line=$words
+    point=${#line}
+    cmd=${words[1]}
+    curr=${words[cwords]}
+    prev=${words[cwords-1]}
+    compadd -- $(COMP_CWORD="$cwords" COMP_LINE="$line" COMP_POINT="$point" \
+        pbs-shell bashcomplete "$cmd" "$curr" "$prev")
+}
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 2/5] api2: add missing token list match_all property

[Dominik Csapak]

to have the proper link between the token list and the sub routes
in the api, include the 'tokenname' property in the token listing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/access/user.rs | 35 ++++++++++++++++++++++++++++-------
 1 file changed, 28 insertions(+), 7 deletions(-)

diff --git a/src/api2/access/user.rs b/src/api2/access/user.rs
index 6a2fe83c..70065986 100644
--- a/src/api2/access/user.rs
+++ b/src/api2/access/user.rs
@@ -655,6 +655,21 @@ pub fn delete_token(
     Ok(())
 }
 
+#[api(
+    properties: {
+        tokenname: { type: Tokenname },
+        token: { type: ApiToken },
+    }
+)]
+#[derive(Serialize, Deserialize)]
+/// A Token Entry of a user
+pub struct TokenInfo {
+    /// The Token name
+    pub tokenname: Tokenname,
+    #[serde(flatten)]
+    pub token: ApiToken,
+}
+
 #[api(
     input: {
         properties: {
@@ -666,7 +681,7 @@ pub fn delete_token(
     returns: {
         description: "List user's API tokens (with config digest).",
         type: Array,
-        items: { type: ApiToken },
+        items: { type: TokenInfo },
     },
     access: {
         permission: &Permission::Or(&[
@@ -680,7 +695,7 @@ pub fn list_tokens(
     userid: Userid,
     _info: &ApiMethod,
     mut rpcenv: &mut dyn RpcEnvironment,
-) -> Result<Vec<ApiToken>, Error> {
+) -> Result<Vec<TokenInfo>, Error> {
 
     let (config, digest) = crate::config::user::config()?;
 
@@ -688,15 +703,21 @@ pub fn list_tokens(
 
     rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into();
 
-    let filter_by_owner = |token: &ApiToken| {
-        if token.tokenid.is_token() {
-           token.tokenid.user() == &userid
+    let filter_by_owner = |token: ApiToken| {
+        if token.tokenid.is_token() && token.tokenid.user() == &userid {
+            let tokenname = token.tokenid.tokenname().unwrap().to_owned();
+            Some(TokenInfo {
+                tokenname,
+                token,
+            })
         } else {
-            false
+            None
         }
     };
 
-    Ok(list.into_iter().filter(filter_by_owner).collect())
+    let res = list.into_iter().filter_map(filter_by_owner).collect();
+
+    Ok(res)
 }
 
 const TOKEN_ITEM_ROUTER: Router = Router::new()
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 3/5] api2: disks/directory: refactor BASE_MOUNT_DIR

[Dominik Csapak]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/node/disks/directory.rs | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 0cb6f4e2..dd6912d3 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -22,6 +22,8 @@ use crate::server::WorkerTask;
 use crate::config::datastore::{self, DataStoreConfig};
 use pbs_config::open_backup_lockfile;
 
+const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
+
 #[api(
     properties: {
         "filesystem": {
@@ -146,7 +148,7 @@ pub fn create_datastore_disk(
         bail!("disk '{}' is already in use.", disk);
     }
 
-    let mount_point = format!("/mnt/datastore/{}", &name);
+    let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
 
     // check if the default path does exist already and bail if it does
     let default_path = std::path::PathBuf::from(&mount_point);
@@ -221,7 +223,7 @@ pub fn create_datastore_disk(
 /// Remove a Filesystem mounted under '/mnt/datastore/<name>'.".
 pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
 
-    let path = format!("/mnt/datastore/{}", name);
+    let path = format!("{}{}", BASE_MOUNT_DIR, name);
     // path of datastore cannot be changed
     let (config, _) = crate::config::datastore::config()?;
     let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 4/5] api2: disks/directory: add 'name' property to directory mount listing

[Dominik Csapak]

so that we have the properties that match with 'match_all'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/node/disks/directory.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index dd6912d3..f9282903 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -38,6 +38,8 @@ const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
 pub struct DatastoreMountInfo {
     /// The path of the mount unit.
     pub unitfile: String,
+    /// The name of the mount
+    pub name: String,
     /// The mount path.
     pub path: String,
     /// The mounted device.
@@ -86,8 +88,15 @@ pub fn  list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
         let config = systemd::config::parse_systemd_mount(&unitfile)?;
         let data: SystemdMountSection = config.lookup("Mount", "Mount")?;
 
+        let name = data
+            .Where
+            .strip_prefix(BASE_MOUNT_DIR)
+            .unwrap_or_else(|| &data.Where)
+            .to_string();
+
         list.push(DatastoreMountInfo {
             unitfile,
+            name,
             device: data.What,
             path: data.Where,
             filesystem: data.Type,
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call

[Dominik Csapak]

to have an api call for api path traversal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/api2/mod.rs      |  4 +---
 src/api2/node/mod.rs | 12 +++++++++++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/src/api2/mod.rs b/src/api2/mod.rs
index 132e2c2a..0ede4165 100644
--- a/src/api2/mod.rs
+++ b/src/api2/mod.rs
@@ -18,14 +18,12 @@ use proxmox::api::router::SubdirMap;
 use proxmox::api::Router;
 use proxmox::list_subdirs_api_method;
 
-const NODES_ROUTER: Router = Router::new().match_all("node", &node::ROUTER);
-
 const SUBDIRS: SubdirMap = &[
     ("access", &access::ROUTER),
     ("admin", &admin::ROUTER),
     ("backup", &backup::ROUTER),
     ("config", &config::ROUTER),
-    ("nodes", &NODES_ROUTER),
+    ("nodes", &node::ROUTER),
     ("ping", &ping::ROUTER),
     ("pull", &pull::ROUTER),
     ("reader", &reader::ROUTER),
diff --git a/src/api2/node/mod.rs b/src/api2/node/mod.rs
index f1a17934..194ec920 100644
--- a/src/api2/node/mod.rs
+++ b/src/api2/node/mod.rs
@@ -315,6 +315,12 @@ fn upgrade_to_websocket(
     .boxed()
 }
 
+#[api]
+/// List Nodes (only for compatiblity)
+fn list_nodes() -> Result<Value, Error> {
+    Ok(json!([ { "node": proxmox::tools::nodename().to_string() } ]))
+}
+
 pub const SUBDIRS: SubdirMap = &[
     ("apt", &apt::ROUTER),
     ("certificates", &certificates::ROUTER),
@@ -338,6 +344,10 @@ pub const SUBDIRS: SubdirMap = &[
     ),
 ];
 
-pub const ROUTER: Router = Router::new()
+pub const ITEM_ROUTER: Router = Router::new()
     .get(&list_subdirs_api_method!(SUBDIRS))
     .subdirs(SUBDIRS);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_NODES)
+    .match_all("node", &ITEM_ROUTER);
-- 
2.30.2

[pbs-devel] applied: [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call

[Thomas Lamprecht]

On 09.09.21 15:48, Dominik Csapak wrote:
> to have an api call for api path traversal
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  src/api2/mod.rs      |  4 +---
>  src/api2/node/mod.rs | 12 +++++++++++-
>  2 files changed, 12 insertions(+), 4 deletions(-)
> 
>

applied this rather independent patch already, thanks!

Any thoughts on providing some resource usage and status info like pve has?

# pvesh get /nodes --output-format json-pretty 
[
   {
      "cpu" : 0.00688381192943118,
      "disk" : 81085153280,
      "id" : "node/d7",
      "level" : "c",
      "maxcpu" : 16,
      "maxdisk" : 133204897792,
      "maxmem" : 6101630976,
      "mem" : 1656844288,
      "node" : "d7",
      "ssl_fingerprint" : "8A:DF:09:A5:D8:CC:42:AD:B1:35:3B:E7:03:30:30:3C:C2:BD:BF:94:FA:F3:CD:46:62:92:6D:34:00:56:CC:8D",
      "status" : "online",
      "type" : "node",
      "uptime" : 1990844
   }
]

Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool

[Hannes Laimer]

I just noticed two things while testing:
 - if an endpoint expects a parameter called 'path'(e.g. POST /config
     /datastore), the parameter conflicts with the api path itself
 - even though commands like verify(POST /admin/datastore/<name>/verify)
or gc(POST /admin/datastore/<name>/gc) are started and run successfully,
`Error: control socket connect failed - Connection refused (os error
111)` is displayed almost always at the end.

Note: For testing I applied the patches on
b65dfff574f44e5a60db3903d2226f4dbe386b7a, I assume the moving of stuff
after that should not effect the functionality

Am 09.09.21 um 15:48 schrieb Dominik Csapak:
> this series adds the tool 'pbs-shell', similar to 'pvesh' and 'pmgsh'
> this tool is intended mainly for debugging, but can be useful for
> api calls not exposed via the client/manager.
> 
> proxmox (and the dependency in proxmox-backup) need to be bumped
> 
> proxmox-backup patches 2-5 are not strictly necessary, but improve
> the api so that the 'ls' command works better
> (there are still some api paths to be fixed)
> 
> proxmox:
> 
> Dominik Csapak (1):
>   proxmox: generate_usage_str: don't require static lifetimes
> 
>  proxmox/src/api/cli/format.rs | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> proxmox-backup:
> 
> Dominik Csapak (5):
>   add 'pbs-shell' utility
>   api2: add missing token list match_all property
>   api2: disks/directory: refactor BASE_MOUNT_DIR
>   api2: disks/directory: add 'name' property to directory mount listing
>   api2: nodes: add missing node list api call
> 
>  Makefile                                     |   2 +
>  debian/pbs-shell.bc                          |   3 +
>  debian/proxmox-backup-server.bash-completion |   1 +
>  debian/proxmox-backup-server.install         |   3 +
>  docs/Makefile                                |   8 +
>  docs/pbs-shell/description.rst               |   3 +
>  docs/pbs-shell/man1.rst                      |  40 ++
>  src/api2/access/user.rs                      |  35 +-
>  src/api2/mod.rs                              |   4 +-
>  src/api2/node/disks/directory.rs             |  15 +-
>  src/api2/node/mod.rs                         |  12 +-
>  src/bin/pbs-shell.rs                         | 502 +++++++++++++++++++
>  zsh-completions/_pbs-shell                   |  13 +
>  13 files changed, 628 insertions(+), 13 deletions(-)
>  create mode 100644 debian/pbs-shell.bc
>  create mode 100644 docs/pbs-shell/description.rst
>  create mode 100644 docs/pbs-shell/man1.rst
>  create mode 100644 src/bin/pbs-shell.rs
>  create mode 100644 zsh-completions/_pbs-shell
> 

Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool

[Dominik Csapak]

thanks for testing!

On 9/13/21 11:36, Hannes Laimer wrote:
> I just noticed two things while testing:
>   - if an endpoint expects a parameter called 'path'(e.g. POST /config
>       /datastore), the parameter conflicts with the api path itself

i'll fix it in the v2 by renaming the variable to 'api-path'. or
should i prefix it with 'pbs-shell' or something?
(it must be something that will not occur in the normal api)

>   - even though commands like verify(POST /admin/datastore/<name>/verify)
> or gc(POST /admin/datastore/<name>/gc) are started and run successfully,
> `Error: control socket connect failed - Connection refused (os error
> 111)` is displayed almost always at the end.

after looking at/testing the code a bit closer, i noticed it cannot
currently work as i do it:
* pbs-shell starts a worker
* pbs-shell connects to 8007 for the log
* proxy tries to query the worker status via command socket, which
   does not exists for the pbs-shell

i'll have to display the task log manually from the pbs-shell
and wait for the worker to finish

while i think it *would* be better if we let the proxy/daemon
start the worker, there is no way to detect api calls
that start a worker before actually executing it...
since it's *only* a debug tool, its still fine i think..

alternatively, we can switch to a http client for executing
the api entirely...

> 
> Note: For testing I applied the patches on
> b65dfff574f44e5a60db3903d2226f4dbe386b7a, I assume the moving of stuff
> after that should not effect the functionality
> 

i'll rebase onto master for the v2

Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool

[Thomas Lamprecht]

disclaimer: I had half an answer here but missed to send it out soon enough, it's somewhat
obsolete due to your v2 but also does not hurt to have on the list so still sending it..

On 13.09.21 12:38, Dominik Csapak wrote:
> thanks for testing!
> 
> On 9/13/21 11:36, Hannes Laimer wrote:
>> I just noticed two things while testing:
>>   - if an endpoint expects a parameter called 'path'(e.g. POST /config
>>       /datastore), the parameter conflicts with the api path itself
> 
> i'll fix it in the v2 by renaming the variable to 'api-path'. or
> should i prefix it with 'pbs-shell' or something?
> (it must be something that will not occur in the normal api)

meh... Ideally we'd use something that the API probably won't be allowed to use
any time in the future (e.g., 🛠️ ;P).

But, `api-path` sounds all right, it's unlikely to be used anytime soon, and
even if we can think through a better solution then, it's a fixed argument any
way I figure; so adapting that wouldn't be an actual breaking change.

So your v2 is fine.

> while i think it *would* be better if we let the proxy/daemon
> start the worker, there is no way to detect api calls
> that start a worker before actually executing it...
> since it's *only* a debug tool, its still fine i think..

I (low-priority) planned to add an actual UPID "type" in PVE since a bit for
API calls that return a worker, doing so in PBS could be used as indication
that a endpoint may return a worker.

> 
> alternatively, we can switch to a http client for executing
> the api entirely...

In that case I'd really like to produce a full-feature API client crate that
can also cope with PVE/PMG (not much to change there). We need that anyway
for future projects and it shouldn't be to hard, can be somewhat modeled
after the pve-api-client perl thingy in API and semantics; it would be great
though if it could cope with either hyper or something simple & sync like
ureq (once the native-tls support pull-request is merged).