* [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool
@ 2021-09-09 13:48 Dominik Csapak
2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes Dominik Csapak
` (6 more replies)
0 siblings, 7 replies; 12+ messages in thread
From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw)
To: pbs-devel
this series adds the tool 'pbs-shell', similar to 'pvesh' and 'pmgsh'
this tool is intended mainly for debugging, but can be useful for
api calls not exposed via the client/manager.
proxmox (and the dependency in proxmox-backup) need to be bumped
proxmox-backup patches 2-5 are not strictly necessary, but improve
the api so that the 'ls' command works better
(there are still some api paths to be fixed)
proxmox:
Dominik Csapak (1):
proxmox: generate_usage_str: don't require static lifetimes
proxmox/src/api/cli/format.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
proxmox-backup:
Dominik Csapak (5):
add 'pbs-shell' utility
api2: add missing token list match_all property
api2: disks/directory: refactor BASE_MOUNT_DIR
api2: disks/directory: add 'name' property to directory mount listing
api2: nodes: add missing node list api call
Makefile | 2 +
debian/pbs-shell.bc | 3 +
debian/proxmox-backup-server.bash-completion | 1 +
debian/proxmox-backup-server.install | 3 +
docs/Makefile | 8 +
docs/pbs-shell/description.rst | 3 +
docs/pbs-shell/man1.rst | 40 ++
src/api2/access/user.rs | 35 +-
src/api2/mod.rs | 4 +-
src/api2/node/disks/directory.rs | 15 +-
src/api2/node/mod.rs | 12 +-
src/bin/pbs-shell.rs | 502 +++++++++++++++++++
zsh-completions/_pbs-shell | 13 +
13 files changed, 628 insertions(+), 13 deletions(-)
create mode 100644 debian/pbs-shell.bc
create mode 100644 docs/pbs-shell/description.rst
create mode 100644 docs/pbs-shell/man1.rst
create mode 100644 src/bin/pbs-shell.rs
create mode 100644 zsh-completions/_pbs-shell
--
2.30.2
^ permalink raw reply [flat|nested] 12+ messages in thread* [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-15 7:40 ` [pbs-devel] applied: " Thomas Lamprecht 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 1/5] add 'pbs-shell' utility Dominik Csapak ` (5 subsequent siblings) 6 siblings, 1 reply; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel this prevents us from using it under certain conditions and it's actually not necessary, so drop them Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- proxmox/src/api/cli/format.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proxmox/src/api/cli/format.rs b/proxmox/src/api/cli/format.rs index 17efcd6..a4fb78d 100644 --- a/proxmox/src/api/cli/format.rs +++ b/proxmox/src/api/cli/format.rs @@ -60,7 +60,7 @@ pub fn generate_usage_str( cli_cmd: &CliCommand, format: DocumentationFormat, indent: &str, - skip_options: &'static [&'static str], + skip_options: &[&str], ) -> String { let arg_param = cli_cmd.arg_param; let fixed_param = &cli_cmd.fixed_param; -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] applied: [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes Dominik Csapak @ 2021-09-15 7:40 ` Thomas Lamprecht 0 siblings, 0 replies; 12+ messages in thread From: Thomas Lamprecht @ 2021-09-15 7:40 UTC (permalink / raw) To: Proxmox Backup Server development discussion, Dominik Csapak On 09.09.21 15:48, Dominik Csapak wrote: > this prevents us from using it under certain conditions and it's > actually not necessary, so drop them > > Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> > --- > proxmox/src/api/cli/format.rs | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > applied this one already, thanks! ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 1/5] add 'pbs-shell' utility 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 2/5] api2: add missing token list match_all property Dominik Csapak ` (4 subsequent siblings) 6 siblings, 0 replies; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel similar to pve/pmg, a user can call the api with this utility without going through the proxy/daemon, as well as list the api endpoints (with child links) and get the api description of endpoints this is mainly intended for debugging, but it is also useful for situations where some api calls do not have an equivalent in a binary and a user does not want to go through the api not implemented are the http2 api calls (since it is a separate api and it wouldn't be that easy to do) there are a few quirks though, related to the 'ls' command: i extract the 'child-link' from the property name of the 'match_all' statement of the router, but this does not always match with the property from the relevant 'get' api call so it fails there (e.g. /tape/drive ) this can be fixed in the respective api calls (e.g. by renaming the parameter that comes from the path) includes bash/zsh completion helpers and a basic manpage Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- Makefile | 2 + debian/pbs-shell.bc | 3 + debian/proxmox-backup-server.bash-completion | 1 + debian/proxmox-backup-server.install | 3 + docs/Makefile | 8 + docs/pbs-shell/description.rst | 3 + docs/pbs-shell/man1.rst | 40 ++ src/bin/pbs-shell.rs | 502 +++++++++++++++++++ zsh-completions/_pbs-shell | 13 + 9 files changed, 575 insertions(+) create mode 100644 debian/pbs-shell.bc create mode 100644 docs/pbs-shell/description.rst create mode 100644 docs/pbs-shell/man1.rst create mode 100644 src/bin/pbs-shell.rs create mode 100644 zsh-completions/_pbs-shell diff --git a/Makefile b/Makefile index c1aecf61..abeaff37 100644 --- a/Makefile +++ b/Makefile @@ -11,6 +11,7 @@ USR_BIN := \ proxmox-backup-client \ proxmox-file-restore \ pxar \ + pbs-shell \ proxmox-tape \ pmtx \ pmt @@ -172,6 +173,7 @@ $(COMPILED_BINS) $(COMPILEDIR)/dump-catalog-shell-cli $(COMPILEDIR)/docgen: .do- --bin proxmox-backup-api \ --bin proxmox-backup-proxy \ --bin proxmox-backup-manager \ + --bin pbs-shell \ --bin docgen $(CARGO) build $(CARGO_BUILD_ARGS) \ --package proxmox-backup-banner \ diff --git a/debian/pbs-shell.bc b/debian/pbs-shell.bc new file mode 100644 index 00000000..3d17187c --- /dev/null +++ b/debian/pbs-shell.bc @@ -0,0 +1,3 @@ +# pbs-shell bash completion + +complete -C 'pbs-shell bashcomplete' pbs-shell diff --git a/debian/proxmox-backup-server.bash-completion b/debian/proxmox-backup-server.bash-completion index a2165699..8d6a7047 100644 --- a/debian/proxmox-backup-server.bash-completion +++ b/debian/proxmox-backup-server.bash-completion @@ -2,3 +2,4 @@ debian/proxmox-backup-manager.bc proxmox-backup-manager debian/proxmox-tape.bc proxmox-tape debian/pmtx.bc pmtx debian/pmt.bc pmt +debian/pbs-shell.bc pbs-shell diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install index 6e2219b4..5e1071fa 100644 --- a/debian/proxmox-backup-server.install +++ b/debian/proxmox-backup-server.install @@ -14,6 +14,7 @@ usr/sbin/proxmox-backup-manager usr/bin/pmtx usr/bin/pmt usr/bin/proxmox-tape +usr/bin/pbs-shell usr/share/javascript/proxmox-backup/index.hbs usr/share/javascript/proxmox-backup/css/ext6-pbs.css usr/share/javascript/proxmox-backup/images @@ -24,6 +25,7 @@ usr/share/man/man1/proxmox-backup-proxy.1 usr/share/man/man1/proxmox-tape.1 usr/share/man/man1/pmtx.1 usr/share/man/man1/pmt.1 +usr/share/man/man1/pbs-shell.1 usr/share/man/man5/acl.cfg.5 usr/share/man/man5/datastore.cfg.5 usr/share/man/man5/user.cfg.5 @@ -38,3 +40,4 @@ usr/share/zsh/vendor-completions/_proxmox-backup-manager usr/share/zsh/vendor-completions/_proxmox-tape usr/share/zsh/vendor-completions/_pmtx usr/share/zsh/vendor-completions/_pmt +usr/share/zsh/vendor-completions/_pbs-shell diff --git a/docs/Makefile b/docs/Makefile index 5e37f7d1..e67df2ea 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -10,6 +10,7 @@ GENERATED_SYNOPSIS := \ pxar/synopsis.rst \ pmtx/synopsis.rst \ pmt/synopsis.rst \ + pbs-shell/synopsis.rst \ config/media-pool/config.rst \ config/tape/config.rst \ config/tape-job/config.rst \ @@ -24,6 +25,7 @@ MAN1_PAGES := \ pxar.1 \ pmtx.1 \ pmt.1 \ + pbs-shell.1 \ proxmox-tape.1 \ proxmox-backup-proxy.1 \ proxmox-backup-client.1 \ @@ -117,6 +119,12 @@ pmt/synopsis.rst: ${COMPILEDIR}/pmt pmt.1: pmt/man1.rst pmt/description.rst pmt/options.rst pmt/synopsis.rst rst2man $< >$@ +pbs-shell/synopsis.rst: ${COMPILEDIR}/pbs-shell + ${COMPILEDIR}/pbs-shell printdoc > pbs-shell/synopsis.rst + +pbs-shell.1: pbs-shell/man1.rst pbs-shell/description.rst pbs-shell/synopsis.rst + rst2man $< >$@ + config/datastore/config.rst: ${COMPILEDIR}/docgen ${COMPILEDIR}/docgen datastore.cfg >$@ diff --git a/docs/pbs-shell/description.rst b/docs/pbs-shell/description.rst new file mode 100644 index 00000000..8dfcae15 --- /dev/null +++ b/docs/pbs-shell/description.rst @@ -0,0 +1,3 @@ +The ``pbs-shell`` command can show and execute api calls and their parameters. +It is mainly intended for use during debugging. + diff --git a/docs/pbs-shell/man1.rst b/docs/pbs-shell/man1.rst new file mode 100644 index 00000000..d0a1d07b --- /dev/null +++ b/docs/pbs-shell/man1.rst @@ -0,0 +1,40 @@ +========================== +pbs-shell +========================== + +.. include:: ../epilog.rst + +------------------------------------------------------------- +Show and execute PBS API calls +------------------------------------------------------------- + +:Author: |AUTHOR| +:Version: Version |VERSION| +:Manual section: 1 + + +Synopsis +========== + +.. include:: synopsis.rst + + +Common Options +============== + +Commands generating output supports the ``--output-format`` +parameter. It accepts the following values: + +:``text``: Text format (default). Human readable. + +:``json``: JSON (single line). + +:``json-pretty``: JSON (multiple lines, nicely formatted). + + +Description +============ + +.. include:: description.rst + +.. include:: ../pbs-copyright.rst diff --git a/src/bin/pbs-shell.rs b/src/bin/pbs-shell.rs new file mode 100644 index 00000000..ce64617b --- /dev/null +++ b/src/bin/pbs-shell.rs @@ -0,0 +1,502 @@ +use anyhow::{bail, format_err, Error}; +use hyper::Method; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use std::collections::HashMap; + +use proxmox::api::{ + api, + cli::*, + format::DocumentationFormat, + schema::{parse_parameter_strings, ApiType, ParameterSchema, Schema}, + ApiHandler, ApiMethod, RpcEnvironment, SubRoute, +}; + +use pbs_client::{connect_to_localhost, display_task_log}; + +const PROG_NAME: &str = "pbs-shell"; + +fn complete_api_path(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, None).await }) +} + +fn complete_api_path_get(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, Some("r")).await }) +} + +fn complete_api_path_set(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, Some("w")).await }) +} + +fn complete_api_path_create(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, Some("c")).await }) +} + +fn complete_api_path_delete(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, Some("d")).await }) +} + +fn complete_api_path_ls(complete_me: &str, _map: &HashMap<String, String>) -> Vec<String> { + pbs_runtime::main(async { complete_api_path_do(complete_me, Some("D")).await }) +} + +async fn complete_api_path_do(mut complete_me: &str, capability: Option<&str>) -> Vec<String> { + if complete_me.is_empty() { + complete_me = "/"; + } + + let mut list = Vec::new(); + + let mut lookup_path = complete_me.to_string(); + let mut filter = ""; + let last_path_index = complete_me.rfind('/'); + if let Some(index) = last_path_index { + if index != complete_me.len() - 1 { + lookup_path = complete_me[..(index + 1)].to_string(); + if index < complete_me.len() - 1 { + filter = &complete_me[(index + 1)..]; + } + } + } + + let uid = nix::unistd::Uid::current(); + + let username = match nix::unistd::User::from_uid(uid) { + Ok(Some(user)) => user.name, + _ => "root@pam".to_string(), + }; + let mut rpcenv = CliEnvironment::new(); + rpcenv.set_auth_id(Some(format!("{}@pam", username))); + + while let Ok(children) = get_api_children(lookup_path.clone(), &mut rpcenv).await { + let old_len = list.len(); + for entry in children { + let name = entry.name; + let caps = entry.capabilities; + + if filter.is_empty() || name.starts_with(filter) { + let mut path = format!("{}{}", lookup_path, name); + if caps.contains('D') { + path.push('/'); + list.push(path.clone()); + } else if let Some(cap) = capability { + if caps.contains(cap) { + list.push(path); + } + } else { + list.push(path); + } + } + } + + if list.len() == 1 && old_len != 1 && list[0].ends_with('/') { + // we added only one match and it was a directory, lookup again + lookup_path = list[0].clone(); + filter = ""; + continue; + } + + break; + } + + list +} + +async fn get_child_links( + path: &str, + rpcenv: &mut dyn RpcEnvironment, +) -> Result<Vec<String>, Error> { + let mut uri_param = HashMap::new(); + let (path, components) = proxmox_backup::tools::normalize_uri_path(&path)?; + + let info = &proxmox_backup::api2::ROUTER + .find_route(&components, &mut uri_param) + .ok_or_else(|| format_err!("no such resource"))?; + + match info.subroute { + Some(SubRoute::Map(map)) => Ok(map.iter().map(|(name, _)| name.to_string()).collect()), + Some(SubRoute::MatchAll { param_name, .. }) => { + let get_call = info.get.ok_or_else(|| format_err!("no such resource"))?; + let list = call_api(get_call, rpcenv, serde_json::to_value(uri_param)?).await?; + Ok(list + .as_array() + .ok_or_else(|| format_err!("{} did not return an array", path))? + .iter() + .map(|item| { + item[param_name] + .as_str() + .map(|c| c.to_string()) + .ok_or_else(|| format_err!("no such property {}", param_name)) + }) + .collect::<Result<Vec<_>, _>>()?) + } + None => bail!("link does not define child links"), + } +} + +fn get_api_method( + method: Method, + path: &str, +) -> Result<(&'static ApiMethod, HashMap<String, String>), Error> { + let mut uri_param = HashMap::new(); + let (path, components) = proxmox_backup::tools::normalize_uri_path(&path)?; + if let Some(method) = + &proxmox_backup::api2::ROUTER.find_method(&components, method.clone(), &mut uri_param) + { + Ok((method, uri_param)) + } else { + bail!("no {} handler defined for '{}'", method, path); + } +} + +fn merge_parameters( + uri_param: HashMap<String, String>, + param: Value, + schema: ParameterSchema, +) -> Result<Value, Error> { + let mut param_list: Vec<(String, String)> = vec![]; + + for (k, v) in uri_param { + param_list.push((k.clone(), v.clone())); + } + + if let Some(map) = param.as_object() { + for (k, v) in map { + param_list.push((k.clone(), v.as_str().unwrap().to_string())); + } + } + + let params = parse_parameter_strings(¶m_list, schema, true)?; + + Ok(params) +} + +async fn call_api( + method: &'static ApiMethod, + rpcenv: &mut dyn RpcEnvironment, + params: Value, +) -> Result<Value, Error> { + match method.handler { + ApiHandler::AsyncHttp(_handler) => { + bail!("not implemented"); + } + ApiHandler::Sync(handler) => (handler)(params, method, rpcenv), + ApiHandler::Async(handler) => (handler)(params, method, rpcenv).await, + } +} + +async fn call_api_and_format_result( + method: Method, + path: String, + mut param: Value, + rpcenv: &mut dyn RpcEnvironment, +) -> Result<(), Error> { + let mut output_format = extract_output_format(&mut param); + let (method, uri_param) = get_api_method(method, &path)?; + let params = merge_parameters(uri_param, param, method.parameters)?; + + let mut result = call_api(method, rpcenv, params).await?; + + if output_format == "text" { + if let Some(upid) = result.as_str() { + let mut client = connect_to_localhost()?; + display_task_log(&mut client, upid, true).await?; + return Ok(()); + } + } + + let options = default_table_format_options(); + let return_type = &method.returns; + if matches!(return_type.schema, Schema::Null) { + output_format = "json-pretty".to_string(); + } + + format_and_print_result_full(&mut result, return_type, &output_format, &options); + + Ok(()) +} + +#[api( + input: { + additional_properties: true, + properties: { + path: { + type: String, + description: "API path.", + }, + "output-format": { + schema: OUTPUT_FORMAT, + optional: true, + }, + }, + }, +)] +/// Call API PUT on <path> +async fn set(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { + call_api_and_format_result(Method::PUT, path, param, rpcenv).await +} + +#[api( + input: { + additional_properties: true, + properties: { + path: { + type: String, + description: "API path.", + }, + "output-format": { + schema: OUTPUT_FORMAT, + optional: true, + }, + }, + }, +)] +/// Call API POST on <path> +async fn create(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { + call_api_and_format_result(Method::POST, path, param, rpcenv).await +} + +#[api( + input: { + additional_properties: true, + properties: { + path: { + type: String, + description: "API path.", + }, + "output-format": { + schema: OUTPUT_FORMAT, + optional: true, + }, + }, + }, +)] +/// Call API GET on <path> +async fn get(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { + call_api_and_format_result(Method::GET, path, param, rpcenv).await +} + +#[api( + input: { + additional_properties: true, + properties: { + path: { + type: String, + description: "API path.", + }, + "output-format": { + schema: OUTPUT_FORMAT, + optional: true, + }, + }, + }, +)] +/// Call API DELETE on <path> +async fn delete(path: String, param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { + call_api_and_format_result(Method::DELETE, path, param, rpcenv).await +} + +#[api( + input: { + properties: { + path: { + type: String, + description: "API path.", + }, + verbose: { + type: Boolean, + description: "Verbose output format.", + optional: true, + default: false, + } + }, + }, +)] +/// Get API usage information for <path> +async fn usage( + path: String, + verbose: bool, + _param: Value, + _rpcenv: &mut dyn RpcEnvironment, +) -> Result<(), Error> { + let docformat = if verbose { + DocumentationFormat::Full + } else { + DocumentationFormat::Short + }; + let mut found = false; + for command in &["get", "set", "create", "delete"] { + let http_method = match *command { + "get" => Method::GET, + "set" => Method::PUT, + "create" => Method::POST, + "delete" => Method::DELETE, + _ => unreachable!(), + }; + let (info, uri_params) = match get_api_method(http_method, &path) { + Ok(some) => some, + Err(_) => continue, + }; + found = true; + + let skip_params: Vec<&str> = uri_params.keys().map(|s| &**s).collect(); + + let cmd = CliCommand::new(info); + let prefix = format!("USAGE: {} {} {}", PROG_NAME, command, path); + + print!( + "{}", + generate_usage_str(&prefix, &cmd, docformat, "", &skip_params) + ); + } + + if !found { + bail!("no such resource '{}'", path); + } + Ok(()) +} + +#[api()] +#[derive(Debug, Serialize, Deserialize)] +/// A child link with capabilities +struct ApiDirEntry { + /// The name of the link + name: String, + /// The capabilities of the path (format Drwcd) + capabilities: String, +} + +const LS_SCHEMA: &proxmox::api::schema::Schema = + &proxmox::api::schema::ArraySchema::new("List of child links", &ApiDirEntry::API_SCHEMA) + .schema(); + +async fn get_api_children( + path: String, + rpcenv: &mut dyn RpcEnvironment, +) -> Result<Vec<ApiDirEntry>, Error> { + let mut res = Vec::new(); + for link in get_child_links(&path, rpcenv).await? { + let path = format!("{}/{}", path, link); + let (path, _) = proxmox_backup::tools::normalize_uri_path(&path)?; + let mut cap = String::new(); + + if get_child_links(&path, rpcenv).await.is_ok() { + cap.push('D'); + } else { + cap.push('-'); + } + + let cap_list = &[ + (Method::GET, 'r'), + (Method::PUT, 'w'), + (Method::POST, 'c'), + (Method::DELETE, 'd'), + ]; + + for (method, c) in cap_list { + if get_api_method(method.clone(), &path).is_ok() { + cap.push(*c); + } else { + cap.push('-'); + } + } + + res.push(ApiDirEntry { + name: link.to_string(), + capabilities: cap, + }); + } + + Ok(res) +} + +#[api( + input: { + properties: { + path: { + type: String, + description: "API path.", + }, + "output-format": { + schema: OUTPUT_FORMAT, + optional: true, + }, + }, + }, +)] +/// Get API usage information for <path> +async fn ls(path: String, mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> { + let output_format = extract_output_format(&mut param); + + let options = TableFormatOptions::new() + .noborder(true) + .noheader(true) + .sortby("name", false); + + let res = get_api_children(path, rpcenv).await?; + + format_and_print_result_full( + &mut serde_json::to_value(res)?, + &proxmox::api::schema::ReturnType { + optional: false, + schema: &LS_SCHEMA, + }, + &output_format, + &options, + ); + + Ok(()) +} + +fn main() -> Result<(), Error> { + let cmd_def = CliCommandMap::new() + .insert( + "get", + CliCommand::new(&API_METHOD_GET) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path_get), + ) + .insert( + "set", + CliCommand::new(&API_METHOD_SET) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path_set), + ) + .insert( + "create", + CliCommand::new(&API_METHOD_CREATE) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path_create), + ) + .insert( + "delete", + CliCommand::new(&API_METHOD_DELETE) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path_delete), + ) + .insert( + "ls", + CliCommand::new(&API_METHOD_LS) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path_ls), + ) + .insert( + "usage", + CliCommand::new(&API_METHOD_USAGE) + .arg_param(&["path"]) + .completion_cb("path", complete_api_path), + ); + + let uid = nix::unistd::Uid::current(); + + let username = match nix::unistd::User::from_uid(uid)? { + Some(user) => user.name, + None => bail!("unable to get user name"), + }; + let mut rpcenv = CliEnvironment::new(); + rpcenv.set_auth_id(Some(format!("{}@pam", username))); + + pbs_runtime::main(run_async_cli_command(cmd_def, rpcenv)); + Ok(()) +} diff --git a/zsh-completions/_pbs-shell b/zsh-completions/_pbs-shell new file mode 100644 index 00000000..507f15ae --- /dev/null +++ b/zsh-completions/_pbs-shell @@ -0,0 +1,13 @@ +#compdef _pbs-shell() pbs-shell + +function _pbs-shell() { + local cwords line point cmd curr prev + cwords=${#words[@]} + line=$words + point=${#line} + cmd=${words[1]} + curr=${words[cwords]} + prev=${words[cwords-1]} + compadd -- $(COMP_CWORD="$cwords" COMP_LINE="$line" COMP_POINT="$point" \ + pbs-shell bashcomplete "$cmd" "$curr" "$prev") +} -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 2/5] api2: add missing token list match_all property 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 1/5] add 'pbs-shell' utility Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 3/5] api2: disks/directory: refactor BASE_MOUNT_DIR Dominik Csapak ` (3 subsequent siblings) 6 siblings, 0 replies; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel to have the proper link between the token list and the sub routes in the api, include the 'tokenname' property in the token listing Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- src/api2/access/user.rs | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/src/api2/access/user.rs b/src/api2/access/user.rs index 6a2fe83c..70065986 100644 --- a/src/api2/access/user.rs +++ b/src/api2/access/user.rs @@ -655,6 +655,21 @@ pub fn delete_token( Ok(()) } +#[api( + properties: { + tokenname: { type: Tokenname }, + token: { type: ApiToken }, + } +)] +#[derive(Serialize, Deserialize)] +/// A Token Entry of a user +pub struct TokenInfo { + /// The Token name + pub tokenname: Tokenname, + #[serde(flatten)] + pub token: ApiToken, +} + #[api( input: { properties: { @@ -666,7 +681,7 @@ pub fn delete_token( returns: { description: "List user's API tokens (with config digest).", type: Array, - items: { type: ApiToken }, + items: { type: TokenInfo }, }, access: { permission: &Permission::Or(&[ @@ -680,7 +695,7 @@ pub fn list_tokens( userid: Userid, _info: &ApiMethod, mut rpcenv: &mut dyn RpcEnvironment, -) -> Result<Vec<ApiToken>, Error> { +) -> Result<Vec<TokenInfo>, Error> { let (config, digest) = crate::config::user::config()?; @@ -688,15 +703,21 @@ pub fn list_tokens( rpcenv["digest"] = proxmox::tools::digest_to_hex(&digest).into(); - let filter_by_owner = |token: &ApiToken| { - if token.tokenid.is_token() { - token.tokenid.user() == &userid + let filter_by_owner = |token: ApiToken| { + if token.tokenid.is_token() && token.tokenid.user() == &userid { + let tokenname = token.tokenid.tokenname().unwrap().to_owned(); + Some(TokenInfo { + tokenname, + token, + }) } else { - false + None } }; - Ok(list.into_iter().filter(filter_by_owner).collect()) + let res = list.into_iter().filter_map(filter_by_owner).collect(); + + Ok(res) } const TOKEN_ITEM_ROUTER: Router = Router::new() -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 3/5] api2: disks/directory: refactor BASE_MOUNT_DIR 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak ` (2 preceding siblings ...) 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 2/5] api2: add missing token list match_all property Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 4/5] api2: disks/directory: add 'name' property to directory mount listing Dominik Csapak ` (2 subsequent siblings) 6 siblings, 0 replies; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- src/api2/node/disks/directory.rs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs index 0cb6f4e2..dd6912d3 100644 --- a/src/api2/node/disks/directory.rs +++ b/src/api2/node/disks/directory.rs @@ -22,6 +22,8 @@ use crate::server::WorkerTask; use crate::config::datastore::{self, DataStoreConfig}; use pbs_config::open_backup_lockfile; +const BASE_MOUNT_DIR: &str = "/mnt/datastore/"; + #[api( properties: { "filesystem": { @@ -146,7 +148,7 @@ pub fn create_datastore_disk( bail!("disk '{}' is already in use.", disk); } - let mount_point = format!("/mnt/datastore/{}", &name); + let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name); // check if the default path does exist already and bail if it does let default_path = std::path::PathBuf::from(&mount_point); @@ -221,7 +223,7 @@ pub fn create_datastore_disk( /// Remove a Filesystem mounted under '/mnt/datastore/<name>'.". pub fn delete_datastore_disk(name: String) -> Result<(), Error> { - let path = format!("/mnt/datastore/{}", name); + let path = format!("{}{}", BASE_MOUNT_DIR, name); // path of datastore cannot be changed let (config, _) = crate::config::datastore::config()?; let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?; -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 4/5] api2: disks/directory: add 'name' property to directory mount listing 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak ` (3 preceding siblings ...) 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 3/5] api2: disks/directory: refactor BASE_MOUNT_DIR Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call Dominik Csapak 2021-09-13 9:36 ` [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Hannes Laimer 6 siblings, 0 replies; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel so that we have the properties that match with 'match_all' Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- src/api2/node/disks/directory.rs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs index dd6912d3..f9282903 100644 --- a/src/api2/node/disks/directory.rs +++ b/src/api2/node/disks/directory.rs @@ -38,6 +38,8 @@ const BASE_MOUNT_DIR: &str = "/mnt/datastore/"; pub struct DatastoreMountInfo { /// The path of the mount unit. pub unitfile: String, + /// The name of the mount + pub name: String, /// The mount path. pub path: String, /// The mounted device. @@ -86,8 +88,15 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> { let config = systemd::config::parse_systemd_mount(&unitfile)?; let data: SystemdMountSection = config.lookup("Mount", "Mount")?; + let name = data + .Where + .strip_prefix(BASE_MOUNT_DIR) + .unwrap_or_else(|| &data.Where) + .to_string(); + list.push(DatastoreMountInfo { unitfile, + name, device: data.What, path: data.Where, filesystem: data.Type, -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak ` (4 preceding siblings ...) 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 4/5] api2: disks/directory: add 'name' property to directory mount listing Dominik Csapak @ 2021-09-09 13:48 ` Dominik Csapak 2021-09-15 9:44 ` [pbs-devel] applied: " Thomas Lamprecht 2021-09-13 9:36 ` [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Hannes Laimer 6 siblings, 1 reply; 12+ messages in thread From: Dominik Csapak @ 2021-09-09 13:48 UTC (permalink / raw) To: pbs-devel to have an api call for api path traversal Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> --- src/api2/mod.rs | 4 +--- src/api2/node/mod.rs | 12 +++++++++++- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/src/api2/mod.rs b/src/api2/mod.rs index 132e2c2a..0ede4165 100644 --- a/src/api2/mod.rs +++ b/src/api2/mod.rs @@ -18,14 +18,12 @@ use proxmox::api::router::SubdirMap; use proxmox::api::Router; use proxmox::list_subdirs_api_method; -const NODES_ROUTER: Router = Router::new().match_all("node", &node::ROUTER); - const SUBDIRS: SubdirMap = &[ ("access", &access::ROUTER), ("admin", &admin::ROUTER), ("backup", &backup::ROUTER), ("config", &config::ROUTER), - ("nodes", &NODES_ROUTER), + ("nodes", &node::ROUTER), ("ping", &ping::ROUTER), ("pull", &pull::ROUTER), ("reader", &reader::ROUTER), diff --git a/src/api2/node/mod.rs b/src/api2/node/mod.rs index f1a17934..194ec920 100644 --- a/src/api2/node/mod.rs +++ b/src/api2/node/mod.rs @@ -315,6 +315,12 @@ fn upgrade_to_websocket( .boxed() } +#[api] +/// List Nodes (only for compatiblity) +fn list_nodes() -> Result<Value, Error> { + Ok(json!([ { "node": proxmox::tools::nodename().to_string() } ])) +} + pub const SUBDIRS: SubdirMap = &[ ("apt", &apt::ROUTER), ("certificates", &certificates::ROUTER), @@ -338,6 +344,10 @@ pub const SUBDIRS: SubdirMap = &[ ), ]; -pub const ROUTER: Router = Router::new() +pub const ITEM_ROUTER: Router = Router::new() .get(&list_subdirs_api_method!(SUBDIRS)) .subdirs(SUBDIRS); + +pub const ROUTER: Router = Router::new() + .get(&API_METHOD_LIST_NODES) + .match_all("node", &ITEM_ROUTER); -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pbs-devel] applied: [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call Dominik Csapak @ 2021-09-15 9:44 ` Thomas Lamprecht 0 siblings, 0 replies; 12+ messages in thread From: Thomas Lamprecht @ 2021-09-15 9:44 UTC (permalink / raw) To: Proxmox Backup Server development discussion, Dominik Csapak On 09.09.21 15:48, Dominik Csapak wrote: > to have an api call for api path traversal > > Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> > --- > src/api2/mod.rs | 4 +--- > src/api2/node/mod.rs | 12 +++++++++++- > 2 files changed, 12 insertions(+), 4 deletions(-) > > applied this rather independent patch already, thanks! Any thoughts on providing some resource usage and status info like pve has? # pvesh get /nodes --output-format json-pretty [ { "cpu" : 0.00688381192943118, "disk" : 81085153280, "id" : "node/d7", "level" : "c", "maxcpu" : 16, "maxdisk" : 133204897792, "maxmem" : 6101630976, "mem" : 1656844288, "node" : "d7", "ssl_fingerprint" : "8A:DF:09:A5:D8:CC:42:AD:B1:35:3B:E7:03:30:30:3C:C2:BD:BF:94:FA:F3:CD:46:62:92:6D:34:00:56:CC:8D", "status" : "online", "type" : "node", "uptime" : 1990844 } ] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak ` (5 preceding siblings ...) 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call Dominik Csapak @ 2021-09-13 9:36 ` Hannes Laimer 2021-09-13 10:38 ` Dominik Csapak 6 siblings, 1 reply; 12+ messages in thread From: Hannes Laimer @ 2021-09-13 9:36 UTC (permalink / raw) To: Proxmox Backup Server development discussion, Dominik Csapak I just noticed two things while testing: - if an endpoint expects a parameter called 'path'(e.g. POST /config /datastore), the parameter conflicts with the api path itself - even though commands like verify(POST /admin/datastore/<name>/verify) or gc(POST /admin/datastore/<name>/gc) are started and run successfully, `Error: control socket connect failed - Connection refused (os error 111)` is displayed almost always at the end. Note: For testing I applied the patches on b65dfff574f44e5a60db3903d2226f4dbe386b7a, I assume the moving of stuff after that should not effect the functionality Am 09.09.21 um 15:48 schrieb Dominik Csapak: > this series adds the tool 'pbs-shell', similar to 'pvesh' and 'pmgsh' > this tool is intended mainly for debugging, but can be useful for > api calls not exposed via the client/manager. > > proxmox (and the dependency in proxmox-backup) need to be bumped > > proxmox-backup patches 2-5 are not strictly necessary, but improve > the api so that the 'ls' command works better > (there are still some api paths to be fixed) > > proxmox: > > Dominik Csapak (1): > proxmox: generate_usage_str: don't require static lifetimes > > proxmox/src/api/cli/format.rs | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > proxmox-backup: > > Dominik Csapak (5): > add 'pbs-shell' utility > api2: add missing token list match_all property > api2: disks/directory: refactor BASE_MOUNT_DIR > api2: disks/directory: add 'name' property to directory mount listing > api2: nodes: add missing node list api call > > Makefile | 2 + > debian/pbs-shell.bc | 3 + > debian/proxmox-backup-server.bash-completion | 1 + > debian/proxmox-backup-server.install | 3 + > docs/Makefile | 8 + > docs/pbs-shell/description.rst | 3 + > docs/pbs-shell/man1.rst | 40 ++ > src/api2/access/user.rs | 35 +- > src/api2/mod.rs | 4 +- > src/api2/node/disks/directory.rs | 15 +- > src/api2/node/mod.rs | 12 +- > src/bin/pbs-shell.rs | 502 +++++++++++++++++++ > zsh-completions/_pbs-shell | 13 + > 13 files changed, 628 insertions(+), 13 deletions(-) > create mode 100644 debian/pbs-shell.bc > create mode 100644 docs/pbs-shell/description.rst > create mode 100644 docs/pbs-shell/man1.rst > create mode 100644 src/bin/pbs-shell.rs > create mode 100644 zsh-completions/_pbs-shell > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool 2021-09-13 9:36 ` [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Hannes Laimer @ 2021-09-13 10:38 ` Dominik Csapak 2021-09-15 5:25 ` Thomas Lamprecht 0 siblings, 1 reply; 12+ messages in thread From: Dominik Csapak @ 2021-09-13 10:38 UTC (permalink / raw) To: Hannes Laimer, Proxmox Backup Server development discussion thanks for testing! On 9/13/21 11:36, Hannes Laimer wrote: > I just noticed two things while testing: > - if an endpoint expects a parameter called 'path'(e.g. POST /config > /datastore), the parameter conflicts with the api path itself i'll fix it in the v2 by renaming the variable to 'api-path'. or should i prefix it with 'pbs-shell' or something? (it must be something that will not occur in the normal api) > - even though commands like verify(POST /admin/datastore/<name>/verify) > or gc(POST /admin/datastore/<name>/gc) are started and run successfully, > `Error: control socket connect failed - Connection refused (os error > 111)` is displayed almost always at the end. after looking at/testing the code a bit closer, i noticed it cannot currently work as i do it: * pbs-shell starts a worker * pbs-shell connects to 8007 for the log * proxy tries to query the worker status via command socket, which does not exists for the pbs-shell i'll have to display the task log manually from the pbs-shell and wait for the worker to finish while i think it *would* be better if we let the proxy/daemon start the worker, there is no way to detect api calls that start a worker before actually executing it... since it's *only* a debug tool, its still fine i think.. alternatively, we can switch to a http client for executing the api entirely... > > Note: For testing I applied the patches on > b65dfff574f44e5a60db3903d2226f4dbe386b7a, I assume the moving of stuff > after that should not effect the functionality > i'll rebase onto master for the v2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool 2021-09-13 10:38 ` Dominik Csapak @ 2021-09-15 5:25 ` Thomas Lamprecht 0 siblings, 0 replies; 12+ messages in thread From: Thomas Lamprecht @ 2021-09-15 5:25 UTC (permalink / raw) To: Proxmox Backup Server development discussion, Dominik Csapak, Hannes Laimer disclaimer: I had half an answer here but missed to send it out soon enough, it's somewhat obsolete due to your v2 but also does not hurt to have on the list so still sending it.. On 13.09.21 12:38, Dominik Csapak wrote: > thanks for testing! > > On 9/13/21 11:36, Hannes Laimer wrote: >> I just noticed two things while testing: >> - if an endpoint expects a parameter called 'path'(e.g. POST /config >> /datastore), the parameter conflicts with the api path itself > > i'll fix it in the v2 by renaming the variable to 'api-path'. or > should i prefix it with 'pbs-shell' or something? > (it must be something that will not occur in the normal api) meh... Ideally we'd use something that the API probably won't be allowed to use any time in the future (e.g., 🛠️ ;P). But, `api-path` sounds all right, it's unlikely to be used anytime soon, and even if we can think through a better solution then, it's a fixed argument any way I figure; so adapting that wouldn't be an actual breaking change. So your v2 is fine. > while i think it *would* be better if we let the proxy/daemon > start the worker, there is no way to detect api calls > that start a worker before actually executing it... > since it's *only* a debug tool, its still fine i think.. I (low-priority) planned to add an actual UPID "type" in PVE since a bit for API calls that return a worker, doing so in PBS could be used as indication that a endpoint may return a worker. > > alternatively, we can switch to a http client for executing > the api entirely... In that case I'd really like to produce a full-feature API client crate that can also cope with PVE/PMG (not much to change there). We need that anyway for future projects and it shouldn't be to hard, can be somewhat modeled after the pve-api-client perl thingy in API and semantics; it would be great though if it could cope with either hyper or something simple & sync like ureq (once the native-tls support pull-request is merged). ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-09-15 9:45 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-09 13:48 [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox 1/1] proxmox: generate_usage_str: don't require static lifetimes Dominik Csapak 2021-09-15 7:40 ` [pbs-devel] applied: " Thomas Lamprecht 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 1/5] add 'pbs-shell' utility Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 2/5] api2: add missing token list match_all property Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 3/5] api2: disks/directory: refactor BASE_MOUNT_DIR Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 4/5] api2: disks/directory: add 'name' property to directory mount listing Dominik Csapak 2021-09-09 13:48 ` [pbs-devel] [PATCH proxmox-backup 5/5] api2: nodes: add missing node list api call Dominik Csapak 2021-09-15 9:44 ` [pbs-devel] applied: " Thomas Lamprecht 2021-09-13 9:36 ` [pbs-devel] [PATCH proxmox/proxmox-backup] add 'pbs-shell' tool Hannes Laimer 2021-09-13 10:38 ` Dominik Csapak 2021-09-15 5:25 ` Thomas Lamprecht
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox