* [pbs-devel] [PATCH vma-to-pbs v3 0/2] add support for notes and logs @ 2024-07-10 14:57 Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 1/2] pass args by reference to create_pbs_backup_task Filip Schauer ` (2 more replies) 0 siblings, 3 replies; 4+ messages in thread From: Filip Schauer @ 2024-07-10 14:57 UTC (permalink / raw) To: pbs-devel Changes since v2: * Split the create_pbs_backup_task argument change into its own commit * Create the HttpClient directly to avoid writing the password into an environment variable * Put the client setup into its own helper function and call it from backup_vma_to_pbs along with set_notes and upload_log Changes since v1: * Set notes and upload log separately via a new connection to the API, instead of hacking notes into the proxmox-backup-qemu C functions and uploading the log as a config. vma-to-pbs: Filip Schauer (2): pass args by reference to create_pbs_backup_task add support for notes and logs Cargo.toml | 8 +++ src/main.rs | 16 ++++++ src/vma2pbs.rs | 133 +++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 147 insertions(+), 10 deletions(-) Summary over all repositories: 3 files changed, 147 insertions(+), 10 deletions(-) -- Generated by git-murpp 0.6.0 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v3 1/2] pass args by reference to create_pbs_backup_task 2024-07-10 14:57 [pbs-devel] [PATCH vma-to-pbs v3 0/2] add support for notes and logs Filip Schauer @ 2024-07-10 14:57 ` Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 2/2] add support for notes and logs Filip Schauer 2024-07-11 8:50 ` [pbs-devel] applied-series: [PATCH vma-to-pbs v3 0/2] " Fabian Grünbichler 2 siblings, 0 replies; 4+ messages in thread From: Filip Schauer @ 2024-07-10 14:57 UTC (permalink / raw) To: pbs-devel Signed-off-by: Filip Schauer <f.schauer@proxmox.com> --- src/vma2pbs.rs | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs index 199cf50..35abdcd 100644 --- a/src/vma2pbs.rs +++ b/src/vma2pbs.rs @@ -52,7 +52,7 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err bail!("{function_name} failed: {pbs_err_str}"); } -fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> { +fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> { println!("PBS repository: {}", args.pbs_repository); if let Some(ns) = &args.namespace { println!("PBS namespace: {}", ns); @@ -65,22 +65,31 @@ fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackup let mut pbs_err: *mut c_char = ptr::null_mut(); - let pbs_repository_cstr = CString::new(args.pbs_repository)?; - let ns_cstr = CString::new(args.namespace.unwrap_or("".to_string()))?; - let backup_id_cstr = CString::new(args.backup_id)?; - let pbs_password_cstr = CString::new(args.pbs_password)?; - let fingerprint_cstr = CString::new(args.fingerprint)?; - let keyfile_cstr = args.keyfile.map(|v| CString::new(v).unwrap()); + let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?; + let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?; + let backup_id_cstr = CString::new(args.backup_id.as_str())?; + let pbs_password_cstr = CString::new(args.pbs_password.as_str())?; + let fingerprint_cstr = CString::new(args.fingerprint.as_str())?; + let keyfile_cstr = args + .keyfile + .as_ref() + .map(|v| CString::new(v.as_str()).unwrap()); let keyfile_ptr = keyfile_cstr .as_ref() .map(|v| v.as_ptr()) .unwrap_or(ptr::null()); - let key_password_cstr = args.key_password.map(|v| CString::new(v).unwrap()); + let key_password_cstr = args + .key_password + .as_ref() + .map(|v| CString::new(v.as_str()).unwrap()); let key_password_ptr = key_password_cstr .as_ref() .map(|v| v.as_ptr()) .unwrap_or(ptr::null()); - let master_keyfile_cstr = args.master_keyfile.map(|v| CString::new(v).unwrap()); + let master_keyfile_cstr = args + .master_keyfile + .as_ref() + .map(|v| CString::new(v.as_str()).unwrap()); let master_keyfile_ptr = master_keyfile_cstr .as_ref() .map(|v| v.as_ptr()) @@ -353,7 +362,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> { }; let vma_reader = VmaReader::new(vma_file)?; - let pbs = create_pbs_backup_task(args)?; + let pbs = create_pbs_backup_task(&args)?; defer! { proxmox_backup_disconnect(pbs); -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v3 2/2] add support for notes and logs 2024-07-10 14:57 [pbs-devel] [PATCH vma-to-pbs v3 0/2] add support for notes and logs Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 1/2] pass args by reference to create_pbs_backup_task Filip Schauer @ 2024-07-10 14:57 ` Filip Schauer 2024-07-11 8:50 ` [pbs-devel] applied-series: [PATCH vma-to-pbs v3 0/2] " Fabian Grünbichler 2 siblings, 0 replies; 4+ messages in thread From: Filip Schauer @ 2024-07-10 14:57 UTC (permalink / raw) To: pbs-devel Allow the user to specify a notes file and a log file to associate with the backup Signed-off-by: Filip Schauer <f.schauer@proxmox.com> --- Cargo.toml | 8 ++++ src/main.rs | 16 ++++++++ src/vma2pbs.rs | 104 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 128 insertions(+) diff --git a/Cargo.toml b/Cargo.toml index 0111362..c62b5e0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,14 +7,22 @@ edition = "2021" [dependencies] anyhow = "1.0" bincode = "1.3" +hyper = "0.14.5" pico-args = "0.4" md5 = "0.7.0" scopeguard = "1.1.0" serde = "1.0" +serde_json = "1.0" serde-big-array = "0.4.1" +proxmox-async = "0.4" proxmox-io = "1.0.1" proxmox-sys = "0.5.0" proxmox-time = "2" +pbs-api-types = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-api-types" } +pbs-client = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-client" } +pbs-datastore = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-datastore" } +pbs-key-config = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-key-config" } +pbs-tools = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-tools" } proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" } diff --git a/src/main.rs b/src/main.rs index 2653d3e..de789c1 100644 --- a/src/main.rs +++ b/src/main.rs @@ -37,6 +37,10 @@ Options: Password file --key-password-file <KEY_PASSWORD_FILE> Key password file + [--notes-file <NOTES_FILE>] + File containing a comment/notes + [--log-file <LOG_FILE>] + Log file -h, --help Print help -V, --version @@ -93,6 +97,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> { let encrypt = args.contains(["-e", "--encrypt"]); let password_file: Option<OsString> = args.opt_value_from_str("--password-file")?; let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?; + let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?; + let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?; match (encrypt, keyfile.is_some()) { (true, false) => bail!("--encrypt requires a --keyfile!"), @@ -170,6 +176,14 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> { None }; + let notes = if let Some(notes_file) = notes_file { + let notes = std::fs::read_to_string(notes_file).context("Could not read notes file")?; + + Some(notes) + } else { + None + }; + let options = BackupVmaToPbsArgs { vma_file_path: vma_file_path.cloned(), pbs_repository, @@ -183,6 +197,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> { fingerprint, compress, encrypt, + notes, + log_file_path, }; Ok(options) diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs index 35abdcd..d2ce437 100644 --- a/src/vma2pbs.rs +++ b/src/vma2pbs.rs @@ -8,6 +8,12 @@ use std::ptr; use std::time::SystemTime; use anyhow::{anyhow, bail, Error}; +use pbs_api_types::{BackupDir, BackupNamespace, BackupType}; +use pbs_client::{BackupRepository, HttpClient, HttpClientOptions}; +use pbs_datastore::DataBlob; +use pbs_key_config::decrypt_key; +use pbs_tools::crypt_config::CryptConfig; +use proxmox_async::runtime::block_on; use proxmox_backup_qemu::{ capi_types::ProxmoxBackupHandle, proxmox_backup_add_config, proxmox_backup_close_image, proxmox_backup_connect, proxmox_backup_disconnect, proxmox_backup_finish, @@ -16,6 +22,7 @@ use proxmox_backup_qemu::{ }; use proxmox_time::epoch_to_rfc3339; use scopeguard::defer; +use serde_json::Value; use crate::vma::VmaReader; @@ -34,6 +41,8 @@ pub struct BackupVmaToPbsArgs { pub fingerprint: String, pub compress: bool, pub encrypt: bool, + pub notes: Option<String>, + pub log_file_path: Option<OsString>, } #[derive(Copy, Clone)] @@ -352,6 +361,89 @@ where Ok(()) } +fn pbs_client_setup(args: &BackupVmaToPbsArgs) -> Result<(HttpClient, String, Value), Error> { + let repo: BackupRepository = args.pbs_repository.parse()?; + let options = HttpClientOptions::new_interactive( + Some(args.pbs_password.clone()), + Some(args.fingerprint.clone()), + ); + let client = HttpClient::new(repo.host(), repo.port(), repo.auth_id(), options)?; + + let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time)); + + let namespace = match &args.namespace { + Some(namespace) => BackupNamespace::new(namespace)?, + None => BackupNamespace::root(), + }; + + let mut request_args = serde_json::to_value(backup_dir)?; + if !namespace.is_root() { + request_args["ns"] = serde_json::to_value(namespace)?; + } + + Ok((client, repo.store().to_owned(), request_args)) +} + +fn upload_log( + client: &HttpClient, + args: &BackupVmaToPbsArgs, + store: &str, + request_args: Value, +) -> Result<(), Error> { + if let Some(log_file_path) = &args.log_file_path { + let path = format!("api2/json/admin/datastore/{}/upload-backup-log", store); + let data = std::fs::read(log_file_path)?; + + let blob = if args.encrypt { + let crypt_config = match &args.keyfile { + None => None, + Some(keyfile) => { + let key = std::fs::read(keyfile)?; + let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> { + match &args.key_password { + Some(key_password) => Ok(key_password.clone().into_bytes()), + None => bail!("no key password provided"), + } + })?; + let crypt_config = CryptConfig::new(key)?; + Some(crypt_config) + } + }; + + DataBlob::encode(&data, crypt_config.as_ref(), args.compress)? + } else { + // fixme: howto sign log? + DataBlob::encode(&data, None, args.compress)? + }; + + let body = hyper::Body::from(blob.into_inner()); + + block_on(async { + client + .upload("application/octet-stream", body, &path, Some(request_args)) + .await + .unwrap(); + }); + } + + Ok(()) +} + +fn set_notes( + client: &HttpClient, + notes: &str, + store: &str, + mut request_args: Value, +) -> Result<(), Error> { + request_args["notes"] = Value::from(notes); + let path = format!("api2/json/admin/datastore/{}/notes", store); + block_on(async { + client.put(&path, Some(request_args)).await.unwrap(); + }); + + Ok(()) +} + pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> { let vma_file: Box<dyn BufRead> = match &args.vma_file_path { Some(vma_file_path) => match File::open(vma_file_path) { @@ -386,6 +478,18 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> { handle_pbs_error(pbs_err, "proxmox_backup_finish")?; } + if args.notes.is_some() || args.log_file_path.is_some() { + let (client, store, request_args) = pbs_client_setup(&args)?; + + if args.log_file_path.is_some() { + upload_log(&client, &args, &store, request_args.clone())?; + } + + if let Some(notes) = args.notes { + set_notes(&client, ¬es, &store, request_args)?; + } + } + let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?; let total_seconds = transfer_duration.as_secs(); let minutes = total_seconds / 60; -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* [pbs-devel] applied-series: [PATCH vma-to-pbs v3 0/2] add support for notes and logs 2024-07-10 14:57 [pbs-devel] [PATCH vma-to-pbs v3 0/2] add support for notes and logs Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 1/2] pass args by reference to create_pbs_backup_task Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 2/2] add support for notes and logs Filip Schauer @ 2024-07-11 8:50 ` Fabian Grünbichler 2 siblings, 0 replies; 4+ messages in thread From: Fabian Grünbichler @ 2024-07-11 8:50 UTC (permalink / raw) To: Proxmox Backup Server development discussion I still hope we manage to extract the backup client properly at some point.. Quoting Filip Schauer (2024-07-10 16:57:30) > Changes since v2: > * Split the create_pbs_backup_task argument change into its own commit > * Create the HttpClient directly to avoid writing the password into an > environment variable > * Put the client setup into its own helper function and call it from > backup_vma_to_pbs along with set_notes and upload_log > > Changes since v1: > * Set notes and upload log separately via a new connection to the API, > instead of hacking notes into the proxmox-backup-qemu C functions and > uploading the log as a config. > > vma-to-pbs: > > Filip Schauer (2): > pass args by reference to create_pbs_backup_task > add support for notes and logs > > Cargo.toml | 8 +++ > src/main.rs | 16 ++++++ > src/vma2pbs.rs | 133 +++++++++++++++++++++++++++++++++++++++++++++---- > 3 files changed, 147 insertions(+), 10 deletions(-) > > > Summary over all repositories: > 3 files changed, 147 insertions(+), 10 deletions(-) > > -- > Generated by git-murpp 0.6.0 > > > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel > > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-07-11 8:50 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-07-10 14:57 [pbs-devel] [PATCH vma-to-pbs v3 0/2] add support for notes and logs Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 1/2] pass args by reference to create_pbs_backup_task Filip Schauer 2024-07-10 14:57 ` [pbs-devel] [PATCH vma-to-pbs v3 2/2] add support for notes and logs Filip Schauer 2024-07-11 8:50 ` [pbs-devel] applied-series: [PATCH vma-to-pbs v3 0/2] " Fabian Grünbichler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox