* [pbs-devel] [PATCH vma-to-pbs v2 0/2] add support for notes and logs
@ 2024-07-10 9:20 Filip Schauer
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time Filip Schauer
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs Filip Schauer
0 siblings, 2 replies; 6+ messages in thread
From: Filip Schauer @ 2024-07-10 9:20 UTC (permalink / raw)
To: pbs-devel
Changes since v1:
* Set notes and upload log separately via a new connection to the API,
instead of hacking notes into the proxmox-backup-qemu C functions and
uploading the log as a config.
vma-to-pbs:
Filip Schauer (2):
bump proxmox-backup-qemu and proxmox-time
add support for notes and logs
Cargo.toml | 10 ++-
debian/control | 2 +-
src/main.rs | 16 ++++
src/vma2pbs.rs | 132 ++++++++++++++++++++++++++++++---
submodules/proxmox-backup-qemu | 2 +-
5 files changed, 149 insertions(+), 13 deletions(-)
Summary over all repositories:
5 files changed, 149 insertions(+), 13 deletions(-)
--
Generated by git-murpp 0.6.0
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time
2024-07-10 9:20 [pbs-devel] [PATCH vma-to-pbs v2 0/2] add support for notes and logs Filip Schauer
@ 2024-07-10 9:20 ` Filip Schauer
2024-07-10 12:19 ` [pbs-devel] applied: " Fabian Grünbichler
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs Filip Schauer
1 sibling, 1 reply; 6+ messages in thread
From: Filip Schauer @ 2024-07-10 9:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 2 +-
debian/control | 2 +-
submodules/proxmox-backup-qemu | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index fb2c641..0111362 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -15,6 +15,6 @@ serde-big-array = "0.4.1"
proxmox-io = "1.0.1"
proxmox-sys = "0.5.0"
-proxmox-time = "1.1.6"
+proxmox-time = "2"
proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" }
diff --git a/debian/control b/debian/control
index 98165dd..473391b 100644
--- a/debian/control
+++ b/debian/control
@@ -65,7 +65,7 @@ Build-Depends: cargo:native (>= 0.65.0~),
librust-proxmox-sys-0.5+crypt-dev (>= 0.5.3-~~),
librust-proxmox-sys-0.5+default-dev (>= 0.5.3-~~),
librust-proxmox-sys-0.5+timer-dev (>= 0.5.3-~~),
- librust-proxmox-time-1+default-dev (>= 1.1.6-~~),
+ librust-proxmox-time-2+default-dev,
librust-proxmox-uuid-1+default-dev,
librust-proxmox-uuid-1+serde-dev,
librust-pxar-0.12+default-dev,
diff --git a/submodules/proxmox-backup-qemu b/submodules/proxmox-backup-qemu
index eaa39af..50ecd20 160000
--- a/submodules/proxmox-backup-qemu
+++ b/submodules/proxmox-backup-qemu
@@ -1 +1 @@
-Subproject commit eaa39af53bded63c642ab891b3cdac7f0457a6a7
+Subproject commit 50ecd20e81354170c24c69bb8d1a936b443dbcca
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs
2024-07-10 9:20 [pbs-devel] [PATCH vma-to-pbs v2 0/2] add support for notes and logs Filip Schauer
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time Filip Schauer
@ 2024-07-10 9:20 ` Filip Schauer
2024-07-10 12:18 ` Fabian Grünbichler
1 sibling, 1 reply; 6+ messages in thread
From: Filip Schauer @ 2024-07-10 9:20 UTC (permalink / raw)
To: pbs-devel
Allow the user to specify a notes file and a log file to associate with
the backup
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Use the proxmox-backup crates from the proxmox-backup-qemu submodule,
because a seperate proxmox-backup submodule leads to "package collision
in the lockfile"
Cargo.toml | 8 +++
src/main.rs | 16 ++++++
src/vma2pbs.rs | 132 +++++++++++++++++++++++++++++++++++++++++++++----
3 files changed, 146 insertions(+), 10 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 0111362..c62b5e0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -7,14 +7,22 @@ edition = "2021"
[dependencies]
anyhow = "1.0"
bincode = "1.3"
+hyper = "0.14.5"
pico-args = "0.4"
md5 = "0.7.0"
scopeguard = "1.1.0"
serde = "1.0"
+serde_json = "1.0"
serde-big-array = "0.4.1"
+proxmox-async = "0.4"
proxmox-io = "1.0.1"
proxmox-sys = "0.5.0"
proxmox-time = "2"
+pbs-api-types = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-api-types" }
+pbs-client = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-client" }
+pbs-datastore = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-datastore" }
+pbs-key-config = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-key-config" }
+pbs-tools = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-tools" }
proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" }
diff --git a/src/main.rs b/src/main.rs
index 2653d3e..de789c1 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -37,6 +37,10 @@ Options:
Password file
--key-password-file <KEY_PASSWORD_FILE>
Key password file
+ [--notes-file <NOTES_FILE>]
+ File containing a comment/notes
+ [--log-file <LOG_FILE>]
+ Log file
-h, --help
Print help
-V, --version
@@ -93,6 +97,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let encrypt = args.contains(["-e", "--encrypt"]);
let password_file: Option<OsString> = args.opt_value_from_str("--password-file")?;
let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
+ let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
+ let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
@@ -170,6 +176,14 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
None
};
+ let notes = if let Some(notes_file) = notes_file {
+ let notes = std::fs::read_to_string(notes_file).context("Could not read notes file")?;
+
+ Some(notes)
+ } else {
+ None
+ };
+
let options = BackupVmaToPbsArgs {
vma_file_path: vma_file_path.cloned(),
pbs_repository,
@@ -183,6 +197,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
fingerprint,
compress,
encrypt,
+ notes,
+ log_file_path,
};
Ok(options)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 199cf50..eaad02e 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -8,6 +8,12 @@ use std::ptr;
use std::time::SystemTime;
use anyhow::{anyhow, bail, Error};
+use pbs_api_types::{BackupDir, BackupNamespace, BackupType};
+use pbs_client::{tools::connect, BackupRepository, HttpClient};
+use pbs_datastore::DataBlob;
+use pbs_key_config::decrypt_key;
+use pbs_tools::crypt_config::CryptConfig;
+use proxmox_async::runtime::block_on;
use proxmox_backup_qemu::{
capi_types::ProxmoxBackupHandle, proxmox_backup_add_config, proxmox_backup_close_image,
proxmox_backup_connect, proxmox_backup_disconnect, proxmox_backup_finish,
@@ -16,6 +22,7 @@ use proxmox_backup_qemu::{
};
use proxmox_time::epoch_to_rfc3339;
use scopeguard::defer;
+use serde_json::Value;
use crate::vma::VmaReader;
@@ -34,6 +41,8 @@ pub struct BackupVmaToPbsArgs {
pub fingerprint: String,
pub compress: bool,
pub encrypt: bool,
+ pub notes: Option<String>,
+ pub log_file_path: Option<OsString>,
}
#[derive(Copy, Clone)]
@@ -52,7 +61,7 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err
bail!("{function_name} failed: {pbs_err_str}");
}
-fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
+fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
println!("PBS repository: {}", args.pbs_repository);
if let Some(ns) = &args.namespace {
println!("PBS namespace: {}", ns);
@@ -65,22 +74,31 @@ fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackup
let mut pbs_err: *mut c_char = ptr::null_mut();
- let pbs_repository_cstr = CString::new(args.pbs_repository)?;
- let ns_cstr = CString::new(args.namespace.unwrap_or("".to_string()))?;
- let backup_id_cstr = CString::new(args.backup_id)?;
- let pbs_password_cstr = CString::new(args.pbs_password)?;
- let fingerprint_cstr = CString::new(args.fingerprint)?;
- let keyfile_cstr = args.keyfile.map(|v| CString::new(v).unwrap());
+ let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?;
+ let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?;
+ let backup_id_cstr = CString::new(args.backup_id.as_str())?;
+ let pbs_password_cstr = CString::new(args.pbs_password.as_str())?;
+ let fingerprint_cstr = CString::new(args.fingerprint.as_str())?;
+ let keyfile_cstr = args
+ .keyfile
+ .as_ref()
+ .map(|v| CString::new(v.as_str()).unwrap());
let keyfile_ptr = keyfile_cstr
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let key_password_cstr = args.key_password.map(|v| CString::new(v).unwrap());
+ let key_password_cstr = args
+ .key_password
+ .as_ref()
+ .map(|v| CString::new(v.as_str()).unwrap());
let key_password_ptr = key_password_cstr
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let master_keyfile_cstr = args.master_keyfile.map(|v| CString::new(v).unwrap());
+ let master_keyfile_cstr = args
+ .master_keyfile
+ .as_ref()
+ .map(|v| CString::new(v.as_str()).unwrap());
let master_keyfile_ptr = master_keyfile_cstr
.as_ref()
.map(|v| v.as_ptr())
@@ -343,6 +361,98 @@ where
Ok(())
}
+fn upload_log_and_notes(args: &BackupVmaToPbsArgs) -> Result<(), Error> {
+ if args.log_file_path.is_none() || args.notes.is_none() {
+ return Ok(());
+ }
+
+ let repo: BackupRepository = args.pbs_repository.parse()?;
+ std::env::set_var("PBS_PASSWORD", &args.pbs_password);
+ let client = connect(&repo)?;
+ let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time));
+
+ let namespace = match &args.namespace {
+ Some(namespace) => BackupNamespace::new(namespace)?,
+ None => BackupNamespace::root(),
+ };
+
+ let mut request_args = serde_json::to_value(backup_dir)?;
+ if !namespace.is_root() {
+ request_args["ns"] = serde_json::to_value(namespace)?;
+ }
+
+ upload_log(&client, args, &repo, request_args.clone())?;
+ upload_notes(&client, args, &repo, request_args)?;
+
+ Ok(())
+}
+
+fn upload_log(
+ client: &HttpClient,
+ args: &BackupVmaToPbsArgs,
+ repo: &BackupRepository,
+ request_args: Value,
+) -> Result<(), Error> {
+ if let Some(log_file_path) = &args.log_file_path {
+ let path = format!(
+ "api2/json/admin/datastore/{}/upload-backup-log",
+ repo.store()
+ );
+
+ let data = std::fs::read(log_file_path)?;
+
+ let blob = if args.encrypt {
+ let crypt_config = match &args.keyfile {
+ None => None,
+ Some(keyfile) => {
+ let key = std::fs::read(keyfile)?;
+ let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
+ match &args.key_password {
+ Some(key_password) => Ok(key_password.clone().into_bytes()),
+ None => bail!("no key password provided"),
+ }
+ })?;
+ let crypt_config = CryptConfig::new(key)?;
+ Some(crypt_config)
+ }
+ };
+
+ DataBlob::encode(&data, crypt_config.as_ref(), args.compress)?
+ } else {
+ // fixme: howto sign log?
+ DataBlob::encode(&data, None, args.compress)?
+ };
+
+ let body = hyper::Body::from(blob.into_inner());
+
+ block_on(async {
+ client
+ .upload("application/octet-stream", body, &path, Some(request_args))
+ .await
+ .unwrap();
+ });
+ }
+
+ Ok(())
+}
+
+fn upload_notes(
+ client: &HttpClient,
+ args: &BackupVmaToPbsArgs,
+ repo: &BackupRepository,
+ mut request_args: Value,
+) -> Result<(), Error> {
+ if let Some(notes) = &args.notes {
+ request_args["notes"] = Value::from(notes.as_str());
+ let path = format!("api2/json/admin/datastore/{}/notes", repo.store());
+ block_on(async {
+ client.put(&path, Some(request_args)).await.unwrap();
+ });
+ }
+
+ Ok(())
+}
+
pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let vma_file: Box<dyn BufRead> = match &args.vma_file_path {
Some(vma_file_path) => match File::open(vma_file_path) {
@@ -353,7 +463,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
};
let vma_reader = VmaReader::new(vma_file)?;
- let pbs = create_pbs_backup_task(args)?;
+ let pbs = create_pbs_backup_task(&args)?;
defer! {
proxmox_backup_disconnect(pbs);
@@ -377,6 +487,8 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
handle_pbs_error(pbs_err, "proxmox_backup_finish")?;
}
+ upload_log_and_notes(&args)?;
+
let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
let total_seconds = transfer_duration.as_secs();
let minutes = total_seconds / 60;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs Filip Schauer
@ 2024-07-10 12:18 ` Fabian Grünbichler
2024-07-10 14:57 ` Filip Schauer
0 siblings, 1 reply; 6+ messages in thread
From: Fabian Grünbichler @ 2024-07-10 12:18 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On July 10, 2024 11:20 am, Filip Schauer wrote:
> Allow the user to specify a notes file and a log file to associate with
> the backup
>
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> ---
> Use the proxmox-backup crates from the proxmox-backup-qemu submodule,
> because a seperate proxmox-backup submodule leads to "package collision
> in the lockfile"
>
> Cargo.toml | 8 +++
> src/main.rs | 16 ++++++
> src/vma2pbs.rs | 132 +++++++++++++++++++++++++++++++++++++++++++++----
> 3 files changed, 146 insertions(+), 10 deletions(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index 0111362..c62b5e0 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -7,14 +7,22 @@ edition = "2021"
> [dependencies]
> anyhow = "1.0"
> bincode = "1.3"
> +hyper = "0.14.5"
> pico-args = "0.4"
> md5 = "0.7.0"
> scopeguard = "1.1.0"
> serde = "1.0"
> +serde_json = "1.0"
> serde-big-array = "0.4.1"
>
> +proxmox-async = "0.4"
> proxmox-io = "1.0.1"
> proxmox-sys = "0.5.0"
> proxmox-time = "2"
>
> +pbs-api-types = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-api-types" }
> +pbs-client = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-client" }
> +pbs-datastore = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-datastore" }
> +pbs-key-config = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-key-config" }
> +pbs-tools = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-tools" }
> proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" }
> diff --git a/src/main.rs b/src/main.rs
> index 2653d3e..de789c1 100644
> --- a/src/main.rs
> +++ b/src/main.rs
> @@ -37,6 +37,10 @@ Options:
> Password file
> --key-password-file <KEY_PASSWORD_FILE>
> Key password file
> + [--notes-file <NOTES_FILE>]
> + File containing a comment/notes
> + [--log-file <LOG_FILE>]
> + Log file
> -h, --help
> Print help
> -V, --version
> @@ -93,6 +97,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> let encrypt = args.contains(["-e", "--encrypt"]);
> let password_file: Option<OsString> = args.opt_value_from_str("--password-file")?;
> let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
> + let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
> + let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
>
> match (encrypt, keyfile.is_some()) {
> (true, false) => bail!("--encrypt requires a --keyfile!"),
> @@ -170,6 +176,14 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> None
> };
>
> + let notes = if let Some(notes_file) = notes_file {
> + let notes = std::fs::read_to_string(notes_file).context("Could not read notes file")?;
> +
> + Some(notes)
> + } else {
> + None
> + };
> +
> let options = BackupVmaToPbsArgs {
> vma_file_path: vma_file_path.cloned(),
> pbs_repository,
> @@ -183,6 +197,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> fingerprint,
> compress,
> encrypt,
> + notes,
> + log_file_path,
> };
>
> Ok(options)
> diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
> index 199cf50..eaad02e 100644
> --- a/src/vma2pbs.rs
> +++ b/src/vma2pbs.rs
> @@ -8,6 +8,12 @@ use std::ptr;
> use std::time::SystemTime;
>
> use anyhow::{anyhow, bail, Error};
> +use pbs_api_types::{BackupDir, BackupNamespace, BackupType};
> +use pbs_client::{tools::connect, BackupRepository, HttpClient};
> +use pbs_datastore::DataBlob;
> +use pbs_key_config::decrypt_key;
> +use pbs_tools::crypt_config::CryptConfig;
> +use proxmox_async::runtime::block_on;
> use proxmox_backup_qemu::{
> capi_types::ProxmoxBackupHandle, proxmox_backup_add_config, proxmox_backup_close_image,
> proxmox_backup_connect, proxmox_backup_disconnect, proxmox_backup_finish,
> @@ -16,6 +22,7 @@ use proxmox_backup_qemu::{
> };
> use proxmox_time::epoch_to_rfc3339;
> use scopeguard::defer;
> +use serde_json::Value;
>
> use crate::vma::VmaReader;
>
> @@ -34,6 +41,8 @@ pub struct BackupVmaToPbsArgs {
> pub fingerprint: String,
> pub compress: bool,
> pub encrypt: bool,
> + pub notes: Option<String>,
> + pub log_file_path: Option<OsString>,
> }
>
> #[derive(Copy, Clone)]
> @@ -52,7 +61,7 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err
> bail!("{function_name} failed: {pbs_err_str}");
> }
>
> -fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
> +fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
this
> println!("PBS repository: {}", args.pbs_repository);
> if let Some(ns) = &args.namespace {
> println!("PBS namespace: {}", ns);
> @@ -65,22 +74,31 @@ fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackup
>
> let mut pbs_err: *mut c_char = ptr::null_mut();
>
> - let pbs_repository_cstr = CString::new(args.pbs_repository)?;
> - let ns_cstr = CString::new(args.namespace.unwrap_or("".to_string()))?;
> - let backup_id_cstr = CString::new(args.backup_id)?;
> - let pbs_password_cstr = CString::new(args.pbs_password)?;
> - let fingerprint_cstr = CString::new(args.fingerprint)?;
> - let keyfile_cstr = args.keyfile.map(|v| CString::new(v).unwrap());
> + let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?;
> + let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?;
> + let backup_id_cstr = CString::new(args.backup_id.as_str())?;
> + let pbs_password_cstr = CString::new(args.pbs_password.as_str())?;
> + let fingerprint_cstr = CString::new(args.fingerprint.as_str())?;
> + let keyfile_cstr = args
> + .keyfile
> + .as_ref()
> + .map(|v| CString::new(v.as_str()).unwrap());
and these
> let keyfile_ptr = keyfile_cstr
> .as_ref()
> .map(|v| v.as_ptr())
> .unwrap_or(ptr::null());
> - let key_password_cstr = args.key_password.map(|v| CString::new(v).unwrap());
> + let key_password_cstr = args
> + .key_password
> + .as_ref()
> + .map(|v| CString::new(v.as_str()).unwrap());
and this
> let key_password_ptr = key_password_cstr
> .as_ref()
> .map(|v| v.as_ptr())
> .unwrap_or(ptr::null());
> - let master_keyfile_cstr = args.master_keyfile.map(|v| CString::new(v).unwrap());
> + let master_keyfile_cstr = args
> + .master_keyfile
> + .as_ref()
> + .map(|v| CString::new(v.as_str()).unwrap());
and this
seem kinda unrelated? some sort of clippy/format fix patch that should
be split out maybe? ah, it's because you need to pass the args to
another helper further below.. that could be split into it's own patch
to clearl separate the interface change and the rest..
> let master_keyfile_ptr = master_keyfile_cstr
> .as_ref()
> .map(|v| v.as_ptr())
> @@ -343,6 +361,98 @@ where
> Ok(())
> }
>
> +fn upload_log_and_notes(args: &BackupVmaToPbsArgs) -> Result<(), Error> {
> + if args.log_file_path.is_none() || args.notes.is_none() {
> + return Ok(());
> + }
> +
> + let repo: BackupRepository = args.pbs_repository.parse()?;
> + std::env::set_var("PBS_PASSWORD", &args.pbs_password);
> + let client = connect(&repo)?;
why not just create the client directly (HttpCLient::new()), then you
don't need to set an env variable here that might leak the password..
> + let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time));
> +
> + let namespace = match &args.namespace {
> + Some(namespace) => BackupNamespace::new(namespace)?,
> + None => BackupNamespace::root(),
> + };
> +
> + let mut request_args = serde_json::to_value(backup_dir)?;
> + if !namespace.is_root() {
> + request_args["ns"] = serde_json::to_value(namespace)?;
> + }
everything up to here could be a helper to get a client..
> +
> + upload_log(&client, args, &repo, request_args.clone())?;
> + upload_notes(&client, args, &repo, request_args)?;
> +
> + Ok(())
> +}
> +
> +fn upload_log(
> + client: &HttpClient,
> + args: &BackupVmaToPbsArgs,
> + repo: &BackupRepository,
> + request_args: Value,
> +) -> Result<(), Error> {
> + if let Some(log_file_path) = &args.log_file_path {
> + let path = format!(
> + "api2/json/admin/datastore/{}/upload-backup-log",
> + repo.store()
> + );
> +
> + let data = std::fs::read(log_file_path)?;
> +
> + let blob = if args.encrypt {
> + let crypt_config = match &args.keyfile {
> + None => None,
> + Some(keyfile) => {
> + let key = std::fs::read(keyfile)?;
> + let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
> + match &args.key_password {
> + Some(key_password) => Ok(key_password.clone().into_bytes()),
> + None => bail!("no key password provided"),
> + }
> + })?;
> + let crypt_config = CryptConfig::new(key)?;
> + Some(crypt_config)
> + }
> + };
> +
> + DataBlob::encode(&data, crypt_config.as_ref(), args.compress)?
> + } else {
> + // fixme: howto sign log?
> + DataBlob::encode(&data, None, args.compress)?
> + };
> +
> + let body = hyper::Body::from(blob.into_inner());
> +
> + block_on(async {
> + client
> + .upload("application/octet-stream", body, &path, Some(request_args))
> + .await
> + .unwrap();
> + });
> + }
> +
> + Ok(())
> +}
> +
> +fn upload_notes(
> + client: &HttpClient,
> + args: &BackupVmaToPbsArgs,
> + repo: &BackupRepository,
> + mut request_args: Value,
> +) -> Result<(), Error> {
> + if let Some(notes) = &args.notes {
> + request_args["notes"] = Value::from(notes.as_str());
> + let path = format!("api2/json/admin/datastore/{}/notes", repo.store());
> + block_on(async {
> + client.put(&path, Some(request_args)).await.unwrap();
> + });
> + }
> +
> + Ok(())
> +}
> +
> pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
> let vma_file: Box<dyn BufRead> = match &args.vma_file_path {
> Some(vma_file_path) => match File::open(vma_file_path) {
> @@ -353,7 +463,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
> };
> let vma_reader = VmaReader::new(vma_file)?;
>
> - let pbs = create_pbs_backup_task(args)?;
> + let pbs = create_pbs_backup_task(&args)?;
>
> defer! {
> proxmox_backup_disconnect(pbs);
> @@ -377,6 +487,8 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
> handle_pbs_error(pbs_err, "proxmox_backup_finish")?;
> }
>
couldn't we just check here if we need to upload logs or notes, and then
do something like:
let (client, request_args) = client_helper(&args)?;
if (args.notes_file.is_some()) {
set_notes(&client, ..)?;
}
if (args.log_file_path.is_some()) {
upload_log(&client, ..)?;
}
> + upload_log_and_notes(&args)?;
> +
> let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
> let total_seconds = transfer_duration.as_secs();
> let minutes = total_seconds / 60;
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] applied: [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time Filip Schauer
@ 2024-07-10 12:19 ` Fabian Grünbichler
0 siblings, 0 replies; 6+ messages in thread
From: Fabian Grünbichler @ 2024-07-10 12:19 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On July 10, 2024 11:20 am, Filip Schauer wrote:
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> ---
> Cargo.toml | 2 +-
> debian/control | 2 +-
> submodules/proxmox-backup-qemu | 2 +-
> 3 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index fb2c641..0111362 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -15,6 +15,6 @@ serde-big-array = "0.4.1"
>
> proxmox-io = "1.0.1"
> proxmox-sys = "0.5.0"
> -proxmox-time = "1.1.6"
> +proxmox-time = "2"
>
> proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" }
> diff --git a/debian/control b/debian/control
> index 98165dd..473391b 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -65,7 +65,7 @@ Build-Depends: cargo:native (>= 0.65.0~),
> librust-proxmox-sys-0.5+crypt-dev (>= 0.5.3-~~),
> librust-proxmox-sys-0.5+default-dev (>= 0.5.3-~~),
> librust-proxmox-sys-0.5+timer-dev (>= 0.5.3-~~),
> - librust-proxmox-time-1+default-dev (>= 1.1.6-~~),
> + librust-proxmox-time-2+default-dev,
> librust-proxmox-uuid-1+default-dev,
> librust-proxmox-uuid-1+serde-dev,
> librust-pxar-0.12+default-dev,
> diff --git a/submodules/proxmox-backup-qemu b/submodules/proxmox-backup-qemu
> index eaa39af..50ecd20 160000
> --- a/submodules/proxmox-backup-qemu
> +++ b/submodules/proxmox-backup-qemu
> @@ -1 +1 @@
> -Subproject commit eaa39af53bded63c642ab891b3cdac7f0457a6a7
> +Subproject commit 50ecd20e81354170c24c69bb8d1a936b443dbcca
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs
2024-07-10 12:18 ` Fabian Grünbichler
@ 2024-07-10 14:57 ` Filip Schauer
0 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-07-10 14:57 UTC (permalink / raw)
To: pbs-devel
Superseded by:
https://lists.proxmox.com/pipermail/pbs-devel/2024-July/010159.html
On 10/07/2024 14:18, Fabian Grünbichler wrote:
> On July 10, 2024 11:20 am, Filip Schauer wrote:
>> Allow the user to specify a notes file and a log file to associate with
>> the backup
>>
>> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
>> ---
>> Use the proxmox-backup crates from the proxmox-backup-qemu submodule,
>> because a seperate proxmox-backup submodule leads to "package collision
>> in the lockfile"
>>
>> Cargo.toml | 8 +++
>> src/main.rs | 16 ++++++
>> src/vma2pbs.rs | 132 +++++++++++++++++++++++++++++++++++++++++++++----
>> 3 files changed, 146 insertions(+), 10 deletions(-)
>>
>> diff --git a/Cargo.toml b/Cargo.toml
>> index 0111362..c62b5e0 100644
>> --- a/Cargo.toml
>> +++ b/Cargo.toml
>> @@ -7,14 +7,22 @@ edition = "2021"
>> [dependencies]
>> anyhow = "1.0"
>> bincode = "1.3"
>> +hyper = "0.14.5"
>> pico-args = "0.4"
>> md5 = "0.7.0"
>> scopeguard = "1.1.0"
>> serde = "1.0"
>> +serde_json = "1.0"
>> serde-big-array = "0.4.1"
>>
>> +proxmox-async = "0.4"
>> proxmox-io = "1.0.1"
>> proxmox-sys = "0.5.0"
>> proxmox-time = "2"
>>
>> +pbs-api-types = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-api-types" }
>> +pbs-client = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-client" }
>> +pbs-datastore = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-datastore" }
>> +pbs-key-config = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-key-config" }
>> +pbs-tools = { path = "submodules/proxmox-backup-qemu/submodules/proxmox-backup/pbs-tools" }
>> proxmox-backup-qemu = { path = "submodules/proxmox-backup-qemu" }
>> diff --git a/src/main.rs b/src/main.rs
>> index 2653d3e..de789c1 100644
>> --- a/src/main.rs
>> +++ b/src/main.rs
>> @@ -37,6 +37,10 @@ Options:
>> Password file
>> --key-password-file <KEY_PASSWORD_FILE>
>> Key password file
>> + [--notes-file <NOTES_FILE>]
>> + File containing a comment/notes
>> + [--log-file <LOG_FILE>]
>> + Log file
>> -h, --help
>> Print help
>> -V, --version
>> @@ -93,6 +97,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
>> let encrypt = args.contains(["-e", "--encrypt"]);
>> let password_file: Option<OsString> = args.opt_value_from_str("--password-file")?;
>> let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
>> + let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
>> + let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
>>
>> match (encrypt, keyfile.is_some()) {
>> (true, false) => bail!("--encrypt requires a --keyfile!"),
>> @@ -170,6 +176,14 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
>> None
>> };
>>
>> + let notes = if let Some(notes_file) = notes_file {
>> + let notes = std::fs::read_to_string(notes_file).context("Could not read notes file")?;
>> +
>> + Some(notes)
>> + } else {
>> + None
>> + };
>> +
>> let options = BackupVmaToPbsArgs {
>> vma_file_path: vma_file_path.cloned(),
>> pbs_repository,
>> @@ -183,6 +197,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
>> fingerprint,
>> compress,
>> encrypt,
>> + notes,
>> + log_file_path,
>> };
>>
>> Ok(options)
>> diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
>> index 199cf50..eaad02e 100644
>> --- a/src/vma2pbs.rs
>> +++ b/src/vma2pbs.rs
>> @@ -8,6 +8,12 @@ use std::ptr;
>> use std::time::SystemTime;
>>
>> use anyhow::{anyhow, bail, Error};
>> +use pbs_api_types::{BackupDir, BackupNamespace, BackupType};
>> +use pbs_client::{tools::connect, BackupRepository, HttpClient};
>> +use pbs_datastore::DataBlob;
>> +use pbs_key_config::decrypt_key;
>> +use pbs_tools::crypt_config::CryptConfig;
>> +use proxmox_async::runtime::block_on;
>> use proxmox_backup_qemu::{
>> capi_types::ProxmoxBackupHandle, proxmox_backup_add_config, proxmox_backup_close_image,
>> proxmox_backup_connect, proxmox_backup_disconnect, proxmox_backup_finish,
>> @@ -16,6 +22,7 @@ use proxmox_backup_qemu::{
>> };
>> use proxmox_time::epoch_to_rfc3339;
>> use scopeguard::defer;
>> +use serde_json::Value;
>>
>> use crate::vma::VmaReader;
>>
>> @@ -34,6 +41,8 @@ pub struct BackupVmaToPbsArgs {
>> pub fingerprint: String,
>> pub compress: bool,
>> pub encrypt: bool,
>> + pub notes: Option<String>,
>> + pub log_file_path: Option<OsString>,
>> }
>>
>> #[derive(Copy, Clone)]
>> @@ -52,7 +61,7 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err
>> bail!("{function_name} failed: {pbs_err_str}");
>> }
>>
>> -fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
>> +fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
> this
>
>> println!("PBS repository: {}", args.pbs_repository);
>> if let Some(ns) = &args.namespace {
>> println!("PBS namespace: {}", ns);
>> @@ -65,22 +74,31 @@ fn create_pbs_backup_task(args: BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackup
>>
>> let mut pbs_err: *mut c_char = ptr::null_mut();
>>
>> - let pbs_repository_cstr = CString::new(args.pbs_repository)?;
>> - let ns_cstr = CString::new(args.namespace.unwrap_or("".to_string()))?;
>> - let backup_id_cstr = CString::new(args.backup_id)?;
>> - let pbs_password_cstr = CString::new(args.pbs_password)?;
>> - let fingerprint_cstr = CString::new(args.fingerprint)?;
>> - let keyfile_cstr = args.keyfile.map(|v| CString::new(v).unwrap());
>> + let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?;
>> + let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?;
>> + let backup_id_cstr = CString::new(args.backup_id.as_str())?;
>> + let pbs_password_cstr = CString::new(args.pbs_password.as_str())?;
>> + let fingerprint_cstr = CString::new(args.fingerprint.as_str())?;
>> + let keyfile_cstr = args
>> + .keyfile
>> + .as_ref()
>> + .map(|v| CString::new(v.as_str()).unwrap());
> and these
>
>> let keyfile_ptr = keyfile_cstr
>> .as_ref()
>> .map(|v| v.as_ptr())
>> .unwrap_or(ptr::null());
>> - let key_password_cstr = args.key_password.map(|v| CString::new(v).unwrap());
>> + let key_password_cstr = args
>> + .key_password
>> + .as_ref()
>> + .map(|v| CString::new(v.as_str()).unwrap());
> and this
>
>> let key_password_ptr = key_password_cstr
>> .as_ref()
>> .map(|v| v.as_ptr())
>> .unwrap_or(ptr::null());
>> - let master_keyfile_cstr = args.master_keyfile.map(|v| CString::new(v).unwrap());
>> + let master_keyfile_cstr = args
>> + .master_keyfile
>> + .as_ref()
>> + .map(|v| CString::new(v.as_str()).unwrap());
> and this
>
> seem kinda unrelated? some sort of clippy/format fix patch that should
> be split out maybe? ah, it's because you need to pass the args to
> another helper further below.. that could be split into it's own patch
> to clearl separate the interface change and the rest..
>
>> let master_keyfile_ptr = master_keyfile_cstr
>> .as_ref()
>> .map(|v| v.as_ptr())
>> @@ -343,6 +361,98 @@ where
>> Ok(())
>> }
>>
>> +fn upload_log_and_notes(args: &BackupVmaToPbsArgs) -> Result<(), Error> {
>> + if args.log_file_path.is_none() || args.notes.is_none() {
>> + return Ok(());
>> + }
>> +
>> + let repo: BackupRepository = args.pbs_repository.parse()?;
>> + std::env::set_var("PBS_PASSWORD", &args.pbs_password);
>> + let client = connect(&repo)?;
> why not just create the client directly (HttpCLient::new()), then you
> don't need to set an env variable here that might leak the password..
>
>> + let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time));
>> +
>> + let namespace = match &args.namespace {
>> + Some(namespace) => BackupNamespace::new(namespace)?,
>> + None => BackupNamespace::root(),
>> + };
>> +
>> + let mut request_args = serde_json::to_value(backup_dir)?;
>> + if !namespace.is_root() {
>> + request_args["ns"] = serde_json::to_value(namespace)?;
>> + }
> everything up to here could be a helper to get a client..
>
>> +
>> + upload_log(&client, args, &repo, request_args.clone())?;
>> + upload_notes(&client, args, &repo, request_args)?;
>> +
>> + Ok(())
>> +}
>> +
>> +fn upload_log(
>> + client: &HttpClient,
>> + args: &BackupVmaToPbsArgs,
>> + repo: &BackupRepository,
>> + request_args: Value,
>> +) -> Result<(), Error> {
>> + if let Some(log_file_path) = &args.log_file_path {
>> + let path = format!(
>> + "api2/json/admin/datastore/{}/upload-backup-log",
>> + repo.store()
>> + );
>> +
>> + let data = std::fs::read(log_file_path)?;
>> +
>> + let blob = if args.encrypt {
>> + let crypt_config = match &args.keyfile {
>> + None => None,
>> + Some(keyfile) => {
>> + let key = std::fs::read(keyfile)?;
>> + let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
>> + match &args.key_password {
>> + Some(key_password) => Ok(key_password.clone().into_bytes()),
>> + None => bail!("no key password provided"),
>> + }
>> + })?;
>> + let crypt_config = CryptConfig::new(key)?;
>> + Some(crypt_config)
>> + }
>> + };
>> +
>> + DataBlob::encode(&data, crypt_config.as_ref(), args.compress)?
>> + } else {
>> + // fixme: howto sign log?
>> + DataBlob::encode(&data, None, args.compress)?
>> + };
>> +
>> + let body = hyper::Body::from(blob.into_inner());
>> +
>> + block_on(async {
>> + client
>> + .upload("application/octet-stream", body, &path, Some(request_args))
>> + .await
>> + .unwrap();
>> + });
>> + }
>> +
>> + Ok(())
>> +}
>> +
>> +fn upload_notes(
>> + client: &HttpClient,
>> + args: &BackupVmaToPbsArgs,
>> + repo: &BackupRepository,
>> + mut request_args: Value,
>> +) -> Result<(), Error> {
>> + if let Some(notes) = &args.notes {
>> + request_args["notes"] = Value::from(notes.as_str());
>> + let path = format!("api2/json/admin/datastore/{}/notes", repo.store());
>> + block_on(async {
>> + client.put(&path, Some(request_args)).await.unwrap();
>> + });
>> + }
>> +
>> + Ok(())
>> +}
>> +
>> pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
>> let vma_file: Box<dyn BufRead> = match &args.vma_file_path {
>> Some(vma_file_path) => match File::open(vma_file_path) {
>> @@ -353,7 +463,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
>> };
>> let vma_reader = VmaReader::new(vma_file)?;
>>
>> - let pbs = create_pbs_backup_task(args)?;
>> + let pbs = create_pbs_backup_task(&args)?;
>>
>> defer! {
>> proxmox_backup_disconnect(pbs);
>> @@ -377,6 +487,8 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
>> handle_pbs_error(pbs_err, "proxmox_backup_finish")?;
>> }
>>
> couldn't we just check here if we need to upload logs or notes, and then
> do something like:
>
> let (client, request_args) = client_helper(&args)?;
> if (args.notes_file.is_some()) {
> set_notes(&client, ..)?;
> }
> if (args.log_file_path.is_some()) {
> upload_log(&client, ..)?;
> }
>
>> + upload_log_and_notes(&args)?;
>> +
>> let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
>> let total_seconds = transfer_duration.as_secs();
>> let minutes = total_seconds / 60;
>> --
>> 2.39.2
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-07-10 14:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-10 9:20 [pbs-devel] [PATCH vma-to-pbs v2 0/2] add support for notes and logs Filip Schauer
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 1/2] bump proxmox-backup-qemu and proxmox-time Filip Schauer
2024-07-10 12:19 ` [pbs-devel] applied: " Fabian Grünbichler
2024-07-10 9:20 ` [pbs-devel] [PATCH vma-to-pbs v2 2/2] add support for notes and logs Filip Schauer
2024-07-10 12:18 ` Fabian Grünbichler
2024-07-10 14:57 ` Filip Schauer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox