* [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory
@ 2024-10-08 14:33 Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 1/5] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:33 UTC (permalink / raw)
To: pbs-devel
When a path to a directory is provided in the vma_file argument, try to
upload all VMA backups in the directory. This also handles compressed
VMA files, notes and logs. If a vmid is specified with --vmid, only the
backups of that particular vmid are uploaded.
Also improve the readability of the log messages to keep track of all
imported backups.
Changed since v1:
* Do not recurse through dump directory
* Compile regex once before iterating over the files in the dump
directory
* Use extract on regex capture groups
* Do not use deprecated method `chrono::NaiveDateTime::timestamp`
* Use proxmox_sys::fs::file_read_optional_string
* Group VMA files by VMID and continue with next VMID on error
* Move the BackupVmaToPbsArgs split into its own commit
* Remove hard coded occurences of 255
* Use level-based logging instead of println
Filip Schauer (5):
split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs
add support for bulk import of a dump directory
remove hard coded values
use level-based logging instead of println
log device upload progress as a percentage
Cargo.toml | 5 +
src/main.rs | 143 ++++++++++++++++++++---
src/vma.rs | 2 +-
src/vma2pbs.rs | 305 ++++++++++++++++++++++++++++++++-----------------
4 files changed, 335 insertions(+), 120 deletions(-)
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 vma-to-pbs 1/5] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
@ 2024-10-08 14:34 ` Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 2/5] add support for bulk import of a dump directory Filip Schauer
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:34 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/main.rs | 17 +++--
src/vma2pbs.rs | 199 ++++++++++++++++++++++++++++---------------------
2 files changed, 126 insertions(+), 90 deletions(-)
diff --git a/src/main.rs b/src/main.rs
index de789c1..3e25591 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -6,7 +6,7 @@ use proxmox_time::epoch_i64;
mod vma;
mod vma2pbs;
-use vma2pbs::{backup_vma_to_pbs, BackupVmaToPbsArgs};
+use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
const CMD_HELP: &str = "\
Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
@@ -184,12 +184,9 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
None
};
- let options = BackupVmaToPbsArgs {
- vma_file_path: vma_file_path.cloned(),
+ let pbs_args = PbsArgs {
pbs_repository,
namespace,
- backup_id: vmid,
- backup_time,
pbs_password,
keyfile,
key_password,
@@ -197,16 +194,24 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
fingerprint,
compress,
encrypt,
+ };
+
+ let vma_args = VmaBackupArgs {
+ vma_file_path: vma_file_path.cloned(),
+ backup_id: vmid,
+ backup_time,
notes,
log_file_path,
};
+ let options = BackupVmaToPbsArgs { pbs_args, vma_args };
+
Ok(options)
}
fn main() -> Result<(), Error> {
let args = parse_args()?;
- backup_vma_to_pbs(args)?;
+ vma2pbs(args)?;
Ok(())
}
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index d2ce437..37ea308 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -29,11 +29,13 @@ use crate::vma::VmaReader;
const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
- pub vma_file_path: Option<OsString>,
+ pub pbs_args: PbsArgs,
+ pub vma_args: VmaBackupArgs,
+}
+
+pub struct PbsArgs {
pub pbs_repository: String,
pub namespace: Option<String>,
- pub backup_id: String,
- pub backup_time: i64,
pub pbs_password: String,
pub keyfile: Option<String>,
pub key_password: Option<String>,
@@ -41,6 +43,12 @@ pub struct BackupVmaToPbsArgs {
pub fingerprint: String,
pub compress: bool,
pub encrypt: bool,
+}
+
+pub struct VmaBackupArgs {
+ pub vma_file_path: Option<OsString>,
+ pub backup_id: String,
+ pub backup_time: i64,
pub notes: Option<String>,
pub log_file_path: Option<OsString>,
}
@@ -61,25 +69,23 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err
bail!("{function_name} failed: {pbs_err_str}");
}
-fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
- println!("PBS repository: {}", args.pbs_repository);
- if let Some(ns) = &args.namespace {
- println!("PBS namespace: {}", ns);
- }
- println!("PBS fingerprint: {}", args.fingerprint);
- println!("compress: {}", args.compress);
- println!("encrypt: {}", args.encrypt);
-
- println!("backup time: {}", epoch_to_rfc3339(args.backup_time)?);
+fn create_pbs_backup_task(
+ pbs_args: &PbsArgs,
+ backup_args: &VmaBackupArgs,
+) -> Result<*mut ProxmoxBackupHandle, Error> {
+ println!(
+ "backup time: {}",
+ epoch_to_rfc3339(backup_args.backup_time)?
+ );
let mut pbs_err: *mut c_char = ptr::null_mut();
- let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?;
- let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?;
- let backup_id_cstr = CString::new(args.backup_id.as_str())?;
- let pbs_password_cstr = CString::new(args.pbs_password.as_str())?;
- let fingerprint_cstr = CString::new(args.fingerprint.as_str())?;
- let keyfile_cstr = args
+ let pbs_repository_cstr = CString::new(pbs_args.pbs_repository.as_str())?;
+ let ns_cstr = CString::new(pbs_args.namespace.as_deref().unwrap_or(""))?;
+ let backup_id_cstr = CString::new(backup_args.backup_id.as_str())?;
+ let pbs_password_cstr = CString::new(pbs_args.pbs_password.as_str())?;
+ let fingerprint_cstr = CString::new(pbs_args.fingerprint.as_str())?;
+ let keyfile_cstr = pbs_args
.keyfile
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -87,7 +93,7 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let key_password_cstr = args
+ let key_password_cstr = pbs_args
.key_password
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -95,7 +101,7 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let master_keyfile_cstr = args
+ let master_keyfile_cstr = pbs_args
.master_keyfile
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -108,14 +114,14 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
pbs_repository_cstr.as_ptr(),
ns_cstr.as_ptr(),
backup_id_cstr.as_ptr(),
- args.backup_time as u64,
+ backup_args.backup_time as u64,
PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE,
pbs_password_cstr.as_ptr(),
keyfile_ptr,
key_password_ptr,
master_keyfile_ptr,
- args.compress,
- args.encrypt,
+ pbs_args.compress,
+ pbs_args.encrypt,
fingerprint_cstr.as_ptr(),
&mut pbs_err,
);
@@ -361,17 +367,24 @@ where
Ok(())
}
-fn pbs_client_setup(args: &BackupVmaToPbsArgs) -> Result<(HttpClient, String, Value), Error> {
- let repo: BackupRepository = args.pbs_repository.parse()?;
+fn pbs_client_setup(
+ pbs_args: &PbsArgs,
+ backup_args: &VmaBackupArgs,
+) -> Result<(HttpClient, String, Value), Error> {
+ let repo: BackupRepository = pbs_args.pbs_repository.parse()?;
let options = HttpClientOptions::new_interactive(
- Some(args.pbs_password.clone()),
- Some(args.fingerprint.clone()),
+ Some(pbs_args.pbs_password.clone()),
+ Some(pbs_args.fingerprint.clone()),
);
let client = HttpClient::new(repo.host(), repo.port(), repo.auth_id(), options)?;
- let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time));
+ let backup_dir = BackupDir::from((
+ BackupType::Vm,
+ backup_args.backup_id.clone(),
+ backup_args.backup_time,
+ ));
- let namespace = match &args.namespace {
+ let namespace = match &pbs_args.namespace {
Some(namespace) => BackupNamespace::new(namespace)?,
None => BackupNamespace::root(),
};
@@ -386,45 +399,44 @@ fn pbs_client_setup(args: &BackupVmaToPbsArgs) -> Result<(HttpClient, String, Va
fn upload_log(
client: &HttpClient,
- args: &BackupVmaToPbsArgs,
+ log_file_path: &OsString,
+ pbs_args: &PbsArgs,
store: &str,
request_args: Value,
) -> Result<(), Error> {
- if let Some(log_file_path) = &args.log_file_path {
- let path = format!("api2/json/admin/datastore/{}/upload-backup-log", store);
- let data = std::fs::read(log_file_path)?;
-
- let blob = if args.encrypt {
- let crypt_config = match &args.keyfile {
- None => None,
- Some(keyfile) => {
- let key = std::fs::read(keyfile)?;
- let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
- match &args.key_password {
- Some(key_password) => Ok(key_password.clone().into_bytes()),
- None => bail!("no key password provided"),
- }
- })?;
- let crypt_config = CryptConfig::new(key)?;
- Some(crypt_config)
- }
- };
-
- DataBlob::encode(&data, crypt_config.as_ref(), args.compress)?
- } else {
- // fixme: howto sign log?
- DataBlob::encode(&data, None, args.compress)?
+ let path = format!("api2/json/admin/datastore/{}/upload-backup-log", store);
+ let data = std::fs::read(log_file_path)?;
+
+ let blob = if pbs_args.encrypt {
+ let crypt_config = match &pbs_args.keyfile {
+ None => None,
+ Some(keyfile) => {
+ let key = std::fs::read(keyfile)?;
+ let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
+ match &pbs_args.key_password {
+ Some(key_password) => Ok(key_password.clone().into_bytes()),
+ None => bail!("no key password provided"),
+ }
+ })?;
+ let crypt_config = CryptConfig::new(key)?;
+ Some(crypt_config)
+ }
};
- let body = hyper::Body::from(blob.into_inner());
+ DataBlob::encode(&data, crypt_config.as_ref(), pbs_args.compress)?
+ } else {
+ // fixme: howto sign log?
+ DataBlob::encode(&data, None, pbs_args.compress)?
+ };
- block_on(async {
- client
- .upload("application/octet-stream", body, &path, Some(request_args))
- .await
- .unwrap();
- });
- }
+ let body = hyper::Body::from(blob.into_inner());
+
+ block_on(async {
+ client
+ .upload("application/octet-stream", body, &path, Some(request_args))
+ .await
+ .unwrap();
+ });
Ok(())
}
@@ -444,8 +456,32 @@ fn set_notes(
Ok(())
}
-pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
- let vma_file: Box<dyn BufRead> = match &args.vma_file_path {
+pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
+ let pbs_args = &args.pbs_args;
+ println!("PBS repository: {}", pbs_args.pbs_repository);
+ if let Some(ns) = &pbs_args.namespace {
+ println!("PBS namespace: {}", ns);
+ }
+ println!("PBS fingerprint: {}", pbs_args.fingerprint);
+ println!("compress: {}", pbs_args.compress);
+ println!("encrypt: {}", pbs_args.encrypt);
+
+ let start_transfer_time = SystemTime::now();
+
+ upload_vma_file(pbs_args, &args.vma_args)?;
+
+ let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
+ let total_seconds = transfer_duration.as_secs();
+ let minutes = total_seconds / 60;
+ let seconds = total_seconds % 60;
+ let milliseconds = transfer_duration.as_millis() % 1000;
+ println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
+
+ Ok(())
+}
+
+fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
+ let vma_file: Box<dyn BufRead> = match &backup_args.vma_file_path {
Some(vma_file_path) => match File::open(vma_file_path) {
Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
Ok(file) => Box::new(BufReader::new(file)),
@@ -454,7 +490,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
};
let vma_reader = VmaReader::new(vma_file)?;
- let pbs = create_pbs_backup_task(&args)?;
+ let pbs = create_pbs_backup_task(pbs_args, backup_args)?;
defer! {
proxmox_backup_disconnect(pbs);
@@ -467,10 +503,6 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
handle_pbs_error(pbs_err, "proxmox_backup_connect")?;
}
- println!("Connected to Proxmox Backup Server");
-
- let start_transfer_time = SystemTime::now();
-
upload_configs(&vma_reader, pbs)?;
upload_block_devices(vma_reader, pbs)?;
@@ -478,24 +510,23 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
handle_pbs_error(pbs_err, "proxmox_backup_finish")?;
}
- if args.notes.is_some() || args.log_file_path.is_some() {
- let (client, store, request_args) = pbs_client_setup(&args)?;
-
- if args.log_file_path.is_some() {
- upload_log(&client, &args, &store, request_args.clone())?;
+ if backup_args.notes.is_some() || backup_args.log_file_path.is_some() {
+ let (client, store, request_args) = pbs_client_setup(pbs_args, backup_args)?;
+
+ if let Some(log_file_path) = &backup_args.log_file_path {
+ upload_log(
+ &client,
+ log_file_path,
+ pbs_args,
+ &store,
+ request_args.clone(),
+ )?;
}
- if let Some(notes) = args.notes {
- set_notes(&client, ¬es, &store, request_args)?;
+ if let Some(notes) = &backup_args.notes {
+ set_notes(&client, notes, &store, request_args)?;
}
}
- let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
- let total_seconds = transfer_duration.as_secs();
- let minutes = total_seconds / 60;
- let seconds = total_seconds % 60;
- let milliseconds = transfer_duration.as_millis() % 1000;
- println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
-
Ok(())
}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 vma-to-pbs 2/5] add support for bulk import of a dump directory
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 1/5] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
@ 2024-10-08 14:34 ` Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 3/5] remove hard coded values Filip Schauer
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:34 UTC (permalink / raw)
To: pbs-devel
When a path to a directory is provided in the vma_file argument, try to
upload all VMA backups in the directory. This also handles compressed
VMA files, notes and logs. If a vmid is specified with --vmid, only the
backups of that particular vmid are uploaded.
This is intended for use on a dump directory:
PBS_FINGERPRINT='PBS_FINGERPRINT' vma-to-pbs \
--repository 'user@realm!token@server:port:datastore' \
/var/lib/vz/dump
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 3 ++
src/main.rs | 124 ++++++++++++++++++++++++++++++++++++++++++++-----
src/vma2pbs.rs | 64 ++++++++++++++++++++++---
3 files changed, 173 insertions(+), 18 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index cd13426..fcbc2a3 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -7,9 +7,12 @@ edition = "2021"
[dependencies]
anyhow = "1.0"
bincode = "1.3"
+chrono = "0.4"
hyper = "0.14.5"
+itertools = "0.10"
pico-args = "0.5"
md5 = "0.7.0"
+regex = "1.7"
scopeguard = "1.1.0"
serde = "1.0"
serde_json = "1.0"
diff --git a/src/main.rs b/src/main.rs
index 3e25591..d001d81 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,26 +1,33 @@
use std::ffi::OsString;
+use std::fs::read_dir;
+use std::path::PathBuf;
use anyhow::{bail, Context, Error};
+use chrono::NaiveDateTime;
+use itertools::Itertools;
use proxmox_sys::linux::tty;
use proxmox_time::epoch_i64;
+use regex::Regex;
mod vma;
mod vma2pbs;
-use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
+use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, Compression, PbsArgs, VmaBackupArgs};
const CMD_HELP: &str = "\
Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
Arguments:
- [vma_file]
+ [vma_file | dump_directory]
Options:
--repository <auth_id@host:port:datastore>
Repository URL
[--ns <NAMESPACE>]
Namespace
- --vmid <VMID>
+ [--vmid <VMID>]
Backup ID
+ If not specified, bulk import all VMA backups in the provided directory.
+ If specified with a dump directory, only import backups of the specified vmid.
[--backup-time <EPOCH>]
Backup timestamp
--fingerprint <FINGERPRINT>
@@ -87,7 +94,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let pbs_repository = args.value_from_str("--repository")?;
let namespace = args.opt_value_from_str("--ns")?;
- let vmid = args.value_from_str("--vmid")?;
+ let vmid = args.opt_value_from_str("--vmid")?;
let backup_time: Option<i64> = args.opt_value_from_str("--backup-time")?;
let backup_time = backup_time.unwrap_or_else(epoch_i64);
let fingerprint = args.opt_value_from_str("--fingerprint")?;
@@ -196,15 +203,110 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
encrypt,
};
- let vma_args = VmaBackupArgs {
- vma_file_path: vma_file_path.cloned(),
- backup_id: vmid,
- backup_time,
- notes,
- log_file_path,
+ let mut grouped_vmas = Vec::new();
+
+ let bulk = if let Some(vma_file_path) = vma_file_path {
+ PathBuf::from(vma_file_path).is_dir()
+ } else {
+ false
};
- let options = BackupVmaToPbsArgs { pbs_args, vma_args };
+ if bulk {
+ let dump_dir_path =
+ PathBuf::from(vma_file_path.expect("no directory specified for bulk import"));
+
+ let re = Regex::new(
+ r"vzdump-qemu-(\d+)-(\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}).vma(|.zst|.lzo|.gz)$",
+ )?;
+
+ let mut vmas = Vec::new();
+
+ for entry in read_dir(dump_dir_path)? {
+ let entry = entry?;
+ let path = entry.path();
+
+ if !path.is_file() {
+ continue;
+ }
+
+ if let Some(file_name) = path.file_name().and_then(|n| n.to_str()) {
+ let Some((_, [backup_id, timestr, ext])) =
+ re.captures(file_name).map(|c| c.extract())
+ else {
+ // Skip the file, since it is not a VMA backup
+ continue;
+ };
+
+ if let Some(ref vmid) = vmid {
+ if backup_id != vmid {
+ // Skip the backup, since it does not match the specified vmid
+ continue;
+ }
+ }
+
+ let compression = match ext {
+ "" => None,
+ ".zst" => Some(Compression::Zstd),
+ ".lzo" => Some(Compression::Lzo),
+ ".gz" => Some(Compression::GZip),
+ _ => continue,
+ };
+
+ let backup_time = NaiveDateTime::parse_from_str(timestr, "%Y_%m_%d-%H_%M_%S")?
+ .and_utc()
+ .timestamp();
+
+ let mut notes_path_os_string: OsString = path.clone().into();
+ notes_path_os_string.push(".notes");
+ let notes_path: PathBuf = notes_path_os_string.into();
+ let notes = proxmox_sys::fs::file_read_optional_string(notes_path)?;
+
+ let mut log_path_os_string: OsString = path.clone().into();
+ log_path_os_string.push(".log");
+ let log_path: PathBuf = log_path_os_string.into();
+ let log_file_path = if log_path.exists() {
+ Some(log_path.to_path_buf().into_os_string())
+ } else {
+ None
+ };
+
+ let backup_args = VmaBackupArgs {
+ vma_file_path: Some(path.clone().into()),
+ compression,
+ backup_id: backup_id.to_string(),
+ backup_time,
+ notes,
+ log_file_path,
+ };
+ vmas.push(backup_args);
+ }
+ }
+
+ vmas.sort_by_key(|d| d.backup_time);
+ grouped_vmas = vmas
+ .into_iter()
+ .into_group_map_by(|d| d.backup_id.clone())
+ .into_values()
+ .collect();
+ grouped_vmas.sort_by_key(|d| d[0].backup_id.clone());
+ } else if let Some(vmid) = vmid {
+ let vmas = vec![VmaBackupArgs {
+ vma_file_path: vma_file_path.cloned(),
+ compression: None,
+ backup_id: vmid,
+ backup_time,
+ notes,
+ log_file_path,
+ }];
+ grouped_vmas.push(vmas);
+ } else {
+ bail!("No vmid specified for single backup file");
+ }
+
+ let options = BackupVmaToPbsArgs {
+ pbs_args,
+ grouped_vmas,
+ };
Ok(options)
}
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 37ea308..05294f3 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -4,6 +4,7 @@ use std::collections::HashMap;
use std::ffi::{c_char, CStr, CString, OsString};
use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
+use std::process::{Command, Stdio};
use std::ptr;
use std::time::SystemTime;
@@ -30,7 +31,7 @@ const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
pub pbs_args: PbsArgs,
- pub vma_args: VmaBackupArgs,
+ pub grouped_vmas: Vec<Vec<VmaBackupArgs>>,
}
pub struct PbsArgs {
@@ -45,8 +46,15 @@ pub struct PbsArgs {
pub encrypt: bool,
}
+pub enum Compression {
+ Zstd,
+ Lzo,
+ GZip,
+}
+
pub struct VmaBackupArgs {
pub vma_file_path: Option<OsString>,
+ pub compression: Option<Compression>,
pub backup_id: String,
pub backup_time: i64,
pub notes: Option<String>,
@@ -468,7 +476,19 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let start_transfer_time = SystemTime::now();
- upload_vma_file(pbs_args, &args.vma_args)?;
+ for vma_group in args.grouped_vmas {
+ for backup_args in vma_group {
+ if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
+ eprintln!(
+ "Failed to upload vma file at {:?} - {}",
+ backup_args.vma_file_path.expect("missing VMA file path"),
+ e
+ );
+ println!("Skipping VMID {}", backup_args.backup_id);
+ break;
+ }
+ }
+ }
let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
let total_seconds = transfer_duration.as_secs();
@@ -481,13 +501,43 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
- let vma_file: Box<dyn BufRead> = match &backup_args.vma_file_path {
- Some(vma_file_path) => match File::open(vma_file_path) {
- Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
- Ok(file) => Box::new(BufReader::new(file)),
+ match &backup_args.vma_file_path {
+ Some(vma_file_path) => println!("Uploading VMA backup from {:?}", vma_file_path),
+ None => println!("Uploading VMA backup from (stdin)"),
+ };
+
+ let vma_file: Box<dyn BufRead> = match &backup_args.compression {
+ Some(compression) => {
+ let vma_file_path = backup_args
+ .vma_file_path
+ .as_ref()
+ .expect("No VMA file path provided");
+ let mut cmd = match compression {
+ Compression::Zstd => {
+ let mut cmd = Command::new("zstd");
+ cmd.args(["-q", "-d", "-c"]);
+ cmd
+ }
+ Compression::Lzo => {
+ let mut cmd = Command::new("lzop");
+ cmd.args(["-d", "-c"]);
+ cmd
+ }
+ Compression::GZip => Command::new("zcat"),
+ };
+ let process = cmd.arg(vma_file_path).stdout(Stdio::piped()).spawn()?;
+ let stdout = process.stdout.expect("Failed to capture stdout");
+ Box::new(BufReader::new(stdout))
+ }
+ None => match &backup_args.vma_file_path {
+ Some(vma_file_path) => match File::open(vma_file_path) {
+ Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
+ Ok(file) => Box::new(BufReader::new(file)),
+ },
+ None => Box::new(BufReader::new(stdin())),
},
- None => Box::new(BufReader::new(stdin())),
};
+
let vma_reader = VmaReader::new(vma_file)?;
let pbs = create_pbs_backup_task(pbs_args, backup_args)?;
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 vma-to-pbs 3/5] remove hard coded values
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 1/5] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 2/5] add support for bulk import of a dump directory Filip Schauer
@ 2024-10-08 14:34 ` Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 4/5] use level-based logging instead of println Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 5/5] log device upload progress as a percentage Filip Schauer
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:34 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/vma.rs | 2 +-
src/vma2pbs.rs | 23 +++++++++++------------
2 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/src/vma.rs b/src/vma.rs
index 518de8a..63ee3b5 100644
--- a/src/vma.rs
+++ b/src/vma.rs
@@ -22,7 +22,7 @@ const VMA_MAX_CONFIGS: usize = 256;
/// Maximum number of block devices
/// See VMA Header in pve-qemu.git/vma_spec.txt
-const VMA_MAX_DEVICES: usize = 256;
+pub const VMA_MAX_DEVICES: usize = 256;
/// VMA magic string
/// See VMA Header in pve-qemu.git/vma_spec.txt
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 05294f3..1eb510c 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -25,7 +25,7 @@ use proxmox_time::epoch_to_rfc3339;
use scopeguard::defer;
use serde_json::Value;
-use crate::vma::VmaReader;
+use crate::vma::{VmaReader, VMA_MAX_DEVICES};
const VMA_CLUSTER_SIZE: usize = 65536;
@@ -173,20 +173,21 @@ where
fn register_block_devices<T>(
vma_reader: &VmaReader<T>,
pbs: *mut ProxmoxBackupHandle,
-) -> Result<[Option<BlockDeviceInfo>; 256], Error>
+) -> Result<[Option<BlockDeviceInfo>; VMA_MAX_DEVICES], Error>
where
T: Read,
{
- let mut block_device_infos: [Option<BlockDeviceInfo>; 256] = [None; 256];
+ let mut block_device_infos: [Option<BlockDeviceInfo>; VMA_MAX_DEVICES] =
+ [None; VMA_MAX_DEVICES];
let mut pbs_err: *mut c_char = ptr::null_mut();
- for device_id in 0..255 {
- if !vma_reader.contains_device(device_id) {
+ for (device_id, block_device_info) in block_device_infos.iter_mut().enumerate() {
+ if !vma_reader.contains_device(device_id.try_into()?) {
continue;
}
- let device_name = vma_reader.get_device_name(device_id)?;
- let device_size = vma_reader.get_device_size(device_id)?;
+ let device_name = vma_reader.get_device_name(device_id.try_into()?)?;
+ let device_size = vma_reader.get_device_size(device_id.try_into()?)?;
println!(
"DEV: dev_id={} size: {} devname: {}",
@@ -206,12 +207,10 @@ where
handle_pbs_error(pbs_err, "proxmox_backup_register_image")?;
}
- let block_device_info = BlockDeviceInfo {
+ *block_device_info = Some(BlockDeviceInfo {
pbs_device_id: pbs_device_id as u8,
device_size,
- };
-
- block_device_infos[device_id as usize] = Some(block_device_info);
+ });
}
Ok(block_device_infos)
@@ -359,7 +358,7 @@ where
let mut pbs_err: *mut c_char = ptr::null_mut();
- for block_device_info in block_device_infos.iter().take(255) {
+ for block_device_info in block_device_infos {
let block_device_info = match block_device_info {
Some(block_device_info) => block_device_info,
None => continue,
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 vma-to-pbs 4/5] use level-based logging instead of println
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
` (2 preceding siblings ...)
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 3/5] remove hard coded values Filip Schauer
@ 2024-10-08 14:34 ` Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 5/5] log device upload progress as a percentage Filip Schauer
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:34 UTC (permalink / raw)
To: pbs-devel
Use log level "info" by default and prevent spamming messages for every
single chunk uploaded. To re-enable these messages, set the RUST_LOG
environment variable to "debug".
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 2 ++
src/main.rs | 14 ++++++++++++--
src/vma2pbs.rs | 38 ++++++++++++++++++++------------------
3 files changed, 34 insertions(+), 20 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index fcbc2a3..bb9c0eb 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,8 +8,10 @@ edition = "2021"
anyhow = "1.0"
bincode = "1.3"
chrono = "0.4"
+env_logger = "0.10"
hyper = "0.14.5"
itertools = "0.10"
+log = "0.4"
pico-args = "0.5"
md5 = "0.7.0"
regex = "1.7"
diff --git a/src/main.rs b/src/main.rs
index d001d81..a9a32cf 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -109,7 +109,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
- (false, true) => println!(
+ (false, true) => log::info!(
"--keyfile given, but --encrypt not set -> backup will be signed, but not encrypted!"
),
_ => {}
@@ -171,7 +171,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Some(key_password)
} else if vma_file_path.is_none() {
- println!(
+ log::info!(
"Please use --key-password-file to provide the password when passing the VMA file \
to stdin, if required."
);
@@ -311,7 +311,17 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Ok(options)
}
+fn init_cli_logger() {
+ env_logger::Builder::from_env(env_logger::Env::new().filter_or("RUST_LOG", "info"))
+ .format_level(false)
+ .format_target(false)
+ .format_timestamp(None)
+ .init();
+}
+
fn main() -> Result<(), Error> {
+ init_cli_logger();
+
let args = parse_args()?;
vma2pbs(args)?;
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 1eb510c..a835366 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -81,8 +81,8 @@ fn create_pbs_backup_task(
pbs_args: &PbsArgs,
backup_args: &VmaBackupArgs,
) -> Result<*mut ProxmoxBackupHandle, Error> {
- println!(
- "backup time: {}",
+ log::info!(
+ "\tbackup time: {}",
epoch_to_rfc3339(backup_args.backup_time)?
);
@@ -151,7 +151,7 @@ where
let config_name = config.name;
let config_data = config.content;
- println!("CFG: size: {} name: {}", config_data.len(), config_name);
+ log::info!("\tCFG: size: {} name: {}", config_data.len(), config_name);
let config_name_cstr = CString::new(config_name)?;
@@ -189,9 +189,11 @@ where
let device_name = vma_reader.get_device_name(device_id.try_into()?)?;
let device_size = vma_reader.get_device_size(device_id.try_into()?)?;
- println!(
- "DEV: dev_id={} size: {} devname: {}",
- device_id, device_size, device_name
+ log::info!(
+ "\tDEV: dev_id={} size: {} devname: {}",
+ device_id,
+ device_size,
+ device_name
);
let device_name_cstr = CString::new(device_name)?;
@@ -275,8 +277,8 @@ where
};
let pbs_upload_chunk = |pbs_chunk_buffer: Option<&[u8]>| {
- println!(
- "Uploading dev_id: {} offset: {:#0X} - {:#0X}",
+ log::debug!(
+ "\tUploading dev_id: {} offset: {:#0X} - {:#0X}",
dev_id,
pbs_chunk_offset,
pbs_chunk_offset + pbs_chunk_size,
@@ -465,25 +467,25 @@ fn set_notes(
pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let pbs_args = &args.pbs_args;
- println!("PBS repository: {}", pbs_args.pbs_repository);
+ log::info!("PBS repository: {}", pbs_args.pbs_repository);
if let Some(ns) = &pbs_args.namespace {
- println!("PBS namespace: {}", ns);
+ log::info!("PBS namespace: {}", ns);
}
- println!("PBS fingerprint: {}", pbs_args.fingerprint);
- println!("compress: {}", pbs_args.compress);
- println!("encrypt: {}", pbs_args.encrypt);
+ log::info!("PBS fingerprint: {}", pbs_args.fingerprint);
+ log::info!("compress: {}", pbs_args.compress);
+ log::info!("encrypt: {}", pbs_args.encrypt);
let start_transfer_time = SystemTime::now();
for vma_group in args.grouped_vmas {
for backup_args in vma_group {
if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
- eprintln!(
+ log::warn!(
"Failed to upload vma file at {:?} - {}",
backup_args.vma_file_path.expect("missing VMA file path"),
e
);
- println!("Skipping VMID {}", backup_args.backup_id);
+ log::info!("Skipping VMID {}", backup_args.backup_id);
break;
}
}
@@ -494,15 +496,15 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let minutes = total_seconds / 60;
let seconds = total_seconds % 60;
let milliseconds = transfer_duration.as_millis() % 1000;
- println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
+ log::info!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
Ok(())
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
match &backup_args.vma_file_path {
- Some(vma_file_path) => println!("Uploading VMA backup from {:?}", vma_file_path),
- None => println!("Uploading VMA backup from (stdin)"),
+ Some(vma_file_path) => log::info!("Uploading VMA backup from {:?}", vma_file_path),
+ None => log::info!("Uploading VMA backup from (stdin)"),
};
let vma_file: Box<dyn BufRead> = match &backup_args.compression {
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 vma-to-pbs 5/5] log device upload progress as a percentage
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
` (3 preceding siblings ...)
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 4/5] use level-based logging instead of println Filip Schauer
@ 2024-10-08 14:34 ` Filip Schauer
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-10-08 14:34 UTC (permalink / raw)
To: pbs-devel
Log the upload progress of a device as a percentage with log level info
every 1000 chunks.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/vma2pbs.rs | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index a835366..817d9c4 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -6,6 +6,7 @@ use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
use std::process::{Command, Stdio};
use std::ptr;
+use std::sync::{Arc, Mutex};
use std::time::SystemTime;
use anyhow::{anyhow, bail, Error};
@@ -233,6 +234,8 @@ where
non_zero_mask: u64,
}
+ let chunk_stats = Arc::new(Mutex::new([0u64; VMA_MAX_DEVICES]));
+
let images_chunks: RefCell<HashMap<u8, HashMap<u64, ImageChunk>>> =
RefCell::new(HashMap::new());
@@ -283,6 +286,14 @@ where
pbs_chunk_offset,
pbs_chunk_offset + pbs_chunk_size,
);
+ let mut chunk_stats_locked = chunk_stats.lock().unwrap();
+ chunk_stats_locked[dev_id as usize] += 1;
+ if (chunk_stats_locked[dev_id as usize] % 1000) == 0 {
+ let percentage =
+ 100 * PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE * chunk_stats_locked[dev_id as usize]
+ / device_size;
+ log::info!("\tUploading dev_id: {} ({}%)", dev_id, percentage);
+ }
let mut pbs_err: *mut c_char = ptr::null_mut();
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-10-08 14:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-10-08 14:33 [pbs-devel] [PATCH v2 vma-to-pbs 0/5] add support for bulk import of a dump directory Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 1/5] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 2/5] add support for bulk import of a dump directory Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 3/5] remove hard coded values Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 4/5] use level-based logging instead of println Filip Schauer
2024-10-08 14:34 ` [pbs-devel] [PATCH v2 vma-to-pbs 5/5] log device upload progress as a percentage Filip Schauer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox