* [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory
@ 2024-10-30 13:55 Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 1/6] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
` (6 more replies)
0 siblings, 7 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
When a path to a directory is provided in the vma_file argument, try to
upload all VMA backups in the directory. This also handles compressed
VMA files, notes and logs. If a vmid is specified with --vmid, only the
backups of that particular vmid are uploaded.
Also improve the readability of the log messages to keep track of all
imported backups.
Changed since v3:
* Mention in the description of the --vmid argument, that it is required
if a single VMA file is provided
* Construct grouped_vmas in place
* Add debug logs when gathering files for bulk import
* Log a summary of the files gathered for bulk import
* Remove the "confusing VMA file path" error message in the second
commit
* Switch chunk_stats from Arc<Mutex<[u64; 256]>> to
Arc<[AtomicU64; 256]> and use fetch_add to atomically increment and
fetch the chunk stat
* Ask for confirmation before bulk import
* Add --yes option to skip the confirmation prompt
Changed since v2:
* Make skipping a VMID on error optional with the --skip-failed option
* Switch log output from stderr to stdout
* Bump itertools to 0.13
Changed since v1:
* Do not recurse through dump directory
* Compile regex once before iterating over the files in the dump
directory
* Use extract on regex capture groups
* Do not use deprecated method `chrono::NaiveDateTime::timestamp`
* Use proxmox_sys::fs::file_read_optional_string
* Group VMA files by VMID and continue with next VMID on error
* Move the BackupVmaToPbsArgs split into its own commit
* Remove hard coded occurences of 255
* Use level-based logging instead of println
Filip Schauer (6):
split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs
add support for bulk import of a dump directory
add option to skip vmids whose backups failed to upload
remove hard coded values
use level-based logging instead of println
log device upload progress as a percentage
Cargo.toml | 5 +
src/main.rs | 192 +++++++++++++++++++++++++++---
src/vma.rs | 2 +-
src/vma2pbs.rs | 310 ++++++++++++++++++++++++++++++++-----------------
4 files changed, 388 insertions(+), 121 deletions(-)
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 1/6] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory Filip Schauer
` (5 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/main.rs | 17 +++--
src/vma2pbs.rs | 199 ++++++++++++++++++++++++++++---------------------
2 files changed, 126 insertions(+), 90 deletions(-)
diff --git a/src/main.rs b/src/main.rs
index de789c1..3e25591 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -6,7 +6,7 @@ use proxmox_time::epoch_i64;
mod vma;
mod vma2pbs;
-use vma2pbs::{backup_vma_to_pbs, BackupVmaToPbsArgs};
+use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
const CMD_HELP: &str = "\
Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
@@ -184,12 +184,9 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
None
};
- let options = BackupVmaToPbsArgs {
- vma_file_path: vma_file_path.cloned(),
+ let pbs_args = PbsArgs {
pbs_repository,
namespace,
- backup_id: vmid,
- backup_time,
pbs_password,
keyfile,
key_password,
@@ -197,16 +194,24 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
fingerprint,
compress,
encrypt,
+ };
+
+ let vma_args = VmaBackupArgs {
+ vma_file_path: vma_file_path.cloned(),
+ backup_id: vmid,
+ backup_time,
notes,
log_file_path,
};
+ let options = BackupVmaToPbsArgs { pbs_args, vma_args };
+
Ok(options)
}
fn main() -> Result<(), Error> {
let args = parse_args()?;
- backup_vma_to_pbs(args)?;
+ vma2pbs(args)?;
Ok(())
}
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index d2ce437..37ea308 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -29,11 +29,13 @@ use crate::vma::VmaReader;
const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
- pub vma_file_path: Option<OsString>,
+ pub pbs_args: PbsArgs,
+ pub vma_args: VmaBackupArgs,
+}
+
+pub struct PbsArgs {
pub pbs_repository: String,
pub namespace: Option<String>,
- pub backup_id: String,
- pub backup_time: i64,
pub pbs_password: String,
pub keyfile: Option<String>,
pub key_password: Option<String>,
@@ -41,6 +43,12 @@ pub struct BackupVmaToPbsArgs {
pub fingerprint: String,
pub compress: bool,
pub encrypt: bool,
+}
+
+pub struct VmaBackupArgs {
+ pub vma_file_path: Option<OsString>,
+ pub backup_id: String,
+ pub backup_time: i64,
pub notes: Option<String>,
pub log_file_path: Option<OsString>,
}
@@ -61,25 +69,23 @@ fn handle_pbs_error(pbs_err: *mut c_char, function_name: &str) -> Result<(), Err
bail!("{function_name} failed: {pbs_err_str}");
}
-fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBackupHandle, Error> {
- println!("PBS repository: {}", args.pbs_repository);
- if let Some(ns) = &args.namespace {
- println!("PBS namespace: {}", ns);
- }
- println!("PBS fingerprint: {}", args.fingerprint);
- println!("compress: {}", args.compress);
- println!("encrypt: {}", args.encrypt);
-
- println!("backup time: {}", epoch_to_rfc3339(args.backup_time)?);
+fn create_pbs_backup_task(
+ pbs_args: &PbsArgs,
+ backup_args: &VmaBackupArgs,
+) -> Result<*mut ProxmoxBackupHandle, Error> {
+ println!(
+ "backup time: {}",
+ epoch_to_rfc3339(backup_args.backup_time)?
+ );
let mut pbs_err: *mut c_char = ptr::null_mut();
- let pbs_repository_cstr = CString::new(args.pbs_repository.as_str())?;
- let ns_cstr = CString::new(args.namespace.as_deref().unwrap_or(""))?;
- let backup_id_cstr = CString::new(args.backup_id.as_str())?;
- let pbs_password_cstr = CString::new(args.pbs_password.as_str())?;
- let fingerprint_cstr = CString::new(args.fingerprint.as_str())?;
- let keyfile_cstr = args
+ let pbs_repository_cstr = CString::new(pbs_args.pbs_repository.as_str())?;
+ let ns_cstr = CString::new(pbs_args.namespace.as_deref().unwrap_or(""))?;
+ let backup_id_cstr = CString::new(backup_args.backup_id.as_str())?;
+ let pbs_password_cstr = CString::new(pbs_args.pbs_password.as_str())?;
+ let fingerprint_cstr = CString::new(pbs_args.fingerprint.as_str())?;
+ let keyfile_cstr = pbs_args
.keyfile
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -87,7 +93,7 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let key_password_cstr = args
+ let key_password_cstr = pbs_args
.key_password
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -95,7 +101,7 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
.as_ref()
.map(|v| v.as_ptr())
.unwrap_or(ptr::null());
- let master_keyfile_cstr = args
+ let master_keyfile_cstr = pbs_args
.master_keyfile
.as_ref()
.map(|v| CString::new(v.as_str()).unwrap());
@@ -108,14 +114,14 @@ fn create_pbs_backup_task(args: &BackupVmaToPbsArgs) -> Result<*mut ProxmoxBacku
pbs_repository_cstr.as_ptr(),
ns_cstr.as_ptr(),
backup_id_cstr.as_ptr(),
- args.backup_time as u64,
+ backup_args.backup_time as u64,
PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE,
pbs_password_cstr.as_ptr(),
keyfile_ptr,
key_password_ptr,
master_keyfile_ptr,
- args.compress,
- args.encrypt,
+ pbs_args.compress,
+ pbs_args.encrypt,
fingerprint_cstr.as_ptr(),
&mut pbs_err,
);
@@ -361,17 +367,24 @@ where
Ok(())
}
-fn pbs_client_setup(args: &BackupVmaToPbsArgs) -> Result<(HttpClient, String, Value), Error> {
- let repo: BackupRepository = args.pbs_repository.parse()?;
+fn pbs_client_setup(
+ pbs_args: &PbsArgs,
+ backup_args: &VmaBackupArgs,
+) -> Result<(HttpClient, String, Value), Error> {
+ let repo: BackupRepository = pbs_args.pbs_repository.parse()?;
let options = HttpClientOptions::new_interactive(
- Some(args.pbs_password.clone()),
- Some(args.fingerprint.clone()),
+ Some(pbs_args.pbs_password.clone()),
+ Some(pbs_args.fingerprint.clone()),
);
let client = HttpClient::new(repo.host(), repo.port(), repo.auth_id(), options)?;
- let backup_dir = BackupDir::from((BackupType::Vm, args.backup_id.clone(), args.backup_time));
+ let backup_dir = BackupDir::from((
+ BackupType::Vm,
+ backup_args.backup_id.clone(),
+ backup_args.backup_time,
+ ));
- let namespace = match &args.namespace {
+ let namespace = match &pbs_args.namespace {
Some(namespace) => BackupNamespace::new(namespace)?,
None => BackupNamespace::root(),
};
@@ -386,45 +399,44 @@ fn pbs_client_setup(args: &BackupVmaToPbsArgs) -> Result<(HttpClient, String, Va
fn upload_log(
client: &HttpClient,
- args: &BackupVmaToPbsArgs,
+ log_file_path: &OsString,
+ pbs_args: &PbsArgs,
store: &str,
request_args: Value,
) -> Result<(), Error> {
- if let Some(log_file_path) = &args.log_file_path {
- let path = format!("api2/json/admin/datastore/{}/upload-backup-log", store);
- let data = std::fs::read(log_file_path)?;
-
- let blob = if args.encrypt {
- let crypt_config = match &args.keyfile {
- None => None,
- Some(keyfile) => {
- let key = std::fs::read(keyfile)?;
- let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
- match &args.key_password {
- Some(key_password) => Ok(key_password.clone().into_bytes()),
- None => bail!("no key password provided"),
- }
- })?;
- let crypt_config = CryptConfig::new(key)?;
- Some(crypt_config)
- }
- };
-
- DataBlob::encode(&data, crypt_config.as_ref(), args.compress)?
- } else {
- // fixme: howto sign log?
- DataBlob::encode(&data, None, args.compress)?
+ let path = format!("api2/json/admin/datastore/{}/upload-backup-log", store);
+ let data = std::fs::read(log_file_path)?;
+
+ let blob = if pbs_args.encrypt {
+ let crypt_config = match &pbs_args.keyfile {
+ None => None,
+ Some(keyfile) => {
+ let key = std::fs::read(keyfile)?;
+ let (key, _created, _) = decrypt_key(&key, &|| -> Result<Vec<u8>, Error> {
+ match &pbs_args.key_password {
+ Some(key_password) => Ok(key_password.clone().into_bytes()),
+ None => bail!("no key password provided"),
+ }
+ })?;
+ let crypt_config = CryptConfig::new(key)?;
+ Some(crypt_config)
+ }
};
- let body = hyper::Body::from(blob.into_inner());
+ DataBlob::encode(&data, crypt_config.as_ref(), pbs_args.compress)?
+ } else {
+ // fixme: howto sign log?
+ DataBlob::encode(&data, None, pbs_args.compress)?
+ };
- block_on(async {
- client
- .upload("application/octet-stream", body, &path, Some(request_args))
- .await
- .unwrap();
- });
- }
+ let body = hyper::Body::from(blob.into_inner());
+
+ block_on(async {
+ client
+ .upload("application/octet-stream", body, &path, Some(request_args))
+ .await
+ .unwrap();
+ });
Ok(())
}
@@ -444,8 +456,32 @@ fn set_notes(
Ok(())
}
-pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
- let vma_file: Box<dyn BufRead> = match &args.vma_file_path {
+pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
+ let pbs_args = &args.pbs_args;
+ println!("PBS repository: {}", pbs_args.pbs_repository);
+ if let Some(ns) = &pbs_args.namespace {
+ println!("PBS namespace: {}", ns);
+ }
+ println!("PBS fingerprint: {}", pbs_args.fingerprint);
+ println!("compress: {}", pbs_args.compress);
+ println!("encrypt: {}", pbs_args.encrypt);
+
+ let start_transfer_time = SystemTime::now();
+
+ upload_vma_file(pbs_args, &args.vma_args)?;
+
+ let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
+ let total_seconds = transfer_duration.as_secs();
+ let minutes = total_seconds / 60;
+ let seconds = total_seconds % 60;
+ let milliseconds = transfer_duration.as_millis() % 1000;
+ println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
+
+ Ok(())
+}
+
+fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
+ let vma_file: Box<dyn BufRead> = match &backup_args.vma_file_path {
Some(vma_file_path) => match File::open(vma_file_path) {
Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
Ok(file) => Box::new(BufReader::new(file)),
@@ -454,7 +490,7 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
};
let vma_reader = VmaReader::new(vma_file)?;
- let pbs = create_pbs_backup_task(&args)?;
+ let pbs = create_pbs_backup_task(pbs_args, backup_args)?;
defer! {
proxmox_backup_disconnect(pbs);
@@ -467,10 +503,6 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
handle_pbs_error(pbs_err, "proxmox_backup_connect")?;
}
- println!("Connected to Proxmox Backup Server");
-
- let start_transfer_time = SystemTime::now();
-
upload_configs(&vma_reader, pbs)?;
upload_block_devices(vma_reader, pbs)?;
@@ -478,24 +510,23 @@ pub fn backup_vma_to_pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
handle_pbs_error(pbs_err, "proxmox_backup_finish")?;
}
- if args.notes.is_some() || args.log_file_path.is_some() {
- let (client, store, request_args) = pbs_client_setup(&args)?;
-
- if args.log_file_path.is_some() {
- upload_log(&client, &args, &store, request_args.clone())?;
+ if backup_args.notes.is_some() || backup_args.log_file_path.is_some() {
+ let (client, store, request_args) = pbs_client_setup(pbs_args, backup_args)?;
+
+ if let Some(log_file_path) = &backup_args.log_file_path {
+ upload_log(
+ &client,
+ log_file_path,
+ pbs_args,
+ &store,
+ request_args.clone(),
+ )?;
}
- if let Some(notes) = args.notes {
- set_notes(&client, ¬es, &store, request_args)?;
+ if let Some(notes) = &backup_args.notes {
+ set_notes(&client, notes, &store, request_args)?;
}
}
- let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
- let total_seconds = transfer_duration.as_secs();
- let minutes = total_seconds / 60;
- let seconds = total_seconds % 60;
- let milliseconds = transfer_duration.as_millis() % 1000;
- println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
-
Ok(())
}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 1/6] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-11-04 13:06 ` Fabian Grünbichler
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 3/6] add option to skip vmids whose backups failed to upload Filip Schauer
` (4 subsequent siblings)
6 siblings, 1 reply; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
When a path to a directory is provided in the vma_file argument, try to
upload all VMA backups in the directory. This also handles compressed
VMA files, notes and logs. If a vmid is specified with --vmid, only the
backups of that particular vmid are uploaded.
This is intended for use on a dump directory:
PBS_FINGERPRINT='PBS_FINGERPRINT' vma-to-pbs \
--repository 'user@realm!token@server:port:datastore' \
/var/lib/vz/dump
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 3 +
src/main.rs | 161 +++++++++++++++++++++++++++++++++++++++++++++----
src/vma2pbs.rs | 64 +++++++++++++++++---
3 files changed, 209 insertions(+), 19 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index cd13426..5c6a175 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -7,9 +7,12 @@ edition = "2021"
[dependencies]
anyhow = "1.0"
bincode = "1.3"
+chrono = "0.4"
hyper = "0.14.5"
+itertools = "0.13"
pico-args = "0.5"
md5 = "0.7.0"
+regex = "1.7"
scopeguard = "1.1.0"
serde = "1.0"
serde_json = "1.0"
diff --git a/src/main.rs b/src/main.rs
index 3e25591..4c5135b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,26 +1,35 @@
use std::ffi::OsString;
+use std::fs::read_dir;
+use std::io::{BufRead, BufReader};
+use std::path::PathBuf;
use anyhow::{bail, Context, Error};
+use chrono::NaiveDateTime;
+use itertools::Itertools;
use proxmox_sys::linux::tty;
use proxmox_time::epoch_i64;
+use regex::Regex;
mod vma;
mod vma2pbs;
-use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
+use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, Compression, PbsArgs, VmaBackupArgs};
const CMD_HELP: &str = "\
Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
Arguments:
- [vma_file]
+ [vma_file | dump_directory]
Options:
--repository <auth_id@host:port:datastore>
Repository URL
[--ns <NAMESPACE>]
Namespace
- --vmid <VMID>
+ [--vmid <VMID>]
Backup ID
+ This is required if a single VMA file is provided.
+ If not specified, bulk import all VMA backups in the provided directory.
+ If specified with a dump directory, only import backups of the specified vmid.
[--backup-time <EPOCH>]
Backup timestamp
--fingerprint <FINGERPRINT>
@@ -41,6 +50,8 @@ Options:
File containing a comment/notes
[--log-file <LOG_FILE>]
Log file
+ -y, --yes
+ Automatic yes to prompts
-h, --help
Print help
-V, --version
@@ -52,7 +63,16 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
args.remove(0); // remove the executable path.
let mut first_later_args_index = 0;
- let options = ["-h", "--help", "-c", "--compress", "-e", "--encrypt"];
+ let options = [
+ "-h",
+ "--help",
+ "-c",
+ "--compress",
+ "-e",
+ "--encrypt",
+ "-y",
+ "--yes",
+ ];
for (i, arg) in args.iter().enumerate() {
if let Some(arg) = arg.to_str() {
@@ -87,7 +107,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let pbs_repository = args.value_from_str("--repository")?;
let namespace = args.opt_value_from_str("--ns")?;
- let vmid = args.value_from_str("--vmid")?;
+ let vmid = args.opt_value_from_str("--vmid")?;
let backup_time: Option<i64> = args.opt_value_from_str("--backup-time")?;
let backup_time = backup_time.unwrap_or_else(epoch_i64);
let fingerprint = args.opt_value_from_str("--fingerprint")?;
@@ -99,6 +119,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
+ let yes = args.contains(["-y", "--yes"]);
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
@@ -196,15 +217,131 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
encrypt,
};
- let vma_args = VmaBackupArgs {
- vma_file_path: vma_file_path.cloned(),
- backup_id: vmid,
- backup_time,
- notes,
- log_file_path,
+ let bulk =
+ vma_file_path
+ .map(PathBuf::from)
+ .and_then(|path| if path.is_dir() { Some(path) } else { None });
+
+ let grouped_vmas = if let Some(dump_dir_path) = bulk {
+ let re = Regex::new(
+ r"vzdump-qemu-(\d+)-(\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}).vma(|.zst|.lzo|.gz)$",
+ )?;
+
+ let mut vmas = Vec::new();
+
+ for entry in read_dir(dump_dir_path)? {
+ let entry = entry?;
+ let path = entry.path();
+
+ if !path.is_file() {
+ continue;
+ }
+
+ if let Some(file_name) = path.file_name().and_then(|n| n.to_str()) {
+ let Some((_, [backup_id, timestr, ext])) =
+ re.captures(file_name).map(|c| c.extract())
+ else {
+ // Skip the file, since it is not a VMA backup
+ continue;
+ };
+
+ if let Some(ref vmid) = vmid {
+ if backup_id != vmid {
+ // Skip the backup, since it does not match the specified vmid
+ continue;
+ }
+ }
+
+ let compression = match ext {
+ "" => None,
+ ".zst" => Some(Compression::Zstd),
+ ".lzo" => Some(Compression::Lzo),
+ ".gz" => Some(Compression::GZip),
+ _ => bail!("Unexpected file extension: {ext}"),
+ };
+
+ let backup_time = NaiveDateTime::parse_from_str(timestr, "%Y_%m_%d-%H_%M_%S")?
+ .and_utc()
+ .timestamp();
+
+ let notes_path = path.with_file_name(format!("{}.notes", file_name));
+ let notes = proxmox_sys::fs::file_read_optional_string(notes_path)?;
+
+ let log_path = path.with_file_name(format!("{}.log", file_name));
+ let log_file_path = if log_path.exists() {
+ Some(log_path.to_path_buf().into_os_string())
+ } else {
+ None
+ };
+
+ let backup_args = VmaBackupArgs {
+ vma_file_path: Some(path.clone().into()),
+ compression,
+ backup_id: backup_id.to_string(),
+ backup_time,
+ notes,
+ log_file_path,
+ };
+ vmas.push(backup_args);
+ }
+ }
+
+ vmas.sort_by_key(|d| d.backup_time);
+ let total_vma_count = vmas.len();
+ let mut grouped_vmas: Vec<_> = vmas
+ .into_iter()
+ .into_group_map_by(|d| d.backup_id.clone())
+ .into_values()
+ .collect();
+ grouped_vmas.sort_by_key(|d| d[0].backup_id.clone());
+
+ println!(
+ "Found {} backup archive(s) of {} different VMID(s):",
+ total_vma_count,
+ grouped_vmas.len()
+ );
+
+ for vma_group in &grouped_vmas {
+ println!(
+ "- VMID {}: {} backups",
+ vma_group[0].backup_id,
+ vma_group.len()
+ );
+ }
+
+ if !yes {
+ loop {
+ eprint!("Proceed with the bulk import? (y/n): ");
+ let mut line = String::new();
+
+ BufReader::new(std::io::stdin()).read_line(&mut line)?;
+ let trimmed = line.trim();
+ if trimmed == "y" || trimmed == "Y" {
+ break;
+ } else if trimmed == "n" || trimmed == "N" {
+ bail!("Bulk import was not confirmed.");
+ }
+ }
+ }
+
+ grouped_vmas
+ } else if let Some(vmid) = vmid {
+ vec![vec![VmaBackupArgs {
+ vma_file_path: vma_file_path.cloned(),
+ compression: None,
+ backup_id: vmid,
+ backup_time,
+ notes,
+ log_file_path,
+ }]]
+ } else {
+ bail!("No vmid specified for single backup file");
};
- let options = BackupVmaToPbsArgs { pbs_args, vma_args };
+ let options = BackupVmaToPbsArgs {
+ pbs_args,
+ grouped_vmas,
+ };
Ok(options)
}
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 37ea308..a3f1eee 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -4,6 +4,7 @@ use std::collections::HashMap;
use std::ffi::{c_char, CStr, CString, OsString};
use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
+use std::process::{Command, Stdio};
use std::ptr;
use std::time::SystemTime;
@@ -30,7 +31,7 @@ const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
pub pbs_args: PbsArgs,
- pub vma_args: VmaBackupArgs,
+ pub grouped_vmas: Vec<Vec<VmaBackupArgs>>,
}
pub struct PbsArgs {
@@ -45,8 +46,15 @@ pub struct PbsArgs {
pub encrypt: bool,
}
+pub enum Compression {
+ Zstd,
+ Lzo,
+ GZip,
+}
+
pub struct VmaBackupArgs {
pub vma_file_path: Option<OsString>,
+ pub compression: Option<Compression>,
pub backup_id: String,
pub backup_time: i64,
pub notes: Option<String>,
@@ -468,7 +476,19 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let start_transfer_time = SystemTime::now();
- upload_vma_file(pbs_args, &args.vma_args)?;
+ for vma_group in args.grouped_vmas {
+ for backup_args in vma_group {
+ if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
+ eprintln!(
+ "Failed to upload vma file at {:?} - {}",
+ backup_args.vma_file_path.unwrap_or("(stdin)".into()),
+ e
+ );
+ println!("Skipping VMID {}", backup_args.backup_id);
+ break;
+ }
+ }
+ }
let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
let total_seconds = transfer_duration.as_secs();
@@ -481,13 +501,43 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
- let vma_file: Box<dyn BufRead> = match &backup_args.vma_file_path {
- Some(vma_file_path) => match File::open(vma_file_path) {
- Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
- Ok(file) => Box::new(BufReader::new(file)),
+ match &backup_args.vma_file_path {
+ Some(vma_file_path) => println!("Uploading VMA backup from {:?}", vma_file_path),
+ None => println!("Uploading VMA backup from (stdin)"),
+ };
+
+ let vma_file: Box<dyn BufRead> = match &backup_args.compression {
+ Some(compression) => {
+ let vma_file_path = backup_args
+ .vma_file_path
+ .as_ref()
+ .expect("No VMA file path provided");
+ let mut cmd = match compression {
+ Compression::Zstd => {
+ let mut cmd = Command::new("zstd");
+ cmd.args(["-q", "-d", "-c"]);
+ cmd
+ }
+ Compression::Lzo => {
+ let mut cmd = Command::new("lzop");
+ cmd.args(["-d", "-c"]);
+ cmd
+ }
+ Compression::GZip => Command::new("zcat"),
+ };
+ let process = cmd.arg(vma_file_path).stdout(Stdio::piped()).spawn()?;
+ let stdout = process.stdout.expect("Failed to capture stdout");
+ Box::new(BufReader::new(stdout))
+ }
+ None => match &backup_args.vma_file_path {
+ Some(vma_file_path) => match File::open(vma_file_path) {
+ Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
+ Ok(file) => Box::new(BufReader::new(file)),
+ },
+ None => Box::new(BufReader::new(stdin())),
},
- None => Box::new(BufReader::new(stdin())),
};
+
let vma_reader = VmaReader::new(vma_file)?;
let pbs = create_pbs_backup_task(pbs_args, backup_args)?;
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 3/6] add option to skip vmids whose backups failed to upload
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 1/6] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 4/6] remove hard coded values Filip Schauer
` (3 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/main.rs | 6 ++++++
src/vma2pbs.rs | 13 ++++++++++---
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/main.rs b/src/main.rs
index 4c5135b..4e3a91b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -50,6 +50,9 @@ Options:
File containing a comment/notes
[--log-file <LOG_FILE>]
Log file
+ --skip-failed
+ Skip VMIDs that failed to be uploaded and continue onto the next VMID if a dump directory
+ is specified.
-y, --yes
Automatic yes to prompts
-h, --help
@@ -70,6 +73,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
"--compress",
"-e",
"--encrypt",
+ "--skip-failed",
"-y",
"--yes",
];
@@ -119,6 +123,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
+ let skip_failed = args.contains("--skip-failed");
let yes = args.contains(["-y", "--yes"]);
match (encrypt, keyfile.is_some()) {
@@ -341,6 +346,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let options = BackupVmaToPbsArgs {
pbs_args,
grouped_vmas,
+ skip_failed,
};
Ok(options)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index a3f1eee..ef9d770 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -32,6 +32,7 @@ const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
pub pbs_args: PbsArgs,
pub grouped_vmas: Vec<Vec<VmaBackupArgs>>,
+ pub skip_failed: bool,
}
pub struct PbsArgs {
@@ -479,13 +480,19 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
for vma_group in args.grouped_vmas {
for backup_args in vma_group {
if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
- eprintln!(
+ let err_msg = format!(
"Failed to upload vma file at {:?} - {}",
backup_args.vma_file_path.unwrap_or("(stdin)".into()),
e
);
- println!("Skipping VMID {}", backup_args.backup_id);
- break;
+
+ if args.skip_failed {
+ eprintln!("{}", err_msg);
+ println!("Skipping VMID {}", backup_args.backup_id);
+ break;
+ } else {
+ bail!(err_msg);
+ }
}
}
}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 4/6] remove hard coded values
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
` (2 preceding siblings ...)
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 3/6] add option to skip vmids whose backups failed to upload Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 5/6] use level-based logging instead of println Filip Schauer
` (2 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/vma.rs | 2 +-
src/vma2pbs.rs | 23 +++++++++++------------
2 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/src/vma.rs b/src/vma.rs
index 518de8a..63ee3b5 100644
--- a/src/vma.rs
+++ b/src/vma.rs
@@ -22,7 +22,7 @@ const VMA_MAX_CONFIGS: usize = 256;
/// Maximum number of block devices
/// See VMA Header in pve-qemu.git/vma_spec.txt
-const VMA_MAX_DEVICES: usize = 256;
+pub const VMA_MAX_DEVICES: usize = 256;
/// VMA magic string
/// See VMA Header in pve-qemu.git/vma_spec.txt
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index ef9d770..497f3ae 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -25,7 +25,7 @@ use proxmox_time::epoch_to_rfc3339;
use scopeguard::defer;
use serde_json::Value;
-use crate::vma::VmaReader;
+use crate::vma::{VmaReader, VMA_MAX_DEVICES};
const VMA_CLUSTER_SIZE: usize = 65536;
@@ -174,20 +174,21 @@ where
fn register_block_devices<T>(
vma_reader: &VmaReader<T>,
pbs: *mut ProxmoxBackupHandle,
-) -> Result<[Option<BlockDeviceInfo>; 256], Error>
+) -> Result<[Option<BlockDeviceInfo>; VMA_MAX_DEVICES], Error>
where
T: Read,
{
- let mut block_device_infos: [Option<BlockDeviceInfo>; 256] = [None; 256];
+ let mut block_device_infos: [Option<BlockDeviceInfo>; VMA_MAX_DEVICES] =
+ [None; VMA_MAX_DEVICES];
let mut pbs_err: *mut c_char = ptr::null_mut();
- for device_id in 0..255 {
- if !vma_reader.contains_device(device_id) {
+ for (device_id, block_device_info) in block_device_infos.iter_mut().enumerate() {
+ if !vma_reader.contains_device(device_id.try_into()?) {
continue;
}
- let device_name = vma_reader.get_device_name(device_id)?;
- let device_size = vma_reader.get_device_size(device_id)?;
+ let device_name = vma_reader.get_device_name(device_id.try_into()?)?;
+ let device_size = vma_reader.get_device_size(device_id.try_into()?)?;
println!(
"DEV: dev_id={} size: {} devname: {}",
@@ -207,12 +208,10 @@ where
handle_pbs_error(pbs_err, "proxmox_backup_register_image")?;
}
- let block_device_info = BlockDeviceInfo {
+ *block_device_info = Some(BlockDeviceInfo {
pbs_device_id: pbs_device_id as u8,
device_size,
- };
-
- block_device_infos[device_id as usize] = Some(block_device_info);
+ });
}
Ok(block_device_infos)
@@ -360,7 +359,7 @@ where
let mut pbs_err: *mut c_char = ptr::null_mut();
- for block_device_info in block_device_infos.iter().take(255) {
+ for block_device_info in block_device_infos {
let block_device_info = match block_device_info {
Some(block_device_info) => block_device_info,
None => continue,
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 5/6] use level-based logging instead of println
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
` (3 preceding siblings ...)
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 4/6] remove hard coded values Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 6/6] log device upload progress as a percentage Filip Schauer
2024-11-04 13:09 ` [pbs-devel] partially-applied: [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Fabian Grünbichler
6 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
Use log level "info" by default and prevent spamming messages for every
single chunk uploaded. To re-enable these messages, set the RUST_LOG
environment variable to "debug".
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 2 ++
src/main.rs | 28 ++++++++++++++++++++++------
src/vma2pbs.rs | 38 ++++++++++++++++++++------------------
3 files changed, 44 insertions(+), 24 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 5c6a175..0f4b2a6 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,8 +8,10 @@ edition = "2021"
anyhow = "1.0"
bincode = "1.3"
chrono = "0.4"
+env_logger = "0.10"
hyper = "0.14.5"
itertools = "0.13"
+log = "0.4"
pico-args = "0.5"
md5 = "0.7.0"
regex = "1.7"
diff --git a/src/main.rs b/src/main.rs
index 4e3a91b..22804ee 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -5,6 +5,7 @@ use std::path::PathBuf;
use anyhow::{bail, Context, Error};
use chrono::NaiveDateTime;
+use env_logger::Target;
use itertools::Itertools;
use proxmox_sys::linux::tty;
use proxmox_time::epoch_i64;
@@ -128,7 +129,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
- (false, true) => println!(
+ (false, true) => log::info!(
"--keyfile given, but --encrypt not set -> backup will be signed, but not encrypted!"
),
_ => {}
@@ -190,7 +191,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Some(key_password)
} else if vma_file_path.is_none() {
- println!(
+ log::info!(
"Please use --key-password-file to provide the password when passing the VMA file \
to stdin, if required."
);
@@ -246,13 +247,17 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let Some((_, [backup_id, timestr, ext])) =
re.captures(file_name).map(|c| c.extract())
else {
- // Skip the file, since it is not a VMA backup
+ log::debug!("Skip \"{file_name}\", since it is not a VMA backup");
continue;
};
if let Some(ref vmid) = vmid {
if backup_id != vmid {
- // Skip the backup, since it does not match the specified vmid
+ log::debug!(
+ "Skip backup with VMID {}, since it does not match specified VMID {}",
+ backup_id,
+ vmid
+ );
continue;
}
}
@@ -300,14 +305,14 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
.collect();
grouped_vmas.sort_by_key(|d| d[0].backup_id.clone());
- println!(
+ log::info!(
"Found {} backup archive(s) of {} different VMID(s):",
total_vma_count,
grouped_vmas.len()
);
for vma_group in &grouped_vmas {
- println!(
+ log::info!(
"- VMID {}: {} backups",
vma_group[0].backup_id,
vma_group.len()
@@ -352,7 +357,18 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Ok(options)
}
+fn init_cli_logger() {
+ env_logger::Builder::from_env(env_logger::Env::new().filter_or("RUST_LOG", "info"))
+ .format_level(false)
+ .format_target(false)
+ .format_timestamp(None)
+ .target(Target::Stdout)
+ .init();
+}
+
fn main() -> Result<(), Error> {
+ init_cli_logger();
+
let args = parse_args()?;
vma2pbs(args)?;
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index 497f3ae..a5907ed 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -82,8 +82,8 @@ fn create_pbs_backup_task(
pbs_args: &PbsArgs,
backup_args: &VmaBackupArgs,
) -> Result<*mut ProxmoxBackupHandle, Error> {
- println!(
- "backup time: {}",
+ log::info!(
+ "\tbackup time: {}",
epoch_to_rfc3339(backup_args.backup_time)?
);
@@ -152,7 +152,7 @@ where
let config_name = config.name;
let config_data = config.content;
- println!("CFG: size: {} name: {}", config_data.len(), config_name);
+ log::info!("\tCFG: size: {} name: {}", config_data.len(), config_name);
let config_name_cstr = CString::new(config_name)?;
@@ -190,9 +190,11 @@ where
let device_name = vma_reader.get_device_name(device_id.try_into()?)?;
let device_size = vma_reader.get_device_size(device_id.try_into()?)?;
- println!(
- "DEV: dev_id={} size: {} devname: {}",
- device_id, device_size, device_name
+ log::info!(
+ "\tDEV: dev_id={} size: {} devname: {}",
+ device_id,
+ device_size,
+ device_name
);
let device_name_cstr = CString::new(device_name)?;
@@ -276,8 +278,8 @@ where
};
let pbs_upload_chunk = |pbs_chunk_buffer: Option<&[u8]>| {
- println!(
- "Uploading dev_id: {} offset: {:#0X} - {:#0X}",
+ log::debug!(
+ "\tUploading dev_id: {} offset: {:#0X} - {:#0X}",
dev_id,
pbs_chunk_offset,
pbs_chunk_offset + pbs_chunk_size,
@@ -466,13 +468,13 @@ fn set_notes(
pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let pbs_args = &args.pbs_args;
- println!("PBS repository: {}", pbs_args.pbs_repository);
+ log::info!("PBS repository: {}", pbs_args.pbs_repository);
if let Some(ns) = &pbs_args.namespace {
- println!("PBS namespace: {}", ns);
+ log::info!("PBS namespace: {}", ns);
}
- println!("PBS fingerprint: {}", pbs_args.fingerprint);
- println!("compress: {}", pbs_args.compress);
- println!("encrypt: {}", pbs_args.encrypt);
+ log::info!("PBS fingerprint: {}", pbs_args.fingerprint);
+ log::info!("compress: {}", pbs_args.compress);
+ log::info!("encrypt: {}", pbs_args.encrypt);
let start_transfer_time = SystemTime::now();
@@ -486,8 +488,8 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
);
if args.skip_failed {
- eprintln!("{}", err_msg);
- println!("Skipping VMID {}", backup_args.backup_id);
+ log::warn!("{}", err_msg);
+ log::info!("Skipping VMID {}", backup_args.backup_id);
break;
} else {
bail!(err_msg);
@@ -501,15 +503,15 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let minutes = total_seconds / 60;
let seconds = total_seconds % 60;
let milliseconds = transfer_duration.as_millis() % 1000;
- println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
+ log::info!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
Ok(())
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
match &backup_args.vma_file_path {
- Some(vma_file_path) => println!("Uploading VMA backup from {:?}", vma_file_path),
- None => println!("Uploading VMA backup from (stdin)"),
+ Some(vma_file_path) => log::info!("Uploading VMA backup from {:?}", vma_file_path),
+ None => log::info!("Uploading VMA backup from (stdin)"),
};
let vma_file: Box<dyn BufRead> = match &backup_args.compression {
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] [PATCH vma-to-pbs v4 6/6] log device upload progress as a percentage
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
` (4 preceding siblings ...)
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 5/6] use level-based logging instead of println Filip Schauer
@ 2024-10-30 13:55 ` Filip Schauer
2024-11-04 13:09 ` [pbs-devel] partially-applied: [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Fabian Grünbichler
6 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-10-30 13:55 UTC (permalink / raw)
To: pbs-devel
Log the upload progress of a device as a percentage with log level info
every 1000 chunks.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/vma2pbs.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index a5907ed..ea5ed2a 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -6,6 +6,8 @@ use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
use std::process::{Command, Stdio};
use std::ptr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
use std::time::SystemTime;
use anyhow::{anyhow, bail, Error};
@@ -234,6 +236,8 @@ where
non_zero_mask: u64,
}
+ let chunk_stats = Arc::new([const { AtomicU64::new(0) }; VMA_MAX_DEVICES]);
+
let images_chunks: RefCell<HashMap<u8, HashMap<u64, ImageChunk>>> =
RefCell::new(HashMap::new());
@@ -284,6 +288,11 @@ where
pbs_chunk_offset,
pbs_chunk_offset + pbs_chunk_size,
);
+ let chunk_stat = chunk_stats[dev_id as usize].fetch_add(1, Ordering::SeqCst);
+ if (chunk_stat % 1000) == 0 {
+ let percentage = 100 * PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE * chunk_stat / device_size;
+ log::info!("\tUploading dev_id: {} ({}%)", dev_id, percentage);
+ }
let mut pbs_err: *mut c_char = ptr::null_mut();
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory Filip Schauer
@ 2024-11-04 13:06 ` Fabian Grünbichler
2024-11-11 13:13 ` Filip Schauer
0 siblings, 1 reply; 10+ messages in thread
From: Fabian Grünbichler @ 2024-11-04 13:06 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On October 30, 2024 2:55 pm, Filip Schauer wrote:
> When a path to a directory is provided in the vma_file argument, try to
> upload all VMA backups in the directory. This also handles compressed
> VMA files, notes and logs. If a vmid is specified with --vmid, only the
> backups of that particular vmid are uploaded.
>
> This is intended for use on a dump directory:
>
> PBS_FINGERPRINT='PBS_FINGERPRINT' vma-to-pbs \
> --repository 'user@realm!token@server:port:datastore' \
> /var/lib/vz/dump
>
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> ---
> Cargo.toml | 3 +
> src/main.rs | 161 +++++++++++++++++++++++++++++++++++++++++++++----
> src/vma2pbs.rs | 64 +++++++++++++++++---
> 3 files changed, 209 insertions(+), 19 deletions(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index cd13426..5c6a175 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -7,9 +7,12 @@ edition = "2021"
> [dependencies]
> anyhow = "1.0"
> bincode = "1.3"
> +chrono = "0.4"
> hyper = "0.14.5"
> +itertools = "0.13"
not needed, see below
> pico-args = "0.5"
> md5 = "0.7.0"
> +regex = "1.7"
> scopeguard = "1.1.0"
> serde = "1.0"
> serde_json = "1.0"
> diff --git a/src/main.rs b/src/main.rs
> index 3e25591..4c5135b 100644
> --- a/src/main.rs
> +++ b/src/main.rs
> @@ -1,26 +1,35 @@
> use std::ffi::OsString;
> +use std::fs::read_dir;
> +use std::io::{BufRead, BufReader};
> +use std::path::PathBuf;
>
> use anyhow::{bail, Context, Error};
> +use chrono::NaiveDateTime;
> +use itertools::Itertools;
> use proxmox_sys::linux::tty;
> use proxmox_time::epoch_i64;
> +use regex::Regex;
>
> mod vma;
> mod vma2pbs;
> -use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
> +use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, Compression, PbsArgs, VmaBackupArgs};
>
> const CMD_HELP: &str = "\
> Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
>
> Arguments:
> - [vma_file]
> + [vma_file | dump_directory]
>
> Options:
> --repository <auth_id@host:port:datastore>
> Repository URL
> [--ns <NAMESPACE>]
> Namespace
> - --vmid <VMID>
> + [--vmid <VMID>]
> Backup ID
> + This is required if a single VMA file is provided.
> + If not specified, bulk import all VMA backups in the provided directory.
> + If specified with a dump directory, only import backups of the specified vmid.
> [--backup-time <EPOCH>]
> Backup timestamp
> --fingerprint <FINGERPRINT>
> @@ -41,6 +50,8 @@ Options:
> File containing a comment/notes
> [--log-file <LOG_FILE>]
> Log file
> + -y, --yes
> + Automatic yes to prompts
> -h, --help
> Print help
> -V, --version
> @@ -52,7 +63,16 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> args.remove(0); // remove the executable path.
>
> let mut first_later_args_index = 0;
> - let options = ["-h", "--help", "-c", "--compress", "-e", "--encrypt"];
> + let options = [
> + "-h",
> + "--help",
> + "-c",
> + "--compress",
> + "-e",
> + "--encrypt",
> + "-y",
> + "--yes",
> + ];
>
> for (i, arg) in args.iter().enumerate() {
> if let Some(arg) = arg.to_str() {
> @@ -87,7 +107,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
>
> let pbs_repository = args.value_from_str("--repository")?;
> let namespace = args.opt_value_from_str("--ns")?;
> - let vmid = args.value_from_str("--vmid")?;
> + let vmid = args.opt_value_from_str("--vmid")?;
> let backup_time: Option<i64> = args.opt_value_from_str("--backup-time")?;
> let backup_time = backup_time.unwrap_or_else(epoch_i64);
> let fingerprint = args.opt_value_from_str("--fingerprint")?;
> @@ -99,6 +119,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
> let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
> let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
> + let yes = args.contains(["-y", "--yes"]);
>
> match (encrypt, keyfile.is_some()) {
> (true, false) => bail!("--encrypt requires a --keyfile!"),
> @@ -196,15 +217,131 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> encrypt,
> };
>
> - let vma_args = VmaBackupArgs {
> - vma_file_path: vma_file_path.cloned(),
> - backup_id: vmid,
> - backup_time,
> - notes,
> - log_file_path,
> + let bulk =
> + vma_file_path
> + .map(PathBuf::from)
> + .and_then(|path| if path.is_dir() { Some(path) } else { None });
> +
> + let grouped_vmas = if let Some(dump_dir_path) = bulk {
grouped_vmas should still be a map, not a vec of vec..
e.g., something like this (requires some more adaptation - while this
could use itertools, I don't think it's worth to pull that in if the
same can be had with a single fold invocation):
@@ -298,12 +298,16 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
vmas.sort_by_key(|d| d.backup_time);
let total_vma_count = vmas.len();
- let mut grouped_vmas: Vec<_> = vmas
- .into_iter()
- .into_group_map_by(|d| d.backup_id.clone())
- .into_values()
- .collect();
- grouped_vmas.sort_by_key(|d| d[0].backup_id.clone());
+ let grouped_vmas = vmas.into_iter().fold(
+ HashMap::new(),
+ |mut grouped: HashMap<String, Vec<VmaBackupArgs>>, vma_args| {
+ grouped
+ .entry(vma_args.backup_id.clone())
+ .or_default()
+ .push(vma_args);
+ grouped
+ },
+ );
log::info!(
"Found {} backup archive(s) of {} different VMID(s):",
@@ -311,12 +315,8 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
grouped_vmas.len()
);
- for vma_group in &grouped_vmas {
- log::info!(
- "- VMID {}: {} backups",
- vma_group[0].backup_id,
- vma_group.len()
- );
+ for (vma_group, vma_args) in &grouped_vmas {
+ log::info!("- VMID {}: {} backups", vma_group, vma_args.len());
}
if !yes {
> + let re = Regex::new(
> + r"vzdump-qemu-(\d+)-(\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}).vma(|.zst|.lzo|.gz)$",
> + )?;
> +
> + let mut vmas = Vec::new();
> +
> + for entry in read_dir(dump_dir_path)? {
> + let entry = entry?;
> + let path = entry.path();
> +
> + if !path.is_file() {
> + continue;
> + }
> +
> + if let Some(file_name) = path.file_name().and_then(|n| n.to_str()) {
> + let Some((_, [backup_id, timestr, ext])) =
> + re.captures(file_name).map(|c| c.extract())
> + else {
> + // Skip the file, since it is not a VMA backup
> + continue;
> + };
> +
> + if let Some(ref vmid) = vmid {
> + if backup_id != vmid {
> + // Skip the backup, since it does not match the specified vmid
> + continue;
> + }
> + }
> +
> + let compression = match ext {
> + "" => None,
> + ".zst" => Some(Compression::Zstd),
> + ".lzo" => Some(Compression::Lzo),
> + ".gz" => Some(Compression::GZip),
> + _ => bail!("Unexpected file extension: {ext}"),
> + };
> +
> + let backup_time = NaiveDateTime::parse_from_str(timestr, "%Y_%m_%d-%H_%M_%S")?
> + .and_utc()
> + .timestamp();
> +
> + let notes_path = path.with_file_name(format!("{}.notes", file_name));
> + let notes = proxmox_sys::fs::file_read_optional_string(notes_path)?;
> +
> + let log_path = path.with_file_name(format!("{}.log", file_name));
> + let log_file_path = if log_path.exists() {
> + Some(log_path.to_path_buf().into_os_string())
> + } else {
> + None
> + };
> +
> + let backup_args = VmaBackupArgs {
> + vma_file_path: Some(path.clone().into()),
> + compression,
> + backup_id: backup_id.to_string(),
> + backup_time,
> + notes,
> + log_file_path,
> + };
> + vmas.push(backup_args);
> + }
> + }
> +
> + vmas.sort_by_key(|d| d.backup_time);
> + let total_vma_count = vmas.len();
> + let mut grouped_vmas: Vec<_> = vmas
> + .into_iter()
> + .into_group_map_by(|d| d.backup_id.clone())
> + .into_values()
> + .collect();
> + grouped_vmas.sort_by_key(|d| d[0].backup_id.clone());
> +
> + println!(
> + "Found {} backup archive(s) of {} different VMID(s):",
> + total_vma_count,
> + grouped_vmas.len()
> + );
if we don't find any, we should print something else here and exit?
> +
> + for vma_group in &grouped_vmas {
> + println!(
> + "- VMID {}: {} backups",
> + vma_group[0].backup_id,
> + vma_group.len()
> + );
> + }
> +
> + if !yes {
> + loop {
> + eprint!("Proceed with the bulk import? (y/n): ");
> + let mut line = String::new();
> +
> + BufReader::new(std::io::stdin()).read_line(&mut line)?;
> + let trimmed = line.trim();
> + if trimmed == "y" || trimmed == "Y" {
> + break;
> + } else if trimmed == "n" || trimmed == "N" {
> + bail!("Bulk import was not confirmed.");
> + }
this maybe should mimic what we do in proxmox_router when prompting for
confirmation? e.g., flush stdout, have a default value, ..?
should we abort after a few loops?
> + }
> + }
> +
> + grouped_vmas
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pbs-devel] partially-applied: [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
` (5 preceding siblings ...)
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 6/6] log device upload progress as a percentage Filip Schauer
@ 2024-11-04 13:09 ` Fabian Grünbichler
6 siblings, 0 replies; 10+ messages in thread
From: Fabian Grünbichler @ 2024-11-04 13:09 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
applied patches 1 and 4
consider patches 3, 5 and 6
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
they were not possible to apply now because patch context relies on
patch #2 being applied as well.
On October 30, 2024 2:55 pm, Filip Schauer wrote:
> When a path to a directory is provided in the vma_file argument, try to
> upload all VMA backups in the directory. This also handles compressed
> VMA files, notes and logs. If a vmid is specified with --vmid, only the
> backups of that particular vmid are uploaded.
>
> Also improve the readability of the log messages to keep track of all
> imported backups.
>
> Changed since v3:
> * Mention in the description of the --vmid argument, that it is required
> if a single VMA file is provided
> * Construct grouped_vmas in place
> * Add debug logs when gathering files for bulk import
> * Log a summary of the files gathered for bulk import
> * Remove the "confusing VMA file path" error message in the second
> commit
> * Switch chunk_stats from Arc<Mutex<[u64; 256]>> to
> Arc<[AtomicU64; 256]> and use fetch_add to atomically increment and
> fetch the chunk stat
> * Ask for confirmation before bulk import
> * Add --yes option to skip the confirmation prompt
>
> Changed since v2:
> * Make skipping a VMID on error optional with the --skip-failed option
> * Switch log output from stderr to stdout
> * Bump itertools to 0.13
>
> Changed since v1:
> * Do not recurse through dump directory
> * Compile regex once before iterating over the files in the dump
> directory
> * Use extract on regex capture groups
> * Do not use deprecated method `chrono::NaiveDateTime::timestamp`
> * Use proxmox_sys::fs::file_read_optional_string
> * Group VMA files by VMID and continue with next VMID on error
> * Move the BackupVmaToPbsArgs split into its own commit
> * Remove hard coded occurences of 255
> * Use level-based logging instead of println
>
> Filip Schauer (6):
> split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs
> add support for bulk import of a dump directory
> add option to skip vmids whose backups failed to upload
> remove hard coded values
> use level-based logging instead of println
> log device upload progress as a percentage
>
> Cargo.toml | 5 +
> src/main.rs | 192 +++++++++++++++++++++++++++---
> src/vma.rs | 2 +-
> src/vma2pbs.rs | 310 ++++++++++++++++++++++++++++++++-----------------
> 4 files changed, 388 insertions(+), 121 deletions(-)
>
> --
> 2.39.5
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory
2024-11-04 13:06 ` Fabian Grünbichler
@ 2024-11-11 13:13 ` Filip Schauer
0 siblings, 0 replies; 10+ messages in thread
From: Filip Schauer @ 2024-11-11 13:13 UTC (permalink / raw)
To: pbs-devel
Superseded by:
https://lists.proxmox.com/pipermail/pbs-devel/2024-November/011353.html
On 04/11/2024 14:06, Fabian Grünbichler wrote:
> grouped_vmas should still be a map, not a vec of vec.. e.g., something
> like this (requires some more adaptation - while this could use
> itertools, I don't think it's worth to pull that in if the same can be
> had with a single fold invocation): @@ -298,12 +298,16 @@ fn
> parse_args() -> Result<BackupVmaToPbsArgs, Error> {
> vmas.sort_by_key(|d|d.backup_time); let total_vma_count = vmas.len();
> - let mut grouped_vmas: Vec<_> = vmas - .into_iter() -
> .into_group_map_by(|d|d.backup_id.clone()) - .into_values() -
> .collect(); - grouped_vmas.sort_by_key(|d|d[0].backup_id.clone()); +
> let grouped_vmas = vmas.into_iter().fold( + HashMap::new(), + |mut
> grouped: HashMap<String, Vec<VmaBackupArgs>>, vma_args|{ + grouped +
> .entry(vma_args.backup_id.clone()) + .or_default() + .push(vma_args);
> + grouped + }, + ); log::info!( "Found {} backup archive(s) of {}
> different VMID(s):", @@ -311,12 +315,8 @@ fn parse_args() ->
> Result<BackupVmaToPbsArgs, Error> { grouped_vmas.len() ); - for
> vma_group in &grouped_vmas { - log::info!( - "- VMID {}: {} backups",
> - vma_group[0].backup_id, - vma_group.len() - ); + for (vma_group,
> vma_args) in &grouped_vmas { + log::info!("- VMID {}: {} backups",
> vma_group, vma_args.len()); } if !yes {
done
On 04/11/2024 14:06, Fabian Grünbichler wrote:
>> + println!(
>> + "Found {} backup archive(s) of {} different VMID(s):",
>> + total_vma_count,
>> + grouped_vmas.len()
>> + );
> if we don't find any, we should print something else here and exit?
done with `bail!` in v5
On 04/11/2024 14:06, Fabian Grünbichler wrote:
>> + if !yes {
>> + loop {
>> + eprint!("Proceed with the bulk import? (y/n): ");
>> + let mut line = String::new();
>> +
>> + BufReader::new(std::io::stdin()).read_line(&mut line)?;
>> + let trimmed = line.trim();
>> + if trimmed == "y" || trimmed == "Y" {
>> + break;
>> + } else if trimmed == "n" || trimmed == "N" {
>> + bail!("Bulk import was not confirmed.");
>> + }
> this maybe should mimic what we do in proxmox_router when prompting
> for confirmation? e.g., flush stdout, have a default value, ..? should
> we abort after a few loops?
Changed in v5 to mimic the behaviour of the confirmation prompt in
proxmox_router. (bail on invalid input)
Also made Y the default choice.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-11-11 13:14 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-10-30 13:55 [pbs-devel] [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 1/6] split BackupVmaToPbsArgs into PbsArgs and VmaBackupArgs Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 2/6] add support for bulk import of a dump directory Filip Schauer
2024-11-04 13:06 ` Fabian Grünbichler
2024-11-11 13:13 ` Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 3/6] add option to skip vmids whose backups failed to upload Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 4/6] remove hard coded values Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 5/6] use level-based logging instead of println Filip Schauer
2024-10-30 13:55 ` [pbs-devel] [PATCH vma-to-pbs v4 6/6] log device upload progress as a percentage Filip Schauer
2024-11-04 13:09 ` [pbs-devel] partially-applied: [PATCH vma-to-pbs v4 0/6] add support for bulk import of a dump directory Fabian Grünbichler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox