* [pbs-devel] [PATCH vma-to-pbs v6 1/4] add support for bulk import of a dump directory
2024-11-13 15:57 [pbs-devel] [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Filip Schauer
@ 2024-11-13 15:57 ` Filip Schauer
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 2/4] add option to skip vmids whose backups failed to upload Filip Schauer
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-11-13 15:57 UTC (permalink / raw)
To: pbs-devel
When a path to a directory is provided in the vma_file argument, try to
upload all VMA backups in the directory. This also handles compressed
VMA files, notes and logs. If a vmid is specified with --vmid, only the
backups of that particular vmid are uploaded.
This is intended for use on a dump directory:
PBS_FINGERPRINT='PBS_FINGERPRINT' vma-to-pbs \
--repository 'user@realm!token@server:port:datastore' \
/var/lib/vz/dump
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 2 +
src/main.rs | 172 +++++++++++++++++++++++++++++++++++++++++++++----
src/vma2pbs.rs | 63 ++++++++++++++++--
3 files changed, 217 insertions(+), 20 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index cd13426..ad80304 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -7,9 +7,11 @@ edition = "2021"
[dependencies]
anyhow = "1.0"
bincode = "1.3"
+chrono = "0.4"
hyper = "0.14.5"
pico-args = "0.5"
md5 = "0.7.0"
+regex = "1.7"
scopeguard = "1.1.0"
serde = "1.0"
serde_json = "1.0"
diff --git a/src/main.rs b/src/main.rs
index 3e25591..de1e6e0 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,26 +1,39 @@
+use std::collections::HashMap;
use std::ffi::OsString;
+use std::fs::read_dir;
+use std::io::{BufRead, BufReader, Write};
+use std::path::PathBuf;
use anyhow::{bail, Context, Error};
+use chrono::NaiveDateTime;
use proxmox_sys::linux::tty;
use proxmox_time::epoch_i64;
+use regex::Regex;
mod vma;
mod vma2pbs;
-use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, PbsArgs, VmaBackupArgs};
+use vma2pbs::{vma2pbs, BackupVmaToPbsArgs, Compression, PbsArgs, VmaBackupArgs};
const CMD_HELP: &str = "\
-Usage: vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
+Single VMA file usage:
+vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> --vmid <VMID> [vma_file]
+
+Bulk import usage:
+vma-to-pbs [OPTIONS] --repository <auth_id@host:port:datastore> [--vmid <VMID>] [dump_directory]
Arguments:
- [vma_file]
+ [vma_file | dump_directory]
Options:
--repository <auth_id@host:port:datastore>
Repository URL
[--ns <NAMESPACE>]
Namespace
- --vmid <VMID>
+ [--vmid <VMID>]
Backup ID
+ This is required if a single VMA file is provided.
+ If not specified, bulk import all VMA backups in the provided directory.
+ If specified with a dump directory, only import backups of the specified vmid.
[--backup-time <EPOCH>]
Backup timestamp
--fingerprint <FINGERPRINT>
@@ -41,6 +54,8 @@ Options:
File containing a comment/notes
[--log-file <LOG_FILE>]
Log file
+ -y, --yes
+ Automatic yes to prompts
-h, --help
Print help
-V, --version
@@ -52,7 +67,16 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
args.remove(0); // remove the executable path.
let mut first_later_args_index = 0;
- let options = ["-h", "--help", "-c", "--compress", "-e", "--encrypt"];
+ let options = [
+ "-h",
+ "--help",
+ "-c",
+ "--compress",
+ "-e",
+ "--encrypt",
+ "-y",
+ "--yes",
+ ];
for (i, arg) in args.iter().enumerate() {
if let Some(arg) = arg.to_str() {
@@ -87,7 +111,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let pbs_repository = args.value_from_str("--repository")?;
let namespace = args.opt_value_from_str("--ns")?;
- let vmid = args.value_from_str("--vmid")?;
+ let vmid: Option<String> = args.opt_value_from_str("--vmid")?;
let backup_time: Option<i64> = args.opt_value_from_str("--backup-time")?;
let backup_time = backup_time.unwrap_or_else(epoch_i64);
let fingerprint = args.opt_value_from_str("--fingerprint")?;
@@ -99,6 +123,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
+ let yes = args.contains(["-y", "--yes"]);
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
@@ -196,15 +221,136 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
encrypt,
};
- let vma_args = VmaBackupArgs {
- vma_file_path: vma_file_path.cloned(),
- backup_id: vmid,
- backup_time,
- notes,
- log_file_path,
+ let bulk =
+ vma_file_path
+ .map(PathBuf::from)
+ .and_then(|path| if path.is_dir() { Some(path) } else { None });
+
+ let grouped_vmas = if let Some(dump_dir_path) = bulk {
+ let re = Regex::new(
+ r"vzdump-qemu-(\d+)-(\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}).vma(|.zst|.lzo|.gz)$",
+ )?;
+
+ let mut vmas = Vec::new();
+
+ for entry in read_dir(dump_dir_path)? {
+ let entry = entry?;
+ let path = entry.path();
+
+ if !path.is_file() {
+ continue;
+ }
+
+ if let Some(file_name) = path.file_name().and_then(|n| n.to_str()) {
+ let Some((_, [backup_id, timestr, ext])) =
+ re.captures(file_name).map(|c| c.extract())
+ else {
+ // Skip the file, since it is not a VMA backup
+ continue;
+ };
+
+ if let Some(ref vmid) = vmid {
+ if backup_id != vmid {
+ // Skip the backup, since it does not match the specified vmid
+ continue;
+ }
+ }
+
+ let compression = match ext {
+ "" => None,
+ ".zst" => Some(Compression::Zstd),
+ ".lzo" => Some(Compression::Lzo),
+ ".gz" => Some(Compression::GZip),
+ _ => bail!("Unexpected file extension: {ext}"),
+ };
+
+ let backup_time = NaiveDateTime::parse_from_str(timestr, "%Y_%m_%d-%H_%M_%S")?
+ .and_utc()
+ .timestamp();
+
+ let notes_path = path.with_file_name(format!("{file_name}.notes"));
+ let notes = proxmox_sys::fs::file_read_optional_string(notes_path)?;
+
+ let log_path = path.with_file_name(format!("{file_name}.log"));
+ let log_file_path = if log_path.exists() {
+ Some(log_path.to_path_buf().into_os_string())
+ } else {
+ None
+ };
+
+ let backup_args = VmaBackupArgs {
+ vma_file_path: Some(path.clone().into()),
+ compression,
+ backup_id: backup_id.to_string(),
+ backup_time,
+ notes,
+ log_file_path,
+ };
+ vmas.push(backup_args);
+ }
+ }
+
+ vmas.sort_by_key(|d| d.backup_time);
+ let total_vma_count = vmas.len();
+ let grouped_vmas = vmas.into_iter().fold(
+ HashMap::new(),
+ |mut grouped: HashMap<String, Vec<VmaBackupArgs>>, vma_args| {
+ grouped
+ .entry(vma_args.backup_id.clone())
+ .or_default()
+ .push(vma_args);
+ grouped
+ },
+ );
+
+ if grouped_vmas.is_empty() {
+ bail!("Did not find any backup archives");
+ }
+
+ println!(
+ "Found {total_vma_count} backup archive(s) of {} different VMID(s):",
+ grouped_vmas.len()
+ );
+
+ for (backup_id, vma_group) in &grouped_vmas {
+ println!("- VMID {backup_id}: {} backups", vma_group.len());
+ }
+
+ if !yes {
+ eprint!("Proceed with the bulk import? (Y/n): ");
+ std::io::stdout().flush()?;
+ let mut line = String::new();
+
+ BufReader::new(std::io::stdin()).read_line(&mut line)?;
+ let trimmed = line.trim();
+ match trimmed {
+ "y" | "Y" | "" => {}
+ "n" | "N" => bail!("Bulk import was not confirmed."),
+ _ => bail!("Unexpected choice '{trimmed}'!"),
+ }
+ }
+
+ grouped_vmas
+ } else if let Some(vmid) = vmid {
+ HashMap::from([(
+ vmid.clone(),
+ vec![VmaBackupArgs {
+ vma_file_path: vma_file_path.cloned(),
+ compression: None,
+ backup_id: vmid,
+ backup_time,
+ notes,
+ log_file_path,
+ }],
+ )])
+ } else {
+ bail!("No vmid specified for single backup file");
};
- let options = BackupVmaToPbsArgs { pbs_args, vma_args };
+ let options = BackupVmaToPbsArgs {
+ pbs_args,
+ grouped_vmas,
+ };
Ok(options)
}
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index a888a7b..b26c62e 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -4,6 +4,7 @@ use std::collections::HashMap;
use std::ffi::{c_char, CStr, CString, OsString};
use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
+use std::process::{Command, Stdio};
use std::ptr;
use std::time::SystemTime;
@@ -30,7 +31,7 @@ const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
pub pbs_args: PbsArgs,
- pub vma_args: VmaBackupArgs,
+ pub grouped_vmas: HashMap<String, Vec<VmaBackupArgs>>,
}
pub struct PbsArgs {
@@ -45,8 +46,15 @@ pub struct PbsArgs {
pub encrypt: bool,
}
+pub enum Compression {
+ Zstd,
+ Lzo,
+ GZip,
+}
+
pub struct VmaBackupArgs {
pub vma_file_path: Option<OsString>,
+ pub compression: Option<Compression>,
pub backup_id: String,
pub backup_time: i64,
pub notes: Option<String>,
@@ -467,7 +475,18 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let start_transfer_time = SystemTime::now();
- upload_vma_file(pbs_args, &args.vma_args)?;
+ for (_, vma_group) in args.grouped_vmas {
+ for backup_args in vma_group {
+ if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
+ eprintln!(
+ "Failed to upload vma file at {:?} - {e}",
+ backup_args.vma_file_path.unwrap_or("(stdin)".into()),
+ );
+ println!("Skipping VMID {}", backup_args.backup_id);
+ break;
+ }
+ }
+ }
let transfer_duration = SystemTime::now().duration_since(start_transfer_time)?;
let total_seconds = transfer_duration.as_secs();
@@ -480,13 +499,43 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
- let vma_file: Box<dyn BufRead> = match &backup_args.vma_file_path {
- Some(vma_file_path) => match File::open(vma_file_path) {
- Err(why) => return Err(anyhow!("Couldn't open file: {}", why)),
- Ok(file) => Box::new(BufReader::new(file)),
+ match &backup_args.vma_file_path {
+ Some(vma_file_path) => println!("Uploading VMA backup from {vma_file_path:?}"),
+ None => println!("Uploading VMA backup from (stdin)"),
+ };
+
+ let vma_file: Box<dyn BufRead> = match &backup_args.compression {
+ Some(compression) => {
+ let vma_file_path = backup_args
+ .vma_file_path
+ .as_ref()
+ .expect("No VMA file path provided");
+ let mut cmd = match compression {
+ Compression::Zstd => {
+ let mut cmd = Command::new("zstd");
+ cmd.args(["-q", "-d", "-c"]);
+ cmd
+ }
+ Compression::Lzo => {
+ let mut cmd = Command::new("lzop");
+ cmd.args(["-d", "-c"]);
+ cmd
+ }
+ Compression::GZip => Command::new("zcat"),
+ };
+ let process = cmd.arg(vma_file_path).stdout(Stdio::piped()).spawn()?;
+ let stdout = process.stdout.expect("Failed to capture stdout");
+ Box::new(BufReader::new(stdout))
+ }
+ None => match &backup_args.vma_file_path {
+ Some(vma_file_path) => match File::open(vma_file_path) {
+ Err(why) => return Err(anyhow!("Couldn't open file: {why}")),
+ Ok(file) => Box::new(BufReader::new(file)),
+ },
+ None => Box::new(BufReader::new(stdin())),
},
- None => Box::new(BufReader::new(stdin())),
};
+
let vma_reader = VmaReader::new(vma_file)?;
let pbs = create_pbs_backup_task(pbs_args, backup_args)?;
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* [pbs-devel] [PATCH vma-to-pbs v6 2/4] add option to skip vmids whose backups failed to upload
2024-11-13 15:57 [pbs-devel] [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Filip Schauer
2024-11-13 15:57 ` [pbs-devel] [PATCH vma-to-pbs v6 1/4] " Filip Schauer
@ 2024-11-13 15:58 ` Filip Schauer
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 3/4] use level-based logging instead of println Filip Schauer
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-11-13 15:58 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/main.rs | 6 ++++++
src/vma2pbs.rs | 13 ++++++++++---
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/main.rs b/src/main.rs
index de1e6e0..e87da87 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -54,6 +54,9 @@ Options:
File containing a comment/notes
[--log-file <LOG_FILE>]
Log file
+ --skip-failed
+ Skip VMIDs that failed to be uploaded and continue onto the next VMID if a dump directory
+ is specified.
-y, --yes
Automatic yes to prompts
-h, --help
@@ -74,6 +77,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
"--compress",
"-e",
"--encrypt",
+ "--skip-failed",
"-y",
"--yes",
];
@@ -123,6 +127,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let key_password_file: Option<OsString> = args.opt_value_from_str("--key-password-file")?;
let notes_file: Option<OsString> = args.opt_value_from_str("--notes-file")?;
let log_file_path: Option<OsString> = args.opt_value_from_str("--log-file")?;
+ let skip_failed = args.contains("--skip-failed");
let yes = args.contains(["-y", "--yes"]);
match (encrypt, keyfile.is_some()) {
@@ -350,6 +355,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let options = BackupVmaToPbsArgs {
pbs_args,
grouped_vmas,
+ skip_failed,
};
Ok(options)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index b26c62e..b7de0cd 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -32,6 +32,7 @@ const VMA_CLUSTER_SIZE: usize = 65536;
pub struct BackupVmaToPbsArgs {
pub pbs_args: PbsArgs,
pub grouped_vmas: HashMap<String, Vec<VmaBackupArgs>>,
+ pub skip_failed: bool,
}
pub struct PbsArgs {
@@ -478,12 +479,18 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
for (_, vma_group) in args.grouped_vmas {
for backup_args in vma_group {
if let Err(e) = upload_vma_file(pbs_args, &backup_args) {
- eprintln!(
+ let err_msg = format!(
"Failed to upload vma file at {:?} - {e}",
backup_args.vma_file_path.unwrap_or("(stdin)".into()),
);
- println!("Skipping VMID {}", backup_args.backup_id);
- break;
+
+ if args.skip_failed {
+ eprintln!("{}", err_msg);
+ println!("Skipping VMID {}", backup_args.backup_id);
+ break;
+ } else {
+ bail!(err_msg);
+ }
}
}
}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* [pbs-devel] [PATCH vma-to-pbs v6 3/4] use level-based logging instead of println
2024-11-13 15:57 [pbs-devel] [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Filip Schauer
2024-11-13 15:57 ` [pbs-devel] [PATCH vma-to-pbs v6 1/4] " Filip Schauer
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 2/4] add option to skip vmids whose backups failed to upload Filip Schauer
@ 2024-11-13 15:58 ` Filip Schauer
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 4/4] log device upload progress as a percentage Filip Schauer
2024-11-22 7:47 ` [pbs-devel] applied-series: [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Fabian Grünbichler
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-11-13 15:58 UTC (permalink / raw)
To: pbs-devel
Use log level "info" by default and prevent spamming messages for every
single chunk uploaded. To re-enable these messages, set the RUST_LOG
environment variable to "debug".
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Cargo.toml | 2 ++
src/main.rs | 28 ++++++++++++++++++++++------
src/vma2pbs.rs | 37 ++++++++++++++++---------------------
3 files changed, 40 insertions(+), 27 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index ad80304..7951bbc 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,7 +8,9 @@ edition = "2021"
anyhow = "1.0"
bincode = "1.3"
chrono = "0.4"
+env_logger = "0.10"
hyper = "0.14.5"
+log = "0.4"
pico-args = "0.5"
md5 = "0.7.0"
regex = "1.7"
diff --git a/src/main.rs b/src/main.rs
index e87da87..f942a73 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -6,6 +6,7 @@ use std::path::PathBuf;
use anyhow::{bail, Context, Error};
use chrono::NaiveDateTime;
+use env_logger::Target;
use proxmox_sys::linux::tty;
use proxmox_time::epoch_i64;
use regex::Regex;
@@ -132,7 +133,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
match (encrypt, keyfile.is_some()) {
(true, false) => bail!("--encrypt requires a --keyfile!"),
- (false, true) => println!(
+ (false, true) => log::info!(
"--keyfile given, but --encrypt not set -> backup will be signed, but not encrypted!"
),
_ => {}
@@ -194,7 +195,7 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Some(key_password)
} else if vma_file_path.is_none() {
- println!(
+ log::info!(
"Please use --key-password-file to provide the password when passing the VMA file \
to stdin, if required."
);
@@ -250,13 +251,17 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
let Some((_, [backup_id, timestr, ext])) =
re.captures(file_name).map(|c| c.extract())
else {
- // Skip the file, since it is not a VMA backup
+ log::debug!("Skip \"{file_name}\", since it is not a VMA backup");
continue;
};
if let Some(ref vmid) = vmid {
if backup_id != vmid {
- // Skip the backup, since it does not match the specified vmid
+ log::debug!(
+ "Skip backup with VMID {}, since it does not match specified VMID {}",
+ backup_id,
+ vmid
+ );
continue;
}
}
@@ -312,13 +317,13 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
bail!("Did not find any backup archives");
}
- println!(
+ log::info!(
"Found {total_vma_count} backup archive(s) of {} different VMID(s):",
grouped_vmas.len()
);
for (backup_id, vma_group) in &grouped_vmas {
- println!("- VMID {backup_id}: {} backups", vma_group.len());
+ log::info!("- VMID {backup_id}: {} backups", vma_group.len());
}
if !yes {
@@ -361,7 +366,18 @@ fn parse_args() -> Result<BackupVmaToPbsArgs, Error> {
Ok(options)
}
+fn init_cli_logger() {
+ env_logger::Builder::from_env(env_logger::Env::new().filter_or("RUST_LOG", "info"))
+ .format_level(false)
+ .format_target(false)
+ .format_timestamp(None)
+ .target(Target::Stdout)
+ .init();
+}
+
fn main() -> Result<(), Error> {
+ init_cli_logger();
+
let args = parse_args()?;
vma2pbs(args)?;
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index b7de0cd..af60e44 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -82,8 +82,8 @@ fn create_pbs_backup_task(
pbs_args: &PbsArgs,
backup_args: &VmaBackupArgs,
) -> Result<*mut ProxmoxBackupHandle, Error> {
- println!(
- "backup time: {}",
+ log::info!(
+ "\tbackup time: {}",
epoch_to_rfc3339(backup_args.backup_time)?
);
@@ -152,7 +152,7 @@ where
let config_name = config.name;
let config_data = config.content;
- println!("CFG: size: {} name: {}", config_data.len(), config_name);
+ log::info!("\tCFG: size: {} name: {config_name}", config_data.len());
let config_name_cstr = CString::new(config_name)?;
@@ -190,10 +190,7 @@ where
let device_name = vma_reader.get_device_name(device_id.try_into()?)?;
let device_size = vma_reader.get_device_size(device_id.try_into()?)?;
- println!(
- "DEV: dev_id={} size: {} devname: {}",
- device_id, device_size, device_name
- );
+ log::info!("\tDEV: dev_id={device_id} size: {device_size} devname: {device_name}");
let device_name_cstr = CString::new(device_name)?;
let pbs_device_id = proxmox_backup_register_image(
@@ -276,10 +273,8 @@ where
};
let pbs_upload_chunk = |pbs_chunk_buffer: Option<&[u8]>| {
- println!(
- "Uploading dev_id: {} offset: {:#0X} - {:#0X}",
- dev_id,
- pbs_chunk_offset,
+ log::debug!(
+ "\tUploading dev_id: {dev_id} offset: {pbs_chunk_offset:#0X} - {:#0X}",
pbs_chunk_offset + pbs_chunk_size,
);
@@ -466,13 +461,13 @@ fn set_notes(
pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let pbs_args = &args.pbs_args;
- println!("PBS repository: {}", pbs_args.pbs_repository);
+ log::info!("PBS repository: {}", pbs_args.pbs_repository);
if let Some(ns) = &pbs_args.namespace {
- println!("PBS namespace: {}", ns);
+ log::info!("PBS namespace: {ns}");
}
- println!("PBS fingerprint: {}", pbs_args.fingerprint);
- println!("compress: {}", pbs_args.compress);
- println!("encrypt: {}", pbs_args.encrypt);
+ log::info!("PBS fingerprint: {}", pbs_args.fingerprint);
+ log::info!("compress: {}", pbs_args.compress);
+ log::info!("encrypt: {}", pbs_args.encrypt);
let start_transfer_time = SystemTime::now();
@@ -485,8 +480,8 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
);
if args.skip_failed {
- eprintln!("{}", err_msg);
- println!("Skipping VMID {}", backup_args.backup_id);
+ log::warn!("{}", err_msg);
+ log::info!("Skipping VMID {}", backup_args.backup_id);
break;
} else {
bail!(err_msg);
@@ -500,15 +495,15 @@ pub fn vma2pbs(args: BackupVmaToPbsArgs) -> Result<(), Error> {
let minutes = total_seconds / 60;
let seconds = total_seconds % 60;
let milliseconds = transfer_duration.as_millis() % 1000;
- println!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
+ log::info!("Backup finished within {minutes} minutes, {seconds} seconds and {milliseconds} ms");
Ok(())
}
fn upload_vma_file(pbs_args: &PbsArgs, backup_args: &VmaBackupArgs) -> Result<(), Error> {
match &backup_args.vma_file_path {
- Some(vma_file_path) => println!("Uploading VMA backup from {vma_file_path:?}"),
- None => println!("Uploading VMA backup from (stdin)"),
+ Some(vma_file_path) => log::info!("Uploading VMA backup from {vma_file_path:?}"),
+ None => log::info!("Uploading VMA backup from (stdin)"),
};
let vma_file: Box<dyn BufRead> = match &backup_args.compression {
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* [pbs-devel] [PATCH vma-to-pbs v6 4/4] log device upload progress as a percentage
2024-11-13 15:57 [pbs-devel] [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Filip Schauer
` (2 preceding siblings ...)
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 3/4] use level-based logging instead of println Filip Schauer
@ 2024-11-13 15:58 ` Filip Schauer
2024-11-22 7:47 ` [pbs-devel] applied-series: [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Fabian Grünbichler
4 siblings, 0 replies; 6+ messages in thread
From: Filip Schauer @ 2024-11-13 15:58 UTC (permalink / raw)
To: pbs-devel
Log the upload progress of a device as a percentage with log level info
every 1000 chunks.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/vma2pbs.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/vma2pbs.rs b/src/vma2pbs.rs
index af60e44..7457b9a 100644
--- a/src/vma2pbs.rs
+++ b/src/vma2pbs.rs
@@ -6,6 +6,8 @@ use std::fs::File;
use std::io::{stdin, BufRead, BufReader, Read};
use std::process::{Command, Stdio};
use std::ptr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
use std::time::SystemTime;
use anyhow::{anyhow, bail, Error};
@@ -229,6 +231,8 @@ where
non_zero_mask: u64,
}
+ let chunk_stats = Arc::new([const { AtomicU64::new(0) }; VMA_MAX_DEVICES]);
+
let images_chunks: RefCell<HashMap<u8, HashMap<u64, ImageChunk>>> =
RefCell::new(HashMap::new());
@@ -277,6 +281,11 @@ where
"\tUploading dev_id: {dev_id} offset: {pbs_chunk_offset:#0X} - {:#0X}",
pbs_chunk_offset + pbs_chunk_size,
);
+ let chunk_stat = chunk_stats[dev_id as usize].fetch_add(1, Ordering::SeqCst);
+ if (chunk_stat % 1000) == 0 {
+ let percentage = 100 * PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE * chunk_stat / device_size;
+ log::info!("\tUploading dev_id: {dev_id} ({percentage}%)");
+ }
let mut pbs_err: *mut c_char = ptr::null_mut();
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* [pbs-devel] applied-series: [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory
2024-11-13 15:57 [pbs-devel] [PATCH vma-to-pbs v6 0/4] add support for bulk import of a dump directory Filip Schauer
` (3 preceding siblings ...)
2024-11-13 15:58 ` [pbs-devel] [PATCH vma-to-pbs v6 4/4] log device upload progress as a percentage Filip Schauer
@ 2024-11-22 7:47 ` Fabian Grünbichler
4 siblings, 0 replies; 6+ messages in thread
From: Fabian Grünbichler @ 2024-11-22 7:47 UTC (permalink / raw)
To: Filip Schauer, pbs-devel
thanks!
Quoting Filip Schauer (2024-11-13 16:57:58)
> When a path to a directory is provided in the vma_file argument, try to
> upload all VMA backups in the directory. This also handles compressed
> VMA files, notes and logs. If a vmid is specified with --vmid, only the
> backups of that particular vmid are uploaded.
>
> Also improve the readability of the log messages to keep track of all
> imported backups.
>
> Changed since v5:
> * Extend the help text by seperate usages for single VMA import and bulk
> import
> * Move variables into format strings where possible
>
> Changed since v4:
> * Switch grouped_vmas from Vec<Vec<>> to HashMap<String, Vec<>>
> * Remove dependency on itertools
> * bail when no backups were found
> * Default to yes on the bulk import confirmation prompt
> * bail on invalid input to the bulk import confirmation prompt
>
> Changed since v3:
> * Mention in the description of the --vmid argument, that it is required
> if a single VMA file is provided
> * Construct grouped_vmas in place
> * Add debug logs when gathering files for bulk import
> * Log a summary of the files gathered for bulk import
> * Remove the "confusing VMA file path" error message in the second
> commit
> * Switch chunk_stats from Arc<Mutex<[u64; 256]>> to
> Arc<[AtomicU64; 256]> and use fetch_add to atomically increment and
> fetch the chunk stat
> * Ask for confirmation before bulk import
> * Add --yes option to skip the confirmation prompt
>
> Changed since v2:
> * Make skipping a VMID on error optional with the --skip-failed option
> * Switch log output from stderr to stdout
> * Bump itertools to 0.13
>
> Changed since v1:
> * Do not recurse through dump directory
> * Compile regex once before iterating over the files in the dump
> directory
> * Use extract on regex capture groups
> * Do not use deprecated method `chrono::NaiveDateTime::timestamp`
> * Use proxmox_sys::fs::file_read_optional_string
> * Group VMA files by VMID and continue with next VMID on error
> * Move the BackupVmaToPbsArgs split into its own commit
> * Remove hard coded occurences of 255
> * Use level-based logging instead of println
>
> Filip Schauer (4):
> add support for bulk import of a dump directory
> add option to skip vmids whose backups failed to upload
> use level-based logging instead of println
> log device upload progress as a percentage
>
> Cargo.toml | 4 +
> src/main.rs | 198 +++++++++++++++++++++++++++++++++++++++++++++----
> src/vma2pbs.rs | 108 +++++++++++++++++++++------
> 3 files changed, 271 insertions(+), 39 deletions(-)
>
> --
> 2.39.5
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 6+ messages in thread