* [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval @ 2024-10-23 9:10 Christian Ebner 2024-10-23 9:10 ` [pbs-devel] [PATCH v2 proxmox 1/5] time: fix typos in `TimeSpan` related docstring Christian Ebner ` (4 more replies) 0 siblings, 5 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:10 UTC (permalink / raw) To: pbs-devel These patches allow to specify a time based or size based interval for progress log output as generated during `proxmox-backup-client backup` runs. The client is extended by optional `progress-interval` and `progress-size-interval` parameters, which allows to set the interval either as `TimeSpan` or `HumanByte` parsable input string, depending on the variant. The parameters are in conflict with each other, only one of them might be set at the same time. If none of the parameters is given, the default time based interval of 1 minute is used. Examples for client invocations are: - no progress logging: `proxmox-backup-client backup root.pxar:/ --progress-interval=0` - time based progress logging with 1min 30s interval `proxmox-backup-client backup root.pxar:/ --progress-interval="1m 30s"` - size based progress logging with 512MiB interval `proxmox-backup-client backup root.pxar:/ --progress-size-interval=512MiB` Lower limits are set to 1s and 100MiB for the corresponding variant in order to prevent excessive log output. If values below the respective limits are given, the progress log output is disabled. Most notable changes since version 1 (thanks to Thomas for feedback): - Split time and size based variants into separate options - Disabling of log output is now handled by setting a value below the minimum. proxmox: Christian Ebner (2): time: fix typos in `TimeSpan` related docstring time: also implement `From<&TimeSpan> for f64` proxmox-time/src/time_span.rs | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) proxmox-backup: Christian Ebner (3): api-types: client: add wrapper api type for TimeSpan client: progress log: factor out log message generation client: progress log: allow to specify backup log interval pbs-api-types/src/client.rs | 49 ++++++++++++++ pbs-api-types/src/lib.rs | 3 + pbs-client/src/backup_writer.rs | 104 +++++++++++++++++++++++++----- proxmox-backup-client/src/main.rs | 54 ++++++++++++++-- 4 files changed, 188 insertions(+), 22 deletions(-) create mode 100644 pbs-api-types/src/client.rs -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 proxmox 1/5] time: fix typos in `TimeSpan` related docstring 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner @ 2024-10-23 9:10 ` Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox 2/5] time: also implement `From<&TimeSpan> for f64` Christian Ebner ` (3 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:10 UTC (permalink / raw) To: pbs-devel Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 1: - not present in previous version proxmox-time/src/time_span.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proxmox-time/src/time_span.rs b/proxmox-time/src/time_span.rs index a1bbbca9..ae311898 100644 --- a/proxmox-time/src/time_span.rs +++ b/proxmox-time/src/time_span.rs @@ -1,14 +1,14 @@ //! Timespans that try to be compatible with the systemd time span format. //! //! Time spans refer to time durations, like [std::time::Duration] but in the format that is -//! targetting human interfaces and that systemd understands. Parts of this documentation have been +//! targeting human interfaces and that systemd understands. Parts of this documentation have been //! adapted from the systemd.time manual page. //! //! The following time units are understood: //! - `nsec`, `ns` (not always accepted by systemd.time) //! - `usec`, `us`, `µs` //! - `msec`, `ms` -//! - `seconds`, s`econd`, `sec`, `s` +//! - `seconds`, `second`, `sec`, `s` //! - `minutes`, `minute`, `min`, `m` //! - `hours`, `hour`, `hr`, `h` //! - `days`, `day`, `d` @@ -26,7 +26,7 @@ //! spaces between units and/or values can be added or omitted. The order of the time values does //! not matter. //! -//! The following examples are all represeting the exact same time span of 1 day 2 hours and 3 +//! The following examples are all representing the exact same time span of 1 day 2 hours and 3 //! minutes: //! //! - `1d 2h 3m` -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 proxmox 2/5] time: also implement `From<&TimeSpan> for f64` 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner 2024-10-23 9:10 ` [pbs-devel] [PATCH v2 proxmox 1/5] time: fix typos in `TimeSpan` related docstring Christian Ebner @ 2024-10-23 9:11 ` Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 3/5] api-types: client: add wrapper api type for TimeSpan Christian Ebner ` (2 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:11 UTC (permalink / raw) To: pbs-devel Extend the already present `From<TimeSpan> for f64` implementation to allow using the reference as well. There is no need to take ownership and consume the `TimeSpan` object for conversion. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 1: - not present in previous version proxmox-time/src/time_span.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/proxmox-time/src/time_span.rs b/proxmox-time/src/time_span.rs index ae311898..2e3ca99e 100644 --- a/proxmox-time/src/time_span.rs +++ b/proxmox-time/src/time_span.rs @@ -128,8 +128,8 @@ pub struct TimeSpan { pub years: u64, } -impl From<TimeSpan> for f64 { - fn from(ts: TimeSpan) -> Self { +impl From<&TimeSpan> for f64 { + fn from(ts: &TimeSpan) -> Self { (ts.seconds as f64) + ((ts.nsec as f64) / 1_000_000_000.0) + ((ts.usec as f64) / 1_000_000.0) @@ -143,6 +143,12 @@ impl From<TimeSpan> for f64 { } } +impl From<TimeSpan> for f64 { + fn from(ts: TimeSpan) -> Self { + Self::from(&ts) + } +} + impl From<std::time::Duration> for TimeSpan { fn from(duration: std::time::Duration) -> Self { let mut duration = duration.as_nanos(); -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 3/5] api-types: client: add wrapper api type for TimeSpan 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner 2024-10-23 9:10 ` [pbs-devel] [PATCH v2 proxmox 1/5] time: fix typos in `TimeSpan` related docstring Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox 2/5] time: also implement `From<&TimeSpan> for f64` Christian Ebner @ 2024-10-23 9:11 ` Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 4/5] client: progress log: factor out log message generation Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 5/5] client: progress log: allow to specify backup log interval Christian Ebner 4 siblings, 0 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:11 UTC (permalink / raw) To: pbs-devel Implementing the `ApiType` for `TimeSpan` directly does not work since `proxmox-schema` depends on `proxmox-time`. Add a wrapper type `TimeInterval` and implement the api type on this as a workaround. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 1: - Move `LogInterval` to client, it is not an api type anymore - Implement `TimeInterval` wrapper for `TimeSpan`, to avoid cyclic create dependencies for `proxmox-schema` and `proxmox-time` if `TimeSpan` would be implemented as api type pbs-api-types/src/client.rs | 49 +++++++++++++++++++++++++++++++++++++ pbs-api-types/src/lib.rs | 3 +++ 2 files changed, 52 insertions(+) create mode 100644 pbs-api-types/src/client.rs diff --git a/pbs-api-types/src/client.rs b/pbs-api-types/src/client.rs new file mode 100644 index 000000000..2d2c864dd --- /dev/null +++ b/pbs-api-types/src/client.rs @@ -0,0 +1,49 @@ +use std::str::FromStr; + +use anyhow::Error; + +use proxmox_schema::{ApiStringFormat, ApiType, Schema, StringSchema}; +use proxmox_time::TimeSpan; + +// Wrapper type for `TimeSpan` to avoid cyclic create dependency for `proxmox-time` and `proxmox-schema` +pub struct TimeInterval(TimeSpan); + +impl FromStr for TimeInterval { + type Err = Error; + + fn from_str(value: &str) -> Result<Self, Self::Err> { + Ok(Self(TimeSpan::from_str(value)?)) + } +} + +impl From<TimeInterval> for TimeSpan { + fn from(time_interval: TimeInterval) -> TimeSpan { + time_interval.0 + } +} + +impl std::fmt::Display for TimeInterval { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + TimeSpan::fmt(&self.0, f) + } +} + +impl ApiType for TimeInterval { + const API_SCHEMA: Schema = StringSchema::new("Time interval") + .format(&ApiStringFormat::VerifyFn(verify_time_interval)) + .schema(); +} + +impl TimeInterval { + pub fn as_f64(&self) -> f64 { + std::primitive::f64::from(&self.0) + } +} + +pub fn verify_time_interval(value: &str) -> Result<(), Error> { + let _: TimeSpan = value.parse()?; + Ok(()) +} + +proxmox_serde::forward_serialize_to_display!(TimeInterval); +proxmox_serde::forward_deserialize_to_from_str!(TimeInterval); diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs index 460c7da7c..bd16ab16c 100644 --- a/pbs-api-types/src/lib.rs +++ b/pbs-api-types/src/lib.rs @@ -93,6 +93,9 @@ pub const GROUP_OR_SNAPSHOT_PATH_REGEX_STR: &str = mod acl; pub use acl::*; +mod client; +pub use client::*; + mod datastore; pub use datastore::*; -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 4/5] client: progress log: factor out log message generation 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner ` (2 preceding siblings ...) 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 3/5] api-types: client: add wrapper api type for TimeSpan Christian Ebner @ 2024-10-23 9:11 ` Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 5/5] client: progress log: allow to specify backup log interval Christian Ebner 4 siblings, 0 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:11 UTC (permalink / raw) To: pbs-devel Move the progress log output into a dedicated helper function, so it can be reused for size based progress log output as well. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 1: - no changes pbs-client/src/backup_writer.rs | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs index 4d2e8a801..37ee39e2e 100644 --- a/pbs-client/src/backup_writer.rs +++ b/pbs-client/src/backup_writer.rs @@ -2,6 +2,7 @@ use std::collections::HashSet; use std::future::Future; use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; +use std::time::Instant; use anyhow::{bail, format_err, Error}; use futures::future::{self, AbortHandle, Either, FutureExt, TryFutureExt}; @@ -690,12 +691,11 @@ impl BackupWriter { Some(tokio::spawn(async move { loop { tokio::time::sleep(tokio::time::Duration::from_secs(60)).await; - - let size = HumanByte::from(stream_len3.load(Ordering::SeqCst)); - let size_uploaded = HumanByte::from(uploaded_len.load(Ordering::SeqCst)); - let elapsed = TimeSpan::from(start_time.elapsed()); - - log::info!("processed {size} in {elapsed}, uploaded {size_uploaded}"); + progress_log( + stream_len3.load(Ordering::SeqCst), + uploaded_len.load(Ordering::SeqCst), + &start_time, + ); } })) } else { @@ -921,3 +921,12 @@ impl BackupWriter { Ok(speed) } } + +#[inline(always)] +fn progress_log(size: usize, size_uploaded: usize, start_time: &Instant) { + let size = HumanByte::from(size); + let size_uploaded = HumanByte::from(size_uploaded); + let elapsed = TimeSpan::from(start_time.elapsed()); + + log::info!("processed {size} in {elapsed}, uploaded {size_uploaded}"); +} -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* [pbs-devel] [PATCH v2 proxmox-backup 5/5] client: progress log: allow to specify backup log interval 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner ` (3 preceding siblings ...) 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 4/5] client: progress log: factor out log message generation Christian Ebner @ 2024-10-23 9:11 ` Christian Ebner 4 siblings, 0 replies; 6+ messages in thread From: Christian Ebner @ 2024-10-23 9:11 UTC (permalink / raw) To: pbs-devel Adds the optional parameters `progress-interval` and `progress-size-interval` which allow to specify the interval to use for backup progress log output. The progress can be specified as time based value given as a `TimeSpan` to `progress-interval` or as size based progress log, giving a `HumanByte` compatible value to `progress-size-interval`. The options are in conflict with each other, only one can be set on invocation. Minimum values of 1s and 100MiB are set for the corresponding variant to protect from excessive output. Values below the respective minimum disable the progress log output althogehter. Examplary client invocations are: - no progress logging: `proxmox-backup-client backup root.pxar:/ --progress-interval=0` - time based progress logging with 1m 30s interval `proxmox-backup-client backup root.pxar:/ --progress-interval="1m 30s"` - size based progress logging with 512MiB interval `proxmox-backup-client backup root.pxar:/ --progress-size-interval=512MiB` Without providing the optional parameter, the current default is set to time based logging with an interval of 1 minute. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- changes since version 1: - Adapt to separate optional parameters for time and size based intervals - Disable logging if given values are below corresponding limits - Define respective minimum values as constants pbs-client/src/backup_writer.rs | 93 +++++++++++++++++++++++++------ proxmox-backup-client/src/main.rs | 54 ++++++++++++++++-- 2 files changed, 126 insertions(+), 21 deletions(-) diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs index 37ee39e2e..f0b31443d 100644 --- a/pbs-client/src/backup_writer.rs +++ b/pbs-client/src/backup_writer.rs @@ -1,5 +1,6 @@ use std::collections::HashSet; use std::future::Future; +use std::str::FromStr; use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Instant; @@ -53,6 +54,7 @@ pub struct UploadOptions { pub compress: bool, pub encrypt: bool, pub fixed_size: Option<u64>, + pub progress_log_interval: Option<LogInterval>, } struct UploadStats { @@ -72,6 +74,19 @@ struct ChunkUploadResponse { size: usize, } +#[derive(Clone)] +pub enum LogInterval { + None, + Size(HumanByte), + Time(TimeSpan), +} + +impl Default for LogInterval { + fn default() -> Self { + Self::Time(TimeSpan::from_str("60s").unwrap()) + } +} + type UploadQueueSender = mpsc::Sender<(MergedChunkInfo, Option<ChunkUploadResponse>)>; type UploadResultReceiver = oneshot::Receiver<Result<(), Error>>; @@ -359,6 +374,7 @@ impl BackupWriter { options.compress, injections, archive, + options.progress_log_interval, ) .await?; @@ -653,6 +669,7 @@ impl BackupWriter { compress: bool, injections: Option<std::sync::mpsc::Receiver<InjectChunks>>, archive: &str, + progress_log_interval: Option<LogInterval>, ) -> impl Future<Output = Result<UploadStats, Error>> { let total_chunks = Arc::new(AtomicUsize::new(0)); let total_chunks2 = total_chunks.clone(); @@ -671,6 +688,8 @@ impl BackupWriter { let injected_len = Arc::new(AtomicUsize::new(0)); let injected_len2 = injected_len.clone(); let uploaded_len = Arc::new(AtomicUsize::new(0)); + let uploaded_len2 = uploaded_len.clone(); + let previous_byte_fraction = Arc::new(AtomicUsize::new(0)); let append_chunk_path = format!("{}_index", prefix); let upload_chunk_path = format!("{}_chunk", prefix); @@ -684,23 +703,34 @@ impl BackupWriter { let index_csum = Arc::new(Mutex::new(Some(openssl::sha::Sha256::new()))); let index_csum_2 = index_csum.clone(); - let progress_handle = if archive.ends_with(".img") - || archive.ends_with(".pxar") - || archive.ends_with(".ppxar") - { - Some(tokio::spawn(async move { - loop { - tokio::time::sleep(tokio::time::Duration::from_secs(60)).await; - progress_log( - stream_len3.load(Ordering::SeqCst), - uploaded_len.load(Ordering::SeqCst), - &start_time, - ); + let mut progress_handle = None; + let mut progress_byte_interval = 0; + match progress_log_interval { + Some(LogInterval::Time(ref time_span)) => { + if archive.ends_with(".img") + || archive.ends_with(".pxar") + || archive.ends_with(".ppxar") + { + let duration = std::primitive::f64::from(time_span.clone()); + progress_handle = Some(tokio::spawn(async move { + loop { + tokio::time::sleep(tokio::time::Duration::from_secs_f64(duration)) + .await; + progress_log( + stream_len3.load(Ordering::SeqCst), + uploaded_len.load(Ordering::SeqCst), + &start_time, + ) + } + })) } - })) - } else { - None - }; + } + Some(LogInterval::Size(ref human_byte)) => { + progress_byte_interval = human_byte.as_u64() as usize + } + Some(LogInterval::None) => {} + None => {} + } stream .inject_reused_chunks(injections, stream_len.clone()) @@ -717,6 +747,15 @@ impl BackupWriter { for chunk in chunks { let offset = stream_len.fetch_add(chunk.size() as usize, Ordering::SeqCst) as u64; + + progress_log_by_byte_interval( + progress_byte_interval, + (offset + chunk.size()) as usize, + &previous_byte_fraction, + &uploaded_len2, + &start_time, + ); + reused_len.fetch_add(chunk.size() as usize, Ordering::SeqCst); injected_len.fetch_add(chunk.size() as usize, Ordering::SeqCst); let digest = chunk.digest(); @@ -734,6 +773,14 @@ impl BackupWriter { total_chunks.fetch_add(1, Ordering::SeqCst); let offset = stream_len.fetch_add(chunk_len, Ordering::SeqCst) as u64; + progress_log_by_byte_interval( + progress_byte_interval, + offset as usize + chunk_len, + &previous_byte_fraction, + &uploaded_len2, + &start_time, + ); + let mut chunk_builder = DataChunkBuilder::new(data.as_ref()).compress(compress); if let Some(ref crypt_config) = crypt_config { @@ -922,6 +969,20 @@ impl BackupWriter { } } +#[inline(always)] +fn progress_log_by_byte_interval( + interval: usize, + pos: usize, + previous: &Arc<AtomicUsize>, + uploaded: &Arc<AtomicUsize>, + start_time: &Instant, +) { + if interval > 0 && pos / interval > previous.load(Ordering::SeqCst) { + previous.store(pos / interval, Ordering::SeqCst); + progress_log(pos, uploaded.load(Ordering::SeqCst), start_time); + } +} + #[inline(always)] fn progress_log(size: usize, size_uploaded: usize, start_time: &Instant) { let size = HumanByte::from(size); diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs index e4034aa99..47afaa446 100644 --- a/proxmox-backup-client/src/main.rs +++ b/proxmox-backup-client/src/main.rs @@ -4,6 +4,7 @@ use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::{Arc, Mutex}; use std::task::Context; +use std::time::Duration; use anyhow::{bail, format_err, Error}; use futures::stream::{StreamExt, TryStreamExt}; @@ -15,20 +16,21 @@ use xdg::BaseDirectories; use pathpatterns::{MatchEntry, MatchType, PatternFlag}; use proxmox_async::blocking::TokioWriterAdapter; +use proxmox_human_byte::HumanByte; use proxmox_io::StdChannelWriter; use proxmox_log::init_cli_logger; use proxmox_router::{cli::*, ApiMethod, RpcEnvironment}; use proxmox_schema::api; use proxmox_sys::fs::{file_get_json, image_size, replace_file, CreateOptions}; -use proxmox_time::{epoch_i64, strftime_local}; +use proxmox_time::{epoch_i64, strftime_local, TimeSpan}; use pxar::accessor::aio::Accessor; use pxar::accessor::{MaybeReady, ReadAt, ReadAtOperation}; use pbs_api_types::{ Authid, BackupDir, BackupGroup, BackupNamespace, BackupPart, BackupType, ClientRateLimitConfig, CryptMode, Fingerprint, GroupListItem, PruneJobOptions, PruneListItem, RateLimitConfig, - SnapshotListItem, StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, - BACKUP_TYPE_SCHEMA, + SnapshotListItem, StorageStatus, TimeInterval, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, + BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, }; use pbs_client::catalog_shell::Shell; use pbs_client::pxar::{ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarPrevRef}; @@ -46,8 +48,8 @@ use pbs_client::tools::{ use pbs_client::{ delete_ticket_info, parse_backup_specification, view_task_result, BackupDetectionMode, BackupReader, BackupRepository, BackupSpecificationType, BackupStats, BackupWriter, - ChunkStream, FixedChunkStream, HttpClient, InjectionData, PxarBackupStream, RemoteChunkReader, - UploadOptions, BACKUP_SOURCE_SCHEMA, + ChunkStream, FixedChunkStream, HttpClient, InjectionData, LogInterval, PxarBackupStream, + RemoteChunkReader, UploadOptions, BACKUP_SOURCE_SCHEMA, }; use pbs_datastore::catalog::{BackupCatalogWriter, CatalogReader, CatalogWriter}; use pbs_datastore::chunk_store::verify_chunk_size; @@ -87,6 +89,9 @@ pub use snapshot::*; mod task; pub use task::*; +const PROGRESS_LOG_TIME_MIN: u64 = 1; +const PROGRESS_LOG_SIZE_MIN: u64 = 100 * 1024 * 1024; + fn record_repository(repo: &BackupRepository) { let base = match BaseDirectories::with_prefix("proxmox-backup") { Ok(v) => v, @@ -734,6 +739,14 @@ fn spawn_catalog_upload( optional: true, default: false, }, + "progress-interval": { + type: TimeInterval, + optional: true, + }, + "progress-size-interval": { + type: HumanByte, + optional: true, + }, } } )] @@ -746,6 +759,8 @@ async fn create_backup( dry_run: bool, skip_e2big_xattr: bool, limit: ClientRateLimitConfig, + progress_interval: Option<TimeInterval>, + progress_size_interval: Option<HumanByte>, _info: &ApiMethod, _rpcenv: &mut dyn RpcEnvironment, ) -> Result<Value, Error> { @@ -782,6 +797,33 @@ async fn create_backup( let empty = Vec::new(); let exclude_args = param["exclude"].as_array().unwrap_or(&empty); + let mut progress_log_interval = LogInterval::default(); + if let Some(time_interval) = progress_interval { + if progress_size_interval.is_some() { + bail!("'progress-interval' and 'progress-size-interval' are in conflict"); + } + + if time_interval.as_f64() >= PROGRESS_LOG_TIME_MIN as f64 { + progress_log_interval = LogInterval::Time(time_interval.into()); + } else { + progress_log_interval = LogInterval::None; + log::info!( + "Log interval '{time_interval}' below minimum '{}', progress log disabled.", + TimeSpan::from(Duration::from_secs(PROGRESS_LOG_TIME_MIN)), + ); + } + } else if let Some(human_byte) = progress_size_interval { + if human_byte.as_u64() >= PROGRESS_LOG_SIZE_MIN { + progress_log_interval = LogInterval::Size(human_byte); + } else { + progress_log_interval = LogInterval::None; + log::info!( + "Log interval '{human_byte}' below minimum '{}', progress log disabled.", + HumanByte::from(PROGRESS_LOG_SIZE_MIN), + ); + } + } + let mut pattern_list = Vec::with_capacity(exclude_args.len()); for entry in exclude_args { let entry = entry @@ -1132,6 +1174,7 @@ async fn create_backup( previous_manifest: previous_manifest.clone(), compress: true, encrypt: crypto.mode == CryptMode::Encrypt, + progress_log_interval: Some(progress_log_interval.clone()), ..UploadOptions::default() }; @@ -1169,6 +1212,7 @@ async fn create_backup( fixed_size: Some(size), compress: true, encrypt: crypto.mode == CryptMode::Encrypt, + progress_log_interval: Some(progress_log_interval.clone()), }; let stats = -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-10-23 9:10 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-10-23 9:10 [pbs-devel] [PATCH v2 proxmox proxmox-backup 0/5] backup client progress log interval Christian Ebner 2024-10-23 9:10 ` [pbs-devel] [PATCH v2 proxmox 1/5] time: fix typos in `TimeSpan` related docstring Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox 2/5] time: also implement `From<&TimeSpan> for f64` Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 3/5] api-types: client: add wrapper api type for TimeSpan Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 4/5] client: progress log: factor out log message generation Christian Ebner 2024-10-23 9:11 ` [pbs-devel] [PATCH v2 proxmox-backup 5/5] client: progress log: allow to specify backup log interval Christian Ebner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox