* [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs
@ 2026-06-10 18:02 Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats` Shan Shaji
` (5 more replies)
0 siblings, 6 replies; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
This series addresses three issues:
- File number statistics not being shown under change detection summary for data and
legacy mode.
- The pogress logs were not showing the number of processed files.
- There was no verbose option.
Sending this as an RFC to get an initial feedback and to make sure I have not
missed anything. Thanks in advance.
Shan Shaji (6):
refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats`
fix #7024: pxar: show archiver summary for legacy and data mode
refactor: use atomic values in `PxarArchiverProgressStats`
fix #7024: cli: show number of processed files during backup
refactor: rename `UploadCounters` to `UploadProgress`
fix #7024: cli: add option to enable verbose logs
pbs-client/src/backup_stats.rs | 9 +-
pbs-client/src/backup_writer.rs | 110 ++++++++----
pbs-client/src/inject_reused_chunks.rs | 8 +-
pbs-client/src/pxar/create.rs | 159 +++++++++++++-----
pbs-client/src/pxar/mod.rs | 3 +-
pbs-client/src/pxar/tools.rs | 10 +-
pbs-client/src/pxar_backup_stream.rs | 13 +-
proxmox-backup-client/src/main.rs | 38 ++++-
.../src/proxmox_restore_daemon/api.rs | 2 +
pxar-bin/src/main.rs | 2 +
tests/catar.rs | 1 +
11 files changed, 271 insertions(+), 84 deletions(-)
--
2.47.3
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats`
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:37 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode Shan Shaji
` (4 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
Earlier, the `ReuseStats` struct was used to count and show summary for
metadata mode. Since, the same counter will be used to show stats for
both legacy and data mode, rename `ReuseStats` struct to
`PxarArchiverProgressStats` and its corresponding propery inside the
archiver to `progress_stats`.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/pxar/create.rs | 46 +++++++++++++++++------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index 304de4d83..c7f26879e 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -150,7 +150,7 @@ pub(crate) struct HardLinkInfo {
}
#[derive(Default)]
-struct ReuseStats {
+struct PxarArchiverProgressStats {
files_reused_count: u64,
files_hardlink_count: u64,
files_reencoded_count: u64,
@@ -193,7 +193,7 @@ struct Archiver {
// an older client without the encode-time check is detected here and the affected file is
// re-encoded instead of reused.
last_reusable_offset: Option<u64>,
- reuse_stats: ReuseStats,
+ progress_stats: PxarArchiverProgressStats,
split_archive: bool,
}
@@ -304,7 +304,7 @@ where
previous_payload_index,
cache: PxarLookaheadCache::new(options.max_cache_size),
last_reusable_offset: None,
- reuse_stats: ReuseStats::default(),
+ progress_stats: PxarArchiverProgressStats::default(),
split_archive,
};
@@ -325,28 +325,28 @@ where
info!("Change detection summary:");
info!(
" - {} total files ({} hardlinks)",
- archiver.reuse_stats.files_reused_count
- + archiver.reuse_stats.files_reencoded_count
- + archiver.reuse_stats.files_hardlink_count,
- archiver.reuse_stats.files_hardlink_count,
+ archiver.progress_stats.files_reused_count
+ + archiver.progress_stats.files_reencoded_count
+ + archiver.progress_stats.files_hardlink_count,
+ archiver.progress_stats.files_hardlink_count,
);
info!(
" - {} unchanged, reusable files with {} data",
- archiver.reuse_stats.files_reused_count,
- HumanByte::from(archiver.reuse_stats.total_reused_payload_size),
+ archiver.progress_stats.files_reused_count,
+ HumanByte::from(archiver.progress_stats.total_reused_payload_size),
);
info!(
" - {} changed or non-reusable files with {} data",
- archiver.reuse_stats.files_reencoded_count,
- HumanByte::from(archiver.reuse_stats.total_reencoded_size),
+ archiver.progress_stats.files_reencoded_count,
+ HumanByte::from(archiver.progress_stats.total_reencoded_size),
);
info!(
" - {} padding in {} partially reused chunks",
HumanByte::from(
- archiver.reuse_stats.total_injected_size
- - archiver.reuse_stats.total_reused_payload_size
+ archiver.progress_stats.total_injected_size
+ - archiver.progress_stats.total_reused_payload_size
),
- archiver.reuse_stats.partial_chunks_count,
+ archiver.progress_stats.partial_chunks_count,
);
}
Ok(())
@@ -971,24 +971,24 @@ impl Archiver {
}
let offset: LinkOffset = if let Some(payload_offset) = payload_offset {
- self.reuse_stats.total_reused_payload_size +=
+ self.progress_stats.total_reused_payload_size +=
file_size + size_of::<pxar::format::Header>() as u64;
- self.reuse_stats.files_reused_count += 1;
+ self.progress_stats.files_reused_count += 1;
encoder
.add_payload_ref(metadata, file_name, file_size, payload_offset)
.await?
} else {
- self.reuse_stats.total_reencoded_size +=
+ self.progress_stats.total_reencoded_size +=
file_size + size_of::<pxar::format::Header>() as u64;
- self.reuse_stats.files_reencoded_count += 1;
+ self.progress_stats.files_reencoded_count += 1;
self.add_regular_file(encoder, fd, file_name, metadata, file_size)
.await?
};
if stat.st_nlink > 1 {
- self.reuse_stats.files_hardlink_count += 1;
+ self.progress_stats.files_hardlink_count += 1;
self.hardlinks
.insert(link_info, (self.path.clone(), offset));
}
@@ -1215,11 +1215,11 @@ impl Archiver {
HumanByte::from(chunk.padding),
HumanByte::from(chunk.size()),
);
- self.reuse_stats.total_injected_size += chunk.size();
- self.reuse_stats.total_injected_count += 1;
+ self.progress_stats.total_injected_size += chunk.size();
+ self.progress_stats.total_injected_count += 1;
if chunk.padding > 0 {
- self.reuse_stats.partial_chunks_count += 1;
+ self.progress_stats.partial_chunks_count += 1;
}
size = size.add(chunk.size());
@@ -2075,7 +2075,7 @@ mod tests {
suggested_boundaries: Some(suggested_boundaries),
cache: PxarLookaheadCache::new(None),
last_reusable_offset: None,
- reuse_stats: ReuseStats::default(),
+ progress_stats: PxarArchiverProgressStats::default(),
split_archive: true,
};
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats` Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:37 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats` Shan Shaji
` (3 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
earlier, the archiver summary was only being shown when the change
detection mode is metadata. To improve this, log the total number of
files and the amount of data re-encoded for legacy and data modes
as well.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/pxar/create.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index c7f26879e..9100ac1a3 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -348,6 +348,15 @@ where
),
archiver.progress_stats.partial_chunks_count,
);
+ } else {
+ info!("Processing summary:");
+ info!(
+ "- {} total files ({} hardlinks) with {} data",
+ archiver.progress_stats.files_reencoded_count
+ + archiver.progress_stats.files_hardlink_count,
+ archiver.progress_stats.files_hardlink_count,
+ HumanByte::from(archiver.progress_stats.total_reencoded_size),
+ );
}
Ok(())
}
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats`
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats` Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup Shan Shaji
` (2 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
Make all fields of the PxarArchiverProgressStats struct atomic.
This enables safe, lock free access to the archiver stats from
within the progress logs while uploading the stream.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/pxar/create.rs | 121 ++++++++++++++++++++++++----------
1 file changed, 85 insertions(+), 36 deletions(-)
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index 9100ac1a3..1246f67d9 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -7,6 +7,7 @@ use std::ops::Range;
use std::os::unix::ffi::OsStrExt;
use std::os::unix::io::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd, RawFd};
use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex, mpsc};
use anyhow::{Context, Error, bail};
@@ -151,14 +152,43 @@ pub(crate) struct HardLinkInfo {
#[derive(Default)]
struct PxarArchiverProgressStats {
- files_reused_count: u64,
- files_hardlink_count: u64,
- files_reencoded_count: u64,
- total_injected_count: u64,
- partial_chunks_count: u64,
- total_injected_size: u64,
- total_reused_payload_size: u64,
- total_reencoded_size: u64,
+ files_reused_count: AtomicU64,
+ files_hardlink_count: AtomicU64,
+ files_reencoded_count: AtomicU64,
+ total_injected_count: AtomicU64,
+ partial_chunks_count: AtomicU64,
+ total_injected_size: AtomicU64,
+ total_reused_payload_size: AtomicU64,
+ total_reencoded_size: AtomicU64,
+}
+
+impl PxarArchiverProgressStats {
+ pub fn total_files_count(&self) -> u64 {
+ self.files_reused_count.load(Ordering::SeqCst)
+ + self.files_hardlink_count.load(Ordering::SeqCst)
+ + self.files_reencoded_count.load(Ordering::SeqCst)
+ }
+
+ pub fn files_reencoded_count(&self) -> u64 {
+ self.files_reencoded_count.load(Ordering::SeqCst)
+ }
+
+ pub fn total_reencoded_size(&self) -> u64 {
+ self.total_reencoded_size.load(Ordering::SeqCst)
+ }
+
+ pub fn files_reused_count(&self) -> u64 {
+ self.files_reused_count.load(Ordering::SeqCst)
+ }
+
+ pub fn total_reused_payload_size(&self) -> u64 {
+ self.total_reused_payload_size.load(Ordering::SeqCst)
+ }
+
+ pub fn padding(&self) -> u64 {
+ self.total_injected_size.load(Ordering::SeqCst)
+ - self.total_reused_payload_size.load(Ordering::SeqCst)
+ }
}
#[derive(Serialize, Deserialize)]
@@ -325,37 +355,40 @@ where
info!("Change detection summary:");
info!(
" - {} total files ({} hardlinks)",
- archiver.progress_stats.files_reused_count
- + archiver.progress_stats.files_reencoded_count
- + archiver.progress_stats.files_hardlink_count,
- archiver.progress_stats.files_hardlink_count,
+ archiver.progress_stats.total_files_count(),
+ archiver
+ .progress_stats
+ .files_hardlink_count
+ .load(Ordering::SeqCst),
);
info!(
" - {} unchanged, reusable files with {} data",
- archiver.progress_stats.files_reused_count,
- HumanByte::from(archiver.progress_stats.total_reused_payload_size),
+ archiver.progress_stats.files_reused_count(),
+ HumanByte::from(archiver.progress_stats.total_reused_payload_size()),
);
info!(
" - {} changed or non-reusable files with {} data",
- archiver.progress_stats.files_reencoded_count,
- HumanByte::from(archiver.progress_stats.total_reencoded_size),
+ archiver.progress_stats.files_reencoded_count(),
+ HumanByte::from(archiver.progress_stats.total_reencoded_size()),
);
info!(
" - {} padding in {} partially reused chunks",
- HumanByte::from(
- archiver.progress_stats.total_injected_size
- - archiver.progress_stats.total_reused_payload_size
- ),
- archiver.progress_stats.partial_chunks_count,
+ HumanByte::from(archiver.progress_stats.padding()),
+ archiver
+ .progress_stats
+ .partial_chunks_count
+ .load(Ordering::SeqCst),
);
} else {
info!("Processing summary:");
info!(
- "- {} total files ({} hardlinks) with {} data",
- archiver.progress_stats.files_reencoded_count
- + archiver.progress_stats.files_hardlink_count,
- archiver.progress_stats.files_hardlink_count,
- HumanByte::from(archiver.progress_stats.total_reencoded_size),
+ " - {} total files ({} hardlinks) with {} data",
+ archiver.progress_stats.total_files_count(),
+ archiver
+ .progress_stats
+ .files_hardlink_count
+ .load(Ordering::SeqCst),
+ HumanByte::from(archiver.progress_stats.total_reencoded_size()),
);
}
Ok(())
@@ -980,24 +1013,34 @@ impl Archiver {
}
let offset: LinkOffset = if let Some(payload_offset) = payload_offset {
- self.progress_stats.total_reused_payload_size +=
- file_size + size_of::<pxar::format::Header>() as u64;
- self.progress_stats.files_reused_count += 1;
+ self.progress_stats.total_reused_payload_size.fetch_add(
+ file_size + size_of::<pxar::format::Header>() as u64,
+ Ordering::SeqCst,
+ );
+ self.progress_stats
+ .files_reused_count
+ .fetch_add(1, Ordering::SeqCst);
encoder
.add_payload_ref(metadata, file_name, file_size, payload_offset)
.await?
} else {
- self.progress_stats.total_reencoded_size +=
- file_size + size_of::<pxar::format::Header>() as u64;
- self.progress_stats.files_reencoded_count += 1;
+ self.progress_stats.total_reencoded_size.fetch_add(
+ file_size + size_of::<pxar::format::Header>() as u64,
+ Ordering::SeqCst,
+ );
+ self.progress_stats
+ .files_reencoded_count
+ .fetch_add(1, Ordering::SeqCst);
self.add_regular_file(encoder, fd, file_name, metadata, file_size)
.await?
};
if stat.st_nlink > 1 {
- self.progress_stats.files_hardlink_count += 1;
+ self.progress_stats
+ .files_hardlink_count
+ .fetch_add(1, Ordering::SeqCst);
self.hardlinks
.insert(link_info, (self.path.clone(), offset));
}
@@ -1224,11 +1267,17 @@ impl Archiver {
HumanByte::from(chunk.padding),
HumanByte::from(chunk.size()),
);
- self.progress_stats.total_injected_size += chunk.size();
- self.progress_stats.total_injected_count += 1;
+ self.progress_stats
+ .total_injected_size
+ .fetch_add(chunk.size(), Ordering::SeqCst);
+ self.progress_stats
+ .total_injected_count
+ .fetch_add(1, Ordering::SeqCst);
if chunk.padding > 0 {
- self.progress_stats.partial_chunks_count += 1;
+ self.progress_stats
+ .partial_chunks_count
+ .fetch_add(1, Ordering::SeqCst);
}
size = size.add(chunk.size());
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
` (2 preceding siblings ...)
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats` Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress` Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs Shan Shaji
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
earlier, the progress logs only reported the processed stream size and
uploaded size. To improve this, `PxarArchiverProgressStats` is now
created earlier and shared between the archiver and the upload stream.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/backup_stats.rs | 5 ++-
pbs-client/src/backup_writer.rs | 39 +++++++++++++++++--
pbs-client/src/pxar/create.rs | 9 +++--
pbs-client/src/pxar/mod.rs | 3 +-
pbs-client/src/pxar_backup_stream.rs | 13 ++++++-
proxmox-backup-client/src/main.rs | 27 +++++++++++--
.../src/proxmox_restore_daemon/api.rs | 1 +
pxar-bin/src/main.rs | 1 +
tests/catar.rs | 1 +
9 files changed, 85 insertions(+), 14 deletions(-)
diff --git a/pbs-client/src/backup_stats.rs b/pbs-client/src/backup_stats.rs
index d7c13784b..aa5ae4889 100644
--- a/pbs-client/src/backup_stats.rs
+++ b/pbs-client/src/backup_stats.rs
@@ -4,6 +4,7 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::time::Duration;
+use crate::pxar::PxarArchiverProgressStats;
use crate::pxar::create::ReusableDynamicEntry;
/// Basic backup run statistics and archive checksum
@@ -50,11 +51,12 @@ pub(crate) struct UploadCounters {
injected_stream_len: Arc<AtomicUsize>,
reused_stream_len: Arc<AtomicUsize>,
total_stream_len: Arc<AtomicUsize>,
+ pub pxar_progress: Option<Arc<PxarArchiverProgressStats>>,
}
impl UploadCounters {
/// Create and zero init new upload counters
- pub(crate) fn new() -> Self {
+ pub(crate) fn new(pxar_progress: Option<Arc<PxarArchiverProgressStats>>) -> Self {
Self {
total_chunk_count: Arc::new(AtomicUsize::new(0)),
injected_chunk_count: Arc::new(AtomicUsize::new(0)),
@@ -63,6 +65,7 @@ impl UploadCounters {
injected_stream_len: Arc::new(AtomicUsize::new(0)),
reused_stream_len: Arc::new(AtomicUsize::new(0)),
total_stream_len: Arc::new(AtomicUsize::new(0)),
+ pxar_progress: pxar_progress,
}
}
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index f0744ecaf..d0246689c 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -28,6 +28,8 @@ use proxmox_human_byte::HumanByte;
use proxmox_log::{Level, debug, enabled, info, trace, warn};
use proxmox_time::TimeSpan;
+use crate::pxar::PxarArchiverProgressStats;
+
use super::backup_stats::{BackupStats, UploadCounters, UploadStats};
use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
use super::merge_known_chunks::{MergeKnownChunks, MergedChunkInfo};
@@ -53,6 +55,7 @@ pub struct UploadOptions {
pub compress: bool,
pub encrypt: bool,
pub index_type: IndexType,
+ pub is_metadata_mode: bool,
}
/// Index type for upload options.
@@ -327,7 +330,7 @@ impl BackupWriter {
.as_u64()
.unwrap();
- let mut counters = UploadCounters::new();
+ let mut counters = UploadCounters::new(None);
let counters_readonly = counters.clone();
let is_fixed_chunk_size = prefix == "fixed";
@@ -377,6 +380,7 @@ impl BackupWriter {
stream,
index_csum_2,
counters_readonly,
+ options.is_metadata_mode,
)
.await?;
@@ -400,6 +404,7 @@ impl BackupWriter {
stream: impl Stream<Item = Result<bytes::BytesMut, Error>>,
options: UploadOptions,
injections: Option<std::sync::mpsc::Receiver<InjectChunks>>,
+ pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
) -> Result<BackupStats, Error> {
let known_chunks = Arc::new(Mutex::new(HashSet::new()));
@@ -478,6 +483,8 @@ impl BackupWriter {
options.compress,
injections,
archive_name,
+ pxar_archive_progress_stats,
+ options.is_metadata_mode,
)
.await?;
@@ -770,8 +777,10 @@ impl BackupWriter {
compress: bool,
injections: Option<std::sync::mpsc::Receiver<InjectChunks>>,
archive: &BackupArchiveName,
+ pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
+ is_metadata_mode: bool,
) -> impl Future<Output = Result<UploadStats, Error>> {
- let mut counters = UploadCounters::new();
+ let mut counters = UploadCounters::new(pxar_archive_progress_stats);
let counters_readonly = counters.clone();
let is_fixed_chunk_size = prefix == "fixed";
@@ -855,6 +864,7 @@ impl BackupWriter {
stream,
index_csum_2,
counters_readonly,
+ is_metadata_mode,
)
}
@@ -866,6 +876,7 @@ impl BackupWriter {
stream: impl Stream<Item = Result<MergedChunkInfo, Error>>,
index_csum: Arc<Mutex<Option<Sha256>>>,
counters: UploadCounters,
+ is_metadata_mode: bool,
) -> impl Future<Output = Result<UploadStats, Error>> {
let append_chunk_path = format!("{prefix}_index");
let upload_chunk_path = format!("{prefix}_chunk");
@@ -889,7 +900,29 @@ impl BackupWriter {
let size_uploaded = HumanByte::from(uploaded_len.load(Ordering::SeqCst));
let elapsed = TimeSpan::from(start_time.elapsed());
- info!("processed {size} in {elapsed}, uploaded {size_uploaded}");
+ if let Some(pxar_progress) = &counters.pxar_progress {
+ let n_files = pxar_progress.total_files_count();
+ let reuse_payload_size = pxar_progress.total_reused_payload_size();
+ let total_of_logical_file_size = HumanByte::from(
+ pxar_progress.total_reencoded_size() + reuse_payload_size,
+ );
+ let total_reuse_payload_size = HumanByte::from(reuse_payload_size);
+ let reuse_file_count = pxar_progress.files_reused_count();
+
+ if is_metadata_mode {
+ info!(
+ "scanned {n_files} files with {total_of_logical_file_size} (reusing {reuse_file_count} files with \
+ {total_reuse_payload_size}) of which processed {size} in {elapsed} and uploaded {size_uploaded}"
+ );
+ } else {
+ info!(
+ "scanned {n_files} files with {total_of_logical_file_size} of which processed {size} \
+ in {elapsed} and uploaded {size_uploaded}"
+ );
+ }
+ } else {
+ info!("processed {size} in {elapsed}, uploaded {size_uploaded}");
+ }
}
}))
} else {
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index 1246f67d9..2d2dfbb3f 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -151,7 +151,7 @@ pub(crate) struct HardLinkInfo {
}
#[derive(Default)]
-struct PxarArchiverProgressStats {
+pub struct PxarArchiverProgressStats {
files_reused_count: AtomicU64,
files_hardlink_count: AtomicU64,
files_reencoded_count: AtomicU64,
@@ -223,7 +223,7 @@ struct Archiver {
// an older client without the encode-time check is detected here and the affected file is
// re-encoded instead of reused.
last_reusable_offset: Option<u64>,
- progress_stats: PxarArchiverProgressStats,
+ progress_stats: Arc<PxarArchiverProgressStats>,
split_archive: bool,
}
@@ -251,6 +251,7 @@ pub async fn create_archive<T, F>(
options: PxarCreateOptions,
forced_boundaries: Option<mpsc::Sender<InjectChunks>>,
suggested_boundaries: Option<mpsc::Sender<u64>>,
+ archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
) -> Result<(), Error>
where
T: SeqWrite + Send,
@@ -334,7 +335,7 @@ where
previous_payload_index,
cache: PxarLookaheadCache::new(options.max_cache_size),
last_reusable_offset: None,
- progress_stats: PxarArchiverProgressStats::default(),
+ progress_stats: archiver_progress_stats.unwrap_or_default(),
split_archive,
};
@@ -2133,7 +2134,7 @@ mod tests {
suggested_boundaries: Some(suggested_boundaries),
cache: PxarLookaheadCache::new(None),
last_reusable_offset: None,
- progress_stats: PxarArchiverProgressStats::default(),
+ progress_stats: Arc::new(PxarArchiverProgressStats::default()),
split_archive: true,
};
diff --git a/pbs-client/src/pxar/mod.rs b/pbs-client/src/pxar/mod.rs
index b88fac33f..d6820d01a 100644
--- a/pbs-client/src/pxar/mod.rs
+++ b/pbs-client/src/pxar/mod.rs
@@ -58,7 +58,8 @@ mod flags;
pub use flags::Flags;
pub use create::{
- MetadataArchiveReader, PxarCreateOptions, PxarPrevRef, PxarWriters, create_archive,
+ MetadataArchiveReader, PxarArchiverProgressStats, PxarCreateOptions, PxarPrevRef, PxarWriters,
+ create_archive,
};
pub use extract::{
ErrorHandler, OverwriteFlags, PxarExtractContext, PxarExtractOptions, create_tar, create_zip,
diff --git a/pbs-client/src/pxar_backup_stream.rs b/pbs-client/src/pxar_backup_stream.rs
index 430eee01f..d1b46cf8a 100644
--- a/pbs-client/src/pxar_backup_stream.rs
+++ b/pbs-client/src/pxar_backup_stream.rs
@@ -20,6 +20,7 @@ use proxmox_log::debug;
use pbs_datastore::catalog::{BackupCatalogWriter, CatalogWriter};
use crate::inject_reused_chunks::InjectChunks;
+use crate::pxar::PxarArchiverProgressStats;
use crate::pxar::create::PxarWriters;
/// Stream implementation to encode and upload .pxar archives.
@@ -50,6 +51,7 @@ impl PxarBackupStream {
options: crate::pxar::PxarCreateOptions,
boundaries: Option<mpsc::Sender<InjectChunks>>,
separate_payload_stream: bool,
+ archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
) -> Result<(Self, Option<Self>), Error> {
let buffer_size = 256 * 1024;
@@ -102,6 +104,7 @@ impl PxarBackupStream {
options,
boundaries,
suggested_boundaries_tx,
+ archiver_progress_stats,
)
.await
{
@@ -145,10 +148,18 @@ impl PxarBackupStream {
options: crate::pxar::PxarCreateOptions,
boundaries: Option<mpsc::Sender<InjectChunks>>,
separate_payload_stream: bool,
+ archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
) -> Result<(Self, Option<Self>), Error> {
let dir = nix::dir::Dir::open(dirname, OFlag::O_DIRECTORY, Mode::empty())?;
- Self::new(dir, catalog, options, boundaries, separate_payload_stream)
+ Self::new(
+ dir,
+ catalog,
+ options,
+ boundaries,
+ separate_payload_stream,
+ archiver_progress_stats,
+ )
}
}
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 2680add78..1036ec7e9 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -31,7 +31,9 @@ use pbs_api_types::{
RateLimitConfig, ServerIdentity, SnapshotListItem, StorageStatus,
};
use pbs_client::catalog_shell::Shell;
-use pbs_client::pxar::{ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarPrevRef};
+use pbs_client::pxar::{
+ ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarArchiverProgressStats, PxarPrevRef,
+};
use pbs_client::tools::{
CHUNK_SIZE_SCHEMA, complete_archive_name, complete_auth_id, complete_backup_group,
complete_backup_snapshot, complete_backup_source, complete_chunk_size,
@@ -268,6 +270,7 @@ async fn backup_directory<P: AsRef<Path>>(
bail!("cannot backup directory with fixed chunk size!");
}
+ let pxar_archiver_progress_stats = Arc::new(PxarArchiverProgressStats::default());
let (payload_boundaries_tx, payload_boundaries_rx) = std::sync::mpsc::channel();
let (pxar_stream, payload_stream) = PxarBackupStream::open(
dir_path.as_ref(),
@@ -275,6 +278,7 @@ async fn backup_directory<P: AsRef<Path>>(
pxar_create_options,
Some(payload_boundaries_tx),
payload_target.is_some(),
+ Some(pxar_archiver_progress_stats.clone()),
)?;
let mut chunk_stream = ChunkStream::new(pxar_stream, chunk_size, None, None);
@@ -289,7 +293,13 @@ async fn backup_directory<P: AsRef<Path>>(
}
});
- let stats = client.upload_stream(archive_name, stream, upload_options.clone(), None);
+ let stats = client.upload_stream(
+ archive_name,
+ stream,
+ upload_options.clone(),
+ None,
+ Some(pxar_archiver_progress_stats.clone()),
+ );
if let Some(mut payload_stream) = payload_stream {
let payload_target = payload_target
@@ -319,6 +329,7 @@ async fn backup_directory<P: AsRef<Path>>(
stream,
upload_options,
Some(payload_injections_rx),
+ Some(pxar_archiver_progress_stats),
);
match futures::join!(stats, payload_stats) {
@@ -359,7 +370,7 @@ async fn backup_image<P: AsRef<Path>>(
}
let stats = client
- .upload_stream(archive_name, stream, upload_options, None)
+ .upload_stream(archive_name, stream, upload_options, None, None)
.await?;
Ok(stats)
@@ -652,7 +663,13 @@ fn spawn_catalog_upload(
tokio::spawn(async move {
let catalog_upload_result = client
- .upload_stream(&CATALOG_NAME, catalog_chunk_stream, upload_options, None)
+ .upload_stream(
+ &CATALOG_NAME,
+ catalog_chunk_stream,
+ upload_options,
+ None,
+ None,
+ )
.await;
if let Err(ref err) = catalog_upload_result {
@@ -1234,6 +1251,7 @@ async fn create_backup(
previous_manifest: previous_manifest.clone(),
compress: true,
encrypt: crypto.mode == CryptMode::Encrypt,
+ is_metadata_mode: detection_mode.is_metadata(),
..UploadOptions::default()
};
@@ -1273,6 +1291,7 @@ async fn create_backup(
index_type: IndexType::Fixed(image_file_size),
compress: true,
encrypt: crypto.mode == CryptMode::Encrypt,
+ is_metadata_mode: false,
};
let stats =
diff --git a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
index ff21c9bee..b7f4fd141 100644
--- a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
+++ b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
@@ -375,6 +375,7 @@ fn extract(
options,
None,
None,
+ None
)
.await
}
diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
index 4b14507ca..953c6e756 100644
--- a/pxar-bin/src/main.rs
+++ b/pxar-bin/src/main.rs
@@ -450,6 +450,7 @@ async fn create_archive(
options,
None,
None,
+ None,
)
.await?;
diff --git a/tests/catar.rs b/tests/catar.rs
index aed23f866..81ca9d8db 100644
--- a/tests/catar.rs
+++ b/tests/catar.rs
@@ -41,6 +41,7 @@ fn run_test(dir_name: &str) -> Result<(), Error> {
options,
None,
None,
+ None,
))?;
Command::new("cmp")
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress`
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
` (3 preceding siblings ...)
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs Shan Shaji
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
Earlier, `UploadCounters` was used for accounting upload stream
progress information. Now the `PxarArchiverProgressStats` has been added
as an optional field inside the struct, which makes the responsibility of
struct not only just counting the progress but also to show the the pxar
archive progress. So, renamed `UploadCounters` to `UploadProgress`.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/backup_stats.rs | 4 +--
pbs-client/src/backup_writer.rs | 37 +++++++++++++-------------
pbs-client/src/inject_reused_chunks.rs | 8 +++---
3 files changed, 25 insertions(+), 24 deletions(-)
diff --git a/pbs-client/src/backup_stats.rs b/pbs-client/src/backup_stats.rs
index aa5ae4889..2866c29a1 100644
--- a/pbs-client/src/backup_stats.rs
+++ b/pbs-client/src/backup_stats.rs
@@ -43,7 +43,7 @@ impl UploadStats {
/// Atomic counters for accounting upload stream progress information
#[derive(Clone)]
-pub(crate) struct UploadCounters {
+pub(crate) struct UploadProgress {
injected_chunk_count: Arc<AtomicUsize>,
known_chunk_count: Arc<AtomicUsize>,
total_chunk_count: Arc<AtomicUsize>,
@@ -54,7 +54,7 @@ pub(crate) struct UploadCounters {
pub pxar_progress: Option<Arc<PxarArchiverProgressStats>>,
}
-impl UploadCounters {
+impl UploadProgress {
/// Create and zero init new upload counters
pub(crate) fn new(pxar_progress: Option<Arc<PxarArchiverProgressStats>>) -> Self {
Self {
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index d0246689c..68379f725 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -30,7 +30,7 @@ use proxmox_time::TimeSpan;
use crate::pxar::PxarArchiverProgressStats;
-use super::backup_stats::{BackupStats, UploadCounters, UploadStats};
+use super::backup_stats::{BackupStats, UploadProgress, UploadStats};
use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
use super::merge_known_chunks::{MergeKnownChunks, MergedChunkInfo};
@@ -330,8 +330,8 @@ impl BackupWriter {
.as_u64()
.unwrap();
- let mut counters = UploadCounters::new(None);
- let counters_readonly = counters.clone();
+ let mut upload_progress = UploadProgress::new(None);
+ let upload_progress_readonly = upload_progress.clone();
let is_fixed_chunk_size = prefix == "fixed";
@@ -343,8 +343,8 @@ impl BackupWriter {
match merged_chunk_info {
MergedChunkInfo::New(ref chunk_info) => {
let chunk_len = chunk_info.chunk_len;
- let offset =
- counters.add_new_chunk(chunk_len as usize, chunk_info.chunk.raw_size());
+ let offset = upload_progress
+ .add_new_chunk(chunk_len as usize, chunk_info.chunk.raw_size());
let end_offset = offset as u64 + chunk_len;
let mut guard = index_csum.lock().unwrap();
let csum = guard.as_mut().unwrap();
@@ -355,7 +355,7 @@ impl BackupWriter {
}
MergedChunkInfo::Known(ref mut known_chunk_list) => {
for (chunk_len, digest) in known_chunk_list {
- let offset = counters.add_known_chunk(*chunk_len as usize);
+ let offset = upload_progress.add_known_chunk(*chunk_len as usize);
let end_offset = offset as u64 + *chunk_len;
let mut guard = index_csum.lock().unwrap();
let csum = guard.as_mut().unwrap();
@@ -379,7 +379,7 @@ impl BackupWriter {
prefix,
stream,
index_csum_2,
- counters_readonly,
+ upload_progress_readonly,
options.is_metadata_mode,
)
.await?;
@@ -780,8 +780,8 @@ impl BackupWriter {
pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
is_metadata_mode: bool,
) -> impl Future<Output = Result<UploadStats, Error>> {
- let mut counters = UploadCounters::new(pxar_archive_progress_stats);
- let counters_readonly = counters.clone();
+ let mut upload_progress = UploadProgress::new(pxar_archive_progress_stats);
+ let upload_progress_readonly = upload_progress.clone();
let is_fixed_chunk_size = prefix == "fixed";
@@ -789,7 +789,7 @@ impl BackupWriter {
let index_csum_2 = index_csum.clone();
let stream = stream
- .inject_reused_chunks(injections, counters.clone())
+ .inject_reused_chunks(injections, upload_progress.clone())
.and_then(move |chunk_info| match chunk_info {
InjectedChunksInfo::Known(chunks) => {
// account for injected chunks
@@ -797,7 +797,7 @@ impl BackupWriter {
let mut guard = index_csum.lock().unwrap();
let csum = guard.as_mut().unwrap();
for chunk in chunks {
- let offset = counters.add_injected_chunk(&chunk) as u64;
+ let offset = upload_progress.add_injected_chunk(&chunk) as u64;
let digest = chunk.digest();
known.push((offset, digest));
let end_offset = offset + chunk.size();
@@ -819,13 +819,14 @@ impl BackupWriter {
let mut known_chunks = known_chunks.lock().unwrap();
let digest = *chunk_builder.digest();
let (offset, res) = if known_chunks.contains(&digest) {
- let offset = counters.add_known_chunk(chunk_len) as u64;
+ let offset = upload_progress.add_known_chunk(chunk_len) as u64;
(offset, MergedChunkInfo::Known(vec![(offset, digest)]))
} else {
match chunk_builder.build() {
Ok((chunk, digest)) => {
- let offset =
- counters.add_new_chunk(chunk_len, chunk.raw_size()) as u64;
+ let offset = upload_progress
+ .add_new_chunk(chunk_len, chunk.raw_size())
+ as u64;
known_chunks.insert(digest);
(
offset,
@@ -863,7 +864,7 @@ impl BackupWriter {
prefix,
stream,
index_csum_2,
- counters_readonly,
+ upload_progress_readonly,
is_metadata_mode,
)
}
@@ -875,7 +876,7 @@ impl BackupWriter {
prefix: &str,
stream: impl Stream<Item = Result<MergedChunkInfo, Error>>,
index_csum: Arc<Mutex<Option<Sha256>>>,
- counters: UploadCounters,
+ upload_progress: UploadProgress,
is_metadata_mode: bool,
) -> impl Future<Output = Result<UploadStats, Error>> {
let append_chunk_path = format!("{prefix}_index");
@@ -891,7 +892,7 @@ impl BackupWriter {
|| archive.ends_with(".pxar.didx")
|| archive.ends_with(".ppxar.didx")
{
- let counters = counters.clone();
+ let counters = upload_progress.clone();
Some(tokio::spawn(async move {
loop {
tokio::time::sleep(tokio::time::Duration::from_secs(60)).await;
@@ -998,7 +999,7 @@ impl BackupWriter {
handle.abort();
}
- futures::future::ok(counters.to_upload_stats(csum, start_time.elapsed()))
+ futures::future::ok(upload_progress.to_upload_stats(csum, start_time.elapsed()))
})
}
diff --git a/pbs-client/src/inject_reused_chunks.rs b/pbs-client/src/inject_reused_chunks.rs
index c85a28a08..314dd94ce 100644
--- a/pbs-client/src/inject_reused_chunks.rs
+++ b/pbs-client/src/inject_reused_chunks.rs
@@ -7,7 +7,7 @@ use anyhow::{Error, anyhow};
use futures::{Stream, ready};
use pin_project_lite::pin_project;
-use crate::backup_stats::UploadCounters;
+use crate::backup_stats::UploadProgress;
use crate::pxar::create::ReusableDynamicEntry;
pin_project! {
@@ -16,7 +16,7 @@ pin_project! {
input: S,
next_injection: Option<InjectChunks>,
injections: Option<mpsc::Receiver<InjectChunks>>,
- counters: UploadCounters,
+ counters: UploadProgress,
}
}
@@ -42,7 +42,7 @@ pub trait InjectReusedChunks: Sized {
fn inject_reused_chunks(
self,
injections: Option<mpsc::Receiver<InjectChunks>>,
- counters: UploadCounters,
+ counters: UploadProgress,
) -> InjectReusedChunksQueue<Self>;
}
@@ -53,7 +53,7 @@ where
fn inject_reused_chunks(
self,
injections: Option<mpsc::Receiver<InjectChunks>>,
- counters: UploadCounters,
+ counters: UploadProgress,
) -> InjectReusedChunksQueue<Self> {
InjectReusedChunksQueue {
input: self,
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
` (4 preceding siblings ...)
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress` Shan Shaji
@ 2026-06-10 18:02 ` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
5 siblings, 1 reply; 13+ messages in thread
From: Shan Shaji @ 2026-06-10 18:02 UTC (permalink / raw)
To: pbs-devel
earlier, there was no option to enable verbose logging inside the CLI.
By default only info level logs were only logged.
To get more information, users had to enable debug logs by setting the
PBS_LOG environment variable, but that was too much information. To fix
this, accept a verbose option and log only necessary information in info
level when the flag is set as true.
Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
---
pbs-client/src/backup_writer.rs | 38 +++++++++++++------
pbs-client/src/pxar/create.rs | 30 +++++++++++++--
pbs-client/src/pxar/tools.rs | 10 ++++-
proxmox-backup-client/src/main.rs | 11 ++++++
.../src/proxmox_restore_daemon/api.rs | 1 +
pxar-bin/src/main.rs | 1 +
6 files changed, 74 insertions(+), 17 deletions(-)
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index 68379f725..15a9bccde 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -29,6 +29,7 @@ use proxmox_log::{Level, debug, enabled, info, trace, warn};
use proxmox_time::TimeSpan;
use crate::pxar::PxarArchiverProgressStats;
+use crate::pxar::tools::log_message;
use super::backup_stats::{BackupStats, UploadProgress, UploadStats};
use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
@@ -56,6 +57,7 @@ pub struct UploadOptions {
pub encrypt: bool,
pub index_type: IndexType,
pub is_metadata_mode: bool,
+ pub verbose: bool,
}
/// Index type for upload options.
@@ -525,20 +527,32 @@ impl BackupWriter {
archive, reused, reused_percent
);
}
- if enabled!(Level::DEBUG) && upload_stats.chunk_count > 0 {
- debug!(
- "{}: Reused {} from {} chunks.",
- archive, upload_stats.chunk_reused, upload_stats.chunk_count
+
+ if (options.verbose || enabled!(Level::DEBUG)) && upload_stats.chunk_count > 0 {
+ log_message(
+ &format!(
+ "{}: Reused {} from {} chunks.",
+ archive, upload_stats.chunk_reused, upload_stats.chunk_count
+ ),
+ options.verbose,
);
- debug!(
- "{}: Average chunk size was {}.",
- archive,
- HumanByte::from(upload_stats.size / upload_stats.chunk_count)
+
+ log_message(
+ &format!(
+ "{}: Average chunk size was {}.",
+ archive,
+ HumanByte::from(upload_stats.size / upload_stats.chunk_count)
+ ),
+ options.verbose,
);
- debug!(
- "{}: Average time per request: {} microseconds.",
- archive,
- (upload_stats.duration.as_micros()) / (upload_stats.chunk_count as u128)
+
+ log_message(
+ &format!(
+ "{}: Average time per request: {} microseconds.",
+ archive,
+ (upload_stats.duration.as_micros()) / (upload_stats.chunk_count as u128)
+ ),
+ options.verbose,
);
}
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index 2d2dfbb3f..85f03d662 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -39,7 +39,7 @@ use crate::inject_reused_chunks::InjectChunks;
use crate::pxar::Flags;
use crate::pxar::look_ahead_cache::{CacheEntry, CacheEntryData, PxarLookaheadCache};
use crate::pxar::metadata::errno_is_unsupported;
-use crate::pxar::tools::assert_single_path_component;
+use crate::pxar::tools::{assert_single_path_component, log_message};
const CHUNK_PADDING_THRESHOLD: f64 = 0.1;
@@ -60,6 +60,8 @@ pub struct PxarCreateOptions {
pub previous_ref: Option<PxarPrevRef>,
/// Maximum number of lookahead cache entries
pub max_cache_size: Option<usize>,
+ /// Enable verbose loging
+ pub verbose: bool,
}
pub type MetadataArchiveReader = Arc<dyn ReadAt + Send + Sync + 'static>;
@@ -225,6 +227,7 @@ struct Archiver {
last_reusable_offset: Option<u64>,
progress_stats: Arc<PxarArchiverProgressStats>,
split_archive: bool,
+ verbose: bool,
}
type Encoder<'a, T> = pxar::encoder::aio::Encoder<'a, T>;
@@ -337,6 +340,7 @@ where
last_reusable_offset: None,
progress_stats: archiver_progress_stats.unwrap_or_default(),
split_archive,
+ verbose: options.verbose,
};
archiver
@@ -511,18 +515,35 @@ impl Archiver {
}
let range =
*offset..*offset + size + size_of::<pxar::format::Header>() as u64;
+
+ if self.verbose {
+ info!("reusable: {file_name:?}");
+ }
+
debug!(
"reusable: {file_name:?} at range {range:?} has unchanged metadata."
);
return Ok(Some(range));
}
- debug!("re-encode: {file_name:?} not a regular file.");
+
+ log_message(
+ &format!("re-encode: {file_name:?} not a regular file."),
+ self.verbose,
+ );
return Ok(None);
}
- debug!("re-encode: {file_name:?} metadata did not match.");
+
+ log_message(
+ &format!("re-encode: {file_name:?} metadata did not match."),
+ self.verbose,
+ );
return Ok(None);
}
- debug!("re-encode: {file_name:?} not found in previous archive.");
+
+ log_message(
+ &format!("re-encode: {file_name:?} not found in previous archive."),
+ self.verbose,
+ );
}
Ok(None)
@@ -2136,6 +2157,7 @@ mod tests {
last_reusable_offset: None,
progress_stats: Arc::new(PxarArchiverProgressStats::default()),
split_archive: true,
+ verbose: false,
};
let accessor = Accessor::new(pxar::PxarVariant::Unified(reader), metadata_size)
diff --git a/pbs-client/src/pxar/tools.rs b/pbs-client/src/pxar/tools.rs
index 0c8c2927e..67d6a740b 100644
--- a/pbs-client/src/pxar/tools.rs
+++ b/pbs-client/src/pxar/tools.rs
@@ -21,7 +21,7 @@ use pbs_datastore::BackupManifest;
use pbs_datastore::dynamic_index::{BufferedDynamicReader, LocalDynamicReadAt};
use pbs_datastore::index::IndexFile;
use pbs_tools::crypt_config::CryptConfig;
-use proxmox_log::{debug, info};
+use proxmox_log::{Level, debug, enabled, info};
use crate::{BackupReader, RemoteChunkReader};
@@ -485,3 +485,11 @@ pub async fn pxar_metadata_catalog_find<'future, T: Clone + Send + Sync + ReadAt
}
Ok(())
}
+
+pub(crate) fn log_message(msg: &str, verbose: bool) {
+ if verbose && !enabled!(Level::DEBUG) {
+ info!("{msg}");
+ } else {
+ debug!("{msg}");
+ }
+}
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 1036ec7e9..52bdbe798 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -805,6 +805,12 @@ fn spawn_catalog_upload(
optional: true,
default: false,
},
+ "verbose": {
+ type: Boolean,
+ description: "Enable verbose logs.",
+ optional: true,
+ default: false,
+ }
}
}
)]
@@ -848,6 +854,8 @@ async fn create_backup(
let include_dev = param["include-dev"].as_array();
+ let verbose = param["verbose"].as_bool().unwrap_or_default();
+
let entries_max = param["entries-max"]
.as_u64()
.unwrap_or(pbs_client::pxar::ENCODER_MAX_ENTRIES as u64);
@@ -1245,6 +1253,7 @@ async fn create_backup(
skip_e2big_xattr,
previous_ref,
max_cache_size,
+ verbose: verbose,
};
let upload_options = UploadOptions {
@@ -1252,6 +1261,7 @@ async fn create_backup(
compress: true,
encrypt: crypto.mode == CryptMode::Encrypt,
is_metadata_mode: detection_mode.is_metadata(),
+ verbose,
..UploadOptions::default()
};
@@ -1292,6 +1302,7 @@ async fn create_backup(
compress: true,
encrypt: crypto.mode == CryptMode::Encrypt,
is_metadata_mode: false,
+ verbose,
};
let stats =
diff --git a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
index b7f4fd141..e890681fc 100644
--- a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
+++ b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
@@ -364,6 +364,7 @@ fn extract(
skip_e2big_xattr: false,
previous_ref: None,
max_cache_size: None,
+ verbose: false
};
let pxar_writer = pxar::PxarVariant::Unified(TokioWriter::new(writer));
diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
index 953c6e756..385f54a0d 100644
--- a/pxar-bin/src/main.rs
+++ b/pxar-bin/src/main.rs
@@ -383,6 +383,7 @@ async fn create_archive(
skip_e2big_xattr: false,
previous_ref: None,
max_cache_size: None,
+ verbose: false,
};
let source = PathBuf::from(source);
--
2.47.3
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats`
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats` Shan Shaji
@ 2026-06-29 8:37 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:37 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
nit: s/refactor:/client: pxar:/ in the commit title.
We commonly use the tags to denote what subsystem of the code in the
respective repository this affects.
On 6/10/26 8:01 PM, Shan Shaji wrote:
> Earlier, the `ReuseStats` struct was used to count and show summary for
> metadata mode. Since, the same counter will be used to show stats for
> both legacy and data mode, rename `ReuseStats` struct to
> `PxarArchiverProgressStats` and its corresponding propery inside the
typo : s/propery/property/
> archiver to `progress_stats`.
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode Shan Shaji
@ 2026-06-29 8:37 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:37 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
On 6/10/26 8:02 PM, Shan Shaji wrote:
> earlier, the archiver summary was only being shown when the change
> detection mode is metadata. To improve this, log the total number of
> files and the amount of data re-encoded for legacy and data modes
> as well.
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
> pbs-client/src/pxar/create.rs | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
> index c7f26879e..9100ac1a3 100644
> --- a/pbs-client/src/pxar/create.rs
> +++ b/pbs-client/src/pxar/create.rs
> @@ -348,6 +348,15 @@ where
> ),
> archiver.progress_stats.partial_chunks_count,
> );
> + } else {
> + info!("Processing summary:");
> + info!(
> + "- {} total files ({} hardlinks) with {} data",
> + archiver.progress_stats.files_reencoded_count
> + + archiver.progress_stats.files_hardlink_count,
nit: the total here is basically the same as for the metadata mode above
(since reuse being 0)
this could therefore be defined as common helper (see suggested diff below).
But maybe even better, since you move these to atomic counters in patch
3 anyways, maybe reorder them and do the conversion first, then you
already have the helper you need at hand anyways.
> + archiver.progress_stats.files_hardlink_count,
> + HumanByte::from(archiver.progress_stats.total_reencoded_size),
> + );
> }
> Ok(())
> }
diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
index 56296171e..e541b829b 100644
--- a/pbs-client/src/pxar/create.rs
+++ b/pbs-client/src/pxar/create.rs
@@ -161,6 +161,14 @@ struct PxarArchiverProgressStats {
total_reencoded_size: u64,
}
+impl PxarArchiverProgressStats {
+ fn total_files_processed(&self) -> u64 {
+ self.files_reused_count
+ + self.files_reencoded_count
+ + self.files_hardlink_count
+ }
+}
+
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(crate) struct PbsClientPrelude {
@@ -325,9 +333,7 @@ where
info!("Change detection summary:");
info!(
" - {} total files ({} hardlinks)",
- archiver.progress_stats.files_reused_count
- + archiver.progress_stats.files_reencoded_count
- + archiver.progress_stats.files_hardlink_count,
+ archiver.progress_stats.total_files_processed(),
archiver.progress_stats.files_hardlink_count,
);
info!(
@@ -352,8 +358,7 @@ where
info!("Processing summary:");
info!(
"- {} total files ({} hardlinks) with {} data",
- archiver.progress_stats.files_reencoded_count
- + archiver.progress_stats.files_hardlink_count,
+ archiver.progress_stats.total_files_processed(),
archiver.progress_stats.files_hardlink_count,
HumanByte::from(archiver.progress_stats.total_reencoded_size),
);
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats`
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats` Shan Shaji
@ 2026-06-29 8:38 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:38 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
nit: s/refactor:/client: pxar:/ in commit title
On 6/10/26 8:01 PM, Shan Shaji wrote:
> Make all fields of the PxarArchiverProgressStats struct atomic.
> This enables safe, lock free access to the archiver stats from
> within the progress logs while uploading the stream.
But that is not what happens in this patch? The progress logging is
introduced in subsequent changes only, so maybe explicitly mention that
this is in preparation for doing this?
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
> pbs-client/src/pxar/create.rs | 121 ++++++++++++++++++++++++----------
> 1 file changed, 85 insertions(+), 36 deletions(-)
>
> diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
> index 9100ac1a3..1246f67d9 100644
> --- a/pbs-client/src/pxar/create.rs
> +++ b/pbs-client/src/pxar/create.rs
> @@ -7,6 +7,7 @@ use std::ops::Range;
> use std::os::unix::ffi::OsStrExt;
> use std::os::unix::io::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd, RawFd};
> use std::path::{Path, PathBuf};
> +use std::sync::atomic::{AtomicU64, Ordering};
> use std::sync::{Arc, Mutex, mpsc};
>
> use anyhow::{Context, Error, bail};
> @@ -151,14 +152,43 @@ pub(crate) struct HardLinkInfo {
>
> #[derive(Default)]
> struct PxarArchiverProgressStats {
> - files_reused_count: u64,
> - files_hardlink_count: u64,
> - files_reencoded_count: u64,
> - total_injected_count: u64,
> - partial_chunks_count: u64,
> - total_injected_size: u64,
> - total_reused_payload_size: u64,
> - total_reencoded_size: u64,
> + files_reused_count: AtomicU64,
> + files_hardlink_count: AtomicU64,
> + files_reencoded_count: AtomicU64,
> + total_injected_count: AtomicU64,
> + partial_chunks_count: AtomicU64,
> + total_injected_size: AtomicU64,
> + total_reused_payload_size: AtomicU64,
> + total_reencoded_size: AtomicU64,
> +}
> +
> +impl PxarArchiverProgressStats {
> + pub fn total_files_count(&self) -> u64 {
nit: non of these helpers need to be exposed as pub. Here they can be
fully contained in the module, on subsequent patches they might the be
relaxed to pub(crate) where needed.
> + self.files_reused_count.load(Ordering::SeqCst)
> + + self.files_hardlink_count.load(Ordering::SeqCst)
> + + self.files_reencoded_count.load(Ordering::SeqCst)
> + }
> +
> + pub fn files_reencoded_count(&self) -> u64 {
> + self.files_reencoded_count.load(Ordering::SeqCst)
> + }
> +
> + pub fn total_reencoded_size(&self) -> u64 {
> + self.total_reencoded_size.load(Ordering::SeqCst)
> + }
> +
> + pub fn files_reused_count(&self) -> u64 {
> + self.files_reused_count.load(Ordering::SeqCst)
> + }
> +
> + pub fn total_reused_payload_size(&self) -> u64 {
> + self.total_reused_payload_size.load(Ordering::SeqCst)
> + }
> +
> + pub fn padding(&self) -> u64 {
> + self.total_injected_size.load(Ordering::SeqCst)
> + - self.total_reused_payload_size.load(Ordering::SeqCst)
> + }
while at it, it would make sense to also implement the helpers for
loading hardlink count and especially for counter increments. That makes
the code below easier to read and improves consistency.
Further, I think the ordering constrains can be relaxed from sequential
consistency to Acquire/Release.
> }
>
> #[derive(Serialize, Deserialize)]
> @@ -325,37 +355,40 @@ where
> info!("Change detection summary:");
> info!(
> " - {} total files ({} hardlinks)",
> - archiver.progress_stats.files_reused_count
> - + archiver.progress_stats.files_reencoded_count
> - + archiver.progress_stats.files_hardlink_count,
> - archiver.progress_stats.files_hardlink_count,
> + archiver.progress_stats.total_files_count(),
> + archiver
> + .progress_stats
> + .files_hardlink_count
> + .load(Ordering::SeqCst),
> );
> info!(
> " - {} unchanged, reusable files with {} data",
> - archiver.progress_stats.files_reused_count,
> - HumanByte::from(archiver.progress_stats.total_reused_payload_size),
> + archiver.progress_stats.files_reused_count(),
> + HumanByte::from(archiver.progress_stats.total_reused_payload_size()),
> );
> info!(
> " - {} changed or non-reusable files with {} data",
> - archiver.progress_stats.files_reencoded_count,
> - HumanByte::from(archiver.progress_stats.total_reencoded_size),
> + archiver.progress_stats.files_reencoded_count(),
> + HumanByte::from(archiver.progress_stats.total_reencoded_size()),
> );
> info!(
> " - {} padding in {} partially reused chunks",
> - HumanByte::from(
> - archiver.progress_stats.total_injected_size
> - - archiver.progress_stats.total_reused_payload_size
> - ),
> - archiver.progress_stats.partial_chunks_count,
> + HumanByte::from(archiver.progress_stats.padding()),
> + archiver
> + .progress_stats
> + .partial_chunks_count
> + .load(Ordering::SeqCst),
> );
> } else {
> info!("Processing summary:");
> info!(
> - "- {} total files ({} hardlinks) with {} data",
> - archiver.progress_stats.files_reencoded_count
> - + archiver.progress_stats.files_hardlink_count,
> - archiver.progress_stats.files_hardlink_count,
> - HumanByte::from(archiver.progress_stats.total_reencoded_size),
> + " - {} total files ({} hardlinks) with {} data",
> + archiver.progress_stats.total_files_count(),
> + archiver
> + .progress_stats
> + .files_hardlink_count
> + .load(Ordering::SeqCst),
> + HumanByte::from(archiver.progress_stats.total_reencoded_size()),
> );
> }
> Ok(())
> @@ -980,24 +1013,34 @@ impl Archiver {
> }
>
> let offset: LinkOffset = if let Some(payload_offset) = payload_offset {
> - self.progress_stats.total_reused_payload_size +=
> - file_size + size_of::<pxar::format::Header>() as u64;
> - self.progress_stats.files_reused_count += 1;
> + self.progress_stats.total_reused_payload_size.fetch_add(
> + file_size + size_of::<pxar::format::Header>() as u64,
> + Ordering::SeqCst,
> + );
> + self.progress_stats
> + .files_reused_count
> + .fetch_add(1, Ordering::SeqCst);
>
> encoder
> .add_payload_ref(metadata, file_name, file_size, payload_offset)
> .await?
> } else {
> - self.progress_stats.total_reencoded_size +=
> - file_size + size_of::<pxar::format::Header>() as u64;
> - self.progress_stats.files_reencoded_count += 1;
> + self.progress_stats.total_reencoded_size.fetch_add(
> + file_size + size_of::<pxar::format::Header>() as u64,
> + Ordering::SeqCst,
> + );
> + self.progress_stats
> + .files_reencoded_count
> + .fetch_add(1, Ordering::SeqCst);
>
> self.add_regular_file(encoder, fd, file_name, metadata, file_size)
> .await?
> };
>
> if stat.st_nlink > 1 {
> - self.progress_stats.files_hardlink_count += 1;
> + self.progress_stats
> + .files_hardlink_count
> + .fetch_add(1, Ordering::SeqCst);
> self.hardlinks
> .insert(link_info, (self.path.clone(), offset));
> }
> @@ -1224,11 +1267,17 @@ impl Archiver {
> HumanByte::from(chunk.padding),
> HumanByte::from(chunk.size()),
> );
> - self.progress_stats.total_injected_size += chunk.size();
> - self.progress_stats.total_injected_count += 1;
> + self.progress_stats
> + .total_injected_size
> + .fetch_add(chunk.size(), Ordering::SeqCst);
> + self.progress_stats
> + .total_injected_count
> + .fetch_add(1, Ordering::SeqCst);
>
> if chunk.padding > 0 {
> - self.progress_stats.partial_chunks_count += 1;
> + self.progress_stats
> + .partial_chunks_count
> + .fetch_add(1, Ordering::SeqCst);
> }
>
> size = size.add(chunk.size());
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress`
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress` Shan Shaji
@ 2026-06-29 8:38 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:38 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
With the suggested rework of the logic in patch 4, this could remain as
is I think.
On 6/10/26 8:02 PM, Shan Shaji wrote:
> Earlier, `UploadCounters` was used for accounting upload stream
> progress information. Now the `PxarArchiverProgressStats` has been added
> as an optional field inside the struct, which makes the responsibility of
> struct not only just counting the progress but also to show the the pxar
> archive progress. So, renamed `UploadCounters` to `UploadProgress`.
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
> pbs-client/src/backup_stats.rs | 4 +--
> pbs-client/src/backup_writer.rs | 37 +++++++++++++-------------
> pbs-client/src/inject_reused_chunks.rs | 8 +++---
> 3 files changed, 25 insertions(+), 24 deletions(-)
>
> diff --git a/pbs-client/src/backup_stats.rs b/pbs-client/src/backup_stats.rs
> index aa5ae4889..2866c29a1 100644
> --- a/pbs-client/src/backup_stats.rs
> +++ b/pbs-client/src/backup_stats.rs
> @@ -43,7 +43,7 @@ impl UploadStats {
>
> /// Atomic counters for accounting upload stream progress information
> #[derive(Clone)]
> -pub(crate) struct UploadCounters {
> +pub(crate) struct UploadProgress {
> injected_chunk_count: Arc<AtomicUsize>,
> known_chunk_count: Arc<AtomicUsize>,
> total_chunk_count: Arc<AtomicUsize>,
> @@ -54,7 +54,7 @@ pub(crate) struct UploadCounters {
> pub pxar_progress: Option<Arc<PxarArchiverProgressStats>>,
> }
>
> -impl UploadCounters {
> +impl UploadProgress {
> /// Create and zero init new upload counters
> pub(crate) fn new(pxar_progress: Option<Arc<PxarArchiverProgressStats>>) -> Self {
> Self {
> diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
> index d0246689c..68379f725 100644
> --- a/pbs-client/src/backup_writer.rs
> +++ b/pbs-client/src/backup_writer.rs
> @@ -30,7 +30,7 @@ use proxmox_time::TimeSpan;
>
> use crate::pxar::PxarArchiverProgressStats;
>
> -use super::backup_stats::{BackupStats, UploadCounters, UploadStats};
> +use super::backup_stats::{BackupStats, UploadProgress, UploadStats};
> use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
> use super::merge_known_chunks::{MergeKnownChunks, MergedChunkInfo};
>
> @@ -330,8 +330,8 @@ impl BackupWriter {
> .as_u64()
> .unwrap();
>
> - let mut counters = UploadCounters::new(None);
> - let counters_readonly = counters.clone();
> + let mut upload_progress = UploadProgress::new(None);
> + let upload_progress_readonly = upload_progress.clone();
>
> let is_fixed_chunk_size = prefix == "fixed";
>
> @@ -343,8 +343,8 @@ impl BackupWriter {
> match merged_chunk_info {
> MergedChunkInfo::New(ref chunk_info) => {
> let chunk_len = chunk_info.chunk_len;
> - let offset =
> - counters.add_new_chunk(chunk_len as usize, chunk_info.chunk.raw_size());
> + let offset = upload_progress
> + .add_new_chunk(chunk_len as usize, chunk_info.chunk.raw_size());
> let end_offset = offset as u64 + chunk_len;
> let mut guard = index_csum.lock().unwrap();
> let csum = guard.as_mut().unwrap();
> @@ -355,7 +355,7 @@ impl BackupWriter {
> }
> MergedChunkInfo::Known(ref mut known_chunk_list) => {
> for (chunk_len, digest) in known_chunk_list {
> - let offset = counters.add_known_chunk(*chunk_len as usize);
> + let offset = upload_progress.add_known_chunk(*chunk_len as usize);
> let end_offset = offset as u64 + *chunk_len;
> let mut guard = index_csum.lock().unwrap();
> let csum = guard.as_mut().unwrap();
> @@ -379,7 +379,7 @@ impl BackupWriter {
> prefix,
> stream,
> index_csum_2,
> - counters_readonly,
> + upload_progress_readonly,
> options.is_metadata_mode,
> )
> .await?;
> @@ -780,8 +780,8 @@ impl BackupWriter {
> pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> is_metadata_mode: bool,
> ) -> impl Future<Output = Result<UploadStats, Error>> {
> - let mut counters = UploadCounters::new(pxar_archive_progress_stats);
> - let counters_readonly = counters.clone();
> + let mut upload_progress = UploadProgress::new(pxar_archive_progress_stats);
> + let upload_progress_readonly = upload_progress.clone();
>
> let is_fixed_chunk_size = prefix == "fixed";
>
> @@ -789,7 +789,7 @@ impl BackupWriter {
> let index_csum_2 = index_csum.clone();
>
> let stream = stream
> - .inject_reused_chunks(injections, counters.clone())
> + .inject_reused_chunks(injections, upload_progress.clone())
> .and_then(move |chunk_info| match chunk_info {
> InjectedChunksInfo::Known(chunks) => {
> // account for injected chunks
> @@ -797,7 +797,7 @@ impl BackupWriter {
> let mut guard = index_csum.lock().unwrap();
> let csum = guard.as_mut().unwrap();
> for chunk in chunks {
> - let offset = counters.add_injected_chunk(&chunk) as u64;
> + let offset = upload_progress.add_injected_chunk(&chunk) as u64;
> let digest = chunk.digest();
> known.push((offset, digest));
> let end_offset = offset + chunk.size();
> @@ -819,13 +819,14 @@ impl BackupWriter {
> let mut known_chunks = known_chunks.lock().unwrap();
> let digest = *chunk_builder.digest();
> let (offset, res) = if known_chunks.contains(&digest) {
> - let offset = counters.add_known_chunk(chunk_len) as u64;
> + let offset = upload_progress.add_known_chunk(chunk_len) as u64;
> (offset, MergedChunkInfo::Known(vec![(offset, digest)]))
> } else {
> match chunk_builder.build() {
> Ok((chunk, digest)) => {
> - let offset =
> - counters.add_new_chunk(chunk_len, chunk.raw_size()) as u64;
> + let offset = upload_progress
> + .add_new_chunk(chunk_len, chunk.raw_size())
> + as u64;
> known_chunks.insert(digest);
> (
> offset,
> @@ -863,7 +864,7 @@ impl BackupWriter {
> prefix,
> stream,
> index_csum_2,
> - counters_readonly,
> + upload_progress_readonly,
> is_metadata_mode,
> )
> }
> @@ -875,7 +876,7 @@ impl BackupWriter {
> prefix: &str,
> stream: impl Stream<Item = Result<MergedChunkInfo, Error>>,
> index_csum: Arc<Mutex<Option<Sha256>>>,
> - counters: UploadCounters,
> + upload_progress: UploadProgress,
> is_metadata_mode: bool,
> ) -> impl Future<Output = Result<UploadStats, Error>> {
> let append_chunk_path = format!("{prefix}_index");
> @@ -891,7 +892,7 @@ impl BackupWriter {
> || archive.ends_with(".pxar.didx")
> || archive.ends_with(".ppxar.didx")
> {
> - let counters = counters.clone();
> + let counters = upload_progress.clone();
> Some(tokio::spawn(async move {
> loop {
> tokio::time::sleep(tokio::time::Duration::from_secs(60)).await;
> @@ -998,7 +999,7 @@ impl BackupWriter {
> handle.abort();
> }
>
> - futures::future::ok(counters.to_upload_stats(csum, start_time.elapsed()))
> + futures::future::ok(upload_progress.to_upload_stats(csum, start_time.elapsed()))
> })
> }
>
> diff --git a/pbs-client/src/inject_reused_chunks.rs b/pbs-client/src/inject_reused_chunks.rs
> index c85a28a08..314dd94ce 100644
> --- a/pbs-client/src/inject_reused_chunks.rs
> +++ b/pbs-client/src/inject_reused_chunks.rs
> @@ -7,7 +7,7 @@ use anyhow::{Error, anyhow};
> use futures::{Stream, ready};
> use pin_project_lite::pin_project;
>
> -use crate::backup_stats::UploadCounters;
> +use crate::backup_stats::UploadProgress;
> use crate::pxar::create::ReusableDynamicEntry;
>
> pin_project! {
> @@ -16,7 +16,7 @@ pin_project! {
> input: S,
> next_injection: Option<InjectChunks>,
> injections: Option<mpsc::Receiver<InjectChunks>>,
> - counters: UploadCounters,
> + counters: UploadProgress,
> }
> }
>
> @@ -42,7 +42,7 @@ pub trait InjectReusedChunks: Sized {
> fn inject_reused_chunks(
> self,
> injections: Option<mpsc::Receiver<InjectChunks>>,
> - counters: UploadCounters,
> + counters: UploadProgress,
> ) -> InjectReusedChunksQueue<Self>;
> }
>
> @@ -53,7 +53,7 @@ where
> fn inject_reused_chunks(
> self,
> injections: Option<mpsc::Receiver<InjectChunks>>,
> - counters: UploadCounters,
> + counters: UploadProgress,
> ) -> InjectReusedChunksQueue<Self> {
> InjectReusedChunksQueue {
> input: self,
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs Shan Shaji
@ 2026-06-29 8:38 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:38 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
one comment inline.
On 6/10/26 8:02 PM, Shan Shaji wrote:
> earlier, there was no option to enable verbose logging inside the CLI.
> By default only info level logs were only logged.
>
> To get more information, users had to enable debug logs by setting the
> PBS_LOG environment variable, but that was too much information. To fix
> this, accept a verbose option and log only necessary information in info
> level when the flag is set as true.
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
> pbs-client/src/backup_writer.rs | 38 +++++++++++++------
> pbs-client/src/pxar/create.rs | 30 +++++++++++++--
> pbs-client/src/pxar/tools.rs | 10 ++++-
> proxmox-backup-client/src/main.rs | 11 ++++++
> .../src/proxmox_restore_daemon/api.rs | 1 +
> pxar-bin/src/main.rs | 1 +
> 6 files changed, 74 insertions(+), 17 deletions(-)
>
> diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
> index 68379f725..15a9bccde 100644
> --- a/pbs-client/src/backup_writer.rs
> +++ b/pbs-client/src/backup_writer.rs
> @@ -29,6 +29,7 @@ use proxmox_log::{Level, debug, enabled, info, trace, warn};
> use proxmox_time::TimeSpan;
>
> use crate::pxar::PxarArchiverProgressStats;
> +use crate::pxar::tools::log_message;
>
> use super::backup_stats::{BackupStats, UploadProgress, UploadStats};
> use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
> @@ -56,6 +57,7 @@ pub struct UploadOptions {
> pub encrypt: bool,
> pub index_type: IndexType,
> pub is_metadata_mode: bool,
> + pub verbose: bool,
> }
>
> /// Index type for upload options.
> @@ -525,20 +527,32 @@ impl BackupWriter {
> archive, reused, reused_percent
> );
> }
> - if enabled!(Level::DEBUG) && upload_stats.chunk_count > 0 {
> - debug!(
> - "{}: Reused {} from {} chunks.",
> - archive, upload_stats.chunk_reused, upload_stats.chunk_count
> +
> + if (options.verbose || enabled!(Level::DEBUG)) && upload_stats.chunk_count > 0 {
> + log_message(
> + &format!(
> + "{}: Reused {} from {} chunks.",
> + archive, upload_stats.chunk_reused, upload_stats.chunk_count
> + ),
> + options.verbose,
> );
> - debug!(
> - "{}: Average chunk size was {}.",
> - archive,
> - HumanByte::from(upload_stats.size / upload_stats.chunk_count)
> +
> + log_message(
> + &format!(
> + "{}: Average chunk size was {}.",
> + archive,
> + HumanByte::from(upload_stats.size / upload_stats.chunk_count)
> + ),
> + options.verbose,
> );
> - debug!(
> - "{}: Average time per request: {} microseconds.",
> - archive,
> - (upload_stats.duration.as_micros()) / (upload_stats.chunk_count as u128)
> +
> + log_message(
> + &format!(
> + "{}: Average time per request: {} microseconds.",
> + archive,
> + (upload_stats.duration.as_micros()) / (upload_stats.chunk_count as u128)
> + ),
> + options.verbose,
> );
> }
>
> diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
> index 2d2dfbb3f..85f03d662 100644
> --- a/pbs-client/src/pxar/create.rs
> +++ b/pbs-client/src/pxar/create.rs
> @@ -39,7 +39,7 @@ use crate::inject_reused_chunks::InjectChunks;
> use crate::pxar::Flags;
> use crate::pxar::look_ahead_cache::{CacheEntry, CacheEntryData, PxarLookaheadCache};
> use crate::pxar::metadata::errno_is_unsupported;
> -use crate::pxar::tools::assert_single_path_component;
> +use crate::pxar::tools::{assert_single_path_component, log_message};
>
> const CHUNK_PADDING_THRESHOLD: f64 = 0.1;
>
> @@ -60,6 +60,8 @@ pub struct PxarCreateOptions {
> pub previous_ref: Option<PxarPrevRef>,
> /// Maximum number of lookahead cache entries
> pub max_cache_size: Option<usize>,
> + /// Enable verbose loging
> + pub verbose: bool,
> }
>
> pub type MetadataArchiveReader = Arc<dyn ReadAt + Send + Sync + 'static>;
> @@ -225,6 +227,7 @@ struct Archiver {
> last_reusable_offset: Option<u64>,
> progress_stats: Arc<PxarArchiverProgressStats>,
> split_archive: bool,
> + verbose: bool,
> }
>
> type Encoder<'a, T> = pxar::encoder::aio::Encoder<'a, T>;
> @@ -337,6 +340,7 @@ where
> last_reusable_offset: None,
> progress_stats: archiver_progress_stats.unwrap_or_default(),
> split_archive,
> + verbose: options.verbose,
> };
>
> archiver
> @@ -511,18 +515,35 @@ impl Archiver {
> }
> let range =
> *offset..*offset + size + size_of::<pxar::format::Header>() as u64;
> +
> + if self.verbose {
> + info!("reusable: {file_name:?}");
> + }
> +
> debug!(
> "reusable: {file_name:?} at range {range:?} has unchanged metadata."
> );
> return Ok(Some(range));
> }
> - debug!("re-encode: {file_name:?} not a regular file.");
> +
> + log_message(
> + &format!("re-encode: {file_name:?} not a regular file."),
> + self.verbose,
> + );
> return Ok(None);
> }
> - debug!("re-encode: {file_name:?} metadata did not match.");
> +
> + log_message(
> + &format!("re-encode: {file_name:?} metadata did not match."),
> + self.verbose,
> + );
> return Ok(None);
> }
> - debug!("re-encode: {file_name:?} not found in previous archive.");
> +
> + log_message(
> + &format!("re-encode: {file_name:?} not found in previous archive."),
> + self.verbose,
> + );
> }
>
> Ok(None)
> @@ -2136,6 +2157,7 @@ mod tests {
> last_reusable_offset: None,
> progress_stats: Arc::new(PxarArchiverProgressStats::default()),
> split_archive: true,
> + verbose: false,
> };
>
> let accessor = Accessor::new(pxar::PxarVariant::Unified(reader), metadata_size)
> diff --git a/pbs-client/src/pxar/tools.rs b/pbs-client/src/pxar/tools.rs
> index 0c8c2927e..67d6a740b 100644
> --- a/pbs-client/src/pxar/tools.rs
> +++ b/pbs-client/src/pxar/tools.rs
> @@ -21,7 +21,7 @@ use pbs_datastore::BackupManifest;
> use pbs_datastore::dynamic_index::{BufferedDynamicReader, LocalDynamicReadAt};
> use pbs_datastore::index::IndexFile;
> use pbs_tools::crypt_config::CryptConfig;
> -use proxmox_log::{debug, info};
> +use proxmox_log::{Level, debug, enabled, info};
>
> use crate::{BackupReader, RemoteChunkReader};
>
> @@ -485,3 +485,11 @@ pub async fn pxar_metadata_catalog_find<'future, T: Clone + Send + Sync + ReadAt
> }
> Ok(())
> }
> +
> +pub(crate) fn log_message(msg: &str, verbose: bool) {
> + if verbose && !enabled!(Level::DEBUG) {
> + info!("{msg}");
> + } else {
> + debug!("{msg}");
> + }
> +}
comment: I do not think this should switch log levels based on the
presence/absence of the verbose flag. Rahter, this should bump the log
level for the message and hide it behind the flag? And set the flag when
the debug level is enabled?
> diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
> index 1036ec7e9..52bdbe798 100644
> --- a/proxmox-backup-client/src/main.rs
> +++ b/proxmox-backup-client/src/main.rs
> @@ -805,6 +805,12 @@ fn spawn_catalog_upload(
> optional: true,
> default: false,
> },
> + "verbose": {
> + type: Boolean,
> + description: "Enable verbose logs.",
> + optional: true,
> + default: false,
> + }
> }
> }
> )]
> @@ -848,6 +854,8 @@ async fn create_backup(
>
> let include_dev = param["include-dev"].as_array();
>
> + let verbose = param["verbose"].as_bool().unwrap_or_default();
> +
> let entries_max = param["entries-max"]
> .as_u64()
> .unwrap_or(pbs_client::pxar::ENCODER_MAX_ENTRIES as u64);
> @@ -1245,6 +1253,7 @@ async fn create_backup(
> skip_e2big_xattr,
> previous_ref,
> max_cache_size,
> + verbose: verbose,
> };
>
> let upload_options = UploadOptions {
> @@ -1252,6 +1261,7 @@ async fn create_backup(
> compress: true,
> encrypt: crypto.mode == CryptMode::Encrypt,
> is_metadata_mode: detection_mode.is_metadata(),
> + verbose,
> ..UploadOptions::default()
> };
>
> @@ -1292,6 +1302,7 @@ async fn create_backup(
> compress: true,
> encrypt: crypto.mode == CryptMode::Encrypt,
> is_metadata_mode: false,
> + verbose,
> };
>
> let stats =
> diff --git a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> index b7f4fd141..e890681fc 100644
> --- a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> +++ b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> @@ -364,6 +364,7 @@ fn extract(
> skip_e2big_xattr: false,
> previous_ref: None,
> max_cache_size: None,
> + verbose: false
> };
>
> let pxar_writer = pxar::PxarVariant::Unified(TokioWriter::new(writer));
> diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
> index 953c6e756..385f54a0d 100644
> --- a/pxar-bin/src/main.rs
> +++ b/pxar-bin/src/main.rs
> @@ -383,6 +383,7 @@ async fn create_archive(
> skip_e2big_xattr: false,
> previous_ref: None,
> max_cache_size: None,
> + verbose: false,
> };
>
> let source = PathBuf::from(source);
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup Shan Shaji
@ 2026-06-29 8:38 ` Christian Ebner
0 siblings, 0 replies; 13+ messages in thread
From: Christian Ebner @ 2026-06-29 8:38 UTC (permalink / raw)
To: Shan Shaji, pbs-devel
Unfortunately passing the pxar progress state along to the upload stream
for logging via the UploadOptions does not lead to nice code here. I
think this needs revisiting.
What if we refactor the progress logging logic to be passed to the
upload stream via e.g. an optional callback method provided via the
UploadOptions?
By this we would have a cleaner interface and no need to mix pxar
archiver related counters with the upload counters AFAICT.
Maybe something along the lines of (untested and incomplete):
diff --git a/pbs-client/src/backup_writer.rs
b/pbs-client/src/backup_writer.rs
index d0246689c..fe48a3f0e 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -48,6 +48,8 @@ impl Drop for BackupWriter {
}
}
+type ProgressLoggerCallback = Box<dyn Fn(&UploadCounters) + Send + Sync
+ 'static>;
+
/// Options for uploading blobs/streams to the server
#[derive(Default, Clone)]
pub struct UploadOptions {
@@ -55,7 +57,7 @@ pub struct UploadOptions {
pub compress: bool,
pub encrypt: bool,
pub index_type: IndexType,
- pub is_metadata_mode: bool,
+ pub(crate) progress_logging_callback:
Option<Arc<ProgressLoggerCallback>>,
}
/// Index type for upload options.
@@ -375,12 +377,11 @@ impl BackupWriter {
let upload_stats = Self::upload_merged_chunk_stream(
self.h2.clone(),
wid,
- archive_name,
prefix,
stream,
index_csum_2,
counters_readonly,
- options.is_metadata_mode,
+ options.progress_logging_callback,
)
.await?;
@@ -482,9 +483,8 @@ impl BackupWriter {
},
options.compress,
injections,
- archive_name,
pxar_archive_progress_stats,
- options.is_metadata_mode,
+ options.progress_logging_callback,
)
.await?;
@@ -776,9 +776,8 @@ impl BackupWriter {
crypt_config: Option<Arc<CryptConfig>>,
compress: bool,
injections: Option<std::sync::mpsc::Receiver<InjectChunks>>,
- archive: &BackupArchiveName,
pxar_archive_progress_stats:
Option<Arc<PxarArchiverProgressStats>>,
- is_metadata_mode: bool,
+ progress_logging_callback: Option<Arc<ProgressLoggerCallback>>,
) -> impl Future<Output = Result<UploadStats, Error>> {
let mut counters =
UploadCounters::new(pxar_archive_progress_stats);
let counters_readonly = counters.clone();
@@ -859,24 +858,22 @@ impl BackupWriter {
Self::upload_merged_chunk_stream(
h2,
wid,
- archive,
prefix,
stream,
index_csum_2,
counters_readonly,
- is_metadata_mode,
+ progress_logging_callback,
)
}
fn upload_merged_chunk_stream(
h2: H2Client,
wid: u64,
- archive: &BackupArchiveName,
prefix: &str,
stream: impl Stream<Item = Result<MergedChunkInfo, Error>>,
index_csum: Arc<Mutex<Option<Sha256>>>,
counters: UploadCounters,
- is_metadata_mode: bool,
+ progress_logging_callback: Option<Arc<ProgressLoggerCallback>>,
) -> impl Future<Output = Result<UploadStats, Error>> {
let append_chunk_path = format!("{prefix}_index");
let upload_chunk_path = format!("{prefix}_chunk");
@@ -887,42 +884,13 @@ impl BackupWriter {
let (upload_queue, upload_result) =
Self::append_chunk_queue(h2.clone(), wid,
append_chunk_path, uploaded_len.clone());
- let progress_handle = if archive.ends_with(".img.fidx")
- || archive.ends_with(".pxar.didx")
- || archive.ends_with(".ppxar.didx")
- {
+ let progress_handle = if let Some(callback) =
progress_logging_callback {
let counters = counters.clone();
+ let callback = Arc::clone(&callback);
+
Some(tokio::spawn(async move {
loop {
- tokio::time::sleep(tokio::time::Duration::from_secs(60)).await;
-
- let size =
HumanByte::from(counters.total_stream_len());
- let size_uploaded =
HumanByte::from(uploaded_len.load(Ordering::SeqCst));
- let elapsed = TimeSpan::from(start_time.elapsed());
-
- if let Some(pxar_progress) = &counters.pxar_progress {
- let n_files = pxar_progress.total_files_count();
- let reuse_payload_size =
pxar_progress.total_reused_payload_size();
- let total_of_logical_file_size = HumanByte::from(
- pxar_progress.total_reencoded_size() +
reuse_payload_size,
- );
- let total_reuse_payload_size =
HumanByte::from(reuse_payload_size);
- let reuse_file_count =
pxar_progress.files_reused_count();
-
- if is_metadata_mode {
- info!(
- "scanned {n_files} files with
{total_of_logical_file_size} (reusing {reuse_file_count} files with \
- {total_reuse_payload_size}) of which
processed {size} in {elapsed} and uploaded {size_uploaded}"
- );
- } else {
- info!(
- "scanned {n_files} files with
{total_of_logical_file_size} of which processed {size} \
- in {elapsed} and uploaded {size_uploaded}"
- );
- }
- } else {
- info!("processed {size} in {elapsed}, uploaded
{size_uploaded}");
- }
+ callback(&counters);
}
}))
} else {
On 6/10/26 8:02 PM, Shan Shaji wrote:
> earlier, the progress logs only reported the processed stream size and
> uploaded size. To improve this, `PxarArchiverProgressStats` is now
> created earlier and shared between the archiver and the upload stream.
>
> Signed-off-by: Shan Shaji <s.shaji@proxmox.com>
> ---
> pbs-client/src/backup_stats.rs | 5 ++-
> pbs-client/src/backup_writer.rs | 39 +++++++++++++++++--
> pbs-client/src/pxar/create.rs | 9 +++--
> pbs-client/src/pxar/mod.rs | 3 +-
> pbs-client/src/pxar_backup_stream.rs | 13 ++++++-
> proxmox-backup-client/src/main.rs | 27 +++++++++++--
> .../src/proxmox_restore_daemon/api.rs | 1 +
> pxar-bin/src/main.rs | 1 +
> tests/catar.rs | 1 +
> 9 files changed, 85 insertions(+), 14 deletions(-)
>
> diff --git a/pbs-client/src/backup_stats.rs b/pbs-client/src/backup_stats.rs
> index d7c13784b..aa5ae4889 100644
> --- a/pbs-client/src/backup_stats.rs
> +++ b/pbs-client/src/backup_stats.rs
> @@ -4,6 +4,7 @@ use std::sync::Arc;
> use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
> use std::time::Duration;
>
> +use crate::pxar::PxarArchiverProgressStats;
> use crate::pxar::create::ReusableDynamicEntry;
>
> /// Basic backup run statistics and archive checksum
> @@ -50,11 +51,12 @@ pub(crate) struct UploadCounters {
> injected_stream_len: Arc<AtomicUsize>,
> reused_stream_len: Arc<AtomicUsize>,
> total_stream_len: Arc<AtomicUsize>,
> + pub pxar_progress: Option<Arc<PxarArchiverProgressStats>>,
no need for this to be pub, the UploadCounters are pub(crate) only anyways.
> }
>
> impl UploadCounters {
> /// Create and zero init new upload counters
> - pub(crate) fn new() -> Self {
> + pub(crate) fn new(pxar_progress: Option<Arc<PxarArchiverProgressStats>>) -> Self {
> Self {
> total_chunk_count: Arc::new(AtomicUsize::new(0)),
> injected_chunk_count: Arc::new(AtomicUsize::new(0)),
> @@ -63,6 +65,7 @@ impl UploadCounters {
> injected_stream_len: Arc::new(AtomicUsize::new(0)),
> reused_stream_len: Arc::new(AtomicUsize::new(0)),
> total_stream_len: Arc::new(AtomicUsize::new(0)),
> + pxar_progress: pxar_progress,
nit: can be a simplified s/pxar_progress: pxar_progress/pxar_progress/
> }
> }
>
> diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
> index f0744ecaf..d0246689c 100644
> --- a/pbs-client/src/backup_writer.rs
> +++ b/pbs-client/src/backup_writer.rs
> @@ -28,6 +28,8 @@ use proxmox_human_byte::HumanByte;
> use proxmox_log::{Level, debug, enabled, info, trace, warn};
> use proxmox_time::TimeSpan;
>
> +use crate::pxar::PxarArchiverProgressStats;
> +
> use super::backup_stats::{BackupStats, UploadCounters, UploadStats};
> use super::inject_reused_chunks::{InjectChunks, InjectReusedChunks, InjectedChunksInfo};
> use super::merge_known_chunks::{MergeKnownChunks, MergedChunkInfo};
> @@ -53,6 +55,7 @@ pub struct UploadOptions {
> pub compress: bool,
> pub encrypt: bool,
> pub index_type: IndexType,
> + pub is_metadata_mode: bool,
> }
>
> /// Index type for upload options.
> @@ -327,7 +330,7 @@ impl BackupWriter {
> .as_u64()
> .unwrap();
>
> - let mut counters = UploadCounters::new();
> + let mut counters = UploadCounters::new(None);
> let counters_readonly = counters.clone();
>
> let is_fixed_chunk_size = prefix == "fixed";
> @@ -377,6 +380,7 @@ impl BackupWriter {
> stream,
> index_csum_2,
> counters_readonly,
> + options.is_metadata_mode,
> )
> .await?;
>
> @@ -400,6 +404,7 @@ impl BackupWriter {
> stream: impl Stream<Item = Result<bytes::BytesMut, Error>>,
> options: UploadOptions,
> injections: Option<std::sync::mpsc::Receiver<InjectChunks>>,
> + pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> ) -> Result<BackupStats, Error> {
> let known_chunks = Arc::new(Mutex::new(HashSet::new()));
>
> @@ -478,6 +483,8 @@ impl BackupWriter {
> options.compress,
> injections,
> archive_name,
> + pxar_archive_progress_stats,
> + options.is_metadata_mode,
> )
> .await?;
>
> @@ -770,8 +777,10 @@ impl BackupWriter {
> compress: bool,
> injections: Option<std::sync::mpsc::Receiver<InjectChunks>>,
> archive: &BackupArchiveName,
> + pxar_archive_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> + is_metadata_mode: bool,
> ) -> impl Future<Output = Result<UploadStats, Error>> {
> - let mut counters = UploadCounters::new();
> + let mut counters = UploadCounters::new(pxar_archive_progress_stats);
> let counters_readonly = counters.clone();
>
> let is_fixed_chunk_size = prefix == "fixed";
> @@ -855,6 +864,7 @@ impl BackupWriter {
> stream,
> index_csum_2,
> counters_readonly,
> + is_metadata_mode,
> )
> }
>
> @@ -866,6 +876,7 @@ impl BackupWriter {
> stream: impl Stream<Item = Result<MergedChunkInfo, Error>>,
> index_csum: Arc<Mutex<Option<Sha256>>>,
> counters: UploadCounters,
> + is_metadata_mode: bool,
> ) -> impl Future<Output = Result<UploadStats, Error>> {
> let append_chunk_path = format!("{prefix}_index");
> let upload_chunk_path = format!("{prefix}_chunk");
> @@ -889,7 +900,29 @@ impl BackupWriter {
> let size_uploaded = HumanByte::from(uploaded_len.load(Ordering::SeqCst));
> let elapsed = TimeSpan::from(start_time.elapsed());
>
> - info!("processed {size} in {elapsed}, uploaded {size_uploaded}");
> + if let Some(pxar_progress) = &counters.pxar_progress {
> + let n_files = pxar_progress.total_files_count();
> + let reuse_payload_size = pxar_progress.total_reused_payload_size();
> + let total_of_logical_file_size = HumanByte::from(
> + pxar_progress.total_reencoded_size() + reuse_payload_size,
> + );
> + let total_reuse_payload_size = HumanByte::from(reuse_payload_size);
> + let reuse_file_count = pxar_progress.files_reused_count();
> +
> + if is_metadata_mode {
> + info!(
> + "scanned {n_files} files with {total_of_logical_file_size} (reusing {reuse_file_count} files with \
> + {total_reuse_payload_size}) of which processed {size} in {elapsed} and uploaded {size_uploaded}"
> + );
> + } else {
> + info!(
> + "scanned {n_files} files with {total_of_logical_file_size} of which processed {size} \
> + in {elapsed} and uploaded {size_uploaded}"
> + );
> + }
> + } else {
> + info!("processed {size} in {elapsed}, uploaded {size_uploaded}");
> + }
> }
> }))
> } else {
> diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs
> index 1246f67d9..2d2dfbb3f 100644
> --- a/pbs-client/src/pxar/create.rs
> +++ b/pbs-client/src/pxar/create.rs
> @@ -151,7 +151,7 @@ pub(crate) struct HardLinkInfo {
> }
>
> #[derive(Default)]
> -struct PxarArchiverProgressStats {
> +pub struct PxarArchiverProgressStats {
> files_reused_count: AtomicU64,
> files_hardlink_count: AtomicU64,
> files_reencoded_count: AtomicU64,
> @@ -223,7 +223,7 @@ struct Archiver {
> // an older client without the encode-time check is detected here and the affected file is
> // re-encoded instead of reused.
> last_reusable_offset: Option<u64>,
> - progress_stats: PxarArchiverProgressStats,
> + progress_stats: Arc<PxarArchiverProgressStats>,
> split_archive: bool,
> }
>
> @@ -251,6 +251,7 @@ pub async fn create_archive<T, F>(
> options: PxarCreateOptions,
> forced_boundaries: Option<mpsc::Sender<InjectChunks>>,
> suggested_boundaries: Option<mpsc::Sender<u64>>,
> + archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> ) -> Result<(), Error>
> where
> T: SeqWrite + Send,
> @@ -334,7 +335,7 @@ where
> previous_payload_index,
> cache: PxarLookaheadCache::new(options.max_cache_size),
> last_reusable_offset: None,
> - progress_stats: PxarArchiverProgressStats::default(),
> + progress_stats: archiver_progress_stats.unwrap_or_default(),
> split_archive,
> };
>
> @@ -2133,7 +2134,7 @@ mod tests {
> suggested_boundaries: Some(suggested_boundaries),
> cache: PxarLookaheadCache::new(None),
> last_reusable_offset: None,
> - progress_stats: PxarArchiverProgressStats::default(),
> + progress_stats: Arc::new(PxarArchiverProgressStats::default()),
> split_archive: true,
> };
>
> diff --git a/pbs-client/src/pxar/mod.rs b/pbs-client/src/pxar/mod.rs
> index b88fac33f..d6820d01a 100644
> --- a/pbs-client/src/pxar/mod.rs
> +++ b/pbs-client/src/pxar/mod.rs
> @@ -58,7 +58,8 @@ mod flags;
> pub use flags::Flags;
>
> pub use create::{
> - MetadataArchiveReader, PxarCreateOptions, PxarPrevRef, PxarWriters, create_archive,
> + MetadataArchiveReader, PxarArchiverProgressStats, PxarCreateOptions, PxarPrevRef, PxarWriters,
> + create_archive,
> };
> pub use extract::{
> ErrorHandler, OverwriteFlags, PxarExtractContext, PxarExtractOptions, create_tar, create_zip,
> diff --git a/pbs-client/src/pxar_backup_stream.rs b/pbs-client/src/pxar_backup_stream.rs
> index 430eee01f..d1b46cf8a 100644
> --- a/pbs-client/src/pxar_backup_stream.rs
> +++ b/pbs-client/src/pxar_backup_stream.rs
> @@ -20,6 +20,7 @@ use proxmox_log::debug;
> use pbs_datastore::catalog::{BackupCatalogWriter, CatalogWriter};
>
> use crate::inject_reused_chunks::InjectChunks;
> +use crate::pxar::PxarArchiverProgressStats;
> use crate::pxar::create::PxarWriters;
>
> /// Stream implementation to encode and upload .pxar archives.
> @@ -50,6 +51,7 @@ impl PxarBackupStream {
> options: crate::pxar::PxarCreateOptions,
> boundaries: Option<mpsc::Sender<InjectChunks>>,
> separate_payload_stream: bool,
> + archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> ) -> Result<(Self, Option<Self>), Error> {
> let buffer_size = 256 * 1024;
>
> @@ -102,6 +104,7 @@ impl PxarBackupStream {
> options,
> boundaries,
> suggested_boundaries_tx,
> + archiver_progress_stats,
> )
> .await
> {
> @@ -145,10 +148,18 @@ impl PxarBackupStream {
> options: crate::pxar::PxarCreateOptions,
> boundaries: Option<mpsc::Sender<InjectChunks>>,
> separate_payload_stream: bool,
> + archiver_progress_stats: Option<Arc<PxarArchiverProgressStats>>,
> ) -> Result<(Self, Option<Self>), Error> {
> let dir = nix::dir::Dir::open(dirname, OFlag::O_DIRECTORY, Mode::empty())?;
>
> - Self::new(dir, catalog, options, boundaries, separate_payload_stream)
> + Self::new(
> + dir,
> + catalog,
> + options,
> + boundaries,
> + separate_payload_stream,
> + archiver_progress_stats,
> + )
> }
> }
>
> diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
> index 2680add78..1036ec7e9 100644
> --- a/proxmox-backup-client/src/main.rs
> +++ b/proxmox-backup-client/src/main.rs
> @@ -31,7 +31,9 @@ use pbs_api_types::{
> RateLimitConfig, ServerIdentity, SnapshotListItem, StorageStatus,
> };
> use pbs_client::catalog_shell::Shell;
> -use pbs_client::pxar::{ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarPrevRef};
> +use pbs_client::pxar::{
> + ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarArchiverProgressStats, PxarPrevRef,
> +};
> use pbs_client::tools::{
> CHUNK_SIZE_SCHEMA, complete_archive_name, complete_auth_id, complete_backup_group,
> complete_backup_snapshot, complete_backup_source, complete_chunk_size,
> @@ -268,6 +270,7 @@ async fn backup_directory<P: AsRef<Path>>(
> bail!("cannot backup directory with fixed chunk size!");
> }
>
> + let pxar_archiver_progress_stats = Arc::new(PxarArchiverProgressStats::default());
> let (payload_boundaries_tx, payload_boundaries_rx) = std::sync::mpsc::channel();
> let (pxar_stream, payload_stream) = PxarBackupStream::open(
> dir_path.as_ref(),
> @@ -275,6 +278,7 @@ async fn backup_directory<P: AsRef<Path>>(
> pxar_create_options,
> Some(payload_boundaries_tx),
> payload_target.is_some(),
> + Some(pxar_archiver_progress_stats.clone()),
> )?;
>
> let mut chunk_stream = ChunkStream::new(pxar_stream, chunk_size, None, None);
> @@ -289,7 +293,13 @@ async fn backup_directory<P: AsRef<Path>>(
> }
> });
>
> - let stats = client.upload_stream(archive_name, stream, upload_options.clone(), None);
> + let stats = client.upload_stream(
> + archive_name,
> + stream,
> + upload_options.clone(),
> + None,
> + Some(pxar_archiver_progress_stats.clone()),
> + );
>
> if let Some(mut payload_stream) = payload_stream {
> let payload_target = payload_target
> @@ -319,6 +329,7 @@ async fn backup_directory<P: AsRef<Path>>(
> stream,
> upload_options,
> Some(payload_injections_rx),
> + Some(pxar_archiver_progress_stats),
> );
>
> match futures::join!(stats, payload_stats) {
> @@ -359,7 +370,7 @@ async fn backup_image<P: AsRef<Path>>(
> }
>
> let stats = client
> - .upload_stream(archive_name, stream, upload_options, None)
> + .upload_stream(archive_name, stream, upload_options, None, None)
> .await?;
>
> Ok(stats)
> @@ -652,7 +663,13 @@ fn spawn_catalog_upload(
>
> tokio::spawn(async move {
> let catalog_upload_result = client
> - .upload_stream(&CATALOG_NAME, catalog_chunk_stream, upload_options, None)
> + .upload_stream(
> + &CATALOG_NAME,
> + catalog_chunk_stream,
> + upload_options,
> + None,
> + None,
> + )
> .await;
>
> if let Err(ref err) = catalog_upload_result {
> @@ -1234,6 +1251,7 @@ async fn create_backup(
> previous_manifest: previous_manifest.clone(),
> compress: true,
> encrypt: crypto.mode == CryptMode::Encrypt,
> + is_metadata_mode: detection_mode.is_metadata(),
> ..UploadOptions::default()
> };
>
> @@ -1273,6 +1291,7 @@ async fn create_backup(
> index_type: IndexType::Fixed(image_file_size),
> compress: true,
> encrypt: crypto.mode == CryptMode::Encrypt,
> + is_metadata_mode: false,
> };
>
> let stats =
> diff --git a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> index ff21c9bee..b7f4fd141 100644
> --- a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> +++ b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
> @@ -375,6 +375,7 @@ fn extract(
> options,
> None,
> None,
> + None
> )
> .await
> }
> diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
> index 4b14507ca..953c6e756 100644
> --- a/pxar-bin/src/main.rs
> +++ b/pxar-bin/src/main.rs
> @@ -450,6 +450,7 @@ async fn create_archive(
> options,
> None,
> None,
> + None,
> )
> .await?;
>
> diff --git a/tests/catar.rs b/tests/catar.rs
> index aed23f866..81ca9d8db 100644
> --- a/tests/catar.rs
> +++ b/tests/catar.rs
> @@ -41,6 +41,7 @@ fn run_test(dir_name: &str) -> Result<(), Error> {
> options,
> None,
> None,
> + None,
> ))?;
>
> Command::new("cmp")
^ permalink raw reply related [flat|nested] 13+ messages in thread
end of thread, other threads:[~2026-06-29 8:38 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-10 18:02 [RFC PATCH proxmox-backup 0/6] fix #7024: client: add file statistics inside logs Shan Shaji
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 1/6] refactor: rename `ReuseStats` struct to `PxarArchiverProgressStats` Shan Shaji
2026-06-29 8:37 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 2/6] fix #7024: pxar: show archiver summary for legacy and data mode Shan Shaji
2026-06-29 8:37 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 3/6] refactor: use atomic values in `PxarArchiverProgressStats` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 4/6] fix #7024: cli: show number of processed files during backup Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 5/6] refactor: rename `UploadCounters` to `UploadProgress` Shan Shaji
2026-06-29 8:38 ` Christian Ebner
2026-06-10 18:02 ` [RFC PATCH proxmox-backup 6/6] fix #7024: cli: add option to enable verbose logs Shan Shaji
2026-06-29 8:38 ` Christian Ebner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox