Re: [PATCH proxmox-backup v8 02/10] tools: implement buffered logger for concurrent log messages

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 22.04.26 um 15:17 schrieb Christian Ebner:
> The BufferedLogger instance collects messages send from different
> sender instances via an async tokio channel and buffers them. Sender
> identify by label and provide a log level for each log line to be
> buffered and flushed.
> 
> On collection, log lines are grouped by label and buffered in
> sequence of arrival per label, up to the configured maximum number of
> per group lines. In addition, the last logged timestamp is updated,
> which will be used to periodically flush log lines per-label with the
> configured interval. There is no guarantee on the order of labels when
> flushing.
> 
> In addition, senders can request flushing at any given point, all logs
> are flushed when closing the sender builder.
> 
> Log output is written based on provided log line level and prefixed
> by the label.
>
> Co-developed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
>  pbs-tools/Cargo.toml             |   3 +
>  pbs-tools/src/buffered_logger.rs | 238 +++++++++++++++++++++++++++++++
>  pbs-tools/src/lib.rs             |   1 +
>  3 files changed, 242 insertions(+)
>  create mode 100644 pbs-tools/src/buffered_logger.rs
> 
> diff --git a/pbs-tools/Cargo.toml b/pbs-tools/Cargo.toml
> index 998e3077e..8bc01e85d 100644
> --- a/pbs-tools/Cargo.toml
> +++ b/pbs-tools/Cargo.toml
> @@ -8,6 +8,7 @@ description = "common tools used throughout pbs"
>  # This must not depend on any subcrates more closely related to pbs itself.
>  [dependencies]
>  anyhow.workspace = true
> +async-trait.workspace = true
>  bytes.workspace = true
>  foreign-types.workspace = true
>  hex.workspace = true
> @@ -17,10 +18,12 @@ openssl.workspace = true
>  serde_json.workspace = true
>  # rt-multi-thread is required for block_in_place
>  tokio = { workspace = true, features = [ "fs", "io-util", "rt", "rt-multi-thread", "sync" ] }
> +tracing.workspace = true
>  
>  proxmox-async.workspace = true
>  proxmox-io = { workspace = true, features = [ "tokio" ] }
>  proxmox-human-byte.workspace = true
> +proxmox-log.workspace = true
>  proxmox-sys.workspace = true
>  proxmox-time.workspace = true
>  
> diff --git a/pbs-tools/src/buffered_logger.rs b/pbs-tools/src/buffered_logger.rs
> new file mode 100644
> index 000000000..4e9bf8651
> --- /dev/null
> +++ b/pbs-tools/src/buffered_logger.rs
> @@ -0,0 +1,238 @@
> +//! Log aggregator to collect and group messages sent from concurrent tasks via
> +//! a tokio channel.

fine for now to introduce this here, especially given the lower friction
for fixes and improvements, which are also more likely to happen initially,
but in the longer term this might be better of in it's own micro-crate or
proxmox-log as opt-in feature.

As while the pbs-tools workspace crate can be fine as staging ground and
maybe for some stuff that's very specific to PBS, but in general I'd like
to see most of it's (somewhat generic) code to move into the common
proxmox.git workspace in some form - existing crate or new (micro) crate.