From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id E19901FF13B for ; Wed, 22 Apr 2026 19:25:08 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A2016264A2; Wed, 22 Apr 2026 19:25:08 +0200 (CEST) Message-ID: <65e4afed-1561-4b27-b44b-ed72678feb15@proxmox.com> Date: Wed, 22 Apr 2026 19:25:04 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH proxmox-backup v8 02/10] tools: implement buffered logger for concurrent log messages To: Christian Ebner , pbs-devel@lists.proxmox.com References: <20260422131820.769620-1-c.ebner@proxmox.com> <20260422131820.769620-3-c.ebner@proxmox.com> Content-Language: en-US From: Thomas Lamprecht In-Reply-To: <20260422131820.769620-3-c.ebner@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776878616838 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.002 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [lib.rs] Message-ID-Hash: XAAWBPWJLVHDQHYI7FEKPHSWP2D2SG62 X-Message-ID-Hash: XAAWBPWJLVHDQHYI7FEKPHSWP2D2SG62 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 22.04.26 um 15:17 schrieb Christian Ebner: > The BufferedLogger instance collects messages send from different > sender instances via an async tokio channel and buffers them. Sender > identify by label and provide a log level for each log line to be > buffered and flushed. > > On collection, log lines are grouped by label and buffered in > sequence of arrival per label, up to the configured maximum number of > per group lines. In addition, the last logged timestamp is updated, > which will be used to periodically flush log lines per-label with the > configured interval. There is no guarantee on the order of labels when > flushing. > > In addition, senders can request flushing at any given point, all logs > are flushed when closing the sender builder. > > Log output is written based on provided log line level and prefixed > by the label. > > Co-developed-by: Fabian Grünbichler > Signed-off-by: Christian Ebner > --- > pbs-tools/Cargo.toml | 3 + > pbs-tools/src/buffered_logger.rs | 238 +++++++++++++++++++++++++++++++ > pbs-tools/src/lib.rs | 1 + > 3 files changed, 242 insertions(+) > create mode 100644 pbs-tools/src/buffered_logger.rs > > diff --git a/pbs-tools/Cargo.toml b/pbs-tools/Cargo.toml > index 998e3077e..8bc01e85d 100644 > --- a/pbs-tools/Cargo.toml > +++ b/pbs-tools/Cargo.toml > @@ -8,6 +8,7 @@ description = "common tools used throughout pbs" > # This must not depend on any subcrates more closely related to pbs itself. > [dependencies] > anyhow.workspace = true > +async-trait.workspace = true > bytes.workspace = true > foreign-types.workspace = true > hex.workspace = true > @@ -17,10 +18,12 @@ openssl.workspace = true > serde_json.workspace = true > # rt-multi-thread is required for block_in_place > tokio = { workspace = true, features = [ "fs", "io-util", "rt", "rt-multi-thread", "sync" ] } > +tracing.workspace = true > > proxmox-async.workspace = true > proxmox-io = { workspace = true, features = [ "tokio" ] } > proxmox-human-byte.workspace = true > +proxmox-log.workspace = true > proxmox-sys.workspace = true > proxmox-time.workspace = true > > diff --git a/pbs-tools/src/buffered_logger.rs b/pbs-tools/src/buffered_logger.rs > new file mode 100644 > index 000000000..4e9bf8651 > --- /dev/null > +++ b/pbs-tools/src/buffered_logger.rs > @@ -0,0 +1,238 @@ > +//! Log aggregator to collect and group messages sent from concurrent tasks via > +//! a tokio channel. fine for now to introduce this here, especially given the lower friction for fixes and improvements, which are also more likely to happen initially, but in the longer term this might be better of in it's own micro-crate or proxmox-log as opt-in feature. As while the pbs-tools workspace crate can be fine as staging ground and maybe for some stuff that's very specific to PBS, but in general I'd like to see most of it's (somewhat generic) code to move into the common proxmox.git workspace in some form - existing crate or new (micro) crate.