Re: [pbs-devel] [PATCH proxmox v2 1/4] log: rename/move init functions

[Wolfgang Bumiller <w.bumiller@proxmox.com>]

On Mon, Feb 17, 2025 at 03:12:23PM +0100, Gabriel Goller wrote:
> On 17.02.2025 14:38, Wolfgang Bumiller wrote:
> > On Mon, Feb 17, 2025 at 02:08:03PM +0100, Gabriel Goller wrote:
> > > On 11.02.2025 10:28, Wolfgang Bumiller wrote:
> > > > On Tue, Feb 11, 2025 at 10:22:44AM +0100, Wolfgang Bumiller wrote:
> > > > > On Mon, Feb 10, 2025 at 05:42:35PM +0100, Gabriel Goller wrote:
> > > > > > On 10.02.2025 15:37, Wolfgang Bumiller wrote:
> > > > > > > On Mon, Dec 09, 2024 at 11:46:03AM +0100, Gabriel Goller wrote:
> > > > > > > > +/// Inits a new tracing logger that prints to stderr or tasklog with the logging level specified in the
> > > > > > > > +/// environment variable `env_var`.
> > > > > > > > +///
> > > > > > > > +/// This logger is task-aware, which means if we are in a PBS task, we will retrieve the task-file
> > > > > > > > +/// and write to it. We'll only write to stderr if we are not in a task. If `env_var` doesn't exist
> > > > > > > > +/// or can't be read, use the `default_log_level`. The output will be very plain: no ansi, no
> > > > > > > > +/// timestamp, no level, just the message and it's
> > > > > > > > +/// fields.
> > > > > > > > +pub fn stderr_or_tasklog(
> > > > > > > > +    env_var: &str,
> > > > > > > > +    default_log_level: LevelFilter,
> > > > > > > > +) -> Result<(), anyhow::Error> {
> > > > > > > > +    let log_level = get_env_variable(env_var, default_log_level);
> > > > > > > > +
> > > > > > > > +    let registry = tracing_subscriber::registry()
> > > > > > > > +        .with(
> > > > > > > > +            plain_stderr_layer()
> > > > > > > > +                .with_filter(filter_fn(|_metadata| !LogContext::exists()))
> > > > > > >
> > > > > > > ^ This condition misses the `Level::ERROR` comparison while being
> > > > > > > suggested as a replacement for `init_cli_logger` which had it (not
> > > > > > > visible in the patch context lines, but it's there).
> > > > > > > If this is done on purpose, please explain it in the commit message.
> > > > > >
> > > > > > Oops, yeah my bad, this should be
> > > > > >
> > > > > >     !LogContext::exists() || *metadata.level() == Level::ERROR
> > > > > >
> > > > > > What do you think about the rest of the patch? I tried to implement this
> > > > > > with a builder pattern as well, but it turned out to be quite tricky
> > > > > > moving the layers around so I just wrote a ton of functions with long
> > > > > > names :(
> > > > >
> > > > > The rest seems fine.
> > > > > It does look like it should be mostly a builder-pattern thing (as it
> > > > > kind of already is, with the final 2 lines being a kind of `.apply()`,
> > > > > but with the names being showing their intended use, it's fine for an
> > > > > `init` module to have specific common setups like this (`init_cli_…`,
> > > > > `…with_pve_format`, etc.)
> > > > >
> > > > > Perhaps the journal/tasklog one could be named "init_daemon_log" (or
> > > > > just have an alias under that name)...
> > > >
> > > > Sorry, I was reading it backwards, we're getting rid of those names...
> > > > That just goes to show I didn't properly think about this... :-)
> > > >
> > > > Now, first of all, having the "descriptive" names there makes sense.
> > > > With the `pve` specific function we then still have a rather specific
> > > > one.
> > > > Therefore, with a specific `init` module, it would IMO be fine to have
> > > > situation-specific names for common setups.
> > > >
> > > > Unless we can actually come up with a builder-pattern variant.
> > > > Perhaps it would have to be the Layer type rather than the subscriber
> > > > we'd need to turn into a builder, while having its final "apply" create
> > > > the subscriber and register it and initialize the LogTracer.
> > > 
> > > I cooked up a simple builder type thingy to build layers:
> > > 
> > > 
> > >     struct LogBuilder {
> > >         global_log_level: LevelFilter,
> > >         layer: Vec<
> > >             Box<dyn tracing_subscriber::Layer<tracing_subscriber::Registry> + Send + Sync + 'static>,
> > >         >,
> > >     }
> > > 
> > > 
> > > And the implementation:
> > > 
> > >     impl LogBuilder {
> > >         pub fn from_env(env_var: &str, default_log_level: LevelFilter) -> LogBuilder {
> > >             let log_level = get_env_variable(env_var, default_log_level);
> > >             LogBuilder {
> > >                 global_log_level: log_level,
> > >                 layer: vec![],
> > >             }
> > >         }
> > > 
> > >         pub fn journald_or_stderr(mut self) -> LogBuilder {
> > >             self.layer.push(
> > >                 journald_or_stderr_layer()
> > >                     .with_filter(self.global_log_level)
> > >                     .boxed(),
> > >             );
> > >             self
> > >         }
> > > 
> > >         pub fn journald_or_stderr_on_logcontext_and_error(mut self) -> LogBuilder {
> > >             self.layer.push(
> > >                 journald_or_stderr_layer()
> > >                     .with_filter(filter_fn(|metadata| {
> > >                         !LogContext::exists() || *metadata.level() == Level::ERROR
> > >                     }))
> > >                     .with_filter(self.global_log_level)
> > >                     .boxed(),
> > >             );
> > >             self
> > >         }
> > > 
> > >         //...
> > > 
> > >         pub fn init(self) -> Result<(), anyhow::Error> {
> > >             let registry = tracing_subscriber::registry().with(self.layer);
> > >             tracing::subscriber::set_global_default(registry)?;
> > > 
> > >             LogTracer::init_with_filter(self.global_log_level.as_log())?;
> > >             Ok(())
> > >         }
> > >     }
> > > 
> > > We could place this in a new builder module and then have the
> > > product-specific functions (e.g. init_pve_log, init_perlmod_log,
> > > init_pbs_log, etc.) in the init module.
> > > 
> > > What do you think?
> > 
> > Hmmm...
> > Those names are a bit long, and still as specific as before, so I'm not
> > sure we win a lot either way.
> > 
> > I'm wondering - if we really have so many specific cases - do we really
> > need them implemented in this crate, rather than where they are used?
> > How many different types of logging layers do we have and where atm?
> 
> We currently have:
>  1) journald (with stderr fallback)

=> builder.journald_logging()

(the fallback could just be implied, or do we ever not want that?)

>  2) stderr

=> builder.stderr_logging()

>  3) stderr (with pve formatting)

=> builder.pve_logging() (#[cfg(feature = "pve")]?)

>  4) pbs tasklog

^ I assume (4) is the "tasklog-or-journald" case?
Not sure where (1) is really the right choice.

Does (4) exist anywhere other than in the API daemons?

=> builder.task_logging()

(the journal-if-not-in-task part could be implied - or do we ever need
to combine the tasklog part differently?)


> 
> But the problem is we have different combinations and filters as well:
> journald and stderr (perlmod), stderr and pbs tasklog (but only when we
> are in a tasklog or when the level is error) (pbs-client), etc.

^ Why is "pbs-client" the example here? IMO this sounds like the API
daemons. The pbs-client *CLI* tool certainly should just log to stderr,
and the "crate" otherwise doesn't decide this.

> 
> The logging gets initiated in:

(list below is reordered)

>  * pbs proxy daemon
>  * pbs api daemon

^ API daemons - so those are case (4) - are they different from one
another?

>  * pbs proxmox-file-restore

^ The VM side? Not sure what it needs, but seems special-enough to have
its own code, unless we just log to the journal anyway

>  * pbs tape pmt
>  * pbs tape pmtx
>  * pbs client
>  * pbs pxar
>  * pbs proxmox-backup-debug
>  * pbs proxmox-backup-manager
>  * pbs proxmox-backup-tape
>  * pbs sg-tape-cmd

^ IIRC all of these are CLI tools and should therefore all be case (2) -
although I don't know about how the tape stuff works.
If they do anything else, it would be good to know why and have this
documented either in proxmox-log or in their logging-init functions.

>  * pbs proxmox-daily-update

The 'daily-update' may be a special case and could use the journal
directly, but may as well be stderr->journald via its `.service`.
But, yeah, obviously it would make actual tracing/debugging easier with
a proper journald-logger here. So another user of case (1).

>  * perlmod

^ For the lack of a better place (as pve, pmg and nftables code probably
all want to share it), having this as a special function in proxmox-log
makes some sense I suppose (but could be feature-guarded).

> 
> Exposing the builder directly without any helper functions would be fine
> as well I reckon. The downside is that the initiation gets more
> "complicated", e.g. (the pbs daemon):
> 
>     LogBuilder::from_env("PBS_LOG", LevelFilter::INFO)
>         .journald_on_no_tasklog_or_error()
>         .tasklog().init()?;

So yes this is probably okay to have, but I really don't think we need
these huge names - like I said, it doesn't make sense to me that
`.tasklog()` should be combined with anything other than this exact one
other thing anyway, so the middle line there could just be left out IMO.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel