[PATCH datacenter-manager/proxmox 0/6] log: allow finegrained control logging levels

[Thomas Ellmenreich <t.ellmenreich@proxmox.com>]

The core of this change is a simple reinterpretation of the logging environment
variables that we are already using. Instead of parsing them as a single level
filter [0], they will now be interpreted as env filters [1]. Since the latter
can still parse the former, this change is backwards compatible and will not
break with any logging environment variables that have been set.

Implementation
--------------

As Gabriel Goller proposed in this [2] Bugzilla enhancement, by using
EnvFilters [1], we can have more granular control over the logging happening in
the different Proxmox products. The first two commits do exactly this and can
thus be applied by themselves. The last three add tests as well as utilities to
better communicate log levels to use cases that don't directly integrate with
tracing.

proxmox 1/6:
    A simple reordering of the logging layers. This had to be done since
    EnvFilter is currently not clonable and can thus be applied only once.
    Previously, the global filter was being applied unnecessarily often, on all
    layers and then on the tracing_log layer as well.

proxmox 2/6:
    Actual implementation of EnvFilters.

proxmox 3/6:
    Simple clean up of empty structs to unit structs and unnecessarily
    explicit module paths that clutter struct definitions.

proxmox 4/6:
    Introduction of tests for our EnvFilter, parser, and constructor. This was
    initially created for me to better understand EnvFilters, but then evolved
    into tests to ensure consistent functionality with our 'default_log_level'.

datacenter-manager 5/6:
    Use proxmox_log::enabled! to check if DEBUG is enabled and use that to
    determine the REST server debug mode.

datacenter-manager 6/6:
    Replaced the PROXMOX_DEBUG environment variable by PDM_LOG, which is more
    in line with the other products.

Different Log Options
---------------------

Combining the different options of the EnvFilter as well as our own default
log level, we have the following cases for logging:

1. Env contains 'simple':
    Meaning that the environment variable contains a simple 'info'... This
    means that that level is taken as the expected one.

2. Env contains 'simple', and 'module':
    Means that the env variable contains a simple log level as well as zero or
    more module specific log levels. As expected the simple level is applied
    as a default and then for the specified modules the defined levels apply.

3. Env only contains 'module':
    If the env only contains a module (or multiple), that is interpreted by
    tracing as disabling the default logging and only enabling logging for the
    module/s

4. Env is empty:
    No logs are printed at all, everything is hidden.

5. Env is not set:
    This is the only time the default comes into play, by being set as the
    default log level.

Notes for the Maintainer
------------------------

For all tracing logging in Proxmox products to support EnvFilters, only
patches 1 and 2 have to be applied. The 4 Patch only contains tests which I
mostly used for understanding, but do still test our defaulting behaviour. And
the rest are small cleanups that might be nice, but not necessary.

Changelog
---------

- RFC v2 -> PATCH v1 (thanks @Lukas)
    * Split the 'proxmox 1/4' commit into two commits. Instead of mixing the
      global filter mix with the cleanup, they are now two different commits.
      To make sure that the core patches can be applied independently, the
      cleanup commit was moved to 'proxmox 3/6'.

    * Fixed some clippy warnings relating to documentation indenting.

    * Added tests that also directly call 'tracing' instead of just calling
      'log', just to confirm that both work and not just 'log'.

    * Introduced a new patch at the end, renaming the PROXMOX_DEBUG
      environment variable to PDM_LOG, as is default for all other products.

- RFC v1 -> v2
    * Removed patch 'v1 proxmox 4/5', which introduced a LoggerConfig returned
      by the Logger::init function. Deemed it unnecessary since I replaced the
      usage for it in patch 'v1 datacenter-manager 5/5' with a simple 'enabled!'
      and the idea of using it in the solution for [3] was abandoned.

    * Cleaned up this cover letter


[0]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.LevelFilter.html
[1]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html
[2]: https://bugzilla.proxmox.com/show_bug.cgi?id=6081
[3]: https://bugzilla.proxmox.com/show_bug.cgi?id=4646


proxmox:

Thomas Ellmenreich (4):
  log: replace per layer filtering by one global filter
  fix #6081: log: replace simple level filter with env filter
  log: cleanup of logger components
  log: add tests to the logger

 proxmox-log/Cargo.toml                |   2 +-
 proxmox-log/src/builder.rs            | 399 +++++++++++++++++++++++---
 proxmox-log/src/lib.rs                |  16 --
 proxmox-log/src/pve_task_formatter.rs |   2 +-
 4 files changed, 362 insertions(+), 57 deletions(-)


proxmox-datacenter-manager:

Thomas Ellmenreich (2):
  api: set REST server debug level based on actual log level
  api: update logging variable

 server/src/bin/proxmox-datacenter-api/main.rs | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)


Summary over all repositories:
  5 files changed, 373 insertions(+), 60 deletions(-)

-- 
Generated by murpp 0.12.0