Re: [pbs-devel] [RFC proxmox-backup 0/2] Tasklog rewrite with tracing

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 23/10/2023 um 11:29 schrieb Dominik Csapak:
> info!("foo"); // logs to task log inside worker, syslog outside
> info!(syslog = true, "foo"); // logs only to syslog, even in worker context
> info!(syslog = true, tasklog = true, "foo"); // log in both task and syslog (if possible)

I find this a bit confusing, especially if one stumbles upon
multiple variants in the same code (path), like e.g., when reading
the first and the second variant, one might think that the first
doesn't have syslog true, or not think that the second has tasklog
false implied.

So if, then I'd rather use negation, i.e., encode that one wants
to avoid logging something to some channel: info!(syslog = false, "..."),
but IMO not really ideal either – while the base idea can be OK, I'D
avoid such flags, they're making it longer and not really less confusing.

We generally want to log to the task-log inside workers, logging to
the syslog is rather the outlier (we overuse syslog currently already
with that downloaded chunk logging) reserved for warnings and errors,
as especially the latter can happen if there are system-wide errors
(like disk failure), so useful to see there.
This is what we're (mostly) doing in perl, connect stdout to the task
log, errors will show up in syslog too, but for targeting the syslogs
explicitly one needs to do so too, while not perfect, that system works
IMO relatively well.

So maybe just add the auto-log stuff from Dominik's variant for plain
info!("") et al., but also log any error to both task and syslog.

To reduce syslog noise we can also limit to only relaying warnings and
higher level to syslog in the "no worker task context case". I.e.:

- if in worker task:
  - log info and higher to worker task log
  - mirror error level to syslog too

- if not in worker task:
  - log warnings and higher to syslog

It's slightly magic, but fit our reality quite well.

For forcing other logs, or a debug level, to a target too, one then
could provide environment variables, or an API command-socket command
for a dynamic change of max-level to log – especially the latter is
a bit orthogonal to this and should be done later.

semi-related: I didn't check the series in detail, so sorry if already
answered there, but how's the story for logging specific spans?

As logging levels are normally rather coarse, especially on complex
tasks one can have a log going on, and so it can be helpful to, e.g.,
enable debug logging only for some specific "topics", especially if
one is debugging an issue that a user faces, but one cannot (easily)
reproduce themselves. But, I guess that is unrelated from your current
work here, or did you already think about that?

(the place I'd want something like that the most is pmxcfs, as there
you can currently only decide if you want very sparse information or
every detail times ten for about every file operation happening, very
frustrating)