public inbox for pmg-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files
@ 2021-10-06 11:46 Oguz Bektas
  2021-10-06 11:51 ` Thomas Lamprecht
  0 siblings, 1 reply; 4+ messages in thread
From: Oguz Bektas @ 2021-10-06 11:46 UTC (permalink / raw)
  To: pmg-devel

with an optional "--maillog" parameter, we parse /var/log/mail.log.*
instead of /var/log/syslog.*

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
---
 src/main.rs | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

diff --git a/src/main.rs b/src/main.rs
index b1d4f8c..13ca63b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -39,6 +39,11 @@ fn main() -> Result<(), Error> {
                 .help("Input file to use instead of /var/log/syslog, or '-' for stdin")
                 .value_name("INPUTFILE"),
         )
+        .arg(
+            Arg::with_name("maillog")
+                .long("maillog")
+                .help("Use /var/log/mail.log* files instead of /var/log/syslog")
+        )
         .arg(
             Arg::with_name("host")
                 .short("h")
@@ -1816,7 +1821,11 @@ impl Parser {
             let filecount = self.count_files_in_time_range();
             for i in (0..filecount).rev() {
                 self.current_month = 0;
-                if let Ok(file) = File::open(LOGFILES[i]) {
+                let mut target_files = LOGFILES;
+                if self.options.maillog {
+                    target_files = MAILLOGFILES;
+                }
+                if let Ok(file) = File::open(target_files[i]) {
                     self.current_file_index = i;
                     if i > 1 {
                         let gzdecoder = read::GzDecoder::new(file);
@@ -1988,6 +1997,8 @@ impl Parser {
             self.options.inputfile = inputfile.to_string();
         }
 
+        self.options.maillog = args.is_present("maillog");
+
         if let Some(start) = args.value_of("start") {
             if let Ok(res) = time::strptime(&start, "%F %T") {
                 self.options.start = mkgmtime(&res);
@@ -2121,6 +2132,7 @@ struct Options {
     end: libc::time_t,
     limit: u64,
     verbose: u32,
+    maillog: bool,
     exclude_greylist: bool,
     exclude_ndr: bool,
 }
@@ -2208,6 +2220,41 @@ fn mkgmtime(tm: &time::Tm) -> libc::time_t {
     res
 }
 
+const MAILLOGFILES: [&str; 32] = [
+    "/var/log/mail.log",
+    "/var/log/mail.log.1",
+    "/var/log/mail.log.2.gz",
+    "/var/log/mail.log.3.gz",
+    "/var/log/mail.log.4.gz",
+    "/var/log/mail.log.5.gz",
+    "/var/log/mail.log.6.gz",
+    "/var/log/mail.log.7.gz",
+    "/var/log/mail.log.8.gz",
+    "/var/log/mail.log.9.gz",
+    "/var/log/mail.log.10.gz",
+    "/var/log/mail.log.11.gz",
+    "/var/log/mail.log.12.gz",
+    "/var/log/mail.log.13.gz",
+    "/var/log/mail.log.14.gz",
+    "/var/log/mail.log.15.gz",
+    "/var/log/mail.log.16.gz",
+    "/var/log/mail.log.17.gz",
+    "/var/log/mail.log.18.gz",
+    "/var/log/mail.log.19.gz",
+    "/var/log/mail.log.20.gz",
+    "/var/log/mail.log.21.gz",
+    "/var/log/mail.log.22.gz",
+    "/var/log/mail.log.23.gz",
+    "/var/log/mail.log.24.gz",
+    "/var/log/mail.log.25.gz",
+    "/var/log/mail.log.26.gz",
+    "/var/log/mail.log.27.gz",
+    "/var/log/mail.log.28.gz",
+    "/var/log/mail.log.29.gz",
+    "/var/log/mail.log.30.gz",
+    "/var/log/mail.log.31.gz",
+];
+
 const LOGFILES: [&str; 32] = [
     "/var/log/syslog",
     "/var/log/syslog.1",
-- 
2.30.2





^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files
  2021-10-06 11:46 [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files Oguz Bektas
@ 2021-10-06 11:51 ` Thomas Lamprecht
  2021-10-06 13:28   ` Stoiko Ivanov
  0 siblings, 1 reply; 4+ messages in thread
From: Thomas Lamprecht @ 2021-10-06 11:51 UTC (permalink / raw)
  To: Oguz Bektas, pmg-devel

On 06.10.21 13:46, Oguz Bektas wrote:
> with an optional "--maillog" parameter, we parse /var/log/mail.log.*
> instead of /var/log/syslog.*

--mail-log

and it may make sense to allow passing a base path to that, that would also allow easier
evaluation like copy of the logs from the production system and run queries on another,
not so important, system.

> 
> Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
> ---
>  src/main.rs | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 48 insertions(+), 1 deletion(-)
> 
> diff --git a/src/main.rs b/src/main.rs
> index b1d4f8c..13ca63b 100644
> --- a/src/main.rs
> +++ b/src/main.rs
> @@ -39,6 +39,11 @@ fn main() -> Result<(), Error> {
>                  .help("Input file to use instead of /var/log/syslog, or '-' for stdin")
>                  .value_name("INPUTFILE"),
>          )
> +        .arg(
> +            Arg::with_name("maillog")
> +                .long("maillog")
> +                .help("Use /var/log/mail.log* files instead of /var/log/syslog")
> +        )
>          .arg(
>              Arg::with_name("host")
>                  .short("h")
> @@ -1816,7 +1821,11 @@ impl Parser {
>              let filecount = self.count_files_in_time_range();
>              for i in (0..filecount).rev() {
>                  self.current_month = 0;
> -                if let Ok(file) = File::open(LOGFILES[i]) {
> +                let mut target_files = LOGFILES;
> +                if self.options.maillog {
> +                    target_files = MAILLOGFILES;
> +                }

that can be done nicer in rust avoiding mut completely, either use the fact
that if statements can return expressions or the match feature, e.g. the former:

let target_files = if self.options.maillog { MAILLOGFILES } else { LOGFILES };

avoiding mut is nicer for saner programs and allows often better optimizations.

> +                if let Ok(file) = File::open(target_files[i]) {
>                      self.current_file_index = i;
>                      if i > 1 {
>                          let gzdecoder = read::GzDecoder::new(file);
> @@ -1988,6 +1997,8 @@ impl Parser {
>              self.options.inputfile = inputfile.to_string();
>          }
>  
> +        self.options.maillog = args.is_present("maillog");
> +
>          if let Some(start) = args.value_of("start") {
>              if let Ok(res) = time::strptime(&start, "%F %T") {
>                  self.options.start = mkgmtime(&res);
> @@ -2121,6 +2132,7 @@ struct Options {
>      end: libc::time_t,
>      limit: u64,
>      verbose: u32,
> +    maillog: bool,
>      exclude_greylist: bool,
>      exclude_ndr: bool,
>  }
> @@ -2208,6 +2220,41 @@ fn mkgmtime(tm: &time::Tm) -> libc::time_t {
>      res
>  }
>  
> +const MAILLOGFILES: [&str; 32] = [
> +    "/var/log/mail.log",
> +    "/var/log/mail.log.1",
> +    "/var/log/mail.log.2.gz",
> +    "/var/log/mail.log.3.gz",
> +    "/var/log/mail.log.4.gz",
> +    "/var/log/mail.log.5.gz",
> +    "/var/log/mail.log.6.gz",
> +    "/var/log/mail.log.7.gz",
> +    "/var/log/mail.log.8.gz",
> +    "/var/log/mail.log.9.gz",
> +    "/var/log/mail.log.10.gz",
> +    "/var/log/mail.log.11.gz",
> +    "/var/log/mail.log.12.gz",
> +    "/var/log/mail.log.13.gz",
> +    "/var/log/mail.log.14.gz",
> +    "/var/log/mail.log.15.gz",
> +    "/var/log/mail.log.16.gz",
> +    "/var/log/mail.log.17.gz",
> +    "/var/log/mail.log.18.gz",
> +    "/var/log/mail.log.19.gz",
> +    "/var/log/mail.log.20.gz",
> +    "/var/log/mail.log.21.gz",
> +    "/var/log/mail.log.22.gz",
> +    "/var/log/mail.log.23.gz",
> +    "/var/log/mail.log.24.gz",
> +    "/var/log/mail.log.25.gz",
> +    "/var/log/mail.log.26.gz",
> +    "/var/log/mail.log.27.gz",
> +    "/var/log/mail.log.28.gz",
> +    "/var/log/mail.log.29.gz",
> +    "/var/log/mail.log.30.gz",
> +    "/var/log/mail.log.31.gz",
> +];


could be an iterator (good if we want to allow an arbitrary base) or using a
const fn (for a fixed one like here)...

> +
>  const LOGFILES: [&str; 32] = [
>      "/var/log/syslog",
>      "/var/log/syslog.1",
> 





^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files
  2021-10-06 11:51 ` Thomas Lamprecht
@ 2021-10-06 13:28   ` Stoiko Ivanov
  2021-10-06 13:41     ` Thomas Lamprecht
  0 siblings, 1 reply; 4+ messages in thread
From: Stoiko Ivanov @ 2021-10-06 13:28 UTC (permalink / raw)
  To: Thomas Lamprecht, Oguz Bektas; +Cc: pmg-devel

On Wed, 6 Oct 2021 13:51:53 +0200
Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:

> On 06.10.21 13:46, Oguz Bektas wrote:
> > with an optional "--maillog" parameter, we parse /var/log/mail.log.*
> > instead of /var/log/syslog.*  
> 
> --mail-log
> 
> and it may make sense to allow passing a base path to that, that would also allow easier
> evaluation like copy of the logs from the production system and run queries on another,
> not so important, system.
+1 for providing a base-path - I'd even go one further and let it deal
with the optionally gzipped logs by matching the filename (for .gz)
instead of the 'index' (assuming that the first and second file are not
compressed, and all consecutive files are).

With that (and the assumption that `logrotate` is used) this would
probably be usable in a configurable way for the GUI+Tracking Center)

Question is from a UX point of view how much a '-i' for single logfile and
a '--mail-log' for a pattern of logfiles makes sense.


> 
> > 
> > Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
> > ---
> >  src/main.rs | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 48 insertions(+), 1 deletion(-)
> > 
> > diff --git a/src/main.rs b/src/main.rs
> > index b1d4f8c..13ca63b 100644
> > --- a/src/main.rs
> > +++ b/src/main.rs
> > @@ -39,6 +39,11 @@ fn main() -> Result<(), Error> {
> >                  .help("Input file to use instead of /var/log/syslog, or '-' for stdin")
> >                  .value_name("INPUTFILE"),
> >          )
> > +        .arg(
> > +            Arg::with_name("maillog")
> > +                .long("maillog")
> > +                .help("Use /var/log/mail.log* files instead of /var/log/syslog")
> > +        )
> >          .arg(
> >              Arg::with_name("host")
> >                  .short("h")
> > @@ -1816,7 +1821,11 @@ impl Parser {
> >              let filecount = self.count_files_in_time_range();
> >              for i in (0..filecount).rev() {
> >                  self.current_month = 0;
> > -                if let Ok(file) = File::open(LOGFILES[i]) {
> > +                let mut target_files = LOGFILES;
> > +                if self.options.maillog {
> > +                    target_files = MAILLOGFILES;
> > +                }  
> 
> that can be done nicer in rust avoiding mut completely, either use the fact
> that if statements can return expressions or the match feature, e.g. the former:
> 
> let target_files = if self.options.maillog { MAILLOGFILES } else { LOGFILES };
> 
> avoiding mut is nicer for saner programs and allows often better optimizations.
> 
> > +                if let Ok(file) = File::open(target_files[i]) {
> >                      self.current_file_index = i;
> >                      if i > 1 {
> >                          let gzdecoder = read::GzDecoder::new(file);
> > @@ -1988,6 +1997,8 @@ impl Parser {
> >              self.options.inputfile = inputfile.to_string();
> >          }
> >  
> > +        self.options.maillog = args.is_present("maillog");
> > +
> >          if let Some(start) = args.value_of("start") {
> >              if let Ok(res) = time::strptime(&start, "%F %T") {
> >                  self.options.start = mkgmtime(&res);
> > @@ -2121,6 +2132,7 @@ struct Options {
> >      end: libc::time_t,
> >      limit: u64,
> >      verbose: u32,
> > +    maillog: bool,
> >      exclude_greylist: bool,
> >      exclude_ndr: bool,
> >  }
> > @@ -2208,6 +2220,41 @@ fn mkgmtime(tm: &time::Tm) -> libc::time_t {
> >      res
> >  }
> >  
> > +const MAILLOGFILES: [&str; 32] = [
> > +    "/var/log/mail.log",
> > +    "/var/log/mail.log.1",
> > +    "/var/log/mail.log.2.gz",
> > +    "/var/log/mail.log.3.gz",
> > +    "/var/log/mail.log.4.gz",
> > +    "/var/log/mail.log.5.gz",
> > +    "/var/log/mail.log.6.gz",
> > +    "/var/log/mail.log.7.gz",
> > +    "/var/log/mail.log.8.gz",
> > +    "/var/log/mail.log.9.gz",
> > +    "/var/log/mail.log.10.gz",
> > +    "/var/log/mail.log.11.gz",
> > +    "/var/log/mail.log.12.gz",
> > +    "/var/log/mail.log.13.gz",
> > +    "/var/log/mail.log.14.gz",
> > +    "/var/log/mail.log.15.gz",
> > +    "/var/log/mail.log.16.gz",
> > +    "/var/log/mail.log.17.gz",
> > +    "/var/log/mail.log.18.gz",
> > +    "/var/log/mail.log.19.gz",
> > +    "/var/log/mail.log.20.gz",
> > +    "/var/log/mail.log.21.gz",
> > +    "/var/log/mail.log.22.gz",
> > +    "/var/log/mail.log.23.gz",
> > +    "/var/log/mail.log.24.gz",
> > +    "/var/log/mail.log.25.gz",
> > +    "/var/log/mail.log.26.gz",
> > +    "/var/log/mail.log.27.gz",
> > +    "/var/log/mail.log.28.gz",
> > +    "/var/log/mail.log.29.gz",
> > +    "/var/log/mail.log.30.gz",
> > +    "/var/log/mail.log.31.gz",
> > +];  
> 
> 
> could be an iterator (good if we want to allow an arbitrary base) or using a
> const fn (for a fixed one like here)...
> 
> > +
> >  const LOGFILES: [&str; 32] = [
> >      "/var/log/syslog",
> >      "/var/log/syslog.1",
> >   
> 
> 
> 
> _______________________________________________
> pmg-devel mailing list
> pmg-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel
> 
> 





^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files
  2021-10-06 13:28   ` Stoiko Ivanov
@ 2021-10-06 13:41     ` Thomas Lamprecht
  0 siblings, 0 replies; 4+ messages in thread
From: Thomas Lamprecht @ 2021-10-06 13:41 UTC (permalink / raw)
  To: Stoiko Ivanov, Oguz Bektas; +Cc: pmg-devel

On 06.10.21 15:28, Stoiko Ivanov wrote:
> On Wed, 6 Oct 2021 13:51:53 +0200
> Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:
> 
>> On 06.10.21 13:46, Oguz Bektas wrote:
>>> with an optional "--maillog" parameter, we parse /var/log/mail.log.*
>>> instead of /var/log/syslog.*  
>> --mail-log
>>
>> and it may make sense to allow passing a base path to that, that would also allow easier
>> evaluation like copy of the logs from the production system and run queries on another,
>> not so important, system.
> +1 for providing a base-path - I'd even go one further and let it deal
> with the optionally gzipped logs by matching the filename (for .gz)
> instead of the 'index' (assuming that the first and second file are not
> compressed, and all consecutive files are).

yeah we can also try either match on every, that way we can better cope with
custom logrotation settings that may gzip only after .3 or so.

All that would be relatively easy to do by implementing the iterator trait for
a struct with a "base_path" member.

> 
> With that (and the assumption that `logrotate` is used) this would
> probably be usable in a configurable way for the GUI+Tracking Center)
> 
> Question is from a UX point of view how much a '-i' for single logfile and
> a '--mail-log' for a pattern of logfiles makes sense.
> 
> 

You have a point there.

First, generally my opinion is that not every option needs a short-op, as that is
something that often leads to some weird excuses/arguments for a specific letter which
a user may the see different.
 
As input is `-i` or `--inputfile` it may be sensible to use --input-base or --input-series
We already use minus as separator in a few places (--message-id, --queue-id, --search-string)
so IMO cannot really care to much for consistency here, and I'd just avoid the short opt for
this rather specific option.





^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-10-06 13:42 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-06 11:46 [pmg-devel] [PATCH pmg-log-tracker] fix #3657: allow parsing mail.log files Oguz Bektas
2021-10-06 11:51 ` Thomas Lamprecht
2021-10-06 13:28   ` Stoiko Ivanov
2021-10-06 13:41     ` Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal