* [pve-devel] [PATCH v3 firewall 0/1] Optional `since` and `until` firewall log filtering
@ 2023-01-19 10:25 Christian Ebner
2023-01-19 10:25 ` [pve-devel] [PATCH v3 firewall 1/1] api: Add optional parameters `since` and `until` for timestamp filter Christian Ebner
0 siblings, 1 reply; 3+ messages in thread
From: Christian Ebner @ 2023-01-19 10:25 UTC (permalink / raw)
To: pve-devel
This patch introduces 2 optional api parameters `since` and `until` to
firewall log endpoints, in order to make them filterable.
Filtering of the firewall logs is performed by a callback function.
---
Changes since RFC version:
- common: Use callback function filter instead of `since` `until` params
- common: code reuse for `dump_logfile` and `dump_fw_logfile`
- firewall: Style fixes and use of callback function
Changes since v1:
- common: Store parameters needed for multiple `dump_logfile_by_filehandle`
invocations in state hash.
- common: Introduce `final` parameter to signal last invocation to
`dump_logfile_by_filehandle`.
- firewall: Moved `dump_fw_logfile` to firewall helper functions
- firewall: Refactor of finding rotated logfiles by use of `dir_glob_foreach` and fixed regex.
- firewall: Avoid error if opening logfile failes with ENOENT
- Whitespace cleanup
Changes since v2:
- use `qr//` syntax for regex in filter callback
- deduplicate code by moving common filter logic to `dump_fw_logfile`
- Fix off by one error in `filecount` for finding final file
Christian Ebner (1):
api: Add optional parameters `since` and `until` for timestamp filter
src/PVE/API2/Firewall/Host.pm | 16 ++++++-
src/PVE/API2/Firewall/VM.pm | 26 ++++++++++--
src/PVE/Firewall/Helpers.pm | 78 +++++++++++++++++++++++++++++++++++
3 files changed, 115 insertions(+), 5 deletions(-)
--
2.30.2
^ permalink raw reply [flat|nested] 3+ messages in thread* [pve-devel] [PATCH v3 firewall 1/1] api: Add optional parameters `since` and `until` for timestamp filter 2023-01-19 10:25 [pve-devel] [PATCH v3 firewall 0/1] Optional `since` and `until` firewall log filtering Christian Ebner @ 2023-01-19 10:25 ` Christian Ebner 2023-01-27 10:04 ` [pve-devel] applied: " Wolfgang Bumiller 0 siblings, 1 reply; 3+ messages in thread From: Christian Ebner @ 2023-01-19 10:25 UTC (permalink / raw) To: pve-devel The optional unix epoch timestamps parameters `since` and `until` are introduced in order to filter firewall logs files. If one of these flags is set, also rotated logfiles are included. This is handled in the `dump_fw_logfile` helper function. Filtering is now performed based on a callback function passed to `dump_fw_logfile`. This patch depends on the corresponding patch in the pve-common repository. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- src/PVE/API2/Firewall/Host.pm | 16 ++++++- src/PVE/API2/Firewall/VM.pm | 26 ++++++++++-- src/PVE/Firewall/Helpers.pm | 78 +++++++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+), 5 deletions(-) diff --git a/src/PVE/API2/Firewall/Host.pm b/src/PVE/API2/Firewall/Host.pm index dfeccd0..0432de2 100644 --- a/src/PVE/API2/Firewall/Host.pm +++ b/src/PVE/API2/Firewall/Host.pm @@ -172,6 +172,18 @@ __PACKAGE__->register_method({ minimum => 0, optional => 1, }, + since => { + type => 'integer', + minimum => 0, + description => "Display log since this UNIX epoch.", + optional => 1, + }, + until => { + type => 'integer', + minimum => 0, + description => "Display log until this UNIX epoch.", + optional => 1, + }, }, }, returns => { @@ -196,8 +208,10 @@ __PACKAGE__->register_method({ my $rpcenv = PVE::RPCEnvironment::get(); my $user = $rpcenv->get_user(); my $node = $param->{node}; + my $filename = "/var/log/pve-firewall.log"; - my ($count, $lines) = PVE::Tools::dump_logfile("/var/log/pve-firewall.log", $param->{start}, $param->{limit}); + my ($count, $lines) = PVE::Firewall::Helpers::dump_fw_logfile( + $filename, $param, undef); $rpcenv->set_result_attrib('total', $count); diff --git a/src/PVE/API2/Firewall/VM.pm b/src/PVE/API2/Firewall/VM.pm index 48b8c5f..fb255e0 100644 --- a/src/PVE/API2/Firewall/VM.pm +++ b/src/PVE/API2/Firewall/VM.pm @@ -176,6 +176,18 @@ sub register_handlers { minimum => 0, optional => 1, }, + since => { + type => 'integer', + minimum => 0, + description => "Display log since this UNIX epoch.", + optional => 1, + }, + until => { + type => 'integer', + minimum => 0, + description => "Display log until this UNIX epoch.", + optional => 1, + }, }, }, returns => { @@ -199,11 +211,17 @@ sub register_handlers { my $rpcenv = PVE::RPCEnvironment::get(); my $user = $rpcenv->get_user(); - my $vmid = $param->{vmid}; + my $filename = "/var/log/pve-firewall.log"; + my $vmid = $param->{'vmid'}; + + my $callback = sub { + my ($line) = @_; + my $reg = "^$vmid "; + return $line =~ m/$reg/; + }; - my ($count, $lines) = PVE::Tools::dump_logfile("/var/log/pve-firewall.log", - $param->{start}, $param->{limit}, - "^$vmid "); + my ($count, $lines) = PVE::Firewall::Helpers::dump_fw_logfile( + $filename, $param, $callback); $rpcenv->set_result_attrib('total', $count); diff --git a/src/PVE/Firewall/Helpers.pm b/src/PVE/Firewall/Helpers.pm index 154fca5..54f5519 100644 --- a/src/PVE/Firewall/Helpers.pm +++ b/src/PVE/Firewall/Helpers.pm @@ -3,6 +3,10 @@ package PVE::Firewall::Helpers; use strict; use warnings; +use Date::Parse qw(str2time); +use Errno qw(ENOENT); +use File::Basename qw(fileparse); +use IO::Zlib; use PVE::Cluster; use PVE::Tools qw(file_get_contents file_set_contents); @@ -52,4 +56,78 @@ sub clone_vmfw_conf { }); } +sub dump_fw_logfile { + my ($filename, $param, $callback) = @_; + my ($start, $limit, $since, $until) = $param->@{qw(start limit since until)}; + + my $filter = sub { + my ($line) = @_; + + if (defined($callback)) { + return undef if !$callback->($line); + } + + if ($since || $until) { + my @words = split / /, $line; + my $timestamp = str2time($words[3], $words[4]); + return undef if $since && $timestamp < $since; + return undef if $until && $timestamp > $until; + } + + return $line; + }; + + if (!defined($since) && !defined($until)) { + return PVE::Tools::dump_logfile($filename, $start, $limit, $filter); + } + + my %state = ( + 'count' => 0, + 'lines' => [], + 'start' => $start, + 'limit' => $limit, + ); + + # Take into consideration also rotated logs + my ($basename, $logdir, $type) = fileparse($filename); + my $regex = qr/^\Q$basename\E(\.[\d]+(\.gz)?)?$/; + my @files = (); + + PVE::Tools::dir_glob_foreach($logdir, $regex, sub { + my ($file) = @_; + push @files, $file; + }); + + @files = reverse sort @files; + + my $filecount = 0; + for my $filename (@files) { + $state{'final'} = $filecount == $#files; + $filecount++; + + my $fh; + if ($filename =~ /\.gz$/) { + $fh = IO::Zlib->new($logdir.$filename, "r"); + } else { + $fh = IO::File->new($logdir.$filename, "r"); + } + + if (!$fh) { + # If file vanished since reading dir entries, ignore + continue if $!{ENOENT}; + + my $lines = $state{'lines'}; + my $count = ++$state{'count'}; + push @$lines, ($count, { n => $count, t => "unable to open file - $!"}); + last; + } + + PVE::Tools::dump_logfile_by_filehandle($fh, $filter, \%state); + + close($fh); + } + + return ($state{'count'}, $state{'lines'}); +} + 1; -- 2.30.2 ^ permalink raw reply [flat|nested] 3+ messages in thread
* [pve-devel] applied: [PATCH v3 firewall 1/1] api: Add optional parameters `since` and `until` for timestamp filter 2023-01-19 10:25 ` [pve-devel] [PATCH v3 firewall 1/1] api: Add optional parameters `since` and `until` for timestamp filter Christian Ebner @ 2023-01-27 10:04 ` Wolfgang Bumiller 0 siblings, 0 replies; 3+ messages in thread From: Wolfgang Bumiller @ 2023-01-27 10:04 UTC (permalink / raw) To: Christian Ebner; +Cc: pve-devel applied, but with a little fixup, see below On Thu, Jan 19, 2023 at 11:25:04AM +0100, Christian Ebner wrote: > The optional unix epoch timestamps parameters `since` and `until` are introduced > in order to filter firewall logs files. If one of these flags is set, also > rotated logfiles are included. This is handled in the `dump_fw_logfile` helper > function. Filtering is now performed based on a callback function passed to > `dump_fw_logfile`. > > This patch depends on the corresponding patch in the pve-common repository. > > Signed-off-by: Christian Ebner <c.ebner@proxmox.com> > --- > src/PVE/API2/Firewall/Host.pm | 16 ++++++- > src/PVE/API2/Firewall/VM.pm | 26 ++++++++++-- > src/PVE/Firewall/Helpers.pm | 78 +++++++++++++++++++++++++++++++++++ > 3 files changed, 115 insertions(+), 5 deletions(-) > > diff --git a/src/PVE/API2/Firewall/Host.pm b/src/PVE/API2/Firewall/Host.pm > index dfeccd0..0432de2 100644 > --- a/src/PVE/API2/Firewall/Host.pm > +++ b/src/PVE/API2/Firewall/Host.pm > @@ -172,6 +172,18 @@ __PACKAGE__->register_method({ > minimum => 0, > optional => 1, > }, > + since => { > + type => 'integer', > + minimum => 0, > + description => "Display log since this UNIX epoch.", > + optional => 1, > + }, > + until => { > + type => 'integer', > + minimum => 0, > + description => "Display log until this UNIX epoch.", > + optional => 1, > + }, > }, > }, > returns => { > @@ -196,8 +208,10 @@ __PACKAGE__->register_method({ > my $rpcenv = PVE::RPCEnvironment::get(); > my $user = $rpcenv->get_user(); > my $node = $param->{node}; > + my $filename = "/var/log/pve-firewall.log"; > > - my ($count, $lines) = PVE::Tools::dump_logfile("/var/log/pve-firewall.log", $param->{start}, $param->{limit}); > + my ($count, $lines) = PVE::Firewall::Helpers::dump_fw_logfile( > + $filename, $param, undef); > > $rpcenv->set_result_attrib('total', $count); > > diff --git a/src/PVE/API2/Firewall/VM.pm b/src/PVE/API2/Firewall/VM.pm > index 48b8c5f..fb255e0 100644 > --- a/src/PVE/API2/Firewall/VM.pm > +++ b/src/PVE/API2/Firewall/VM.pm > @@ -176,6 +176,18 @@ sub register_handlers { > minimum => 0, > optional => 1, > }, > + since => { > + type => 'integer', > + minimum => 0, > + description => "Display log since this UNIX epoch.", > + optional => 1, > + }, > + until => { > + type => 'integer', > + minimum => 0, > + description => "Display log until this UNIX epoch.", > + optional => 1, > + }, > }, > }, > returns => { > @@ -199,11 +211,17 @@ sub register_handlers { > > my $rpcenv = PVE::RPCEnvironment::get(); > my $user = $rpcenv->get_user(); > - my $vmid = $param->{vmid}; > + my $filename = "/var/log/pve-firewall.log"; > + my $vmid = $param->{'vmid'}; > + > + my $callback = sub { > + my ($line) = @_; > + my $reg = "^$vmid "; > + return $line =~ m/$reg/; > + }; > > - my ($count, $lines) = PVE::Tools::dump_logfile("/var/log/pve-firewall.log", > - $param->{start}, $param->{limit}, > - "^$vmid "); > + my ($count, $lines) = PVE::Firewall::Helpers::dump_fw_logfile( > + $filename, $param, $callback); > > $rpcenv->set_result_attrib('total', $count); > > diff --git a/src/PVE/Firewall/Helpers.pm b/src/PVE/Firewall/Helpers.pm > index 154fca5..54f5519 100644 > --- a/src/PVE/Firewall/Helpers.pm > +++ b/src/PVE/Firewall/Helpers.pm > @@ -3,6 +3,10 @@ package PVE::Firewall::Helpers; > use strict; > use warnings; > > +use Date::Parse qw(str2time); > +use Errno qw(ENOENT); > +use File::Basename qw(fileparse); > +use IO::Zlib; > use PVE::Cluster; > use PVE::Tools qw(file_get_contents file_set_contents); > > @@ -52,4 +56,78 @@ sub clone_vmfw_conf { > }); > } > > +sub dump_fw_logfile { > + my ($filename, $param, $callback) = @_; > + my ($start, $limit, $since, $until) = $param->@{qw(start limit since until)}; > + > + my $filter = sub { > + my ($line) = @_; > + > + if (defined($callback)) { > + return undef if !$callback->($line); > + } > + > + if ($since || $until) { > + my @words = split / /, $line; > + my $timestamp = str2time($words[3], $words[4]); > + return undef if $since && $timestamp < $since; > + return undef if $until && $timestamp > $until; > + } > + > + return $line; > + }; > + > + if (!defined($since) && !defined($until)) { > + return PVE::Tools::dump_logfile($filename, $start, $limit, $filter); > + } > + > + my %state = ( > + 'count' => 0, > + 'lines' => [], > + 'start' => $start, > + 'limit' => $limit, > + ); > + > + # Take into consideration also rotated logs > + my ($basename, $logdir, $type) = fileparse($filename); > + my $regex = qr/^\Q$basename\E(\.[\d]+(\.gz)?)?$/; > + my @files = (); > + > + PVE::Tools::dir_glob_foreach($logdir, $regex, sub { > + my ($file) = @_; > + push @files, $file; > + }); > + > + @files = reverse sort @files; > + > + my $filecount = 0; > + for my $filename (@files) { > + $state{'final'} = $filecount == $#files; > + $filecount++; > + > + my $fh; > + if ($filename =~ /\.gz$/) { > + $fh = IO::Zlib->new($logdir.$filename, "r"); > + } else { > + $fh = IO::File->new($logdir.$filename, "r"); > + } > + > + if (!$fh) { > + # If file vanished since reading dir entries, ignore > + continue if $!{ENOENT}; ^ Just noticed this `continue` which should be `next` in perl. This is one of the nastier things in perl that only trigger at run-time even though IMO it's supposed to check that at compile time. ...Except it *can't* because it also allows `continue` to bail out of multiple levels of function calls back to a `given{}` block... *sigh*... perl... just... WHY... ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-01-27 10:04 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-19 10:25 [pve-devel] [PATCH v3 firewall 0/1] Optional `since` and `until` firewall log filtering Christian Ebner 2023-01-19 10:25 ` [pve-devel] [PATCH v3 firewall 1/1] api: Add optional parameters `since` and `until` for timestamp filter Christian Ebner 2023-01-27 10:04 ` [pve-devel] applied: " Wolfgang Bumiller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox