public inbox for pmg-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Csapak <d.csapak@proxmox.com>
To: pmg-devel@lists.proxmox.com
Subject: [pmg-devel] [PATCH pmg-api v2 09/10] RuleCache: implement and/invert for what matches
Date: Wed, 21 Feb 2024 13:24:35 +0100	[thread overview]
Message-ID: <20240221122439.1281024-10-d.csapak@proxmox.com> (raw)
In-Reply-To: <20240221122439.1281024-1-d.csapak@proxmox.com>

Since what matches are not a simple boolean match, but also can contain
"marks" to mark specific parts of the mail, we must implement some
custom logic for and/invert here.

The goal here is to define that groups are on a per part level,
but the rule operates on the whole mail.

To achieve this we have two different and/invert combine functions, one
for the group level and one for the whole what match.

For per group and/inversion we and 'and-combine' and invert the list of
marks, so if it matches part 1,2 of 1,2,3 the inversion would return 3.

For the rule it only matters if the and/inversion part matches at all,
regardless of the marks. If it matches, the marks will be or'ed.

With this, one can represent many different scenarios that were not
possible before.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/PMG/RuleCache.pm     | 164 +++++++++++++++++++++++++++++++++++++--
 src/PMG/RuleDB/Remove.pm |  12 ++-
 2 files changed, 167 insertions(+), 9 deletions(-)

diff --git a/src/PMG/RuleCache.pm b/src/PMG/RuleCache.pm
index d0fa1f8..14da88f 100644
--- a/src/PMG/RuleCache.pm
+++ b/src/PMG/RuleCache.pm
@@ -332,29 +332,146 @@ sub what_match {
 	return ($marks, $spaminfo);
     }
 
+    my $what_matches = {};
+
     for my $group ($what->{groups}->@*) {
+	my $group_matches = {};
+	my $and = $group->{and};
+	my $invert = $group->{invert};
 	for my $obj ($group->{objects}->@*) {
 	    if (!$obj->can('what_match_targets')) {
-		if (my $match = $obj->what_match($queue, $element, $msginfo, $dbh)) {
-		    for my $target ($msginfo->{targets}->@*) {
-			push $marks->{$target}->@*, $match->@*;
+		my $match = $obj->what_match($queue, $element, $msginfo, $dbh);
+		for my $target ($msginfo->{targets}->@*) {
+		    if (defined($match)) {
+			push $group_matches->{$target}->@*, $match;
+		    } else {
+			push $group_matches->{$target}->@*, undef;
 		    }
 		}
 	    } else {
-		if (my $target_info = $obj->what_match_targets($queue, $element, $msginfo, $dbh)) {
-		    foreach my $k (keys $target_info->%*) {
-			push $marks->{$k}->@*, $target_info->{$k}->{marks}->@*;
+		my $target_info = $obj->what_match_targets($queue, $element, $msginfo, $dbh);
+		for my $target ($msginfo->{targets}->@*) {
+		    my $match = $target_info->{$target};
+		    if (defined($match)) {
+			push $group_matches->{$target}->@*, $match->{marks};
 			# only save spaminfo once
-			$spaminfo = $target_info->{$k}->{spaminfo} if !defined($spaminfo);
+			$spaminfo = $match->{spaminfo} if !defined($spaminfo);
+		    } else {
+			push $group_matches->{$target}->@*, undef;
 		    }
 		}
 	    }
 	}
+
+	for my $target (keys $group_matches->%*) {
+	    my $matches = group_match_and_invert($group_matches->{$target}, $and, $invert, $msginfo);
+	    push $what_matches->{$target}->@*, $matches;
+	}
+    }
+
+    for my $target (keys $what_matches->%*) {
+	my $target_marks = what_match_and_invert($what_matches->{$target}, $what->{and}, $what->{invert});
+	$marks->{$target} = $target_marks;
     }
 
     return ($marks, $spaminfo);
 }
 
+# combines matches of groups
+# this is only binary, and if it matches, 'or' combines the marks
+# so that all found marks are included
+#
+# this way we can create rules like:
+#
+# ---
+# What is and combined:
+# group1: match filename .*\.pdf
+# group2: spamlevel >= 3
+# ACTION: remove attachments
+# ---
+# which would remove attachments for all *.pdf filenames where
+# the spamlevel is >= 3
+sub what_match_and_invert($$$) {
+    my ($matches, $and, $invert) = @_;
+
+    my $match_result = match_list_with_mode($matches, $and, $invert, sub {
+	my ($match) = @_;
+	return defined($match);
+    });
+
+    if ($match_result) {
+	my $res = [];
+	for my $match ($matches->@*) {
+	    push $res->@*, $match->@* if defined($match);
+	}
+	return $res;
+    } else {
+	return undef;
+    }
+}
+
+# combines group matches according to and/invert
+# since we want match groups per mime part, we must
+# look at the marks and possibly invert them
+sub group_match_and_invert($$$$) {
+    my ($group_matches, $and, $invert, $msginfo) = @_;
+
+    my $encountered_parts = 0;
+    if ($and) {
+	my $set = {};
+	my $count = scalar($group_matches->@*);
+	for my $match ($group_matches->@*) {
+	    if (!defined($match)) {
+		$set = {};
+		last;
+	    }
+
+	    if (scalar($match->@*) > 0) {
+		$encountered_parts = 1;
+		$set->{$_}++ for $match->@*;
+	    } else {
+		$set->{$_}++ for (1..$msginfo->{max_aid});
+	    }
+	}
+
+	$group_matches = undef;
+	for my $key (keys $set->%*) {
+	    if ($set->{$key} == $count) {
+		push $group_matches->@*, $key;
+	    }
+	}
+	if (defined($group_matches) && scalar($group_matches->@*) == $count && !$encountered_parts) {
+	    $group_matches = [];
+	}
+    } else {
+	my $set = {};
+	for my $match ($group_matches->@*) {
+	    next if !defined($match);
+	    if (scalar($match->@*) == 0) {
+		$set->{$_} = 1 for (1..$msginfo->{max_aid});
+	    } else {
+		$encountered_parts = 1;
+		$set->{$_} = 1 for $match->@*;
+	    }
+	}
+
+	my $count = scalar(keys $set->%*);
+	if ($count == $msginfo->{max_aid} && !$encountered_parts) {
+	    $group_matches = [];
+	} elsif ($count == 0) {
+	    $group_matches = undef;
+	} else {
+	    $group_matches = [keys $set->%*];
+	}
+    }
+
+    if ($invert) {
+	$group_matches = invert_mark_list($group_matches, $msginfo->{max_aid});
+    }
+
+    return $group_matches;
+}
+
 # calls sub with each element of $list, and and/ors/inverts the result
 sub match_list_with_mode($$$$) {
     my ($list, $and, $invert, $sub) = @_;
@@ -374,4 +491,37 @@ sub match_list_with_mode($$$$) {
     return $and != $invert;
 }
 
+# inverts a list of marks with the remaining ones of the mail
+# examples:
+# mail has [1,2,3,4,5]
+#
+# undef => [1,2,3,4,5]
+# [1,2] => [3,4,5]
+# [1,2,3,4,5] => undef
+# [] => undef // [] means the whole mail matched
+sub invert_mark_list($$) {
+    my ($list, $max_aid) = @_;
+
+    if (defined($list)) {
+	my $length = scalar($list->@*);
+	if ($length == 0 || $length == ($max_aid - 1)) {
+	    return undef;
+	}
+    }
+
+    $list //= [];
+
+    my $set = {};
+    $set->{$_} = 1 for $list->@*;
+
+    my $new_list = [];
+    for (my $i = 1; $i <= $max_aid; $i++) {
+	if (!$set->{$i}) {
+	    push $new_list->@*, $i;
+	}
+    }
+
+    return $new_list;
+}
+
 1;
diff --git a/src/PMG/RuleDB/Remove.pm b/src/PMG/RuleDB/Remove.pm
index 3acc861..7cc06b1 100644
--- a/src/PMG/RuleDB/Remove.pm
+++ b/src/PMG/RuleDB/Remove.pm
@@ -209,7 +209,14 @@ sub execute {
 	return if !$found_mark;
     }
 
-    my $subgroups = $mod_group->subgroups ($targets);
+    my $subgroups;
+    if ($marks->{spaminfo}) {
+	# when there was a spam check in the rule, we might have different marks for
+	# different targets, so simply copy the mail for each target that matches
+	$subgroups = $mod_group->explode($targets);
+    } else {
+	$subgroups = $mod_group->subgroups ($targets);
+    }
 
     my $html = PMG::Utils::subst_values($self->{text}, $vars);
 
@@ -263,7 +270,8 @@ sub execute {
 
 	$self->{message_seen} = 0;
 
-	# since currently all marks are equal for all target, just use the first one
+	# if we only had a spam/virus check, the marks are identical
+	# otherwise we get a subgroup per target anyway
 	my $match_marks = $marks->{$tg->[0]};
 
 	$self->delete_marked_parts($queue, $entity, $html, $rtype, $match_marks, $rulename);
-- 
2.30.2





  parent reply	other threads:[~2024-02-21 12:25 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-02-21 12:24 [pmg-devel] [PATCH pmg-api/docs/gui v2] implement and combination and inversion of groups and objects Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 01/10] RuleCache: reorganize how we gather marks and spaminfo Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 02/10] api: refactor rule parameters Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 03/10] add objectgroup attributes and/invert Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 04/10] add rule attributes and/invert (for each relevant type) Dominik Csapak
2024-02-22  6:46   ` Thomas Lamprecht
2024-02-22  7:34     ` Dominik Csapak
2024-02-22  7:38       ` Thomas Lamprecht
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 05/10] RuleCache: load rule/objectgroup attributes from database Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 06/10] RuleCache: implement and/invert for when/from/to Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 07/10] MailQueue: return maximum AID Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 08/10] ModGroup: add possibility to explode to all targets Dominik Csapak
2024-02-21 12:24 ` Dominik Csapak [this message]
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-api v2 10/10] pmgdb: extend dump output to include add/invert Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-docs v2 1/1] rule system: explain new and mode and invert flag Dominik Csapak
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-gui v2 1/2] rules: use tree panel instead of grouping feature of the grid Dominik Csapak
2024-02-21 17:42   ` Thomas Lamprecht
2024-02-21 12:24 ` [pmg-devel] [PATCH pmg-gui v2 2/2] rules/objects: add mode selector dropdown Dominik Csapak
2024-02-21 18:31   ` Thomas Lamprecht
2024-02-21 18:36 ` [pmg-devel] applied-partially: [PATCH pmg-api/docs/gui v2] implement and combination and inversion of groups and objects Stoiko Ivanov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240221122439.1281024-10-d.csapak@proxmox.com \
    --to=d.csapak@proxmox.com \
    --cc=pmg-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal