[pmg-devel] [PATCH pmg-api/pmg-gui v2] content-type filter: add content-only option

[pmg-devel] [PATCH pmg-api/pmg-gui v2] content-type filter: add content-only option

[Stoiko Ivanov]

v1->v2:
Based on Domink's feedback reworked the series - it now offers only one
optional new attribute for the content-type filter ('only-content') to
indicate that it should only compare the content-type obtained via
magic/file-conents. The other 2 match options can already be expressed
with the Filenamefilter ('application/pdf' -> '.*\.pdf') and the Match
Field filter (to match the 'Content-type' header)

original cover-letter for v1[0]:
The following patch series was started shortly after the release of PMG 8.1,
but I did not find the time to get it in shape for sending.
They follow the patches for adding 'top' and 'add_separator' to the
Disclaimer action.

The current content-type filter can sometimes surprise users (e.g.
https://bugzilla.proxmox.com/show_bug.cgi?id=5618#c2 and
https://bugzilla.proxmox.com/show_bug.cgi?id=2691#c0 ,but also a few
cases in our technical support-channels come up here and there):

It matches if any of:
* content-type header
* file-magic
* filename (extension)
match the content type, the what-object matches.
by adding an attribute for each of the sources users can restrict which
of the sources should be taken into consideration

the first patches for both repositories are independent (I just ran into
them while looking into this).

minimally tested locally, by sending a plain-text file called
testtext.pdf, and a pdf-file renamed to have a `.docx` suffix.

[0]
https://lore.proxmox.com/pmg-devel/20250212151241.91077-1-s.ivanov@proxmox.com/ 

pmg-api:
Stoiko Ivanov (2):
  ruledb: disclaimer: simplify update-case
  ruledb: content-type: add flag for matching only based on
    magic/content

 src/PMG/RuleDB/ContentTypeFilter.pm | 75 ++++++++++++++++++++++++++---
 src/PMG/RuleDB/Disclaimer.pm        |  8 ++-

pmg-gui:
Stoiko Ivanov (2):
  rules/object: remove icon from remove button
  rules/content-typefilter: add checkbox for file content only matching

 js/ObjectGroup.js |  3 +--
 js/Utils.js       | 11 +++++++++++
 2 files changed, 12 insertions(+), 2 deletions(-)

-- 
2.39.5



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

[pmg-devel] [PATCH pmg-api v2 1/2] ruledb: disclaimer: simplify update-case

[Stoiko Ivanov]

as one object(_id) belongs is of one object type, we can safely remove
all associated attribut entries from the database at once instead of
running 2 separate statements to delete each individually

noticed this while adding new attribut entries to the content-type
check.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/RuleDB/Disclaimer.pm | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/src/PMG/RuleDB/Disclaimer.pm b/src/PMG/RuleDB/Disclaimer.pm
index 574f62e..cb9b5fb 100644
--- a/src/PMG/RuleDB/Disclaimer.pm
+++ b/src/PMG/RuleDB/Disclaimer.pm
@@ -113,11 +113,9 @@ sub save {
 	    "UPDATE Object SET Value = ? WHERE ID = ?",
 	    undef, $value, $self->{id});
 
-	for my $prop (qw(top separator)) {
-	    $ruledb->{dbh}->do(
-		"DELETE FROM Attribut WHERE Name = ? and Object_ID = ?",
-		undef, $prop,  $self->{id});
-	}
+	$ruledb->{dbh}->do(
+	    "DELETE FROM Attribut WHERE Object_ID = ?",
+		undef, $self->{id});
     } else {
 	# insert
 
-- 
2.39.5



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

[pmg-devel] [PATCH pmg-api v2 2/2] ruledb: content-type: add flag for matching only based on magic/content

[Stoiko Ivanov]

our current content-type matching is sensibly quite cautious in
matching if any available information indicates a potential match:
* mime-type detection based on file contents
* mime-type detection based on file suffix
* content-type header

Sometimes this can lead to surprises (e.g. when a MUA sets the
filetype of a pdf to application/octet-stream (the default type if no
information is available), or a filter for zip-files matching
docx-files.

This change gives users the option to restrict matching only on the
content as detected by xdg_mime_get_mime_type_for_data.

This is a fix for the intial request in #2691 and addresses the
suggestion from Friedrich from:
https://bugzilla.proxmox.com/show_bug.cgi?id=5618#c2

matches on the other items can be created with Match Field objects
(for the content-type header) and Filename (for the match based on the
provided filename - combinations of those should give us the complete
flexibility.

inspired by the changes for disclaimer released with PMG 8.1:
51d1507 ("fix #2430: ruledb disclaimer: make separator configurable")

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
would be grateful for suggestions that are a better fit than 'only-content'!

 src/PMG/RuleDB/ContentTypeFilter.pm | 75 ++++++++++++++++++++++++++---
 1 file changed, 68 insertions(+), 7 deletions(-)

diff --git a/src/PMG/RuleDB/ContentTypeFilter.pm b/src/PMG/RuleDB/ContentTypeFilter.pm
index 0199311..550a880 100644
--- a/src/PMG/RuleDB/ContentTypeFilter.pm
+++ b/src/PMG/RuleDB/ContentTypeFilter.pm
@@ -26,7 +26,7 @@ sub otype_text {
 }
 
 sub new {
-    my ($type, $fvalue, $ogroup) = @_;
+    my ($type, $fvalue, $ogroup, $only_content) = @_;
 
     my $class = ref($type) || $type;
 
@@ -36,6 +36,7 @@ sub new {
     }
 
     my $self = $class->SUPER::new('content-type', $fvalue, $ogroup);
+    $self->{only_content} = $only_content;
 
     return $self;
 }
@@ -52,9 +53,50 @@ sub load_attr {
 	$obj->{field_value} = $nt;
     }
 
+    my $sth = $ruledb->{dbh}->prepare(
+	"SELECT * FROM Attribut WHERE Object_ID = ?");
+
+    $sth->execute($id);
+
+    $obj->{only_content} = 0;
+
+    while (my $ref = $sth->fetchrow_hashref()) {
+	if ($ref->{name} eq 'only_content') {
+	    $obj->{only_content} = $ref->{value};
+	}
+    }
+
+    $sth->finish();
+
+    $obj->{id} = $id;
+
+    $obj->{digest} = Digest::SHA::sha1_hex( $id, $value, $ogroup, $obj->{only_content});
+
     return $obj;
 }
 
+sub save {
+    my ($self, $ruledb) = @_;
+
+    if (defined($self->{id})) {
+	#update - clean old attribut entries
+	$ruledb->{dbh}->do(
+	    "DELETE FROM Attribut WHERE Object_ID = ?",
+	    undef, $self->{id});
+    }
+
+    $self->{id} = $self->SUPER::save($ruledb);
+
+    if (defined($self->{only_content})) {
+	$ruledb->{dbh}->do(
+	    "INSERT INTO Attribut (Value, Name, Object_ID) VALUES (?, 'only_content', ?) ".
+	    "ON CONFLICT(Object_ID, Name) DO UPDATE SET Value = Excluded.Value ",
+	    undef, $self->{only_content},  $self->{id});
+    }
+
+    return $self->{id};
+}
+
 sub parse_entity {
     my ($self, $entity) = @_;
 
@@ -78,12 +120,16 @@ sub parse_entity {
 
 	my $glob_ct = $entity->{PMX_glob_ct};
 
-	if ($header_ct && $header_ct =~ m|$self->{field_value}|) {
-	    push @$res, $id;
-	} elsif ($magic_ct && $magic_ct =~ m|$self->{field_value}|) {
-	    push @$res, $id;
-	} elsif ($glob_ct && $glob_ct =~ m|$self->{field_value}|) {
+	my $check_only_content = ${self}->{only_content} // 1;
+
+	if ($magic_ct && $magic_ct =~ m|$self->{field_value}|) {
 	    push @$res, $id;
+	} elsif (!$check_only_content) {
+	    if ($header_ct && $header_ct =~ m|$self->{field_value}|) {
+		push @$res, $id;
+	    } elsif ($glob_ct && $glob_ct =~ m|$self->{field_value}|) {
+		push @$res, $id;
+	    }
 	}
     }
 
@@ -112,19 +158,34 @@ sub properties {
 	    pattern => '[0-9a-zA-Z\/\\\[\]\+\-\.\*\_]+',
 	    maxLength => 1024,
 	},
+	'only-content' => {
+	    description => "use content-type from scanning only (ignore filename and header)",
+	    type => 'boolean',
+	    optional => 1,
+	    default => 0,
+	},
     };
 }
 
 sub get {
     my ($self) = @_;
 
-    return { contenttype => $self->{field_value} };
+    return {
+	contenttype => $self->{field_value},
+	'only-content' => $self->{only_content},
+    };
 }
 
 sub update {
     my ($self, $param) = @_;
 
     $self->{field_value} = $param->{contenttype};
+
+    if (defined($param->{'only-content'}) && $param->{'only-content'} == 1) {
+	$self->{only_content} = 1;
+    } else {
+	delete $self->{only_content};
+    }
 }
 
 1;
-- 
2.39.5



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

Re: [pmg-devel] [PATCH pmg-api v2 2/2] ruledb: content-type: add flag for matching only based on magic/content

[Friedrich Weber]

On 18/02/2025 14:54, Stoiko Ivanov wrote:
> our current content-type matching is sensibly quite cautious in
> matching if any available information indicates a potential match:
> * mime-type detection based on file contents
> * mime-type detection based on file suffix
> * content-type header
> 
> Sometimes this can lead to surprises (e.g. when a MUA sets the
> filetype of a pdf to application/octet-stream (the default type if no
> information is available), or a filter for zip-files matching
> docx-files.
> 
> This change gives users the option to restrict matching only on the
> content as detected by xdg_mime_get_mime_type_for_data.
> 
> This is a fix for the intial request in #2691 and addresses the
> suggestion from Friedrich from:
> https://bugzilla.proxmox.com/show_bug.cgi?id=5618#c2


Thanks for tackling this! I think having a flag like only-content makes
sense.

I tested this a bit and there seems to be one issue, steps to reproduce:

- add a What object with a Content Type Filter for application/pdf,
enable the new "Ignore header information" flag

- create a rule that blocks incoming mails matching this What object

- send an email with a random 1K blob as attachment that sets
Content-Type: application/pdf and some non-descriptive filename for the
attachment:

swaks --from [...] --to [...] --server [...] --attach-type
application/pdf --attach-name foo.bin --attach <(dd if=/dev/urandom
bs=1k count=1)

The email is blocked by the rule. But I would expect it to be accepted,
because the `xdg_mime_get_mime_type_for_data` shouldn't recognize the
random blob as PDF, and the user-provided Content-Type application/pdf
should be ignored.

I think the email is accepted because the magic ct [1] defaults to the
user-provided Content-Type and since `xdg_mime_get_mime_type_for_data`
returns application/octet-stream, we're keep it at the user-provided
Content-Type. I guess it would be nicer if the magic wouldn't default to
the user-provided Content-Type if "Ignore header information" is
enabled, but I'm not sure how easily this can be done.

[1]
https://git.proxmox.com/?p=pmg-api.git;a=blob;f=src/PMG/Utils.pm;h=0b8945f245;hb=6bbc222#l623

> 
> matches on the other items can be created with Match Field objects
> (for the content-type header) and Filename (for the match based on the
> provided filename - combinations of those should give us the complete
> flexibility.
> 
> inspired by the changes for disclaimer released with PMG 8.1:
> 51d1507 ("fix #2430: ruledb disclaimer: make separator configurable")
> 
> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
> ---
> would be grateful for suggestions that are a better fit than 'only-content'!
> 
>  src/PMG/RuleDB/ContentTypeFilter.pm | 75 ++++++++++++++++++++++++++---
>  1 file changed, 68 insertions(+), 7 deletions(-)
> 
> diff --git a/src/PMG/RuleDB/ContentTypeFilter.pm b/src/PMG/RuleDB/ContentTypeFilter.pm
> index 0199311..550a880 100644
> --- a/src/PMG/RuleDB/ContentTypeFilter.pm
> +++ b/src/PMG/RuleDB/ContentTypeFilter.pm
> @@ -26,7 +26,7 @@ sub otype_text {
>  }
>  
>  sub new {
> -    my ($type, $fvalue, $ogroup) = @_;
> +    my ($type, $fvalue, $ogroup, $only_content) = @_;
>  
>      my $class = ref($type) || $type;
>  
> @@ -36,6 +36,7 @@ sub new {
>      }
>  
>      my $self = $class->SUPER::new('content-type', $fvalue, $ogroup);
> +    $self->{only_content} = $only_content;
>  
>      return $self;
>  }
> @@ -52,9 +53,50 @@ sub load_attr {
>  	$obj->{field_value} = $nt;
>      }
>  
> +    my $sth = $ruledb->{dbh}->prepare(
> +	"SELECT * FROM Attribut WHERE Object_ID = ?");
> +
> +    $sth->execute($id);
> +
> +    $obj->{only_content} = 0;
> +
> +    while (my $ref = $sth->fetchrow_hashref()) {
> +	if ($ref->{name} eq 'only_content') {
> +	    $obj->{only_content} = $ref->{value};
> +	}
> +    }
> +
> +    $sth->finish();
> +
> +    $obj->{id} = $id;
> +
> +    $obj->{digest} = Digest::SHA::sha1_hex( $id, $value, $ogroup, $obj->{only_content});
> +
>      return $obj;
>  }
>  
> +sub save {
> +    my ($self, $ruledb) = @_;
> +
> +    if (defined($self->{id})) {
> +	#update - clean old attribut entries
> +	$ruledb->{dbh}->do(
> +	    "DELETE FROM Attribut WHERE Object_ID = ?",
> +	    undef, $self->{id});
> +    }
> +
> +    $self->{id} = $self->SUPER::save($ruledb);
> +
> +    if (defined($self->{only_content})) {
> +	$ruledb->{dbh}->do(
> +	    "INSERT INTO Attribut (Value, Name, Object_ID) VALUES (?, 'only_content', ?) ".
> +	    "ON CONFLICT(Object_ID, Name) DO UPDATE SET Value = Excluded.Value ",
> +	    undef, $self->{only_content},  $self->{id});
> +    }
> +
> +    return $self->{id};
> +}
> +
>  sub parse_entity {
>      my ($self, $entity) = @_;
>  
> @@ -78,12 +120,16 @@ sub parse_entity {
>  
>  	my $glob_ct = $entity->{PMX_glob_ct};
>  
> -	if ($header_ct && $header_ct =~ m|$self->{field_value}|) {
> -	    push @$res, $id;
> -	} elsif ($magic_ct && $magic_ct =~ m|$self->{field_value}|) {
> -	    push @$res, $id;
> -	} elsif ($glob_ct && $glob_ct =~ m|$self->{field_value}|) {
> +	my $check_only_content = ${self}->{only_content} // 1;
> +
> +	if ($magic_ct && $magic_ct =~ m|$self->{field_value}|) {
>  	    push @$res, $id;
> +	} elsif (!$check_only_content) {
> +	    if ($header_ct && $header_ct =~ m|$self->{field_value}|) {
> +		push @$res, $id;
> +	    } elsif ($glob_ct && $glob_ct =~ m|$self->{field_value}|) {
> +		push @$res, $id;
> +	    }
>  	}
>      }
>  
> @@ -112,19 +158,34 @@ sub properties {
>  	    pattern => '[0-9a-zA-Z\/\\\[\]\+\-\.\*\_]+',
>  	    maxLength => 1024,
>  	},
> +	'only-content' => {
> +	    description => "use content-type from scanning only (ignore filename and header)",
> +	    type => 'boolean',
> +	    optional => 1,
> +	    default => 0,
> +	},
>      };
>  }
>  
>  sub get {
>      my ($self) = @_;
>  
> -    return { contenttype => $self->{field_value} };
> +    return {
> +	contenttype => $self->{field_value},
> +	'only-content' => $self->{only_content},
> +    };
>  }
>  
>  sub update {
>      my ($self, $param) = @_;
>  
>      $self->{field_value} = $param->{contenttype};
> +
> +    if (defined($param->{'only-content'}) && $param->{'only-content'} == 1) {
> +	$self->{only_content} = 1;
> +    } else {
> +	delete $self->{only_content};
> +    }
>  }
>  
>  1;



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

Re: [pmg-devel] [PATCH pmg-api v2 2/2] ruledb: content-type: add flag for matching only based on magic/content

[Stoiko Ivanov]

On Tue, Feb 18, 2025 at 06:18:13PM +0100, Friedrich Weber wrote:
> On 18/02/2025 14:54, Stoiko Ivanov wrote:
> > our current content-type matching is sensibly quite cautious in
> > matching if any available information indicates a potential match:
> > * mime-type detection based on file contents
> > * mime-type detection based on file suffix
> > * content-type header
> > 
> > Sometimes this can lead to surprises (e.g. when a MUA sets the
> > filetype of a pdf to application/octet-stream (the default type if no
> > information is available), or a filter for zip-files matching
> > docx-files.
> > 
> > This change gives users the option to restrict matching only on the
> > content as detected by xdg_mime_get_mime_type_for_data.
> > 
> > This is a fix for the intial request in #2691 and addresses the
> > suggestion from Friedrich from:
> > https://bugzilla.proxmox.com/show_bug.cgi?id=5618#c2
> 
> 
> Thanks for tackling this! I think having a flag like only-content makes
> sense.
> 
> I tested this a bit and there seems to be one issue, steps to reproduce:
> 
> - add a What object with a Content Type Filter for application/pdf,
> enable the new "Ignore header information" flag
> 
> - create a rule that blocks incoming mails matching this What object
> 
> - send an email with a random 1K blob as attachment that sets
> Content-Type: application/pdf and some non-descriptive filename for the
> attachment:
> 
> swaks --from [...] --to [...] --server [...] --attach-type
> application/pdf --attach-name foo.bin --attach <(dd if=/dev/urandom
> bs=1k count=1)
> 
> The email is blocked by the rule. But I would expect it to be accepted,
> because the `xdg_mime_get_mime_type_for_data` shouldn't recognize the
> random blob as PDF, and the user-provided Content-Type application/pdf
> should be ignored.
> 
> I think the email is accepted because the magic ct [1] defaults to the
> user-provided Content-Type and since `xdg_mime_get_mime_type_for_data`
> returns application/octet-stream, we're keep it at the user-provided
> Content-Type. I guess it would be nicer if the magic wouldn't default to
> the user-provided Content-Type if "Ignore header information" is
> enabled, but I'm not sure how easily this can be done.
> 
> [1]
> https://git.proxmox.com/?p=pmg-api.git;a=blob;f=src/PMG/Utils.pm;h=0b8945f245;hb=6bbc222#l623

Thanks big-time for the testing, issue-finding and analysis of the cause!
reworked the content-type finding in Utils.pm - after quickly checking
where we rely on that information:
https://lore.proxmox.com/pmg-devel/20250219121851.110090-1-s.ivanov@proxmox.com/T/#t



>..snip..


_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

[pmg-devel] [PATCH pmg-gui v2 1/2] rules/object: remove icon from remove button

[Stoiko Ivanov]

the icons were introduced to the listing, and it seems their rendering
when removing was not noticed - w/o this the message is e.g.:
"Are you sure you want to remove entry
'<span class="fa-fw fa fa-file-image-o'"></span> Content Type Filter:
..."

Fixes: ea4f2a7 ("add icons to the object types")

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 js/ObjectGroup.js | 3 +--
 js/Utils.js       | 5 +++++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/js/ObjectGroup.js b/js/ObjectGroup.js
index 3c8de64..1807e97 100644
--- a/js/ObjectGroup.js
+++ b/js/ObjectGroup.js
@@ -125,8 +125,7 @@ Ext.define('PMG.ObjectGroup', {
 	    },
 	    callback: reload,
 	    getRecordName: function(rec) {
-		return PMG.Utils.format_otype(rec.data.otype) +
-		    ': ' + rec.data.descr;
+		return PMG.Utils.format_otype_subject(rec.data.otype) + ': ' + rec.data.descr;
 	    },
 	    waitMsgTarget: me,
 	});
diff --git a/js/Utils.js b/js/Utils.js
index 9b5f054..94e3c95 100644
--- a/js/Utils.js
+++ b/js/Utils.js
@@ -128,6 +128,11 @@ Ext.define('PMG.Utils', {
 	return icon + text;
     },
 
+    format_otype_subject: function(otype) {
+	let editor = PMG.Utils.object_editors[otype];
+	return editor.subject ?? 'unknown';
+    },
+
     format_otype: function(otype) {
 	let editor = PMG.Utils.object_editors[otype];
 	let iconCls = 'fa fa-question-circle';
-- 
2.39.5



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel

[pmg-devel] [PATCH pmg-gui v2 2/2] rules/content-typefilter: add checkbox for file content only matching

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 js/Utils.js | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/js/Utils.js b/js/Utils.js
index 94e3c95..257226d 100644
--- a/js/Utils.js
+++ b/js/Utils.js
@@ -416,6 +416,12 @@ Ext.define('PMG.Utils', {
 		    allowBlank: false,
 		    reset: Ext.emptyFn,
 		},
+		{
+		    xtype: 'proxmoxcheckbox',
+		    name: 'only-content',
+		    fieldLabel: gettext("Ignore header information"),
+		    uncheckedValue: '0',
+		},
 	    ],
 	},
 	3004: {
-- 
2.39.5



_______________________________________________
pmg-devel mailing list
pmg-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel