[pmg-devel] [PATCH pmg-api v3 0/6] add support for locally configured SA channels

[pmg-devel] [PATCH pmg-api v3 0/6] add support for locally configured SA channels

[Stoiko Ivanov]

v2->v3:
* incorporated Thomas' feedback (Thanks!)
* added minimal tests for the sa-channel file parser
* fixed two small glitches in the Makefiles (new patches 1 and 2) - noticed
  while adding the tests

original cover-letter for v2:
v1->v2:
* addressed Fabian's feedback (where appropriate more details are added to the
individual patches)

original cover-letter:

With the recent announcement by KAM about the availability of a signed
update channel for the KAM.cf (and other) ruleset [0], I thought that this
might be a better suited way to provide regular updates to KAM.cf compared
to the initial patch I sent some time ago [1].

The upside of using sa-update is that it does lint the ruleset before trying
to pull the updates, and afterwards as well. GPG signed updates by KAM can
also be considered production worthy.

I tried to align the implementation to the channel.d mechanism mentioned in
[0] since it seems:
* somewhat sensible (these days my first choice would not be shell-code
  snippets)
* one mechanism of potentially having a distributable way of providing
  the necessary configuration for an external SA ruleset - and the only one
  I'm aware of, which is not a self-tailored script

The patchset was tested with the file provided at [0] on my test
installation

Should this be accepted we could ship
/etc/mail/spamassassin/channel.d/KAM_channel.conf in proxmox-spamassassin


[0] https://mcgrail.com/template/kam.cf_channel
[1] https://lists.proxmox.com/pipermail/pmg-devel/2020-November/001397.html


Stoiko Ivanov (6):
  buildsys: fix check target in main Makefile
  buildsys: fix PERLLIB setting in tests/Makefile
  add helper for parsing SA channel.d files
  api: spamassassin: read local channels
  api: spamassassin: update local channels
  pmg-daily: run sa-update for local channels

 Makefile                               |  2 +-
 src/PMG/API2/SpamAssassin.pm           | 89 +++++++++++++++-----------
 src/PMG/Utils.pm                       | 60 +++++++++++++++++
 src/bin/pmg-daily                      |  9 ++-
 src/tests/KAM_channel.conf             | 34 ++++++++++
 src/tests/Makefile                     |  3 +-
 src/tests/missing_gpg_key_channel.conf |  2 +
 src/tests/missing_keyid.conf           |  7 ++
 src/tests/test_sa_channel_parser.pl    | 75 ++++++++++++++++++++++
 9 files changed, 241 insertions(+), 40 deletions(-)
 create mode 100644 src/tests/KAM_channel.conf
 create mode 100644 src/tests/missing_gpg_key_channel.conf
 create mode 100644 src/tests/missing_keyid.conf
 create mode 100755 src/tests/test_sa_channel_parser.pl

-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 1/6] buildsys: fix check target in main Makefile

[Stoiko Ivanov]

the tests are in src/ - the test target in the main Makefile still
tries to run them in the main directory.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index f6bf7c6..081676a 100644
--- a/Makefile
+++ b/Makefile
@@ -27,7 +27,7 @@ upload: ${DEB}
 
 .PHONY: check
 check:
-	make -C tests check
+	make -C src/tests check
 
 .PHONY: clean distclean
 distclean: clean
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 2/6] buildsys: fix PERLLIB setting in tests/Makefile

[Stoiko Ivanov]

Make variables don't need quoting - and with quotes the PERLLIB is
literally set to ".." instead of .. (only the latter works)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/tests/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/tests/Makefile b/src/tests/Makefile
index 84075c4..79310b3 100644
--- a/src/tests/Makefile
+++ b/src/tests/Makefile
@@ -1,6 +1,6 @@
 #export TESTDB = "Proxmox_testdb"
 
-export PERLIB = ".."
+export PERLLIB = ..
 
 all:
 
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 3/6] add helper for parsing SA channel.d files

[Stoiko Ivanov]

RHEL/CentOS based SpamAssassin implementations ship an update script,
which reads shell snippets from
/etc/mail/spamassassin/channel.d/*.conf and uses the information there
to update SA rules from the configured channels [0].

Noticed the existence of this directory/mechanism while reading the
announcement of the updatechannel for the KAM ruleset [1].

Parsing the file as text, instead of sourcing it in a shell, since I
hope that the channel files distributed don't rely on running commands
to get the ruleset url and gpg key.

The parser has some minimal tests added (inspired by the
convert_size_test.pl from pve-common)

[0] https://src.fedoraproject.org/rpms/spamassassin/blob/master/f/sa-update.cronscript
[1] https://mcgrail.com/template/kam.cf_channel

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
v2->v3:
* refactored read_sa_channel into a sub directly in PMG::Utils to make
  it testable
* added tests (hopefully I copied from an appropriate place ;)


 src/PMG/Utils.pm                       | 32 +++++++++++
 src/tests/KAM_channel.conf             | 34 ++++++++++++
 src/tests/Makefile                     |  1 +
 src/tests/missing_gpg_key_channel.conf |  2 +
 src/tests/missing_keyid.conf           |  7 +++
 src/tests/test_sa_channel_parser.pl    | 75 ++++++++++++++++++++++++++
 6 files changed, 151 insertions(+)
 create mode 100644 src/tests/KAM_channel.conf
 create mode 100644 src/tests/missing_gpg_key_channel.conf
 create mode 100644 src/tests/missing_keyid.conf
 create mode 100755 src/tests/test_sa_channel_parser.pl

diff --git a/src/PMG/Utils.pm b/src/PMG/Utils.pm
index d3fae9e..e3863b0 100644
--- a/src/PMG/Utils.pm
+++ b/src/PMG/Utils.pm
@@ -1442,5 +1442,37 @@ sub domain_regex {
     return $regex;
 }
 
+sub read_sa_channel {
+    my ($filename) = @_;
+
+    my $content = PVE::Tools::file_get_contents($filename);
+    my $channel = {
+	filename => $filename,
+    };
+
+    ($channel->{keyid}) = ($content =~ /^KEYID=([a-fA-F0-9]+)$/m);
+    die "no KEYID in $filename!\n" if !defined($channel->{keyid});
+    ($channel->{channelurl}) = ($content =~ /^CHANNELURL=(.+)$/m);
+    die "no CHANNELURL in $filename!\n" if !defined($channel->{channelurl});
+    ($channel->{gpgkey}) = ($content =~ /(?:^|\n)(-----BEGIN PGP PUBLIC KEY BLOCK-----.+-----END PGP PUBLIC KEY BLOCK-----)(?:\n|$)/s);
+    die "no GPG public key in $filename!\n" if !defined($channel->{gpgkey});
+
+    return $channel;
+};
+
+sub local_spamassassin_channels {
+
+    my $res = [];
+
+    my $local_channel_dir = '/etc/mail/spamassassin/channel.d/';
+
+    PVE::Tools::dir_glob_foreach($local_channel_dir, '.*\.conf', sub {
+	my ($filename) = @_;
+	my $channel = read_sa_channel($local_channel_dir.$filename);
+	push(@$res, $channel);
+    });
+
+    return $res;
+}
 
 1;
diff --git a/src/tests/KAM_channel.conf b/src/tests/KAM_channel.conf
new file mode 100644
index 0000000..50b8bc8
--- /dev/null
+++ b/src/tests/KAM_channel.conf
@@ -0,0 +1,34 @@
+CHANNELURL=kam.sa-channels.mcgrail.com
+KEYID=24C063D8
+# Ignore everything below.
+return 0
+
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBF96bE0BEADsT1xRD2l19kmUSg9XMfRUtJbMGa9YAQ0a2fayT9IdmR38J4o3
+Ln2fIR0CMa81Q+mi7pSdTpHGqR3t5GjmDGcCN8kwoHbmm0t5F9gK0tFAXThf+e40
+kMdzLNzled4+5D83VyKCNaPm1tmogzYKKIEzTHCqQ7TdahWZDRDFiZJWFkd/9miE
+kURY2uWLCttF+4Aa2AOHUg/7q00NSR8S0jWpLzpVNjbgi/jjkCafhpSZ56aqXHk3
+QrTwJj3sznrLb9TkVZoXFKbBCh15m7mf5VVJVEZpj3BsvbcZJPnBFkCrzPjfShRz
+lttRyiCFflOIcDrClg62tA/a1BmdUuIB5ktdCX8gB0F4t+9MhqgF89vT/OQpxywv
+/QmuvKZzl77TQcLFHDlS+TKjLI6RdM3xuto1B8aSIYpKslnVpYuMpxNsvouAiQig
+5qKBzYMbFCVge8Kjvcs6znxsPyjkCWgZVbf7ev7v+h71kkVfJ2TRR52ty/vsh82c
+LYEaIB8CKYTstf69EOEQEhqMVNfhzuEb22ueYtAQSsnpLgGii0PwAFfSB4puzEUI
+ItJVmD4DviD7ZfZnT8dR2bsysV4BF8s2dKX0KDnBAkzhlc30/iwt8j8bZXx3Evau
+Ci+sFvBRMbpJJbVH8AJT7/dImn1ZqbK7jaZkFMticGBBWaKee8NYmF+KKwARAQAB
+tDdLZXZpbiBBLiBNY0dyYWlsIChLQU0gQ2hhbm5lbCkgPGthbWNoYW5uZWxAbWNn
+cmFpbC5jb20+iQJOBBMBCAA4FiEEIdlxQicskGb8qnkrShVtpSTAY9gFAl96bE0C
+GwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQShVtpSTAY9hQZRAA5i8RkBCH
+zjY/xHAoIUa4u9Di52I8t8IKHuIbH5a1TfShT8uj38ucmc/gRWMoOu1Tef9G2DdJ
+FQc7KOA9GcGyGl1C2gfoTJEqBSNJTgJVfmHQ1Ef0ucNSjYFD3H0eFGTIuoSFy3Mi
+g7CzxfhIJXIn4JW9sNwICH/7pOLke5Ihd5WvyOqU13FrfGemRbilviG73HYoy+Fh
+4R9A1MLF3I0zVG5nszfn5CjSVG3c+Buj7Gk1d67noINbhCs2IPnyuOSvfrZc5wx1
+ImCS8BpmGjXqaXZAIWLIhpMXvRiboGxX1zzRZLoz7Y5Y5h1MfnY2ASDMddmJpgOv
+Vey/acAB4+6TtCgXmA6Wy8xmsqlId4qBocxX/jCMJ8OsuueYE6eF2jzS/JfbTndA
+7pHOnCoR+ndMra5vaX8MYyGKqxxWyBoKWGgeBs8fSMwHAqRIo9GHWK67nBX0x39U
+x9G0yn/A2dhaGqhui8xrcAHg/OGJErOlDw7YBeVX0RiS6awPyk9fo0IsGN0po2VX
+bd9H8DKz1CXBLNZRG0vn5mViSOBzZeGU+K9aAs58GZ46LKA3YfWJ4s5W8BS+J3Ia
+TFpq8U+OO/BSmOkMHZ+OPKWSlxNitFTyQsIdtS1PfqqYc+MK312LdmvrG2KWXE3N
+EnuBffLm6uSOHJA6/0r6THJkffDSuvqM5yU=
+=GVCC
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/src/tests/Makefile b/src/tests/Makefile
index 79310b3..dc35796 100644
--- a/src/tests/Makefile
+++ b/src/tests/Makefile
@@ -10,6 +10,7 @@ check:
 	./print_testdb.pl > testdb.txt.new
 	diff -u testdb.txt testdb.txt.new
 	./test_greylist.pl
+	./test_sa_channel_parser.pl
 
 #	test_config.pl		\
 #	test_mimetype.pl	\
diff --git a/src/tests/missing_gpg_key_channel.conf b/src/tests/missing_gpg_key_channel.conf
new file mode 100644
index 0000000..2bf59f3
--- /dev/null
+++ b/src/tests/missing_gpg_key_channel.conf
@@ -0,0 +1,2 @@
+CHANNELURL=missing.gpg.key.invalid
+KEYID=AAAAAAAA
diff --git a/src/tests/missing_keyid.conf b/src/tests/missing_keyid.conf
new file mode 100644
index 0000000..fe4eacf
--- /dev/null
+++ b/src/tests/missing_keyid.conf
@@ -0,0 +1,7 @@
+CHANNELURL=kam.sa-channels.mcgrail.com
+# Ignore everything below.
+return 0
+
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/src/tests/test_sa_channel_parser.pl b/src/tests/test_sa_channel_parser.pl
new file mode 100755
index 0000000..fcbb859
--- /dev/null
+++ b/src/tests/test_sa_channel_parser.pl
@@ -0,0 +1,75 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+use Test::More;
+
+use PMG::Utils;
+
+my $kam_key = qq{-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBF96bE0BEADsT1xRD2l19kmUSg9XMfRUtJbMGa9YAQ0a2fayT9IdmR38J4o3
+Ln2fIR0CMa81Q+mi7pSdTpHGqR3t5GjmDGcCN8kwoHbmm0t5F9gK0tFAXThf+e40
+kMdzLNzled4+5D83VyKCNaPm1tmogzYKKIEzTHCqQ7TdahWZDRDFiZJWFkd/9miE
+kURY2uWLCttF+4Aa2AOHUg/7q00NSR8S0jWpLzpVNjbgi/jjkCafhpSZ56aqXHk3
+QrTwJj3sznrLb9TkVZoXFKbBCh15m7mf5VVJVEZpj3BsvbcZJPnBFkCrzPjfShRz
+lttRyiCFflOIcDrClg62tA/a1BmdUuIB5ktdCX8gB0F4t+9MhqgF89vT/OQpxywv
+/QmuvKZzl77TQcLFHDlS+TKjLI6RdM3xuto1B8aSIYpKslnVpYuMpxNsvouAiQig
+5qKBzYMbFCVge8Kjvcs6znxsPyjkCWgZVbf7ev7v+h71kkVfJ2TRR52ty/vsh82c
+LYEaIB8CKYTstf69EOEQEhqMVNfhzuEb22ueYtAQSsnpLgGii0PwAFfSB4puzEUI
+ItJVmD4DviD7ZfZnT8dR2bsysV4BF8s2dKX0KDnBAkzhlc30/iwt8j8bZXx3Evau
+Ci+sFvBRMbpJJbVH8AJT7/dImn1ZqbK7jaZkFMticGBBWaKee8NYmF+KKwARAQAB
+tDdLZXZpbiBBLiBNY0dyYWlsIChLQU0gQ2hhbm5lbCkgPGthbWNoYW5uZWxAbWNn
+cmFpbC5jb20+iQJOBBMBCAA4FiEEIdlxQicskGb8qnkrShVtpSTAY9gFAl96bE0C
+GwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQShVtpSTAY9hQZRAA5i8RkBCH
+zjY/xHAoIUa4u9Di52I8t8IKHuIbH5a1TfShT8uj38ucmc/gRWMoOu1Tef9G2DdJ
+FQc7KOA9GcGyGl1C2gfoTJEqBSNJTgJVfmHQ1Ef0ucNSjYFD3H0eFGTIuoSFy3Mi
+g7CzxfhIJXIn4JW9sNwICH/7pOLke5Ihd5WvyOqU13FrfGemRbilviG73HYoy+Fh
+4R9A1MLF3I0zVG5nszfn5CjSVG3c+Buj7Gk1d67noINbhCs2IPnyuOSvfrZc5wx1
+ImCS8BpmGjXqaXZAIWLIhpMXvRiboGxX1zzRZLoz7Y5Y5h1MfnY2ASDMddmJpgOv
+Vey/acAB4+6TtCgXmA6Wy8xmsqlId4qBocxX/jCMJ8OsuueYE6eF2jzS/JfbTndA
+7pHOnCoR+ndMra5vaX8MYyGKqxxWyBoKWGgeBs8fSMwHAqRIo9GHWK67nBX0x39U
+x9G0yn/A2dhaGqhui8xrcAHg/OGJErOlDw7YBeVX0RiS6awPyk9fo0IsGN0po2VX
+bd9H8DKz1CXBLNZRG0vn5mViSOBzZeGU+K9aAs58GZ46LKA3YfWJ4s5W8BS+J3Ia
+TFpq8U+OO/BSmOkMHZ+OPKWSlxNitFTyQsIdtS1PfqqYc+MK312LdmvrG2KWXE3N
+EnuBffLm6uSOHJA6/0r6THJkffDSuvqM5yU=
+=GVCC
+-----END PGP PUBLIC KEY BLOCK-----};
+
+my $tests = [
+    [
+	'./KAM_channel.conf', # input filename
+	{                   # result structure
+	    filename => './KAM_channel.conf',
+	    channelurl => 'kam.sa-channels.mcgrail.com',
+	    keyid => '24C063D8',
+	    gpgkey => $kam_key,
+	},
+	undef,              # error string
+    ],
+    [
+	'./missing_gpg_key_channel.conf',
+	undef,
+	'no GPG public key in ./missing_gpg_key_channel.conf!',
+    ],
+    [
+	'./missing_keyid.conf',
+	undef,
+	'no KEYID in ./missing_keyid.conf!',
+    ],
+];
+
+foreach my $test (@$tests) {
+    my ($filename, $expect, $error) = @$test;
+
+    my $result = eval { PMG::Utils::read_sa_channel($filename); };
+    my $err = $@;
+
+    if ($error) {
+	like($err, qr/^\Q$error\E/, "expected error for $filename: $error");
+    } else {
+	is_deeply($result, $expect, "channel file: $filename parsed correctly");
+    }
+}
+
+done_testing();
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 4/6] api: spamassassin: read local channels

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/API2/SpamAssassin.pm | 83 +++++++++++++++++++++---------------
 1 file changed, 49 insertions(+), 34 deletions(-)

diff --git a/src/PMG/API2/SpamAssassin.pm b/src/PMG/API2/SpamAssassin.pm
index eab02d9..6b9f8f9 100644
--- a/src/PMG/API2/SpamAssassin.pm
+++ b/src/PMG/API2/SpamAssassin.pm
@@ -80,50 +80,65 @@ __PACKAGE__->register_method({
 	my ($param) = @_;
 
 	my $saversion = $Mail::SpamAssassin::VERSION;
-	my $channelfile = "/var/lib/spamassassin/$saversion/updates_spamassassin_org.cf";
+	my $sa_update_dir = "/var/lib/spamassassin/$saversion/";
+
+	my $check_channel = sub {
+	    my ($channel) = @_;
+
+	    # see sa-update source:
+	    my $channel_file_base = $channel;
+	    $channel_file_base =~ s/[^A-Za-z0-9-]+/_/g;
+	    my $channelfile = "${sa_update_dir}${channel_file_base}.cf";
+
+	    my $mtime = -1;
+	    my $version = -1;
+	    my $newversion = -1;
+
+	    if (-f $channelfile) {
+		# stat metadata cf file
+		$mtime = (stat($channelfile))[9]; # 9 is mtime
+
+		# parse version from metadata cf file
+		my $metadata = PVE::Tools::file_read_firstline($channelfile);
+		if ($metadata =~ m/\s([0-9]+)$/) {
+		    $version = $1;
+		} else {
+		    warn "invalid metadata in '$channelfile'\n";
+		}
+	    }
+	    # call sa-update to see if updates are available
 
-	my $mtime = -1;
-	my $version = -1;
-	my $newversion = -1;
+	    my $cmd = "$SAUPDATE -v --checkonly --channel $channel";
+	    PVE::Tools::run_command($cmd, noerr => 1, logfunc => sub {
+		my ($line) = @_;
 
-	if (-f $channelfile) {
-	    # stat metadata cf file
-	    $mtime = (stat($channelfile))[9]; # 9 is mtime
+		if ($line =~ m/Update available for channel \S+: -?[0-9]+ -> ([0-9]+)/) {
+		    $newversion = $1;
+		}
+	    });
 
-	    # parse version from metadata cf file
-	    my $metadata = PVE::Tools::file_read_firstline($channelfile);
-	    if ($metadata =~ m/\s([0-9]+)$/) {
-		$version = $1;
-	    } else {
-		warn "invalid metadata in '$channelfile'\n";
-	    }
-	}
-	# call sa-update to see if updates are available
+	    my $result = {
+		channel => $channel,
+	    };
 
-	my $cmd = "$SAUPDATE -v --checkonly";
-	PVE::Tools::run_command($cmd, noerr => 1, logfunc => sub {
-	    my ($line) = @_;
+	    $result->{version} = $version if $version > -1;
+	    $result->{update_version} = $newversion if $newversion > -1;
+	    $result->{last_updated} = $mtime if $mtime > -1;
 
-	    if ($line =~ m/Update available for channel \S+: -?[0-9]+ -> ([0-9]+)/) {
-		$newversion = $1;
+	    if ($newversion > $version) {
+		$result->{update_avail} = 1;
+	    } else {
+		$result->{update_avail} = 0;
 	    }
-	});
-
-	my $result = {
-	    channel => 'updates.spamassassin.org',
+	    return $result;
 	};
 
-	$result->{version} = $version if $version > -1;
-	$result->{update_version} = $newversion if $newversion > -1;
-	$result->{last_updated} = $mtime if $mtime > -1;
+	my @channels = ('updates.spamassassin.org');
 
-	if ($newversion > $version) {
-	    $result->{update_avail} = 1;
-	} else {
-	    $result->{update_avail} = 0;
-	}
+	my $localchannels = PMG::Utils::local_spamassassin_channels();
+	push(@channels, map { $_->{channelurl} } @$localchannels);
 
-	return [$result];
+	return [ map { $check_channel->($_) } @channels];
     }});
 
 __PACKAGE__->register_method({
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 5/6] api: spamassassin: update local channels

[Stoiko Ivanov]

This patch adds a helper to loop over all present Spamassassin
channels files in /etc/mail/spamassassin/channel.d and:
* import the included gpg key into sa-update's keyring
* run sa-update for each channel separately

the verbose argument of the helper is for reusing the code in
pmg-daily (where we only want to log errors and be less informative)

the $SA_UPDATE variable hardcoding the path of /usr/bin/sa-update was
dropped in favor of using 'sa-update' without path since we do have a
sensible setting of PATH everywhere, and hardcoding paths is
problematic (especially in usr-merged systems).

The choice of invoking sa-update for each channel separately, instead
of providing multiple '--channel' and '--gpgkey' options to a single
command was made to prevent downloading signatures, which were signed
by a key not configured for the channel.

Importing gpg-keys is also done with individual sa-update invocations,
because sa-update only imports the last present --import argument
(wrong use of Getopt::Long)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/API2/SpamAssassin.pm |  8 ++++----
 src/PMG/Utils.pm             | 28 ++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/src/PMG/API2/SpamAssassin.pm b/src/PMG/API2/SpamAssassin.pm
index 6b9f8f9..441fd7e 100644
--- a/src/PMG/API2/SpamAssassin.pm
+++ b/src/PMG/API2/SpamAssassin.pm
@@ -18,8 +18,6 @@ use Mail::SpamAssassin;
 
 use base qw(PVE::RESTHandler);
 
-my $SAUPDATE = '/usr/bin/sa-update';
-
 __PACKAGE__->register_method ({
     name => 'index',
     path => '',
@@ -108,7 +106,7 @@ __PACKAGE__->register_method({
 	    }
 	    # call sa-update to see if updates are available
 
-	    my $cmd = "$SAUPDATE -v --checkonly --channel $channel";
+	    my $cmd = "sa-update -v --checkonly --channel $channel";
 	    PVE::Tools::run_command($cmd, noerr => 1, logfunc => sub {
 		my ($line) = @_;
 
@@ -171,9 +169,11 @@ __PACKAGE__->register_method({
 		$ENV{http_proxy} = $http_proxy;
 	    }
 
-	    my $cmd = "$SAUPDATE -v";
+	    my $cmd = "sa-update -v";
 
 	    PVE::Tools::run_command($cmd, noerr => 1);
+
+	    PMG::Utils::update_local_spamassassin_channels(1);
 	};
 
 	return $rpcenv->fork_worker('saupdate', undef, $authuser, $realcmd);
diff --git a/src/PMG/Utils.pm b/src/PMG/Utils.pm
index e3863b0..149bcdc 100644
--- a/src/PMG/Utils.pm
+++ b/src/PMG/Utils.pm
@@ -1475,4 +1475,32 @@ sub local_spamassassin_channels {
     return $res;
 }
 
+sub update_local_spamassassin_channels {
+    my ($verbose) = @_;
+    # import all configured channel's gpg-keys to sa-update's keyring
+    my $localchannels = PMG::Utils::local_spamassassin_channels();
+    for my $channel (@$localchannels) {
+	my $importcmd = ['sa-update', '--import', $channel->{filename}];
+	push @$importcmd, '-v' if $verbose;
+
+	print "Importing gpg key from $channel->{filename}\n" if $verbose;
+	PVE::Tools::run_command($importcmd);
+    }
+
+    my $fresh_updates = 0;
+
+    for my $channel (@$localchannels) {
+	my $cmd = ['sa-update', '--channel', $channel->{channelurl}, '--gpgkey', $channel->{keyid}];
+	push @$cmd, '-v' if $verbose;
+
+	print "Updating $channel->{channelurl}\n" if $verbose;
+	my $ret = PVE::Tools::run_command($cmd, noerr => 1);
+	die "updating $channel->{channelurl} failed - sa-update exited with $ret\n" if $ret >= 2;
+
+	$fresh_updates = 1 if $ret == 0;
+    }
+
+    return $fresh_updates
+}
+
 1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 6/6] pmg-daily: run sa-update for local channels

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/bin/pmg-daily | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/src/bin/pmg-daily b/src/bin/pmg-daily
index 32ccb95..8865c94 100755
--- a/src/bin/pmg-daily
+++ b/src/bin/pmg-daily
@@ -73,12 +73,19 @@ if (my $http_proxy = $cfg->get('admin', 'http_proxy')) {
 }
 
 # update spamassassin rules
+my $restart_filter = 0;
 if (system('sa-update') == 0) {
     # if the exit code is 0, new updates were downloaded
     # then restart the pmg-smtp-filter to load the new rules
-    PMG::Utils::service_cmd('pmg-smtp-filter', 'restart');
+    $restart_filter = 1;
 }
 
+eval {
+    $restart_filter ||= PMG::Utils::update_local_spamassassin_channels(0);
+};
+syslog('err', "$@") if $@;
+
+PMG::Utils::service_cmd('pmg-smtp-filter', 'restart') if $restart_filter;
 # run bayes database maintainance
 system('sa-learn --force-expire >/dev/null 2>&1');
 
-- 
2.20.1

[pmg-devel] applied-series: [PATCH pmg-api v3 0/6] add support for locally configured SA channels

[Thomas Lamprecht]

On 19.01.21 11:38, Stoiko Ivanov wrote:
> Stoiko Ivanov (6):
>   buildsys: fix check target in main Makefile
>   buildsys: fix PERLLIB setting in tests/Makefile
>   add helper for parsing SA channel.d files
>   api: spamassassin: read local channels
>   api: spamassassin: update local channels
>   pmg-daily: run sa-update for local channels
> 
>  Makefile                               |  2 +-
>  src/PMG/API2/SpamAssassin.pm           | 89 +++++++++++++++-----------
>  src/PMG/Utils.pm                       | 60 +++++++++++++++++
>  src/bin/pmg-daily                      |  9 ++-
>  src/tests/KAM_channel.conf             | 34 ++++++++++
>  src/tests/Makefile                     |  3 +-
>  src/tests/missing_gpg_key_channel.conf |  2 +
>  src/tests/missing_keyid.conf           |  7 ++
>  src/tests/test_sa_channel_parser.pl    | 75 ++++++++++++++++++++++
>  9 files changed, 241 insertions(+), 40 deletions(-)
>  create mode 100644 src/tests/KAM_channel.conf
>  create mode 100644 src/tests/missing_gpg_key_channel.conf
>  create mode 100644 src/tests/missing_keyid.conf
>  create mode 100755 src/tests/test_sa_channel_parser.pl
> 

applied series, much thanks!