[pmg-devel] [PATCH spamassassin/pmg-api ] ship complete KAM ruleset

[pmg-devel] [PATCH spamassassin/pmg-api ] ship complete KAM ruleset

[Stoiko Ivanov]

A few warnings were observed, when upgrading to proxmox-spamassassin 4.0
and to a pmg-api > 7.2-5:
* the sa-rules we ship in proxmox-spamassassin still recorded SA version
  3.4.6 and spamassassin warned about that - this is fixed independently,
  by pushing updates SA rules directly (after running our
  regression_tests)
* the KAM.cf file as of recently depends on the other files in the
  complete ruleset as well:
  ```
  WARNING: HashBL: PCCC_HASHBL_FREEMAIL acl 'freemail' not defined
  ```

This patchset addresses the latter issue by shipping the complete KAM
ruleset with each proxmox-spamassassin package.
The download using sa-update is based on the code in pmg-api, which does
the same in pmg-daily

I'm sending only the actual code and doc changes - and, if approved, would
push the actual rule-update and removal directly instead of sending
multiple megabytes of spamassassin rule definitions (which would not pass
our mail-gateway anyways :)

testing this is best done by applying the patches and running
`make update-kam`

With using kam_sa-channels_mcgrail_com.cf to include all files in the
ruleset the adaptation of the local.cf template (pmg-api patch 1/1) would
need a versioned dependeny bump on proxmox-spamassassin

In my tests spamassassin (pmg-smtp-filter) quite happily ignored the
missing /usr/share/spamassassin-extra/KAM.cf file - so a Breaks in the
other direction should not be needed.

proxmox-spamassassin:
Stoiko Ivanov (2):
  kam-rules: ship full ruleset not only KAM.cf
  buildsys: update outdated docs on updating rulesets

 Makefile | 41 +++++++++++++++++------------------------
 1 file changed, 17 insertions(+), 24 deletions(-)

pmg-api:
Stoiko Ivanov (1):
  templates: adapt to new path for KAM rules in proxmox-spamassassin

 src/templates/local.cf.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.30.2

[pmg-devel] [PATCH spamassassin 1/2] kam-rules: ship full ruleset not only KAM.cf

[Stoiko Ivanov]

The KAM ruleset consists of more files than just KAM.cf - and with the
enabling of the HashBL module in SA 4.0.0 the KAM.cf depends on
settings in other files from the ruleset. This leads to warnings when
starting pmg-smtp-filter on a freshly installed system (more exactly
on a system where pmg-daily did not already download the complete KAM
ruleset to /var/lib)

This patch follows the logic we use in pmg-daily and ships the
complete ruleset in /usr/share/spamassassin-extra

KAM.cf itself was also moved to the kam.sa-channels.mcgrail.org
subdirectory so that we can use the downloaded cf file to get the
correct includes

Given that the default installations from PMG 6.4 on download and use
the complete ruleset the potential for regression should be quite
small.

I shortly considered adding a symlink from KAM.cf pointing to
kam_sa-channels_mcgrail_com/KAM.cf so that users with unadapted
templates still load at least KAM.cf - but since SpamAssassin simply
ignores non-existing files, and most setups, which have modified
templates probably ran pmg-daily at some point resulting in the files
from /var/lib/spamassassin being preferred I left that out.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 Makefile | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index 65474a1..a0f6bb9 100644
--- a/Makefile
+++ b/Makefile
@@ -26,7 +26,9 @@ DBG_DEB = proxmox-spamassassin-dbgsym_${DEB_VERSION_UPSTREAM_REVISION}_amd64.deb
 DEBS = ${DEB} ${DBG_DEB}
 DSC = proxmox-spamassassin_${DEB_VERSION_UPSTREAM_REVISION}.dsc
 
-EXTRA_RULES = KAM.cf
+KAM_CHANNEL_FILE = debian/tree/etc/mail/spamassassin/channel.d/KAM_channel.conf
+KAM_KEYID = $(shell awk -F '=' '/^KEYID/ { print $$2 }' ${KAM_CHANNEL_FILE})
+
 
 SA_UPDATE_GPG_DIR = .sa-update-gpghome
 
@@ -53,12 +55,12 @@ ${NPKGNAME}.orig.tar.gz: upstream/
 	cp -a upstream ${NPKGNAME}
 	tar -czf $@ ${NPKGNAME}
 
-${NPKGNAME}: sa-updates ${EXTRA_RULES} ${NPKGNAME}.orig.tar.gz
+${NPKGNAME}: sa-updates kam-updates ${NPKGNAME}.orig.tar.gz
 	cp -a debian $@
 	mkdir -p $@/debian/tree/usr/share/spamassassin
 	rsync -av sa-updates/ $@/debian/tree/usr/share/spamassassin
 	mkdir -p $@/debian/tree/usr/share/spamassassin-extra
-	cp -a ${EXTRA_RULES} $@/debian/tree/usr/share/spamassassin-extra
+	rsync -av kam-updates/ $@/debian/tree/usr/share/spamassassin-extra
 
 ${OPKGNAME}.tar.gz:
 	rm -f $@*
@@ -67,15 +69,12 @@ ${OPKGNAME}.tar.gz:
 	mv $@.asc.tmp $@.asc
 	gpgv --keyring ./sa-releasekey.gpg $@.asc $@
 
-KAM.cf:
-	wget https://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf -O $@.tmp
-	mv KAM.cf.tmp $@
-
 ${SA_UPDATE_GPG_DIR}/.prepared: ${OPKGNAME}.tar.gz
 	rm -rf ${SA_UPDATE_GPG_DIR}
 	mkdir --mode=0700 -p ${SA_UPDATE_GPG_DIR}
 	tar --strip-components 2 -xzf ${OPKGNAME}.tar.gz ${OPKGNAME}/rules/sa-update-pubkey.txt
 	sa-update --gpghomedir ${SA_UPDATE_GPG_DIR} --import sa-update-pubkey.txt
+	sa-update --gpghomedir ${SA_UPDATE_GPG_DIR} --import ${KAM_CHANNEL_FILE}
 	rm sa-update-pubkey.txt
 	touch $@
 
@@ -85,13 +84,20 @@ update-sa: ${SA_UPDATE_GPG_DIR}/.prepared
 	sa-update --gpghomedir ${SA_UPDATE_GPG_DIR} --updatedir updates.tmp --channel updates.spamassassin.org
 	rsync -av --delete --exclude=local.cf --exclude=regression_tests.cf --exclude=*.txt  --exclude=MIRRORED.BY updates.tmp/updates_spamassassin_org/ sa-updates
 
+.PHONY: update-kam
+update-kam: ${SA_UPDATE_GPG_DIR}/.prepared
+	rm -rf kam-updates.tmp
+	sa-update --gpghomedir ${SA_UPDATE_GPG_DIR} --gpgkey ${KAM_KEYID} --updatedir kam-updates.tmp --channel kam.sa-channels.mcgrail.com
+	rm -rf kam-updates.tmp/kam_sa-channels_mcgrail_com/MIRRORED.BY
+	mv kam-updates.tmp kam-updates
+
 .PHONY: upload
 upload: ${DEBS}
 	tar cf - ${DEBS} | ssh -X repoman@repo.proxmox.com -- upload --product pmg --dist bullseye
 
 .PHONY: clean distclean
 clean:
-	rm -rf *~ debian/*~ *.deb proxmox-spamassassin_* updates.tmp ${SA_UPDATE_GPG_DIR} ${OPKGNAME}.tar.gz.*
+	rm -rf *~ debian/*~ *.deb proxmox-spamassassin_* updates.tmp kam-updates.tmp ${SA_UPDATE_GPG_DIR} ${OPKGNAME}.tar.gz.*
 
 distclean: clean
 	rm -rf  ${OPKGNAME}.*
-- 
2.30.2

[pmg-devel] [PATCH spamassassin 2/2] buildsys: update outdated docs on updating rulesets

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 Makefile | 19 +++----------------
 1 file changed, 3 insertions(+), 16 deletions(-)

diff --git a/Makefile b/Makefile
index a0f6bb9..c97bd95 100644
--- a/Makefile
+++ b/Makefile
@@ -2,22 +2,9 @@ include /usr/share/dpkg/pkg-info.mk
 
 # howto add rule updates:
 
-# edit paths ($srcdir) in changes.pl 
-# update changes.diff - verify changes!
-
-# dig  -t any 3.4.3.updates.spamassassin.org
-# wget http://spamassassin.kluge.net/updates/501214.tar.gz
-# wget http://spamassassin.kluge.net/updates/501214.tar.gz.asc
-# gpg --verify 501214.tar.gz.asc 501214.tar.gz
-#
-# or just use: make sa-updates.tgz (from the last stable version)
-# warning: start with an empty sa-updates.tgz, install the package,
-# the run sa-update to generate the updates
-#
-# Note: we also add KAM.cf from
-# http://www.pccc.com/downloads/SpamAssassin/contrib/KAM.cf
-#
-# edit debin/rules to apply updates
+# running `make update-sa` updates the core SA rules (in the sa-updates directory)
+# running `make update-kam` updates the KAM ruleset (in the kam-updates directory)
+
 
 OPKGNAME = Mail-SpamAssassin-${DEB_VERSION_UPSTREAM}
 NPKGNAME = proxmox-spamassassin_${DEB_VERSION_UPSTREAM}
-- 
2.30.2

[pmg-devel] [PATCH pmg-api 1/1] templates: adapt to new path for KAM rules in proxmox-spamassassin

[Stoiko Ivanov]

This changes the included KAM.cf to the one downloaded from the
kam-channel, which in turn includes all files contained in the
rule-set

This commit needs a versioned dependency bump on proxmox-spamassassin

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/templates/local.cf.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/templates/local.cf.in b/src/templates/local.cf.in
index f1106fe..da66538 100644
--- a/src/templates/local.cf.in
+++ b/src/templates/local.cf.in
@@ -25,5 +25,5 @@ whitelist_bounce_relays [% composed.wl_bounce_relays %]
 score ANY_BOUNCE_MESSAGE [% pmg.spam.bounce_score %]
 [% END %]
 
-include /usr/share/spamassassin-extra/KAM.cf
+include /usr/share/spamassassin-extra/kam_sa-channels_mcgrail_com.cf
 
-- 
2.30.2

[pmg-devel] applied-series: [PATCH spamassassin/pmg-api ] ship complete KAM ruleset

[Thomas Lamprecht]

Am 24/03/2023 um 13:46 schrieb Stoiko Ivanov:
> A few warnings were observed, when upgrading to proxmox-spamassassin 4.0
> and to a pmg-api > 7.2-5:
> * the sa-rules we ship in proxmox-spamassassin still recorded SA version
>   3.4.6 and spamassassin warned about that - this is fixed independently,
>   by pushing updates SA rules directly (after running our
>   regression_tests)
> * the KAM.cf file as of recently depends on the other files in the
>   complete ruleset as well:
>   ```
>   WARNING: HashBL: PCCC_HASHBL_FREEMAIL acl 'freemail' not defined
>   ```
> 
> This patchset addresses the latter issue by shipping the complete KAM
> ruleset with each proxmox-spamassassin package.
> The download using sa-update is based on the code in pmg-api, which does
> the same in pmg-daily
> 
> I'm sending only the actual code and doc changes - and, if approved, would
> push the actual rule-update and removal directly instead of sending
> multiple megabytes of spamassassin rule definitions (which would not pass
> our mail-gateway anyways :)

just push it to a staff repo or the like next time, then I can
pull all and could have bumped this now already ;-)

> 
> testing this is best done by applying the patches and running
> `make update-kam`
> 
> With using kam_sa-channels_mcgrail_com.cf to include all files in the
> ruleset the adaptation of the local.cf template (pmg-api patch 1/1) would
> need a versioned dependeny bump on proxmox-spamassassin
> 
> In my tests spamassassin (pmg-smtp-filter) quite happily ignored the
> missing /usr/share/spamassassin-extra/KAM.cf file - so a Breaks in the
> other direction should not be needed.
> 
> proxmox-spamassassin:
> Stoiko Ivanov (2):
>   kam-rules: ship full ruleset not only KAM.cf
>   buildsys: update outdated docs on updating rulesets
> 
>  Makefile | 41 +++++++++++++++++------------------------
>  1 file changed, 17 insertions(+), 24 deletions(-)
> 
> pmg-api:
> Stoiko Ivanov (1):
>   templates: adapt to new path for KAM rules in proxmox-spamassassin
> 
>  src/templates/local.cf.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 


applied, thanks!

Re: [pmg-devel] applied-series: [PATCH spamassassin/pmg-api ] ship complete KAM ruleset

[Stoiko Ivanov]

On Sun, 26 Mar 2023 18:11:41 +0200
Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:

> Am 24/03/2023 um 13:46 schrieb Stoiko Ivanov:
> > A few warnings were observed, when upgrading to proxmox-spamassassin 4.0
> > and to a pmg-api > 7.2-5:
> > * the sa-rules we ship in proxmox-spamassassin still recorded SA version
> >   3.4.6 and spamassassin warned about that - this is fixed independently,
> >   by pushing updates SA rules directly (after running our
> >   regression_tests)
> > * the KAM.cf file as of recently depends on the other files in the
> >   complete ruleset as well:
> >   ```
> >   WARNING: HashBL: PCCC_HASHBL_FREEMAIL acl 'freemail' not defined
> >   ```
> > 
> > This patchset addresses the latter issue by shipping the complete KAM
> > ruleset with each proxmox-spamassassin package.
> > The download using sa-update is based on the code in pmg-api, which does
> > the same in pmg-daily
> > 
> > I'm sending only the actual code and doc changes - and, if approved, would
> > push the actual rule-update and removal directly instead of sending
> > multiple megabytes of spamassassin rule definitions (which would not pass
> > our mail-gateway anyways :)  
> 
> just push it to a staff repo or the like next time, then I can
> pull all and could have bumped this now already ;-)

dunno why I did not do that ... - sorry for causing more work

pushed the ruleset I tested with.

Thanks for checking the patches!

> 
> > 
> > testing this is best done by applying the patches and running
> > `make update-kam`
> > 
> > With using kam_sa-channels_mcgrail_com.cf to include all files in the
> > ruleset the adaptation of the local.cf template (pmg-api patch 1/1) would
> > need a versioned dependeny bump on proxmox-spamassassin
> > 
> > In my tests spamassassin (pmg-smtp-filter) quite happily ignored the
> > missing /usr/share/spamassassin-extra/KAM.cf file - so a Breaks in the
> > other direction should not be needed.
> > 
> > proxmox-spamassassin:
> > Stoiko Ivanov (2):
> >   kam-rules: ship full ruleset not only KAM.cf
> >   buildsys: update outdated docs on updating rulesets
> > 
> >  Makefile | 41 +++++++++++++++++------------------------
> >  1 file changed, 17 insertions(+), 24 deletions(-)
> > 
> > pmg-api:
> > Stoiko Ivanov (1):
> >   templates: adapt to new path for KAM rules in proxmox-spamassassin
> > 
> >  src/templates/local.cf.in | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >   
> 
> 
> applied, thanks!