[pmg-devel] [PATCH pmg-api v3 0/5] improve local mail injection and add smtputf8 support

[Stoiko Ivanov <s.ivanov@proxmox.com>]

Thanks to Dominik's feedback and testing this series so well we found one
issue:
* reinject_local_mail does encode the envelope-addresses (if they are not
purely ascii) - but the quarantine delivery passes the data already utf-8
encoded to it (since it's read this way from the database)

This would have been an issue previously as well - but since the
quarantine delivery never got smtputf8 support it was not noticed.

We discussed the rest of the feedback off-list - to summarize it:
* reinject_local_mail already does take the smtputf8 setting from pmg.conf
  into consideration (pmg-api patch 2/5)

changes v2 -> v3:
* patch 5/5 for pmg-api was added to properly decode database results
  before quarantine delivery or adding addresses to the user welcomelist
  or blocklists
* other patches are left as they were

original cover-letter for v2:
This series is the improved second part of:
https://lists.proxmox.com/pipermail/pmg-devel/2023-January/002257.html

and incorporates a few other improvments, which came up during implementation.

Changes from v1:
1 dropped the already applied and released stop-gap fix
2 added 'reinject_local_mail` helper to scan and check the mail, instead
  of checking with mail_needs_smtputf8 at all call-sites  (Thanks to Thomas
  for the suggestion!)
3 while add it went ahead and added smtputf8 as new option to the mail section
  of pmg.conf (reason for the docs and gui patch)
4 during testing noted the duplication of reinject_mail in
  deliver_quarantined_mail (with the latter not receiving any of the
  improvments for the former) leading to the refactoring in api-patch 3 and 4

point 4 is independent of the rest, but review should be easier after looking
through the other patches.

the docs-patch is needed (as build-depends), due to the new pmg.conf option

original cover-letter for v1:
This series addresses an issue some of our users reported in our
community forum with pmg-api_7.2-3:
* the decision if smtputf8 is needed is based on the envelope-addresses
  and the headers (if they contain non-ascii characters we use SMTPUTF8
* this breaks environments where SMTPUTF8 is disabled (mostly because
  some downstream servers do not support this), but mails still contain
  non-ascii data (while this is against the relevant rfc, which say that
  header-data must contain only ascii characters (and should be encoded
  with MIME-words otherwise), it is seemingly quite common in the wild)

one testmail I got from a user from the forum had the From header
correctly encoded, but added an X-DFrom header with the unencoded from.

The first patch simply drops the header-inspection and should enable
most of the reporters to receive mail again.

The second patch tries to address the smtputf8 issue a bit differently
than what we currently do
- For mails received via SMTP it simply sets SMTPUTF8 if the original
  postfix processes sent pmg-smtp-filter the mail with the flag, and
  does not set it otherwise
- For locally generated mail it detects if its needed by checking the
  envelope-addresses and the headers for non-ascii characters.

This should follow postfix own functioning quite closely:
https://www.postfix.org/SMTPUTF8_README.html

processing by pmg-smtp-filter should not change the need for the flag,
since we don't rewrite envelope-addresses, and modify filed does
mime-encode the resulting header.

Sending as two patches, since the first one would be good to get out
soon (as it's affecting a few setups), while the second one might
benefit from a bit more testing (I did some tests, which all looked
good, but might have overlooked some cases)

pmg-api:
Stoiko Ivanov (5):
  smtputf8: keep smtputf8 from incoming postfix, detect for local mail
  config: make smtputf8 configurable through the API
  reinject mail: improve error logging
  quarantine: use reinject_local_mail to deliver quarantined mail
  api: quarantine: decode addresses before delivery/userlisting

 src/PMG/API2/Quarantine.pm | 12 ++++---
 src/PMG/Config.pm          |  7 +++++
 src/PMG/Quarantine.pm      | 64 +++++++++-----------------------------
 src/PMG/RuleDB/Notify.pm   |  2 +-
 src/PMG/SMTP.pm            |  3 +-
 src/PMG/Utils.pm           | 53 +++++++++++++++++++++----------
 src/templates/main.cf.in   |  4 +++
 7 files changed, 73 insertions(+), 72 deletions(-)

pmg-docs:
Stoiko Ivanov (1):
  doc-generator: add new option smtputf8

 gen-pmg.conf.5-opts.pl | 1 +
 1 file changed, 1 insertion(+)

pmg-gui:
Stoiko Ivanov (1):
  mail proxy options: add smtputf8 checkbox

 js/MailProxyOptions.js | 3 +++
 1 file changed, 3 insertions(+)

-- 
2.30.2