From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2964099AEF for ; Tue, 14 Nov 2023 14:00:17 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7FD9A1FE28 for ; Tue, 14 Nov 2023 14:00:15 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 14 Nov 2023 14:00:09 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1F10A428D7 for ; Tue, 14 Nov 2023 14:00:09 +0100 (CET) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Tue, 14 Nov 2023 13:59:08 +0100 Message-Id: <20231114130000.565122-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.162 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH v2 many 00/52] revamp notifications; smtp endpoints; system mail X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2023 13:00:17 -0000 Note: For simplicity, the series merges the three series' into one large one. Otherwise the cross-deps would have been really messy to manage on the list. ## Notification revamp: This series replaces notification filters and groups with notification matchers. Instead of having a per-notification event target/policy setting (at the moment stored in datacenter.cfg and jobs.cfg), this shifts the routing part into the matcher completely. Config example, I think this demonstrates the principle quite nicely: sendmail: default-target mailto-user root@pam matcher: fencing-for-node mode all # all match-directives have to match, default match-field exact:hostname=pve.example.com match-field exact:type=fencing target default-target --> Send all fencing notifications for a certain host to a certain target. Right now, there are three different match-directives: - match-field: exact/regex match for notification metadata fields - match-severity: match notification severities (info,notice,warning,error) - match-calender: match notification timestamp example: match-calendar mon..fri 8-12 The old target/policy based notification was already in the pvetest repository. Thus we take special care that there is no breakage when the notification system encounters old settings/configuration keys. It will clean them out/migrate them if possible. What I tested: - Made sure existing notifications continue to work (replication/fencing in a cluster setup, backups, system updates) - Made sure that the 'legacy' mailto parameter for backups also works - Tested the new UI for notification matchers - Tested whether old config keys for filters and groups break anything Followup work in the near future: - UI code for notification matcher config is a bit messy, I will send a cleanup-patch - main focus right now was to get it working - Mark 'mailto' in backup jobs as deprecated in UI - while also migrating automatically to the new system (create an endpoint/matcher when creating/updating a backup job) Changes in this series revision: - Added built-in default config. builtins can be freely modified (or disabled, if one has no use for them). If changed, the settings are stored in /etc/pve/notifications.cfg. To reset to defaults, one can simply delete this entry (from config, via API, via GUI). - We also return a 'origin' paramter from certain API calls, which tells us if it is a built-in, a user-created config entry or modified built-in entry. - Simplied permission system, only evaluate perms for /mapping/notifications. Also switch from plural to singular form. ## SMTP Endpoints: This patch series adds support for a new notification endpoint type, smtp. As the name suggests, this new endpoint allows PVE to talk to SMTP server directly, without using the system's MTA (postfix). On the Rust side, these patches add a new dependency to the `lettre` crate for SMTP communication. This crate was chosen as it is: - by far the most popular mailing crate for Rust - well maintained - has reasonable dependencies - has async support, enabling us to asyncify the proxmox-notify crate at some point, if needed Tested against: - the gmail SMTP server - the posteo SMTP server - our own webmail SMTP server This series also required updating the 'lettre' crate since one of lettre's deps was bumped to a new version by us. Changes in new, merged patch series: - Added origin/disabled params Changes since v3: - Rebased on top of the matcher-based notification revamp - Removed 'filter' setting from target configuration - Pulled in required patches from 'system mail forwarding' patch series Changes since v2: - Rebased proxmox-widget-toolkit onto the latest master to avoid any conflicts. Changes since v1: - Rebased on top of [1] - Added a mechanism for mails forwarded by `proxmox-mail-forward` These are forwarded inline as "message/rfc822" to avoid having to rewrite mail headers (otherwise, some SMTP relays might reject the mail, because the `From` header of the forwarded mail does not match the mail account) [1] https://lists.proxmox.com/pipermail/pve-devel/2023-August/058956.html [2] https://lists.proxmox.com/pipermail/pve-devel/2023-October/059299.html [3] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059818.html [4] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059843.html [5] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059872.html ## System mail forwarding The aim of this patch series is to adapt `proxmox-mail-forward` so that it forwards emails that were sent to the local root user through the `proxmox_notify` crate. A short summary of the status quo: Any mail that is sent to the local `root` user is forwarded by postfix to the `proxmox-mail-forward` binary, which receives the mail via STDIN. `proxmox-mail-forward` looks up the email address configured for the `root@pam` user in /etc/{proxmox-backup,pve}/user.cfg and then forwards the mail to this address by calling `sendmail` This patch series modifies `proxmox-mail-forward` in the following way: `proxmox-mail-forward` instantiates the configuration for `proxmox_notify` by reading `/etc/{proxmox-backup,pve}/notifications.cfg. The forwarding behavior is the following: - PVE installed: Use PVE's notifications.cfg - PBS installed: Use PBS's notifications.cfg if present. If not, use an empty configuration and add a default sendmail target and a matcher - this is needed because notifications are not yet integrated in PBS. In that way, the forwarding behavior is still the same as before on PBS (forward to root@pam via sendmail). - PVE/PBS co-installed: Use PVE's config *and* PBS's config. If PBS's notifications.cfg does not exist, a default sendmail target will *not* be added, to avoid forwarding the same mail twice. For co-installations we assume for now that PVE has a sensible matcher/target config for forwarded mails. Changelog: - Merged series: no changes - v1 -> v2: - Rebased - Apply the same fix for the PVE context as in [1] - v2 -> v3: - Rebased on top of matcher-based notification system: This simplifies proxmox-mail-forward by a great deal, since notification routing is moved into the matcher. This means proxmox-mail-forward does not need to read /etc/pve/datacenter.cfg any more to determine the target for the notification. [1] https://lists.proxmox.com/pipermail/pve-devel/2023-October/059294.html [2] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059818.html [3] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059872.html [4] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059894.html [5] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059899.html [6] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059900.html debcargo-conf: Lukas Wagner (2): cherry-pick chumsky 0.9.2 from debian unstable update lettre to 0.11.1 src/chumsky/debian/changelog | 5 ++ src/chumsky/debian/copyright | 39 +++++++++++ src/chumsky/debian/copyright.debcargo.hint | 51 ++++++++++++++ src/chumsky/debian/debcargo.toml | 2 + src/lettre/debian/changelog | 10 +++ .../debian/patches/downgrade_fastrand.patch | 13 ++++ .../debian/patches/downgrade_idna.patch | 13 ++++ src/lettre/debian/patches/downgrade_url.patch | 13 ++++ .../patches/remove_unused_features.patch | 69 ++++++++++--------- src/lettre/debian/patches/series | 4 +- .../patches/upgrade_quoted_printable.patch | 13 ---- 11 files changed, 185 insertions(+), 47 deletions(-) create mode 100644 src/chumsky/debian/changelog create mode 100644 src/chumsky/debian/copyright create mode 100644 src/chumsky/debian/copyright.debcargo.hint create mode 100644 src/chumsky/debian/debcargo.toml create mode 100644 src/lettre/debian/patches/downgrade_fastrand.patch create mode 100644 src/lettre/debian/patches/downgrade_idna.patch create mode 100644 src/lettre/debian/patches/downgrade_url.patch delete mode 100644 src/lettre/debian/patches/upgrade_quoted_printable.patch proxmox: Lukas Wagner (13): notify: introduce Error::Generic notify: factor out notification content into its own type notify: replace filters and groups with matcher-based system notify: add calendar matcher notify: matcher: introduce common trait for match directives notify: let a matcher always match if it has no matching directives sys: email: add `forward` notify: add mechanisms for email message forwarding notify: add PVE/PBS context notify: add 'smtp' endpoint notify: add api for smtp endpoints notify: add 'disable' parameter for matchers and targets. notify: add built-in config and 'origin' parameter Cargo.toml | 2 + proxmox-notify/Cargo.toml | 11 +- proxmox-notify/examples/render.rs | 4 +- proxmox-notify/src/api/common.rs | 6 +- proxmox-notify/src/api/filter.rs | 231 --------- proxmox-notify/src/api/gotify.rs | 22 +- proxmox-notify/src/api/group.rs | 259 ---------- proxmox-notify/src/api/matcher.rs | 265 ++++++++++ proxmox-notify/src/api/mod.rs | 146 ++---- proxmox-notify/src/api/sendmail.rs | 24 +- proxmox-notify/src/api/smtp.rs | 362 ++++++++++++++ proxmox-notify/src/config.rs | 57 ++- proxmox-notify/src/context.rs | 21 - proxmox-notify/src/context/common.rs | 27 + proxmox-notify/src/context/mod.rs | 43 ++ proxmox-notify/src/context/pbs.rs | 146 ++++++ proxmox-notify/src/context/pve.rs | 98 ++++ proxmox-notify/src/endpoints/common/mail.rs | 24 + proxmox-notify/src/endpoints/common/mod.rs | 2 + proxmox-notify/src/endpoints/gotify.rs | 53 +- proxmox-notify/src/endpoints/mod.rs | 4 + proxmox-notify/src/endpoints/sendmail.rs | 114 ++--- proxmox-notify/src/endpoints/smtp.rs | 263 ++++++++++ proxmox-notify/src/filter.rs | 193 +------ proxmox-notify/src/group.rs | 40 +- proxmox-notify/src/lib.rs | 526 +++++++++++--------- proxmox-notify/src/matcher.rs | 500 +++++++++++++++++++ proxmox-notify/src/renderer/mod.rs | 15 +- proxmox-notify/src/schema.rs | 11 +- proxmox-sys/src/email.rs | 52 +- 30 files changed, 2341 insertions(+), 1180 deletions(-) delete mode 100644 proxmox-notify/src/api/filter.rs delete mode 100644 proxmox-notify/src/api/group.rs create mode 100644 proxmox-notify/src/api/matcher.rs create mode 100644 proxmox-notify/src/api/smtp.rs delete mode 100644 proxmox-notify/src/context.rs create mode 100644 proxmox-notify/src/context/common.rs create mode 100644 proxmox-notify/src/context/mod.rs create mode 100644 proxmox-notify/src/context/pbs.rs create mode 100644 proxmox-notify/src/context/pve.rs create mode 100644 proxmox-notify/src/endpoints/common/mail.rs create mode 100644 proxmox-notify/src/endpoints/common/mod.rs create mode 100644 proxmox-notify/src/endpoints/smtp.rs create mode 100644 proxmox-notify/src/matcher.rs proxmox-perl-rs: Lukas Wagner (5): notify: adapt to new matcher-based notification routing notify: add bindings for smtp API calls pve-rs: notify: remove notify_context for PVE notify: add 'disable' parameter notify: support 'origin' paramter common/src/notify.rs | 288 +++++++++++++++++++++-------------- pve-rs/Cargo.toml | 2 +- pve-rs/src/lib.rs | 7 +- pve-rs/src/notify_context.rs | 117 -------------- 4 files changed, 180 insertions(+), 234 deletions(-) delete mode 100644 pve-rs/src/notify_context.rs pve-cluster: Lukas Wagner (1): notify: adapt to matcher based notification system src/PVE/Notify.pm | 101 +++++++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 54 deletions(-) pve-guest-common: Lukas Wagner (1): vzdump: deprecate mailto/mailnotification/notification-{target,policy} src/PVE/VZDump/Common.pm | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) pve-ha-manager: Lukas Wagner (1): env: switch to matcher-based notification system src/PVE/HA/Env/PVE2.pm | 10 ++-------- src/PVE/HA/NodeStatus.pm | 11 +++++++++-- 2 files changed, 11 insertions(+), 10 deletions(-) pve-manager: Lukas Wagner (12): api: notification: remove notification groups api: notification: add new matcher-based notification API ui: dc: remove unneeded notification events panel vzdump: adapt to new matcher based notification system api: apt: adapt to matcher-based notifications api: replication: adapt to matcher-based notification system test: fix vzdump notification test ui: vzdump: remove left-overs from target/policy based notifications ui: dc: config: show notification panel again notify: add API routes for smtp endpoints api: notification: add disable and origin params api: notification: simplify ACLs for notification PVE/API2/APT.pm | 27 +- PVE/API2/Cluster/Notifications.pm | 815 ++++++++++-------- PVE/API2/Replication.pm | 25 +- PVE/API2/VZDump.pm | 8 +- PVE/VZDump.pm | 40 +- test/vzdump_notification_test.pl | 6 +- www/manager6/Makefile | 4 - www/manager6/dc/Backup.js | 81 +- www/manager6/dc/Config.js | 28 +- www/manager6/dc/NotificationEvents.js | 276 ------ .../form/NotificationPolicySelector.js | 1 - www/manager6/window/Backup.js | 35 +- 12 files changed, 527 insertions(+), 819 deletions(-) delete mode 100644 www/manager6/dc/NotificationEvents.js proxmox-widget-toolkit: Lukas Wagner (10): notification ui: add target selector for matcher notification ui: remove filter setting for targets notification ui: remove notification groups notification ui: rename filter to matcher notification: matcher: add UI for matcher editing notification ui: unprotected mailto-root target noficiation: matcher edit: make 'field' an editable combobox panel: notification: add gui for SMTP endpoints notification ui: add enable checkbox for targets/matchers notification ui: add column for 'origin' src/Makefile | 6 +- src/Schema.js | 10 +- src/data/model/NotificationConfig.js | 6 +- src/form/NotificationFilterSelector.js | 58 -- src/panel/EmailRecipientPanel.js | 88 ++ src/panel/GotifyEditPanel.js | 39 +- src/panel/NotificationConfigView.js | 80 +- src/panel/NotificationGroupEditPanel.js | 183 ---- src/panel/SendmailEditPanel.js | 91 +- src/panel/SmtpEditPanel.js | 204 +++++ src/window/NotificationFilterEdit.js | 109 --- src/window/NotificationMatcherEdit.js | 1066 +++++++++++++++++++++++ 12 files changed, 1482 insertions(+), 458 deletions(-) delete mode 100644 src/form/NotificationFilterSelector.js create mode 100644 src/panel/EmailRecipientPanel.js delete mode 100644 src/panel/NotificationGroupEditPanel.js create mode 100644 src/panel/SmtpEditPanel.js delete mode 100644 src/window/NotificationFilterEdit.js create mode 100644 src/window/NotificationMatcherEdit.js pve-docs: Lukas Wagner (5): notifications: update docs to for matcher-based notifications notifications: document SMTP endpoints notifications: document 'comment' option for targets/matchers notifications: add documentation for system mail forwarding notifications: change to simplified ACL structure. notifications.adoc | 332 +++++++++++++++++++++++++++++++++------------ 1 file changed, 245 insertions(+), 87 deletions(-) proxmox-mail-forward: Lukas Wagner (2): feed forwarded mails into proxmox_notify update d/control Cargo.toml | 6 +- debian/control | 6 +- src/main.rs | 255 +++++++++++++++++++++++-------------------------- 3 files changed, 125 insertions(+), 142 deletions(-) Summary over all repositories: 77 files changed, 5150 insertions(+), 3040 deletions(-) -- murpp v0.4.0