From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 365129B2A0 for ; Wed, 24 May 2023 15:58:11 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0DDE51F7CD for ; Wed, 24 May 2023 15:57:39 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 24 May 2023 15:57:29 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 51C7D46E14 for ; Wed, 24 May 2023 15:57:29 +0200 (CEST) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Wed, 24 May 2023 15:56:07 +0200 Message-Id: <20230524135649.934881-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.162 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH v2 cluster/guest-common/manager/ha-manager/proxmox{, -perl-rs} 00/42] fix #4156: introduce new notification module X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2023 13:58:11 -0000 The purpose of this patch series is to overhaul the existing mail notification infrastructure in Proxmox VE. The series replaces calls to 'sendmail' with calls to a new, configurable notification module. The module was designed to support multiple notification endpoints, 'sendmail' using the system's sendmail command being the first one. As a proof of the extensibility of the current approach, the 'gotify' [1] plugin was also implemented in this series. Concepts: - Endpoints: An endpoint is responsible for sending a notification to some external entity, e.g. by calling `sendmail` to send a mail, or by performing REST API calls to a gotify server. Currently, there are two types of endpoints, `sendmail` and `gotify`. - Channels: Logically, channel can be thought of as a 'group of endpoints'. Each endpoint can be included in one or more channels. If one is using the notification API to send a notification, a channel has to be specified. The notification will then be forwarded to all endpoints included in that channel. Logically they decouple endpoints from notification senders - for instance, a backup job configuration would need to contain references to potentially multiple endpoints, or, a alternatively, always notify via *all* endpoints. The latter would potentially shift more configuration effort to filters, for instance if some backup jobs should only notify via *some* endpoints. I think the group/channel-based approach provides a relatively nice middle ground. - Filters: Every endpoint can also have a filter. Filters allow filtering notifications based on severity (info, notice, warning, error) or notification properties (metadata included in a notification, they are also the base for the template rendering). Filters allow AND/OR/NOT conditions and using sub-filters to allow arbitrarily complex filter structures. Conceptually, the new notification backend consists of three separate parts: - A new `proxmox-notify` crate, implemented in Rust. The crate contains the endpoint/filter/channel implementations, configuration parsing/writing (passed in/out as a string), template rendering, etc. - Glue code in `proxmox-perl-rs`, in order to be able to make calls to the `proxmox-notify` crate from Perl - A light-weight wrapper module `PVE::Notify`, implemented in Perl and living in `pve-manager` for now. It provides some helper functions and is responsible for reading/writing the configuration files, passing the configuration to the Rust part as a string. As of now, there were four different event sources: - Backup Jobs/One-off backups - APT update notifications - Replication failures - Node Fencing As a part of this patch series, all four were switched over to use the new `PVE::Notify` package to send notifications. For backup jobs, it is now possible to choose between 'E-Mail' or 'channel-based' notifications. This was done so that - we don't break existing configurations where the `mailto` option is set - there is a shortcut in case somebody really only ever cares about email notifications. Under the hood, both use the new notification backend. The 'E-Mail' option simply creates a temporary channel as well a temporary 'sendmail' endpoint. Since there is no way to configure endpoints/channels from the GUI yet, the control field for backup jobs where one can choose between "E-Mail" and "Channel" based notifications is disabled right now and always set to email. IMO it felt a bit weird being able to select a notification without being able to create/configure one from the GUI. APT/Replication/Node fencing do not yet have a way to configure a notification channel, so they use the same 'E-Mail' approach, sending mails to `root` via a temporary channel. Follow-up work (in no particular order): - Documentation (once the current approach has been approved) - Add a GUI/CLI for managing channels/endpoints, later also filters - Allow configuring a notification channel for APT/Repl/Fencing - In the future, the API might be changed/extended so that supports "registering" notifications. This allows us to a.) generate a list of all possible notification sources in the system b.) allows users to easily create filters for specific notification events. In my head, using the notification module could look like this then: # Global context my backup_failed_notification = PVE::Notify::register({ 'id' => 'backup-failed', 'severity' => 'error', 'properties' => ['host', 'vmlist', 'logs'], 'title' => '{{ host }}: Backup failed' 'body' => <<'EOF' A backup has failed for the following VMs: {{ vmlist }} {{ logs }} EOF }); # Later, to send the notification: PVE::Notify::send(backup_failed_notification->instantiate({ 'host' => 'earth', 'vmlist' => ... , 'logs' => ... , })); - proxmox-mail-forward could be integrated as well. This would feed e.g. zfs-zed events into our notification infrastructure. Special care must be taken to not create recursive notification loops (e.g. zed sends to root, forwarder uses notification module, a configured sendmail endpoint sends to root, forwarder uses module --> loop) - Maybe add some CLI so that admins can send notifications in scripts (an API endpoint callable via pvesh might be enough for a start). This should be done once everything is sufficiently stable (e.g. templating helpers, etc.) - Add more notification events - Add other endpoints, e.g. webhook, a generic SMTP, etc. - Integrate the new module into the other products [1] https://gotify.net/ [2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526 Changes from v1: - Some renaming: - PVE::Notification -> PVE::Notify - proxmox-notification -> proxmox-notify - Split configuration for gotify endpoints into a public part in `notifications.cfg` and a private part for the token in `priv/notifications.cfg` - Add template-based notification rendering (`proxmox`), including helpers for: - tables - pretty printed JSON - duration, timestamps - byte sizes - Add notification channels (repo `proxmox`) - Add API routes for channels, endpoints, filters (implementation in `proxmox-notify`, glue code in `proxmox-perl-rs` and handler in `pve-manager`) - Integrated new notification channels in backup jobs/one-off backups (repo `pve-manager`) - Replication/APT/Fencing use an 'anonymous' channel with a temporary sendmail endpoint, sending mails to `root` - Added new options for backup jobs - Reworked git history Versions of this patch series: v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html proxmox: Lukas Wagner (17): add `proxmox-human-byte` crate human-byte: move tests to their own sub-module add proxmox-notify crate notify: add debian packaging notify: preparation for the first endpoint plugin notify: preparation for the API notify: api: add API for sending notifications/testing endpoints notify: add notification channels notify: api: add API for channels notify: add sendmail plugin notify: api: add API for sendmail endpoints notify: add gotify endpoint notify: api: add API for gotify endpoints notify: add notification filter mechanism notify: api: add API for filters notify: add template rendering notify: add example for template rendering Cargo.toml | 4 + proxmox-human-byte/Cargo.toml | 15 + proxmox-human-byte/debian/changelog | 5 + proxmox-human-byte/debian/control | 43 ++ proxmox-human-byte/debian/copyright | 16 + proxmox-human-byte/debian/debcargo.toml | 7 + proxmox-human-byte/src/lib.rs | 363 +++++++++++++++ proxmox-notify/Cargo.toml | 28 ++ proxmox-notify/debian/changelog | 5 + proxmox-notify/debian/control | 31 ++ proxmox-notify/debian/copyright | 16 + proxmox-notify/debian/debcargo.toml | 7 + proxmox-notify/examples/render.rs | 63 +++ proxmox-notify/src/api/channel.rs | 253 ++++++++++ proxmox-notify/src/api/common.rs | 46 ++ proxmox-notify/src/api/filter.rs | 366 +++++++++++++++ proxmox-notify/src/api/gotify.rs | 294 ++++++++++++ proxmox-notify/src/api/mod.rs | 111 +++++ proxmox-notify/src/api/sendmail.rs | 263 +++++++++++ proxmox-notify/src/channel.rs | 53 +++ proxmox-notify/src/config.rs | 103 ++++ proxmox-notify/src/endpoints/gotify.rs | 139 ++++++ proxmox-notify/src/endpoints/mod.rs | 4 + proxmox-notify/src/endpoints/sendmail.rs | 106 +++++ proxmox-notify/src/filter.rs | 498 ++++++++++++++++++++ proxmox-notify/src/lib.rs | 567 +++++++++++++++++++++++ proxmox-notify/src/renderer/html.rs | 100 ++++ proxmox-notify/src/renderer/mod.rs | 359 ++++++++++++++ proxmox-notify/src/renderer/plaintext.rs | 141 ++++++ proxmox-notify/src/renderer/table.rs | 24 + proxmox-notify/src/schema.rs | 43 ++ 31 files changed, 4073 insertions(+) create mode 100644 proxmox-human-byte/Cargo.toml create mode 100644 proxmox-human-byte/debian/changelog create mode 100644 proxmox-human-byte/debian/control create mode 100644 proxmox-human-byte/debian/copyright create mode 100644 proxmox-human-byte/debian/debcargo.toml create mode 100644 proxmox-human-byte/src/lib.rs create mode 100644 proxmox-notify/Cargo.toml create mode 100644 proxmox-notify/debian/changelog create mode 100644 proxmox-notify/debian/control create mode 100644 proxmox-notify/debian/copyright create mode 100644 proxmox-notify/debian/debcargo.toml create mode 100644 proxmox-notify/examples/render.rs create mode 100644 proxmox-notify/src/api/channel.rs create mode 100644 proxmox-notify/src/api/common.rs create mode 100644 proxmox-notify/src/api/filter.rs create mode 100644 proxmox-notify/src/api/gotify.rs create mode 100644 proxmox-notify/src/api/mod.rs create mode 100644 proxmox-notify/src/api/sendmail.rs create mode 100644 proxmox-notify/src/channel.rs create mode 100644 proxmox-notify/src/config.rs create mode 100644 proxmox-notify/src/endpoints/gotify.rs create mode 100644 proxmox-notify/src/endpoints/mod.rs create mode 100644 proxmox-notify/src/endpoints/sendmail.rs create mode 100644 proxmox-notify/src/filter.rs create mode 100644 proxmox-notify/src/lib.rs create mode 100644 proxmox-notify/src/renderer/html.rs create mode 100644 proxmox-notify/src/renderer/mod.rs create mode 100644 proxmox-notify/src/renderer/plaintext.rs create mode 100644 proxmox-notify/src/renderer/table.rs create mode 100644 proxmox-notify/src/schema.rs proxmox-perl-rs: Lukas Wagner (7): log: set default log level to 'info', add product specific logging env var add PVE::RS::Notify module notify: add api for sending notifications/testing endpoints notify: add api for notification channels notify: add api for sendmail endpoints notify: add api for gotify endpoints notify: add api for notification filters common/src/logger.rs | 12 +- pmg-rs/src/lib.rs | 2 +- pve-rs/Cargo.toml | 1 + pve-rs/Makefile | 1 + pve-rs/src/lib.rs | 3 +- pve-rs/src/notify.rs | 411 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 426 insertions(+), 4 deletions(-) create mode 100644 pve-rs/src/notify.rs pve-cluster: Lukas Wagner (1): cluster files: add notifications.cfg src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-guest-common: Lukas Wagner (1): vzdump: add config options for new notification backend src/PVE/VZDump/Common.pm | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) pve-manager: Lukas Wagner (15): test: fix names of .PHONY targets add PVE::Notify module vzdump: send notifications via new notification module test: rename mail_test.pl to vzdump_notification_test.pl api: apt: send notification via new notification module api: replication: send notifications via new notification module ui: backup: allow to select notification channel for notifications ui: backup: adapt backup job details to new notification params ui: backup: allow to set notification-{channel,mode} for one-off backups api: prepare api handler module for notification config api: add api routes for notification channels api: add api routes for sendmail endpoints api: add api routes for gotify endpoints api: add api routes for notification filters ui: backup: disable notification mode selector for now PVE/API2/APT.pm | 73 +- PVE/API2/Cluster.pm | 7 + PVE/API2/Cluster/Makefile | 1 + PVE/API2/Cluster/Notifications.pm | 1262 +++++++++++++++++ PVE/API2/Replication.pm | 75 +- PVE/API2/VZDump.pm | 2 +- PVE/Makefile | 1 + PVE/Notify.pm | 84 ++ PVE/VZDump.pm | 323 +++-- test/Makefile | 16 +- ...il_test.pl => vzdump_notification_test.pl} | 36 +- www/manager6/Makefile | 4 +- www/manager6/dc/Backup.js | 78 +- www/manager6/dc/BackupJobDetail.js | 24 +- .../form/NotificationChannelSelector.js | 47 + www/manager6/form/NotificationModeSelector.js | 8 + ...ector.js => NotificationPolicySelector.js} | 1 + www/manager6/window/Backup.js | 35 +- 18 files changed, 1863 insertions(+), 214 deletions(-) create mode 100644 PVE/API2/Cluster/Notifications.pm create mode 100644 PVE/Notify.pm rename test/{mail_test.pl => vzdump_notification_test.pl} (62%) create mode 100644 www/manager6/form/NotificationChannelSelector.js create mode 100644 www/manager6/form/NotificationModeSelector.js rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%) pve-ha-manager: Lukas Wagner (1): manager: send notifications via new notification module src/PVE/HA/Env.pm | 6 ++--- src/PVE/HA/Env/PVE2.pm | 27 ++++++++++++++++++--- src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++---------------- src/PVE/HA/Sim/Env.pm | 10 ++++++-- 4 files changed, 66 insertions(+), 29 deletions(-) Summary over all repositories: 62 files changed, 6458 insertions(+), 249 deletions(-) Generated by murpp v0.3.0 -- 2.30.2