From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 67F749595 for ; Thu, 3 Aug 2023 14:18:02 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 8AEE0280F for ; Thu, 3 Aug 2023 14:17:40 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 3 Aug 2023 14:17:33 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8C46142608 for ; Thu, 3 Aug 2023 14:17:33 +0200 (CEST) From: Lukas Wagner To: pve-devel@lists.proxmox.com Date: Thu, 3 Aug 2023 14:16:49 +0200 Message-Id: <20230803121719.519207-1-l.wagner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.101 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH v6 many 00/30] fix #4156: introduce new notification system X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2023 12:18:02 -0000 # Overview The purpose of this patch series is to overhaul the existing mail notification infrastructure in Proxmox VE. The series replaces calls to 'sendmail' with calls to a new, configurable notification module. The module was designed to support multiple notification endpoints, 'sendmail' using the system's sendmail command being the first one. As a proof of the extensibility of the current approach, the 'gotify' [1] plugin was also implemented. The patch series also includes groups. They allow to send a notification to multiple endpoints at the same time. Furthermore, there are filters. Endpoints and groups can configure filters to determine if a notification should be sent. For now, filters can only be configured based on notification severity. A short summary of what is included in this patch series: - Sendmail endpoint plugin: uses the system's `sendmail` command to send - well - mail. The sendmail plugin sends multi-part mails containing HTML as well as plain text. - Gotify endpoint plugin: sends a notification to a gotify server - Groups: As for any notification event one is only able to select a single target, groups can be created to notify multiply endpoints at the same time - Filters: Endpoints and groups can also have filtering: The filter can match on the notification's metadata (only severity for now) to determine if it will be sent or not. Filters can be easily extended in the future to match on other structured metadata as well. - REST API for managing endpoints, groups and filters - Overhauled GUI for backup jobs/one-off backups - here the use can now select a notification target - GUI for configuring the other notification events (APT, replication, fencing) - here the user can configure *when* and *where* to send a notification - Notification rendering based on templates: From a single template, the system can render notifications to either plain text or HTML. # Configuration For every single notification event (backup jobs, replication, APT, fencing), it is now possible to configure a.) whether to send a notification at all b.) where to send the notification (target) For backup jobs, this is configured in the respective backup job config in `jobs.cfg`, everything else is configured using an extended `notify` configuration key in `datacenter.cfg`: root@pve:~# cat /etc/pve/datacenter.cfg keyboard: en-us notify: target-fencing=foo,fencing=always,package-updates=auto,target-package-updates=bar root@pve:~# cat /etc/pve/jobs.cfg vzdump: backup-2e021835-88ea ... notes-template {{guestname}} notification-policy always notification-target mail-to-root ... Targets (endpoint plugins or groups) and filters are configured in two new configuration files: `notifications.cfg` and `priv/notifications.cfg` - the latter is only readable by root and is used to store any sensitive information for notification endpoint plugins (tokens, passwords, etc.). These two new configuration files are opaque to the Perl code, it only interacts with it through the bindings to the Rust implementation. root@pve:~# cat /etc/pve/notifications.cfg filter: only-errors min-severity error sendmail: sm1 mailto-user root@pam mailto user1@example.com mailto user2@example.com filter on-errors gotify: gt1 filter only-errors server https:// root@pve:~# cat /etc/pve/priv/notifications.cfg gotify: gt1 token foobar # Permissions The new notification module is fully integrated in the permission system. For every target or filter, there exists a corresponding ACL path /mapping/notifications/. A user must have the Mapping.Use permissions on that path to send the notification/use the filter. Mapping.Modify and Mapping.Audit are needed for writing/reading a target's/filter's config. For groups, the user must have the Mapping.Use permission for every single endpoint included in the group. If a group/endpoint has a filter, the user must have the Mapping.Use permissions for that as well. # Backwards Compatibility Great care was taken so that existing setups work exactly as before. Also, every new feature of this series is strictly opt-in, meaning that if no settings are changed, the system *should* behave exactly as before with regards to sending notifications. The formatting of the notification emails has changed *slightly*. If any users relied on that, they might require some changes (e.g. if they scraped the mails to get the results from backup jobs). # A note to testers Since this patch series covers a lot of different repos, it's a bit cumbersome to set it up correctly. Thus, I've built .deb files that you can just install (staging repos should be configured). The packages are at `iso/packages/notification` on nasi. I'll try to rebuild them regularly with my changes rebased on the current master branches. Holler at me if you think anything is broken or if the packages do not install properly. The recommend setup to test this is a 3 node cluster (fencing notifications) with ZFS storage (for replication notifications). For update notifications, you can use `pvesh create /node//apt/update --notify 1`. You might need to clear/rm `/var/lib/pve-manager/pkgupdates` first. # Follow-up work (in no particular order) - Revisit the template helpers. Once they are considered 'stable', we could offer an API for users to send notifications themselves. - Maybe add some fancier styling to the HTML mails? e.g. a light, but recognizable Proxmox-colored style? Unfortunately HTML mails have no proper support for CSS and require inline styling - In the future, the API might be changed/extended so that supports "registering" notifications. This also allows us to specify a 'contract' on what properties are guaranteed to be included with a specific notification. Consequently, this allows us to: a.) generate a list of all possible notification sources in the system b.) add more sophisticated filtering (e.g. match property for c.) add other endpoints types where a fixed set of metadata fields makes sense (e.g. webhook) In my head, using the notification module could look like this then: # Global context my backup_failed_notification = PVE::Notify::register({ 'id' => 'backup-failed', 'severity' => 'error', 'properties' => ['host', 'vmlist', 'logs'], 'title' => '{{ host }}: Backup failed' 'body' => <<'EOF' A backup has failed for the following VMs: {{ vmlist }} {{ logs }} EOF }); # Later, to send the notification: PVE::Notify::send(target, backup_failed_notification->instantiate({ 'host' => 'earth', 'vmlist' => ... , 'logs' => ... , })); - proxmox-mail-forward could be integrated as well. This would feed e.g. zfs-zed events into our notification infrastructure. Special care must be taken to not create recursive notification loops (e.g. zed sends to root, forwarder uses notification module, a configured sendmail endpoint sends to root, forwarder uses module --> loop) - Maybe add some CLI so that admins can send notifications in scripts (an API endpoint callable via pvesh might be enough for a start). - Add more notification events - the 'chattier' ones should probably be disabled by default - Add other endpoints, e.g. webhook, a generic mail endpoint that sends emails directly via SMTP, etc. - Integrate the new module into the other products - for PBS, this should be relatively straight-forward # Version bumps - pve-manager requires: widget-toolkit pve-doc-generator - pve-ha-manager requires libpve-notify-perl [1] https://gotify.net/ [2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526 # Changelog Changes since v5: - pve-manager: - Fix bug in permission check in the API for testing targets - Fix eslint warnings (missing trailing commas) - Reflowed some commit messages (70 chars) - widget-toolkit: - Increase width of the endpoint edit window in preparation for endpoints configuration panels with 2 columns Changes since v4: - pve-manager: - Minor code style improvements in pve-manager, based on Wolfgang's and Thomas' feedback - thanks! - Extended some commit messages - Change naming of anonymous, virtual groups/endpoints - Allow any user to see/test the 'mail-to-root' target. The target is exempt from permission checks to ensure backwards compat, so it makes sense to display it in the notification target panel. Changes since v3: - tried to document needed version bumps in cover letter - proxmox-perl-rs: Fixed an issue with unicode encoding by taking the config file contents as &[u8] and decoding instead of &str. - proxmox-section-config: derive Clone for SectionConfigData, so that proxmox-notify's `Config` can derive Clone as well - proxmox-schema: add schema for comments - proxmox-notify: update d/copyright to new format - proxmox-notify: use `OnceCell` instead of `Mutex` for `Context` - minor stylistic code improvements, based on the feedback from the reviews (thanks!) - see changelog in commits for details Changes since v2: - Transformed 'channels' into 'groups'. Allow to notify via a single endpoint or via a group (containing multiple endpoints) - Dropped some of the features for filters for now, such as sub-filters and property-matchers. The property matching would require use to stabilize notification properties, which will be done later. Right now, the notifications only have the properties required for rendering the notification template. - Groups can now also have filters - Check if a filter/group/endpoint is still used before deleting it - Ensure that filter/group/endpoint names are unique - Add new options to datacenter.cfg, allowing to configure the target and whether to notify at all for all notification events - Added new GUI panels, one for modifying the new settings in datacenter.cfg, the other one for adding/modifying/deleting notification targets and filters - Integrate notification targets in the backup job UI - Full integration with the permission system - Allow users as mail recipients, looking up their email address in the user configuration file `user.cfg`. - Look up mail-from and author from datacenter.cfg, if it is not configured for a sendmail endpoint - Support proxies for gotify endpoints. Proxy config is read from datacenter.cfg - Move `PVE::Notify` to `pve-cluster`, building a new `libpve-notify-perl` package - The API now uses the new array type - Add always-available `mail-to-root` target, which is used as a fallback if no other target is configured for an event - Template rendering: Don't HTML-escape if rendering to plain text - many other minor changes Changes since v1: - Some renaming: - PVE::Notification -> PVE::Notify - proxmox-notification -> proxmox-notify - Split configuration for gotify endpoints into a public part in `notifications.cfg` and a private part for the token in `priv/notifications.cfg` - Add template-based notification rendering (`proxmox`), including helpers for: - tables - pretty printed JSON - duration, timestamps - byte sizes - Add notification channels (repo `proxmox`) - Add API routes for channels, endpoints, filters (implementation in `proxmox-notify`, glue code in `proxmox-perl-rs` and handler in `pve-manager`) - Integrated new notification channels in backup jobs/one-off backups (repo `pve-manager`) - Replication/APT/Fencing use an 'anonymous' channel with a temporary sendmail endpoint, sending mails to `root` - Added new options for backup jobs - Reworked git history Versions of this patch series: v5: https://lists.proxmox.com/pipermail/pve-devel/2023-July/058467.html v4: https://lists.proxmox.com/pipermail/pve-devel/2023-July/058315.html v3: https://lists.proxmox.com/pipermail/pve-devel/2023-July/058158.html v2: https://lists.proxmox.com/pipermail/pve-devel/2023-May/056927.html v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html pve-ha-manager: Lukas Wagner (1): manager: send notifications via new notification module debian/control | 2 ++ src/PVE/HA/Env.pm | 6 ++--- src/PVE/HA/Env/PVE2.pm | 21 +++++++++------- src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++---------------- src/PVE/HA/Sim/Env.pm | 10 ++++++-- 5 files changed, 57 insertions(+), 34 deletions(-) pve-manager: Lukas Wagner (23): d/control: add dependency to `libpve-notify-perl` vzdump: send notifications via new notification module test: rename mail_test.pl to vzdump_notification_test.pl api: apt: send notification via new notification module api: replication: send notifications via new notification module api: prepare api handler module for notification config api: notification: add api routes for groups api: notification: add api routes for sendmail endpoints api: notification: add api routes for gotify endpoints api: notification: add api routes for filters api: notification: allow fetching notification targets api: notification: allow to test targets api: notification: disallow removing targets if they are used ui: backup: allow to select notification target for jobs ui: backup: adapt backup job details to new notification params ui: backup: allow to set notification-target for one-off backups ui: allow to configure notification event -> target mapping ui: add notification target configuration panel ui: perm path: add ACL paths for notifications, usb and pci mappings ui: perm path: increase width of the perm path selector combobox ui: dc: remove notify key from datacenter option view vzdump: use as a convention for virtual endpoints/groups api: notification: make the 'mail-to-root' target visible to any user PVE/API2/APT.pm | 99 +- PVE/API2/Cluster.pm | 7 + PVE/API2/Cluster/Makefile | 1 + PVE/API2/Cluster/Notifications.pm | 1332 +++++++++++++++++ PVE/API2/Replication.pm | 63 +- PVE/API2/VZDump.pm | 10 +- PVE/VZDump.pm | 335 +++-- debian/control | 2 + test/Makefile | 8 +- ...il_test.pl => vzdump_notification_test.pl} | 36 +- www/manager6/Makefile | 5 +- www/manager6/data/PermPathStore.js | 3 + www/manager6/dc/Backup.js | 84 +- www/manager6/dc/BackupJobDetail.js | 20 +- www/manager6/dc/Config.js | 28 + www/manager6/dc/NotificationEvents.js | 277 ++++ www/manager6/dc/OptionView.js | 20 - www/manager6/form/NotificationModeSelector.js | 8 + ...ector.js => NotificationPolicySelector.js} | 1 + .../form/NotificationTargetSelector.js | 54 + www/manager6/form/PermPathSelector.js | 1 + www/manager6/window/Backup.js | 35 +- 22 files changed, 2176 insertions(+), 253 deletions(-) create mode 100644 PVE/API2/Cluster/Notifications.pm rename test/{mail_test.pl => vzdump_notification_test.pl} (62%) create mode 100644 www/manager6/dc/NotificationEvents.js create mode 100644 www/manager6/form/NotificationModeSelector.js rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%) create mode 100644 www/manager6/form/NotificationTargetSelector.js proxmox-widget-toolkit: Lukas Wagner (5): notification: add gui for sendmail notification endpoints notification: add gui for gotify notification endpoints notification: add gui for notification groups notification: allow to select filter for notification targets notification: add ui for managing notification filters src/Makefile | 8 + src/Schema.js | 18 ++ src/data/model/NotificationConfig.js | 17 ++ src/form/NotificationFilterSelector.js | 58 +++++ src/panel/GotifyEditPanel.js | 53 ++++ src/panel/NotificationConfigView.js | 319 ++++++++++++++++++++++++ src/panel/NotificationGroupEditPanel.js | 183 ++++++++++++++ src/panel/SendmailEditPanel.js | 139 +++++++++++ src/window/EndpointEditBase.js | 57 +++++ src/window/NotificationFilterEdit.js | 109 ++++++++ 10 files changed, 961 insertions(+) create mode 100644 src/data/model/NotificationConfig.js create mode 100644 src/form/NotificationFilterSelector.js create mode 100644 src/panel/GotifyEditPanel.js create mode 100644 src/panel/NotificationConfigView.js create mode 100644 src/panel/NotificationGroupEditPanel.js create mode 100644 src/panel/SendmailEditPanel.js create mode 100644 src/window/EndpointEditBase.js create mode 100644 src/window/NotificationFilterEdit.js pve-docs: Lukas Wagner (1): add documentation for the new notification system notifications.adoc | 159 +++++++++++++++++++++++++++++++++++++++++++ pve-admin-guide.adoc | 2 + pve-gui.adoc | 2 + vzdump.adoc | 5 ++ 4 files changed, 168 insertions(+) create mode 100644 notifications.adoc Summary over all repositories: 41 files changed, 3362 insertions(+), 287 deletions(-) -- murpp v0.4.0