[pve-devel] [PATCH v4 many 00/69] fix #4156: introduce new notification system

[Lukas Wagner <l.wagner@proxmox.com>]

# Overview

The purpose of this patch series is to overhaul the existing mail
notification infrastructure in Proxmox VE.
The series replaces calls to 'sendmail' with calls to a
new, configurable notification module. The module was designed to
support multiple notification endpoints, 'sendmail' using the system's
sendmail command being the first one. As a proof of the extensibility
of the current approach, the 'gotify' [1] plugin was also implemented.
The patch series also includes groups. They allow to send a notification
to multiple endpoints at the same time. Furthermore, there are filters.
Endpoints and groups can configure filters to determine if a notification
should be sent. For now, filters can only be configured based on notification
severity.

A short summary of what is included in this patch series:
  - Sendmail endpoint plugin: uses the system's `sendmail` command 
    to send - well - mail. The sendmail plugin sends multi-part mails
    containing HTML as well as plain text.
  - Gotify endpoint plugin: sends a notification to a gotify server
  - Groups: As for any notification event one is only able to select a single
    target, groups can be created to notify multiply endpoints at the same time
  - Filters: Endpoints and groups can also have filtering: The filter 
    can match on the notification's metadata (only severity for now) to 
    determine if it will be sent or not. Filters can be easily extended in 
    the future to match on other structured metadata as well.
  - REST API for managing endpoints, groups and filters
  - Overhauled GUI for backup jobs/one-off backups - here the use can now 
    select a notification target
  - GUI for configuring the other notification events 
    (APT, replication, fencing) - here the user can configure *when* and
    *where* to send a notification
  - Notification rendering based on templates: 
    From a single template, the system can render notifications to either
    plain text or HTML.

# Configuration

For every single notification event (backup jobs, replication, APT, fencing),
it is now possible to configure
  a.) whether to send a notification at all
  b.) where to send the notification (target)

For backup jobs, this is configured in the respective backup job config in 
`jobs.cfg`, everything else is configured using an extended `notify`
configuration key in `datacenter.cfg`:

    root@pve:~# cat /etc/pve/datacenter.cfg 
    keyboard: en-us
    notify: target-fencing=foo,fencing=always,package-updates=auto,target-package-updates=bar
  
    root@pve:~# cat /etc/pve/jobs.cfg
    vzdump: backup-2e021835-88ea
      ...
      notes-template {{guestname}}
      notification-policy always
      notification-target mail-to-root
      ...

Targets (endpoint plugins or groups) and filters are configured in two new 
configuration files: `notifications.cfg` and `priv/notifications.cfg` - 
the latter is only readable by root and is used to store any sensitive
information for notification endpoint plugins (tokens, passwords, etc.).
These two new configuration files are opaque to the Perl code, it only 
interacts with it through the bindings to the Rust implementation.

    root@pve:~# cat /etc/pve/notifications.cfg
    filter: only-errors
      min-severity error

    sendmail: sm1
      mailto-user root@pam
      mailto user1@example.com
      mailto user2@example.com
      filter on-errors

    gotify: gt1
      filter only-errors
      server https://<IP:port>
  
    root@pve:~# cat /etc/pve/priv/notifications.cfg
    gotify: gt1
      token foobar

# Permissions

The new notification module is fully integrated in the permission system.
For every target or filter, there exists a corresponding ACL path
/mapping/notifications/<name>. A user must have the Mapping.Use permissions
on that path to send the notification/use the filter. Mapping.Modify and 
Mapping.Audit are needed for writing/reading a target's/filter's config.
For groups, the user must have the Mapping.Use permission for every 
single endpoint included in the group. If a group/endpoint has a filter,
the user must have the Mapping.Use permissions for that as well.

# Backwards Compatibility

Great care was taken so that existing setups work exactly as before.
Also, every new feature of this series is strictly opt-in, meaning that 
if no settings are changed, the system *should* behave exactly as before with
regards to sending notifications. The formatting of the notification emails 
has changed *slightly*. If any users relied on that, they might require some
changes (e.g. if they scraped the mails to get the results from backup jobs).

# A note to testers

Since this patch series covers a lot of different repos, it's a bit 
cumbersome to set it up correctly. Thus, I've built .deb files that you can
just install (staging repos should be configured). 
The packages are at `iso/packages/notification` on nasi. I'll try 
to rebuild them regularly with my changes rebased on the current master
branches. Holler at me if you think anything is broken or if the packages
do not install properly.

The changes are also available on my staff repos. Development happens on the
`notification` branches, the version posted here corresponds to the
`notification_v4` tag.

The recommend setup to test this is a 3 node cluster (fencing notifications)
with ZFS storage (for replication notifications). For update notifications,
you can use `pvesh create /node/<hostname>/apt/update --notify 1`.
You might need to clear/rm `/var/lib/pve-manager/pkgupdates` first.

# Follow-up work (in no particular order)

  - Revisit the template helpers. Once they are considered 'stable', we 
    could offer an API for users to send notifications themselves.

  - Maybe add some fancier styling to the HTML mails? 
    e.g. a light, but recognizable Proxmox-colored style? Unfortunately 
    HTML mails have no proper support for CSS and require inline styling

  - In the future, the API might be changed/extended so that supports
    "registering" notifications.
    This also allows us to specify a 'contract' on what properties
    are guaranteed to be included with a specific notification.
    Consequently, this allows us to:
      a.) generate a list of all possible notification sources in the system 
      b.) add more sophisticated filtering (e.g. match property for 
      c.) add other endpoints types where a fixed set of metadata fields makes
          sense (e.g. webhook)

    In my head, using the notification module could look like this
    then:

        # Global context
        my backup_failed_notification = PVE::Notify::register({
          'id' => 'backup-failed',
          'severity' => 'error',
          'properties' => ['host', 'vmlist', 'logs'],
          'title' => '{{ host }}: Backup failed'
          'body' => <<'EOF'
        A backup has failed for the following VMs: {{ vmlist }}

        {{ logs }}
        EOF
        });

        # Later, to send the notification:
        PVE::Notify::send(target, backup_failed_notification->instantiate({
          'host' => 'earth',
          'vmlist' => ... ,
          'logs' => ... ,
        }));

  - proxmox-mail-forward could be integrated as well. This would feed
    e.g. zfs-zed events into our notification infrastructure. Special
    care must be taken to not create recursive notification loops
    (e.g. zed sends to root, forwarder uses notification module, a
    configured sendmail endpoint sends to root, forwarder uses module
    --> loop)

  - Maybe add some CLI so that admins can send notifications in
    scripts (an API endpoint callable via pvesh might be enough for a
    start).
  - Add more notification events - the 'chattier' ones should probably be 
    disabled by default
  - Add other endpoints, e.g. webhook, a generic mail endpoint that 
    sends emails directly via SMTP, etc.
  - Integrate the new module into the other products
    - for PBS, this should be relatively straight-forward

# Version bumps
- proxmox-notify requires proxmox-section-config and proxmox-schema
  (first two patches in for the `proxmox` repo)
- proxmox-perl-rs requires proxmox-notify
- pve-cluster requires proxmox-perl-rs
- pve-manager requires:
    libpve-notify-perl (built from pve-cluster sources)
    libpve-cluster-perl/pve-cluster
    widget-toolkit
    guest-common
    common
    pve-doc-generator
- pve-ha-manager requires libpve-notify-perl

[1] https://gotify.net/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526

# Changelog

Changes since v3:
  - tried to document needed version bumps in cover letter
  - proxmox-perl-rs: Fixed an issue with unicode encoding by taking
    the config file contents as &[u8] and decoding instead of &str.
  - proxmox-section-config: derive Clone for SectionConfigData, so that 
    proxmox-notify's `Config` can derive Clone as well
  - proxmox-schema: add schema for comments
  - proxmox-notify: update d/copyright to new format
  - proxmox-notify: use `OnceCell` instead of `Mutex` for `Context`
  - minor stylistic code improvements, based on the feedback from the reviews
    (thanks!) - see changelog in commits for details

Changes since v2:
  - Transformed 'channels' into 'groups'. Allow to notify via a single endpoint
    or via a group (containing multiple endpoints)
  - Dropped some of the features for filters for now, such as sub-filters and 
    property-matchers. The property matching would require use to stabilize
    notification properties, which will be done later. Right now, the
    notifications only have the properties required for rendering the 
    notification template.
  - Groups can now also have filters
  - Check if a filter/group/endpoint is still used before deleting it
  - Ensure that filter/group/endpoint names are unique
  - Add new options to datacenter.cfg, allowing to configure the target
    and whether to notify at all for all notification events
  - Added new GUI panels, one for modifying the new settings in 
    datacenter.cfg, the other one for adding/modifying/deleting notification
    targets and filters
  - Integrate notification targets in the backup job UI
  - Full integration with the permission system
  - Allow users as mail recipients, looking up their email address in the user
    configuration file `user.cfg`.
  - Look up mail-from and author from datacenter.cfg, if it is not configured 
    for a sendmail endpoint
  - Support proxies for gotify endpoints. Proxy config is read from 
    datacenter.cfg
  - Move `PVE::Notify` to `pve-cluster`, building a new `libpve-notify-perl`
    package
  - The API now uses the new array type
  - Add always-available `mail-to-root` target, which is used as a fallback
    if no other target is configured for an event
  - Template rendering: Don't HTML-escape if rendering to plain text
  - many other minor changes
    
Changes since v1:
  - Some renaming:
    - PVE::Notification -> PVE::Notify
    - proxmox-notification -> proxmox-notify
  - Split configuration for gotify endpoints into a public part in
    `notifications.cfg` and a private part for the token in 
    `priv/notifications.cfg`
  - Add template-based notification rendering (`proxmox`), including helpers 
    for: 
    - tables
    - pretty printed JSON
    - duration, timestamps
    - byte sizes
  - Add notification channels (repo `proxmox`)
  - Add API routes for channels, endpoints, filters (implementation in 
    `proxmox-notify`, glue code in `proxmox-perl-rs` and handler in 
    `pve-manager`)
  - Integrated new notification channels in backup jobs/one-off backups (repo 
    `pve-manager`)
  - Replication/APT/Fencing use an 'anonymous' channel with a temporary 
    sendmail endpoint, sending mails to `root`
  - Added new options for backup jobs
  - Reworked git history

Versions of this patch series:
v3: https://lists.proxmox.com/pipermail/pve-devel/2023-July/058158.html
v2: https://lists.proxmox.com/pipermail/pve-devel/2023-May/056927.html
v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html



proxmox:

Lukas Wagner (25):
  section-config: derive Clone for SectionConfigData
  schema: add schema/format for comments
  add proxmox-notify crate
  notify: preparation for the first endpoint plugin
  notify: preparation for the API
  notify: api: add API for sending notifications/testing endpoints
  notify: add sendmail plugin
  notify: api: add API for sendmail endpoints
  notify: add gotify endpoint
  notify: api: add API for gotify endpoints
  notify: add notification groups
  notify: api: add API for groups
  notify: add notification filter mechanism
  notify: api: add API for filters
  notify: add template rendering
  notify: add example for template rendering
  notify: add context
  notify: sendmail: allow users as recipients
  notify: sendmail: query default author/mailfrom from context
  notify: gotify: add proxy support
  notify: api: allow to query entities referenced by filter/target
  notify: on deletion, check if a filter/endp. is still used by anything
  notify: ensure that filter/group/endpoint names are unique
  notify: additional logging when sending a notification
  notify: add debian packaging

 Cargo.toml                               |   3 +
 proxmox-notify/Cargo.toml                |  29 ++
 proxmox-notify/debian/changelog          |   5 +
 proxmox-notify/debian/control            | 112 +++++
 proxmox-notify/debian/copyright          |  18 +
 proxmox-notify/debian/debcargo.toml      |   7 +
 proxmox-notify/examples/render.rs        |  63 +++
 proxmox-notify/src/api/common.rs         |  55 +++
 proxmox-notify/src/api/filter.rs         | 227 +++++++++
 proxmox-notify/src/api/gotify.rs         | 292 +++++++++++
 proxmox-notify/src/api/group.rs          | 261 ++++++++++
 proxmox-notify/src/api/mod.rs            | 370 ++++++++++++++
 proxmox-notify/src/api/sendmail.rs       | 288 +++++++++++
 proxmox-notify/src/config.rs             | 103 ++++
 proxmox-notify/src/context.rs            |  19 +
 proxmox-notify/src/endpoints/gotify.rs   | 153 ++++++
 proxmox-notify/src/endpoints/mod.rs      |   4 +
 proxmox-notify/src/endpoints/sendmail.rs | 147 ++++++
 proxmox-notify/src/filter.rs             | 199 ++++++++
 proxmox-notify/src/group.rs              |  49 ++
 proxmox-notify/src/lib.rs                | 592 +++++++++++++++++++++++
 proxmox-notify/src/renderer/html.rs      | 100 ++++
 proxmox-notify/src/renderer/mod.rs       | 366 ++++++++++++++
 proxmox-notify/src/renderer/plaintext.rs | 141 ++++++
 proxmox-notify/src/renderer/table.rs     |  24 +
 proxmox-notify/src/schema.rs             |  27 ++
 proxmox-schema/src/api_types.rs          |   8 +
 proxmox-section-config/src/lib.rs        |   2 +-
 28 files changed, 3663 insertions(+), 1 deletion(-)
 create mode 100644 proxmox-notify/Cargo.toml
 create mode 100644 proxmox-notify/debian/changelog
 create mode 100644 proxmox-notify/debian/control
 create mode 100644 proxmox-notify/debian/copyright
 create mode 100644 proxmox-notify/debian/debcargo.toml
 create mode 100644 proxmox-notify/examples/render.rs
 create mode 100644 proxmox-notify/src/api/common.rs
 create mode 100644 proxmox-notify/src/api/filter.rs
 create mode 100644 proxmox-notify/src/api/gotify.rs
 create mode 100644 proxmox-notify/src/api/group.rs
 create mode 100644 proxmox-notify/src/api/mod.rs
 create mode 100644 proxmox-notify/src/api/sendmail.rs
 create mode 100644 proxmox-notify/src/config.rs
 create mode 100644 proxmox-notify/src/context.rs
 create mode 100644 proxmox-notify/src/endpoints/gotify.rs
 create mode 100644 proxmox-notify/src/endpoints/mod.rs
 create mode 100644 proxmox-notify/src/endpoints/sendmail.rs
 create mode 100644 proxmox-notify/src/filter.rs
 create mode 100644 proxmox-notify/src/group.rs
 create mode 100644 proxmox-notify/src/lib.rs
 create mode 100644 proxmox-notify/src/renderer/html.rs
 create mode 100644 proxmox-notify/src/renderer/mod.rs
 create mode 100644 proxmox-notify/src/renderer/plaintext.rs
 create mode 100644 proxmox-notify/src/renderer/table.rs
 create mode 100644 proxmox-notify/src/schema.rs


proxmox-perl-rs:

Lukas Wagner (10):
  add PVE::RS::Notify module
  notify: add api for sending notifications/testing endpoints
  notify: add api for notification groups
  notify: add api for sendmail endpoints
  notify: add api for gotify endpoints
  notify: add api for notification filters
  notify: sendmail: support the `mailto-user` parameter
  notify: implement context for getting default author/mailfrom
  notify: add context for getting http_proxy from datacenter.cfg
  notify: add wrapper for `get_referenced_entities`

 pve-rs/Cargo.toml    |   2 +
 pve-rs/Makefile      |   1 +
 pve-rs/src/lib.rs    |   4 +-
 pve-rs/src/notify.rs | 560 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 566 insertions(+), 1 deletion(-)
 create mode 100644 pve-rs/src/notify.rs


pve-cluster:

Lukas Wagner (3):
  cluster files: add notifications.cfg
  datacenter: add APT/fencing/replication notification configuration
  add libpve-notify-perl package

 debian/control                    |   9 ++
 debian/libpve-notify-perl.docs    |   1 +
 debian/libpve-notify-perl.install |   1 +
 src/PVE/Cluster.pm                |   2 +
 src/PVE/DataCenterConfig.pm       |  63 ++++++++++++-
 src/PVE/Makefile                  |   2 +-
 src/PVE/Notify.pm                 | 145 ++++++++++++++++++++++++++++++
 src/pmxcfs/status.c               |   2 +
 8 files changed, 222 insertions(+), 3 deletions(-)
 create mode 100644 debian/libpve-notify-perl.docs
 create mode 100644 debian/libpve-notify-perl.install
 create mode 100644 src/PVE/Notify.pm


pve-guest-common:

Lukas Wagner (1):
  vzdump: add config options for new notification backend

 src/PVE/VZDump/Common.pm | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)


pve-common:

Lukas Wagner (1):
  JSONSchema: increase maxLength of config-digest to 64

 src/PVE/JSONSchema.pm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)


pve-ha-manager:

Lukas Wagner (1):
  manager: send notifications via new notification module

 debian/control           |  2 ++
 src/PVE/HA/Env.pm        |  6 ++---
 src/PVE/HA/Env/PVE2.pm   | 21 +++++++++-------
 src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++----------------
 src/PVE/HA/Sim/Env.pm    | 10 ++++++--
 5 files changed, 57 insertions(+), 34 deletions(-)


pve-manager:

Lukas Wagner (22):
  test: fix names of .PHONY targets
  d/control: add dependency to `libpve-notify-perl`
  vzdump: send notifications via new notification module
  test: rename mail_test.pl to vzdump_notification_test.pl
  api: apt: send notification via new notification module
  api: replication: send notifications via new notification module
  api: prepare api handler module for notification config
  api: notification: add api routes for groups
  api: notification: add api routes for sendmail endpoints
  api: notification: add api routes for gotify endpoints
  api: notification: add api routes for filters
  api: notification: allow fetching notification targets
  api: notification: allow to test targets
  api: notification: disallow removing targets if they are used
  ui: backup: allow to select notification target for jobs
  ui: backup: adapt backup job details to new notification params
  ui: backup: allow to set notification-target for one-off backups
  ui: allow to configure notification event -> target mapping
  ui: add notification target configuration panel
  ui: perm path: add ACL paths for notifications, usb and pci mappings
  ui: perm path: increase width of the perm path selector combobox
  ui: dc: remove notify key from datacenter option view

 PVE/API2/APT.pm                               |   99 +-
 PVE/API2/Cluster.pm                           |    7 +
 PVE/API2/Cluster/Makefile                     |    1 +
 PVE/API2/Cluster/Notifications.pm             | 1356 +++++++++++++++++
 PVE/API2/Replication.pm                       |   63 +-
 PVE/API2/VZDump.pm                            |   10 +-
 PVE/VZDump.pm                                 |  335 ++--
 debian/control                                |    2 +
 test/Makefile                                 |   16 +-
 ...il_test.pl => vzdump_notification_test.pl} |   36 +-
 www/manager6/Makefile                         |    5 +-
 www/manager6/data/PermPathStore.js            |    3 +
 www/manager6/dc/Backup.js                     |   84 +-
 www/manager6/dc/BackupJobDetail.js            |   20 +-
 www/manager6/dc/Config.js                     |   28 +
 www/manager6/dc/NotificationEvents.js         |  277 ++++
 www/manager6/dc/OptionView.js                 |   20 -
 www/manager6/form/NotificationModeSelector.js |    8 +
 ...ector.js => NotificationPolicySelector.js} |    1 +
 .../form/NotificationTargetSelector.js        |   54 +
 www/manager6/form/PermPathSelector.js         |    1 +
 www/manager6/window/Backup.js                 |   35 +-
 22 files changed, 2207 insertions(+), 254 deletions(-)
 create mode 100644 PVE/API2/Cluster/Notifications.pm
 rename test/{mail_test.pl => vzdump_notification_test.pl} (62%)
 create mode 100644 www/manager6/dc/NotificationEvents.js
 create mode 100644 www/manager6/form/NotificationModeSelector.js
 rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%)
 create mode 100644 www/manager6/form/NotificationTargetSelector.js


proxmox-widget-toolkit:

Lukas Wagner (5):
  notification: add gui for sendmail notification endpoints
  notification: add gui for gotify notification endpoints
  notification: add gui for notification groups
  notification: allow to select filter for notification targets
  notification: add ui for managing notification filters

 src/Makefile                            |   8 +
 src/Schema.js                           |  18 ++
 src/data/model/NotificationConfig.js    |  17 ++
 src/form/NotificationFilterSelector.js  |  58 +++++
 src/panel/GotifyEditPanel.js            |  53 ++++
 src/panel/NotificationConfigView.js     | 319 ++++++++++++++++++++++++
 src/panel/NotificationGroupEditPanel.js | 183 ++++++++++++++
 src/panel/SendmailEditPanel.js          | 139 +++++++++++
 src/window/EndpointEditBase.js          |  57 +++++
 src/window/NotificationFilterEdit.js    | 109 ++++++++
 10 files changed, 961 insertions(+)
 create mode 100644 src/data/model/NotificationConfig.js
 create mode 100644 src/form/NotificationFilterSelector.js
 create mode 100644 src/panel/GotifyEditPanel.js
 create mode 100644 src/panel/NotificationConfigView.js
 create mode 100644 src/panel/NotificationGroupEditPanel.js
 create mode 100644 src/panel/SendmailEditPanel.js
 create mode 100644 src/window/EndpointEditBase.js
 create mode 100644 src/window/NotificationFilterEdit.js


pve-docs:

Lukas Wagner (1):
  add documentation for the new notification system

 notifications.adoc   | 159 +++++++++++++++++++++++++++++++++++++++++++
 pve-admin-guide.adoc |   2 +
 pve-gui.adoc         |   2 +
 vzdump.adoc          |   5 ++
 4 files changed, 168 insertions(+)
 create mode 100644 notifications.adoc


Summary over all repositories:
  83 files changed, 7868 insertions(+), 297 deletions(-)

-- 
murpp v0.4.0