[pve-devel] [PATCH v2 cluster/guest-common/manager/ha-manager/proxmox{, -perl-rs} 00/42] fix #4156: introduce new notification module

[Lukas Wagner <l.wagner@proxmox.com>]

The purpose of this patch series is to overhaul the existing mail
notification infrastructure in Proxmox VE.
The series replaces calls to 'sendmail' with calls to a
new, configurable notification module. The module was designed to
support multiple notification endpoints, 'sendmail' using the system's
sendmail command being the first one. As a proof of the extensibility
of the current approach, the 'gotify' [1] plugin was also implemented
in this series.

Concepts:
  - Endpoints: 
    An endpoint is responsible for sending a notification to some external
    entity, e.g. by calling `sendmail` to send a mail, or by performing REST
    API calls to a gotify server.
    Currently, there are two types of endpoints, `sendmail` and 
    `gotify`.

  - Channels:
    Logically, channel can be thought of as a 'group of endpoints'. Each
    endpoint can be included in one or more channels. If one is using the 
    notification API to send a notification, a channel has to be specified. 
    The notification will then be forwarded to all endpoints included in that
    channel.
    Logically they decouple endpoints from notification senders - for instance,
    a backup job configuration would need to contain references to potentially
    multiple  endpoints, or, a alternatively, always notify via *all* endpoints. 
    The latter would potentially shift more configuration effort to filters, for
    instance if some backup jobs should only notify via *some* endpoints.
    I think the group/channel-based approach provides a relatively nice middle
    ground.

  - Filters:
    Every endpoint can also have a filter. Filters allow filtering
    notifications based on severity (info, notice, warning, error) or
    notification properties (metadata included in a notification, they are
    also the base for the template rendering).
    Filters allow AND/OR/NOT conditions and using sub-filters to allow
    arbitrarily complex filter structures.

Conceptually, the new notification backend consists of three separate parts:
  - A new `proxmox-notify` crate, implemented in Rust. The crate contains 
    the endpoint/filter/channel implementations, configuration parsing/writing
    (passed in/out as a string), template rendering, etc.

  - Glue code in `proxmox-perl-rs`, in order to be able to make calls to the 
    `proxmox-notify` crate from Perl

  - A light-weight wrapper module `PVE::Notify`, implemented in Perl and
    living in `pve-manager` for now. It provides some helper functions and 
    is responsible for reading/writing the configuration files, passing the 
    configuration to the Rust part as a string.

As of now, there were four different event sources:
  - Backup Jobs/One-off backups
  - APT update notifications
  - Replication failures
  - Node Fencing

As a part of this patch series, all four were switched over to use the new
`PVE::Notify` package to send notifications.
For backup jobs, it is now possible to choose between 'E-Mail' or 
'channel-based' notifications. 
This was done so that 
  - we don't break existing configurations where the `mailto` option is set
  - there is a shortcut in case somebody really only ever cares about email
    notifications.
Under the hood, both use the new notification backend. The 'E-Mail' option 
simply creates a temporary channel as well a temporary 'sendmail' endpoint.

Since there is no way to configure endpoints/channels from the GUI yet,
the control field for backup jobs where one can choose between
"E-Mail" and "Channel" based notifications is disabled right now and always
set to email. IMO it felt a bit weird being able to select a notification
without being able to create/configure one from the GUI.

APT/Replication/Node fencing do not yet have a way to configure a notification
channel, so they use the same 'E-Mail' approach, sending mails to `root` via
a temporary channel.

Follow-up work (in no particular order):
  - Documentation (once the current approach has been approved)
  - Add a GUI/CLI for managing channels/endpoints, later also filters
  - Allow configuring a notification channel for APT/Repl/Fencing
  - In the future, the API might be changed/extended so that supports
    "registering" notifications. This allows us to a.) generate a
    list of all possible notification sources in the system b.) allows
    users to easily create filters for specific notification events.
    In my head, using the notification module could look like this
    then:

    # Global context
    my backup_failed_notification = PVE::Notify::register({
      'id' => 'backup-failed',
      'severity' => 'error',
      'properties' => ['host', 'vmlist', 'logs'],
      'title' => '{{ host }}: Backup failed'
      'body' => <<'EOF'
    A backup has failed for the following VMs: {{ vmlist }}

    {{ logs }}
    EOF
    });

    # Later, to send the notification:
    PVE::Notify::send(backup_failed_notification->instantiate({
      'host' => 'earth',
      'vmlist' => ... ,
      'logs' => ... ,
    }));

  - proxmox-mail-forward could be integrated as well. This would feed
    e.g. zfs-zed events into our notification infrastructure. Special
    care must be taken to not create recursive notification loops
    (e.g. zed sends to root, forwarder uses notification module, a
    configured sendmail endpoint sends to root, forwarder uses module
    --> loop)

  - Maybe add some CLI so that admins can send notifications in
    scripts (an API endpoint callable via pvesh might be enough for a
    start). This should be done once everything is sufficiently stable 
    (e.g. templating helpers, etc.)

  - Add more notification events
  - Add other endpoints, e.g. webhook, a generic SMTP, etc.
  - Integrate the new module into the other products

[1] https://gotify.net/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526

Changes from v1:
  - Some renaming:
    - PVE::Notification -> PVE::Notify
    - proxmox-notification -> proxmox-notify
  - Split configuration for gotify endpoints into a public part in
    `notifications.cfg` and a private part for the token in 
    `priv/notifications.cfg`
  - Add template-based notification rendering (`proxmox`), including helpers 
    for: 
    - tables
    - pretty printed JSON
    - duration, timestamps
    - byte sizes
  - Add notification channels (repo `proxmox`)
  - Add API routes for channels, endpoints, filters (implementation in 
    `proxmox-notify`, glue code in `proxmox-perl-rs` and handler in 
    `pve-manager`)
  - Integrated new notification channels in backup jobs/one-off backups (repo 
    `pve-manager`)
  - Replication/APT/Fencing use an 'anonymous' channel with a temporary 
    sendmail endpoint, sending mails to `root`
  - Added new options for backup jobs
  - Reworked git history

Versions of this patch series:
v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html


proxmox:

Lukas Wagner (17):
  add `proxmox-human-byte` crate
  human-byte: move tests to their own sub-module
  add proxmox-notify crate
  notify: add debian packaging
  notify: preparation for the first endpoint plugin
  notify: preparation for the API
  notify: api: add API for sending notifications/testing endpoints
  notify: add notification channels
  notify: api: add API for channels
  notify: add sendmail plugin
  notify: api: add API for sendmail endpoints
  notify: add gotify endpoint
  notify: api: add API for gotify endpoints
  notify: add notification filter mechanism
  notify: api: add API for filters
  notify: add template rendering
  notify: add example for template rendering

 Cargo.toml                               |   4 +
 proxmox-human-byte/Cargo.toml            |  15 +
 proxmox-human-byte/debian/changelog      |   5 +
 proxmox-human-byte/debian/control        |  43 ++
 proxmox-human-byte/debian/copyright      |  16 +
 proxmox-human-byte/debian/debcargo.toml  |   7 +
 proxmox-human-byte/src/lib.rs            | 363 +++++++++++++++
 proxmox-notify/Cargo.toml                |  28 ++
 proxmox-notify/debian/changelog          |   5 +
 proxmox-notify/debian/control            |  31 ++
 proxmox-notify/debian/copyright          |  16 +
 proxmox-notify/debian/debcargo.toml      |   7 +
 proxmox-notify/examples/render.rs        |  63 +++
 proxmox-notify/src/api/channel.rs        | 253 ++++++++++
 proxmox-notify/src/api/common.rs         |  46 ++
 proxmox-notify/src/api/filter.rs         | 366 +++++++++++++++
 proxmox-notify/src/api/gotify.rs         | 294 ++++++++++++
 proxmox-notify/src/api/mod.rs            | 111 +++++
 proxmox-notify/src/api/sendmail.rs       | 263 +++++++++++
 proxmox-notify/src/channel.rs            |  53 +++
 proxmox-notify/src/config.rs             | 103 ++++
 proxmox-notify/src/endpoints/gotify.rs   | 139 ++++++
 proxmox-notify/src/endpoints/mod.rs      |   4 +
 proxmox-notify/src/endpoints/sendmail.rs | 106 +++++
 proxmox-notify/src/filter.rs             | 498 ++++++++++++++++++++
 proxmox-notify/src/lib.rs                | 567 +++++++++++++++++++++++
 proxmox-notify/src/renderer/html.rs      | 100 ++++
 proxmox-notify/src/renderer/mod.rs       | 359 ++++++++++++++
 proxmox-notify/src/renderer/plaintext.rs | 141 ++++++
 proxmox-notify/src/renderer/table.rs     |  24 +
 proxmox-notify/src/schema.rs             |  43 ++
 31 files changed, 4073 insertions(+)
 create mode 100644 proxmox-human-byte/Cargo.toml
 create mode 100644 proxmox-human-byte/debian/changelog
 create mode 100644 proxmox-human-byte/debian/control
 create mode 100644 proxmox-human-byte/debian/copyright
 create mode 100644 proxmox-human-byte/debian/debcargo.toml
 create mode 100644 proxmox-human-byte/src/lib.rs
 create mode 100644 proxmox-notify/Cargo.toml
 create mode 100644 proxmox-notify/debian/changelog
 create mode 100644 proxmox-notify/debian/control
 create mode 100644 proxmox-notify/debian/copyright
 create mode 100644 proxmox-notify/debian/debcargo.toml
 create mode 100644 proxmox-notify/examples/render.rs
 create mode 100644 proxmox-notify/src/api/channel.rs
 create mode 100644 proxmox-notify/src/api/common.rs
 create mode 100644 proxmox-notify/src/api/filter.rs
 create mode 100644 proxmox-notify/src/api/gotify.rs
 create mode 100644 proxmox-notify/src/api/mod.rs
 create mode 100644 proxmox-notify/src/api/sendmail.rs
 create mode 100644 proxmox-notify/src/channel.rs
 create mode 100644 proxmox-notify/src/config.rs
 create mode 100644 proxmox-notify/src/endpoints/gotify.rs
 create mode 100644 proxmox-notify/src/endpoints/mod.rs
 create mode 100644 proxmox-notify/src/endpoints/sendmail.rs
 create mode 100644 proxmox-notify/src/filter.rs
 create mode 100644 proxmox-notify/src/lib.rs
 create mode 100644 proxmox-notify/src/renderer/html.rs
 create mode 100644 proxmox-notify/src/renderer/mod.rs
 create mode 100644 proxmox-notify/src/renderer/plaintext.rs
 create mode 100644 proxmox-notify/src/renderer/table.rs
 create mode 100644 proxmox-notify/src/schema.rs


proxmox-perl-rs:

Lukas Wagner (7):
  log: set default log level to 'info', add product specific logging env
    var
  add PVE::RS::Notify module
  notify: add api for sending notifications/testing endpoints
  notify: add api for notification channels
  notify: add api for sendmail endpoints
  notify: add api for gotify endpoints
  notify: add api for notification filters

 common/src/logger.rs |  12 +-
 pmg-rs/src/lib.rs    |   2 +-
 pve-rs/Cargo.toml    |   1 +
 pve-rs/Makefile      |   1 +
 pve-rs/src/lib.rs    |   3 +-
 pve-rs/src/notify.rs | 411 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 426 insertions(+), 4 deletions(-)
 create mode 100644 pve-rs/src/notify.rs


pve-cluster:

Lukas Wagner (1):
  cluster files: add notifications.cfg

 src/PVE/Cluster.pm  | 2 ++
 src/pmxcfs/status.c | 2 ++
 2 files changed, 4 insertions(+)


pve-guest-common:

Lukas Wagner (1):
  vzdump: add config options for new notification backend

 src/PVE/VZDump/Common.pm | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)


pve-manager:

Lukas Wagner (15):
  test: fix names of .PHONY targets
  add PVE::Notify module
  vzdump: send notifications via new notification module
  test: rename mail_test.pl to vzdump_notification_test.pl
  api: apt: send notification via new notification module
  api: replication: send notifications via new notification module
  ui: backup: allow to select notification channel for notifications
  ui: backup: adapt backup job details to new notification params
  ui: backup: allow to set notification-{channel,mode} for one-off
    backups
  api: prepare api handler module for notification config
  api: add api routes for notification channels
  api: add api routes for sendmail endpoints
  api: add api routes for gotify endpoints
  api: add api routes for notification filters
  ui: backup: disable notification mode selector for now

 PVE/API2/APT.pm                               |   73 +-
 PVE/API2/Cluster.pm                           |    7 +
 PVE/API2/Cluster/Makefile                     |    1 +
 PVE/API2/Cluster/Notifications.pm             | 1262 +++++++++++++++++
 PVE/API2/Replication.pm                       |   75 +-
 PVE/API2/VZDump.pm                            |    2 +-
 PVE/Makefile                                  |    1 +
 PVE/Notify.pm                                 |   84 ++
 PVE/VZDump.pm                                 |  323 +++--
 test/Makefile                                 |   16 +-
 ...il_test.pl => vzdump_notification_test.pl} |   36 +-
 www/manager6/Makefile                         |    4 +-
 www/manager6/dc/Backup.js                     |   78 +-
 www/manager6/dc/BackupJobDetail.js            |   24 +-
 .../form/NotificationChannelSelector.js       |   47 +
 www/manager6/form/NotificationModeSelector.js |    8 +
 ...ector.js => NotificationPolicySelector.js} |    1 +
 www/manager6/window/Backup.js                 |   35 +-
 18 files changed, 1863 insertions(+), 214 deletions(-)
 create mode 100644 PVE/API2/Cluster/Notifications.pm
 create mode 100644 PVE/Notify.pm
 rename test/{mail_test.pl => vzdump_notification_test.pl} (62%)
 create mode 100644 www/manager6/form/NotificationChannelSelector.js
 create mode 100644 www/manager6/form/NotificationModeSelector.js
 rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%)


pve-ha-manager:

Lukas Wagner (1):
  manager: send notifications via new notification module

 src/PVE/HA/Env.pm        |  6 ++---
 src/PVE/HA/Env/PVE2.pm   | 27 ++++++++++++++++++---
 src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++----------------
 src/PVE/HA/Sim/Env.pm    | 10 ++++++--
 4 files changed, 66 insertions(+), 29 deletions(-)


Summary over all repositories:
  62 files changed, 6458 insertions(+), 249 deletions(-)

Generated by murpp v0.3.0
-- 
2.30.2