[pbs-devel] [PATCH many v3 00/42] integrate notification system

[Lukas Wagner <l.wagner@proxmox.com>]

These patches integrate the notification system which was introduced in Proxmox VE
in 8.1 into Proxmox Backup Server.

Some highlights/noteworthy details from the series:
  - notification template files are installed in /usr/share/proxmox-backup/templates/default
    and are rendered when a notification is sent
  - since sending notifications needs to be done from a privileged context
    (to read protected passwords/tokens from the priv config file),
    we queue notifications to be sent in /var/lib/proxmox-backup/notifications
    and periodically send any queued notifications via a worker in the
    privileged process
  - The API endpoint paths are prefixed with /config/notifications
  - API endpoints which read/modify notification system config require
    Sys.Audit (read) or Sys.Modify (modify/test endpoint) permissions
    on /system/notifications

  - tape-{backup,restore} settings and datastore options now have a
    'notification-mode' parameter, which allows to choose between the
    'legacy' behavior (sendmail to a selected user's email address) and
    the event-based notification system. If the parameter is not set, we
    default to the legacy behavior in order to keep existing behavior
    as is for now. For new datastores/tape backup jobs created
    interactively from the UI, we set the parameter to 'notification-system'
    and opt in into the new system by default. For the CLI/API we don't
    so that we don't cause problems with any automations/scripts -
    there I'd change the default to the new system with the next major release

Rough edges:
  - Datastore option view in UI could be improved. When 'notification-mode' is
    set to 'notification-system', we should indicate that 'mailto-user' and the
    other notification settings have no effect. Already did that in the edit
    window, but in the grid panel I did not find a quick way to do that.

Prerequisites:

  - This patch series requires the patches for 'proxmox' from
  https://lists.proxmox.com/pipermail/pve-devel/2024-April/063493.html
  to be applied first

  - The dependency for proxmox-widget-toolkit should be bumped to at least
  4.1.4 (we need: "utils: add mechanism to add and override translatable notification event
  descriptions in the product specific UIs")
  Also, while not strictly needed, the patches for widget-toolkit from
  'notification metadata matching improvments' [v6]
  make creating matchers much easier and *should* also be applied - the required API 
  endpoints were already implemented by this series.

Changes since v2:
  - consolidate verify/prune/sync/tape-job to job-id
  - add a note to docs that package update notifications
    are only sent by nodes with an active subscription

Changes since v1:
  - proxmox_notify: Change default-matcher to exclude 'prune success' notifications
  - proxmox_notify: Move sendmail/forward functions from proxmox_sys
  - proxmox_notify: Improve docs for API types
  - proxmox-widget-toolkit: allow to override default mail author
  - Add documentation, man pages, etc. (mostly copied from PVE, some
    stylistic touchups, some rephrasing)
  - Add additional matchable metadata fields, allowing to match on
    individual jobs:
    - verification-job
    - prune-job
    - sync-job
    - tape-job
  - Sort queued notifications by timestamp before sending
  - Fix missing hostname in package-update notification
  - Change default mail author in the UI from Proxmox VE to Proxmox Backup Server ($hostname)

proxmox:

Lukas Wagner (5):
  notify: expose `config` module
  notify: use std::sync::OnceCell instead of lazy_static!
  notify: pbs-context: exclude successful prunes in default matcher
  notify: endpoints: matcher: improve descriptions for API types
  notify: add getter for notification timestamp

 proxmox-notify/Cargo.toml                |  1 -
 proxmox-notify/src/config.rs             | 23 ++++++++++++-------
 proxmox-notify/src/context/pbs.rs        |  5 ++++-
 proxmox-notify/src/endpoints/gotify.rs   |  6 ++---
 proxmox-notify/src/endpoints/sendmail.rs | 15 ++++++++-----
 proxmox-notify/src/endpoints/smtp.rs     | 28 +++++++++++++++---------
 proxmox-notify/src/lib.rs                | 11 ++++++----
 proxmox-notify/src/matcher.rs            | 18 +++++++--------
 8 files changed, 66 insertions(+), 41 deletions(-)


proxmox-widget-toolkit:

Lukas Wagner (1):
  sendmail: smtp: allow to overide default mail author

 src/Schema.js                  | 9 +++++++++
 src/panel/SendmailEditPanel.js | 2 +-
 src/panel/SmtpEditPanel.js     | 2 +-
 src/window/EndpointEditBase.js | 1 +
 4 files changed, 12 insertions(+), 2 deletions(-)


proxmox-backup:

Lukas Wagner (36):
  pbs-config: add module for loading notification config
  server: rename email_notifications module to notifications
  notifications: allow sending notifications via proxmox_notify
  buildsys: install templates for test notifications
  pbs-config: acl: add /system/notifications as known ACL path
  api: add endpoints for querying/testing notification targets
  api: add endpoints for notification matchers
  api: add endpoints for sendmail targets
  api: add endpoints for smtp targets
  api: add endpoints for gotify targets
  api: add endpoints for querying known notification values/fields
  api-types: api: datatore: add notification-mode parameter
  api-types: api: tape: add notification-mode parameter
  server: notifications: send GC notifications via notification system
  server: notifications: send prune notifications via notification
    system
  server: notifications: send verify notifications via notification
    system
  server: notifications: send sync notifications via notification system
  server: notifications: send update notifications via notification
    system
  server: notifications: send acme notifications via notification system
  server: notifications: send tape notifications via notification system
  ui: add notification config panel
  ui: tape backup job: add selector for notification-mode
  ui: tape backup: add selector for 'notification-mode'
  ui: tape restore: add 'notification-mode' parameter
  ui: datastore options: add 'notification-mode' parameter
  ui: utils: add overrides for known notification metadata fields/values
  ui: datastore edit: make new stores use notification system by default
  ui: permissions paths: add /system/notifications to combobox
  proxmox-backup-manager: add CLI for notification targets
  proxmox-backup-manager: add CLI for notification matchers
  proxmox-backup-manager: add CLI for gotify endpoints
  proxmox-backup-manager: add CLI for sendmail endpoints
  proxmox-backup-manager: add CLI for SMTP endpoints
  docgen: generate synopsis for notifications{-priv,}.cfg
  docs: add documentation for notification system
  ui: util: override default mail author for sendmail/smtp targets

 Cargo.toml                                    |   3 +
 Makefile                                      |   5 +-
 debian/proxmox-backup-server.install          |  31 +
 docs/Makefile                                 |   6 +-
 docs/conf.py                                  |   2 +
 docs/config/notifications-priv/format.rst     |   1 +
 docs/config/notifications-priv/man5.rst       |  24 +
 docs/config/notifications/format.rst          |  28 +
 docs/config/notifications/man5.rst            |  24 +
 docs/configuration-files.rst                  |  30 +
 docs/index.rst                                |   1 +
 docs/notifications.rst                        | 213 +++++
 pbs-api-types/src/datastore.rs                |  22 +
 pbs-api-types/src/jobs.rs                     |   8 +-
 pbs-config/Cargo.toml                         |   1 +
 pbs-config/src/acl.rs                         |   3 +-
 pbs-config/src/lib.rs                         |   1 +
 pbs-config/src/notifications.rs               |  41 +
 src/api2/config/datastore.rs                  |   9 +
 src/api2/config/mod.rs                        |   2 +
 src/api2/config/notifications/gotify.rs       | 190 +++++
 src/api2/config/notifications/matchers.rs     | 170 ++++
 src/api2/config/notifications/mod.rs          | 210 +++++
 src/api2/config/notifications/sendmail.rs     | 178 ++++
 src/api2/config/notifications/smtp.rs         | 191 +++++
 src/api2/config/notifications/targets.rs      |  63 ++
 src/api2/config/tape_backup_job.rs            |   8 +
 src/api2/pull.rs                              |  10 +-
 src/api2/tape/backup.rs                       |  62 +-
 src/api2/tape/restore.rs                      |  46 +-
 src/bin/docgen.rs                             |   4 +
 src/bin/proxmox-backup-api.rs                 |  11 +
 src/bin/proxmox-backup-manager.rs             |   2 +
 src/bin/proxmox-backup-proxy.rs               |   1 +
 src/bin/proxmox_backup_manager/mod.rs         |   2 +
 .../notifications/gotify.rs                   |  93 +++
 .../notifications/matchers.rs                 |  93 +++
 .../notifications/mod.rs                      |  21 +
 .../notifications/sendmail.rs                 |  94 +++
 .../notifications/smtp.rs                     |  96 +++
 .../notifications/targets.rs                  |  51 ++
 src/server/email_notifications.rs             | 763 ------------------
 src/server/gc_job.rs                          |  10 +-
 src/server/mod.rs                             |   4 +-
 src/server/notifications.rs                   | 566 +++++++++++++
 src/server/verify_job.rs                      |  10 +-
 src/tape/drive/mod.rs                         |  22 +-
 src/tape/mod.rs                               |  27 +
 src/tape/pool_writer/mod.rs                   |  11 +-
 templates/Makefile                            |  41 +
 templates/default/acme-err-body.txt.hbs       |   7 +
 templates/default/acme-err-subject.txt.hbs    |   1 +
 templates/default/gc-err-body.txt.hbs         |   8 +
 templates/default/gc-err-subject.txt.hbs      |   1 +
 templates/default/gc-ok-body.txt.hbs          |  23 +
 templates/default/gc-ok-subject.txt.hbs       |   1 +
 .../default/package-updates-body.txt.hbs      |   8 +
 .../default/package-updates-subject.txt.hbs   |   1 +
 templates/default/prune-err-body.txt.hbs      |  10 +
 templates/default/prune-err-subject.txt.hbs   |   1 +
 templates/default/prune-ok-body.txt.hbs       |  10 +
 templates/default/prune-ok-subject.txt.hbs    |   1 +
 templates/default/sync-err-body.txt.hbs       |  14 +
 templates/default/sync-err-subject.txt.hbs    |   5 +
 templates/default/sync-ok-body.txt.hbs        |  14 +
 templates/default/sync-ok-subject.txt.hbs     |   5 +
 .../default/tape-backup-err-body.txt.hbs      |  26 +
 .../default/tape-backup-err-subject.txt.hbs   |   5 +
 templates/default/tape-backup-ok-body.txt.hbs |  27 +
 .../default/tape-backup-ok-subject.txt.hbs    |   5 +
 templates/default/tape-load-body.txt.hbs      |  15 +
 templates/default/tape-load-subject.txt.hbs   |   1 +
 templates/default/test-body.html.hbs          |   1 +
 templates/default/test-body.txt.hbs           |   1 +
 templates/default/test-subject.txt.hbs        |   1 +
 templates/default/verify-err-body.txt.hbs     |  14 +
 templates/default/verify-err-subject.txt.hbs  |   1 +
 templates/default/verify-ok-body.txt.hbs      |  10 +
 templates/default/verify-ok-subject.txt.hbs   |   1 +
 www/Makefile                                  |   1 +
 www/NavigationTree.js                         |   6 +
 www/OnlineHelpInfo.js                         |  24 +
 www/Utils.js                                  |  39 +
 www/config/NotificationConfigView.js          |  11 +
 www/datastore/OptionView.js                   |  15 +
 www/form/PermissionPathSelector.js            |   1 +
 www/tape/window/TapeBackup.js                 |  25 +
 www/tape/window/TapeBackupJob.js              |  24 +
 www/tape/window/TapeRestore.js                |  17 +
 www/window/DataStoreEdit.js                   |  13 +
 www/window/NotifyOptions.js                   |  38 +
 91 files changed, 3068 insertions(+), 869 deletions(-)
 create mode 100644 docs/config/notifications-priv/format.rst
 create mode 100644 docs/config/notifications-priv/man5.rst
 create mode 100644 docs/config/notifications/format.rst
 create mode 100644 docs/config/notifications/man5.rst
 create mode 100644 docs/notifications.rst
 create mode 100644 pbs-config/src/notifications.rs
 create mode 100644 src/api2/config/notifications/gotify.rs
 create mode 100644 src/api2/config/notifications/matchers.rs
 create mode 100644 src/api2/config/notifications/mod.rs
 create mode 100644 src/api2/config/notifications/sendmail.rs
 create mode 100644 src/api2/config/notifications/smtp.rs
 create mode 100644 src/api2/config/notifications/targets.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/gotify.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/matchers.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/mod.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/sendmail.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/smtp.rs
 create mode 100644 src/bin/proxmox_backup_manager/notifications/targets.rs
 delete mode 100644 src/server/email_notifications.rs
 create mode 100644 src/server/notifications.rs
 create mode 100644 templates/Makefile
 create mode 100644 templates/default/acme-err-body.txt.hbs
 create mode 100644 templates/default/acme-err-subject.txt.hbs
 create mode 100644 templates/default/gc-err-body.txt.hbs
 create mode 100644 templates/default/gc-err-subject.txt.hbs
 create mode 100644 templates/default/gc-ok-body.txt.hbs
 create mode 100644 templates/default/gc-ok-subject.txt.hbs
 create mode 100644 templates/default/package-updates-body.txt.hbs
 create mode 100644 templates/default/package-updates-subject.txt.hbs
 create mode 100644 templates/default/prune-err-body.txt.hbs
 create mode 100644 templates/default/prune-err-subject.txt.hbs
 create mode 100644 templates/default/prune-ok-body.txt.hbs
 create mode 100644 templates/default/prune-ok-subject.txt.hbs
 create mode 100644 templates/default/sync-err-body.txt.hbs
 create mode 100644 templates/default/sync-err-subject.txt.hbs
 create mode 100644 templates/default/sync-ok-body.txt.hbs
 create mode 100644 templates/default/sync-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-err-body.txt.hbs
 create mode 100644 templates/default/tape-backup-err-subject.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-body.txt.hbs
 create mode 100644 templates/default/tape-backup-ok-subject.txt.hbs
 create mode 100644 templates/default/tape-load-body.txt.hbs
 create mode 100644 templates/default/tape-load-subject.txt.hbs
 create mode 100644 templates/default/test-body.html.hbs
 create mode 100644 templates/default/test-body.txt.hbs
 create mode 100644 templates/default/test-subject.txt.hbs
 create mode 100644 templates/default/verify-err-body.txt.hbs
 create mode 100644 templates/default/verify-err-subject.txt.hbs
 create mode 100644 templates/default/verify-ok-body.txt.hbs
 create mode 100644 templates/default/verify-ok-subject.txt.hbs
 create mode 100644 www/config/NotificationConfigView.js


Summary over all repositories:
  103 files changed, 3146 insertions(+), 912 deletions(-)

-- 
Generated by git-murpp 0.7.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel